Add soitto/convoprobe plugin v0.0.17

[ShuntaroOkuma]

Plugin Submission Form

1. Metadata

• Plugin Author: soitto
• Plugin Name: convoprobe
• Repository URL: https://github.com/ShuntaroOkuma/convoprobe-plugin

2. Submission Type

• New plugin submission
• Version update for existing plugin

3. Description

ConvoProbe is a multi-turn chatbot QA testing tool. The Dify plugin lets users execute a saved ConvoProbe scenario against any Chat-type app in their workspace via Reverse Invocation, then post the resulting transcript back to ConvoProbe for LLM-as-Judge scoring. Useful for regression-testing chatbots, comparing prompts, and producing reviewable QA evidence.

The Tool plugin (run_scenario) takes a scenario_id and a target app-selector and returns the run id + transcript URL.

4. Checklist

• I have read and followed the Publish to Dify Marketplace guidelines
• I have read and comply with the Plugin Developer Agreement
• I confirm my plugin works properly on both Dify Community Edition and Cloud Version
• I confirm my plugin has been thoroughly tested for completeness and functionality
• My plugin brings new value to Dify

5. Documentation Checklist

• Step-by-step setup instructions
• Detailed usage instructions
• All required APIs and credentials are clearly listed
• Connection requirements and configuration details
• Link to the repository for the plugin source code

6. Privacy Protection Information

Data Collection

The plugin transfers:

• Scenario definitions from the ConvoProbe Backend to the plugin runtime (read direction)
• Chatbot prompts and responses, plus latency/error metadata and the resulting transcript, from the plugin to the ConvoProbe Backend (write direction) for LLM-as-Judge scoring and review

No personally identifiable information is collected by the plugin itself. The plugin does NOT transfer Dify workspace API keys, workspace member identities, or files/attachments outside the test scenario. Full details are in PRIVACY.md, bundled in the .difypkg.

Privacy Policy

• I confirm that I have prepared and included a privacy policy in my plugin package based on the Plugin Privacy Protection Guidelines

[crazywoola]

✅ LGTM

Decision: Approve

Local Check Results

Check	Status	Detail	Required action
PR content language	✅ Pass	PR title/body CJK ratio=0.0% (zh=0, en=1800, ignored_zh=0, allowed_zh<=0)	None.
Project structure	✅ Pass	All expected files present: manifest.yaml, README.md, PRIVACY.md. _assets/: yes.	None.
Manifest author	✅ Pass	author is valid.	None.
Icon validation	✅ Pass	icon exists: _assets/icon.svg	None.
Version check	✅ Pass	version 0.0.17 is available.	None.
README language	✅ Pass	README.md CJK ratio=0.0% (zh=0, en=1431, allowed_zh<=0)	None.
PRIVACY.md	✅ Pass	PRIVACY.md exists and is non-empty.	None.
Dependency install	✅ Pass	requirements installed successfully.	None.
dify_plugin version	✅ Pass	dify_plugin version 0.9.0 satisfies >= 0.5.0.	None.
Install test	✅ Pass	plugin install test passed.	None.
Packaging test	✅ Pass	packaging check passed.	None.

[crazywoola]

Thanks for the update. The PR is approved, but pre-check-plugin is currently failing again, so we cannot merge it yet.

Could you please check the CI logs, fix the failing check, and push an update? Once CI is green again, we can take another look and proceed with merge if everything still looks good.

[ShuntaroOkuma]

Thanks for the LGTM @crazywoola — the CI failure looks like a workflow-side regex bug, not a plugin issue. I reproduced the failure locally and verified the fix; details below.

Root cause

.github/workflows/pre-check-plugin.yaml detects the installed dify_plugin version via:

dify_version=$(pip list | grep -o 'dify_plugin\s\+[0-9.]\+' | awk '{print $2}' || echo "not_found")

• SDK ≤ 0.7.4 appears in pip list as dify_plugin (underscore) — the regex matches.

• SDK 0.9.0 normalizes to dify-plugin (hyphen) — the regex matches nothing. Because grep -o | awk exits 0 with empty output on a miss, dify_version becomes the empty string (not "not_found"), the if "$dify_version" = "not_found" check is skipped, and the script falls into the else-branch with an empty version string. Version('') > Version('0.0.1b64') raises packaging.version.InvalidVersion, the python -c call exits non-zero, and the script lands on the INSTALL_METHOD=aws_lambda fallback. SDK 0.9.0's DifyPluginEnv.INSTALL_METHOD enum only accepts local | remote | serverless, so pydantic raises:

  pydantic_core._pydantic_core.ValidationError: 1 validation error for DifyPluginEnv
  INSTALL_METHOD
    Input should be 'local', 'remote' or 'serverless' [type=enum, input_value='aws_lambda', input_type=str]
  

This matches our CI log exactly (job 77994468503, line ~1620), and contrasts with #2462 (dify_plugin==0.5.1, PASSED) where the same step logs Found dify_plugin version: 0.5.1 → serverless.

Local verification (before / after)

I extracted the workflow's bash block verbatim into a script and ran it against two clean venvs (one with dify_plugin==0.9.0, one with dify_plugin==0.7.4), then re-ran with the proposed regex change:

SDK in venv	pip list row	OLD workflow script	NEW workflow script
0.9.0	dify-plugin 0.9.0 (hyphen)	Found dify_plugin version: → Version('') → InvalidVersion → aws_lambda (matches CI failure)	Found dify_plugin version: 0.9.0 → serverless ✅
0.7.4	dify_plugin 0.7.4 (underscore)	Found dify_plugin version: 0.7.4 → serverless ✅	Found dify_plugin version: 0.7.4 → serverless ✅ (no regression)

So the failure is fully explained by the regex mismatch on SDK 0.9.0+ and the empty-output fall-through, and the proposed fix corrects 0.9.0 without regressing 0.7.4.

Proposed fix

Two changes in the same block:

1. Broaden the regex to accept both naming forms: dify[-_]plugin.
2. Explicitly fold the empty-output case into not_found (since grep -o | awk on a miss exits 0, the original || echo "not_found" never fired).

-              dify_version=$(pip list | grep -o 'dify_plugin\s\+[0-9.]\+' | awk '{print $2}' || echo "not_found")
+              dify_version=$(pip list | grep -oE 'dify[-_]plugin[[:space:]]+[0-9][0-9.a-zA-Z]*' | awk '{print $2}')
+              if [ -z "$dify_version" ]; then
+                dify_version="not_found"
+              fi

I'll open a separate PR against the workflow with this patch (commit ready, push pending a token-scope refresh on my side) and link it back here. In the meantime, would it be possible to re-run this check (or merge as-is, given the LGTM and the failure being workflow-side)? Happy to follow up either way.

[ShuntaroOkuma]

Follow-up: the workflow patch is up as #2463 (single-file diff, +5/-1). Once it lands, this CI run should pass on re-run.

[ShuntaroOkuma]

@crazywoola CI is green now after #2463. Could you merge when you have a moment? Thanks!