diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5b89401..0aab7bc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -6,8 +6,9 @@ name: ci # + marketplace.json/plugins/ sync + version bumps on release PRs # + post-merge version backstop on pushes to main # link-check: CI-02: the tools/check-links.sh sweep -# compile-check: EVAL-02 compile the baked JCSMP samples verbatim -# trigger-evals: lightweight skill-triggering check; self-skips green without a key +# jcsmp-sample-check: EVAL-02 compile the baked JCSMP samples verbatim +# python-sample-check: check the baked Python samples against the latest solace-pubsubplus +# trigger-evals: lightweight skill-triggering check; disabled until CI may hold an API key on: # Deliberately no `edited` type: running on title/body edits either re-runs # the paid eval matrix for a no-code change or, if jobs are gated, lets the @@ -50,7 +51,7 @@ jobs: # missing license are warnings only. Also validate the Claude # plugin/marketplace manifests. validate: - runs-on: ubuntu-latest + runs-on: ubuntu-26.04 timeout-minutes: 10 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 @@ -99,7 +100,7 @@ jobs: # CI-02: sweep the plugins/ tree for broken and soft-404 documentation links. link-check: - runs-on: ubuntu-latest + runs-on: ubuntu-26.04 timeout-minutes: 15 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 @@ -108,8 +109,8 @@ jobs: # EVAL-02: compile the baked JCSMP samples verbatim against the live-resolved # sol-jcsmp. Pure compile gate, no API budget and no credential. - compile-check: - runs-on: ubuntu-latest + jcsmp-sample-check: + runs-on: ubuntu-26.04 timeout-minutes: 25 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 @@ -118,8 +119,23 @@ jobs: distribution: temurin java-version: 11 cache: maven - - name: Compile baked samples (EVAL-02 hard gate) - run: ./plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/compile.sh + - name: Check baked JCSMP samples (EVAL-02 hard gate) + run: ./plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/check-samples.sh + + # Check the baked Python samples against the latest solace-pubsubplus from PyPI: + # py_compile, then import each sample and build its API objects against a + # resolvable host without a broker. Python has no compile step, so this is the + # Python counterpart of jcsmp-sample-check. No API budget and no credential. + python-sample-check: + runs-on: ubuntu-26.04 + timeout-minutes: 15 + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0 + with: + python-version: '3.11' + - name: Check baked Python samples + run: ./plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check-samples.sh # Lightweight skill-triggering check. For every case in each plugin's # evals/trigger-evals.json, run the prompt with only that plugin loaded and @@ -133,8 +149,12 @@ jobs: # run summary. The secret is legal in the job env but illegal in a job-level # if, so the guard reads HAS_KEY at step level and the real key is scoped to # the run step only. + # Disabled: no policy permits an API key on this public repo, so the job can + # only self-skip. Run the trigger evals locally instead; remove this `if` to + # re-enable the job once a key is allowed. trigger-evals: - runs-on: ubuntu-latest + if: false + runs-on: ubuntu-26.04 timeout-minutes: 120 # per model leg; 57 cases x 3 runs (~50 min observed) strategy: fail-fast: false # each model is an independent check diff --git a/.gitignore b/.gitignore index 7a26224..34d846d 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,5 @@ # GSD planning directory (local-only) .planning/ + +# Python bytecode cache +__pycache__/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..db331da --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,72 @@ +# CLAUDE.md + +Orientation for AI coding agents in this repository. For humans, the [README](README.md) covers installation, development, CI, and releases, and the [evals README](plugins/solace-messaging-skills/evals/README.md) covers the eval corpora and runners. + +This repository is a Claude Code plugin marketplace (`solace-agent-plugins`) of Agent Skills for building Solace messaging applications. The product is Markdown skill files, reference samples, and eval corpora. Nothing is compiled or packaged. + +## Where things live + +| Path | What it is | +|------|------------| +| `.claude-plugin/marketplace.json` | The marketplace. One entry per plugin directory. | +| `plugins//.claude-plugin/plugin.json` | Plugin manifest, including the plugin's version. | +| `plugins//skills//SKILL.md` | A skill. The frontmatter `description` alone decides when the skill fires. | +| `plugins/solace-messaging-skills/skills/solace-application-development/references/.md` | Per-API entry file (`jcsmp.md`, `python.md`). Mode files, baked samples, and scripts sit in the matching `references//` folder. | +| `plugins//evals/` | Trigger and output eval corpora. Maintainer tooling, not shipped functionality. | +| `tools/` | CI check scripts, the two eval runners, and `output-evals.schema.json`. | +| `.github/workflows/ci.yml` | The only CI workflow. | + +Nothing in the repository is generated or vendored. The Eclipse files under `references/jcsmp/evals/sample-check/` are IDE metadata for the JCSMP sample check; leave them alone. + +## Commands + +Run from the repository root. These match the README. + +```shell +gh skill publish --dry-run # skills against the Agent Skills spec (gh 2.90 or later) +claude plugin validate . --strict # marketplace and plugin manifests +./tools/check-plugin-versions.sh --sync-only # marketplace.json matches plugins/ +./tools/check-links.sh plugins # broken and soft-404 doc links +./plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/check-samples.sh # check the baked JCSMP samples (compile and config smoke) +./plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check-samples.sh # check the baked Python samples (python3 and PyPI access) +claude --plugin-dir plugins/solace-messaging-skills # load the plugin from the clone +``` + +The commands above need no broker and no API key. The evals need an exported `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`; an existing `claude` login does not work, because the runners use a scratch config directory. + +```shell +./tools/run-trigger-evals.sh --model haiku # then --model sonnet +./tools/run-output-evals.sh --model sonnet # then --model opus; a full leg takes about an hour +./tools/run-output-evals.sh --case # one output eval case +``` + +The trigger runner has no single-case mode; set `TRIGGER_EVAL_RUNS=1` for a quicker check. Only the output evals can touch a broker, and only when all four `OUTPUT_EVAL_BROKER_*` variables are exported (see the evals README). + +## Conventions + +- Never hardcode a library version in skill content. Generated builds resolve the latest release at generation time (Invariant 3 in the application-development `SKILL.md`). +- Link documentation to the live `docs.solace.com` `.md` URL. Do not copy doc content into a skill. Every link under `plugins/` must pass `check-links.sh`. +- A new skill needs cases in both `trigger-evals.json` (positive and near-miss negative prompts) and `output-evals.json` (validate the latter against `tools/output-evals.schema.json`). +- A new, removed, or renamed plugin directory needs the matching `marketplace.json` change in the same pull request. +- A new baked JCSMP sample must be added to the sample roster in `jcsmp/evals/sample-check/check-samples.sh`, or the sample check does not cover it. A new baked Python sample goes into the roster in `python/evals/sample-check/check-samples.sh`; `check_samples.py` picks it up from there. +- Do not put exact counts (of cases, skills, or graders) in READMEs or skill docs. They go stale. + +## Gotchas / do not + +- **Do not bump a `plugin.json` version in a feature pull request.** Versions change only when a release is prepared. Claude Code caches installed plugins by version, so the bump is what ships a release, and a stray bump confuses what users have. Changes under `plugins//evals/`, `tools/`, `.github/`, or the repository root never need a bump. +- **Both repositories are public.** Never write internal links, internal system names, credentials, API keys, or broker details into files, commit messages, branch names, or pull request text. +- **Run `gh repo set-default SolaceDev/agent-plugins` in a fresh clone.** With the `upstream` remote present, a bare `gh pr` command otherwise targets `SolaceProducts/agent-plugins`. +- **A green `trigger-evals` check asserts nothing.** The job is disabled (`if: false`) because the public repositories hold no API key. Run the evals locally and record the results in the pull request description. +- **A trigger eval leg can go red on one `must_pass` case from model variance.** If the prompt and skill descriptions did not change, re-run the full leg before you treat it as a regression. +- **Keep shell scripts on LF line endings.** `.gitattributes` enforces it. Bash rejects CRLF scripts, and an output eval grader compares the generated `verify.sh` byte for byte with the bundled copy in `references/jcsmp/scripts/`. +- **Keep the pins in `ci.yml`.** Every `uses:` step is pinned to a commit SHA with the tag in a trailing comment, and the Claude Code CLI version is pinned in `CLAUDE_CODE_VERSION`. Update a SHA and its comment together. +- **`check-links.sh` sweeps `plugins/` only.** It does not check links in the root README or this file. +- **A tool permission denial during an output eval is an infrastructure failure, not a skill failure.** Managed settings, hooks, or local command shims can deny a command. Fix the environment, then re-run. + +## Agent workflow + +- Open pull requests as drafts. Watch the checks with `gh pr checks --watch`, fix failures, and mark the pull request ready only when all checks are green. +- Print a failed CI job's log with `gh run view --log-failed`, then reproduce it with the matching command above. +- For a change to skill content or skill descriptions, run the trigger evals on Haiku and Sonnet and the output evals on Sonnet and Opus, and record the results in the pull request description. +- Do not pick reviewers. Ask the user who to tag. +- Never merge a release pull request. A human reviewer merges it. diff --git a/README.md b/README.md index c94a5b6..38e9548 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # AI Agent Plugins by Solace -A collection of Solace AI agent plugins that package skills for building event-driven messaging applications. These skills help developers using AI coding assistants (such as Claude Code) design and implement Solace messaging applications that follow Solace best practices out of the gate. The solace-application-development skill designs and generates Solace messaging applications (publishing to topics, consuming from queues, and guaranteed messaging), starting with the Solace Messaging API for JCSMP. A companion solace-topic-best-practices skill looks up the canonical Topic Architecture Best Practices page online and applies it to topic-hierarchy decisions. +A collection of Solace AI agent plugins that package skills for building event-driven messaging applications. These skills help developers using AI coding assistants (such as Claude Code) design and implement Solace messaging applications that follow Solace best practices out of the gate. The solace-application-development skill designs and generates Solace messaging applications (publishing to topics, consuming from queues, and guaranteed messaging), with the Solace JCSMP API and the Solace Python API. A companion solace-topic-best-practices skill looks up the canonical Topic Architecture Best Practices page online and applies it to topic-hierarchy decisions. > [!WARNING] > These skills are powered by language models, which are nondeterministic and may change over time. Generated code can vary between runs and may be incorrect, insecure, incomplete, or unsuitable for your environment. You are responsible for reviewing, testing, and validating all generated code before use, especially before deploying to production. @@ -42,7 +42,7 @@ This repository includes the following skills: | Skill | Description | |-------|-------------| -| **solace-application-development** | Routes Solace messaging design and implementation requests to grounded Solace documentation. Full guidance for messaging pattern selection, runnable code generation, and message round trip verification arrives in upcoming releases. | +| **solace-application-development** | Front door for Solace API development (JCSMP and Python). Routes each request to the right API, helps choose a messaging pattern, and generates a runnable application from reference samples, grounded in canonical Solace documentation. | | **solace-topic-best-practices** | Navigation-only lookup skill that reads the canonical Topic Architecture Best Practices page online and applies it to topic-hierarchy decisions, without generating application code. | | **solace-messaging-feedback** | Formats the current session into copy-paste-ready feedback about the skills, routed by support-contract status: contract holders get an email to Solace Support for bugs and a Solace Ideas portal submission for feature ideas, users without a contract get a Solace Community post for both. Formatter only: it produces the text for you to review and send or post, and never sends anything itself. | @@ -62,8 +62,9 @@ agent-plugins/ │ │ └── SKILL.md # Navigation-only manifest; reads the live docs page │ └── solace-messaging-feedback/ # Feedback formatter skill │ └── SKILL.md # Formats and routes session feedback (Support, Community, or Ideas portal) -├── tools/ # Re-runnable scripts (check-links, run-trigger-evals, run-output-evals) +├── tools/ # CI check scripts and the trigger and output eval runners ├── README.md # This file +├── CLAUDE.md # Orientation for AI coding agents that work in this repository └── .claude-plugin/ # Claude marketplace definition (marketplace.json) ``` @@ -73,6 +74,10 @@ Each plugin ships a trigger eval corpus under `plugins//evals/` that che Each plugin also ships an output eval corpus in the same directory that checks a skill's output honors the skill contract (design before code, the door question, scrubbed feedback drafts, grounded answers). Output evals run locally only. See the [output evals section](plugins/solace-messaging-skills/evals/README.md#output-evals) of the same README for the corpus format, the graders, and how to run them. +## Contributing + +This repository does not accept external pull requests or GitHub issues. If you have a Solace support contract, report bugs to Solace Support and suggest features on the [Solace Ideas portal](https://ideas.solace.com/ideas). Otherwise, post both on the [Solace Community](https://community.solace.com/). The `solace-messaging-feedback` skill formats a session into a ready-to-send report for each channel. + ## License This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details. diff --git a/plugins/solace-messaging-skills/.claude-plugin/plugin.json b/plugins/solace-messaging-skills/.claude-plugin/plugin.json index d41a724..bdc384b 100644 --- a/plugins/solace-messaging-skills/.claude-plugin/plugin.json +++ b/plugins/solace-messaging-skills/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "solace-messaging-skills", "description": "Skills for building applications on the Solace platform. Includes solace-application-development (designs and generates Solace messaging code grounded in canonical Solace docs), solace-topic-best-practices (looks up the canonical Topic Architecture Best Practices page online for topic-hierarchy questions), and solace-messaging-feedback (formats a session into copy-paste-ready feedback routed to the Solace Ideas portal, Solace Support, or the Solace Community)", - "version": "0.1.1", + "version": "0.2.0", "license": "Apache-2.0", "author": { "name": "Solace" diff --git a/plugins/solace-messaging-skills/evals/README.md b/plugins/solace-messaging-skills/evals/README.md index e8b4661..9d4c61e 100644 --- a/plugins/solace-messaging-skills/evals/README.md +++ b/plugins/solace-messaging-skills/evals/README.md @@ -8,23 +8,23 @@ To run the evals manually, you need the `claude` CLI and `jq` on your PATH, plus ```shell export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN -./tools/run-trigger-evals.sh # defaults to claude-haiku-4-5 -./tools/run-trigger-evals.sh --model claude-sonnet-5 +./tools/run-trigger-evals.sh # defaults to a Haiku model +./tools/run-trigger-evals.sh --model sonnet # the latest Sonnet ``` Each prompt runs three times and the verdict is the majority result. Set `TRIGGER_EVAL_RUNS=1` for a quicker, cheaper local check, and `TRIGGER_EVAL_MODEL` to change the default model. A run passes when at least 90% of cases match their expectation, and any infrastructure failure (such as a missing credential, a malformed corpus, or zero discovered cases) fails the run regardless of the rate. A case may also set `"must_pass": true`, and any must-pass failure fails the run regardless of the pooled rate. Note that a negative case asserts only that the named skill stays silent, so another skill in the plugin may legitimately fire on the same prompt. ## Continuous integration -In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs this same script on every pull request as a two model matrix (`claude-haiku-4-5` and `claude-sonnet-5`), each model an independent check, and self-skips green when the `ANTHROPIC_API_KEY` secret is absent. +The `trigger-evals` job in `.github/workflows/ci.yml` is disabled (`if: false`), because no policy permits an API key on this public repository. Run the trigger evals locally as shown above. To re-enable the job once a key is allowed, remove the `if`. It then runs this same script on every pull request as a two model matrix (Haiku and Sonnet), each model an independent check, and self-skips green when the `ANTHROPIC_API_KEY` secret is absent. # Output evals Trigger evals ask whether the right skill fires. Output evals ask the next question: does the skill's output honor the skill contract? The corpus at `plugins//evals/output-evals.json` holds the cases for the plugin's skills. Each case carries an `id`, the target `skill`, an ordered `user_turns` array of scripted user messages (multi-turn cases continue one session via `--resume`), an optional `max_agent_turns` budget (the model's replies and tool calls allowed per user turn), an optional `must_pass` flag, and a `graders` array. A case passes only when the target skill fired, every grader passed, and no user turn hit the `max_agent_turns` cap. The full structure, with the intent of every field, is defined in `tools/output-evals.schema.json` (JSON Schema, draft 2020-12). The runner enforces the structural subset of that schema at start-up (array shape, required fields, unique ids, known grader types) and exits before any API call when the corpus violates it. Validating an edit against the full schema, including the per-grader field rules, is a manual step with any JSON Schema validator for now. -For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The compile case also carries an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. +For the application-development skill, the cases cover these contract points end to end: the broker-access question, design mode entered rather than skipped, `solace-design.md` saved on request by the end of design mode, the three-way door question (Quickstart, Solace Suggested, Custom), the Solace Suggested secure connection and HA failover question, the Quickstart posture (plaintext capable, single project, no HA question), and the AI-generated notice at the top of every generated file. The Python cases check the same points for the Solace Python API, and one more: a request that must be persisted routes to Guaranteed Pub/Sub, because the Python API has no guaranteed request-reply. The JCSMP and Python Quickstart implement cases also carry an opt-in live round trip: when the four `OUTPUT_EVAL_BROKER_*` variables are set, the runner executes the generated project's own `verify.sh` against that broker. Without them the grader skips, the run says so up front and in its summary, and no case ever requires broker credentials. -Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `java_disclaimer`, `compile`, `maven_release_match`, `live_verify`) check the transcript and the generated files mechanically, including a real `mvn compile` of the generated project, a match of the generated pom against the live sol-jcsmp `` on Maven Central, and, when a broker is configured, a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. +Two grader families exist. Deterministic graders (`assistant_grep`, `tool_use`, `file_exists`, `file_grep`, `disclaimer`, `compile`, `release_match`, `live_verify`) check the transcript and the generated files mechanically. The API-specific graders `disclaimer`, `compile`, `release_match`, and `live_verify` take an `api` field (`jcsmp`, the default, or `python`). For a JCSMP project, `compile` runs a real `mvn compile`, and `release_match` matches the generated pom against the live sol-jcsmp `` on Maven Central. For a Python project, `compile` installs each `requirements.txt` into a fresh virtual environment outside the project, runs `py_compile` on every script, and imports each top-level module beside the `requirements.txt`, and `release_match` matches the `solace-pubsubplus` pin against the live PyPI `info.version`. The `disclaimer` grader checks the header of every `.java` or `.py` file. When a broker is configured, `live_verify` runs a real `verify.sh roundtrip` of the generated project. The `llm_judge` grader sends the transcript to a fixed judge model for routing and grounding criteria that a grep cannot decide. Negative cases assert the absence of forbidden behavior: code before a design confirm, a support email before the support-contract confirmation, unscrubbed identifiers in a feedback draft, and memory-derived debug steps. Run `./tools/run-output-evals.sh --help` for the full grader reference. ## How the transcript maps onto the corpus @@ -37,7 +37,7 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json | Grader `turn` | Selects which `turn.jsonl` file the grader reads. It is not a field inside the file. | | `skill` (the implicit skill-fired gate) | An `assistant` event whose `message.content[]` holds a `tool_use` block with `name` `Skill`. Its `input.skill` is `plugin:skill-name`; the runner strips the prefix. | | `assistant_grep` | `assistant` events, `message.content[]` blocks with `type` `text`. Thinking blocks and tool payloads are excluded. | -| `tool_use` | `assistant` events, blocks with `type` `tool_use`. The runner matches `name` and greps the compact JSON of `input`. | +| `tool_use` | `assistant` events, blocks with `type` `tool_use`. The runner matches `name` and greps the compact JSON of `input`, minus a Bash call's `description` field (narration, not the command). | | `llm_judge` with `webfetch_urls` or `webfetch_results` | WebFetch `tool_use` blocks supply the URLs. `user` events whose `tool_result.tool_use_id` matches a WebFetch block supply the fetched content. | | Session continuity (`--resume`) | `session_id` on the `result` event, with the `init` event as the fallback. | | INFRA verdict for a turn | A missing `result` event, or `result.is_error` true with a `subtype` other than `error_max_turns`. | @@ -48,10 +48,11 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json - The `claude` CLI (Claude Code) on your PATH. - `jq` and `curl` on your PATH. -- `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the cases that compile generated code need it, and the runner checks for it only when such a case is selected. +- `mvn` (Apache Maven) with a JDK 11 or newer on your PATH. Only the JCSMP cases that compile generated code need it, and the runner checks for it only when such a case is selected. +- `python3` with the `venv` module and `pip`, for the cases that generate Solace Python projects. A generated Python project installs `solace-pubsubplus` into its own virtual environment, and its `verify.sh` runs that environment's interpreter. The Python `compile` grader builds a separate virtual environment of its own. The runner checks for `python3` only when such a case is selected. - An exported credential, either `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN`. An existing `claude` login does not satisfy this: the runner uses a throwaway config directory so that only the plugin under test is loaded, and that directory has no login. Claude seat holders can mint a token with `claude setup-token`. -- `timeout` from GNU coreutils (or Homebrew's `gtimeout`) on your PATH, to cap each `mvn compile` and `verify.sh` run at 600 seconds. Optional: without it those steps run unbounded, and the runner prints a warning. On macOS, install it with `brew install coreutils`. -- Network access to `repo1.maven.org` (Maven Central metadata and dependencies) and `docs.solace.com` (the skills fetch documentation pages). +- `timeout` from GNU coreutils (or Homebrew's `gtimeout`) on your PATH, to cap each `mvn compile`, `pip install`, and `verify.sh` run at 600 seconds. Optional: without it those steps run unbounded, and the runner prints a warning. On macOS, install it with `brew install coreutils`. +- Network access to `repo1.maven.org` (Maven Central metadata and dependencies), `pypi.org` and `files.pythonhosted.org` (the `solace-pubsubplus` release metadata and package), and `docs.solace.com` (the skills fetch documentation pages). - Optional: the four `OUTPUT_EVAL_BROKER_*` variables for the live round trip against a real broker (see below). - On macOS, `caffeinate` to keep the machine awake for a full leg. - On Windows, Git Bash. The repository's `.gitattributes` checks shell scripts out with LF endings, which bash and the byte-identity grader on `verify.sh` both need. A checkout made with `core.autocrlf=true` before that file existed keeps its CRLF copies, and git reports the tree clean because it treats them as unchanged, so `git pull` and `git checkout -- .` do not rewrite them. Re-clone, or delete the tracked scripts and check them out again: `git ls-files -z '*.sh' | xargs -0 rm -f && git checkout -- .`. @@ -60,18 +61,18 @@ Run from the repository root: ```shell export ANTHROPIC_API_KEY= # or CLAUDE_CODE_OAUTH_TOKEN -./tools/run-output-evals.sh # defaults to claude-sonnet-5 -./tools/run-output-evals.sh --model claude-opus-5 +./tools/run-output-evals.sh # defaults to a Sonnet model +./tools/run-output-evals.sh --model opus # the latest Opus ./tools/run-output-evals.sh --case appdev-quickstart-implement-full ``` -Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. +Run the suite for both the latest Sonnet and the latest Opus (`--model sonnet` and `--model opus`) before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default a pinned Sonnet model) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails. -The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`. +The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the JCSMP and Python Quickstart implement cases are `must_pass`. ### Live verification against a real broker -The compile case's `live_verify` grader runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables exported in your shell: +The `live_verify` grader of each Quickstart implement case runs the generated project's `verify.sh roundtrip` against a broker you name. It is opt-in and needs all four of these variables exported in your shell: ```shell export OUTPUT_EVAL_BROKER_HOST=tcp://:55555 @@ -80,6 +81,14 @@ export OUTPUT_EVAL_BROKER_USER= export OUTPUT_EVAL_BROKER_PASSWORD= ``` -Do not put these values in a file inside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model, and hands them to `verify.sh` only as CLI arguments. +Do not put these values in a file inside the repository; the repository is public. The runner strips the export attribute from the four variables so the values never reach the subject model. A JCSMP project gets them as `verify.sh` CLI arguments. A Python project gets them in a `config.json` that the runner writes into the generated project, in the run's work directory, after the subject model has finished. + +The Python API has no default trust store, so a Python live round trip against a `tcps://` host also needs this variable: + +```shell +export OUTPUT_EVAL_BROKER_TRUST_STORE= +``` + +The runner writes it into the generated `config.json` as `solace.messaging.tls.trust-store-path`. For Solace Cloud, the **Connect** tab of the service has a **Download PEM** link for that certificate. When the host is `tcps://`, a selected case runs a Python live round trip, and this variable is not set, the runner exits with an error before any case runs. A JCSMP project does not use it. The runner prints the live-verify state before the first case starts and again in the summary. When none of the four variables is exported, the grader skips and the case line says so. When some but not all are exported, the runner exits with an error before any case runs. When `verify.sh` reports a broker or credential error (its exit code 2), the case is an INFRA failure, not a skill failure. Use a dedicated eval message VPN or service: the generated subscriber provisions a durable queue on every run, so queues accumulate, and the client username needs permission to create endpoints. diff --git a/plugins/solace-messaging-skills/evals/output-evals.json b/plugins/solace-messaging-skills/evals/output-evals.json index 7b2a7ac..c91b2f1 100644 --- a/plugins/solace-messaging-skills/evals/output-evals.json +++ b/plugins/solace-messaging-skills/evals/output-evals.json @@ -61,9 +61,9 @@ { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, { "type": "file_grep", "glob": "pom.xml", "pattern": "|", "expect": "absent" }, { "type": "compile" }, - { "type": "maven_release_match" }, + { "type": "release_match" }, { "type": "live_verify" }, - { "type": "java_disclaimer" }, + { "type": "disclaimer" }, { "type": "file_grep", "glob": "*.java", "pattern": "^import .*\\*;", "expect": "absent" }, { "type": "file_grep", "glob": "pom.xml", "pattern": "com.solacesystems", "fixed": true, "expect": "present" }, { "type": "file_exists", "glob": "log4j2.xml", "expect": "present" }, @@ -89,7 +89,7 @@ { "type": "tool_use", "tool": "*", "input_pattern": "repo1\\.maven\\.org/maven2/com/solacesystems/sol-jcsmp/maven-metadata\\.xml", "expect": "present" }, { "type": "tool_use", "tool": "Bash", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, { "type": "tool_use", "tool": "WebFetch", "input_pattern": "solrsearch|search\\.maven\\.org", "expect": "absent" }, - { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app)", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "(\"command\":\"|[^\"'`#\\\\])(\\./|(ba)?sh )verify\\.sh +(consumer|publisher|roundtrip|app)", "ignore_heredoc_bodies": true, "expect": "absent" }, { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" } ] }, @@ -123,7 +123,7 @@ "max_agent_turns": 80, "graders": [ { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 3 }, - { "type": "java_disclaimer" }, + { "type": "disclaimer" }, { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "SSL_VALIDATE_CERTIFICATE_DATE", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "setProperty\\([^)]*SSL_TRUST_STORE", "expect": "absent" }, @@ -132,10 +132,303 @@ { "type": "compile" }, { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REQUEST_RECEIVED", "fixed": true, "expect": "present" }, { "type": "file_grep", "glob": "*.java", "pattern": "VERIFY: REPLY_RECEIVED", "fixed": true, "expect": "present" }, - { "type": "tool_use", "tool": "Bash", "input_pattern": "verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "(\"command\":\"|[^\"'`#\\\\])(\\./|(ba)?sh )verify\\.sh +(consumer|publisher|roundtrip|app|direct|guaranteed-request-reply|direct-request-reply)", "ignore_heredoc_bodies": true, "expect": "absent" }, { "type": "assistant_grep", "pattern": "verify.sh", "fixed": true, "expect": "present" } ] }, + { + "id": "appdev-python-design-summary-positive", + "skill": "solace-application-development", + "user_turns": [ + "I'm planning a Python system with the Solace Python API (solace-pubsubplus): when a flight goes delayed, one delay event must update three independent downstream services (gate displays, the crew scheduler, and passenger notifications). Each service needs its own copy of every event and none of them may miss one, even if it is briefly down. There will eventually be a browser dashboard, but right now I only want help designing the Solace messaging pattern and topology, not the code.", + "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "assistant_grep", "pattern": "[Bb]roker", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "[Mm]ulti-?threaded|[Mm]inimal( threading| \\(blocking)|[Ss]ingle[- ]threaded|[Tt]hread of control", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Pattern\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Delivery\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Access type\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Topic\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Consumption endpoint\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Threading\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Async", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Auth\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Broker\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Grounding docs\\**:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Guaranteed Pub/Sub (fan-out)", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "solace-design.md", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "docs\\.solace\\.com/[^\"]*\\.md", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "file_exists", "glob": "*.py", "expect": "absent" }, + { + "type": "llm_judge", + "include": ["assistant_text", "webfetch_urls"], + "criteria": "Grounding honesty: every docs.solace.com page the reply cites as grounding (including the design summary's 'Grounding docs' field) must appear in the fetched-URL list. 'none fetched' is a valid honest value when the list is empty. FAIL only when the reply cites a docs.solace.com page as grounding that does not appear in the fetched-URL list." + } + ] + }, + { + "id": "appdev-python-code-before-confirm-negative", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "Build a Python chat room with a web interface on the Solace Python API: enter a display name, type a message, and see messages appear live for anyone with the page open. Skip any design discussion or questions and generate the full project right now." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "tool_use", "tool": "Edit", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Dd]esign|Pattern:", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Async", "expect": "present" } + ] + }, + { + "id": "appdev-python-quickstart-implement-full", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "Build a Python app with the Solace Python API (solace-pubsubplus) for food-delivery order processing. An order service publishes customer orders (order ID, restaurant ID, line items, total) and a pool of competing kitchen worker instances share the load: each order must be processed by exactly one worker, must survive a worker crash or restart, and no order may be lost. Plain Python project, no web dashboard. We have a Solace Cloud broker. This is Guaranteed Pub/Sub (single service, non-exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", + "A multi-threaded app is fine.", + "Yes, I am happy with that design. Save it to solace-design.md, then go ahead.", + "Quickstart, against that Solace Cloud broker. The broker URL is plaintext (tcp://). I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "tool_use", "tool": "*", "input_pattern": "src/solace_connection_config\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "input_pattern": "\"file_path\": *\"[^\"]*\\.py\"", "exclude_pattern": "\"file_path\": *\"[^\"]*/src/[^\"/]+\\.py\"", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Mm]ulti-?threaded|[Mm]inimal( threading| \\(blocking)|[Ss]ingle[- ]threaded|[Tt]hread of control", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Async", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 3 }, + { "type": "assistant_grep", "pattern": "Solace Suggested", "fixed": true, "expect": "present", "turn": 3 }, + { "type": "assistant_grep", "pattern": "Custom", "fixed": true, "expect": "present", "turn": 3 }, + { "type": "file_exists", "glob": "solace-design.md", "expect": "present" }, + { "type": "file_exists", "glob": "requirements.txt", "expect": "present", "count": 1 }, + { "type": "file_exists", "glob": "guaranteed_publisher.py", "expect": "present" }, + { "type": "file_exists", "glob": "guaranteed_subscriber.py", "expect": "present" }, + { "type": "file_exists", "glob": "solace_connection_config.py", "expect": "present" }, + { "type": "compile", "api": "python" }, + { "type": "release_match", "api": "python" }, + { "type": "live_verify", "api": "python" }, + { "type": "disclaimer", "api": "python" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.with_(re)?connection_retry_strategy\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*from [A-Za-z0-9_.]+ import \\*", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*(import|from) multiprocessing", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: CONNECTED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: MESSAGE_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "config.example.json", "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "solace.messaging.transport.host", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcp://HOST:55555", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_VPN", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_USERNAME", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "YOUR_PASSWORD", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": ".venv", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/python/scripts/verify.sh" }, + { "type": "file_exists", "glob": "verify-hooks.sh", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^START_CMD=.*src/guaranteed_subscriber\\.py", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^TRIGGER_CMD=.*src/guaranteed_publisher\\.py", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^READY_MARKER=", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^PASS_MARKER=", "expect": "present" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "pypi\\.org/pypi/solace-pubsubplus/json", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "py_compile", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "(\"command\":\"|[^\"'`#\\\\])(\\./|(ba)?sh )verify\\.sh +(consumer|publisher|roundtrip|app|direct)", "ignore_heredoc_bodies": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" }, + { "type": "assistant_grep", "pattern": "[Tt]rust[ -]?store", "expect": "present", "turn": 4 } + ] + }, + { + "id": "appdev-python-debug-redirect-negative", + "skill": "solace-application-development", + "must_pass": true, + "user_turns": [ + "My food-delivery kitchen worker uses the Solace Python API (solace-pubsubplus), and its PersistentMessageReceiver now fails to start with a PubSubPlusClientError saying it could not bind to the queue. Walk me through debugging this step by step." + ], + "max_agent_turns": 15, + "graders": [ + { "type": "assistant_grep", "pattern": "https://docs.solace.com/API/Messaging-APIs/Python-API/python-home.md", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "*", "expect": "absent" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { + "type": "llm_judge", + "criteria": "The user asked for step-by-step debugging of a Solace Python API error. The skill's contract says the documentation redirect is the ENTIRE debug answer: acknowledge the problem, link the canonical Solace Python API documentation, and stop. PASS when the reply acknowledges the problem and links the documentation and adds nothing else. Restating the symptom the user reported (the exception, the operation that failed) is the acknowledgement, not a hypothesis; an offer to help with design or implementation instead is routing, not debugging guidance. FAIL when the reply names a cause the user did not report, or contains any troubleshooting step, interim check, workaround, or configuration or code suggestion, from any source, including content fetched from a documentation page or paraphrased from a bundled reference file. Restating the rule does not excuse guidance that follows it." + } + ] + }, + { + "id": "appdev-python-suggested-secure", + "skill": "solace-application-development", + "user_turns": [ + "Build a Python app with the Solace Python API (solace-pubsubplus) for food-delivery order processing: an order service publishes customer orders and a kitchen worker processes them. No order may be lost, even if the kitchen worker restarts. Plain Python, no dashboard. We have a TLS-enabled Solace Cloud broker. This is Guaranteed Pub/Sub (single service, exclusive) with PERSISTENT delivery; derive the topic and queue names yourself.", + "I am not sure, go with the default.", + "Yes, I am happy with that design. No need to save it, go ahead.", + "Solace Suggested, against that Solace Cloud broker.", + "No HA failover needed. I will put the broker's CA certificate into a trust store directory myself. I will not share credentials in the chat; compile only and hand me the commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "tool_use", "tool": "*", "input_pattern": "src/solace_connection_config\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "input_pattern": "\"file_path\": *\"[^\"]*\\.py\"", "exclude_pattern": "\"file_path\": *\"[^\"]*/src/[^\"/]+\\.py\"", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Mm]ulti-?threaded|[Mm]inimal( threading| \\(blocking)|[Ss]ingle[- ]threaded|[Tt]hread of control", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Async", "expect": "present" }, + { "type": "assistant_grep", "pattern": "[Ff]ailover", "expect": "present", "turn": 4 }, + { "type": "assistant_grep", "pattern": "ght_client_certs.md", "fixed": true, "expect": "present" }, + { "type": "disclaimer", "api": "python" }, + { "type": "file_exists", "glob": "requirements.txt", "expect": "present", "count": 2 }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "count": 2, "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/python/scripts/verify.sh" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present", "count": 2 }, + { "type": "compile", "api": "python" }, + { "type": "file_grep", "glob": "*.py", "pattern": "CERT_VALIDATED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "CERT_REJECT_EXPIRED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "CERT_VALIDATE_SERVERNAME", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*(CERT_VALIDATED|CERT_REJECT_EXPIRED|CERT_VALIDATE_SERVERNAME)\\]?[[:space:]]*[:=][[:space:]]*False", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.without_certificate_validation\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "PERSISTENT_DMQ_ELIGIBLE", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.with_reconnection_retry_strategy\\(", "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*(\\.with_connection_retry_strategy\\(|CONNECTION_RETRIES_PER_HOST)", "expect": "absent" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcps://", "fixed": true, "expect": "all_files" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "solace.messaging.tls.trust-store-path", "fixed": true, "expect": "all_files" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "all_files" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "(\"command\":\"|[^\"'`#\\\\])(\\./|(ba)?sh )verify\\.sh +(consumer|publisher|roundtrip|app|direct)", "ignore_heredoc_bodies": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "--subscriber-dir", "fixed": true, "expect": "present" } + ] + }, + { + "id": "appdev-python-persisted-request-routing", + "skill": "solace-application-development", + "user_turns": [ + "I'm designing a Python system with the Solace Python API where a pharmacy service requests refill authorization from a doctor's-office service and waits for the answer. The refill request must not be lost, even when the doctor's-office service is down at the moment it is sent. Right now I only want help designing the Solace messaging pattern and topology, not the code.", + "We have a Solace Cloud broker available. Go with your recommendations, but do not save anything to a file yet." + ], + "max_agent_turns": 25, + "graders": [ + { "type": "tool_use", "tool": "WebFetch", "input_pattern": "Python-API-Request-Reply\\.md", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Pattern[*:`[:space:]]+Guaranteed Pub/Sub \\(", "expect": "present" }, + { "type": "assistant_grep", "pattern": "[Mm]ulti-?threaded|[Mm]inimal( threading| \\(blocking)|[Ss]ingle[- ]threaded|[Tt]hread of control", "expect": "present", "turn": 1 }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Async", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Pattern[*:`[:space:]]+Request-Reply", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "[Cc]orrelation", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "expect": "absent" }, + { "type": "file_exists", "glob": "*", "expect": "absent" } + ] + }, + { + "id": "appdev-python-minimal-threading-guaranteed-implement", + "skill": "solace-application-development", + "user_turns": [ + "Build a Python app with the Solace Python API for food-delivery order processing: an order service publishes customer orders and a kitchen worker processes them. No order may be lost, even if the kitchen worker restarts, and only one worker is active at a time. Plain Python project, no web dashboard. We have a Solace Cloud broker. This is Guaranteed Pub/Sub (single service, exclusive) with PERSISTENT delivery; derive the topic and queue names yourself. Keep the messaging data path on a single thread of control with minimal threading: the publisher's main thread sends each order and waits for its acknowledgement, and the worker's main loop pulls each message and acknowledges it itself. No callbacks on the messaging data path.", + "Yes, I am happy with that design. No need to save it, go ahead.", + "Quickstart, against that Solace Cloud broker. The broker URL is plaintext (tcp://). I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "tool_use", "tool": "*", "input_pattern": "src/solace_connection_config\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "input_pattern": "\"file_path\": *\"[^\"]*\\.py\"", "exclude_pattern": "\"file_path\": *\"[^\"]*/src/[^\"/]+\\.py\"", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Minimal", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "file_exists", "glob": "guaranteed_publisher_blocking.py", "expect": "present" }, + { "type": "file_exists", "glob": "guaranteed_subscriber_blocking.py", "expect": "present" }, + { "type": "file_exists", "glob": "guaranteed_publisher.py", "expect": "absent" }, + { "type": "file_exists", "glob": "guaranteed_subscriber.py", "expect": "absent" }, + { "type": "file_exists", "glob": "solace_connection_config.py", "expect": "present" }, + { "type": "file_exists", "glob": "requirements.txt", "expect": "present", "count": 1 }, + { "type": "compile", "api": "python" }, + { "type": "release_match", "api": "python" }, + { "type": "live_verify", "api": "python" }, + { "type": "disclaimer", "api": "python" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.publish_await_acknowledgement\\(", "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.publish_await_acknowledgement\\([^)]*None", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_message\\(", "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_message\\([[:space:]]*\\)", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_message\\([^)]*None", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_async\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.set_message_publish_receipt_listener\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*time\\.sleep\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.with_(re)?connection_retry_strategy\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*from [A-Za-z0-9_.]+ import \\*", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*(import|from) multiprocessing", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: CONNECTED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: QUEUE_BOUND", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: PUBLISH_ACKED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: MESSAGE_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "config.example.json", "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcp://HOST:55555", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/python/scripts/verify.sh" }, + { "type": "file_exists", "glob": "verify-hooks.sh", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^START_CMD=.*src/guaranteed_subscriber_blocking\\.py", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^TRIGGER_CMD=.*src/guaranteed_publisher_blocking\\.py", "expect": "present" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-guaranteed-publisher-blocking-sample\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-guaranteed-subscriber-blocking-sample\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-guaranteed-(publisher|subscriber)-sample\\.py", "expect": "absent" }, + { "type": "tool_use", "tool": "*", "input_pattern": "pypi\\.org/pypi/solace-pubsubplus/json", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "py_compile", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "(\"command\":\"|[^\"'`#\\\\])(\\./|(ba)?sh )verify\\.sh +(consumer|publisher|roundtrip|app|direct)", "ignore_heredoc_bodies": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "./verify.sh", "fixed": true, "expect": "present" } + ] + }, + { + "id": "appdev-python-minimal-threading-direct-implement", + "skill": "solace-application-development", + "user_turns": [ + "Build a Python app with the Solace Python API for a plant-monitoring system: a telemetry publisher pushes temperature readings from the factory floor several times a second, and a live dashboard service consumes them to redraw a wallboard. A dropped reading makes no difference because the next one supersedes it, and nothing needs to be stored. Plain Python project, no web dashboard. We have a Solace Cloud broker. This is Direct Pub/Sub with DIRECT delivery; derive the topic yourself. Keep the dashboard's messaging path on a single thread of control with minimal threading: its main loop pulls the next reading itself instead of taking a callback.", + "Yes, I am happy with that design. No need to save it, go ahead.", + "Quickstart, against that Solace Cloud broker. The broker URL is plaintext (tcp://). I will not put credentials in the chat, so compile only and hand me the exact verify commands to run myself." + ], + "max_agent_turns": 80, + "graders": [ + { "type": "tool_use", "tool": "*", "input_pattern": "src/solace_connection_config\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "Write", "input_pattern": "\"file_path\": *\"[^\"]*\\.py\"", "exclude_pattern": "\"file_path\": *\"[^\"]*/src/[^\"/]+\\.py\"", "expect": "absent" }, + { "type": "assistant_grep", "pattern": "Threading[*:`[:space:]]+Minimal", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Pattern[*:`[:space:]]+Direct Pub/Sub", "expect": "present" }, + { "type": "assistant_grep", "pattern": "Quickstart", "fixed": true, "expect": "present", "turn": 2 }, + { "type": "file_exists", "glob": "direct_publisher.py", "expect": "present" }, + { "type": "file_exists", "glob": "direct_subscriber_blocking.py", "expect": "present" }, + { "type": "file_exists", "glob": "direct_subscriber.py", "expect": "absent" }, + { "type": "file_exists", "glob": "solace_connection_config.py", "expect": "present" }, + { "type": "file_exists", "glob": "requirements.txt", "expect": "present", "count": 1 }, + { "type": "compile", "api": "python" }, + { "type": "release_match", "api": "python" }, + { "type": "disclaimer", "api": "python" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_message\\(", "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_message\\([[:space:]]*\\)", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_message\\([^)]*None", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.receive_async\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.set_publish_failure_listener\\(", "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.publish_await_acknowledgement\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*create_persistent_message_(publisher|receiver)_builder\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*time\\.sleep\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*\\.with_(re)?connection_retry_strategy\\(", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[[:space:]]*from [A-Za-z0-9_.]+ import \\*", "expect": "absent" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: CONNECTED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: SUBSCRIBED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "VERIFY: MESSAGE_RECEIVED", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": "*.py", "pattern": "^[^#]*VERIFY: PUBLISH_ACKED", "expect": "absent" }, + { "type": "file_exists", "glob": "config.example.json", "expect": "present" }, + { "type": "file_grep", "glob": "config.example.json", "pattern": "tcp://HOST:55555", "fixed": true, "expect": "present" }, + { "type": "file_grep", "glob": ".gitignore", "pattern": "config.json", "fixed": true, "expect": "present" }, + { "type": "file_exists", "glob": "verify.sh", "expect": "present", "identical_to": "plugins/solace-messaging-skills/skills/solace-application-development/references/python/scripts/verify.sh" }, + { "type": "file_exists", "glob": "verify-hooks.sh", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^START_CMD=.*src/direct_subscriber_blocking\\.py", "expect": "present" }, + { "type": "file_grep", "glob": "verify-hooks.sh", "pattern": "^TRIGGER_CMD=.*src/direct_publisher\\.py", "expect": "present" }, + { "type": "file_exists", "glob": "solace-verification-checklist.md", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-direct-publisher-sample\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-direct-subscriber-blocking-sample\\.py", "expect": "present" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-direct-subscriber-sample\\.py", "expect": "absent" }, + { "type": "tool_use", "tool": "*", "input_pattern": "python-guaranteed-[a-z-]+\\.py", "expect": "absent" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "py_compile", "expect": "present" }, + { "type": "tool_use", "tool": "Bash", "input_pattern": "(\"command\":\"|[^\"'`#\\\\])(\\./|(ba)?sh )verify\\.sh +(consumer|publisher|roundtrip|app|direct)", "ignore_heredoc_bodies": true, "expect": "absent" }, + { "type": "assistant_grep", "pattern": "./verify.sh direct", "fixed": true, "expect": "present" } + ] + }, { "id": "feedback-email-before-confirm-negative", "skill": "solace-messaging-feedback", diff --git a/plugins/solace-messaging-skills/evals/trigger-evals.json b/plugins/solace-messaging-skills/evals/trigger-evals.json index 750d510..7f7cf6f 100644 --- a/plugins/solace-messaging-skills/evals/trigger-evals.json +++ b/plugins/solace-messaging-skills/evals/trigger-evals.json @@ -12,9 +12,16 @@ {"skill":"solace-application-development","prompt":"For the logistics platform we run in Java on our Solace event broker, a ShipmentTracker service publishes a DeliveryException event whenever a package misses its checkpoint. Three separate services each need to act on it independently: a CustomerNotifier emails the customer, a ReroutePlanner recalculates the route, and an AnalyticsIngest service records it for reporting. Each is owned by a different team, each needs its own full copy of every event, and none of them can miss one even if that service is down for maintenance. How should I structure the messaging so every one of these services reliably receives every event on its own?","should_trigger":true,"must_pass":true}, {"skill":"solace-application-development","prompt":"We've standardized on the Solace JMS API. How do I create a JMS ConnectionFactory and a MessageProducer to send to a queue in Java?","should_trigger":false}, - {"skill":"solace-application-development","prompt":"I'm writing my Solace client in Python with the solace-pubsubplus library. How do I set up the messaging service and a persistent receiver?","should_trigger":false}, - {"skill":"solace-application-development","prompt":"We've got a warehouse-robotics system where two Python services communicate through our Solace event broker using the solace-pubsubplus library. A RobotDispatcher publishes task assignments and a RobotController consumes them and must not lose any while a robot reboots. How should I structure the messaging between these two Python services?","should_trigger":false}, + {"skill":"solace-application-development","prompt":"I'm writing my Solace client in Python with the solace-pubsubplus library. How do I set up the messaging service and a persistent receiver?","should_trigger":true}, + {"skill":"solace-application-development","prompt":"We've got a warehouse-robotics system where two Python services communicate through our Solace event broker using the solace-pubsubplus library. A RobotDispatcher publishes task assignments and a RobotController consumes them and must not lose any while a robot reboots. How should I structure the messaging between these two Python services?","should_trigger":true,"must_pass":true}, + {"skill":"solace-application-development","prompt":"We're building a Python inventory service on the Solace Python API. Stock-level updates must not be lost while the service restarts. I want to keep the app single-threaded: no callbacks on the messaging path, the main loop should pull each message and acknowledge it itself, and the publisher should wait for each acknowledgement before it sends the next one. How should I structure this?","should_trigger":true}, + {"skill":"solace-application-development","prompt":"My Python worker uses a thread pool and the GIL is limiting it. Should I move the work to multiprocessing or asyncio, and how do I keep a single thread of control over the job queue?","should_trigger":false}, {"skill":"solace-application-development","prompt":"Show me how to publish to a Solace topic from C# using the .NET messaging API.","should_trigger":false}, + {"skill":"solace-application-development","prompt":"I'm writing a Go service with the Solace PubSub+ Messaging API for Go. How do I build a persistent message publisher and handle publish receipts?","should_trigger":false}, + {"skill":"solace-application-development","prompt":"I'm on the Solace Messaging API for Java (com.solace:solace-messaging-client). How do I set up a MessagingService and a PersistentMessageReceiver in my Java service to consume from a durable queue?","should_trigger":false}, + {"skill":"solace-application-development","prompt":"In our browser dashboard we use the solclientjs library. How do I subscribe to a Solace topic from JavaScript and update the page when messages arrive?","should_trigger":false}, + {"skill":"solace-application-development","prompt":"I have a Python asyncio service that calls a few REST APIs concurrently. How do I limit it to five requests in flight at once with a semaphore?","should_trigger":false}, + {"skill":"solace-application-development","prompt":"I'm using confluent-kafka in Python. How do I write a consumer that commits offsets only after each message has been processed?","should_trigger":false}, {"skill":"solace-application-development","prompt":"I need a Kafka producer and consumer in Java for our clickstream pipeline. How do I configure the KafkaProducer with the right key and value serializers?","should_trigger":false}, {"skill":"solace-application-development","prompt":"How do I provision a durable queue named q.orders.intake and set its max-spool-usage on our Solace broker through the SEMP API and PubSub+ Manager? I'm configuring the broker, not writing a client.","should_trigger":false}, {"skill":"solace-application-development","prompt":"How do I model our order and shipment events and the relationships between them in Solace Event Portal?","should_trigger":false}, diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/SKILL.md b/plugins/solace-messaging-skills/skills/solace-application-development/SKILL.md index a291599..41b40aa 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/SKILL.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/SKILL.md @@ -1,6 +1,6 @@ --- name: solace-application-development -description: "Use when the user mentions JCSMP, the com.solacesystems:sol-jcsmp library, JCSMPSession, publishing to a Solace topic, consuming from a Solace queue, or guaranteed messaging, OR asks broadly to build or design a Solace messaging app in Java (publisher, consumer, pub/sub). This is the single front door for Solace API development: it grounds the work in canonical Solace docs and routes the request to the right API (JCSMP today), helping choose a messaging pattern or generate a runnable Maven app. Do NOT trigger for an explicitly named non JCSMP API, which beats the generic Java signal: Solace JMS, or the Solace Python, Go, .NET, or JavaScript APIs. Also do NOT trigger for Apache Kafka, broker administration and configuration, Solace Event Portal event modeling, Schema Registry, OpenTelemetry tracing, or topic-architecture and topic-hierarchy design questions (use solace-topic-best-practices)." +description: "Use when the user mentions JCSMP, the com.solacesystems:sol-jcsmp library, JCSMPSession, the Solace Python API (solace-pubsubplus), MessagingService, PersistentMessageReceiver, publishing to a Solace topic, consuming from a Solace queue, or guaranteed messaging, OR asks broadly to build or design a Solace messaging app in Java or Python (publisher, consumer, pub/sub). This is the single front door for Solace API development: it grounds the work in canonical Solace docs and routes to the right API, helping choose a messaging pattern or generate a runnable app. Do NOT trigger for an explicitly named other Solace API, which beats the generic Java signal: Solace JMS, the Solace Messaging API for Java, or the Solace Go, .NET, or JavaScript APIs. Also do NOT trigger for Apache Kafka (any language), broker administration and configuration, Solace Event Portal event modeling, Schema Registry, OpenTelemetry tracing, or topic-architecture and topic-hierarchy design questions (use solace-topic-best-practices)." allowed-tools: Read Write Edit Bash Grep Glob WebFetch --- @@ -13,6 +13,7 @@ This skill is the front door for building Solace messaging applications grounded **Do NOT read all reference files upfront. Read ONLY what you need, when you need it.** - User wants a JCSMP application (design help or code generation) then read `references/jcsmp.md`, which selects the right mode within the JCSMP API. +- User wants a Solace Python API application (design help or code generation) then read `references/python.md`, which selects the right mode within the Python API. - Read a mode file only at the moment a step needs it. - Most requests need the API's entry file plus one mode file, not the whole tree. @@ -30,22 +31,25 @@ Determine which Solace API the developer is building against, then read that API | User intent | API | What to do | |---|---|---| -| Anything about JCSMP, the `com.solacesystems:sol-jcsmp` library, a Java publisher or consumer, publishing to a topic, or consuming from a queue with guaranteed messaging | **JCSMP** | Read `references/jcsmp.md` | +| Anything about JCSMP, the `com.solacesystems:sol-jcsmp` library, `JCSMPSession`, or a Java publisher or consumer (publishing to a topic, or consuming from a queue with guaranteed messaging) | **JCSMP** | Read `references/jcsmp.md` | +| Anything about the Solace Python API, the `solace-pubsubplus` library, `MessagingService`, `PersistentMessageReceiver`, or a Python publisher or consumer | **Python** | Read `references/python.md` | -If the API is unclear but the request is clearly about Solace messaging in Java, default to **JCSMP**. Mode detection (Design, Implement) then happens in that API's entry file. +If the API is unclear but the request is clearly about Solace messaging in Java, default to **JCSMP**. If the API is unclear but the request is clearly about Solace messaging in Python, default to **Python**. If the request names no language at all (for example "build me a Solace publisher"), ask one question before anything else: JCSMP (Java) or Python? Files in the workspace (a `pom.xml`, `.py` files, a `requirements.txt`) are facts to report while you ask, never the answer. The one exception is a saved `solace-design.md` that the developer asks you to build from: its title (`Solace JCSMP Design Summary` or `Solace Python Design Summary`) names the API, so read that API's entry file, whose design-contract gate restates the summary and takes the developer's confirm. The Solace Messaging API for Java (`com.solace:solace-messaging-client`) shares class names such as `MessagingService` and `PersistentMessageReceiver` with the Python API, but it is neither JCSMP nor Python: when a Java request names that API or its `com.solace` package, say in one line that this skill does not cover it yet, and stop. Mode detection (Design, Implement) then happens in that API's entry file. ## Invariants Non-negotiable rules that apply across every API. Apply all. 1. **Content sourcing**: Every named entity (API, parameter, configuration value) must be traceable to a canonical Solace source. Do not assert behavior, defaults, or best practices that are not in the grounding documentation. -2. **WebFetch-on-demand doc grounding**: This skill bundles no documentation. When a step needs documentation content to ground a generation or design decision, WebFetch the live canonical page on demand (its `docs.solace.com` `.md` URL, the same link the reference files carry), then quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. Do not restate doc content in the skill; link the live `.md` URL. Never substitute another channel for that WebFetch: grounding taken from a docs chatbot, a search tool, or decompiling a jar does not satisfy this invariant, and a page never fetched in the session must never be cited as grounding. A short list of references stays live for the same reason it always has (Javadoc HTML, the JCSMP API Release Notes, Maven Central, the tutorials, and the GitHub samples) because those have no `docs.solace.com` `.md` form. -3. **No hardcoded versions**: Never pin a library version anywhere in skill content. The generated build resolves the latest release of every dependency at generation time from the AUTHORITATIVE Maven repository metadata; each API's entry file names the exact coordinate and carries the exact lookup command. Never read a version from the legacy `search.maven.org/solrsearch` index: it lags the repository metadata and reports stale versions. Security floors (a minimum version below which known CVEs live) are enforced as checks on the resolved result, never as pins. -4. **AI-assisted disclaimer header**: Every source file generated under this skill starts with this exact line: `AI-assisted code. Review before production use.` Directly below it: `See the verification checklist: solace-verification-checklist.md`. This applies to EVERY generated source file, including custom builds, web apps, and embedded shapes that adapt the reference samples outside the canonical generator. The pom and other non-source artifacts are exempt; the bundled reference samples themselves carry no header. -5. **Verification artifacts, always**: Any session that generates Solace code from this skill writes a tailored `solace-verification-checklist.md` beside the code as generation output, copies the bundled `verify.sh` (plus its generated `verify-hooks.sh`) into the project, and ends in a real verify run or the compile-only fallback with the exact handed-back commands. The `VERIFY:` markers are non-optional in generated Solace messaging code, whatever the app shape. An improvised check (for example curl against the app's own HTTP API) may TRIGGER traffic but never renders the verdict; the verdict comes from the markers and the verify exit code. -6. **Live-broker heads-up**: Immediately before starting ANY process that will connect to a live broker — verify.sh, `java -jar`, a `mvn` run, a run script, anything — print one line that names the broker host and states the actual broker-side effects of this run (the queues it provisions, the connections it opens, the messages it publishes), scaled to the real app. Environment discovery is never consent and never an answer: a running local broker, an existing config file, or found credentials are facts to report, not answers to consume; report them, then still ask which broker the developer wants to target. +2. **WebFetch-on-demand doc grounding**: This skill bundles no documentation. When a step needs documentation content to ground a generation or design decision, WebFetch the live canonical page on demand (its `docs.solace.com` `.md` URL, the same link the reference files carry), then quote or summarize the fetched page. Do not answer from memory and do not paraphrase guidance the page does not contain. Do not restate doc content in the skill; link the live `.md` URL. Never substitute another channel for that WebFetch: grounding taken from a docs chatbot, a search tool, or decompiling a jar does not satisfy this invariant, and a page never fetched in the session must never be cited as grounding. A short list of references stays live for the same reason it always has (the JCSMP Javadoc and the Python API reference, the JCSMP and Python API Release Notes, Maven Central and PyPI, the tutorials, and the GitHub samples) because those have no `docs.solace.com` `.md` form. Each API's entry file lists its own live exceptions. +3. **No hardcoded versions**: Never pin a library version anywhere in skill content. The generated build resolves the latest release of every dependency at generation time from the API's AUTHORITATIVE registry: the Maven repository metadata for JCSMP, the PyPI JSON `info.version` for Python. Each API's entry file names the exact coordinate and carries the exact lookup command. Never read a JCSMP version from the legacy `search.maven.org/solrsearch` index: it lags the repository metadata and reports stale versions. Security floors (a minimum version below which known CVEs live) are enforced as checks on the resolved result, never as pins. +4. **AI-assisted disclaimer header**: Every source file generated under this skill starts with this exact line: `AI-assisted code. Review before production use.` Directly below it: `See the verification checklist: solace-verification-checklist.md`. This applies to EVERY generated source file, including custom builds, web apps, and embedded shapes that adapt the reference samples outside the canonical generator. Build files (the pom, `requirements.txt`) and other non-source artifacts are exempt; the bundled reference samples themselves carry no header. +5. **Verification artifacts, always**: Any session that generates Solace code from this skill writes a tailored `solace-verification-checklist.md` beside the code as generation output, copies the API's own bundled verify script (plus its generated `verify-hooks.sh`) into the project, and ends in a real verify run or the compile-only fallback with the exact handed-back commands. The API's entry file names its verify script. Never claim a verified run without the script. The `VERIFY:` markers are non-optional in generated Solace messaging code, whatever the app shape and whatever the API. An improvised check (for example curl against the app's own HTTP API) may TRIGGER traffic but never renders the verdict; the verdict comes from the markers and the verify exit code. +6. **Live-broker heads-up**: Immediately before starting ANY process that will connect to a live broker — verify.sh, `java -jar`, a `mvn` run, a `python` run, a run script, anything — print one line that names the broker host and states the actual broker-side effects of this run (the queues it provisions, the connections it opens, the messages it publishes), scaled to the real app. Environment discovery is never consent and never an answer: a running local broker, an existing config file, or found credentials are facts to report, not answers to consume; report them, then still ask which broker the developer wants to target. ## Reference Files (read on-demand only) - `references/jcsmp.md`: the JCSMP entry file (mode detection, the JCSMP coordinate, and the live JCSMP doc links); read this for any JCSMP request. -- `references/jcsmp/`: all JCSMP API content (mode files, the reference samples, the verify script, and the compile fixture), reached through `references/jcsmp.md`. +- `references/jcsmp/`: all JCSMP API content (mode files, the reference samples, the verify script, and the sample check), reached through `references/jcsmp.md`. +- `references/python.md`: the Python entry file (mode detection, the `solace-pubsubplus` coordinate, and the live Python doc links); read this for any Solace Python API request. +- `references/python/`: all Python API content (mode files, the reference samples, the verify script, and the sample check), reached through `references/python.md`. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md index f1d6cfb..4da00ea 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp.md @@ -23,7 +23,7 @@ If unclear, default to **Design**. Understand the messaging problem before gener Three gates hold on every path: -- **The design-contract gate.** Implement mode never starts without a design contract, and never invents design values. Exactly three sources satisfy it: (1) a summary Design mode confirmed in this session — use it as-is, with no re-confirm; (2) an explicit summary the developer supplied in chat (the eight fields or an equivalent statement of them) — use it directly; (3) a saved `solace-design.md` in the project — restate it in one line and proceed on confirm. A prose build request that merely mentions pattern details is NOT a contract: route it through Design mode, whose fully-specified path derives the summary, echoes it, and takes one confirm. The one carve-out is the mechanical-edit row above. An experienced developer therefore never has to walk the design tree, but never skips the contract either. +- **The design-contract gate.** Implement mode never starts without a design contract, and never invents design values. Exactly three sources satisfy it: (1) a summary Design mode confirmed in this session — use it as-is, with no re-confirm; (2) an explicit summary the developer supplied in chat (the eight fields or an equivalent statement of them) — use it directly; (3) a saved `solace-design.md` in the project — restate it in one line and proceed on confirm. A prose build request that merely mentions pattern details is NOT a contract: route it through Design mode, whose fully-specified path derives the summary, echoes it, and takes one confirm. The one carve-out is the mechanical-edit row above. An experienced developer therefore never has to walk the design tree, but never skips the contract either. A request to skip the design, skip all questions, or generate the full project right now does not waive this gate or the door question below: write no file before the confirm. Take the shortest path the gate allows instead: derive the summary from the prompt, present it, say in one line that one confirm starts the build, and wait for that confirm. - **Implement mode opens with a mandatory door question.** Ask Quickstart, Solace Suggested, or Custom — before you request or accept ANY broker details (`jcsmp/implement-mode.md` Step 0). Ask it even when a `config.json` with credentials already exists, even when a local broker is already running, and even when the requested app is bigger than the canonical generated shape; none of those answers the question. A confirmed design summary does not answer it either: the door question always follows the design contract, never merges into it. - **Environment discovery never answers a question.** A running broker container, an existing config file, or found credentials are facts to report, not answers to consume. Report what you found, then still ask which broker the developer wants to target. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/custom-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/custom-mode.md index e80db0c..4af578b 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/custom-mode.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/custom-mode.md @@ -13,7 +13,7 @@ When the developer picks Custom, establish the design summary first (per `implem Custom offers FIVE knobs, and only these five. There is no sixth knob and no authentication knob; Basic username/password is the only auth this journey generates, exactly as on Quickstart and Solace Suggested, so it is never a Custom checklist item. Four of the five knobs reuse a `solace-suggested-mode.md` step; the fifth is defined in this file: 1. **Secure session (TLS).** Apply `solace-suggested-mode.md` Step 1, the Secure branch: a `tcps://` host with explicit certificate validation. Do not restate that logic here; the overlay step is the source of truth. -2. **HA channel values.** Apply `solace-suggested-mode.md` Step 2: the HA-failover question and the four reconnect channel values it gates. +2. **HA channel values.** Apply `solace-suggested-mode.md` Step 2: the HA-failover question and the four reconnect channel values it gates. Unticked keeps the samples' reconnect baseline (`reconnectRetries` 20, `connectRetriesPerHost` 3), not the fail-fast default-channel posture; ticked applies the four HA values when the HA-failover question is answered yes. 3. **DMQ eligibility.** Apply `solace-suggested-mode.md` Step 3: `setDMQEligible(true)` on every PERSISTENT publish, with the broker-side prerequisite stated as the developer's responsibility. 4. **Decoupled projects.** Apply `solace-suggested-mode.md` Step 4: two separate, independent single-module Maven projects, one per role. 5. **Admin-provisioned queue.** The one knob that is a real code change rather than a reused overlay step. It is defined in full in the "Admin-provisioned-queue knob" section below. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/.gitignore b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/.gitignore similarity index 100% rename from plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/.gitignore rename to plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/.gitignore diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/ConfigSmokeTest.java b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/ConfigSmokeTest.java similarity index 96% rename from plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/ConfigSmokeTest.java rename to plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/ConfigSmokeTest.java index 64757a6..976f394 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/ConfigSmokeTest.java +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/ConfigSmokeTest.java @@ -1,5 +1,5 @@ /* - * Runtime smoke test for SolaceConnectionConfig, driven by compile.sh. + * Runtime smoke test for SolaceConnectionConfig, driven by check-samples.sh. * * NOT a reference sample: compiling the helper never executes its config.json * parser or CLI-args fallback, so this tiny driver is compiled against the @@ -9,7 +9,7 @@ * value, blank CLI arg). * * One case per JVM invocation: the parser reads a CWD-relative config.json and - * the fail-fast paths throw/exit, so compile.sh drives each case from its own + * the fail-fast paths throw/exit, so check-samples.sh drives each case from its own * working directory and checks the exit code. The case name is argv[0]; any * connection arguments the case needs are built here, not taken from argv. */ diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/compile.sh b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/check-samples.sh similarity index 99% rename from plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/compile.sh rename to plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/check-samples.sh index cab3887..a307688 100755 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/compile.sh +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/check-samples.sh @@ -23,7 +23,7 @@ set -euo pipefail # Resolve this script's own directory so sample/pom paths are independent of CWD. SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" POM="${SCRIPT_DIR}/pom.xml" -# references/jcsmp/ lives two levels up from compile-fixture/ (evals/ -> jcsmp/). +# references/jcsmp/ lives two levels up from sample-check/ (evals/ -> jcsmp/). REFERENCES_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" # Per-sample source paths. Each new baked sample a later per-pattern plan adds is # a ONE-LINE append here, mirroring this shape: diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/pom.xml b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/pom.xml similarity index 88% rename from plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/pom.xml rename to plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/pom.xml index eb9016d..6dcc805 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/compile-fixture/pom.xml +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check/pom.xml @@ -5,13 +5,13 @@ 4.0.0 com.solace.samples - jcsmp-compile-fixture + jcsmp-sample-check 0.0.1-SNAPSHOT jar UTF-8 11 - RESOLVED_AT_GENERATION_TIME @@ -42,12 +42,12 @@ org.apache.logging.log4j log4j-api - 2.26.0 + 2.26.1 org.apache.logging.log4j log4j-core - 2.26.0 + 2.26.1 diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md index 76b78ee..af910ec 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/implement-mode.md @@ -126,7 +126,7 @@ Hold every invariant while adapting: keep the shared connection helper, the sess **Preserve the marker contract.** A variant adapts the SHAPE, never the marker strings. It MUST still emit its leaf's `VERIFY:` markers, character for character, so the fixed `verify.sh` stages drive it unchanged (this is exactly why a durable-reply-queue requestor still prints `VERIFY: REPLY_RECEIVED`). Never rename or drop a marker to fit a variant. -**Demo harness is not application logic.** The samples carry demo-harness elements that exist only to make a standalone run observable: the ENTER-to-quit `System.in` loop, the once-per-second stats-printing thread, the pacing `Thread.sleep` between sends, and the rotating example payload. When the developer's request describes a real application domain, do NOT carry these into the generated classes: replace the example payload and send cadence with the application's real messages and triggers, and keep only the lifecycle pieces the leaf's steps call for (the SIGINT hook on the long-running role, the self-exit on the foreground role). When the request IS a demo or a try-it run, keeping the harness is fine. Two things are NOT demo harness: the `trace(...)` narration method (keep it; applications replace its single body to route narration to their logging or reporting system) and the `VERIFY:` markers, which are emitted through `trace(...)` like every other status line. Because `verify.sh` greps captured stdout for the markers and for the shutdown hook's `Shutdown signal received` proof line (both flow through `trace(...)`), the generated `trace(...)` body MUST stay `System.out.println` through the Step 5 verification stages; rerouting that body to a logger or reporting system is the developer's step AFTER the verify stages pass, never part of generation. Never treat the logging backend (Step 3) as demo harness: the backend and its config file ship in every generated app. +**Demo harness is not application logic.** The samples carry demo-harness elements that exist only to make a standalone run observable: the ENTER-to-quit `System.in` loop, the once-per-second stats-printing thread, the continuous publish loop, and the rotating example payload. When the developer's request describes a real application domain, do NOT carry these into the generated classes: replace the example payload and the publish loop with the application's real messages and triggers, and never add a delay or rate limit between sends that the application does not ask for, and keep only the lifecycle pieces the leaf's steps call for (the SIGINT hook on the long-running role, the self-exit on the foreground role). When the request IS a demo or a try-it run, keeping the harness is fine. Two things are NOT demo harness: the `trace(...)` narration method (keep it; applications replace its single body to route narration to their logging or reporting system) and the `VERIFY:` markers, which are emitted through `trace(...)` like every other status line. Because `verify.sh` greps captured stdout for the markers and for the shutdown hook's `Shutdown signal received` proof line (both flow through `trace(...)`), the generated `trace(...)` body MUST stay `System.out.println` through the Step 5 verification stages; rerouting that body to a logger or reporting system is the developer's step AFTER the verify stages pass, never part of generation. Never treat the logging backend (Step 3) as demo harness: the backend and its config file ship in every generated app. **Comments follow their code.** The samples' comments are part of the reference, not decoration: they carry the documented best practices (the session-independent `createMessage`, the binary payload and the legacy `XML` naming, `getData()` versus `getBytes()`, registering a session event handler at session creation, the ACK/NACK handling options, the correlation-key rule) into the code a developer reads long after this session. Generation carries them under ONE contract, on every door and every app shape: @@ -164,7 +164,7 @@ Invoke the matching stage after generating each class, passing the connection pa 1. After generating the Subscriber: `./verify.sh consumer `. It waits for `VERIFY: QUEUE_BOUND` in the Subscriber output (queue provisioned, subscribed, flow started). 2. After generating the Publisher: `./verify.sh publisher `. NOTE: this stage starts the Subscriber first to bind the queue, so it requires BOTH classes; it waits for `VERIFY: PUBLISH_ACKED` in the Publisher's own output (connect, then publish ACK). -3. At the end: `./verify.sh roundtrip `. It starts the Subscriber, runs the Publisher, and waits for `VERIFY: MESSAGE_RECEIVED` in the Subscriber output (a published PERSISTENT message lands on the queue and is consumed and ACKed). +3. At the end: `./verify.sh roundtrip `. It starts the Subscriber, runs the Publisher, and waits for `VERIFY: PUBLISH_ACKED` in the Publisher output and then a new `VERIFY: MESSAGE_RECEIVED` in the Subscriber output (a published PERSISTENT message lands on the queue and is consumed and ACKed; a message an earlier run left on the durable queue does not count). The stages above (`consumer`, `publisher`, `roundtrip`) verify the guaranteed pub/sub leaves. `verify.sh` also accepts three further stage names for the other leaves: `direct` (the Direct Pub/Sub leaf), `direct-request-reply` (the Request-Reply (Direct) leaf), and `guaranteed-request-reply` (the Request-Reply (Guaranteed) leaf). All are runnable. The two request-reply stages are replier-first single round-trips: each starts the Replier first (the readiness gate on `VERIFY: SUBSCRIBED`), then runs the Requestor in the foreground and passes on `VERIFY: REPLY_RECEIVED` in the Requestor log (with `VERIFY: REQUEST_RECEIVED` in the Replier log as richer evidence). The `guaranteed-request-reply` stage differs only in that the Replier provisions a durable request queue (an endpoint-management denial classifies as an environment failure, exit 2) and the Requestor uses a temporary reply queue plus a blocking `flow.receive` rather than the direct `Requestor` convenience. @@ -182,13 +182,13 @@ PASS_MARKER='VERIFY: MESSAGE_RECEIVED' # marker proving the round trip Generate `verify-hooks.sh` with ONLY these four assignments. The app stage imports exactly these four names from a child shell and ignores everything else in the file, so an extra variable, a `cd`, or a function never reaches the script's own state; keep the file to the four lines all the same. Give a network trigger its own timeout (the `--max-time` above): the stage also bounds the trigger externally and force-kills one that never returns, but a self-timing trigger leaves cleaner evidence. -The `app` stage starts `START_CMD` in the background and captures its output, waits for `VERIFY: CONNECTED` and then `READY_MARKER` (an environment signature or a timeout before that classifies as exit 2 or exit 1), runs `TRIGGER_CMD` under a bounded deadline (a trigger that never returns is force-killed, and the marker watch still decides the verdict), waits for `PASS_MARKER` in the app's captured output, then stops the app with SIGINT and classifies. Set `READY_MARKER`/`PASS_MARKER` to the chosen leaf's own markers. A curl inside `TRIGGER_CMD` is exactly the right use of curl: it triggers, and the markers judge. On the canonical two-class shape, generate `verify-hooks.sh` too, carrying the classic commands (`START_CMD='mvn -q exec:java@subscriber'`, `TRIGGER_CMD='mvn -q exec:java@publisher'`) so the file documents the same contract everywhere; the fixed stages do not read it. +The `app` stage starts `START_CMD` in the background and captures its output, waits for `VERIFY: CONNECTED` and then `READY_MARKER` (an environment signature or a timeout before that classifies as exit 2 or exit 1), runs `TRIGGER_CMD` under a bounded deadline (a trigger that never returns is force-killed, and the marker watch still decides the verdict), waits for a new `PASS_MARKER` in the app's captured output (one already there before the trigger does not count), then stops the app with SIGINT and classifies. Set `READY_MARKER`/`PASS_MARKER` to the chosen leaf's own markers. A curl inside `TRIGGER_CMD` is exactly the right use of curl: it triggers, and the markers judge. On the canonical two-class shape, generate `verify-hooks.sh` too, carrying the classic commands (`START_CMD='mvn -q exec:java@subscriber'`, `TRIGGER_CMD='mvn -q exec:java@publisher'`) so the file documents the same contract everywhere; the fixed stages do not read it. ### Interpret the exit code -- **Exit 0 (stage passes).** The stage milestone appeared and the run tore down cleanly (for the fixed stages that includes the graceful-shutdown proof line and a clean JVM exit, 0 or 130). Advance to the next stage. -- **Exit 1 (code failure).** Compile failed, a marker never appeared, a process died before its milestone, or a run hung and had to be force-killed. Enter the bounded fix loop below. -- **Exit 2 (environment failure).** A doc-traceable JCSMP connection/auth signature appeared (the broker is unreachable or the credentials are wrong). STOP. Do NOT enter the fix loop. The code is not the problem. +- **Exit 0 (stage passes).** The stage milestone appeared in this run and the run tore down cleanly (for the fixed stages that includes a foreground role that exited with status 0 on its own, the graceful-shutdown proof line, and a clean JVM exit, 0 or 130). Advance to the next stage. +- **Exit 1 (code failure).** Compile failed, a marker never appeared, a process died before its milestone, the foreground role exited non-zero, or a run hung and had to be force-killed. Enter the bounded fix loop below. An exit 1 before any process starts is a missing precondition instead (a usage error, or a missing `pom.xml` or `verify-hooks.sh`), and stderr names it: fix that and re-run, without counting it as a fix-loop attempt. +- **Exit 2 (environment failure).** A doc-traceable JCSMP connection/auth signature appeared (the broker is unreachable or the credentials are wrong), or the long-running role was still connecting when the readiness deadline passed and never printed `VERIFY: CONNECTED` (an unreachable host, or a firewall that blocks the port). STOP. Do NOT enter the fix loop. The code is not the problem. **Bounded fix loop (exit 1 only).** Diagnose the captured output, fix the code, and re-run the same stage, up to 3 automatic attempts per stage. After the third failure, stop fixing: summarize exactly what you tried across the attempts and what the captured output shows, then hand the decision to the developer. Report the evidence, not just "it failed". diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-publisher-sample.java b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-publisher-sample.java index 93c618f..69beb5f 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-publisher-sample.java +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-publisher-sample.java @@ -57,7 +57,6 @@ public class DirectPublisher { private static final String APP_NAME = DirectPublisher.class.getSimpleName(); static final String TOPIC_PREFIX = "solace/samples/"; // used as the topic "root" private static final String API = "JCSMP"; - private static final int APPROX_MSG_RATE_PER_SEC = 100; private static final int PAYLOAD_SIZE = 512; // remember to add log4j2.xml to your classpath @@ -189,15 +188,10 @@ private static void runPublishLoop() throws IOException { if (e instanceof JCSMPTransportException) { // all reconnect attempts failed isShutdown = true; // let's quit; or, could initiate a new connection attempt } - } finally { // add a delay between messages - try { - Thread.sleep(1000 / APPROX_MSG_RATE_PER_SEC); // do Thread.sleep(0) for max speed - // Note: STANDARD Edition Solace PubSub+ broker is limited to 10k msg/s max ingress - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); // restore the interrupt status, do not swallow it - isShutdown = true; - } } + // no delay between messages: the loop publishes as fast as send() returns, the way an + // application publishes each message as soon as it has one + // Note: STANDARD Edition Solace broker is limited to 10k msg/s max ingress } isShutdown = true; } diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-requestor-sample.java b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-requestor-sample.java index 3bfe9a2..a564530 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-requestor-sample.java +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-direct-requestor-sample.java @@ -60,8 +60,7 @@ public class DirectRequestor { private static final String API = "JCSMP"; // POSITIVE blocking-request timeout in milliseconds: requestor.request(...) blocks up // to this long for the correlated reply before throwing JCSMPRequestTimeoutException. - // A positive timeout is the direct reliability knob (NOT a burst); 3000 ms matches the - // upstream DirectRequestorBlocking sample. + // A positive timeout is the direct reliability knob (NOT a burst). private static final int REQUEST_TIMEOUT_MS = 3000; private static JCSMPSession session; diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-publisher-sample.java b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-publisher-sample.java index a4d0ef1..e0e8a5f 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-publisher-sample.java +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-publisher-sample.java @@ -58,7 +58,6 @@ public class GuaranteedPublisher { static final String TOPIC_PREFIX = "solace/samples/"; // used as the topic "root" private static final String API = "JCSMP"; private static final int PUBLISH_WINDOW_SIZE = 255; // max guaranteed-publish window (range 1-255) - private static final int APPROX_MSG_RATE_PER_SEC = 100; private static final int PAYLOAD_SIZE = 512; // remember to add log4j2.xml to your classpath @@ -220,15 +219,10 @@ private static void runPublishLoop() throws IOException, InterruptedException { if (e instanceof JCSMPTransportException) { // all reconnect attempts failed isShutdown = true; // let's quit; or, could initiate a new connection attempt } - } finally { // add a delay between messages - try { - Thread.sleep(1000 / APPROX_MSG_RATE_PER_SEC); // do Thread.sleep(0) for max speed - // Note: STANDARD Edition Solace PubSub+ broker is limited to 10k msg/s max ingress - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); // restore the interrupt status, do not swallow it - isShutdown = true; - } } + // no delay between messages: the loop publishes as fast as send() returns, the way an + // application publishes each message as soon as it has one + // Note: STANDARD Edition Solace broker is limited to 10k msg/s max ingress } isShutdown = true; } diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-replier-sample.java b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-replier-sample.java index 471a19c..bfbacd2 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-replier-sample.java +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-replier-sample.java @@ -157,8 +157,7 @@ public void handleEvent(SessionEventArgs event) { final Queue requestQueue = JCSMPFactory.onlyInstance().createQueue(REQUEST_QUEUE_NAME); // ELEVATION: provision the durable request queue in-process and map the request - // topic onto it at startup, so a fresh broker works out of the box (the upstream - // sample provisions with flag 0, which throws "already exists" on a re-run). + // topic onto it at startup, so a fresh broker works out of the box. // best practice: confirm the broker allows client-side endpoint management first if (!session.isCapable(CapabilityType.ENDPOINT_MANAGEMENT)) { System.err.println("This client/broker does not allow client-side endpoint management; " @@ -168,8 +167,8 @@ public void handleEvent(SessionEventArgs event) { EndpointProperties endpointProps = new EndpointProperties(); endpointProps.setAccessType(EndpointProperties.ACCESSTYPE_EXCLUSIVE); // single-replier sample endpointProps.setPermission(EndpointProperties.PERMISSION_CONSUME); - // provision the durable request queue IDEMPOTENTLY so a re-run is safe (NOT the - // upstream flag 0, which throws on the second run) + // provision the durable request queue IDEMPOTENTLY so a re-run is safe (flag 0 would + // throw "already exists" on the second run) session.provision(requestQueue, endpointProps, JCSMPSession.FLAG_IGNORE_ALREADY_EXISTS); // map the request topic onto the queue with a subscription (PERSISTENT pub/sub onto // a queue); WAIT_FOR_CONFIRM blocks until the broker confirms the subscription diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-subscriber-sample.java b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-subscriber-sample.java index 6f13cbd..b2471c2 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-subscriber-sample.java +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/jcsmp-guaranteed-subscriber-sample.java @@ -153,8 +153,7 @@ public void handleEvent(SessionEventArgs event) { final Queue queue = JCSMPFactory.onlyInstance().createQueue(QUEUE_NAME); // ELEVATION: provision the durable queue in-process and map the topic onto it - // at startup, so a fresh broker works out of the box (the upstream sample expects - // the queue to pre-exist and errors out otherwise). + // at startup, so a fresh broker works out of the box. // best practice: confirm the broker allows client-side endpoint management first if (!session.isCapable(CapabilityType.ENDPOINT_MANAGEMENT)) { System.err.println("This client/broker does not allow client-side endpoint management; " diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/prerequisites.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/prerequisites.md index f817f62..0902c14 100644 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/prerequisites.md +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/prerequisites.md @@ -11,7 +11,7 @@ Read this when a developer does not yet have a broker to build against. Both Des You need one reachable broker. The primary, recommended choice is [Solace Cloud](https://docs.solace.com/Get-Started/Getting-Started-Try-Broker.md): it is the simplest path to a running broker for a greenfield app and for the run-and-observe round-trip. Default to it unless the developer has a reason not to. -As brief alternatives, the same page also documents a self-hosted Software Broker (run locally via container or VM) and an Appliance (existing hardware the developer already operates) for developers who cannot use Solace Cloud. +As a brief alternative, the same page also covers setting up a free broker locally, for developers who cannot use Solace Cloud. The skill does not provision or configure the broker. It assumes the broker is reachable and that the developer has connection details (host, message VPN, client username, password). A broker discovered running in the environment (for example a local container) is a fact to report, never an answer: still ask which broker the developer wants to target. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/scripts/verify.sh b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/scripts/verify.sh index 3e915fc..c21f173 100755 --- a/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/scripts/verify.sh +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/scripts/verify.sh @@ -24,16 +24,22 @@ # foreground and lets it exit on its own, watches each process's OWN captured # output for a fixed VERIFY: marker line, sends SIGINT to the subscriber to # exercise its graceful-shutdown hook, and classifies the outcome by exit code: -# 0 = stage passed (marker seen, the shutdown hook ran, AND the subscriber -# exited cleanly. A clean exit is exit code 0 OR 130: the JVM reports 130 +# 0 = stage passed (marker seen in this run, the foreground role exited with +# status 0 on its own, the shutdown hook ran, AND the subscriber exited +# cleanly. A clean exit is exit code 0 OR 130: the JVM reports 130 # (128+SIGINT) when it terminates AFTER running its shutdown hooks, which -# is the expected, graceful outcome here) +# is the expected, graceful outcome here. The app stage instead passes on +# READY_MARKER, then a new PASS_MARKER after the trigger, and a bounded +# teardown; the app's shutdown is not checked) # 1 = code failure (compile failed, a marker never appeared, a process died -# before its milestone, the shutdown hook did not run, or a run hung and -# had to be force-killed) +# before its milestone, the foreground role exited non-zero, the shutdown +# hook did not run, or a run hung and had to be force-killed), OR a +# precondition miss before any process starts (a usage error, or a missing +# pom.xml or verify-hooks.sh); stderr names which # 2 = environment failure (broker/credentials: a doc-traceable JCSMP error -# signature appeared in either process log; the code is not the problem, -# so the agent must NOT enter the fix loop) +# signature appeared in either process log, or the long-running role was +# still connecting at the readiness deadline with no VERIFY: CONNECTED; the +# code is not the problem, so the agent must NOT enter the fix loop) # # SIGINT delivery under `mvn exec:java` (the D-06 mechanism, why `set -m`): # in a NON-interactive shell without job control, a command started in the @@ -215,7 +221,8 @@ echo "── Live-run heads-up: stage '$STAGE' starts processes that connect to # stage waits for a role-specific milestone in that role's OWN log: # publisher → VERIFY: PUBLISH_ACKED in PUB_LOG (the publish ACK callback fired) # consumer → VERIFY: QUEUE_BOUND in SUB_LOG (flow.start() bound the durable queue) -# roundtrip → VERIFY: MESSAGE_RECEIVED in SUB_LOG (a published message was consumed + ACKed) +# roundtrip → VERIFY: PUBLISH_ACKED in PUB_LOG, then a NEW VERIFY: MESSAGE_RECEIVED in SUB_LOG +# (a published message was consumed + ACKed; a stale queued one does not count) # Reaching any stage marker already proves the connect happened for that process. # These strings form a fixed contract with the VERIFY: markers that # implement-mode.md Step 4 instructs the generated app to println. The two MUST @@ -223,7 +230,7 @@ echo "── Live-run heads-up: stage '$STAGE' starts processes that connect to case "$STAGE" in publisher) ;; # waits for VERIFY: PUBLISH_ACKED in PUB_LOG consumer) ;; # waits for VERIFY: QUEUE_BOUND in SUB_LOG - roundtrip) ;; # waits for VERIFY: MESSAGE_RECEIVED in SUB_LOG + roundtrip) ;; # waits for VERIFY: PUBLISH_ACKED, then a new VERIFY: MESSAGE_RECEIVED in SUB_LOG direct) ;; # direct pub/sub stage direct-request-reply) ;; # direct request-reply stage guaranteed-request-reply) ;; # guaranteed request-reply stage @@ -278,6 +285,32 @@ TIMEOUT_S=30 # marker-watch deadline (Claude's discretion per CONTEXT) PUBLISH_WAIT_S=30 # foreground-publisher deadline before escalating to a code failure SHUTDOWN_WAIT_S=15 # bounded post-SIGINT wait before escalating to SIGTERM/SIGKILL +# A long-running role that is still alive at the readiness deadline and never +# printed VERIFY: CONNECTED is still connecting: an unreachable host or a blocked +# port, where the connect attempt (and its retries) outlasts TIMEOUT_S without +# logging an error. That is the environment, not the code, so exit 2. +classify_stalled_connect() { # + if [ "$status" -ne 2 ] && [ "$3" -eq 0 ] && kill -0 "$1" 2>/dev/null && ! grep -qF "VERIFY: CONNECTED" "$2"; then + status=2 + CONNECT_STALLED=1 + fi +} + +# Print the count of in once it has stopped changing for 2 seconds +# (bounded by TIMEOUT_S), so messages an earlier run left on a durable queue, which +# arrive just after the bind, are all counted before the publisher starts. +settled_count() { # + local n m settle_deadline=$(( SECONDS + TIMEOUT_S )) + n=$(grep -cF "$2" "$1" || true) + while [ "$SECONDS" -lt "$settle_deadline" ]; do + sleep 2 + m=$(grep -cF "$2" "$1" || true) + [ "$m" -eq "$n" ] && break + n=$m + done + echo "$n" +} + # ── Bounded long-running shutdown (shared by the env-failure teardown and STAGE D) ─ # Send SIGINT to the process GROUP of the PID passed as $1, poll up to # SHUTDOWN_WAIT_S for the JVM to exit on its own, then escalate SIGTERM then @@ -379,6 +412,7 @@ trap cleanup EXIT status=1 # default to code failure until a stage milestone is confirmed DIRECT_ZERO_RECEIPT=0 # set by run_direct_stage when the at-most-once message dropped post-gate (D-05) RR_NO_REPLY=0 # set by run_request_reply_stage when the replier subscribed but no reply came back +CONNECT_STALLED=0 # set when the long-running role was still connecting at the readiness deadline # ── run_guaranteed_stage: the guaranteed pub/sub flow (publisher|consumer|roundtrip) ─ # The entire STAGE A..D body is lifted here VERBATIM (GEN-06 byte-stability): no @@ -428,6 +462,7 @@ while [ "$SECONDS" -lt "$deadline" ]; do fi sleep 1 done +classify_stalled_connect "$SUB_PID" "$SUB_LOG" "$bound" # If the broker/credentials are wrong the subscriber already proved it; tear it # down (bounded, same SIGINT/SIGTERM/SIGKILL escalation as STAGE D) and classify @@ -448,7 +483,13 @@ else # (subscriber-first ordering, D-07-R11). The wait is BOUNDED: a publisher that # hangs past PUBLISH_WAIT_S is escalated and classified as a code failure so # the unattended script NEVER hangs. + published=0 + stale=0 if { [ "$STAGE" = "publisher" ] || [ "$STAGE" = "roundtrip" ]; } && [ "$bound" -eq 1 ]; then + # A receipt already in the log is a message an earlier run left on the durable + # queue, delivered at bind time; the round trip must see a receipt beyond these. + stale=$(settled_count "$SUB_LOG" "VERIFY: MESSAGE_RECEIVED") + [ "$stale" -eq 0 ] || echo "── $stale message(s) left on the queue by an earlier run arrived before the publisher started; they do not count ──" >&2 echo "── Running publisher (mvn -q exec:java@publisher), watching for: VERIFY: PUBLISH_ACKED ──" set -m if [ "$TWO_PROJECT" -eq 1 ]; then @@ -479,32 +520,37 @@ else if [ "$pub_hung" -eq 1 ]; then # The publisher did not exit on its own within the deadline; escalate - # and treat the hang as a code failure (status stays 1). + # and record the hang (pub_rc 124), which fails the stage. echo "── Publisher did not exit within ${PUBLISH_WAIT_S}s; escalating (SIGTERM, then SIGKILL) ──" >&2 kill -TERM "$PUB_PID" 2>/dev/null || true sleep 2 kill -KILL "$PUB_PID" 2>/dev/null || true wait "$PUB_PID" 2>/dev/null || true + pub_rc=124 # synthetic "timed out" code; never a clean exit else - wait "$PUB_PID" 2>/dev/null || true + pub_rc=0 + wait "$PUB_PID" 2>/dev/null || pub_rc=$? fi if grep -qE "$ENV_SIGNATURES" "$PUB_LOG"; then status=2 # environment failure (broker/credentials), bypass the fix loop - elif [ "$STAGE" = "publisher" ] && grep -qF "VERIFY: PUBLISH_ACKED" "$PUB_LOG"; then - status=0 # the publish ACK callback fired + elif [ "$pub_rc" -eq 0 ] && grep -qF "VERIFY: PUBLISH_ACKED" "$PUB_LOG"; then + published=1 # the publish ACK callback fired and the publisher exited 0 + if [ "$STAGE" = "publisher" ]; then status=0; fi fi fi # ── STAGE C: confirm receipt on the SUBSCRIBER's log (roundtrip only) ──────── - if [ "$STAGE" = "roundtrip" ] && [ "$status" -ne 2 ]; then + # One receipt more than before the publisher ran. A publisher that never acked, + # exited non-zero, or hung cannot pass the round trip. + if [ "$STAGE" = "roundtrip" ] && [ "$published" -eq 1 ]; then rt_deadline=$(( SECONDS + TIMEOUT_S )) while [ "$SECONDS" -lt "$rt_deadline" ]; do if grep -qE "$ENV_SIGNATURES" "$SUB_LOG"; then status=2 break fi - if grep -qF "VERIFY: MESSAGE_RECEIVED" "$SUB_LOG"; then + if [ "$(grep -cF "VERIFY: MESSAGE_RECEIVED" "$SUB_LOG" || true)" -gt "$stale" ]; then status=0 # a published message was consumed + ACKed break fi @@ -598,6 +644,7 @@ run_direct_stage() { fi sleep 1 done + classify_stalled_connect "$SUB_PID" "$SUB_LOG" "$subscribed" if [ "$status" -eq 2 ]; then # Broker/credentials are wrong; tear the subscriber down (bounded) and @@ -639,8 +686,10 @@ run_direct_stage() { sleep 2 kill -KILL "$PUB_PID" 2>/dev/null || true wait "$PUB_PID" 2>/dev/null || true + pub_rc=124 # synthetic "timed out" code; never a clean exit else - wait "$PUB_PID" 2>/dev/null || true + pub_rc=0 + wait "$PUB_PID" 2>/dev/null || pub_rc=$? fi # ── Pass condition: at least one VERIFY: MESSAGE_RECEIVED in the subscriber ─ @@ -652,6 +701,9 @@ run_direct_stage() { status=2 # environment failure (broker/credentials), bypass the fix loop break fi + if [ "$pub_rc" -ne 0 ]; then + break # the publisher exited non-zero or hung (code failure) + fi if grep -qF "VERIFY: MESSAGE_RECEIVED" "$SUB_LOG"; then status=0 # at least one direct message was received (D-04 pass) break @@ -666,7 +718,7 @@ run_direct_stage() { # clean, but not one message of the burst arrived) stays status 1; the case-1 print is # annotated with the Direct at-most-once caveat below (D-05). An env match # already set status 2 above. - if [ "$status" -ne 0 ] && [ "$status" -ne 2 ]; then + if [ "$status" -ne 0 ] && [ "$status" -ne 2 ] && [ "$pub_rc" -eq 0 ]; then DIRECT_ZERO_RECEIPT=1 fi @@ -775,6 +827,7 @@ run_request_reply_stage() { fi sleep 1 done + classify_stalled_connect "$SUB_PID" "$SUB_LOG" "$subscribed" if [ "$status" -eq 2 ]; then # Broker/credentials are wrong; tear the replier down (bounded) and classify @@ -819,8 +872,10 @@ run_request_reply_stage() { sleep 2 kill -KILL "$PUB_PID" 2>/dev/null || true wait "$PUB_PID" 2>/dev/null || true + pub_rc=124 # synthetic "timed out" code; never a clean exit else - wait "$PUB_PID" 2>/dev/null || true + pub_rc=0 + wait "$PUB_PID" 2>/dev/null || pub_rc=$? fi # ── Pass condition: VERIFY: REPLY_RECEIVED in the requestor's OWN log, AND ─ @@ -828,14 +883,14 @@ run_request_reply_stage() { # match in either log maps to status 2. if grep -qE "$ENV_SIGNATURES" "$SUB_LOG" || grep -qE "$ENV_SIGNATURES" "$PUB_LOG"; then status=2 # environment failure (broker/credentials), bypass the fix loop - elif grep -qF "VERIFY: REPLY_RECEIVED" "$PUB_LOG" && grep -qF "VERIFY: REQUEST_RECEIVED" "$SUB_LOG"; then + elif [ "$pub_rc" -eq 0 ] && grep -qF "VERIFY: REPLY_RECEIVED" "$PUB_LOG" && grep -qF "VERIFY: REQUEST_RECEIVED" "$SUB_LOG"; then status=0 # the request landed on the replier and the correlated reply came back fi # A clean requestor run with no reply (subscription was confirmed, the requestor # ran and self-exited, but no VERIFY: REPLY_RECEIVED) stays status 1; the case-1 # print is annotated with the no-reply caveat below. An env match already set 2. - if [ "$status" -ne 0 ] && [ "$status" -ne 2 ]; then + if [ "$status" -ne 0 ] && [ "$status" -ne 2 ] && [ "$pub_rc" -ne 124 ] && ! grep -qF "VERIFY: REPLY_RECEIVED" "$PUB_LOG"; then RR_NO_REPLY=1 fi @@ -933,6 +988,10 @@ run_app_stage() { # response never came back. Reusing PUB_PID (not a new name) keeps the # EXIT trap's group-kill coverage over an orphaned trigger. sleep 2 + # A PASS_MARKER already in the log (a backlog the app consumed at startup) + # does not count; the trigger must produce a new one. + local seen + seen=$(settled_count "$SUB_LOG" "$PASS_MARKER") echo "── Running trigger (TRIGGER_CMD from verify-hooks.sh), watching for: $PASS_MARKER ──" set -m bash -c "$TRIGGER_CMD" >"$PUB_LOG" 2>&1 & @@ -957,7 +1016,7 @@ run_app_stage() { status=2 break fi - if grep -qF "$PASS_MARKER" "$SUB_LOG"; then + if [ "$(grep -cF "$PASS_MARKER" "$SUB_LOG" || true)" -gt "$seen" ]; then status=0 # the pass marker appeared in the app's own captured output break fi @@ -986,6 +1045,13 @@ case "$STAGE" in *) echo "unknown stage: $STAGE" >&2; usage ;; esac +# A connection error can be logged only after the stage gave up (for example during +# the SIGINT teardown, while the API was still retrying the connect), so look for +# the environment signatures again before classifying a code failure. +if [ "$status" -eq 1 ] && { grep -qE "$ENV_SIGNATURES" "$SUB_LOG" || grep -qE "$ENV_SIGNATURES" "$PUB_LOG"; }; then + status=2 +fi + # ── Print the captured evidence, exit classified (the EXIT trap removes the ──── # temp logs, V8) ───────────────────────────────────────────────────────────────── echo "── Captured subscriber output ──────────────────────────────────────" @@ -999,17 +1065,25 @@ case "$status" in if [ "$STAGE" = "app" ]; then echo "PASS (app): the readiness and pass markers were observed in the app's own captured output, and the app was stopped with a bounded teardown." else - echo "PASS ($STAGE): the stage milestone was observed in the role's own log, the subscriber's shutdown hook ran ('$SHUTDOWN_LINE'), and the subscriber exited cleanly (exit 0 or 130)." + echo "PASS ($STAGE): the stage milestone was observed in the role's own log, the foreground role exited with status 0, the subscriber's shutdown hook ran ('$SHUTDOWN_LINE'), and the subscriber exited cleanly (exit 0 or 130)." + fi + ;; + 2) + if [ "$CONNECT_STALLED" -eq 1 ]; then + echo "ENV FAILURE ($STAGE): the long-running role was still connecting to ${HOST:-the broker in config.json} after ${TIMEOUT_S}s and never printed VERIFY: CONNECTED. The host is unreachable or a firewall blocks the port (connect retries can stretch this). Check the host and port. NOT entering the fix loop." >&2 + else + echo "ENV FAILURE ($STAGE): a JCSMP connection/auth signature matched in a process log (broker or credentials are wrong, not the code). NOT entering the fix loop." >&2 fi ;; - 2) echo "ENV FAILURE ($STAGE): a JCSMP connection/auth signature matched in a process log (broker or credentials are wrong, not the code). NOT entering the fix loop." >&2 ;; *) if [ "$DIRECT_ZERO_RECEIPT" -eq 1 ]; then echo "CODE FAILURE ($STAGE): the subscription was confirmed (VERIFY: SUBSCRIBED) and the direct publisher ran clean, but not one message of the burst was received (VERIFY: MESSAGE_RECEIVED never appeared). Direct messaging is at-most-once: a whole burst dropped after a confirmed subscription points at the topic subscription, the topic string, or a slow-consumer egress discard, not a transient race. Check that the subscriber's subscription matches the publisher's topic." >&2 elif [ "$RR_NO_REPLY" -eq 1 ]; then echo "CODE FAILURE ($STAGE): the replier subscribed (VERIFY: SUBSCRIBED) and the requestor ran and self-exited, but no correlated reply came back (VERIFY: REPLY_RECEIVED never appeared in the requestor log). Check the requestor's request topic against the replier's subscription, the reply-to handling (the requestor's createRequestor / the replier's sendReply onto getReplyTo), and whether VERIFY: REQUEST_RECEIVED appeared in the replier log: its absence means the request never reached the replier, its presence means the reply was lost on the way back." >&2 else - echo "CODE FAILURE ($STAGE): the stage milestone was not observed, a process died before its milestone, the shutdown hook did not run, or a run hung and had to be force-killed." >&2 + fg_note="" + [ "$STAGE" = "app" ] || [ "${pub_rc:-0}" -eq 0 ] || fg_note=" The foreground role exited with status $pub_rc (124: it did not exit on its own and was force-killed)." + echo "CODE FAILURE ($STAGE): the stage milestone was not observed, a process died before its milestone, the foreground role exited non-zero, the shutdown hook did not run, or a run hung and had to be force-killed.$fg_note" >&2 fi ;; esac diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/python.md new file mode 100644 index 0000000..b774b00 --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python.md @@ -0,0 +1,83 @@ +# Python API + +The Python entry file of the Solace API development skill. The Solace Messaging API for Python is published on PyPI as `solace-pubsubplus`. This file detects the developer's intent and selects the right Python mode; the detailed guidance lives in the lazy-loaded mode files under `python/`. + +The cross-cutting invariants (content sourcing, WebFetch-on-demand doc grounding, and no hardcoded versions) are owned by the umbrella `SKILL.md`. Apply them here; this file does not restate them. The Python-specific points below are the package to depend on and the canonical pages to ground in. + +## Mode Detection + +Determine the user's intent and enter the appropriate mode. Routing is keyed on the design contract, not on the phrasing of the request: Implement mode never starts without a confirmed design (the design-contract gate below), so a "build me..." prompt is NOT an Implement signal by itself. + +| User intent | Mode | What to do | +|---|---|---| +| "Help me choose the Solace messaging pattern for a new Python app" / "Topic or queue?" / "What delivery semantics should I use?" | **Design** | Read `python/design-mode.md` | +| Build or generate a NEW app or messaging component with NO valid design contract yet ("Build me a Solace Python publisher/consumer", "Generate a Python project with solace-pubsubplus", a web app / dashboard / service, or any embedded shape where the Python messaging layer lives inside a larger application) | **Design first** | Read `python/design-mode.md`. A prompt that already states the design resolves on its fully-specified path in ONE confirm; a bare prompt walks the tree. The confirmed summary then feeds Implement. | +| A build request WITH a valid design contract (see the design-contract gate below) | **Implement** | Read `python/implement-mode.md`; for an embedded shape generate the messaging layer per its leaf rules and verify with the `app` stage of `verify.sh` (its Step 5) | +| An edit to an EXISTING app that changes the messaging topology (the topology rule below) | **Design first** | Re-enter `python/design-mode.md` to re-confirm the affected summary fields, then Implement applies the change | +| A mechanical edit to an EXISTING app with no topology change (reconnect handling, payload format, logging, renames, ack tuning) | **Implement** | Read `python/implement-mode.md`; no design pass | +| "My Python app is throwing on connect" / "Why is my receiver not binding the queue?" | **Debug** | Debug mode is not yet available in this release. Redirect the user to the canonical Solace Python API documentation: [Python API Home](https://docs.solace.com/API/Messaging-APIs/Python-API/python-home.md). The redirect is the ENTIRE debug answer: acknowledge the problem in one line, link the page, and stop. Do not generate debugging guidance from memory. That prohibition covers root-cause hypotheses, likely causes, interim checks or "things worth checking in the meantime", workarounds, and configuration or code suggestions. Restating this rule does not license adding such guidance after the redirect. | + +If unclear, default to **Design**. Understand the messaging problem before generating code. + +**The topology rule** decides between the two existing-app rows. An edit changes the messaging topology when it changes the structure the design summary records: a new app or messaging component (a new publisher, receiver, requestor, or replier, including a leaf change such as single-service growing into fan-out), a pattern change, a delivery-mode change, a consumption-endpoint change, or a queue access-type change. That list is the core; also treat an unlisted change as topology when your judgment says it alters the message flow between the apps and the broker. Everything else is a mechanical edit. + +Three gates hold on every path: + +- **The design-contract gate.** Implement mode never starts without a design contract, and never invents design values. Exactly three sources satisfy it: (1) a summary Design mode confirmed in this session: use it as-is, with no re-confirm; (2) an explicit summary the developer supplied in chat (every field of the summary or an equivalent statement of them): use it directly; (3) a saved `solace-design.md` in the project: restate it in one line and proceed on confirm. A summary with no `Threading` field reads as `Threading: Async`; Implement mode derives it and does not ask. A prose build request that merely mentions pattern details is NOT a contract: route it through Design mode, whose fully-specified path derives the summary, echoes it, and takes one confirm. The one carve-out is the mechanical-edit row above. An experienced developer therefore never has to walk the design tree, but never skips the contract either. A request to skip the design, skip all questions, or generate the full project right now does not waive this gate or the door question below: write no file before the confirm. Take the shortest path the gate allows instead: derive the summary from the prompt (an unstated `Threading` records `Async`, named as the default; `python/design-mode.md` question 3), present it, say in one line that one confirm starts the build, and wait for that confirm. +- **Implement mode opens with a mandatory door question.** Ask Quickstart, Solace Suggested, or Custom, before you request or accept ANY broker details (`python/implement-mode.md` Step 0). Ask it even when a `config.json` with credentials already exists, even when a local broker is already running, and even when the requested app is bigger than the canonical generated shape; none of those answers the question. A confirmed design summary does not answer it either: the door question always follows the design contract, never merges into it. +- **Environment discovery never answers a question.** A running broker container, an existing config file, or found credentials are facts to report, not answers to consume. Report what you found, then still ask which broker the developer wants to target. + +## Python package + +The single Solace dependency for this API is the PyPI package `solace-pubsubplus`, imported as `solace.messaging`. Resolve the latest release at generation time (the no-hardcoded-versions invariant); never pin a `solace-pubsubplus` version in skill content. Resolve it from the authoritative PyPI JSON API with exactly this command, and read `info.version`: + +```bash +curl -s https://pypi.org/pypi/solace-pubsubplus/json \ + | python3 -c 'import json, sys; print(json.load(sys.stdin)["info"]["version"])' +``` + +The package's `requires_python` value on PyPI is an install floor, not a support statement. Solace supports only the Python versions that are in active support and also have security update support (see `python/prerequisites.md`). + +## Verify script + +The Python API bundles `python/scripts/verify.sh`, a stage-dispatched run-and-observe script (`verify.sh ...`) that Implement mode copies into every generated project and runs against a reachable broker (`python/implement-mode.md` Step 5). It is a separate script from the JCSMP one: it runs each project's `.venv` interpreter, not Maven. Without real broker values, Implement mode ends in the compile-only fallback of the same step and never claims a verified run (SKILL.md Invariant 5). + +## Canonical doc links + +Python-specific grounding. Each page below is a live `docs.solace.com` `.md` URL, WebFetched on demand; the live exceptions are listed separately because they have no `docs.solace.com` `.md` form. + +- [Python API Home](https://docs.solace.com/API/Messaging-APIs/Python-API/python-home.md): entry point for the Python API. +- [Python API Developer Guide](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Dev-Guide.md): the index of the developer guide pages. +- [Messaging Service](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Messaging-Service.md): building and connecting the `MessagingService`. +- [Publishing Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Publish.md) and [Consuming Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Receive.md): the guaranteed pub/sub pages. +- [Request-Reply Messaging in the Solace Python API](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Request-Reply.md): request-reply in the Python API, which uses direct messages only. +- [Creating Queues with the Solace Python API](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Create-Queues.md): queue provisioning from the client. +- [Supported Environments](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-supported-Environments.md): Python versions, platforms, and OpenSSL. +- [Solace C API Best Practices](https://docs.solace.com/API/API-Developer-Guide/C-API-Best-Practices.md): the Python API has no best practices page of its own. It wraps the C API, so the reference samples ground their callback, reconnect, acknowledgement, and time-to-live practices here. +- For the broader Solace grounding, see the shared [Solace Core Concepts](https://docs.solace.com/Get-Started/event-mesh-basics.md) page. + +Live exceptions (these have no `docs.solace.com` `.md` form; use the live URL directly): + +- [Python API reference](https://docs.solace.com/API-Developer-Online-Ref-Documentation/python/index.html) (Sphinx HTML, one large page per sub-package) +- [Python API Release Notes](https://products.solace.com/download/PYTHON_API_RN) +- [solace-pubsubplus on PyPI](https://pypi.org/project/solace-pubsubplus/) (its project description carries the `multiprocessing` restriction, which no `docs.solace.com` page states) + +## Mode and reference files (read on-demand only) + +- `python/prerequisites.md`: broker acquisition and the supported environments (route here first if the developer has no reachable broker). +- `python/design-mode.md`: choose and confirm the messaging pattern and topology before any code is generated; every new build without a design contract routes here first. +- `python/implement-mode.md`: generate a runnable Python project from a confirmed design; its Step 4 dispatches onto the per-leaf wiring files below. +- `python/implement-guaranteed-pubsub.md`: the Guaranteed Pub/Sub leaf wiring (read from implement-mode Step 4). +- `python/implement-direct-pubsub.md`: the Direct Pub/Sub leaf wiring (read from implement-mode Step 4). +- `python/implement-request-reply.md`: the Request-Reply (Direct) leaf wiring (read from implement-mode Step 4). +- `python/solace-suggested-mode.md`: the leaf-agnostic Solace Suggested overlay (TLS secure session with the developer's trust store, DMQ eligibility on persistent publishes, HA failover, separate Python projects) reached from implement-mode Step 0 on the Solace Suggested path. +- `python/custom-mode.md`: the thin a la carte Custom overlay reached from implement-mode Step 0 on the Custom path; a leaf-aware checklist of the hardening knobs (secure session, DMQ, HA, decoupled projects, admin-provisioned queue) that generates exactly the ticked subset, reusing the Solace Suggested overlay steps for the mapped knobs. +- `python/python-guaranteed-publisher-sample.py`: best-practices publisher sample (basic-auth connect, PERSISTENT publish to a topic, publish receipt handling, graceful shutdown). +- `python/python-guaranteed-subscriber-sample.py`: best-practices receiver sample (provisions a durable queue plus a topic subscription, CLIENT-ack receiver). +- `python/python-guaranteed-publisher-blocking-sample.py` and `python/python-guaranteed-subscriber-blocking-sample.py`: the guaranteed pub/sub samples for the minimal-threading shape (`Threading: Minimal (blocking publish and receive)` in the design summary): blocking `publish_await_acknowledgement()` per message, and a main loop that pulls with `receive_message()` and ACKs on the application's own thread. +- `python/python-solace-connection-config.py`: the shared connection-config helper every generated app uses. +- `python/python-direct-publisher-sample.py` and `python/python-direct-subscriber-sample.py`: the direct pub/sub samples (bounded-buffer publish with a publish failure listener; direct topic subscription with discard detection). +- `python/python-direct-subscriber-blocking-sample.py`: the direct subscriber for the minimal-threading shape (a main loop that pulls with `receive_message()`); the direct publisher is the same sample on both shapes. +- `python/python-direct-requestor-blocking-sample.py`, `python/python-direct-requestor-sample.py`, and `python/python-direct-replier-sample.py`: the direct request-reply samples (blocking and non-blocking requestor; replier answering with `replier.reply()`). +- `python/verification-checklist.md`: the master verification checklist template that implement-mode Step 6 tailors into every generated project. +- `python/scripts/verify.sh`: the bundled verify script that Implement mode copies into every generated project; the `app` stage drives web and embedded shapes through a generated `verify-hooks.sh`. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/custom-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/custom-mode.md new file mode 100644 index 0000000..cecbeeb --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/custom-mode.md @@ -0,0 +1,69 @@ +# Custom Mode + +The Custom overlay for Implement mode of the Python API. This file is reached from `implement-mode.md` Step 0 when the developer chooses the Custom door. Custom is a la carte: the developer ticks which hardening knobs to include, and generation applies exactly the ticked subset and nothing more. It is a thin overlay. It does NOT re-describe how each knob works. For the four knobs that map to a `solace-suggested-mode.md` step, it points at that step and reuses it, so `solace-suggested-mode.md` stays the single source of truth for each hardening delta. + +This overlay defers to `implement-mode.md` for all leaf and sample mechanics, exactly the way `solace-suggested-mode.md` does: the Step 2 design-summary input contract, the Step 4 leaf dispatch (which leaf wiring file each leaf reads, which sample pair the `Threading` field selects, and the near-verbatim adaptation discipline), the AI-assisted disclaimer header on every generated script, explicit imports (never `from solace.messaging... import *`), and the fixed `VERIFY:` marker contract. Custom changes NO `VERIFY:` marker string. Any live run on the Custom path prints the same one-line live-broker heads-up (SKILL.md Invariant 6) immediately before it starts: name the broker host and the run's actual broker-side effects. + +## The checklist model: one up-front multi-select + +When the developer picks Custom, establish the design summary first (per `implement-mode.md` Step 2, or reuse one already in hand), THEN present a SINGLE up-front multi-select checklist of the knobs that apply to the resulting `Pattern` leaf, capture the ticked subset once, and generate exactly that subset. Do NOT walk the knobs one at a time as sequential per-knob questions. One checklist, one capture, one generation pass. The property that defines Custom is "exactly the ticked subset, nothing more", so a knob that was not ticked leaves the generated output at its base shape for that concern. + +## The five knobs + +Custom offers FIVE knobs, and only these five. There is no sixth knob and no authentication knob; basic username/password is the only auth this journey generates, exactly as on Quickstart and Solace Suggested, so it is never a Custom checklist item. Four of the five knobs reuse a `solace-suggested-mode.md` step; the fifth is defined in this file: + +1. **Secure session (TLS).** Apply `solace-suggested-mode.md` Step 1: a `tcps://` host, explicit certificate validation, and the developer's trust store directory in `config.json`. Do not restate that logic here; the overlay step is the source of truth. +2. **HA reconnect values.** Apply `solace-suggested-mode.md` Step 2: the HA-failover question and the retry strategies and retries-per-host value it gates. Unticked keeps the samples' reconnect baseline (20 retries 3000 ms apart), not the fail-fast API defaults; ticked applies the HA values when the HA-failover question is answered yes. +3. **DMQ eligibility.** Apply `solace-suggested-mode.md` Step 3: `PERSISTENT_DMQ_ELIGIBLE` set explicitly on the persistent publisher's message template, with the broker-side prerequisite stated as the developer's responsibility. +4. **Decoupled projects.** Apply `solace-suggested-mode.md` Step 4: separate, independent Python projects, one per role. +5. **Admin-provisioned queue.** The one knob that is a real code change rather than a reused overlay step. It is defined in full in the "Admin-provisioned-queue knob" section below. + +For knobs 1 through 4, point at the named overlay step and reuse it. Do NOT duplicate the knob logic in this file. + +## The Quickstart-shaped floor + +Zero knobs ticked yields a Quickstart-shaped base app: one Python project with both scripts, a self-provisioned durable queue on the guaranteed leaves, the Quickstart transport rule (the scheme the developer's host carries, asked in `implement-mode.md` Step 1), and the `config.json`-or-command-line connection convention. Each ticked knob layers its Suggested delta on top of that floor. There is NO minimum-one-knob guard: zero ticks is a valid result, and it is the Quickstart-shaped base app. One exception to the Quickstart shape: the floor KEEPS the reference samples' reconnect baseline (20 retries 3000 ms apart); the fail-fast API-default posture is the Quickstart door's alone, and on Custom the HA knob decides between that baseline (unticked) and the HA values (ticked). Custom therefore spans from about Quickstart (nothing ticked) to about Solace Suggested (everything ticked). + +When the secure-session knob is unticked, the base follows the Quickstart transport rule: `implement-mode.md` Step 1 asks the scheme. On a plaintext `tcp://` host the base carries an honest plaintext caveat: unencrypted transport, suitable for a dev or test broker with no TLS, not for real credentials or real data over untrusted networks. On a `tcps://` host, apply the Quickstart trust store guidance from `implement-mode.md` Step 0 and never downgrade the host to `tcp://`. When the secure-session knob is ticked, the session upgrades to the `solace-suggested-mode.md` Step 1 `tcps://` path. + +The command-line fallback of the shared helper follows the knobs: it stays when both the secure-session and the decoupled-projects knobs are unticked (the Quickstart convention), and it is dropped, as on Solace Suggested, when either one is ticked, so the trust store path and the per-project credentials come only from each project's gitignored `config.json`. The helper requires the trust store key only when the secure-session knob is ticked. With the knob unticked on a `tcps://` host, the helper passes the trust store key through from `config.json` as on Quickstart, and the command-line fallback cannot carry it. + +## Leaf-aware offering + +Only offer knobs that apply to the design summary's chosen `Pattern` leaf. Read the `Pattern` field the same way `implement-mode.md` Step 2 does, then filter the checklist: + +- Secure session, HA reconnect values, and decoupled projects apply on EVERY leaf. +- DMQ eligibility and the admin-provisioned queue apply ONLY on the leaves that carry persistent delivery and a durable queue: the three Guaranteed Pub/Sub leaves. The Python API has no guaranteed request-reply leaf. +- HIDE both DMQ eligibility and the admin-provisioned queue on the direct at-most-once leaves (Direct Pub/Sub and Request-Reply (Direct)). Those leaves have no queue, so both knobs would be a silent no-op there; a developer must not be able to tick a knob that does nothing. + +## Admin-provisioned-queue knob (the one real code change) + +Ticking the admin-provisioned-queue knob changes the generated code so the subscriber binds a pre-existing, administrator-owned queue instead of provisioning its own. The administrator owns the queue lifecycle AND its topic subscription on the broker; the subscriber only binds a persistent receiver to that already-provisioned, already-subscribed endpoint. Ground it in [Creating Queues with the Solace Python API](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Create-Queues.md) (`CREATE_ON_START` creates the queue given to `build()`, and the default, `DO_NOT_CREATE`, disables any attempt to create missing resources) and [Consuming Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Receive.md). WebFetch these live pages on demand; do not paste doc content. + +This is a generation-time OMISSION from the generated output. It is NEVER an edit to any `.py` reference sample and never touches any `VERIFY:` marker string. + +On **Guaranteed Pub/Sub**, ticking the knob edits the subscriber's generation. DROP these two from the generated subscriber's receiver builder: + +- `with_missing_resources_creation_strategy(MissingResourcesCreationStrategy.CREATE_ON_START)` and its then-unused import (the in-process queue creation). +- `with_subscriptions([TopicSubscription.of(topic)])` and its then-unused import (the client-side topic-to-queue mapping; the administrator configured this subscription on the broker). + +KEEP the rest of the subscriber unchanged: + +- The local `Queue.durable_exclusive_queue(queue_name)` or `Queue.durable_non_exclusive_queue(queue_name)` handle from the `Access type` field (the receiver still needs a queue object to bind to; the access type itself is now the administrator's broker setting, so state in a comment that it must match). +- `with_message_client_acknowledgement()`, `with_activation_passivation_support(...)`, `receive_async(...)` (or the `receive_message(...)` loop on the minimal-threading shape), `start()`, and `ack()` after processing. +- The sample's `PubSubPlusClientError` branch around `start()`: it now reports a queue the administrator has not provisioned. + +All of the Guaranteed Pub/Sub leaf's `VERIFY:` markers still emit, unchanged: `VERIFY: CONNECTED` after connect, `VERIFY: QUEUE_BOUND` after `start()`, and `VERIFY: MESSAGE_RECEIVED` after `ack()`. The publisher is untouched, so its `VERIFY: PUBLISH_ACKED` is unaffected. On the fan-out leaf, the drop applies to every subscriber script. + +Because the ticked app no longer provisions anything, a round trip needs the administrator to have provisioned the queue and its subscription first. Say so in the handed-back next steps and record it in the checklist under Your responsibility. + +## Return to the implement-mode contracts + +After capturing the ticked subset and applying the deltas above, return to `implement-mode.md` for the unchanged leaf mechanics: the Step 4 dispatch generates the chosen leaf's scripts and writes the tailored `solace-verification-checklist.md` into each project as generation output (one copy per project when the decoupled-projects knob is ticked, otherwise one at the project root), and Step 6 resolves each item to its state for the Custom ticked subset, invites the developer's additions, and reports the same in chat. This overlay selects which hardening deltas to layer; it does not replace those contracts. + +## Grounding references + +Live `docs.solace.com` `.md` pages (WebFetch on demand): + +- [Creating Queues with the Solace Python API](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Create-Queues.md) +- [Consuming Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Receive.md) diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/design-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/design-mode.md new file mode 100644 index 0000000..983fe0b --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/design-mode.md @@ -0,0 +1,116 @@ +# Design Mode + +Help the developer choose the right Solace messaging pattern and topology for a Python application before any code is generated, grounded in canonical Solace docs. Work through the steps in order. Read this file's grounding links only as each step needs them; link the canonical doc and state the decision in one line, never paraphrase doc content. + +The output of this mode is a filled-in design summary the developer can carry into Implement mode. + +## Step 0: Confirm broker access and grounding + +Before triaging anything, confirm the developer has access to a reachable broker. Confirm that they HAVE one; Design mode only records the broker TYPE (for example, Solace Cloud) in the summary, so do NOT ask them to paste connection details (host, message VPN, client username, password) into the chat. Those values are handled later in Implement mode, where they go into a gitignored `config.json` or, on Quickstart, the command line. A broker discovered running in the environment (a local container, an existing config file) is a fact to report, never an answer: still confirm which broker the developer wants to target. + +If the developer has no reachable broker, route through `prerequisites.md` first (broker acquisition, Solace Cloud recommended), then return here. + +## Step 1: Read the prompt for stated intent before asking anything + +Design mode is ask-or-recommend, not interrogate-first. Before walking the tree, read what the developer already told you and try to resolve a pattern up front. Match the prompt against this signal table first; it is the deterministic path. + +| Stated intent in the prompt | Recommended leaf (or branch to lock if partial) | +|---|---| +| "request a reply", "needs a correlated response", "ask and wait for an answer" | Request-Reply (Direct) (the delivery question still applies; see the no-guaranteed-request-reply rule below) | +| "the request must not be lost", "the request must survive the responder being down", "persistent request" | a Guaranteed Pub/Sub leaf with an application correlation id (the no-guaranteed-request-reply rule below; never Request-Reply (Direct)) | +| "fan out to many services", "broadcast to several independent apps", "every team gets its own copy" | Guaranteed Pub/Sub (fan-out) | +| "competing workers", "share the load across consumers", "scale out processing of one stream" | Guaranteed Pub/Sub (single service, non-exclusive) | +| "one active consumer", "strict ordering", "only one worker at a time" | Guaranteed Pub/Sub (single service, exclusive) | +| "fire-and-forget telemetry", "high-rate metrics, occasional loss is fine", "live ticks, no persistence" | Direct Pub/Sub | +| "must not lose messages", "survive consumer downtime", "persistent delivery" | a Guaranteed leaf (let the remaining questions pick which) | +| "keep it single-threaded", "no callbacks on the data path", "minimal threading", "poll for messages", "pull each message and ack it myself", "one thread of control" | lock the `Threading` field to `Minimal (blocking publish and receive)` (the pattern questions still apply; Request-Reply (Direct) records `n/a (the requestor blocks on each reply; the replier answers from a receive_async() callback)` instead, so the developer sees that the leaf meets the preference on the requestor side only). A prompt that states callbacks, handler threads, or a multi-threaded app locks it to `Async` the same way. | + +If a row matches, or if your own judgment maps an unmatched prompt cleanly to one of the five leaves, take the recommend path. Do not default to a full interrogation just because the prompt is not a verbatim table row; fall back to your own judgment to pick the closest leaf, then confirm. + +**Recommend path (intent is clear).** Recommend the leaf in ONE line and confirm, for example "Sounds like Guaranteed Pub/Sub (fan-out), confirm?", then proceed on the developer's confirm. One line plus one confirm, not a rationale paragraph and not a silent zero-confirm jump. + +**Threading is never inferred.** Question 3 (Step 2) applies to every Direct Pub/Sub and guaranteed publish/subscribe design. Only an explicit statement in the prompt (the threading row above) answers it; a use case, a rate, or a resolved leaf does not. A prompt that settles everything except threading is on the partial-intent path, so ask question 3 even when the leaf is clear: ask it in ONE message that also names the resolved leaf in one line, and take the developer's answer as the leaf confirm too, then present the summary (Step 4). The one exception is a request to skip all questions (the design-contract gate in `python.md`): record `Threading: Async` and say in the summary that it is the default. + +**Fully-specified path (the prompt answers every applicable tree question).** When the prompt resolves the leaf AND answers every Step 2 question that applies to that leaf, including the threading question, skip the separate leaf confirm. Derive the full summary, present it (Step 4), and let the close-out question carry the single confirm: one confirm covers the leaf and the design. Do not ask a leaf confirm and then a close-out confirm; that is two questions where the prompt earned one. + +**Partial-intent path (intent is partial).** When the prompt answers some branches but not all, lock the branches the prompt already answers and ask ONLY the remaining tree questions, in the Step 2 order, skipping every question the prompt already settled. Reach a leaf, then confirm it the same one-line way. + +**Unclear path (no stated intent).** When the prompt states no usable intent, walk all five tree questions in Step 2 order to reach a leaf. + +**Variant of a leaf (the prompt states a requirement the base shape does not meet).** A leaf is a pattern FAMILY, not a fixed topology. Alongside the intent match, read the prompt for reliability or topology requirements that the chosen leaf's DEFAULT shape does not satisfy (for example "the consumer must dedupe redeliveries"). When you find one, the leaf still resolves to one of the five strings verbatim, but the design is a VARIANT of that leaf: its default sample shape is adapted to meet the requirement. Name the deviation in the same one-line confirm and carry the actual topology into the Step 4 summary. Keep a variant within the same pattern family and to topology or reliability knobs the canonical docs ground; if a requirement leaves the Solace topic and queue publish and subscribe model, or needs behavior the docs cannot ground, say so and treat it as off-catalog rather than inventing it. + +## Step 2: Walk the decision tree to a leaf + +Ask only the questions the prompt left open, in this order. Before you answer each question you do walk, WebFetch its grounding page first (even when the prompt pre-answers the question), then link the canonical doc and state the decision in one line, quoting one line from the fetched page; never paraphrase guidance the page does not contain. A page you did not fetch in this session must not appear in the summary's Grounding docs field. + +1. **Interaction shape: request-reply or publish/subscribe?** Grounded in [Message Exchange Patterns](https://docs.solace.com/Get-Started/message-exchange-patterns.md). Decision: a request that expects a correlated reply is Request-Reply; an event fanned out to one or many independent consumers is publish/subscribe. +2. **Delivery guarantee: direct or guaranteed?** This ONE question applies to both interaction shapes, so ask it next regardless of the answer to question 1. Grounded in [Message Delivery Modes](https://docs.solace.com/Get-Started/message-delivery-modes.md). Decision: direct is fire-and-forget with no broker-side persistence, for high-rate flows that tolerate occasional loss; guaranteed (persistent) is persisted and survives consumer downtime. +3. **Threading on the messaging data path: multi-threaded, or minimal threading? (Direct Pub/Sub and guaranteed publish/subscribe branches only.)** Grounded in [Consuming Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Receive.md) on the guaranteed branch and in [Consuming Direct Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-DM-Receive.md) on the direct branch; both pages state that `receive_message(timeout)` blocks the routine until a message arrives, the timeout passes, or a service interruption occurs, and the first states that `receive_async()` invokes the callback on a new Python thread for every message. Frame the choice as WHERE the application's thread of control sits on the messaging data path, never as speed: make no throughput claim for either answer. Name the two answers so the developer can tell them apart. The first answer can be called multi-threaded, async, callback, or push; the second can be called minimal threading, blocking, pull, or single thread of control. Any of these names is fine, and the developer's reply maps onto the matching answer. The reason the question exists, stated in one line when you ask it: in Python the GIL lets only one thread run Python code at a time, so the extra API threads of the callback shape buy less than they do in other languages, and a developer who wants one thread of control can have it. Three answers: + - **Multi-threaded app** (the push model): the API pushes each received message, and on the guaranteed branch each publish receipt, to an application callback on an API thread, so the application manages the concurrency between its own threads and the API's threads. This is the async shape and the current default. Record `Threading: Async`. + - **Minimal threading preferred** (the pull model): the application's own thread pulls each message with a blocking `receive_message(timeout)` call, and on the guaranteed branch publishes each message with a blocking `publish_await_acknowledgement()` call, so no application callback runs on the data path and no application state is shared with an API thread; only the service event listeners still run on API threads. The cost is one message in flight at a time on each side. This is the blocking shape. Record `Threading: Minimal (blocking publish and receive)`. + - **Unsure**: the async shape. Record `Threading: Async`. + + A prompt that already states either answer (the threading row in the Step 1 table) skips this question. Otherwise ALWAYS ask it, with the one skip-all-questions exception in Step 1: never infer the answer from the use case and never record `Async` without asking. Request-Reply (Direct) never reaches it: its base shape already blocks on the reply, and its summary records `Threading: n/a`. When the prompt stated a minimal-threading preference, the line reads `Threading: n/a (the requestor blocks on each reply; the replier answers from a receive_async() callback)`, so the developer sees that the leaf does not meet the preference on the replier side; a blocking replier is not a shape this skill generates. +4. **Number of consuming services (guaranteed publish/subscribe branch only).** Decision: a single consuming service is one logical consumer of the stream; fan-out is several independent services that each need their own full copy of every event. +5. **Competing consumers (guaranteed publish/subscribe, single service branch only).** Grounded in [Topic Endpoints and Queues](https://docs.solace.com/Get-Started/topic-endpoints-queues.md). Decision: exclusive gives one active consumer at a time (strict ordering, hot standby); non-exclusive lets multiple competing workers share the queue and split the load. + +Question 3 applies on the Direct Pub/Sub and the guaranteed publish/subscribe branches. Questions 4 and 5 apply ONLY on the guaranteed publish/subscribe branch. Request-reply resolves at the delivery-guarantee answer and never reaches questions 3 to 5; direct publish/subscribe resolves after question 3. + +**No guaranteed request-reply in the Python API.** When question 1 resolves to Request-Reply and question 2 resolves to guaranteed (the request must be persisted), there is no Python request-reply leaf for it. WebFetch [Request-Reply Messaging in the Solace Python API](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Request-Reply.md) and quote its note that request-reply messaging can only be used with direct messages in the Python API. Then route to Guaranteed Pub/Sub, never to Request-Reply (Direct): the request travels as a persistent message to a durable queue, and the application sets a correlation id on it and matches the answer by that id. Continue the tree at question 3 (the threading question, then the two consumer questions) to pick the Guaranteed Pub/Sub leaf, then record the design as a VARIANT of that leaf (Step 1): name the application correlation id in the `Consumption endpoint` field with a short inline reason, for example `durable queue q.acme.orders + topic subscription; application correlation id (persisted request, no Python guaranteed request-reply)`. Tell the developer that Implement generates the request leg only: the answer path (how the consumer returns the answer and how the requestor matches it by the correlation id) is theirs to build, and the checklist records it as developer-owned. Never offer Request-Reply (Direct) for a request that must be persisted, never invent a guaranteed request-reply leaf, and never record `Request-Reply (Guaranteed)` in a Python summary. + +## Step 3: Resolve to one of the five leaves + +The tree resolves to exactly five leaves. Spell the chosen leaf in the summary exactly as written here; these strings are the contract Implement mode reads. + +- **Request-Reply (Direct)** GENERATED. Request-reply over direct messaging. +- **Direct Pub/Sub** GENERATED. Publish/subscribe over direct messaging, fire-and-forget. +- **Guaranteed Pub/Sub (single service, exclusive)** GENERATED. One durable queue, exclusive access, one active consumer at a time. +- **Guaranteed Pub/Sub (single service, non-exclusive)** GENERATED. One durable queue, non-exclusive access, competing workers share the load. +- **Guaranteed Pub/Sub (fan-out)** SNIPPET. One durable queue plus a topic subscription per independent consuming service. Implement mode reuses the single-service guaranteed generator, so there is no dedicated fan-out sample; the leaf is realized by repeating the single-service queue-and-subscription setup once per consuming service. + +Inside the guaranteed publish/subscribe single-service leaf, the exclusive versus non-exclusive choice is realized as a queue-factory SNIPPET, not a separate sample: the same generated app builds an exclusive or a non-exclusive durable queue based on the developer's answer to question 4. + +**A leaf can carry a variant.** The five strings name the pattern family; a design may adapt a leaf's default topology to a stated requirement (the variant path in Step 1). In every case the `Pattern` string stays one of the five verbatim, and the deviation is recorded in the topology fields of the Step 4 summary (for example the `Consumption endpoint`), never by inventing a sixth leaf string. + +**Topic hierarchy and queue name.** Always publish to topics; never address a queue directly. Do NOT ask the developer to choose or design the topic string. Instead, apply the co-installed solace-topic-best-practices skill (which reads the canonical Topic Architecture Best Practices page live) to the developer's use case, derive the recommended topic hierarchy, and recommend it in ONE line; use that recommended topic without making the developer pick it. Do not design the hierarchy here by hand and do not restate that skill's guidance. Likewise, for any leaf that consumes from a durable queue, derive the queue name automatically rather than asking: follow the `q.`-prefixed dotted convention (for example the topic `acme/orders/new` maps to the queue `q.acme.orders`), recommend it in one line, and record it. The developer overrides the recommended topic or queue name ONLY if they explicitly ask to. + +**Traps to avoid:** + +- **Publishing direct-to-queue instead of pub/sub.** Publish to a topic; let the durable queue carry a topic subscription. ([Topic Endpoints and Queues](https://docs.solace.com/Get-Started/topic-endpoints-queues.md).) +- **Text payload instead of binary.** Default to a binary (`bytearray`) payload; the receiver reads it back with `get_payload_as_bytes()`. +- **Direct where guaranteed is required.** Use persistent delivery whenever the design cannot tolerate message loss or must survive consumer downtime. ([Message Delivery Modes](https://docs.solace.com/Get-Started/message-delivery-modes.md).) +- **A topic endpoint instead of a queue-with-topic-subscription.** Promote the durable queue. ([Topic Endpoints and Queues](https://docs.solace.com/Get-Started/topic-endpoints-queues.md).) +- **A guaranteed request-reply leaf.** The Python API has none; follow the rule at the end of Step 2. + +## Step 4: Emit the design summary + +Fill in this ONE unified design summary every run, in chat, with the confirmed values. The same fields appear every run regardless of which leaf the tree reached, so Implement mode reads one stable structure. Use explicit `n/a` for any field the chosen pattern does not apply. The title names the API, so a saved `solace-design.md` routes back to the Python entry file (the saved-design exception in SKILL.md API Selection). + +``` +Solace Python Design Summary +- Pattern: +- Delivery: +- Access type: +- Topic: +- Consumption endpoint: +- Threading: +- Auth: Basic username/password +- Broker: Solace Cloud +- Grounding docs: +``` + +The fields are fixed: Pattern, Delivery, Access type, Topic, Consumption endpoint, Threading, Auth, Broker, Grounding docs. Formatting is at your discretion, but always present every field in that list, every run, with explicit `n/a` where a field does not apply. The JCSMP summary keeps its own, shorter field list; the two summaries route by their title and are no longer the same shape. The `Pattern` value stays one of the five leaf strings verbatim even for a variant; the deviation lives in the topology fields, never in a new leaf string. + +Field semantics: + +- **Pattern.** Exactly one of the five leaf strings, spelled verbatim: `Request-Reply (Direct)`, `Direct Pub/Sub`, `Guaranteed Pub/Sub (single service, exclusive)`, `Guaranteed Pub/Sub (single service, non-exclusive)`, `Guaranteed Pub/Sub (fan-out)`. This is the contract Implement mode reads, so the spelling is not cosmetic. +- **Delivery.** `Direct` for the two direct leaves; `Guaranteed (persistent)` for the three guaranteed leaves. +- **Access type.** `Exclusive` or `Non-exclusive` for the two guaranteed single-service leaves; `n/a` for every other leaf. Surface it explicitly even though it is also encoded in the Pattern string. +- **Consumption endpoint.** The pattern-specific place the consuming side reads from. Record the ACTUAL topology the design requires; the defaults are `direct topic subscription` for Direct Pub/Sub, `durable queue + topic subscription` for the three guaranteed leaves, and `API-managed reply (RequestReplyMessagePublisher)` for Request-Reply (Direct). For the durable-queue leaves, include the auto-derived queue name (for example `durable queue q.acme.orders + topic subscription`) so Implement mode reads the concrete queue name here. For a variant, record the variant topology here with a short inline reason. +- **Topic.** The recommended topic string or hierarchy, derived by applying the solace-topic-best-practices skill to the use case. +- **Threading.** `Async` or `Minimal (blocking publish and receive)` on the Direct Pub/Sub and Guaranteed Pub/Sub leaves, from question 3; `n/a` for Request-Reply (Direct). Spell the value verbatim: it is the contract Implement mode reads to pick the sample pair. `Async` selects the receive-callback and publish-receipt samples. `Minimal (blocking publish and receive)` selects the blocking samples: `receive_message()` on both pub/sub branches, plus `publish_await_acknowledgement()` on the guaranteed branch (the direct publisher has no receipt to await, so it is the same script on both answers). A summary with no `Threading` line reads as `Async` (a summary typed in chat, or a `solace-design.md` saved before the field existed). +- **Auth.** Basic username/password. +- **Broker.** Default to `Solace Cloud` and record it without asking. Switch to another type (Software Broker or Appliance) ONLY if the developer explicitly insists. +- **Grounding docs.** ONLY the pages this session actually WebFetched. Never list a page you did not fetch. Write `none fetched` when no page was fetched. + +After presenting the summary in chat, close Design mode explicitly so the developer knows exactly what happens next; do NOT just display the summary and stop. On the fully-specified path (Step 1) this close-out is the ONLY confirm of the run; it carries the leaf confirmation too. If the design is a variant of its leaf, state the deviation and its rationale in one line here too. Ask them directly, in one step, both whether they are happy with this design AND whether to save it to `solace-design.md` in their project (for example: "Happy with this design? If so, I can save it to `solace-design.md` and move into Implement mode to generate the runnable Python project."). Write the file only on their OK; never write it unprompted. Once they confirm, state the next step plainly: the work moves into Implement mode, which generates the runnable Python project from this summary. Implement mode treats this summary, whether it lives in the chat or in the saved `solace-design.md`, as its input contract. Implement mode OPENS with its own Step 0 door question (Quickstart, Solace Suggested, or Custom), asked before any broker details are requested or accepted; an existing `config.json` or a running broker does not answer it, so ask it on the way in. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check-samples.sh b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check-samples.sh new file mode 100755 index 0000000..9899894 --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check-samples.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# +# Sample check for the baked Solace Python samples. Python has no compile step, so the +# gate installs the latest solace-pubsubplus, runs py_compile on every baked sample, +# and runs check_samples.py, which imports each sample, builds its API objects without a +# broker, and exercises the shared connection helper. The script's exit code IS the +# gate: exit 0 only if every check passes. +# +# Runnable locally with one command from any working directory. Resolves the +# solace-pubsubplus version from PyPI at run time, so no version number is committed +# anywhere (SKILL.md Invariant 3). All work happens in a throwaway temp directory, so +# the repository tree is left unmodified. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# references/python/ lives two levels up from sample-check/ (evals/ -> python/). +REFERENCES_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" +PYTHON="${PYTHON:-python3}" + +# Sample roster: each baked sample and the name a generated project gives it +# (implement-mode.md Step 4). A new baked sample is a one-line append here. +SAMPLES=( + "python-solace-connection-config.py:solace_connection_config.py" + "python-guaranteed-publisher-sample.py:guaranteed_publisher.py" + "python-guaranteed-subscriber-sample.py:guaranteed_subscriber.py" + "python-guaranteed-publisher-blocking-sample.py:guaranteed_publisher_blocking.py" + "python-guaranteed-subscriber-blocking-sample.py:guaranteed_subscriber_blocking.py" + "python-direct-publisher-sample.py:direct_publisher.py" + "python-direct-subscriber-sample.py:direct_subscriber.py" + "python-direct-subscriber-blocking-sample.py:direct_subscriber_blocking.py" + "python-direct-requestor-blocking-sample.py:direct_requestor_blocking.py" + "python-direct-requestor-sample.py:direct_requestor.py" + "python-direct-replier-sample.py:direct_replier.py" +) + +# Existence guard: fail early if any input is missing, including a new roster entry +# whose sample is not yet committed. +for entry in "${SAMPLES[@]}"; do + src="${REFERENCES_DIR}/${entry%%:*}" + [[ -f "$src" ]] || { echo "ERROR: required input not found: $src" >&2; exit 1; } +done +for f in "${SCRIPT_DIR}/check_samples.py" "${REFERENCES_DIR}/scripts/verify.sh"; do + [[ -f "$f" ]] || { echo "ERROR: required input not found: $f" >&2; exit 1; } +done + +# (1) Resolve the latest solace-pubsubplus release from the PyPI JSON API, the same +# source implement-mode.md Step 3 uses. On any failure, exit non-zero rather than +# guessing a version. +echo "Resolving the latest solace-pubsubplus version from PyPI..." +VERSION="$(curl -s --max-time 20 https://pypi.org/pypi/solace-pubsubplus/json \ + | "$PYTHON" -c 'import json, sys; print(json.load(sys.stdin)["info"]["version"])' 2>/dev/null || true)" +if [[ -z "${VERSION// /}" ]]; then + echo "ERROR: could not resolve the latest solace-pubsubplus version from PyPI" >&2 + echo " (network failure or unreachable API). Not guessing a version." >&2 + exit 1 +fi +echo "Resolved solace-pubsubplus version: ${VERSION}" + +# (2) Build a throwaway working tree and virtual environment. +WORK_DIR="$(mktemp -d)" +cleanup() { rm -rf "$WORK_DIR"; } +trap cleanup EXIT + +"$PYTHON" -m venv "${WORK_DIR}/.venv" +VENV_PY="${WORK_DIR}/.venv/bin/python" +echo "Installing solace-pubsubplus==${VERSION} with $("$VENV_PY" --version)..." +"$VENV_PY" -m pip install -q --disable-pip-version-check "solace-pubsubplus==${VERSION}" + +# (3) Copy the samples under their generated names. Content is verbatim; only the +# file name changes, so the samples import each other as a generated project does. +for entry in "${SAMPLES[@]}"; do + cp "${REFERENCES_DIR}/${entry%%:*}" "${WORK_DIR}/${entry##*:}" +done +cp "${SCRIPT_DIR}/check_samples.py" "${WORK_DIR}/check_samples.py" + +# (4) py_compile every sample. +echo "Running py_compile on every baked sample..." +( cd "$WORK_DIR" && "$VENV_PY" -m py_compile ./*.py ) +echo "PASS: py_compile" + +# (5) Import each sample and build its API objects without a broker, then exercise +# the connection helper. +echo "Building the API objects against tcp://localhost:55555 without a broker..." +( cd "$WORK_DIR" && "$VENV_PY" check_samples.py ) + +# (6) The bundled verify script must at least parse. +bash -n "${REFERENCES_DIR}/scripts/verify.sh" +echo "PASS: scripts/verify.sh parses" + +# (7) The bundled verify script must resolve the fixed script names of both threading +# shapes (implement-mode.md Step 4) without a broker. Single-project mode stops at the +# long-running role's missing .venv before it looks at the foreground role, so the +# fixture uses separate-projects mode with a stub interpreter in the long-running +# role's directory: with both scripts present under src/ the stage must get past both +# names to the foreground role's virtual-environment check, and with either script +# missing it must name that script in that directory. +echo "Checking that scripts/verify.sh resolves the fixed script names..." +verify_stderr() { # : verify.sh's stderr in separate-projects mode, no broker + ( bash "${REFERENCES_DIR}/scripts/verify.sh" "$1" --subscriber-dir "$2" --publisher-dir "$3" 2>&1 >/dev/null ) || true +} +make_roles() { # sets SUB and PUB to fresh role directories; SUB carries a stub interpreter + SUB="$(mktemp -d "${WORK_DIR}/sub.XXXXXX")"; PUB="$(mktemp -d "${WORK_DIR}/pub.XXXXXX")" + mkdir -p "${SUB}/src" "${SUB}/.venv/bin" "${PUB}/src" + : > "${SUB}/.venv/bin/python"; chmod +x "${SUB}/.venv/bin/python" +} +expect_resolved() { # : the stage resolves both names + make_roles; : > "${SUB}/src/$2"; : > "${PUB}/src/$3" + grep -qF "no virtual environment in: $(cd "${PUB}" && pwd)" <<<"$(verify_stderr "$1" "${SUB}" "${PUB}")" \ + || { echo "ERROR: verify.sh $1 did not resolve: $2 $3" >&2; exit 1; } +} +expect_missing() { # : the stage names the absent role's default + make_roles; [ "$2" = - ] || : > "${SUB}/src/$2"; [ "$3" = - ] || : > "${PUB}/src/$3" + local dir; if [ "$4" = sub ]; then dir="${SUB}"; else dir="${PUB}"; fi + grep -qF "no src/$5 found in: $(cd "${dir}" && pwd)" <<<"$(verify_stderr "$1" "${SUB}" "${PUB}")" \ + || { echo "ERROR: verify.sh $1 did not report src/$5 as missing in the $4 directory" >&2; exit 1; } +} +expect_resolved consumer guaranteed_subscriber.py guaranteed_publisher.py +expect_resolved roundtrip guaranteed_subscriber_blocking.py guaranteed_publisher_blocking.py +expect_resolved direct direct_subscriber.py direct_publisher.py +expect_resolved direct direct_subscriber_blocking.py direct_publisher.py +expect_resolved direct-request-reply direct_replier.py direct_requestor_blocking.py +expect_resolved direct-request-reply direct_replier.py direct_requestor.py +expect_missing consumer - guaranteed_publisher.py sub guaranteed_subscriber.py +expect_missing roundtrip guaranteed_subscriber.py - pub guaranteed_publisher.py +expect_missing direct - direct_publisher.py sub direct_subscriber.py +expect_missing direct-request-reply direct_replier.py - pub direct_requestor_blocking.py +echo "PASS: scripts/verify.sh resolves the fixed script names of both threading shapes" + +echo "PASS: the baked Python samples against solace-pubsubplus ${VERSION}" diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check_samples.py b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check_samples.py new file mode 100644 index 0000000..5bc89c1 --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/evals/sample-check/check_samples.py @@ -0,0 +1,122 @@ +"""Sample check for the baked Solace Python samples, run by check-samples.sh. + +check-samples.sh copies every sample into the working directory under its generated name and +runs this file with the virtual environment's interpreter. Without a broker, it: + +1. imports each sample module (each keeps its __main__ guard, so nothing connects), +2. calls each sample's setup_solace(), which builds the MessagingService and the + role's publisher, receiver, or request-reply objects, then teardown_solace() on the + never-connected service, and +3. exercises the shared solace_connection_config helper's config.json, command-line + fallback, and fail-fast paths. + +build() resolves the host, so every build uses tcp://localhost:55555, which resolves +whether or not a broker listens there. Any failure raises, and the non-zero exit fails +the gate. +""" + +import glob +import importlib +import json +import os +import tempfile +import threading + +# Every sample check-samples.sh copied in, so a new roster entry is covered without a +# second list to keep in step. The shared helper has no setup_solace() of its own. +NOT_SAMPLES = {"check_samples", "solace_connection_config"} +MODULES = sorted( + name + for name in (os.path.splitext(path)[0] for path in glob.glob("*.py")) + if name not in NOT_SAMPLES +) + +HOST_KEY = "solace.messaging.transport.host" +VPN_KEY = "solace.messaging.service.vpn-name" +USER_KEY = "solace.messaging.authentication.basic.username" +PASSWORD_KEY = "solace.messaging.authentication.basic.password" + +PROPERTIES = { + HOST_KEY: "tcp://localhost:55555", + VPN_KEY: "smoke-vpn", + USER_KEY: "smoke-user", + PASSWORD_KEY: "smoke-pass", +} + + +def build_samples() -> None: + if not MODULES: + raise RuntimeError("no sample modules found in the working directory") + for name in MODULES: + module = importlib.import_module(name) + state = module.setup_solace(dict(PROPERTIES), threading.Event()) + module.teardown_solace(state) + print(f"PASS: {name} builds its service and role objects without a broker", flush=True) + + +def expect_exit(call, case: str) -> None: + try: + call() + except SystemExit as exit_error: + if exit_error.code == 1: + return + raise AssertionError(f"{case}: exit code {exit_error.code}, expected 1") from exit_error + raise AssertionError(f"{case}: expected the helper to exit") + + +def expect_value_error(call, case: str) -> None: + try: + call() + except ValueError: + return + raise AssertionError(f"{case}: expected a ValueError") + + +def check_config_helper() -> None: + from solace_connection_config import load_service_properties + + start_dir = os.getcwd() + with tempfile.TemporaryDirectory() as work: + os.chdir(work) + try: + def write_config(config: dict) -> None: + with open("config.json", "w", encoding="utf-8") as config_file: + json.dump(config, config_file) + + # a valid config.json round-trips every key, including a pass-through key + write_config({**PROPERTIES, "solace.messaging.tls.trust-store-path": "/smoke/trust"}) + loaded = load_service_properties([], "smoke") + assert loaded[HOST_KEY] == PROPERTIES[HOST_KEY], loaded + assert loaded["solace.messaging.tls.trust-store-path"] == "/smoke/trust", loaded + + # config.json wins over command-line arguments + loaded = load_service_properties(["tcp://other:55555", "v", "u"], "smoke") + assert loaded[HOST_KEY] == PROPERTIES[HOST_KEY], loaded + + # a missing required key fails fast + write_config({VPN_KEY: "smoke-vpn", USER_KEY: "smoke-user"}) + expect_value_error(lambda: load_service_properties([], "smoke"), "missing-key") + + # a present but blank required value fails fast + write_config({**PROPERTIES, HOST_KEY: " "}) + expect_value_error(lambda: load_service_properties([], "smoke"), "blank-value") + + # with no config.json, the command-line arguments are the source, and an + # omitted password is sent as an empty string + os.remove("config.json") + loaded = load_service_properties(["tcp://localhost:55555", "smoke-vpn", "smoke-user"], "smoke") + assert loaded[USER_KEY] == "smoke-user" and loaded[PASSWORD_KEY] == "", loaded + + # a blank command-line argument fails fast + expect_value_error(lambda: load_service_properties(["", "smoke-vpn", "smoke-user"], "smoke"), "args-blank") + + # neither source: print the usage line and exit 1 + expect_exit(lambda: load_service_properties([], "smoke"), "no-source") + finally: + os.chdir(start_dir) + print("PASS: solace_connection_config file, fallback, and fail-fast paths", flush=True) + + +if __name__ == "__main__": + build_samples() + check_config_helper() diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-direct-pubsub.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-direct-pubsub.md new file mode 100644 index 0000000..8b604ae --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-direct-pubsub.md @@ -0,0 +1,39 @@ +# Implement: Direct Pub/Sub leaf + +The per-leaf wiring for the Direct Pub/Sub leaf of the Python API. Read this file from `implement-mode.md` Step 4 when the `Pattern` field is `Direct Pub/Sub`. The shared Step 4 rules in `implement-mode.md` (the disclaimer header, the shared connection helper, the fixed file names, the concurrency rules, the variant discipline, the demo-harness rule, the comments rule, and the Quickstart fail-fast rule) apply here unchanged. + +## Direct pub/sub flow + +Generation reads the direct publisher sample and ONE direct subscriber sample, picked by the design summary's `Threading` field (`implement-mode.md` Step 4), and adapts each into its own generated script, the same near-verbatim discipline the guaranteed flow uses: `direct_publisher.py` is a near-verbatim adaptation of the publisher sample, and `direct_subscriber.py` (or `direct_subscriber_blocking.py` on the blocking shape) is a near-verbatim adaptation of the chosen subscriber sample. Read ONLY the subscriber sample the field selects. Direct messaging is at-most-once: there is no broker receipt and no redelivery, so this leaf has NO queue, NO provisioning, NO client acknowledgement, and NO publish receipt listener. + +- `python-direct-publisher-sample.py` (both threading shapes): basic-auth connect, a continuous DIRECT binary publish loop with a rotating payload to a dynamic topic, a publish failure listener for the asynchronous failures the API reports, capacity-bounded back pressure, service event listeners registered before `connect()`, and a graceful SIGINT/SIGTERM shutdown. No receipt listener and no user context (direct has no broker receipt). Generation STRIPS the continuous loop, the rotating payload, the dynamic topic, and the stats thread down to a short bounded burst that self-exits (publisher steps below). The direct publisher has no blocking variant: `publish()` returns at once because direct has no receipt to await, and the publish failure listener stays on both shapes as the only channel for an asynchronous send failure. +- `python-direct-subscriber-sample.py` (`Threading: Async`, or no `Threading` field): basic-auth connect, a plain topic subscription on a direct receiver (no queue, no provisioning), capacity-bounded back pressure with discard detection, service event listeners registered before `connect()`, and a graceful SIGINT/SIGTERM shutdown. The subscriber is the long-running SIGINT target. +- `python-direct-subscriber-blocking-sample.py` (`Threading: Minimal (blocking publish and receive)`): the same subscription, back pressure, and discard detection, but NO `receive_async()` handler: the main loop pulls each message with `receive_message(timeout)` and a POSITIVE timeout and processes it on the application's own thread, and shutdown calls `terminate(0)`. + +Each script builds its OWN `MessagingService` (two services, one per script, matching the samples). Keep the samples' structure: the shared state dataclass, `setup_solace(...)` that builds the service and the receiver or publisher before any connection exists, `connect_solace(...)`, the main loop, and `teardown_solace(...)` in `main()`'s `finally`. Keep every listener class; they are part of the best-practices contract and must not be stripped during adaptation. + +**Ground before you write.** Before writing either script, WebFetch [Publishing Direct Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-DM-Publish.md) and [Consuming Direct Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-DM-Receive.md) in this session (SKILL.md Invariant 2). + +## Subscriber script (`direct_subscriber.py`, or `direct_subscriber_blocking.py` on the blocking shape) + +The subscriber subscribes directly to the topic (no queue), receives on a callback (callback shape) or pulls on its main loop (blocking shape), and is the long-running process the developer runs FIRST. Start the file with the disclaimer header, then wire it in this sequence: + +1. Build the connection properties with `load_service_properties(args, APP_NAME)`, then the service with `MessagingService.builder().from_properties(properties).build()` (on Quickstart without the reconnection strategy call, per the fail-fast rule). +2. Build the direct receiver as the sample does: `create_direct_message_receiver_builder()` with `with_subscriptions([TopicSubscription.of(topic)])` for the design summary's topic and `on_back_pressure_drop_latest(RECEIVE_BUFFER_CAPACITY)`, then `build()`. Register the three service event listeners BEFORE `connect()`. There is NO queue and NO missing-resources creation strategy on the direct path. +3. `messaging_service.connect()`. Immediately after `connect()` returns, emit the connect marker: `trace("VERIFY: CONNECTED")`. +4. Callback shape: `receiver.receive_async(handler)` BEFORE `receiver.start()`, as the sample orders them, then `receiver.start()`. Blocking shape: NO `receive_async()`; `receiver.start()` alone, as the blocking sample does, and the loop of step 5 pulls the messages. `start()` applies the subscription and blocks until the broker confirms it. On both shapes, immediately after `start()` returns, emit the subscribed marker: `trace("VERIFY: SUBSCRIBED")`. **No reapply-subscriptions setting.** The JCSMP direct subscriber must set `REAPPLY_SUBSCRIPTIONS` to `true`, or its subscription is lost on an automatic reconnect. The Python API has no such setting and needs none: the [Messaging Service](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Messaging-Service.md) page states that the API re-applies the application's direct subscriptions on reconnect. Do not search for or invent a Python equivalent. Keep the sample's `on_reconnected` comment that says so. When reconnect attempts are exhausted, the same page states that the application must recreate the session with all its subscriptions; the sample's `on_service_interrupted` handler covers that case by shutting down. +5. Callback shape: in the handler's `on_message`, keep the sample's payload read and its discard check (`get_message_discard_notification()`). Blocking shape: in the main loop, keep the sample's `message = receiver.receive_message(RECEIVE_TIMEOUT_MS)` call with its POSITIVE timeout in milliseconds and its `PubSubPlusClientError` branch, and hand each returned message to the sample's `process_message(...)` with the same payload read and discard check; `None` means a timeout, and the loop goes round to its shutdown check. On both shapes, after the discard check, emit the receipt marker: `trace("VERIFY: MESSAGE_RECEIVED")`. There is NO `ack()` on the direct path. +6. KEEP the long-running wait loop (on the blocking shape, the `receive_message()` loop itself, which checks the shutdown event each time the call returns), the SIGINT/SIGTERM handler that only sets the shutdown event, and `teardown_solace(...)` with `terminate(grace_period)` and `disconnect()`. On the blocking shape keep the sample's `TERMINATE_GRACE_PERIOD_MS` of 0 and its `IncompleteMessageDeliveryError` warning: the loop has stopped, so nothing drains the buffer, and dropping its direct messages is a normal at-most-once stop. The subscriber is the SIGINT target, and its shutdown line (`Shutdown signal received ...`) stays a `trace(...)` call. + +## Publisher script (`direct_publisher.py`) + +The publisher only connects and publishes a short bounded burst of DIRECT messages, then exits on its own. It does NOT provision anything and does NOT wait for a key press or a signal. Start the file with the disclaimer header, then wire it in this sequence: + +1. Build the connection properties and the service exactly as the subscriber does (step 1 above), then the publisher with `create_direct_message_publisher_builder().on_back_pressure_wait(PUBLISH_BUFFER_CAPACITY).build()`. Keep `set_publish_failure_listener(...)`: direct has no broker receipt, so a buffered message that cannot be sent is reported only through that listener. Register the three service event listeners BEFORE `connect()`. +2. `messaging_service.connect()`. Immediately after `connect()` returns, emit the connect marker: `trace("VERIFY: CONNECTED")`. Then `publisher.start()`. +3. Publish a SHORT BOUNDED BURST to the design summary's topic with `publish(payload, Topic.of(topic))` in a small fixed-count loop, then return from the publish step. REMOVE the sample's continuous loop, its rotating payload and dynamic topic, and its stats thread. Keep the sample's `PubSubPlusClientError` branch, which sets the shared state's `exit_code` to 1. `teardown_solace(...)` in `main()`'s `finally` calls `terminate(grace_period)`, which gives the buffered messages up to the grace period to leave before `disconnect()`. Keep the SIGINT/SIGTERM handler, so Ctrl-C during a back pressure wait still reaches teardown. When the request describes a real application, its own messages replace the burst (the demo-harness rule), and each payload stays a `bytearray` or a `str`, never `bytes`. That replacement changes the payload and the topic, never the shape: a request that describes a continuous or high-rate feed (readings several times a second, a telemetry stream) still gets a publisher that sends a bounded batch of the application's real messages and exits on its own, because the `direct` stage waits for that exit. Do not keep the sample's open-ended loop behind an environment-variable cap, and do not move the project to the `app` stage to drive it; the continuous trigger is the developer's integration step after verification. +4. The publisher emits NO send-side marker. Direct has no broker receipt, so there is no `PUBLISH_ACKED` analog; a marker after `publish()` would only prove the local call returned. The proof of the round trip is `VERIFY: MESSAGE_RECEIVED` in the subscriber output. + +## The marker contract + +These three `VERIFY:` marker strings (`VERIFY: CONNECTED`, `VERIFY: SUBSCRIBED`, `VERIFY: MESSAGE_RECEIVED`) are the same strings the JCSMP Direct Pub/Sub leaf emits, character for character. They are read PER PROCESS: the subscriber output carries `SUBSCRIBED` and `MESSAGE_RECEIVED`, and both scripts print `CONNECTED`. `verify.sh` reads these same strings in each process's own output (`implement-mode.md` Step 5), so never change one. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-guaranteed-pubsub.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-guaranteed-pubsub.md new file mode 100644 index 0000000..81bd483 --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-guaranteed-pubsub.md @@ -0,0 +1,62 @@ +# Implement: Guaranteed Pub/Sub leaves + +The per-leaf wiring for the three Guaranteed Pub/Sub leaves of the Python API. Read this file from `implement-mode.md` Step 4 when the `Pattern` field is `Guaranteed Pub/Sub (single service, exclusive)`, `Guaranteed Pub/Sub (single service, non-exclusive)`, or `Guaranteed Pub/Sub (fan-out)`. The shared Step 4 rules in `implement-mode.md` (the disclaimer header, the shared connection helper, the fixed file names, the concurrency rules, the variant discipline, the demo-harness rule, the comments rule, and the Quickstart fail-fast rule) apply here unchanged. + +## Guaranteed pub/sub flow + +The skill ships two best-practices reference sample PAIRS for this flow, and the design summary's `Threading` field picks one (`implement-mode.md` Step 4): `Async`, or no `Threading` field, picks the callback pair; `Minimal (blocking publish and receive)` picks the blocking pair. Read ONLY the chosen pair, by their exact relative paths, and adapt each into its own generated script. There is no merge and no third skeleton: the generated publisher is a near-verbatim adaptation of the chosen publisher sample, and the generated subscriber is a near-verbatim adaptation of the chosen subscriber sample. + +The callback pair (`Threading: Async`), generated as `guaranteed_publisher.py` and `guaranteed_subscriber.py`: + +- `python-guaranteed-publisher-sample.py`: basic-auth connect, persistent binary publish to a topic from a message template that sets the time-to-live once, publish receipt (ACK/NACK) handling, capacity-bounded back pressure, service event listeners registered before `connect()`, and a graceful SIGINT/SIGTERM shutdown that drains outstanding receipts. +- `python-guaranteed-subscriber-sample.py`: basic-auth connect, durable queue plus topic subscription provisioned at start, a client-ack receiver that ACKs only after processing, receiver state and termination handling, service event listeners registered before `connect()`, and a graceful SIGINT/SIGTERM shutdown. + +The blocking pair (`Threading: Minimal (blocking publish and receive)`), generated as `guaranteed_publisher_blocking.py` and `guaranteed_subscriber_blocking.py`: + +- `python-guaranteed-publisher-blocking-sample.py`: the same connect and message template, but one blocking `publish_await_acknowledgement()` per message with a POSITIVE acknowledgement timeout, NO publish receipt listener and NO back-pressure setting (one message is in flight at a time, so the publish buffer cannot grow), the "Discard the Message" handling of a timeout, a NACK, or an overflow from the Publishing Persistent Messages page, and the same graceful shutdown. The main thread publishes each message and handles its ACK or NACK itself. +- `python-guaranteed-subscriber-blocking-sample.py`: the same provisioning and client-ack receiver, but NO `receive_async()` handler: the main loop pulls each message with `receive_message(timeout)` and a POSITIVE timeout, processes it, and ACKs it on the application's own thread, and shutdown calls `terminate(0)`. + +Each script builds its OWN `MessagingService` (two services, one per script, matching the samples). Keep the samples' structure: the shared state dataclass, `setup_solace(...)` that builds the service and the receiver or publisher before any connection exists, `connect_solace(...)`, the main loop, and `teardown_solace(...)` in `main()`'s `finally`. Keep every listener class; they are part of the best-practices contract and must not be stripped during adaptation. + +**Ground before you write.** Before writing either script, WebFetch [Publishing Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Publish.md) and [Consuming Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Receive.md) in this session (SKILL.md Invariant 2). The second page also grounds the `receive_async()` and acknowledgement rules in `implement-mode.md` Step 4's concurrency section. + +The three guaranteed pub/sub leaves generate the same publisher and subscriber. The exclusive versus non-exclusive choice is one queue-factory line (read the `Access type` summary field). The fan-out leaf reuses the single-service subscriber once per consuming service (see Fan-out below). The two threading shapes share the sequences below; a step that differs names both shapes. + +## Subscriber script (`guaranteed_subscriber.py`, or `guaranteed_subscriber_blocking.py` on the blocking shape) + +The subscriber provisions the durable queue and the topic subscription, and it is the long-running process the developer runs FIRST. The publisher does NOT provision anything. Start the file with the disclaimer header, then wire the subscriber in this sequence, each step grounded in the pages linked in `implement-mode.md`'s Grounding references: + +1. Build the connection properties with `load_service_properties(args, APP_NAME)`, then the service with `MessagingService.builder().from_properties(properties).build()` (on Quickstart without the reconnection strategy call, per the fail-fast rule). +2. Build the queue from the design summary's `Consumption endpoint` queue name and `Access type` field (see the access-type snippet below). +3. Build the persistent receiver as the sample does: `create_persistent_message_receiver_builder()` with `with_missing_resources_creation_strategy(MissingResourcesCreationStrategy.CREATE_ON_START)`, `with_subscriptions([TopicSubscription.of(topic)])` for the design summary's topic, `with_message_client_acknowledgement()`, and `with_activation_passivation_support(...)`, then `build(queue)`. Keep the receiver termination listener and register the three service event listeners BEFORE `connect()`. See [Creating Queues with the Solace Python API](https://docs.solace.com/API/API-Developer-Guide-Python/Python-API-Create-Queues.md). +4. `messaging_service.connect()`. Immediately after `connect()` returns, emit the connect marker: `trace("VERIFY: CONNECTED")`. +5. Callback shape: `receiver.receive_async(handler)` BEFORE `receiver.start()`, as the sample orders them, then `receiver.start()`. Blocking shape: NO `receive_async()`; `receiver.start()` alone, as the blocking sample does, and the loop of step 6 pulls the messages. On both shapes, immediately after `start()` returns without raising, emit the queue-bound marker: `trace("VERIFY: QUEUE_BOUND")`. Keep the sample's `PubSubPlusClientError` branch around `start()`: a client that may not create endpoints fails there, and the error message tells the developer what to do. +6. Callback shape: in the handler's `on_message`, keep the sample's try/except/finally and call `receiver.ack(message)` only after processing. Blocking shape: in the main loop, keep the sample's `message = receiver.receive_message(RECEIVE_TIMEOUT_MS)` call with its POSITIVE timeout in milliseconds and its `PubSubPlusClientError` branch, and hand each returned message to the sample's `process_message(...)`, whose try/except/finally calls `receiver.ack(message)` only after processing; `None` means a timeout, and the loop goes round to its shutdown check. On both shapes, immediately after `ack()` in the `finally`, emit the round-trip marker: `trace("VERIFY: MESSAGE_RECEIVED")`. See [Consuming Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Receive.md). +7. KEEP the long-running wait loop (on the blocking shape, the `receive_message()` loop itself, which checks the shutdown event each time the call returns), the SIGINT/SIGTERM handler that only sets the shutdown event, and `teardown_solace(...)` with `terminate(grace_period)` and `disconnect()`. On the blocking shape keep the sample's `TERMINATE_GRACE_PERIOD_MS` of 0 and its `IncompleteMessageDeliveryError` warning: the loop has stopped, so nothing drains the receiver's buffer, and a message that was never ACKed stays on the queue for redelivery. The subscriber is the SIGINT target, and its shutdown line (`Shutdown signal received ...`) stays a `trace(...)` call. + +**Access type (exclusive vs non-exclusive): one line, not a separate sample.** The `Access type` field picks the queue factory: + +```python +queue = Queue.durable_exclusive_queue(queue_name) # one active consumer / ordered failover +# queue = Queue.durable_non_exclusive_queue(queue_name) # competing consumers share the load +``` + +When the `Access type` field reads exclusive, keep the `durable_exclusive_queue` line: only one bound receiver is active at a time, and the receiver state listener reports the passive standby. When it reads non-exclusive, use the `durable_non_exclusive_queue` line instead, so several subscriber processes bound to the same queue share the messages. Emit exactly one of the two lines. Ground the choice in [Topic Endpoints and Queues](https://docs.solace.com/Get-Started/topic-endpoints-queues.md). + +**Fan-out.** On the fan-out leaf, every consuming service gets its own durable queue with the same topic subscription, so each service receives its own full copy of every event. Generate one subscriber script per consuming service: `guaranteed_subscriber.py` for the first service, and `guaranteed_subscriber_.py` for each further service (on the blocking shape, `guaranteed_subscriber_blocking.py` and `guaranteed_subscriber_blocking_.py`), where `` is a lowercase Python identifier (letters, digits, and underscores, for example `billing` or `audit_log`) so the import smoke can import the module, each with its own queue name from the design summary and an exclusive durable queue. The `verify.sh` stages drive the first subscriber script only, so hand back one run command per further subscriber in Step 5, and add each script to the import smoke. + +## Publisher script (`guaranteed_publisher.py`, or `guaranteed_publisher_blocking.py` on the blocking shape) + +The publisher only connects and publishes; it does NOT provision the queue or the subscription. It must exit on its own after the broker acknowledges its messages, so the developer does not have to interrupt it. Start the file with the disclaimer header, then wire the publisher in this sequence: + +1. Build the connection properties and the service exactly as the subscriber does (step 1 above), then the publisher. Callback shape: `create_persistent_message_publisher_builder().on_back_pressure_wait(PUBLISH_BUFFER_CAPACITY).build()`. Blocking shape: `create_persistent_message_publisher_builder().build()` with NO back-pressure setting, as the blocking sample explains: one message is in flight at a time, so the publish buffer cannot grow. Register the three service event listeners BEFORE `connect()`. +2. `messaging_service.connect()`. Immediately after `connect()` returns, emit the connect marker: `trace("VERIFY: CONNECTED")`. +3. `publisher.start()`. Callback shape: then `publisher.set_message_publish_receipt_listener(...)` AFTER `start()`, as the sample orders them. In the receipt handler's ACK branch, emit the publish-ACK marker: `trace("VERIFY: PUBLISH_ACKED")`. CRITICAL: the sample logs the ACK with `logger.debug`, which the INFO-level logging never shows, so the marker would never appear. Emit it through `trace(...)`, not through the logger. In the NACK branch, keep the sample's warning and also set the shared state's `exit_code` to 1, so a rejected message makes the publisher exit non-zero. Pass the shared state to the handler to do this. Blocking shape: NO receipt listener; the marker moves into step 4. +4. Publish persistent messages to the design summary's topic as the sample does: one `message_builder()` template with `PERSISTENT_TIME_TO_LIVE` set once, and `build(payload, additional_message_properties={APPLICATION_MESSAGE_ID: msg_id})` per message, where `payload` is a `bytearray` or a `str`, never `bytes` (the demo-harness rule in `implement-mode.md`). Callback shape: `publish(message, topic, user_context=msg_id)`. Blocking shape: `publish_await_acknowledgement(message, topic, PUBLISH_ACK_TIMEOUT_MS)` with the sample's POSITIVE timeout in milliseconds and no `user_context`; a normal return IS the broker's ACK, so emit `trace("VERIFY: PUBLISH_ACKED")` immediately after the call returns. Keep the blocking sample's three discard branches (`PubSubTimeoutError`, the NACK error classes, and `PublisherOverflowError`), which log the message id at INFO and continue (the Discard the Message strategy from the Publishing Persistent Messages page), but set the shared state's `exit_code` to 1 in EACH of them, the same way the callback shape's NACK branch does: the loop still sends the remaining messages, and the publisher exits non-zero at the end, so a run that lost one message out of many fails the `verify.sh` stage instead of passing on the markers of the others. Keep its `PubSubPlusClientError` branch too, which sets `exit_code` to 1 and stops. See [Publishing Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Publish.md). +5. EXIT on its own: replace the sample's continuous publish loop with the application's own messages, or with one message on a try-it run, then return from the publish step. `teardown_solace(...)` in `main()`'s `finally` calls `terminate(grace_period)`. Callback shape: it sends what is buffered and waits for the broker's receipt of every published message, so the `VERIFY: PUBLISH_ACKED` marker prints before the script exits. Blocking shape: every call has already returned with its ACK and printed the marker, so `terminate()` returns at once; keep the sample's `IncompleteMessageDeliveryError` branch for a message the API never handed to the transport. Drop the stats thread (or the blocking sample's inline rate report) with the loop. Keep the SIGINT/SIGTERM handler, so Ctrl-C during a back pressure wait or an acknowledgement wait still reaches teardown. + +**Persisted-request variant (application correlation id).** When the `Consumption endpoint` field records an application correlation id (a persisted request that `design-mode.md` routed here), generate the request leg only. The publisher adds `message_properties.CORRELATION_ID: msg_id` to the same `additional_message_properties` dict in step 4 (on either threading shape), and the subscriber stays unchanged. Do NOT generate an answer path: no reply topic, no reply subscription, no reply publish, and no `VERIFY:` marker beyond the four below. List the answer path under Developer-owned items in the checklist (`implement-mode.md` Step 6), and name it in the close-out as the developer's next step. + +## The marker contract + +These four `VERIFY:` marker strings (`VERIFY: CONNECTED`, `VERIFY: PUBLISH_ACKED`, `VERIFY: QUEUE_BOUND`, `VERIFY: MESSAGE_RECEIVED`) are the same strings the JCSMP guaranteed pub/sub leaf emits, character for character. They are read PER PROCESS: the subscriber output carries `QUEUE_BOUND` and `MESSAGE_RECEIVED`, the publisher output carries `PUBLISH_ACKED`, and both print `CONNECTED`. `verify.sh` reads these same strings in each process's own output (`implement-mode.md` Step 5), so never change one. diff --git a/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-mode.md b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-mode.md new file mode 100644 index 0000000..f63df74 --- /dev/null +++ b/plugins/solace-messaging-skills/skills/solace-application-development/references/python/implement-mode.md @@ -0,0 +1,288 @@ +# Implement Mode + +Generate a complete, runnable Python project from a confirmed design summary. The canonical output is TWO scripts (a publisher and a subscriber, or a requestor and a replier) plus the shared connection-config helper in ONE project directory with its own virtual environment. Each script is a near-verbatim adaptation of its reference sample. Near-verbatim adaptation of two samples is more reliable than merging them into one program, and a real deployment lifts each script into its own service or project. + +Implement mode is NOT only for the canonical shape. A web app, a dashboard, a service, or any embedded shape where the Python messaging layer lives inside a larger application follows the SAME steps: the same door question, the same leaf wiring for the messaging layer, the same generation outputs, and the same verification fallback. Building an off-catalog shape never waives a step in this file. + +Work through the steps in order. Read this file's grounding links only as each step needs them. Link the canonical doc and state the decision in one line, never paraphrase doc content. Resolve every dependency version at generation time; pin nothing in skill content (SKILL.md Invariant 3). + +**Coverage in this release.** Implement mode generates all five Python leaves on all three doors and verifies them with the bundled `scripts/verify.sh` (Step 5). + +## Generation outputs: the file contract + +Every Implement run writes ALL of these as generation output, on every app shape. None of them is gated on consent, and none of them waits for the end of the session: + +1. The generated scripts for the chosen leaf, plus the shared `solace_connection_config.py` helper, all under `src/` at the project root. Every source file starts with the AI-assisted disclaimer header (Step 4). +2. `requirements.txt` with the `solace-pubsubplus` version resolved per Step 3. +3. `config.example.json` (placeholders) and a `.gitignore` that ignores `config.json` and the virtual environment (Step 4). +4. A tailored `solace-verification-checklist.md` at each project root (tailoring rules in Step 6; the WRITE happens here at generation time, like the scripts). +5. A copy of the bundled Python `scripts/verify.sh` at each project root, plus a generated `verify-hooks.sh` beside it (Step 5), so the verification commands resolve from inside the project. + +**Project layout.** Every `.py` file the project generates goes under `src/`, and nothing else does. The project root holds `requirements.txt`, `.venv/`, `config.example.json`, `config.json`, `.gitignore`, `verify.sh`, `verify-hooks.sh`, `solace-design.md`, and `solace-verification-checklist.md`, so code and Markdown never share a directory. `src/` is a plain directory, not a package: no `__init__.py` and no `pyproject.toml`. Every script runs from the project root as `.venv/bin/python src/