Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion plugins/solace-messaging-skills/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Each `turnN.jsonl` in the work directory is the raw `--output-format stream-json
| Grader `turn` | Selects which `turn<N>.jsonl` file the grader reads. It is not a field inside the file. |
| `skill` (the implicit skill-fired gate) | An `assistant` event whose `message.content[]` holds a `tool_use` block with `name` `Skill`. Its `input.skill` is `plugin:skill-name`; the runner strips the prefix. |
| `assistant_grep` | `assistant` events, `message.content[]` blocks with `type` `text`. Thinking blocks and tool payloads are excluded. |
| `tool_use` | `assistant` events, blocks with `type` `tool_use`. The runner matches `name` and greps the compact JSON of `input`. |
| `tool_use` | `assistant` events, blocks with `type` `tool_use`. The runner matches `name` and greps the compact JSON of `input`, minus a Bash call's `description` field (narration, not the command). |
| `llm_judge` with `webfetch_urls` or `webfetch_results` | WebFetch `tool_use` blocks supply the URLs. `user` events whose `tool_result.tool_use_id` matches a WebFetch block supply the fetched content. |
| Session continuity (`--resume`) | `session_id` on the `result` event, with the `init` event as the fallback. |
| INFRA verdict for a turn | A missing `result` event, or `result.is_error` true with a `subtype` other than `error_max_turns`. |
Expand Down
150 changes: 138 additions & 12 deletions plugins/solace-messaging-skills/evals/output-evals.json

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions plugins/solace-messaging-skills/evals/trigger-evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
{"skill":"solace-application-development","prompt":"We've standardized on the Solace JMS API. How do I create a JMS ConnectionFactory and a MessageProducer to send to a queue in Java?","should_trigger":false},
{"skill":"solace-application-development","prompt":"I'm writing my Solace client in Python with the solace-pubsubplus library. How do I set up the messaging service and a persistent receiver?","should_trigger":true},
{"skill":"solace-application-development","prompt":"We've got a warehouse-robotics system where two Python services communicate through our Solace event broker using the solace-pubsubplus library. A RobotDispatcher publishes task assignments and a RobotController consumes them and must not lose any while a robot reboots. How should I structure the messaging between these two Python services?","should_trigger":true,"must_pass":true},
{"skill":"solace-application-development","prompt":"We're building a Python inventory service on the Solace Python API. Stock-level updates must not be lost while the service restarts. I want to keep the app single-threaded: no callbacks on the messaging path, the main loop should pull each message and acknowledge it itself, and the publisher should wait for each acknowledgement before it sends the next one. How should I structure this?","should_trigger":true},
{"skill":"solace-application-development","prompt":"My Python worker uses a thread pool and the GIL is limiting it. Should I move the work to multiprocessing or asyncio, and how do I keep a single thread of control over the job queue?","should_trigger":false},
{"skill":"solace-application-development","prompt":"Show me how to publish to a Solace topic from C# using the .NET messaging API.","should_trigger":false},
{"skill":"solace-application-development","prompt":"I'm writing a Go service with the Solace PubSub+ Messaging API for Go. How do I build a persistent message publisher and handle publish receipts?","should_trigger":false},
{"skill":"solace-application-development","prompt":"I'm on the Solace Messaging API for Java (com.solace:solace-messaging-client). How do I set up a MessagingService and a PersistentMessageReceiver in my Java service to consume from a durable queue?","should_trigger":false},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ If unclear, default to **Design**. Understand the messaging problem before gener

Three gates hold on every path:

- **The design-contract gate.** Implement mode never starts without a design contract, and never invents design values. Exactly three sources satisfy it: (1) a summary Design mode confirmed in this session: use it as-is, with no re-confirm; (2) an explicit summary the developer supplied in chat (the eight fields or an equivalent statement of them): use it directly; (3) a saved `solace-design.md` in the project: restate it in one line and proceed on confirm. A prose build request that merely mentions pattern details is NOT a contract: route it through Design mode, whose fully-specified path derives the summary, echoes it, and takes one confirm. The one carve-out is the mechanical-edit row above. An experienced developer therefore never has to walk the design tree, but never skips the contract either. A request to skip the design, skip all questions, or generate the full project right now does not waive this gate or the door question below: write no file before the confirm. Take the shortest path the gate allows instead: derive the summary from the prompt, present it, say in one line that one confirm starts the build, and wait for that confirm.
- **The design-contract gate.** Implement mode never starts without a design contract, and never invents design values. Exactly three sources satisfy it: (1) a summary Design mode confirmed in this session: use it as-is, with no re-confirm; (2) an explicit summary the developer supplied in chat (every field of the summary or an equivalent statement of them): use it directly; (3) a saved `solace-design.md` in the project: restate it in one line and proceed on confirm. A summary with no `Threading` field reads as `Threading: Async`; Implement mode derives it and does not ask. A prose build request that merely mentions pattern details is NOT a contract: route it through Design mode, whose fully-specified path derives the summary, echoes it, and takes one confirm. The one carve-out is the mechanical-edit row above. An experienced developer therefore never has to walk the design tree, but never skips the contract either. A request to skip the design, skip all questions, or generate the full project right now does not waive this gate or the door question below: write no file before the confirm. Take the shortest path the gate allows instead: derive the summary from the prompt (an unstated `Threading` records `Async`, named as the default; `python/design-mode.md` question 3), present it, say in one line that one confirm starts the build, and wait for that confirm.
- **Implement mode opens with a mandatory door question.** Ask Quickstart, Solace Suggested, or Custom, before you request or accept ANY broker details (`python/implement-mode.md` Step 0). Ask it even when a `config.json` with credentials already exists, even when a local broker is already running, and even when the requested app is bigger than the canonical generated shape; none of those answers the question. A confirmed design summary does not answer it either: the door question always follows the design contract, never merges into it.
- **Environment discovery never answers a question.** A running broker container, an existing config file, or found credentials are facts to report, not answers to consume. Report what you found, then still ask which broker the developer wants to target.

Expand Down Expand Up @@ -74,8 +74,10 @@ Live exceptions (these have no `docs.solace.com` `.md` form; use the live URL di
- `python/custom-mode.md`: the thin a la carte Custom overlay reached from implement-mode Step 0 on the Custom path; a leaf-aware checklist of the hardening knobs (secure session, DMQ, HA, decoupled projects, admin-provisioned queue) that generates exactly the ticked subset, reusing the Solace Suggested overlay steps for the mapped knobs.
- `python/python-guaranteed-publisher-sample.py`: best-practices publisher sample (basic-auth connect, PERSISTENT publish to a topic, publish receipt handling, graceful shutdown).
- `python/python-guaranteed-subscriber-sample.py`: best-practices receiver sample (provisions a durable queue plus a topic subscription, CLIENT-ack receiver).
- `python/python-guaranteed-publisher-blocking-sample.py` and `python/python-guaranteed-subscriber-blocking-sample.py`: the guaranteed pub/sub samples for the minimal-threading shape (`Threading: Minimal (blocking publish and receive)` in the design summary): blocking `publish_await_acknowledgement()` per message, and a main loop that pulls with `receive_message()` and ACKs on the application's own thread.
- `python/python-solace-connection-config.py`: the shared connection-config helper every generated app uses.
- `python/python-direct-publisher-sample.py` and `python/python-direct-subscriber-sample.py`: the direct pub/sub samples (bounded-buffer publish with a publish failure listener; direct topic subscription with discard detection).
- `python/python-direct-subscriber-blocking-sample.py`: the direct subscriber for the minimal-threading shape (a main loop that pulls with `receive_message()`); the direct publisher is the same sample on both shapes.
- `python/python-direct-requestor-blocking-sample.py`, `python/python-direct-requestor-sample.py`, and `python/python-direct-replier-sample.py`: the direct request-reply samples (blocking and non-blocking requestor; replier answering with `replier.reply()`).
- `python/verification-checklist.md`: the master verification checklist template that implement-mode Step 6 tailors into every generated project.
- `python/scripts/verify.sh`: the bundled verify script that Implement mode copies into every generated project; the `app` stage drives web and embedded shapes through a generated `verify-hooks.sh`.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The Custom overlay for Implement mode of the Python API. This file is reached from `implement-mode.md` Step 0 when the developer chooses the Custom door. Custom is a la carte: the developer ticks which hardening knobs to include, and generation applies exactly the ticked subset and nothing more. It is a thin overlay. It does NOT re-describe how each knob works. For the four knobs that map to a `solace-suggested-mode.md` step, it points at that step and reuses it, so `solace-suggested-mode.md` stays the single source of truth for each hardening delta.

This overlay defers to `implement-mode.md` for all leaf and sample mechanics, exactly the way `solace-suggested-mode.md` does: the Step 2 design-summary input contract, the Step 4 leaf dispatch (which leaf wiring file and samples each leaf reads, and the near-verbatim adaptation discipline), the AI-assisted disclaimer header on every generated script, explicit imports (never `from solace.messaging... import *`), and the fixed `VERIFY:` marker contract. Custom changes NO `VERIFY:` marker string. Any live run on the Custom path prints the same one-line live-broker heads-up (SKILL.md Invariant 6) immediately before it starts: name the broker host and the run's actual broker-side effects.
This overlay defers to `implement-mode.md` for all leaf and sample mechanics, exactly the way `solace-suggested-mode.md` does: the Step 2 design-summary input contract, the Step 4 leaf dispatch (which leaf wiring file each leaf reads, which sample pair the `Threading` field selects, and the near-verbatim adaptation discipline), the AI-assisted disclaimer header on every generated script, explicit imports (never `from solace.messaging... import *`), and the fixed `VERIFY:` marker contract. Custom changes NO `VERIFY:` marker string. Any live run on the Custom path prints the same one-line live-broker heads-up (SKILL.md Invariant 6) immediately before it starts: name the broker host and the run's actual broker-side effects.

## The checklist model: one up-front multi-select

Expand Down Expand Up @@ -50,7 +50,7 @@ On **Guaranteed Pub/Sub**, ticking the knob edits the subscriber's generation. D
KEEP the rest of the subscriber unchanged:

- The local `Queue.durable_exclusive_queue(queue_name)` or `Queue.durable_non_exclusive_queue(queue_name)` handle from the `Access type` field (the receiver still needs a queue object to bind to; the access type itself is now the administrator's broker setting, so state in a comment that it must match).
- `with_message_client_acknowledgement()`, `with_activation_passivation_support(...)`, `receive_async(...)`, `start()`, and `ack()` after processing.
- `with_message_client_acknowledgement()`, `with_activation_passivation_support(...)`, `receive_async(...)` (or the `receive_message(...)` loop on the minimal-threading shape), `start()`, and `ack()` after processing.
- The sample's `PubSubPlusClientError` branch around `start()`: it now reports a queue the administrator has not provisioned.

All of the Guaranteed Pub/Sub leaf's `VERIFY:` markers still emit, unchanged: `VERIFY: CONNECTED` after connect, `VERIFY: QUEUE_BOUND` after `start()`, and `VERIFY: MESSAGE_RECEIVED` after `ack()`. The publisher is untouched, so its `VERIFY: PUBLISH_ACKED` is unaffected. On the fan-out leaf, the drop applies to every subscriber script.
Expand Down
Loading
Loading