Repository navigation
Conversation
- Add question 3 (threading) to python/design-mode.md after the delivery question on the Direct Pub/Sub and guaranteed pub/sub branches, with a Step 1 signal row and the ninth Threading summary field (n/a on Request-Reply (Direct)) - Dispatch the sample pair on the Threading field in python/implement-mode.md; a missing field reads as Async and Implement does not ask - Give implement-guaranteed-pubsub.md and implement-direct-pubsub.md both shapes: publish_await_acknowledgement() plus receive_message() on the guaranteed leaf, the direct publisher plus receive_message() on Direct Pub/Sub - Resolve the blocking script names in verify.sh with one resolve_script helper and test the resolution offline in check-samples.sh - Cover the blocking shapes in the verification checklist, the Custom and Solace Suggested overlays, and python.md - Grep the Threading field in the Python design output eval; add minimal-threading output cases for the guaranteed and direct leaves and two trigger cases
aelsammak
marked this pull request as ready for review
October 27, 2026 12:35
aelsammak
marked this pull request as draft
October 27, 2026 12:54
- Treat the Python threading question as open on every pub/sub design unless the prompt states a threading preference; a resolved leaf no longer makes the prompt fully specified - Ask the question in one message with the one-line leaf name, so the answer also confirms the leaf - Keep Async as the recorded default on the skip-all-questions path and for an Implement-mode summary with no Threading field - Add a threading answer turn to the Quickstart and Solace Suggested implement cases and shift their turn-scoped graders - Grep for the threading question and Threading: Async on the affected Python design and implement cases
aelsammak
marked this pull request as ready for review
October 27, 2026 17:03
mtite-sol
reviewed
Oct 27, 2026
- Place every generated .py file under src/ at the project root so code and the Markdown artifacts never share a directory; the root keeps the config files, requirements.txt, .venv, .gitignore, the verify scripts, and the Markdown artifacts - Run the scripts from the project root as .venv/bin/python src/<script>.py so config.json is still read from the working directory and the sibling import of solace_connection_config resolves - Resolve, compile, and preflight the scripts under src/ in verify.sh; move the sample-check resolution fixtures under src/ - Import the modules from src/ in the output-eval compile grader; add layout graders to the four Python implement cases - Add the layout item to the verification checklist and a per-role src/ to the Solace Suggested overlay
- Replace "nine fields" and "all nine" in python.md, design-mode.md, and implement-mode.md with wording that points at the field list, so a future field changes one list and no sentence count - Keep the named field list in design-mode.md as the contract; the design-summary eval greps each field by name - Describe the JCSMP summary as the shorter list instead of "eight fields" - Follow the review suggestion on PR #17
mtite-sol
reviewed
Oct 27, 2026
mtite-sol
reviewed
Oct 27, 2026
| 4. Publish persistent messages to the design summary's topic as the sample does: one `message_builder()` template with `PERSISTENT_TIME_TO_LIVE` set once, `build(payload, additional_message_properties={APPLICATION_MESSAGE_ID: msg_id})` per message, and `publish(message, topic, user_context=msg_id)`. See [Publishing Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Publish.md). | ||
| 5. EXIT on its own: replace the sample's continuous publish loop with the application's own messages, or with one message on a try-it run, then return from the publish step. `teardown_solace(...)` in `main()`'s `finally` calls `terminate(grace_period)`, which sends what is buffered and waits for the broker's receipt of every published message, so the `VERIFY: PUBLISH_ACKED` marker prints before the script exits. Drop the stats thread with the loop. Keep the SIGINT/SIGTERM handler, so Ctrl-C during a back pressure wait still reaches teardown. | ||
| 3. `publisher.start()`. Callback shape: then `publisher.set_message_publish_receipt_listener(...)` AFTER `start()`, as the sample orders them. In the receipt handler's ACK branch, emit the publish-ACK marker: `trace("VERIFY: PUBLISH_ACKED")`. CRITICAL: the sample logs the ACK with `logger.debug`, which the INFO-level logging never shows, so the marker would never appear. Emit it through `trace(...)`, not through the logger. In the NACK branch, keep the sample's warning and also set the shared state's `exit_code` to 1, so a rejected message makes the publisher exit non-zero. Pass the shared state to the handler to do this. Blocking shape: NO receipt listener; the marker moves into step 4. | ||
| 4. Publish persistent messages to the design summary's topic as the sample does: one `message_builder()` template with `PERSISTENT_TIME_TO_LIVE` set once, and `build(payload, additional_message_properties={APPLICATION_MESSAGE_ID: msg_id})` per message. Callback shape: `publish(message, topic, user_context=msg_id)`. Blocking shape: `publish_await_acknowledgement(message, topic, PUBLISH_ACK_TIMEOUT_MS)` with the sample's POSITIVE timeout in milliseconds and no `user_context`; a normal return IS the broker's ACK, so emit `trace("VERIFY: PUBLISH_ACKED")` immediately after the call returns. Keep the blocking sample's three discard branches (`PubSubTimeoutError`, the NACK error classes, and `PublisherOverflowError`), which log the message id at INFO and continue (the Discard the Message strategy from the Publishing Persistent Messages page; `verify.sh` judges the stage by the marker, so a discarded message fails the stage whatever the exit code), and its `PubSubPlusClientError` branch, which sets `exit_code` to 1 and stops. See [Publishing Persistent Messages](https://docs.solace.com/API/API-Developer-Guide-Python/Python-PM-Publish.md). |
There was a problem hiding this comment.
On your exit-code question: I'd make the blocking shape match the async one. verify.sh passes the publisher only on exit 0 plus VERIFY: PUBLISH_ACKED. The async wiring sets exit_code = 1 on a NACK, so a NACK fails the stage. The blocking sample's timeout/NACK branches only log, so an app that publishes several orders and gets one NACK exits 0 with markers from the others and passes. "A discarded message fails the stage whatever the exit code" only holds for a one-message run.
Suggest: keep the Discard strategy (log and continue), but set exit_code = 1 in the NACK branch, and arguably the timeout branch.
There was a problem hiding this comment.
Step 4 reads right now, thanks. One thing on the new grader, though: ^[^#]*exit_code *= *1 in guaranteed_publisher_blocking.py already matches four lines the sample carries (the PubSubPlusClientError, IncompleteMessageDeliveryError, failed terminate() and service-interruption branches), so a generation without the discard-branch change still passes it. file_grep has no count option, so I'd either swap it for an llm_judge check on the discard branches or drop it rather than keep a grader that can't fail.
Author
There was a problem hiding this comment.
Agreed, dropped in 716f80b. A file_grep cannot tie the line to the discard branches. The llm_judge grader only sees the transcript and the fetched pages, not the generated files, so that check needs a runner change first. Step 4 stays the guard.
mtite-sol
reviewed
Oct 27, 2026
mtite-sol
reviewed
Oct 27, 2026
mtite-sol
requested changes
Oct 27, 2026
mtite-sol
left a comment
mtite-sol
left a comment
There was a problem hiding this comment.
It's looking good! Just added some comments in-line for you to look at.
- Prove both script names per stage in the sample-check resolution fixture: separate-projects mode with a stub interpreter in the long-running role's directory reaches the foreground role's check, and a missing script is named in its own directory - Set exit_code to 1 in each of the blocking publisher's three discard branches so a run that loses one message out of many fails the verify.sh stage, matching the callback shape's NACK branch; grade it on the guaranteed minimal case - Put src/ on the canonical-shape verify-hooks.sh example and name the blocking pair for the minimal shape; grep the hooks commands on the three Python implement cases - Record the replier's receive_async() callback in the Request-Reply (Direct) Threading line when the prompt asked for minimal threading
- Remove the file_grep grader for exit_code = 1 in guaranteed_publisher_blocking.py from the minimal-threading guaranteed implement case - The bundled blocking sample already carries four matching lines, so a generation without the discard-branch change passed it - file_grep is line-based and llm_judge sees no generated files, so no grader can tie the line to the discard branches - Step 4 of implement-guaranteed-pubsub.md stays the guard for the discard-branch exit code
mtite-sol
approved these changes
Oct 27, 2026
mtite-sol
left a comment
mtite-sol
left a comment
There was a problem hiding this comment.
Changes LGTM! Just one comment on the sonnet failure being under discussion - is this something you plan on addressing in a follow up PR? Just want to make sure this wasn't something you forgot about.
Author
Yup, working on the sonnet issue in the background! I will get this fixed as part of this PR 😄 |
…se positives - Make the trust store line its own closing step in implement-mode.md, so a plaintext Quickstart close carries it - State in implement-direct-pubsub.md that a continuous feed still gets a bounded batch that exits, with no env-var cap and no app stage; the generated two-script project never takes the app stage - Name the threading answers with the labels multi-threaded and minimal threading in design-mode.md question 3 - Drop a Bash call's description field in the runner's tool_use matching and make the six forbidden-verify patterns context-aware, so quoted doc text no longer matches - Note the description drop in the evals README
aelsammak
added a commit
that referenced
this pull request
Oct 28, 2026
* EBP-3103: Add best-practice Python pub/sub reference samples (#8) * EBP-3103: Add best-practice Python pub/sub reference samples - Add python-solace-connection-config.py plus guaranteed and direct publisher and subscriber samples under references/python, derived from the Solace Python samples and elevated to the Python developer guide and C API best practices - Register service listeners before connect(), use a 3000 ms reconnect interval, pair reject back pressure with a readiness listener, check publish receipts, subscribe the CREATE_ON_START queue to the topic, client-ack after processing, and shut down gracefully on SIGINT and SIGTERM - Read config.json or command-line arguments through the shared helper, mirroring jcsmp-solace-connection-config.java, with no version pin - Reword JCSMP sample comments to drop the legacy product name and the comparisons to upstream samples - Verify locally against solace-pubsubplus resolved from PyPI with a no-broker smoke test and live round trips per pair * EBP-3103: Remove PEP 563 future import from Python samples - Drop `from __future__ import annotations` and its explanatory comment from the five Python reference samples - The built-in generics in the signatures run natively on every Python release that Solace supports, so the shim added nothing - Match the upstream SolaceSamples/solace-samples-python style, which carries no future import - Verified: all five modules import with solace-pubsubplus installed * EBP-3103: Condense the HA failover comment in the Python samples - Replace the ten-line HA and replication comment with a five-line HA-only comment that mirrors the JCSMP sample - List the C API Best Practices values as Python API calls: connection and reconnection retry strategies plus the CONNECTION_RETRIES_PER_HOST transport property - Drop the replication guidance, the forever_retry remark, and the comma-separated host list - Shorten the back pressure comment above PUBLISH_BUFFER_CAPACITY to one line in both publisher samples * EBP-3103: Condense the Python connection-config helper docstring - Shorten the module docstring of python-solace-connection-config.py while keeping every contract point (key names, pass-through, override order, gitignore rule, empty-password handling) - Move the AI-assisted code header block to the end of the docstring to match the other Python samples * EBP-3103: Disable the trigger-evals CI job - Add a job-level `if: false` to trigger-evals: no policy permits an API key on this public repo, so the job could only self-skip and asserted nothing - Keep the job definition so it can be re-enabled by removing the `if` once a key is allowed; trigger evals continue to run locally * EBP-3103: Ignore Python bytecode caches - Add __pycache__/ to .gitignore so byte-compiling the Python reference samples leaves no untracked files * EBP-3103: Fix shutdown and exit-code handling in the Python samples - Register the SIGINT/SIGTERM handlers before setup_solace(), so a SIGTERM during the blocking connect() still reaches teardown instead of killing the process - Return exit code 1 from every failure exit (failed queue bind, service interruption, receiver termination, publish() failure); main() now returns the code and __main__ passes it to sys.exit() - Correct the guaranteed subscriber's terminate() comments: the receiver turns TERMINATED once its buffer is empty, so the ack() of the message still in the handler can fail and the message is redelivered - Verified against a live broker: both round trips exit 0, and the bind failure, a service interruption (reconnect exhausted), and SIGTERM during connect each run teardown and exit 1 * EBP-3103: Explain the terminate() grace period in the Python samples - Describe why the samples bound terminate(): a predictable shutdown time, for example within a process supervisor's stop timeout, so the disconnect still runs - State the API default (600000 ms) as a fact and invite adjusting the value to suit the application * EBP-3103: Fix review findings in the Python samples and CI docs - Name MessageDestinationDoesNotExistError for the no-subscription-match NACK, matching the library's subcode mapping - Drop "a broker rejection" from the direct publisher failure listener comment; broker events never reach that listener - Split terminate() handling into IncompleteMessageDeliveryError and other client errors, and exit non-zero on both - Update the exit_code comments to match, and drop the inaccurate "failed bind" case in the guaranteed subscriber - Describe the disabled trigger-evals CI job in the evals README and point to local runs - Include *.py in the link check so the Python sample header URLs are swept * EBP-3103: Stop naming specific model versions in the evals README - Use the claude CLI aliases (sonnet, opus) in the run examples so they always select the latest model in each family - Describe the runner defaults, the CI matrix, and the judge by model family instead of pinned IDs - Remove the stale claude-opus-5 reference * EBP-3103: Restructure the Python samples as functions and bound buffers - Replace the Java-style app classes with module-level functions (main, setup_solace, run_publish_loop or await_messages, teardown_solace) that share one dataclass state - Keep the listener classes, since the Python API requires listener subclasses; they now take the shared state - Replace the SolaceConnectionConfig class with one load_service_properties() function that returns the from_properties() dict - Use X | None type hints for the Python 3.10+ floor - Publishers: on_back_pressure_wait(50) replaces on_back_pressure_reject(1000), so the readiness listener and the PublisherOverflowError retry are removed - Direct subscriber: bound the receive buffer with on_back_pressure_drop_latest(50) - Verified with the smoke test, both live round trips, and a frozen-proxy test of a blocked publish() (returns on drain, or raises after the reconnect budget) * EBP-3103: Build Solace resources before connect in the Python samples - setup_solace() now builds the service and the publisher or receiver before any connection exists, and returns a fully populated state, so no state object is created empty and filled in later (no | None fields) - New connect_solace() connects the service and starts the endpoint; main() calls setup_solace() before its try and runs teardown in finally - Remove the reconnect pause from both publishers: a guaranteed publish waits in the buffer during a reconnect, and the direct publisher now documents that messages published during a reconnect are lost and reported through the publish failure listener, pointing to guaranteed messaging for lossless delivery - Raise the back pressure buffer from 50 to 1000, the value the Python developer guide uses in its examples (the API has no default capacity), and note that it counts messages and is tunable to deployment memory * EBP-3103: Publish direct messages builderless and template guaranteed ones - Direct publisher passes the payload straight to publish() with no OutboundMessageBuilder, and shows additional_message_properties for per-message headers; a comment notes the builder suits "templated" messages with many fixed headers - Guaranteed publisher uses the builder as a template: the time-to-live (new MESSAGE_TTL_MS, 60 s) is set once, since build() copies the template into each message, and the per-message application message id goes through build()'s additional_message_properties so it never changes the template * EBP-3103: Remove the publish rate throttle from the samples - Remove APPROX_MSG_RATE_PER_SEC and the delay after each send from the Python and JCSMP direct and guaranteed publisher samples, so the loop publishes as fast as send() or publish() returns, the way an application publishes each message as soon as it has one - Update the demo-harness rule in implement-mode.md: the continuous publish loop replaces the pacing Thread.sleep in the list, and generated applications never add a delay or rate limit between sends that the application does not ask for - Output evals not run for the implement-mode.md change * EBP-3103: Let configuration override the reconnect defaults in Python - Set the reconnection retry strategy before from_properties() in all four samples: the builder merges calls in order and a later call wins, so the code's strategy is now a default that reconnection keys in the properties (for example config.json) override; update the connection config helper docstring to match - Load the Solace properties in main() and pass them to setup_solace(), so converting the application's input (config.json or argv) stays out of the Solace setup - Log a buffered-message loss at direct publisher shutdown as a warning and keep exit code 0, since direct (at-most-once) messaging allows it * EBP-3103: Set the message handler before starting the Python receivers - Call receive_async() before receiver.start() in both subscriber samples, so the handler is in place when delivery begins and messages do not wait in the API buffer - Keep receive_async() after connect(), since _can_receive_message() requires a connected service - Address PR #8 review comment on the guaranteed subscriber sample * EBP-3103: Add an async suffix to the Python sample names - Rename the four Python samples to python-<delivery>-<role>-async-sample.py - Update the generated project file names to match, for example guaranteed_subscriber_async.py - Use the async/sync terms from the Python developer guide, so later sync samples (publish_await_acknowledgement, receive_message) fit the scheme - Add guidance to the guaranteed publisher on how to choose the message TTL: at least the longest consumer recovery time (restart plus reconnect), or a multiple of it * EBP-3104: Add best-practice Python request-reply reference samples (#9) - Add python-direct-requestor-blocking-sample.py (publish_await_response with a positive reply timeout and one retry on PubSubTimeoutError), python-direct-requestor-sample.py (non-blocking publish() returning a Future per request, then collected), and python-direct-replier-sample.py (receive_async with a RequestMessageHandler, a None-replier guard, and reply()) - Direct only: the Python API has no guaranteed request-reply, so there is no counterpart to the JCSMP guaranteed requestor and replier - Follow the EBP-3103 conventions: shared connection-config helper, listeners registered before connect(), 3000 ms reconnect interval, SIGINT/SIGTERM shutdown, bounded terminate(), exit code 1 on failure and 0 on a shutdown signal - Document that the request-reply builders have no back-pressure setting and that their from_properties() applies no property - Rename the four EBP-3103 samples to drop the -async suffix: a -blocking suffix now marks only a variant that waits for the broker's answer, matching the SolaceSamples naming - Correct a stale guaranteed publisher comment that said the direct publisher pauses during a reconnect - Verified against solace-pubsubplus 1.11.0: pyflakes clean, a no-broker smoke test, and live round trips from both requestors to the replier * EBP-3105: Add Python API routing and the guaranteed pub/sub Quickstart (#10) * EBP-3105: Add Python API routing and the guaranteed pub/sub Quickstart - Route Solace Python API requests (solace-pubsubplus, MessagingService, PersistentMessageReceiver) to a new references/python.md entry file; remove Python from the description's exclusions and keep JMS, Go, .NET, and JavaScript excluded - Ask "JCSMP (Java) or Python?" when a Solace build request names no language; workspace files are facts to report, not the answer - Generalize Invariants 3 and 5: the registry per API (Maven metadata for JCSMP, PyPI info.version for Python) and the API's own bundled verify script, with the compile-only fallback for an API that has none yet - Add python/prerequisites.md, design-mode.md (five leaves; no guaranteed request-reply in the Python API), implement-mode.md, and implement-guaranteed-pubsub.md, mirroring the JCSMP files - Quickstart generates a venv project with requirements.txt pinned to the version PyPI reports at generation time, a gitignored config.json, and the four JCSMP VERIFY: markers; it ends in py_compile plus an import smoke and handed-back run commands until the Python verify.sh lands - Document the concurrency rules: receive_async() is not asyncio and calls the handler on a new thread per message, and the API cannot be used with multiprocessing - The direct pub/sub and direct request-reply leaves, and the Solace Suggested and Custom doors, stop with "not yet available" for Python until EBP-3106 - Flip the two Python trigger cases to positives (the guaranteed pub/sub one must_pass) and add Go, JavaScript, asyncio, and Kafka Python near-miss negatives * EBP-3105: Apply review fixes to the Python routing and Quickstart path - Scope the JCSMP routing row to Java, so a request with no language gets the JCSMP or Python question - Let a saved solace-design.md title select the API, and restore the input contract sentence in Python design mode - Exclude the Solace Messaging API for Java, which shares MessagingService and PersistentMessageReceiver with Python; add a negative trigger case - Add the environment failure rule to the manual run, so a broker or credential failure never leads to code changes - Scale the live broker heads-up, run commands, and round trip items to every fan-out subscriber; require an identifier for <service> - Keep the requirements.txt pin when PyPI is unreachable - Add header, checklist, and version drift checks to Step 5 - Align the interim checklist with the JCSMP master: doc link per item, command line credentials under Your responsibility, stronger auth and grounding items - Ground the asyncio ack rule in the Consuming Persistent Messages page, and add traps for the parametrized_retry doc comment and PUBLISH_ACKED without delivery - Name the missing verify script and the Request-Reply page in python.md - Fix the Try Broker description in both prerequisites files to match the page * EBP-3229: Add CLAUDE.md and refresh the root README (#11) - Add a root CLAUDE.md for AI coding agents: where the skills, references, evals, and tools live, the exact check and eval commands, repo conventions, a gotchas section, and the PR workflow - Update the README intro and the solace-application-development row to describe the current JCSMP and Python coverage instead of future releases - Add a Contributing section that routes bugs and feature requests to Solace Support, the Solace Ideas portal, or the Solace Community - List CLAUDE.md in the repository layout and describe tools/ by role * EBP-3106: Add Python direct leaves and the Solace Suggested and Custom overlays (#12) * EBP-3106: Add Python direct leaves and the Solace Suggested and Custom overlays - Add implement-direct-pubsub.md and implement-request-reply.md leaf wiring files for the Python API, mirroring the JCSMP leaf files and their VERIFY: marker strings; the blocking requestor is the default and the non-blocking requestor is a variant - Add solace-suggested-mode.md (TLS with explicit certificate checks and the developer's trust store directory, the HA question mapped to the RetryStrategy values, DMQ eligibility on the persistent publisher, separate Python projects) and custom-mode.md (the same five a la carte knobs as JCSMP) - Remove the "not yet available" stops for the direct leaves and the hardened doors from implement-mode.md and design-mode.md, and extend the run commands, the manual-run rule, and the interim checklist to every leaf and door - Route a prompt that needs a persisted request to Guaranteed Pub/Sub with an application correlation id, never to Request-Reply (Direct) - Explain the trust store to the developer: the Python API has no default trust store, so a tcps:// host fails at build() without one; Quickstart now asks plaintext or secure and gives the same guidance on a secure host - Note that the Python API re-applies direct subscriptions on reconnect, so the JCSMP reapply-subscriptions property has no Python equivalent * EBP-3106: Hold the design gate when the user asks to skip questions - State in the jcsmp.md and python.md design-contract gate that a request to skip the design or all questions does not waive the gate or the door question - Direct the shortest allowed path instead: derive the summary, present it, and wait for one confirm before writing any file - Sonnet failed appdev-code-before-confirm-negative in 2 of 2 runs before the change and passed 3 of 3 after it - Add an exclude_pattern to the appdev-quickstart-implement-full verify.sh grader so verify.sh text inside a heredoc file write is not counted as a run * EBP-3106: Align Python Custom TLS and checklist rules with JCSMP - Ask the scheme question on Custom when the secure-session knob is unticked, and give a tcps:// host the Quickstart trust store guidance, example config, next steps, and checklist items - Split the DMQ checklist item into the API eligibility flag and the broker-side DMQ, which always stays the developer's responsibility - Move the HA reconnect values to Delivered only when the HA-failover question is answered yes, matching the JCSMP checklist * EBP-3106: Generate only the request leg for a persisted Python request - Record only the application correlation id in the design summary, and tell the developer the answer path is theirs to build - Add a persisted-request variant to the Guaranteed Pub/Sub leaf: the publisher sets CORRELATION_ID, no reply wiring or extra VERIFY markers, and the answer path is listed as developer-owned - Name the reconnect baseline on the unticked HA knob in the Python and JCSMP Custom modes, so the checklist no longer calls it the fail-fast API defaults * EBP-3107: Add Python verify.sh, checklist template, and smoke fixture (#13) * EBP-3107: Add Python verify.sh, checklist template, and smoke fixture - Add references/python/scripts/verify.sh: stage-dispatched run-and-observe verification (publisher, consumer, roundtrip, direct, direct-request-reply, app) with the same VERIFY: markers and 0/1/2 exit contract as the JCSMP script - Run each role with its project's .venv interpreter; connection arguments are optional when config.json is present, and separate projects use the --subscriber-dir/--publisher-dir and --replier-dir/--requestor-dir flags - Classify Python API connection, authentication, and TLS signatures (observed live) and the queue bind denial as environment failures (exit 2) - Move the interim checklist into references/python/verification-checklist.md and rewrite implement-mode Steps 4 to 6 around the bundled script - Add the smoke fixture (latest solace-pubsubplus from PyPI, py_compile, build every sample's API objects without a broker, config helper paths) and the python-smoke-check CI job beside compile-check - Update SKILL.md Invariant 5, python.md, the Suggested overlay, the leaf files, the evals README prerequisites, and the CLAUDE.md commands * EBP-3107: Rename the offline sample checks to <api>-sample-check - Rename the CI jobs compile-check and python-smoke-check to jcsmp-sample-check and python-sample-check, so every API's offline check of its baked samples (no broker, no API key) has one name pattern - Move each fixture to references/<api>/evals/sample-check/ with the entry script check-samples.sh (Python driver check_samples.py) and rename the fixture pom artifactId to jcsmp-sample-check - Update the CI step names, CLAUDE.md commands and roster convention, and the SKILL.md reference-file lines - The scheme extends to the Java and Go APIs as java-sample-check and go-sample-check * EBP-3107: Close false passes in verify.sh for both APIs - Require PUBLISH_ACKED for roundtrip and a MESSAGE_RECEIVED beyond those already logged, so a message an earlier run left on the durable queue no longer passes a broken publisher - Record the foreground role's exit code (124 on a hang) and fail the publisher, roundtrip, direct, and request-reply stages when it is non-zero - Count PASS_MARKER lines before the app-stage trigger so a startup backlog does not pass the stage - Refuse Python connection arguments when config.json exists, because the scripts prefer config.json and the heads-up named a different broker - Derive the Python sample-check module list from the copied samples instead of a second hand-kept list - Scope the exit-code docs: the app-stage pass rule, the foreground exit code, and precondition exits that do not count as fix-loop attempts - Apply the same verify.sh fixes to the shipped JCSMP script and fix the stale compile.sh comments * EBP-3107: Pin CI runners to ubuntu-26.04 and Python to 3.11 - Replace every ubuntu-latest runner label with ubuntu-26.04 so a runner image change cannot alter CI without a commit - Pin the python-sample-check job to Python 3.11, the oldest Python version still in support, instead of 3.x * EBP-3107: Classify stalled connects and settle stale-message counts - Classify a long-running role that is still connecting at the readiness deadline (no VERIFY: CONNECTED) as an environment failure, exit 2, naming the host; an unroutable host or a blocked port previously gave exit 1 in both APIs, and a refused port did too with the Python HA retry values - Re-check the environment signatures after the stage before reporting a code failure, and match Python's "Session connect timeout" - Take the stale MESSAGE_RECEIVED and PASS_MARKER counts only after they stop changing for 2 seconds, so a queued message delivered just after the bind no longer passes roundtrip - Show the request-reply "no reply" diagnosis whenever the requestor exited on its own without a reply, since the Python requestors exit 1 in that case - Add the stalled connect to the exit 2 text in both implement-mode.md files and script headers * EBP-3108: Add Python output evals and API-aware graders (#15) * EBP-3108: Add Python output evals and API-aware graders - Give compile, release_match, disclaimer, and live_verify one api field (jcsmp default, python); release_match replaces maven_release_match and disclaimer replaces java_disclaimer - Python compile installs each requirements.txt into a fresh venv, then runs py_compile and an import smoke; Python release_match checks the PyPI pin - Python live_verify writes config.json after the subject model finishes; add OUTPUT_EVAL_BROKER_TRUST_STORE for tcps:// hosts (no default trust store) - Add file_exists count, and skip .venv and __pycache__ in file globs - Add six Python cases mirroring the JCSMP set, plus persisted-request routing - Exclude heredoc checklist text from the verify.sh run check in both Suggested cases (false positive on Sonnet) - Document the Python graders and the trust store variable in evals/README.md * EBP-3108: Address PR review findings in output evals - Limit the verify.sh heredoc exclusion to text inside a heredoc body - Delete config.json after the Python live round trip and fail as INFRA when the write fails - Correct README claims about compile cases, the import smoke, and timeout scope * EBP-3108: Strip heredoc bodies before tool_use matching - Replace the four heredoc exclude_pattern regexes with an opt-in ignore_heredoc_bodies flag on the tool_use grader, so each grader keeps one copy of the verify.sh verb list - Add a strip_heredoc_bodies jq function that drops heredoc bodies from Bash commands before tojson, closing the hole where a lowercase or digit delimiter (EOF2, Eof) hid a real verify.sh run - Escape the quote class as \\x27 so the jq string literal compiles; a compile error would silently empty every tool_use grader - Add ignore_heredoc_bodies to the tool_use schema and the runner grader notes * EBP-3125: Add blocking Python publish and receive reference samples (#16) * EBP-3125: Add blocking Python publish and receive reference samples - Add python-guaranteed-publisher-blocking-sample.py: publish_await_acknowledgement() with a positive timeout, no receipt listener; a timeout, NACK, overflow, or other publish failure stops the loop with exit code 1 - Add python-guaranteed-subscriber-blocking-sample.py: receive_message(timeout), process, then ack() on the main thread - Add python-direct-subscriber-blocking-sample.py: receive_message(timeout) loop with discard detection - Blocking subscribers call terminate(0): no handler thread drains the buffer after the loop stops; unacked guaranteed messages are redelivered, buffered direct messages drop - Keep the async twins' structure; no time.sleep(), no receive_async(), rate reports inline instead of a stats thread - Add the three samples to the Python sample-check roster * EBP-3125: Correct terminate() and ack() comments in the blocking samples - Reword the publisher's terminate() comments and log text: without a receipt listener the API waits only for a message still unsent in its publish buffer, and does not wait for or report a sent message whose ACK is late - Reword the guaranteed subscriber's ack() failure example: ack() still works while the receiver is terminating and raises only once the receiver is terminated by the broker or the service is down - Verified against the solace-pubsubplus publisher and receiver implementation during the PR review * EBP-3125: Discard failed publishes in the blocking publisher - Handle a timeout, a NACK, or a buffer overflow with the "Discard the Message" strategy from the Python developer guide (Strategies for Handling Publish Receipt Errors): log at INFO and continue with the next message - Keep ERROR, exit code 1, and stop only for any other PubSubPlusClientError, where sending cannot continue - Note that a message discarded after a timeout can still be delivered, and name the wait-and-retry and fixed-retry alternatives - Addresses PR review feedback on log levels and stopping at the first failure * EBP-3125: Base the blocking publisher ACK timeout comment on the C API ACK timer - Explain PUBLISH_ACK_TIMEOUT_MS from the C API publish ACK timer: an unACKed message is resent every 2000 ms, so 5000 ms allows three attempts - State the convention: keep the value a little above a multiple of 2000 ms - Addresses PR review feedback on how the ACK timeout is chosen * EBP-3331: Add the Python design-tree threading question (#17) * EBP-3331: Add the Python design-tree threading question - Add question 3 (threading) to python/design-mode.md after the delivery question on the Direct Pub/Sub and guaranteed pub/sub branches, with a Step 1 signal row and the ninth Threading summary field (n/a on Request-Reply (Direct)) - Dispatch the sample pair on the Threading field in python/implement-mode.md; a missing field reads as Async and Implement does not ask - Give implement-guaranteed-pubsub.md and implement-direct-pubsub.md both shapes: publish_await_acknowledgement() plus receive_message() on the guaranteed leaf, the direct publisher plus receive_message() on Direct Pub/Sub - Resolve the blocking script names in verify.sh with one resolve_script helper and test the resolution offline in check-samples.sh - Cover the blocking shapes in the verification checklist, the Custom and Solace Suggested overlays, and python.md - Grep the Threading field in the Python design output eval; add minimal-threading output cases for the guaranteed and direct leaves and two trigger cases * EBP-3331: Always ask the threading question unless the prompt states it - Treat the Python threading question as open on every pub/sub design unless the prompt states a threading preference; a resolved leaf no longer makes the prompt fully specified - Ask the question in one message with the one-line leaf name, so the answer also confirms the leaf - Keep Async as the recorded default on the skip-all-questions path and for an Implement-mode summary with no Threading field - Add a threading answer turn to the Quickstart and Solace Suggested implement cases and shift their turn-scoped graders - Grep for the threading question and Threading: Async on the affected Python design and implement cases * EBP-3331: Put generated Python scripts under src/ - Place every generated .py file under src/ at the project root so code and the Markdown artifacts never share a directory; the root keeps the config files, requirements.txt, .venv, .gitignore, the verify scripts, and the Markdown artifacts - Run the scripts from the project root as .venv/bin/python src/<script>.py so config.json is still read from the working directory and the sibling import of solace_connection_config resolves - Resolve, compile, and preflight the scripts under src/ in verify.sh; move the sample-check resolution fixtures under src/ - Import the modules from src/ in the output-eval compile grader; add layout graders to the four Python implement cases - Add the layout item to the verification checklist and a per-role src/ to the Solace Suggested overlay * EBP-3331: Name the design summary fields without a count - Replace "nine fields" and "all nine" in python.md, design-mode.md, and implement-mode.md with wording that points at the field list, so a future field changes one list and no sentence count - Keep the named field list in design-mode.md as the contract; the design-summary eval greps each field by name - Describe the JCSMP summary as the shorter list instead of "eight fields" - Follow the review suggestion on PR #17 * EBP-3331: Apply the review fixes on the threading question PR - Prove both script names per stage in the sample-check resolution fixture: separate-projects mode with a stub interpreter in the long-running role's directory reaches the foreground role's check, and a missing script is named in its own directory - Set exit_code to 1 in each of the blocking publisher's three discard branches so a run that loses one message out of many fails the verify.sh stage, matching the callback shape's NACK branch; grade it on the guaranteed minimal case - Put src/ on the canonical-shape verify-hooks.sh example and name the blocking pair for the minimal shape; grep the hooks commands on the three Python implement cases - Record the replier's receive_async() callback in the Request-Reply (Direct) Threading line when the prompt asked for minimal threading * EBP-3331: Drop the blocking publisher exit-code grader - Remove the file_grep grader for exit_code = 1 in guaranteed_publisher_blocking.py from the minimal-threading guaranteed implement case - The bundled blocking sample already carries four matching lines, so a generation without the discard-branch change passed it - file_grep is line-based and llm_judge sees no generated files, so no grader can tie the line to the discard branches - Step 4 of implement-guaranteed-pubsub.md stays the guard for the discard-branch exit code * EBP-3331: Fix the Sonnet output eval misses and the verify grader false positives - Make the trust store line its own closing step in implement-mode.md, so a plaintext Quickstart close carries it - State in implement-direct-pubsub.md that a continuous feed still gets a bounded batch that exits, with no env-var cap and no app stage; the generated two-script project never takes the app stage - Name the threading answers with the labels multi-threaded and minimal threading in design-mode.md question 3 - Drop a Bash call's description field in the runner's tool_use matching and make the six forbidden-verify patterns context-aware, so quoted doc text no longer matches - Note the description drop in the evals README * Chore(deps): Bump org.apache.logging.log4j:log4j-api from 2.26.0 to 2.26.1 in /plugins/solace-messaging-skills/skills/solace-application-development/references/jcsmp/evals/sample-check (#14) * Chore(deps): Bump org.apache.logging.log4j:log4j-api Bumps org.apache.logging.log4j:log4j-api from 2.26.0 to 2.26.1. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-version: 2.26.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): Bump org.apache.logging.log4j:log4j-core from 2.26.0 to 2.26.1 - Keep log4j-core on the same version as log4j-api, which this PR bumps to 2.26.1 - The pin is fixture-only tooling for the JCSMP sample check; it is not skill content --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adi El-Sammak <71227923+aelsammak@users.noreply.github.com> * EBP-3341: Require bytearray or str payloads and relax threading labels (#18) - Add a payload type rule to the Python demo-harness rule: a bytes payload fails in build() and publish(), so wrap encoded text in bytearray - Repeat the rule in the guaranteed, direct, and request-reply publish steps and add a checklist conformance item - Accept equivalent names for the two threading answers in the design tree - Relax the matching output eval grader pattern in four Python cases * EBP-3341: Bump solace-messaging-skills to 0.2.0 (#19) - Bump the plugin version from 0.1.1 to 0.2.0 for the release - Minor bump because the release adds Solace Python API coverage and the skill now triggers on Python prompts --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this change?
EBP-3125 added blocking Python samples, but the skill had no path to them. This change adds a threading question to the Python design tree and routes the minimal-threading answer to the blocking samples. It also puts every generated Python script under
src/, so code and Markdown no longer share a directory.How is this accomplished?
design-mode.md: question 3 (after delivery, before consumer count) on the Direct Pub/Sub and guaranteed pub/sub branches, always asked unless the prompt states the answer. Ninth summary fieldThreading: <Async | Minimal (blocking publish and receive) | n/a>; Request-Reply (Direct) recordsn/a. A Step 1 signal row locks the field from prompt intent.implement-mode.md: nine-field contract. A missingThreadingfield reads asAsyncwith no question. The dispatch table selects the sample pair byThreading; the fixed names gainguaranteed_publisher_blocking.py,guaranteed_subscriber_blocking.py, anddirect_subscriber_blocking.py.publish_await_acknowledgement()plusreceive_message(); Direct Pub/Sub on Minimal keeps the direct publisher plusreceive_message().verify.sh: aresolve_scripthelper falls back to the blocking name, as the requestor stage already did.check-samples.shtests the resolution offline.python.mdcover the blocking shapes.Threadinggrep on the Python design case, two new minimal-threading output cases (guaranteed and direct), two trigger cases.src/layout: every generated.pyfile goes undersrc/, a plain directory with no package files; the root keeps the config files,requirements.txt,.venv,.gitignore, the verify scripts, and the Markdown artifacts. Scripts run from the root as.venv/bin/python src/<script>.py.verify.shresolves, compiles, and preflights undersrc/; the sample check, the output-eval compile grader, and layout graders on the four Python implement cases follow.appstage; question 3 names its answers multi-threaded and minimal threading; the runner drops a Bash call'sdescriptionfield before tool_use matching, and the six forbidden-verify patterns require an invocation context, so quoted doc text no longer matches.Anything reviews should focus on/be aware of?
Local checks green (plugin validate, version sync, link check, Python sample check, output-evals schema). A stub
verify.shround trip passed in single-project and two-project mode with thesrc/layout. Noplugin.jsonbump.Evals (2026-10-26, threading question): output, 3 touched Python cases, Sonnet 3/3 and Opus 3/3, live verify skipped (no broker). Trigger: Sonnet 64/64, Opus 64/64, Haiku 4.5 60/64 (4 feedback-skill prompts), Haiku 5.5 58/64 (6 Java design prompts). The Haiku failures sit on unchanged prompts and descriptions; both new trigger cases pass on every leg. Evals (2026-10-27,
src/layout, the four Python implement cases): Opus 4/4; Sonnet 3/4, live verify skipped. The Sonnet failure is the direct minimal-threading case, twice on the same grader: the model kept a continuous publisher for a "several times a second" prompt and handed back./verify.sh appinstead of./verify.sh direct. Its files and layout were correct. After the count-free field wording (8eceeff), the Python design-summary case passes on Sonnet and Opus. After the review fixes (997e45d), the four affected Python cases (full Quickstart, both minimal-threading cases, design summary): Opus 4/4; Sonnet 3/4 with the direct case passing this time and the full Quickstart case failing twice, once on a turn-4 trust-store grep that the kept transcript satisfies and once on the turn-1 threading-question grep, where the model asked the question with the summary's field values instead of the grader's words. After the Sonnet fixes (e6ef623): Sonnet full Quickstart 2/2, direct 2/2, design summary, suggested secure, and persisted routing 1/1 each; Opus full Quickstart, direct, design summary, and persisted routing 1/1, suggested secure 0/1 on a forbidden-verify false positive (sedandgrepon the checklist text), which the context-aware pattern clears on replay. Live verify skipped.exit_code = 1in each discard branch, so a lost message fails the stage as on the callback shape; the sample-check fixture proves both script names per stage; the hooks example and graders carrysrc/; Request-Reply (Direct) names the replier'sreceive_async()callback in itsThreadingline when the prompt asked for minimal threading.time.sleep()grep in the new cases would also fail a docs-style retry backoff.must_passyet; full output legs did not run../verify.sh <stage>inside a Bash call still matches the forbidden-verify pattern; the two false positives seen (adescriptionfield, asedof the checklist text) no longer do.