Repository navigation
test(verify): add a project verify skill and document it in AGENTS.md - #273
Conversation
Agents could only prove a change with the all-or-nothing verify:local run. The skill keeps a disposable Postgres/Redis/API stack up for iteration, maps each kind of change to its required gates, and records redacted evidence. journeys.sh asserts the feature map's client journeys against it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…erify skill scripts
Name the shapes those comments explained: exec_in_stack_env runs in a
subshell so the server's PID is $!, SERVER_ENV is split into an array,
register_for_teardown records the stack before anything can fail, and
isToolFailure detects {"ok":false} MCP replies.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 13 files
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
…and the verify skill Both checked that an integration run executed tests with none failed or skipped, in two copies. assert-integration-report.mjs is now the single check, with node tests for empty, failed and skipped reports. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
@coderabbitai review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77fc44ddfe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Redact sensitive keys inside JSON-looking strings and sensitive headers, cap every recorded body at 4 KiB, keep SERVER_ENV values out of logs and doctor output, write state.env shell-safely and owner-only, record the server PID before readiness polling, keep doctor running when the server is down, restore the default server env when a journey fails, accept only suite entry points in `suite`, and correct the auth, exec, sandbox and prerequisite docs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summary
Until now an agent could only prove a change with
pnpm verify:local, which runs everything in one go. This PR adds a project skill at.claude/skills/verify/so agents can work and verify on their own:SKILL.mdlists, for each source area, the checks to run while iterating and the gate that counts as completion evidence.scripts/stack.sh. It keeps a disposable Postgres/Redis/API stack running between calls. It reusesinfra/docker-compose.verify.ymlwith a unique Compose project, random loopback ports, a cleanenv -i, and migrations on startup. Subcommands:up,restart,down: start, rebuild, and stop the stack.downstops only the PID it started and keeps the evidence.doctor: checks service health, PID and port ownership, readiness, and token auth. It fails whendist/is older thansrc/or whennode_modulesdoesn't matchpnpm-lock.yaml.api,mcp,psql: send an authenticated request, call one MCP tool, or inspect the database.suite <name>: runs anyscripts/verify/*.mjssuite, orintegration [filter], against the stack.features/*.mdcovers auth, sandboxes, exec, files, ingest, and MCP.scripts/journeys.shruns those journeys with 41 assertions in about 8 seconds.scripts/verify/.run/stack-<id>/transcript.jsonlmasks the stack secret, JWTs, and token, secret, password, and authorization keys.scripts/verifysuites. Agents without Claude Code can run the scripts directly.No product code changes; nothing in
src/is touched. The one CI-gate edit:scripts/verify-local.shnow callsscripts/verify/assert-integration-report.mjsinstead of an inline heredoc. It runs the same three assertions with the same messages, andstack.sh suite integrationcalls the same file. Node tests cover empty, failed, and skipped reports.Test plan
Run in this worktree on top of
e9cf495:pnpm checkpasses: 2334 unit tests and 154 TypeScript SDK tests.up, thendoctor(all ok), all 6 journeys (41/41), thendown. No containers, volumes, listeners, or server process remain, and the evidence directory survives.scripts/verifysuite passes throughstack.sh suite: contracts, output-cap, shutdown, sdk-clients, worker-failures, sse-finalization, input-budgets, tenants, ingest-admission, and integration (265 tests, 0 skipped).large-fileswas not run because it needs a cgroup v2 host.src/api/routes/exec.tsmadedoctorreport the stale build andjourneys.sh execfail with the expected and actual values. Reverting the change made it pass again.up, anddownstill cleans up;doctor;X-Auth-Secretand the returned token are recorded as[REDACTED]. A scan found no JWTs or secrets in the transcript.pnpm verify:architecturepassed while a stack was up, so the two coexist.pnpm check:scripts(including the new tests),pnpm lint, andpnpm verify:localall pass (265 integration tests, 0 skipped). On the stack,suite integrationpasses on a real report and fails on the same report with one skipped test.Follow-ups (not in this PR)
bash_execreports denial as{"exitCode":1,"stderr":"forbidden"}, and file and sandbox tools report it as{"ok":false}. Both useisError:false. SettingisError:true(insrc/api/mcp/tool-errors.ts) would match the MCP spec, but it is a contract change.stack.shrepeats about 15 lines of compose and env setup fromverify-local.sh.🤖 Generated with Claude Code
Summary by cubic
Adds a
verifyproject skill so agents can prove changes against a live, disposable Postgres/Redis/API stack instead of only running the fullpnpm verify:localgate.New Features
scripts/stack.shkeeps a disposable Postgres/Redis/API stack running between calls, withup,restart,doctor,api,mcp,psql,suite, anddownsubcommands.SKILL.mdmaps each source area to the checks to run while iterating and the gate that counts as completion evidence.features/*.mdandscripts/journeys.shdrive client-visible journeys for auth, sandboxes, exec, files, ingest, and MCP with about 40 assertions in roughly 10 seconds.transcript.jsonl; bodies are capped at 4 KiB and the stack secret, JWTs, and sensitive keys and header values (including inside JSON) are redacted.AGENTS.mddocuments the workflow; no product code changes, and existing gates likeverify:localcan run alongside the stack.verify-local.sh, is now shared asscripts/verify/assert-integration-report.mjswith node tests.Written for commit 2b4408d. Summary will update on new commits.