Repository navigation
fix(brain): keep other-host prompts within the action schema - #684
Conversation
vladimirrott
left a comment
vladimirrott
left a comment
There was a problem hiding this comment.
Thank you, @QinXi-ai. The test that reads every action name out of the rendered prompt and checks it against that host's propose_plan enum is the right shape for #672, and it covers the next drift too: if anyone adds a sentence naming an action the schema withholds on some family, it goes red. Reviewed at 3fb7dc5.
Nothing here blocks on your side.
What I checked, with your code run in a container (rootless podman, no network):
- The other families are untouched. I rendered the prompt on
mainand on your head for no hint, Fedora 42, Ubuntu 24.04, Debian 13 and Arch, each with and without a saved preference. Eight of the ten are byte-identical. Only the two Arch renders differ (29,587 → 27,178 bytes). So every existing E2E story sees the same prompt it passed with, and the live-story checkCLAUDE.mdasks for on prompt changes has nothing to re-measure. - The Arch diff reads right.
GetSystemStateis gone from the risk table, the no-params list, the history finish and the preference finish. The old text also told an Arch host "Distro family is unknown" and describedGetSystemStateas returning rpm-ostree deployment data; both are gone.refuseis a real planning tool (planning_tools/refuse.rs), so the overview example points at something the model can call. - Your tests bite. I ran your three tests against four mutations, restoring the file between each:
M1 delete `Some(DISTRO_FAMILY_OTHER) => render_other_prompt(...)` (the fix reverted)
every_prompt_action_is_offered_by_its_plan_schema ... FAILED
Some(DistroHint { id: "arch", family: "other", version: None }): unoffered actions ["GetSystemState"]
other_prompt_preserves_examples_and_has_an_available_preference_finish ... FAILED
M2 other-host {FINISH_ACTION} -> "GetSystemState" 2 failed
M3 other-host {STATE_ACTION_ENTRY} -> "GetSystemState, " every_prompt_action_is_offered_by_its_plan_schema FAILED
M4 drop the other-host {SERVICE_DISAMBIGUATION} substitution
no_rendered_prompt_leaks_an_unsubstituted_placeholder ... FAILED
the other prompt shipped {SERVICE_DISAMBIGUATION}
The unmutated run and a run after the last restore both printed running 3 tests and 3 passed.
The red rust check is the test baseline and nothing else: test_baseline: rust suite has 2000 tests, baseline says 1998. All 2,000 tests passed in that job. CONTRIBUTING.md puts that figure on whoever merges, so it's mine, along with bringing the branch up to date. main is two commits ahead and neither touches sysknife-brain.
One optional thought. The Arch copy of Example F is a second text of that example, and an edit to the shared one won't reach it. It also drops the line "call get_system_state (planning tool) first ← FORBIDDEN" that the shared copy carries. I'm happy to take it as it is; if you'd rather keep that line, it's a one-line change.
I'll take it from here. Every open issue in your area has a pending offer right now; when one frees up I'll point you at it.
…fix/672-other-host-prompt
Main reached 2002 with lacs-project#626; this branch adds two tests (every_prompt_action_is_offered_by_its_plan_schema and other_prompt_preserves_examples_and_has_an_available_preference_finish), so the artifact and the three published figures move together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU
vladimirrott
left a comment
vladimirrott
left a comment
There was a problem hiding this comment.
Thanks, this is a careful fix. Routing other hosts to their own renderer is the right call, and so is telling the model to refuse an OS/hardware overview instead of inventing a snapshot. The test is better than the fix needs: every_prompt_action_is_offered_by_its_plan_schema checks every action name the prompt mentions against the propose_plan enum for seven hints. Any future prompt edit that names an action the schema won't accept now fails, on any family.
I proved it at 3fb7dc51 in a no-network container. The test passes as written. With the Some(DISTRO_FAMILY_OTHER) => render_other_prompt(...) arm deleted, so Arch falls back to the generic prompt, the same test FAILED. I also checked that refuse and ListJobHistory are both real tools here, and that {STATE_ACTION} can't be a substring of {STATE_ACTION_ENTRY}, so the order of the replace chain is safe.
The only red check was rust, and only on test_baseline (2000 against 1998). #626 merged in the meantime and moved main to 2002. Since you allowed maintainer edits, I pushed two commits to your branch: a merge of current main, and 4f6616c, which moves the artifact and the three published figures to 2004. Nothing else changed, and the evidence check passes on it. I'll merge once CI is green on that head, then write the CHANGELOG line at release.
Main moved to 2,004 Rust tests (lacs-project#626, lacs-project#684); this branch adds seven, so the artifact and the three published figures read 2,011. No other conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU
Minor release. #627 keys request_hash with an HMAC under the audit key (chain_version 4; previews stored by an older release fail closed after upgrading) and #639 changes the history request a newer CLI sends, so a caller relying on the old behaviour notices: the middle digit moves. Also: the planner sees only its host's distribution (#626, #656, #658, #684), checkpoint and the MCP audit tools share the audit-store resolver (#636), every workflow job has a time limit (#662), a weekly cargo audit (#640), release hardening (#635, #637, #660), and every published figure is rendered from its source (#685). Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Fixes #672. On detected
otherhosts, the prompt no longer recommends a state action excluded bypropose_plan. Neither existing OS snapshot action is offered there, so the renderer explicitly directs unsupported overview requests torefuse, keeps the six worked examples, and uses the available read-onlyListJobHistorystep to finish history and preference-only intents.Fedora, Ubuntu, Debian and no-hint rendering retain their existing text. Shared template expansion was compared byte for byte with the base. Action schemas and daemon/client eligibility fences are unchanged.
Related Issue
Closes #672.
Validation
unoffered actions ["GetSystemState"]on Arch.docs/action-compatibility.md.cargo nextest run --locked --offline -p sysknife-brain: 364 passed, 0 skipped.cargo clippy --locked --offline -p sysknife-brain --all-targets --all-features -- -D warnings, formatting and whitespace checks passed.3fb7dc5, all checks other than the Rust job pass (container-smokeis intentionally skipped). The Rust job is still installing Tauri system dependencies, before compilation or tests.Notes for Reviewers
Two Rust tests were added. Full local workspace measurement is blocked by missing GLib/GIO/GObject development libraries; the recorded test baseline and published counts are left unchanged per CONTRIBUTING's fallback.
scripts/ci-local.sh --no-postgreswas run and was incomplete: workspace Rust checks hit those libraries, frontend checks hit the local Node/tooling setup, and several required lint/security helpers are absent. The final local-run summary is included below; it remains incomplete after correcting the WSL Git worktree paths.Live model/VM story validation has not been performed. The deterministic checks establish prompt/schema consistency, not live model behavior. This contribution is AI-assisted.
Local CI summary