fix: stop the auditor leaking reasoning and tool tokens into public comments

[heysanil]

Since defaulting both model tiers to DeepSeek V4 Flash via OpenRouter, docs-sentinel has been posting corrupted PR comments. Two distinct bugs, both observed on heysanil/invisible-string:

Tool-call framing leak (#8, #11) — the entire comment body was </｜DSML｜parameter> fragments (｜ = U+FF5C, DeepSeek's native tool-call tokens escaping OpenRouter's Anthropic-compat shim). The auditor's real verdict was destroyed, yet the comment still rendered under "no documentation drift detected." A destroyed audit looked identical to a clean pass.

Reasoning leak (#9, #10, #12) — full chain-of-thought published, conclusion buried at the end.

Both entered at one line: jq -r '.result' treated free-form LLM prose as a trusted structured field and piped it into a public comment and a git commit body.

Two live security defects found during review

Neither was known when this work started; both are exploitable on the current default config.

• Bash(git diff:*) was a file-write primitive. --allowed-tools prefix-matching makes git diff --output=<path> reachable — verified writing 19,551 bytes. That defeats the assumption guardrail.sh documents in a comment: it enumerates only tracked modifications, so any created file went uninspected.
• The model token was written to disk where the auditor could read it. audit.yml wrote ANTHROPIC_AUTH_TOKEN into $GITHUB_ENV, a file the granted Read tool can open.

Separately bad; together a read primitive plus a write primitive in a job holding contents: write.

Approach

Reasoning and tool calls are now excluded structurally, not by pattern-matching. claude -p --output-format stream-json emits typed content blocks, so extraction keeps only {type:"text"} — thinking and tool_use cannot reach the summary by construction.

gate → build-context ─┐
                      ▼
   ┌──────────── run-auditor.sh ─────────────────┐
   │  claude --output-format stream-json         │
   │    → classify-run    (did it complete?)     │
   │    → extract-summary (typed text blocks)    │
   │    → validate-summary (shape + tree agree)  │
   │  ↑── reset tree, regenerate context, retry ─┘
   └──────────────────┬──────────────────────────┘
                      ▼
      guardrail → compose → commit
                      ▼
        decide-outcome → render-status  (always runs)

Execution vs hygiene failures are kept apart. A run that completed but produced unusable prose keeps its doc edits. A run that didn't complete has all edits discarded — the guardrail proves paths and churn are legal, not that a half-finished edit set is coherent.

One renderer owns every outcome. The "no documentation drift detected" heading is now reachable only from audit succeeded ∧ not degraded ∧ guardrail passed ∧ nothing changed. Every other terminal state renders as inconclusive with a specific reason, or as infrastructure failure.

Verified end to end against the live model

Not just unit tests — the real prompt through the real CLI:

• Drift case: repo with a 3400→3500 port change and a stale README. The auditor emitted 4 thinking blocks and 7 tool_use blocks; the pipeline returned only Subject: docs: sync… plus one bullet. Zero reasoning, zero framing, README correctly fixed.
• No-drift case: the model produced exactly the No documentation updates needed — form the validator requires, and passed.
• PR #8 payload replayed through the full pipeline: renders "Docs audit — inconclusive."

Changes

Area	Change
Security	Bash grant removed; token injected in-process; persist-credentials: false on audit-main
Engine	classify-run · extract-summary · validate-summary · run-auditor · decide-outcome · render-status
Guardrail	Rejects untracked files (the backstop the tool allowlist alone was providing)
Prompt	Fenced summary contract; no shell access
Docs	README degradation modes; corrected a stale "draft PR" claim

93 bats tests (was 39), shellcheck and actionlint clean.

Review

Ten tasks, each implemented by a fresh subagent and independently reviewed, plus a whole-branch review. Reviewers mutation-tested rather than read diffs, which caught two defects in the plan itself:

• A jq selector crashed on non-object JSON (.type on a scalar is an error, not null) — same pattern found and fixed pre-emptively in a second script.
• Bare [[ ]] assertions don't fail bats tests on bash 3.2 — only the last statement decides. Deleting core logic passed all 8 tests. All 35 assertions guarded; the suite is now load-bearing.

Known follow-ups (not blocking)

• audit-pr's checkout keeps a contents: write token in .git/config while the model runs with unrestricted Read; audit-main already sets persist-credentials: false.
• No per-attempt timeout: a hung first attempt exhausts the step budget and loses the retry, mislabelled as infra.
• A hygiene-degraded run on the default branch opens a docs-sync PR whose body reads confident; the inconclusive signal only reaches the step summary.

Follow-on work

This was split out of a larger design. docs/superpowers/specs/2026-08-08-pi-harness-and-model-benchmark-design.md covers migrating to the Pi harness and building a model benchmark — justified by benchmark fairness (today every non-Anthropic model is measured through a shim Anthropic models bypass) rather than by this incident.

Model defaults changed in this PR

Also swaps the default model routing, which is the other half of the response to the DeepSeek incident:

Input	Was	Now
model	~deepseek/deepseek-v4-flash-latest	z-ai/glm-5.2
small-model	~deepseek/deepseek-v4-flash-latest	deepseek/deepseek-v4-flash-0731
effort	high	xhigh
model-capabilities	effort,thinking,…	effort,xhigh_effort,thinking,…

Two things worth noting. Both slugs are now pinned to concrete versions rather than ~…-latest floating aliases, so a given workflow SHA always audits with the same models. And xhigh_effort had to be added to model-capabilities — Claude Code gates reasoning by pattern-matching the model ID, so without that capability declared it clamps xhigh back to high and the new effort default would have been a silent no-op.