UVH-IN1: file a failed tool-call assertion at selection, not user value

[chelojimenez]

UVH-IN1 — inspector lane, step 2. Analyzer 5 → 6. Stacked on #4488 (UVH-IN5); GitHub retargets to main once that merges.

The mis-routing

stepsToPromptTurns promotes only toolCalledWith into expectedToolCalls, where the selection matcher grades it. The other three tool assertions — toolCalledAtLeastOnce, toolNeverCalled, firstToolWas — fall through to per-turn checks and reach the analyzer as predicate results, which deriveUserValue graded as user value.

So a case asserting "tool get_project was never called" reported userValue: failed / predicateFailed with selection: passed. The chain said the user did not get what they asked for; what actually happened is that the model picked the wrong tool.

Across one audited prod window that single mis-routing is most of the gap between 2 selection failures and 12 user-value ones — the report card blaming the wrong stage on exactly the failures operators most need attributed.

The matrix — the three kinds do not share applicability

predicate	failure files as	expects a call?
toolCalledAtLeastOnce	missingToolCall	yes
firstToolWas	unexpectedToolCall	yes
toolNeverCalled	unexpectedToolCall	no

toolNeverCalled is the asymmetry that makes this a matrix rather than a set: a case whose only tool assertion forbids a call expects none, and turning call on for it would demand evidence of the very thing the case exists to rule out.

Passing rows route too, not only failing ones. A case whose only selection assertion is "never call the admin tool", which did not call it, has measured selection and found it sound — notMeasured would understate what the run established.

Routed, not copied. A failure filed at both stages would double-count one defect and make firstFailedStage depend on which stage a reader looked at first. For the same reason buildStageAuthoredCase stops counting these in assertionCount, so a toolNeverCalled-only case no longer reports a permanent user-value gap that no author could ever close.

toolCalledWith is deliberately untouched: it is already matcher-graded, and re-reading its point-in-time predicate row here would let a raw residual contradict the adjudicated verdict.

Backward compatibility, and why the bump is cheap

The predicate discriminator was always present at runtime — PredicateResult carries the whole predicate — and a cast in finalize-iteration.ts was erasing it. It now crosses with the row, and a row without one grades exactly as before.

That is why the analyzer bump to 6 changed no recorded row in the historical-parity corpus: all 19 records reproduce byte-for-byte, and only their version stamp moved. STAGE_REASONS does not move either (the routing re-uses missingToolCall / unexpectedToolCall), so the backend mirror needs no re-pin.

Verdict / gate / count neutrality

Nothing in iteration-verdict.ts, eval-gate*.ts, eval-run-exit-code.ts, sdk/gates.ts or the tallies reads stage rows. Confirmed empirically:

• CLI suite 1182 tests passed — gate exit codes unchanged.
• runner-parity snapshots: 42 changed lines, all of them the version stamp, verified by filtering the diff.

Verification

• sdk: 6881 passed (313 files). New per-kind matrix tests cover each kind's reason, that passing rows are selection evidence, that a routed failure leaves userValue, precedence when a missing matcher call and a predicate both fail, the mixed case where every turn passed but a predicate failed, and that a row with no discriminator still grades as user value.
• stage-inputs: per-kind expectsToolCall / assertionCount coverage, including that a widget assertion (keyed by kind, not type) is never mistaken for a predicate kind.
• inspector server 700 passed; typecheck clean across sdk/cli/client.
• Two fixture repairs worth calling out: the decision-summary corpus hardcoded 6 as "a version newer than ours", which stopped exercising version-ahead the moment the analyzer reached 6 — it now uses a far-future constant, and the test asserts known by meaning (STAGE_ANALYZER_VERSION) rather than duplicating the fixture comparison with two magic numbers.

Intended consequence

This widens the D7 metadata-attribution judge's candidate population, which gates on firstFailedStage === "selection". That is intended — the reason more selection failures now reach it is that they were always selection failures. The judge's per-run job has no candidate cap, so a volume metric before widening evaluate-enabled stays on the operator residual list.

---

Generated by Claude Code

---

Note

Medium Risk
Changes eval stage attribution and firstFailedStage/failureCategory for tool predicates, which affects dashboards and the D7 judge, though iteration pass/fail and gates are unchanged.

Overview
UVH-IN1 fixes mis-attribution: toolCalledAtLeastOnce, firstToolWas, and toolNeverCalled no longer fail at userValue (predicateFailed); they are graded at selection with missingToolCall or unexpectedToolCall. toolCalledWith stays on the matcher path.

The stage analyzer bumps to v6. Predicate rows now keep their predicate.type through finalize-iteration (a cast had been dropping it). deriveSelection consumes failed (and passing) selection predicates; deriveUserValue excludes them so defects are not double-counted. buildStageAuthoredCase and the SDK mapping mirror a per-kind matrix: positive kinds set expectsToolCall; toolNeverCalled does not; selection kinds are omitted from assertionCount.

Pass/fail verdicts and gates are unchanged. Rows without a predicate discriminator behave as before. More trials surface firstFailedStage === "selection" for the D7 metadata judge by design.

^{Reviewed by Cursor Bugbot for commit 7983531. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Summary by cubic

Fixes UVH-IN1: tool-call assertions other than toolCalledWith now file their failures at the selection stage instead of userValue, so "the model picked the wrong tool" is no longer misreported as "the user didn't get what they asked for."

Selection routing

• toolCalledAtLeastOnce fails as missingToolCall; firstToolWas and toolNeverCalled fail as unexpectedToolCall.
• toolNeverCalled expects no call, so unlike the other two it doesn't make selection applicable.
• Passing predicate rows are selection evidence too, filing passed/observed rather than notMeasured.
• Failures are routed, not copied, so these kinds are excluded from assertion counts and a toolNeverCalled-only case reports no permanent user-value gap.
• toolCalledWith stays matcher-graded and is deliberately untouched.

Compatibility

• Rows without a predicate discriminator grade exactly as before; the analyzer bump to 6 changed no recorded rows, only version stamps.
• Verdicts, gate exit codes, and pass/fail counts are unchanged.
• The D7 metadata-attribution judge sees more selection candidates because it gates on firstFailedStage === "selection" — intended, since those were always selection failures.

^{Written for commit 7983531. Summary will update on new commits.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_bfca785d-af31-4b1d-a645-5dcfafcec77a)

[chelojimenez]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

Internal preview

Preview URL: https://mcp-inspector-pr-4490.up.railway.app
Deployed commit: da44b9d
PR head commit: 7983531
Backend target: staging fallback.
Health: ✅ Convex reachable
Access is employee-only in non-production environments.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c1c8e063-6702-4a5c-a921-7a4cc605a9dc)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_4b7d670e-e83b-4f59-82e8-f57b5caa178b)