UVH-IN7: an observed tool error reaches response, even unauthored

[chelojimenez]

UVH-IN7 — inspector lane, step 3. Analyzer 6 → 7. Stacked on #4490 (UVH-IN1) → #4488 (UVH-IN5).

This is the fix for the O0 investigation: the 8 prod trials that were fully green in the chain while their legacy verdict said failed.

What those trials actually were

A case can author only transcript predicates — nothing about tools at all — and still have a tool fail on the server during the run. response was notApplicable for such a case, because applicability was decided purely from what the case authored. So when failOnToolError failed the trial on exactly that tool error, the chain had every applicable stage green and no row able to say why the verdict was red.

It was never a verdict bug. It was a chain with no vocabulary for what happened — which is why the right fix is an analyzer change rather than copy that papers over the disagreement.

The change

An observed errored tool call now makes response measurable on its own, filing failed / toolError under serverData like any other server-answered error.

Applicability and the deriver share one predicate (hasObservedToolFailure) rather than two copies of the condition, so a stage cannot be switched on by one rule and then found empty by the other.

Two boundaries are deliberate:

• A span carrying an mcpErrorCode does not turn the stage on. It never reached the server's handler, so it stays a setup fact — attributing our own transport failure to the server is precisely the mis-attribution this module exists to prevent.
• A case with no tool failure is unchanged: response stays notApplicable, because there is still nothing for it to decide.

Why this doesn't break the "computed before evidence" rule

The applicability table's whole point is that an inapplicable stage can never be reported as an evidence gap. This entry reads a single positive observation, not an absence — and a stage turned on by observed evidence cannot then be reported as a gap, because deriveResponse holds the very span that turned it on. That reasoning is written at the call site so the next reader doesn't have to reconstruct it.

Verification

• 5 new tests: the recovered-tool-error case files response: failed / toolError with category serverData and firstFailedStage: response; the chain no longer goes all-green while the verdict fails (the O0 shape, asserted directly); a transport-local error does not turn the stage on; no errored span leaves an unauthored response inapplicable; an already-authored case is unaffected.
• Mutation-checked: removing the new applicability clause fails exactly the two O0 tests and nothing else.
• sdk: 6886 passed (313 files). Inspector server: 700 passed. CLI: 1182 passed — gate exit codes unchanged.
• runner-parity snapshots: 84 diff lines, all the version stamp, verified by filtering.
• Parity corpus and the stage-analytics golden re-stamped to 7; every recorded row still reproduces byte-for-byte.

STAGE_REASONS does not move (toolError already existed), so the backend mirror needs no re-pin for this bump.

Note for the reader

Post-IN7, the failOnToolError: false variant now reports response: failed / toolError while the trial verdict passes. That is intended and is the honest reading: the chain reports what the server did, and the trial's own policy decides whether that fails the run. The two are allowed to differ — what was wrong before was the chain having nothing to say.

---

Generated by Claude Code

---

Note

Medium Risk
Changes core eval stage applicability and failure attribution for transcript-only cases; behavior shifts for trials where tool errors were previously invisible in the stage chain.

Overview
Fixes eval stage derivation so predicate-only cases can still surface server tool failures on the response stage instead of leaving it notApplicable while the legacy verdict fails on failOnToolError.

hasObservedToolFailure is introduced and used for both response applicability and deriveResponse, so an observed content/domain tool error files failed / toolError (serverData) without applicability and derivation drifting apart. Errored spans with an mcpErrorCode (transport/setup) still do not turn response on; runs with no tool failure behave as before.

STAGE_ANALYZER_VERSION bumps 6 → 7; STAGE_REASONS is unchanged. Tests, parity corpus, and runner snapshots are re-stamped for the new version.

^{Reviewed by Cursor Bugbot for commit dbb9fb3. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Summary by cubic

Fixes the stage analyzer so an observed tool error makes response measurable even when the case authored nothing about tools. Previously response was notApplicable for such cases, so a server tool error that failed the trial left every applicable stage green with no row saying why the verdict was red. Now the chain files failed / toolError under serverData instead.

Note

• Applicability and the deriver now share a single hasObservedToolFailure predicate.
• A span carrying an mcpErrorCode stays a setup fact and does not turn the stage on.
• A case with no tool failure keeps response as notApplicable.
• Analyzer bumps to 7; STAGE_REASONS is unchanged, so the backend mirror needs no re-pin.

^{Written for commit dbb9fb3. Summary will update on new commits.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_1be93e91-c1dd-4b66-bd30-cb8277c2b848)

[chelojimenez]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

Internal preview

Preview URL: https://mcp-inspector-pr-4491.up.railway.app
Deployed commit: 3ca69a8
PR head commit: dbb9fb3
Backend target: staging fallback.
Health: ✅ Convex reachable
Access is employee-only in non-production environments.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_de307d95-4a1f-4d82-9a60-a3e6f23dd3ca)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_030d07d2-4f4b-4296-8f77-f692359ad86f)