UVH-IN2: attribute a model-provider failure to us, not to the server - #4492
UVH-IN2: attribute a model-provider failure to us, not to the server#4492chelojimenez wants to merge 5 commits into
Conversation
20 trials in one audited prod window failed on "credit balance too low…
Anthropic API". The step errored, the asserts were skipped, and the chain
ended `userValue: notMeasured / noEvidenceCaptured` with NO failure
category at all — our provider's outage presented as an unattributed
server failure, in a report card whose entire job is to say whose side
broke.
The layer that failed is now tagged at the catch site and carried into the
chain. `providerError` marks the stages a model-call failure left blank,
and `categoryFor` files the run under `setup` — the existing bucket for our
own side breaking, so no new category was needed.
CLASSIFIED STRUCTURALLY, never by reading the message.
`drive-hosted-eval-turn.ts` already knows which layer it is in: `failTurn`
is the engine's stream, `mapThrownTurnError` names its own call site, and
"pre-turn setup" is the one that never reached the model. A text classifier
would be one provider's wording away from mis-attributing a whole class of
run. The engine's `code`/`httpStatus` ride along as diagnostics and are
deliberately NOT part of the decision.
Three boundaries are deliberate:
- Only BLANK rows are re-labelled. A provider dying at turn 4 does not
un-observe turns 1-3, so a stage with its own evidence keeps its row.
- Never `failed`. A run that could not be attempted has measured nothing
about the server, and inflating a server failure rate with our own
outage is the mis-attribution this reason exists to prevent.
- A broken grader still outranks it; `evaluator` is never folded in.
`providerError` is broader than its name — outage, exhausted credits, rate
limit, our own spend guardrails — and what they share is that OUR side of
the call broke. That is said in the reason's own docblock rather than left
for a reader to infer.
Analyzer 7 -> 8. This bump moves `STAGE_REASONS`; the backend mirror
already carries the member (UVH-BE1 shipped it deliberately ahead), so
nothing quarantines during the deploy window.
Stated limitation, unchanged: the legacy verdict still counts these trials
failed. Verdict population is a customer gate change and stays deferred.
SDK 6891 tests, inspector server 700, CLI 1182 — gate exit codes unchanged.
Parity corpus and snapshots differ only in the version stamp, verified by
filtering the diff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
GitHub Actions
Inactive
dosubot
Bot
added
size:XL
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_999d5047-b1dc-436a-ab31-d32d518eccc5) |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eaae5e4ed5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // UVH-IN2: the layer that raised the fatal error, so the chain can say a | ||
| // provider outage was ours rather than filing it against the server. | ||
| ...(iterationStepError ? { stepError: iterationStepError } : {}), |
There was a problem hiding this comment.
When orgByokRuntime.kind === "local", execution is dispatched to runLocalIteration rather than this hosted path (evals-runner.ts:2500-2545). A provider failure delivered through runDirectChatTurn's onError leaves no response messages, so drive-local-eval-turn.ts:391-418 records an iterationError, but the local finish call at evals-runner.ts:3848-3937 never supplies stepError. Consequently these completed local-BYOK trials still derive blank stage reasons and no failure category, preserving the exact misattribution this change is intended to eliminate; carry the model error source through the local accumulator and both local finalization paths as well.
Useful? React with 👍 / 👎.
| return { | ||
| kind: "failed" as const, | ||
| ...failure, | ||
| errorSource: "model" as const, | ||
| ...(lastEngineError?.code ? { errorCode: lastEngineError.code } : {}), |
There was a problem hiding this comment.
For harness-backed evals, runHarnessTurn reports every failure through onEngineError, including pre-model setup failures such as missing projectId, disabled broker delivery, or sandbox reservation failure (run-harness-turn.ts:869-965 and 1360-1369). Those events reach this generic failure branch and are unconditionally tagged as model, so the persisted stages falsely say the model provider failed even though no provider call occurred. Preserve a structured source from the harness or classify its setup checks separately instead of treating every engine error as a provider error.
Useful? React with 👍 / 👎.
Internal previewPreview URL: https://mcp-inspector-pr-4492.up.railway.app |
…vc-mcp-eval-reporting-gyycwl-in2
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_d6381f0e-a180-41a6-ba0a-585c8bf0f9bb) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00c7af91c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| errorSource: "model" as const, | ||
| ...(lastEngineError?.code ? { errorCode: lastEngineError.code } : {}), | ||
| ...(typeof lastEngineError?.httpStatus === "number" | ||
| ? { errorHttpStatus: lastEngineError.httpStatus } |
There was a problem hiding this comment.
For every hosted or cloud-model failure returned here, buildHostedStepHandlers converts the outcome into a StepEngineOutcome at step-handlers.ts:253-265 and 302-315 but copies only iterationError and iterationErrorDetails. Consequently executeSteps never receives errorSource, iterationStepError remains undefined, and the new providerError attribution is not produced for any normal hosted provider failure. Forward the new source/code/status fields through both hosted handler paths.
Useful? React with 👍 / 👎.
| predicateResults, | ||
| widgetRenderObservations, | ||
| stageToolErrors, | ||
| ...(args.stepError ? { stepError: args.stepError } : {}), |
There was a problem hiding this comment.
When a provider-failed iteration later receives a judge verdict, this transient stepError is unavailable to the second pass: it is used to build the initial stage rows but is not stored in iteration metadata or returned by the judge-stage backend, while judge-second-pass.ts:334-349 calls buildStageMetadata without it and then overwrites the derivation-owned stage keys wholesale. Such iterations therefore lose providerError and the setup category as soon as the judge pass runs; persist and reload the classified source for that re-derivation.
Useful? React with 👍 / 👎.
| const outcome = await handlers.onFollowUp!({ | ||
| text, | ||
| stepIndex, | ||
| turnOrdinal: turn, | ||
| }); |
There was a problem hiding this comment.
When a widget ui/message follow-up hits a provider failure, the handler outcome includes the new attribution fields, but drainAndDriveFollowUps reduces it to only outcome.iterationError and runFollowUps constructs a result with no errorSource, code, details, or status. Thus this hosted failure path remains unattributed even after the main hosted handler bridge is fixed; return or propagate the complete StepEngineOutcome from the follow-up loop.
Useful? React with 👍 / 👎.
Both found by Codex review, both verified against the code, and both are this
same defect wearing a different coat — attributing to the model a failure that
never reached it, or attributing nothing at all where it should.
HARNESS SETUP FAILURES WERE CALLED PROVIDER FAILURES. `failTurn` said "every
path through here is the engine's stream failing". True of the chat engine,
false of the harness: `runHarnessTurn` wraps its whole turn — preparation
included — in one try (the throws for a missing projectId, a missing auth
bearer and disabled broker credential delivery are all inside it), and reports
every one through the same `onEngineError` a provider outage uses. So our own
setup bug was filed as the provider's.
The engine now reports the PHASE it failed in, derived from the trace-started
flag the emitter already holds to decide whether a turn happened at all —
never from reading the message. An engine that reports no phase still means
`model`: every emitter that omits it today is a real stream failure, and
defaulting the other way would un-attribute the outages this work exists for.
THE LOCAL PATH CARRIED NO SOURCE AT ALL. With orgByokRuntime.kind === "local",
execution goes to runLocalIteration, and neither of its finish calls supplied a
stepError — so local-BYOK trials that died on the model call still finalized
with blank stage reasons and no failure category. The original fix never
touched that path.
The local driver now records the layer at both error sites and threads it
through both finish paths:
- an empty model stream is unambiguously the model call;
- a non-tool error span is the model call ONLY when its category is `llm`.
That branch selects every non-tool error span, and connection, discovery
and oauth spans all reach it — tagging those `model` would blame the
provider for a server we could not reach.
Anything else leaves the source unset. An absent source attributes nothing,
which is the right floor: no attribution beats a confident wrong one.
Both decisions are extracted as pure functions so the choice can be read and
tested on its own rather than inferred from a call site's position.
Mutation-checked: ignoring the phase, tagging every span, and tagging none each
fail exactly their intended tests. Server evals + harness suites: 1079 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_8b0ff13e-1118-470e-89ee-4b26ef13b35a) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ec733dd0a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| r.state === "notMeasured" && BLANK.includes(r.reason) | ||
| ? { ...r, reason: "providerError" as const } |
There was a problem hiding this comment.
When a model failure occurs on a turn that expects a tool call, the runner still builds a prompt summary with that call in missing, so deriveSelection has already produced failed/missingToolCall before this function runs. Because this condition only relabels notMeasured rows, categoryFor retains selection rather than setup, and the provider outage is still reported as a model-selection defect. Preserve evidence from completed earlier turns, but suppress matcher/predicate failures attributable to the specific aborted turn before deriving the stages.
Useful? React with 👍 / 👎.
| // delivery all land here, and none of them ever reached a model. The | ||
| // trace-started flag used just above to decide whether a turn happened | ||
| // at all is the same signal for whose layer failed. | ||
| phase: driver?.traceStarted ? "stream" : "setup", |
There was a problem hiding this comment.
The new phase fix uses driver.traceStarted, but driver is not created until after await agent.stream(...) or await agent.continueStream(...) returns. If that call itself rejects—for example on an immediate provider authentication, quota, or rate-limit response—the catch reports phase: "setup" even though the model request was attempted, so the hosted path will omit providerError. Track entry into the model invocation separately from creation of the trace driver; this is fresh evidence beyond the earlier setup-attribution comment because the added phase logic now misclassifies the inverse failure direction.
Useful? React with 👍 / 👎.
|
Both Codex findings were right. Verified against the code and fixed in P1 — the local BYOK path carried no source at all. Confirmed: The local driver now records the layer at both of its error sites and threads it through both finish paths. An empty model stream is unambiguously the model call. A non-tool error span is the model call only when its category is P2 — harness setup errors were tagged The engine now reports the phase it failed in, derived from the trace-started flag the emitter already holds to decide whether a turn happened at all — never from reading the message, consistent with the rest of this PR. An engine reporting no phase still means Both decisions are extracted as pure functions ( Generated by Claude Code |
GitHub Actions
Error
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_96011c4e-b8c9-4a2b-8c54-2e0dbaaf94d8) |
…able
Two further findings. The first is the one that mattered: the fix was INERT on
the shape it exists for.
A MISSING TOOL CALL IS NOT A SELECTION DEFECT WHEN THE PROVIDER NEVER LET US
MAKE IT. applyProviderError re-labelled only rows that measured nothing — but a
case expecting a tool call whose provider died already carries
`selection: failed / missingToolCall`, written by the matcher before the chain
is derived. So firstFailedStage stayed `selection`, categoryFor returned
`selection`, and the outage was filed as a model-selection defect: the exact
misattribution this reason was built to remove, on the commonest case in the
corpus.
The line the fix turns on:
- An ABSENCE verdict (no call arrived, an assertion over output did not hold,
the judge scored a truncated transcript low) is only sound if the run was
allowed to finish. When our own model call died first, "it did not happen"
has a second explanation that outranks the accusation and we cannot tell
which is true. Those rows become notMeasured/providerError, and their
evidence goes with the verdict it supported — a notMeasured row must not
still argue for a failure it no longer claims.
- A PRESENCE verdict stands: an unexpected call was really made, arguments
really mismatched, a tool really errored, a render really failed.
connectFailed and toolsListFailed matter most — they happen BEFORE any
model call, so a server that would not connect is never excused by a
provider error that came later. Laundering a genuine server defect would
be the worse bug of the two.
THE PHASE FLAG WAS READ TOO LATE. It used driver.traceStarted, and the driver
is only built after agent.stream(...) RESOLVES — so an immediate provider
rejection (auth, quota, a rate limit) read as our setup failing when the model
had in fact been asked. The flag is now set immediately before the call, so it
marks the handover rather than a successful one.
Mutation-checked in both directions: never withdrawing a failed row reproduces
the original bug, withdrawing every failed row launders the server defects, and
keeping stale evidence leaves a withdrawn row arguing for its old verdict.
SDK 6894 passed; CLI gate suite 1182 passed, 0 failed; harness + evals 1079.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
69e7cdc to
f9817f1
Compare
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_b73a904e-53d0-4fe4-9ec2-b38869ce9b7a) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9817f1b3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| r.state === "failed" && | ||
| PROVIDER_UNKNOWABLE_FAILURES.includes(r.reason) | ||
| ) { |
There was a problem hiding this comment.
When a provider error accompanies missingToolCall, the positional pass has already changed call, response, and userValue to notReached/earlierStageFailed before this branch converts selection from failed to notMeasured. Fresh evidence in this revision is that the conversion now happens after that positional rewrite, so the result has no failed row or firstFailedStage while three rows still claim an earlier stage failed. Recompute positional states after withdrawing the failure, or relabel those dependent rows, so the chain remains internally consistent.
Useful? React with 👍 / 👎.
| providerError: | ||
| "the model provider failed the call, so the run never reached the server", |
There was a problem hiding this comment.
In a multi-turn iteration where earlier turns successfully called the MCP server and a later model call fails, providerError can coexist with passed call/response rows, yet every renderer expands it to the claim that the run “never reached the server.” This contradicts the preserved earlier evidence and can send operators investigating the wrong timeline; phrase the label as the provider preventing this stage or turn from completing rather than claiming the server was never reached.
Useful? React with 👍 / 👎.
|
Both right again, and the P1 is the most important finding on this PR — the fix was inert on the shape it exists for. Fixed in A missing tool call was still a selection defect. Confirmed by reading The line the fix turns on:
An absence verdict is only sound if the run was allowed to finish; when our own model call died first, "it did not happen" has a second explanation that outranks the accusation and we cannot tell which is true. A presence verdict was actually observed and stands whatever killed the turn afterwards. The withdrawn row's evidence goes with the verdict it supported: a The phase flag was read too late. Also right — Mutation-checked in both directions: never withdrawing a failed row reproduces the original bug; withdrawing every failed row breaks the two laundering tests; keeping stale evidence leaves the withdrawn row arguing for its old verdict. SDK 6,894 passed; CLI gate suite 1,182 passed, 0 failed; harness + evals 1,079 passed. (Also: the previous push had picked up ~60 lines of unrelated prettier trailing-comma churn in Generated by Claude Code |
… label Two follow-ups on the withdrawal added last round. Both are defects I introduced with that fix. THE CASCADE RAN BEFORE THE WITHDRAWAL. The positional pass reads `failed` rows to decide which later stages "never ran". Withdrawing a provider-blocked failure AFTER it had run left call, response and userValue still saying `earlierStageFailed` while no stage failed and no firstFailedStage existed — three rows citing a failure the chain no longer records. A chain that argues with itself is worse than either reading alone. The withdrawal now happens first, so the cascade sees the rows as they will actually be reported: a provider outage marks the later stages `providerError`, which is WHY they were not measured, rather than blaming a stage that is no longer failed. A failure the provider did NOT explain still cascades exactly as before — `unexpectedToolCall` survives the withdrawal, stays the first failed row, and the stages after it still read `notReached`. Repaired, not disabled. `applyProviderError` stays in `finalize` as well, because the early-return paths never reach the cascade; it is idempotent, so the second pass over already-converted rows finds nothing to change. THE REASON'S LABEL SPOKE FOR THE RUN. It read "the model provider failed the call, so the run never reached the server" — but the reason is applied PER ROW, so a multi-turn iteration whose provider died at turn 4 keeps its earlier measured rows, and that claim would sit directly beside a `call: passed` that disproves it. It now says "so this stage was never measured", which is true of the row it labels. Mutation-checked: restoring the old ordering reproduces the self-contradicting chain, and restoring the run-level wording fails the label's scope test. SDK 6896 passed; CLI gate suite 1182 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_900e8d0e-1130-482c-b0b4-ccd8bc9a57a0) |
|
Both right, and both are defects I introduced with last round's withdrawal. Fixed in The cascade ran before the withdrawal. Confirmed by reading the order: the positional pass computes The withdrawal now runs first, so the cascade sees the rows as they will actually be reported. The later stages read The cascade is repaired, not disabled, and there's a test for that half too: The label spoke for the run. Also right — Mutation-checked: restoring the old ordering reproduces the self-contradicting chain; restoring the run-level wording fails the scope test. SDK 6,896 passed; CLI gate suite 1,182 passed, 0 failed. Three rounds in, this is converging — 8 findings, then 5, then 2 — and each has been specific and verifiable rather than a reshape of an earlier one, so it's been worth working through. If a fourth round lands against these fixes I'll stop pushing and summarise what's still flagged instead of iterating further. Generated by Claude Code |
2 participants
UVH-IN2 — inspector lane, step 4. Analyzer 7 → 8. Stacked on #4491 → #4490 → #4488.
The defect
20 trials in one audited prod window failed on "credit balance too low… Anthropic API". The step errored, the asserts were skipped, and the chain ended
userValue: notMeasured / noEvidenceCapturedwith no failure category at all.Our provider's outage, presented as an unattributed server failure — in a report card whose entire job is to say whose side broke.
The fix
The layer that failed is tagged at the catch site and carried into the chain.
providerErrormarks the stages a model-call failure left blank, andcategoryForfiles the run undersetup— the existing bucket for our own side breaking, so no new category was needed.Structural, never a message classifier
drive-hosted-eval-turn.tsalready knows which layer it is in, so nothing needs to be inferred from error text:failTurn(engine stream failed)modelmapThrownTurnError(…, "runAssistantTurn")modelmapThrownTurnError(…, "pre-turn setup")setupA text classifier would be one provider's wording away from mis-attributing a whole class of run. The engine's
code/httpStatusare preserved and ride along as diagnostics for a reader — deliberately not part of the decision.Worth recording for the next reader: I checked whether codes alone could carry this. They can't.
MCPJamEngineErrorEventpopulatescode/httpStatusonly at its first fire site (a non-OK/streamresponse with a guardrail-shaped body), and the backend's structured codes (wallet_locked,billing_limit_reached,byo_provider_credential_failed) are our own guardrails, not the upstream provider's error — so the Anthropic credit case arrives message-only. The catch site is the only honest signal, which is what the original plan specified.Three deliberate boundaries
failed. A run that could not be attempted has measured nothing about the server; inflating a server failure rate with our own outage is exactly the mis-attribution this reason exists to prevent.evaluatoris never folded into another category.On the name
providerErroris broader than it sounds: outage, exhausted credits, rate limit, and our own spend guardrails all land here. What they share is that our side of the call broke — the only distinction the chain needs in order to stop blaming the server. That is written into the reason's own docblock rather than left for a reader to infer.Verification
providerErrorwith categorysetupand nothingfailed; measured stages keep their rows; a setup-layer error is not a provider error; an unclassified error changes nothing (the compatibility floor); a broken grader still wins.sdk6891 passed; inspector server 700 passed; CLI 1182 passed — gate exit codes unchanged.STAGE_REASONSmoves in this bump, and the backend mirror already carries the member — backend #1193 shipped it deliberately ahead of this PR, so nothing quarantines during the deploy window. That was the whole point of splitting it out.Stated limitation
The legacy verdict still counts these trials failed. Changing verdict population is a customer gate change and stays deferred behind its own product decision and release note — this PR fixes attribution, not the gate.
Generated by Claude Code
Note
Medium Risk
Touches eval failure attribution across hosted, harness, and local BYOK paths and bumps stage analyzer metadata customers read in report cards; behavior is heavily tested but misclassification would skew failure bucketing.
Overview
When our model call fails (credits, rate limits, provider outage), eval stage chains no longer leave blank
noEvidenceCapturedrows with no failure category or file the outage as a server-side selection defect.SDK (analyzer 7 → 8): Adds
providerErrorand threadsstepError.sourcefrom runners intoderiveStageResults. Model-layer failures re-label unmeasured stages and withdraw absence verdicts (e.g.missingToolCall) that are unknowable after a provider death, while presence/server observations stay.categoryFormaps these runs tosetup. Provider withdrawal runs before the positional cascade so downstream rows do not citeearlierStageFailedfor a failure that was withdrawn.Inspector runners: Catch sites tag
modelvssetupstructurally (not from error text) and passstepErrorthrough finalize. Hosted turns usefailedLayerForEngineErroron enginephase. Harness setsmodelInvokedbeforeagent.streamand reportsphase: setup | streamononEngineError. Local BYOK setsstepErrorSourceon empty streams and on non-tool error spans only when category isllm(modelLayerForErrorSpan). Step executor propagateserrorSource/ codes to iteration results.Tests and parity snapshots bump to analyzer 8. Legacy trial verdicts still count as failed; only attribution changes.
Reviewed by Cursor Bugbot for commit 0440ac3. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
UVH-IN2 — fixes eval report attribution so a model-provider failure is filed as ours, never blamed on the server or left uncategorized.
providerError(per-stage: "this stage was never measured") and the run is categorizedsetup.selection: failed / missingToolCall— are withdrawn toproviderError, since "it did not happen" is unknowable when our call died first; presence verdicts still stand, as do measured stages.providerErrorinstead ofearlierStageFailedwith no failed stage behind them.projectId, auth bearer, broker delivery) were taggedmodelthrough the shared stream callback; the engine now reports the phase it failed in, derived from a flag set right before the model call rather than after it resolves, so those land onsetup.failed, and a broken grader still outranks it.Written for commit 0440ac3. Summary will update on new commits.