UVH-IN2: attribute a model-provider failure to us, not to the server

.changeset/evals-provider-error-attribution.md

@@ -0,0 +1,24 @@
+---
+"@mcpjam/sdk": patch
+"@mcpjam/inspector": patch
+---
+
+A model-provider failure is attributed to us, not filed against the server
+
+20 trials in one audited prod window failed on "credit balance too low… Anthropic API". The step errored, the asserts were skipped, and the chain ended `userValue: notMeasured / noEvidenceCaptured` with **no failure category at all** — our provider's outage presented as an unattributed server failure, in a report card whose entire job is to say whose side broke.
+
+The layer that failed is now tagged at the catch site and carried into the chain. `providerError` marks the stages a model-call failure left blank, and `categoryFor` files the run under `setup` — the existing bucket for our own side breaking, so no new category was needed.
+
+**Classified structurally, never by reading the message.** `drive-hosted-eval-turn.ts` already knows which layer it is in: `failTurn` is the engine's stream, `mapThrownTurnError` names its own call site, and "pre-turn setup" is the one that never reached the model. A text classifier would be one provider's wording away from mis-attributing a whole class of run. The engine's `code` and `httpStatus` ride along as diagnostics for a reader, and are deliberately _not_ part of the decision.
+
+Three boundaries are deliberate:
+
+- **Only blank rows are re-labelled.** A provider dying at turn 4 does not un-observe turns 1–3, so a stage with its own evidence keeps its own row.
+- **Never `failed`.** A run that could not be attempted has measured nothing about the server, and inflating a server failure rate with our own outage is the mis-attribution this reason exists to prevent.
+- **A broken grader still outranks it.** `evaluator` is never folded into another category.
+
+`providerError` is broader than its name: it covers a provider outage, an exhausted credit balance, a rate limit, and our own spend guardrails. What they share is that _our_ side of the call broke, which is the only distinction the chain needs to stop blaming the server. That is stated in the reason's own docblock.
+
+Analyzer 7 → 8. This bump moves `STAGE_REASONS`, and the backend mirror already carries the member — it shipped deliberately ahead of this change — so nothing quarantines during the deploy window.
+
+**Stated limitation, unchanged:** the legacy verdict still counts these trials failed. Changing verdict population is a customer gate change and remains deferred behind its own product decision and release note.

.changeset/evals-provider-error-catch-sites.md

@@ -0,0 +1,22 @@
+---
+"@mcpjam/inspector": patch
+---
+
+Provider-error attribution reaches the local path, and stops over-claiming on the harness
+
+Two catch sites turned out to be broader than the comments above them claimed. Both were found in review, and both are the provider-error defect wearing a different coat — attributing to the model a failure that never reached it, which misleads a report card exactly as much as the original did.
+
+**The harness reports setup failures through the same callback as stream failures.** `runHarnessTurn` wraps its entire turn — preparation included — in one `try`, so a missing `projectId`, a missing auth bearer, or disabled broker credential delivery arrives at `onEngineError` looking precisely like a provider outage. `failTurn` tagged every one of them `model`, which files our own setup bug as the provider's.
+
+The engine now reports the **phase** it failed in, derived from the trace-started flag the emitter already holds to decide whether a turn happened at all — never from reading the message. `failTurn` reads that phase. An engine that reports none still means `model`: every emitter that omits it today is a real stream failure, and defaulting the other way would un-attribute the outages this work was built for.
+
+**The local path carried no source at all.** When `orgByokRuntime.kind === "local"`, execution goes to `runLocalIteration` rather than the hosted driver, and neither of its finish calls supplied a `stepError`. So local-BYOK trials that died on the model call still finalized with blank stage reasons and no failure category — the exact misattribution the original change removed, surviving untouched on the path it never covered.
+
+The local driver now records the layer at both of its error sites, and threads it through both finish paths:
+
+- **An empty model stream** is unambiguously the model call — no other layer reaches that branch.
+- **A non-tool error span** is the model call **only when its category is `llm`**. That branch selects every non-tool error span, and `connection`, `discovery` and `oauth` spans are all reachable there; tagging those `model` would blame the provider for a server we could not reach.
+
+Anything else leaves the source unset. An absent source attributes nothing, which is the right floor — no attribution is strictly better than a confident wrong one.
+
+Both decisions are extracted as small pure functions (`failedLayerForEngineError`, `modelLayerForErrorSpan`) so the choice can be read and tested on its own rather than inferred from a call site's position. Mutation-checked: ignoring the phase, tagging every span, and tagging none each fail exactly their intended tests.

.changeset/evals-provider-error-chain-consistency.md

@@ -0,0 +1,15 @@
+---
+"@mcpjam/sdk": patch
+---
+
+Withdrawing a provider-blocked failure no longer leaves the chain arguing with itself
+
+Two follow-ups on the withdrawal added in the previous round.
+
+**The cascade ran before the withdrawal.** The positional pass reads `failed` rows to decide which later stages "never ran". Withdrawing a provider-blocked failure *after* it had run left `call`, `response` and `userValue` still saying `earlierStageFailed` while no stage failed and no `firstFailedStage` existed — three rows citing a failure the chain no longer records.
+
+The withdrawal now happens first, so the cascade sees the rows as they will actually be reported: a provider outage marks the later stages `providerError`, which is *why* they were not measured, rather than blaming a stage that is no longer failed. A failure the provider did **not** explain still cascades exactly as before — `unexpectedToolCall` survives the withdrawal, stays the first failed row, and the stages after it still read `notReached`. The cascade is repaired, not disabled.
+
+**The reason's label spoke for the run.** It read *"the model provider failed the call, so the run never reached the server"* — but `providerError` is applied per row, so a multi-turn iteration whose provider died at turn 4 keeps its earlier measured rows. The run-level claim would sit directly beside a `call: passed` that disproves it, and send a reader after the wrong timeline. It now says *"…so this stage was never measured"*, which is true of the row it labels.
+
+Mutation-checked: restoring the old ordering reproduces the self-contradicting chain, and restoring the run-level wording fails the label's scope test.

.changeset/evals-provider-error-withdraws-absences.md

@@ -0,0 +1,19 @@
+---
+"@mcpjam/sdk": patch
+"@mcpjam/inspector": patch
+---
+
+A provider outage withdraws the failures it made unknowable, and is dated from the handover
+
+Two further review findings, and the first is the one that mattered most: the provider-error fix was **inert on the shape it exists for**.
+
+**A missing tool call is not a selection defect when the provider never let us make it.** `applyProviderError` re-labelled only rows that measured *nothing*. But a case expecting a tool call whose provider died already has `selection: failed / missingToolCall` written by the matcher before the chain is derived — so `firstFailedStage` stayed `selection`, `categoryFor` returned `selection`, and the outage was filed as a model-selection defect. The exact misattribution this reason was built to remove, on the commonest case in the corpus.
+
+The fix turns on a distinction worth stating plainly:
+
+- An **absence** verdict — no call arrived, an assertion over the output did not hold, the judge scored a truncated transcript low — is only sound if the run was allowed to finish. When our own model call died first, "it did not happen" has a second explanation that outranks the accusation, and we cannot tell which is true. Those rows become `notMeasured / providerError`, and their evidence goes with the verdict it supported.
+- A **presence** verdict stands: an unexpected call was really made, arguments really mismatched, a tool really errored, a render really failed. `connectFailed` and `toolsListFailed` matter most here — they happen *before* any model call, so a server that would not connect is never excused by a provider error that came later. Letting a provider blip launder a genuine server defect would be the worse bug of the two.
+
+**The phase flag was read too late.** It used `driver.traceStarted`, and the driver is only built after `agent.stream(...)` *resolves* — so an immediate provider rejection (auth, quota, a rate limit) was reported as our setup failing when the model had in fact been asked. The flag is now set immediately before the call, marking the handover itself rather than a successful one.
+
+Mutation-checked in both directions: never withdrawing a failed row reproduces the original bug; withdrawing every failed row launders the server defects; and keeping stale evidence on a withdrawn row leaves it arguing for a failure it no longer claims. SDK 6,894 passed; CLI gate suite 1,182 passed, 0 failed; server harness + evals 1,079 passed.