UVH-IN3: name a verdict/chain disagreement instead of calling it missing data

.changeset/evals-disagreement-claims-less.md

@@ -0,0 +1,21 @@
+---
+"@mcpjam/sdk": patch
+---
+
+The verdict/chain disagreement claim narrows to what the row actually proves
+
+Two review findings, both cases of a claim reaching past its evidence — which is the specific failure this wording was introduced to fix, so getting it wrong here would have been particularly poor.
+
+**A chain with an unmeasured stage is incomplete, not green.** The predicate asked for "something passed and nothing failed". A chain with connection and discovery `passed` and selection `notMeasured / noEvidenceCaptured` satisfies that — while the verdict may be failing on exactly the stage the chain could not read. That is a measurement gap, and calling it a conflict sends someone looking for a contradiction that is not there.
+
+Every applicable stage must now have `passed`. `notApplicable` is the one state that does not block the claim: a stage the case never exercises is out of scope rather than missing evidence, and requiring it to pass would make the claim unreachable for any case that does not use all six stages.
+
+**The stale-analyzer wording named a cause the version does not establish.** It read *"…predates the analyzer that reports an errored tool call on an unauthored case — re-run the case to attribute it"*. But the version only proves what the analyzer was **able to see**; it is not evidence that such a call occurred. Every legacy row with any other uncategorised disagreement was being sent after one specific finding.
+
+It now says only what the row establishes — that the chain was derived by an analyzer measuring strictly less than the current one, so re-deriving may attribute what this one could not:
+
+> the recorded verdict disagrees with the measured chain; this run's chain was derived by an older analyzer that measures less than the current one — re-run the case before investigating further
+
+That keeps "re-run" as a real instruction without attaching a cause to it.
+
+Mutation-checked: reverting to the weak predicate, refusing `notApplicable`, and restoring the tool-error wording each fail exactly their intended tests. SDK 6,899 passed; CLI gate suite 1,182 passed, 0 failed — verdicts unchanged, as for every step in this program.

.changeset/evals-verdict-chain-disagreement.md

@@ -0,0 +1,23 @@
+---
+"@mcpjam/sdk": patch
+---
+
+A verdict that disagrees with the chain is named as a disagreement, not as missing information
+
+When a case failed but the chain recorded no failure category, the decision summary said one thing: `inspect the case trace; no failure category was recorded`. That sentence covers two runs that are not alike, and it describes the more interesting one wrongly.
+
+If the chain never validated, or measured nothing, then information really is absent and there is nothing more to say. But if the chain validated, every applicable stage came back `passed`, and the recorded verdict still says failed — that is not an absence. It is two things we hold in conflict, and sending someone to look for a missing measurement is sending them after the wrong thing.
+
+That run now reads: `the recorded verdict disagrees with the measured chain; inspect the case trace`.
+
+**The claim is asserted only when it is structurally established** — the chain validated, at least one stage actually `passed`, no stage `failed`, and the verdict is `failed`. Each of those four does real work:
+
+- **Something must have passed.** A policy-blocked run has all six stages `notMeasured / blockedByPolicy`, so nothing failed — but nothing was measured either, and there is nothing for a verdict to disagree _with_.
+- **Nothing may have failed.** `failureCategory` is read off the stored row, and the derivation schema pins only `firstFailedStage` to the failed row, never the category. A row can therefore validate carrying a `failed` stage and no category; calling that "the chain found nothing wrong" would be contradicted by the row itself.
+- **The verdict must say failed.** A trial that recorded no verdict is still diagnosed, and its chain can be entirely green. Nothing was decided, so nothing is in conflict.
+
+**One cause is knowable rather than guessed.** A chain derived before analyzer 7 could not report an errored tool call on a case that authored no tool expectation — every applicable stage green, the verdict red, and no row in which to say why. For those rows the wording names that cause and says to re-run, because a newer analyzer will attribute it. This is read from the row's own `stageAnalyzerVersion`, never inferred from the shape of the failure.
+
+**Nothing else guesses at a cause.** The assembler cannot see one from here, and a guess dressed as a finding is exactly what this vocabulary exists to prevent.
+
+Post-IN7 this fires rarely by design: it now serves legacy-analyzer rows and residual unknowns. Wording only — the contract stays `.strict()` at schemaVersion 1, no stage row, state, reason or category moves, and no verdict changes.

sdk/src/contract/decision-labels.ts

@@ -197,6 +197,40 @@ export const NEXT_ACTION_BY_FAILURE_CATEGORY = Object.freeze({
 export const DECISION_SUMMARY_FALLBACK_NEXT_ACTION =
   "inspect the case trace; no failure category was recorded";
 
+/**
+ * The action when the recorded verdict and the measured chain DISAGREE.
+ *
+ * A narrower, and therefore more useful, statement than the fallback above:
+ * the chain validated, every applicable stage came back ok, and the verdict
+ * still says failed. "No failure category was recorded" is true of that run
+ * but describes it as an absence of information, when in fact two things we
+ * hold are in conflict — which is a different thing to go and look at.
+ *
+ * Named as a disagreement and nothing more. The chain cannot see WHY from
+ * here, and a guess at the cause dressed as a finding is exactly what this
+ * whole vocabulary exists to prevent.
+ */
+export const DECISION_SUMMARY_VERDICT_CHAIN_DISAGREEMENT_NEXT_ACTION =
+  "the recorded verdict disagrees with the measured chain; inspect the case trace";
+
+/**
+ * The same disagreement on a run whose chain predates analyzer 7.
+ *
+ * What the version proves is NARROW, and the first draft of this line over-read
+ * it. A pre-7 analyzer could not report an errored tool call on a case that
+ * authored no tool expectation — but that is a statement about what the
+ * analyzer was ABLE to see, never evidence that such a call occurred. Naming
+ * the tool error as the cause would have sent a reader after a specific
+ * finding on every legacy row, whatever actually went wrong.
+ *
+ * So this says only what the row itself establishes: the chain was derived by
+ * an analyzer that measures strictly less than the current one, and
+ * re-deriving may therefore attribute what this one could not. That makes
+ * "re-run" a real instruction without attaching a cause to it.
+ */
+export const DECISION_SUMMARY_STALE_ANALYZER_DISAGREEMENT_NEXT_ACTION =
+  "the recorded verdict disagrees with the measured chain; this run's chain was derived by an older analyzer that measures less than the current one — re-run the case before investigating further";
+
 /** Every vocabulary this module renders, for tests that assert totality. */
 export const DECISION_LABEL_VOCABULARIES = Object.freeze({
   stages: USER_VALUE_STAGES,

sdk/src/contract/decision-summary.ts

@@ -72,10 +72,13 @@ import {
 import { opaqueIdSchema } from "./identity.js";
 import {
   DECISION_SUMMARY_FALLBACK_NEXT_ACTION,
+  DECISION_SUMMARY_STALE_ANALYZER_DISAGREEMENT_NEXT_ACTION,
+  DECISION_SUMMARY_VERDICT_CHAIN_DISAGREEMENT_NEXT_ACTION,
   NEXT_ACTION_BY_FAILURE_CATEGORY,
 } from "./decision-labels.js";
 import {
   STAGE_ANALYZER_VERSION,
+  STAGE_ANALYZER_VERSION_EVIDENCE_TRIGGERED_RESPONSE,
   stageDerivationSchema,
   stageResultRowSchema,
   type StageResultRow,
@@ -826,10 +829,72 @@ function assembleDiagnostic(
     evidence: assembleEvidence(input, iteration, chain),
     nextAction: category
       ? NEXT_ACTION_BY_FAILURE_CATEGORY[category]
-      : DECISION_SUMMARY_FALLBACK_NEXT_ACTION,
+      : uncategorisedNextAction(chain, iteration.result),
   };
 }
 
+/**
+ * What to tell a reader when the chain established no failure category.
+ *
+ * Two of these runs are not the same thing, and the old single line described
+ * both as an absence of information:
+ *
+ *   - The chain measured everything it could and found nothing wrong, while
+ *     the recorded verdict says failed. That is not missing information — it
+ *     is two things we hold disagreeing, which is a different investigation.
+ *   - Anything else (an unverified chain, a stage that never measured, a run
+ *     that did not fail): no category and nothing more to say.
+ *
+ * The disagreement is asserted only when all four halves of it are
+ * STRUCTURALLY established: the chain validated, at least one stage actually
+ * passed, every applicable stage passed, and the verdict is `failed`. Nothing
+ * here guesses at a cause — the chain cannot see one from here, and a guess
+ * dressed as a finding is what this vocabulary exists to prevent.
+ */
+function uncategorisedNextAction(
+  chain: EvalRunDecisionChain,
+  result: EvalRunDecisionIterationInput["result"]
+): string {
+  if (chain.status !== "verified" || result !== "failed") {
+    return DECISION_SUMMARY_FALLBACK_NEXT_ACTION;
+  }
+  // A disagreement needs the chain to be COMPLETE and clean, not merely
+  // un-failed. Every stage must have `passed`, except the ones the case does
+  // not exercise at all.
+  //
+  // "Something passed and nothing failed" is too weak, and its gap is the same
+  // mistake in slower motion: a chain with connection and discovery passed and
+  // selection `notMeasured / noEvidenceCaptured` has nothing failed and
+  // something measured, but the verdict may be failing on exactly the stage
+  // the chain could not read. That is a measurement GAP, and telling someone
+  // two things they hold are in conflict sends them looking for a
+  // contradiction that is not there.
+  //
+  // `notApplicable` is the one state that does not block the claim: a stage
+  // the case never exercises is not missing evidence, it is out of scope, and
+  // requiring it to pass would make the claim unreachable for every case that
+  // does not use all six stages.
+  const measuredSomething = chain.stages.some((row) => row.state === "passed");
+  const everyApplicableStagePassed = chain.stages.every(
+    (row) => row.state === "passed" || row.state === "notApplicable"
+  );
+  if (!measuredSomething || !everyApplicableStagePassed) {
+    return DECISION_SUMMARY_FALLBACK_NEXT_ACTION;
+  }
+
+  // A pre-7 chain gets a different INSTRUCTION, not a different diagnosis.
+  //
+  // The version establishes that this analyzer measured strictly less than the
+  // current one — before 7 an errored tool call on a case with no authored
+  // tool expectation had no stage able to report it — so re-deriving may
+  // attribute what this row could not. It does NOT establish that such a call
+  // occurred, which is why the wording names the analyzer and not a cause.
+  return chain.analyzerVersion !== undefined &&
+    chain.analyzerVersion < STAGE_ANALYZER_VERSION_EVIDENCE_TRIGGERED_RESPONSE
+    ? DECISION_SUMMARY_STALE_ANALYZER_DISAGREEMENT_NEXT_ACTION
+    : DECISION_SUMMARY_VERDICT_CHAIN_DISAGREEMENT_NEXT_ACTION;
+}
+
 function assembleChain(
   iteration: EvalRunDecisionIterationInput
 ): EvalRunDecisionChain {

sdk/src/contract/stage-derivation.ts

@@ -84,6 +84,18 @@ import {
  */
 export const STAGE_ANALYZER_VERSION = 8;
 
+/**
+ * The 7 above, named — the first analyzer that can report an errored tool call
+ * on a case which authored no tool expectation.
+ *
+ * A reader comparing a stored row against this can tell a chain that found
+ * nothing wrong from one that had no row in which to say so, which is the
+ * difference between an open question and a known, closable one. It lives
+ * beside the history rather than beside the copy that reads it so that a
+ * future bump is edited where the bump is already being written down.
+ */
+export const STAGE_ANALYZER_VERSION_EVIDENCE_TRIGGERED_RESPONSE = 7;
+
 /**
  * Why a stage landed where it did.
  *