fix(evals): preserve first-pass and retry evidence

[proffesor-for-testing]

fix(evals): preserve first-pass and retry evidence

Problem

Agentic QE's skill evaluator retries failed cases by default, but a successful retry replaces the first failure before metrics and learning consume the run. At protected main ffc0c6a5173874070bb2d07b29e9ae2a7383ed55 (v3.14.3):

• ParallelEvalConfig.retryFailedTests defaults to true.
• EvalWorker.executeBatch() replaces the first failed TestCaseResult with a successful retry and decrements the worker's failure count.
• executeTestCase(..., isRetry) accepts an isRetry argument but does not record or use it. TestCaseResult has no run, attempt, predecessor, retry, provider-request, token, or terminal-disposition fields.
• runEvalParallel() computes one pass rate from the replaced results, records that score and those results to SkillValidationLearner, and returns no first-pass or attempt-level denominator.
• The retry integration test verifies only that extra model calls occurred and that at least three final cases passed. It does not require conservation of the failed attempt, separate first-pass/eventual metrics, or truthful cost and instability reporting.

Therefore a fail-then-pass case can become indistinguishable from a first-pass success. The headline pass rate, confidence history, cross-model analysis, and feedback loop can inherit best-of-two selection while reporting one apparent attempt and omitting the first call's cost, latency, and failure evidence. This is a static finding; no production promotion was attempted and no benchmark result was recomputed today.

Why now

Four new or newly surfaced primary sources point to the same contract:

1. Project APE's verifier benchmark keeps failed calls and refusals in the miss denominator, separates deterministic scoring from stochastic execution, and measures repeatability on unchanged cases. Aggregate recall moved only 0–4 points between runs while individual outcomes flipped by as much as 28%; adding a second error reduced per-defect recall for 17 of 18 configurations (median drop four points). A final aggregate can therefore look stable while case-level reliability is not.
2. Inspect AI commit fa74574 fixed retries/requeues that inherited previous-attempt events in shared logs. Its tests cover same/new retry identities, failed synchronization, and overlapping restart races, and refuse to publish a mixed-attempt row.
3. OpenAI Codex commit 63eb71c binds late tool-result metadata to the original execution across waits and retries, but invalidates the binding when reused cells, duplicate IDs, or conflicting outputs make attribution ambiguous.
4. The public Agent Benchmark methodology reports first-pass success separately from cost/time/tokens to accepted result and requires failed attempts and unreached acceptance to remain in denominators. Its official graders/results are sealed, so it is methodological support, not independently reproduced performance evidence.

Proposed outcome

P0 — make attempt evidence append-only and generation-bound

Introduce a versioned per-case ledger rather than replacing results:

type AttemptDisposition =
  | 'passed'
  | 'quality-failed'
  | 'timeout'
  | 'refused'
  | 'provider-error'
  | 'cancelled'
  | 'unavailable';

interface EvalAttemptReceipt {
  schemaVersion: 'eval-attempt/v1';
  runId: string;
  testId: string;
  attemptId: string;
  attemptIndex: number;
  predecessorAttemptId: string | null;
  disposition: AttemptDisposition;
  startedAt: string;
  finishedAt: string;
  model: string;
  provider?: string;
  providerRequestId?: string;
  harnessRevision: string;
  suiteVersion: string;
  suiteHash: string;
  oracleVersion: string;
  seed?: number;
  tokensUsed?: number;
  latencyMs: number;
  score?: number;
  sanitizedError?: string;
}

• Never overwrite or delete an earlier attempt because a later one passed.
• Bind every late result to one (runId, testId, attemptId) generation. If identity is duplicated, reused, or ambiguous, retain the event as unattributed evidence and fail the affected case closed rather than guessing.
• Give retry admission an explicit reason and policy version. A quality failure, timeout, refusal, provider failure, and unavailable executor are not interchangeable.
• Persist the complete attempt ledger before publishing aggregate results or learning evidence. A partial write must not certify a complete run.

P0 — separate capability from recovery and instability

Report at least:

• firstPassPassRate: one predeclared first attempt per eligible case;
• eventualPassRate: cases with any eligible pass within the declared retry budget;
• retryRecoveryRate: first-pass failures later recovered;
• regressionOnRetryRate: first-pass passes that fail an explicit replicate, when replication is requested;
• attempt-disposition counts, missing/unavailable counts, total calls, tokens, cost basis, and elapsed time;
• per-case agreement/flip matrix for exact replicates.

Do not relabel eventual success as first-pass capability. Existing suites may opt into an explicit compatibility policy, but the report and receipt must name it and must not silently compare it with first-pass runs.

P1 — make learning and trust decisions retry-aware

• SkillValidationLearner must receive the attempt ledger plus a declared aggregation policy, not only the selected final result.
• Default retry-recovered cases to unstable/recovered, not ordinary positive evidence. Promotion may use them only under an explicit policy with separate weights and minimum independent support.
• Exclude provider/runtime unavailability from the quality denominator only when the report keeps it in an instrument-validity denominator; never drop it entirely.
• Preserve rejected, failed, null, and ambiguous attempts for audit and calibration while keeping raw prompts/results and secrets out of durable learning artifacts.
• Compose with feat(learning): qualify successful trajectories before durable promotion #638's learning-admission contract and feat(verification): qualify judge repeatability before automation #658's repeatability policy; do not create a second promotion mechanism.

P2 — harden concurrent and resumed evaluation

• Ensure worker batching, retries, cancellation, resume, and shared persistence cannot mix generations or attach late events to a newer attempt.
• If a retry/requeue overlaps synchronization, publish either a complete old generation or a complete new generation, never a hybrid.
• Carry exact executor, model/provider, harness, suite, oracle, and revision identity through CLI, MCP, exported JSON, and learning records.
• Add a migration disposition for legacy single-result records (legacy-final-only) rather than inferring first-pass success.

Acceptance criteria

• A fail-then-pass case retains both ordered attempts and reports first-pass failure plus eventual recovery.
• A pass-then-fail replicate retains both attempts and reports the flip; aggregate stability cannot hide case instability.
• TestCaseResult or its replacement contains stable run/test/attempt identity, attempt index, predecessor, disposition, and revision-bound provenance.
• isRetry is removed or becomes observable evidence; no ignored retry marker remains.
• First-pass, eventual, recovery, flip, unavailable, and cost metrics have explicit non-overlapping definitions and denominators.
• Failed calls, refusals, timeouts, cancellations, and provider errors remain visible and cannot silently disappear from every denominator.
• Late metadata/results attach only to an unambiguous attempt generation; duplicate/reused identities fail closed.
• SkillValidationLearner cannot treat retry recovery as ordinary first-pass positive evidence under the default policy.
• A partial persistence/sync failure cannot publish a complete aggregate or learning outcome.
• Parallel workers, requeues, and resume cannot mix events from different attempts.
• CLI, MCP, JSON export, and stored learning evidence preserve the same attempt semantics.
• Legacy final-only results are explicitly labeled and excluded from first-pass comparisons unless re-executed.

Validation experiments

1. Fail → pass: deterministic executor fails attempt 0 and passes attempt 1. Assert two receipts, first-pass failure, eventual recovery, and full call/token/time accounting.
2. Pass → fail replicate: request an exact replicate and assert the flip matrix records disagreement rather than retaining only the preferred result.
3. Failure classes: inject quality failure, timeout, refusal, provider error, cancellation, and unavailable executor; verify distinct dispositions and denominators.
4. Late result: delay attempt 0 until attempt 1 starts. Its result must remain bound to attempt 0 or be marked unattributed, never certify attempt 1.
5. Duplicate/reused IDs: reuse test/attempt IDs and provide conflicting outputs. Aggregation and learning must fail closed.
6. Parallel collision: run identical test IDs in different batches/runs with inverted completion order. No event or result may cross generations.
7. Sync race: retry/requeue while the ledger is being persisted, plus injected write failure. Readers see no mixed attempt and the next sync conserves all committed evidence.
8. Learning mutation: remove the retry-recovered disposition before SkillValidationLearner; contract tests must reject the promotion input.
9. Legacy import: ingest a final-only historical record. It remains legacy-final-only and cannot populate first-pass metrics.
10. Cost accounting: compare retry-off and retry-on with identical first attempts; first-pass metrics match, while eventual outcome and total cost differ transparently.
11. Multi-defect stress: run one-defect and two-defect variants on the same cases; report paired per-defect change rather than unrelated aggregates.
12. Privacy: seed secrets in model output/error text; receipts retain provenance and disposition without raw sensitive content.

Relationship to existing work

• feat(benchmarks): qualify paired comparisons before winner promotion #692 defines qualified paired benchmark comparisons and stable task/attempt alignment; this issue supplies truthful attempt evidence for the skill-eval producer.
• feat(verification): qualify judge repeatability before automation #658 qualifies judge repeatability; this issue preserves executions needed to measure it.
• feat(learning): qualify successful trajectories before durable promotion #638 governs learning admission; this issue prevents retry-selected evidence from reaching that gate as an apparent first-pass success.
• feat(reliability): make session resume fail-closed and crash-consistent #659 governs crash-consistent session resume; this issue covers evaluation-attempt identity and aggregation, while reusing its commit/generation principles.
• fix(observability): expose audit-buffer loss and evidence completeness #695 covers bounded audit-buffer completeness; this issue covers benchmark attempt conservation.
• fix(coordination): make cancellation abort in-flight work #707 covers cancellation/quiescence; cancelled eval attempts should compose with its receipts.
• fix(test-execution): reject incomplete Vitest execution evidence #712 covers owned Vitest report and runner-exit truth; this issue targets the separate LLM skill-evaluation path.

Evidence limits and non-goals

• The Agentic QE finding is static exact-head evidence; today did not execute a real model eval or reproduce a production promotion.
• Project APE is a domain-specific injected-defect benchmark, not proof that its numeric rates transfer to QE skill evaluation.
• The Agent Benchmark public release withholds official graders and results; only its attempt-aware methodology is used here.
• This issue does not remove retries, prescribe one universal stochastic metric, or duplicate the general statistics/promotion machinery in feat(benchmarks): qualify paired comparisons before winner promotion #692.