feat: add ScenarioRank Phase 3A evaluation harness - #8
Draft
abdo2006-dev wants to merge 6 commits into
Draft
Conversation
Adds decision-benchmark-v1: 16 fully synthetic decision cases (21 scenario executions) covering basic ranking, multi-scenario behaviour, evidence quality, input-permutation robustness, and pairing. The benchmark separates two things that are routinely conflated. Each case carries deterministic_expectations (objectively checkable: candidate and scenario coverage, pair completeness, logical-stage accounting, whether the reported winner matches the deterministic ranking) and rubric_dimensions (human judgment: grounding, trade-off clarity, uncertainty handling). No case hardcodes one "correct" natural-language answer: where several winners are defensible, allowed_winner_ids lists all of them, and where the evidence is too thin or contradictory to justify any winner, the case makes no claim at all and checks instead that every candidate is flagged for human review. Versioning is enforced rather than documented: case IDs are immutable, a meaning change requires a new benchmark_version, a cosmetic change increments metadata_revision only, and a loader refuses an unsupported schema_version rather than attempting a best-effort read of an unknown format. Every case declares synthetic: true and data_policy: "synthetic-only" as schema literals, not booleans a later case could quietly flip. Every candidate, company, and record is invented. Case inputs are validated against the production evaluationRequestSchema, so the benchmark cannot describe a request the server would reject.
Adds the harness that executes decision-benchmark-v1 against the real production pipeline: real prompts, real schemas, real deterministic scoring, real batch-identity validation. Eleven deterministic graders cover contract validity, candidate coverage, scenario coverage, ranking consistency, score integrity, pairing integrity, pipeline accounting, not_measured honesty, winner expectations, unsupported claims, and uncertainty acknowledgement. Score integrity recomputes every recomputable deterministic value using the production formulas in server/domain/scoring.js; weighted_fit_score is excluded and the reason is stated, because normalised criterion weights are not part of the public response. The unsupported-claim checks are deliberately conservative and scoped to model-authored narrative fields only, so the pipeline's own honest "has not been measured" wording cannot trip them. The narrative-contradiction check is name-based and reports itself skipped when the winner's display name is shared by another candidate, rather than guessing. Seven offline fake-provider profiles: three valid, and four deliberately invalid so the graders are proven to catch real defects rather than only ever observed passing. Scores are keyed by candidate ID, never by array position, which is what makes the candidate-order permutation a test of the pipeline instead of a test of the fixture. Live mode is gated hard and was never executed: it requires --live, an API key, an explicit positive budget, and a deliberate case selection; it refuses CI by default; it refuses a model with no recorded pricing, because a budget that cannot be computed cannot be enforced; and it stops before starting any execution whose worst case would breach the limit. The provider factory is imported lazily, so a refused invocation never constructs an OpenAI client. Comparison returns improved/regressed/unchanged/inconclusive. Only required invariants move the verdict; cost, token, and duration deltas are reported raw and explicitly marked not_assessed, because two runs cannot support a significance claim. An output change with no invariant change is inconclusive, not unchanged. Artifacts go to .eval-runs/ and are schema-validated and scanned for secrets and absolute paths before they are written, not after. A case may declare known_defects for documented, pre-existing product defects. Those failures stop gating the exit status, but a case-level check raises a required failure if a known defect ever stops reproducing, so a suppression cannot outlive the defect it describes.
Adds 309 evaluation tests in their own Vitest project, so frontend, backend, and evaluation counts stay independently reportable. None of them makes a network request and none of them calls OpenAI. Every grader is proven twice: once that it passes a genuinely good result produced by executing the real pipeline, and once that it catches a specific, realistic defect. A grader that has only ever been seen to pass is not evidence of anything. Coverage includes benchmark schema rejection paths (duplicate case IDs, unsupported schema versions, invalid tags, malformed candidate IDs, missing synthetic-data policy, mismatched pair counts and stage counts, winner expectations naming unknown candidates); case validation against the production request contract; runner behaviour with every network primitive disabled and no API key present; determinism of decision content across repeated runs; live-mode refusals and budget arithmetic driven entirely by injected values; comparison verdicts including the incompatible-benchmark refusals; and permutation-variant linkage. Repository-protection tests enforce the boundaries rather than trusting them: production imports nothing from evals/, .eval-runs is ignored by git itself, no committed evaluation file contains a secret or an absolute machine path, the request observer records no payloads, every CLI supports --help and exits nonzero on a required failure, and no dependency was added. Wires npm run eval:validate/fixtures/live/compare, adds evals/ to lint:server, includes test:evals in npm test, and ignores .eval-runs/.
Adds docs/evaluation/ (architecture, benchmark v1, human review guide, runbook) and ADR-0009 recording the local-first evaluation decision and the benchmark-versioning policy. ADR-0009 explains why the hosted OpenAI Evals API was deliberately not integrated in this phase: a hosted service sees prompts and completions but cannot observe the deterministic layer most of these checks target, it requires network access and spend for every run, and it would reintroduce, at the evaluation layer, the vendor coupling the provider contract removed. The seams are drawn so it can be added later as a provider factory plus a reporter. Documents why evaluation precedes prompt optimisation, what a fixture run does and does not prove, deterministic versus qualitative grading, live-run safety and budget controls, artifact privacy, and the explicit limits of the comparison command. Records the real defect the harness found on its first run and deliberately did not fix. SR-P3A-001: computeRiskAdjustedScore can return a negative value for a weak candidate, while completedPipelineResponseSchema bounds risk_adjusted_score to 0-100 and server/http/routes.js validates its own response before sending — so a run containing a weak enough candidate returns a generic 500 after every OpenAI call has already been paid for. Phase 3A is scoped to measurement and forbidden from changing scoring or contracts, and either candidate fix moves the baseline the harness measures from, so it is tracked as a known defect (KNOWN_LIMITATIONS.md P0.7) rather than patched. Also records a second gap surfaced while building case-002: ScenarioRank has no near-tie uncertainty signal at all, so a decision separated by noise is never flagged for human review (P2.5). States plainly, in the docs and inside every run artifact, that this is a development benchmark: not scientifically validated, not representative of real hiring decisions, not evidence of fairness or demographic neutrality, not a legal-compliance test, not a calibrated-confidence benchmark, and not a production service-level objective. Phase 3B has not started.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — do not merge.
Phase 3A remains an evaluation-only harness over the real production pipeline. This PR does not change production scoring, ranking, pairing, prompts, provider behavior, HTTP behavior, frontend behavior, or shared response contracts. SR-P3A-001 remains intentionally unfixed. No live evaluation or OpenAI request was made; Phase 3B remains unstarted.
Final hardening
unexpected_failure, thenbaseline_change_required,pass_with_known_defects, andclean_pass; Markdown reports full fixture and defect-comparison counts.eval:update-integrityis explicit, local-only, requires a reason and version/metadata change, and writes digest provenance without committing.Current offline baseline
pass_with_known_defects: 16/16 cases complete without unexpected failure; 12 clean cases; 8 expected observations across 4 affected executions; 0 unexpected failures and 0 unexpected defect resolutions.Verification
npm run lint,npm run lint:server,npm run typecheck, all project guards,npm run build, andnode --check server.mjspassed.npm run eval:validateand repeatednpm run eval:fixtures -- --no-writepassed.npm testpassed: 646 tests (103 frontend + 217 server + 326 evaluation).npm lspassed;git diff --checkpassed.npm auditcould not contact npm's audit endpoint in the restricted environment.