From ea26c401fa519c94ac661cb673b95c2d14e7ddc5 Mon Sep 17 00:00:00 2001 From: Akshaya Shanbhogue Date: Mon, 21 Sep 2026 20:52:50 -0700 Subject: [PATCH 1/3] feat(criteria): add system_one_judge, a typed-rubric grader on a System One model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A System One model (TypeSafe's jev) generates no text: it reads one state and answers a map of typed questions with calibrated probabilities in a single round trip. That removes the two things a text judge has to defend against — an unparseable verdict and a tool call the model may decline to make — so the rubric the author writes IS the grading schema. The grade stays on our side: each question resolves to a value in [0,1] via its own expected/values, and the criterion score is their weighted mean. That buys determinism, a grade that can be recomputed from an archived transcript, and findings that print the arithmetic per question instead of arguing it in prose. A choice question's options may be written as a bare list when the names speak for themselves; it widens to the option->description map the API wants, with null descriptions, which the live endpoint accepts. Order is preserved and a repeated option is a load-time error. The judge reads the files it is given plus, opt-in per criterion, the agent's own final message (include_agent_output), its tool-call trajectory (include_tool_calls) and the simulated dialog (include_dialog) — so a rubric can grade how the agent worked and whether its summary was honest, not only the artifact left behind. tasks/smoke_system_one_judge.yaml exercises all three primitives over all three of those state slices, and runs in CI's smoke-pass bucket beside smoke_llm_judge. tests/test_system_one_judge_live.py pins the external wire shape (notably the STRING level keys in a score answer's probabilities) in the live-tests job, since the unit tests mock the invoker and would keep agreeing with a stale contract. Both need the TYPESAFE_API_KEY repo secret. A missing key escalates the row to ERROR rather than scoring it 0.0, so each job preflights it and fails with a message naming the key instead of reporting an unexplained tasks_errored. Deliberately not shared with llm_judge: checker_context.api_route resolves a TEXT judge model, which is not a substitute, and a transport failure escalates the row rather than scoring an ungraded row 0.0. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/contracts.md | 42 +++ .env.example | 8 + .github/workflows/pr-checks.yml | 50 ++- docs/TASK_DEFINITION_GUIDE.md | 63 +++- evalboard/lib/pricing.generated.ts | 2 + plugins/coder-eval/reference/criteria.md | 30 ++ src/coder_eval/criteria/system_one_judge.py | 157 +++++++++ src/coder_eval/evaluation/judge_system_one.py | 107 ++++++ .../evaluation/system_one_scoring.py | 178 ++++++++++ src/coder_eval/harbor/portability.py | 8 +- src/coder_eval/models/__init__.py | 24 +- src/coder_eval/models/criteria.py | 181 +++++++++- src/coder_eval/models/judge_defaults.py | 9 + src/coder_eval/models/results.py | 8 +- src/coder_eval/models/system_one.py | 186 ++++++++++ src/coder_eval/pricing.py | 3 + src/coder_eval/reports/html.py | 4 +- tasks/README.md | 7 +- tasks/smoke_system_one_judge.yaml | 105 ++++++ tests/test_success_criterion_union.py | 4 + tests/test_system_one_judge.py | 332 ++++++++++++++++++ tests/test_system_one_judge_live.py | 82 +++++ 22 files changed, 1569 insertions(+), 21 deletions(-) create mode 100644 src/coder_eval/criteria/system_one_judge.py create mode 100644 src/coder_eval/evaluation/judge_system_one.py create mode 100644 src/coder_eval/evaluation/system_one_scoring.py create mode 100644 src/coder_eval/models/system_one.py create mode 100644 tasks/smoke_system_one_judge.yaml create mode 100644 tests/test_system_one_judge.py create mode 100644 tests/test_system_one_judge_live.py diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index 0c8ee894c..60fc3210a 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -391,3 +391,45 @@ with a plain synchronous call, because offloading a single fast syscall only wid cancellation window. The cleanup itself is deliberately synchronous: a bare `await` inside `finally` is cancellable, and cancelling as that line is reached would skip cleanup and leak the copy with no reaper. + +## System One rubric scoring + +A System One model (TypeSafe's `jev`) does not generate text. It reads one state and answers +a map of typed questions — `noul` (P(yes)), `choice` (a distribution over options), `score` (a +distribution over ordered levels) — with calibrated probabilities. That inverts what a judge +criterion has to defend against. There is no prose to parse, so the whole `submit_verdict` +tool channel has no counterpart here: the response schema is fixed by the questions that were +asked, and a model that answers off-schema is a wire fault, not a grading fault. It also means +there is no system prompt to hold an identity, so the transcript's system-prompt slot carries +the rubric as sent — that, not a persona, is what a reviewer needs to replay a grade. + +### The score is ours, not the model's + +The model reports a distribution; the grade is arithmetic we do on it. Each question resolves +to a value in [0.0, 1.0] through the author's own `expected` / `values`, and the criterion +score is their weighted mean. Keeping that reduction on our side buys three things a text +judge cannot have: the same answers always produce the same grade, the arithmetic is printable +(`findings` carries one line per question, with the value and the weight), and a rubric can be +re-scored from an archived transcript without another call. + +`scoring: expected` weights every outcome by its probability, so a model that is genuinely +torn lands mid-scale instead of being rounded into a confident-looking verdict — the point of +a calibrated model. `argmax` exists for the case where partial credit is misleading rather +than informative: a gate. The default is `expected` because discarding the confidence is the +lossy choice and should be the one you ask for. + +A distribution that is absent, non-numeric or sums to zero falls back to the point answer +(`choice` / `score` / `noul`) rather than grading as 0.0 — a broken `probabilities` block is a +provider fault, and the point answer is still a real answer. A question that is *unanswered*, +or answered with the wrong primitive, is the opposite case: it scores 0.0 at its full weight +and names itself in `findings`, because the rubric asked something the grade depends on and +dropping it would quietly inflate the mean. + +### What it does not share with `llm_judge` + +It does not read `checker_context.api_route`. The eval route resolves a TEXT judge model, and +substituting one for a System One model is not a fallback, it is a different API. The +credential comes from the env var *named* by `api_key_env`, so only the name is ever stored on +the criterion or persisted into a run record. A transport failure raises +`JudgeInfrastructureError` and escalates the row, rather than following `llm_judge`'s +unconfigured-transport arm into a scored 0.0 — an ungraded row must not read as a failed one. diff --git a/.env.example b/.env.example index cb67f4953..a6ed76dd1 100644 --- a/.env.example +++ b/.env.example @@ -29,6 +29,14 @@ LOG_TO_FILE=false # Set to true to enable file logging # AWS_BEARER_TOKEN_BEDROCK="" # AWS_REGION="us-east-1" # BEDROCK_MODEL="eu.anthropic.claude-sonnet-4-5-20250929-v1:0" + +# TypeSafe System One (the system_one_judge criterion, model `jev`). NOT routed +# through API_BACKEND — a System One model answers typed questions with +# probabilities rather than text, so it is its own endpoint and its own key. +# The criterion stores only this variable's NAME (api_key_env), never the token. +# Unlike llm_judge, a missing key escalates the row to ERROR instead of scoring +# it 0.0: an ungraded row must not read as a failed one. +# TYPESAFE_API_KEY="" # BEDROCK_SMALL_MODEL="eu.anthropic.claude-haiku-4-5" # Codex agent settings (requires the [codex] extra: `uv sync --extra codex`). diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 502f7141d..e8d999ef6 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -488,16 +488,21 @@ jobs: AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }} AWS_REGION: ${{ secrets.AWS_REGION }} BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }} - # tasks_run for --tags smoke-pass. 8 task files (hello_date, dataset_example, - # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test, - # anti_cheat_reference, record_cli_responses); dataset_example fans out to 2 - # inline rows, so 9 sub-tasks. If you add/remove a smoke-pass task or change - # the dataset row count, bump these. + # Grades smoke_system_one_judge. Not a Bedrock credential: the System One + # judge calls TypeSafe directly and ignores the run's API backend. A + # missing key escalates that task to ERROR rather than failing a criterion, + # so the preflight step below fails loudly instead. + TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }} + # tasks_run for --tags smoke-pass. 9 task files (hello_date, dataset_example, + # smoke_llm_judge, smoke_agent_judge, smoke_system_one_judge, byod_smoke_test, + # agentless_smoke_test, anti_cheat_reference, record_cli_responses); + # dataset_example fans out to 2 inline rows, so 10 sub-tasks. If you + # add/remove a smoke-pass task or change the dataset row count, bump these. # # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not # match — the smoke-pass step names its path explicitly. Keep that in sync. - EXPECTED_SMOKE_PASS_RUN: "9" - EXPECTED_SMOKE_PASS_SUCCEEDED: "9" + EXPECTED_SMOKE_PASS_RUN: "10" + EXPECTED_SMOKE_PASS_SUCCEEDED: "10" # smoke-fail bucket: three tasks expected to fail. # 1. smoke_negative_path: file_contains criterion is unsatisfiable # (sentinel-string regression detection for success-checker). @@ -571,6 +576,15 @@ jobs: # record_cli_responses is the record_cli per-invocation-response probe and # is also driver: docker, so it needs that same image; it is flat in # tasks/, so the glob already matches it. + - name: Verify smoke secrets present + # smoke_system_one_judge grades through TypeSafe. Without the key the + # criterion raises JudgeInfrastructureError and the task lands in + # tasks_errored, which reads as "the harness broke" rather than "the + # secret is missing". Fail here, where the message says which. + run: | + : "${TYPESAFE_API_KEY:?TYPESAFE_API_KEY missing — needed by smoke_system_one_judge}" + echo "All smoke secrets present." + - name: Run smoke-pass bucket (expect all to succeed) run: | .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \ @@ -697,6 +711,8 @@ jobs: AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }} AWS_REGION: ${{ secrets.AWS_REGION }} BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }} + # The System One judge's own endpoint — unrelated to either route above. + TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }} steps: - name: Checkout code @@ -746,6 +762,7 @@ jobs: : "${AWS_BEARER_TOKEN_BEDROCK:?AWS_BEARER_TOKEN_BEDROCK missing}" : "${AWS_REGION:?AWS_REGION missing}" : "${BEDROCK_MODEL:?BEDROCK_MODEL missing}" + : "${TYPESAFE_API_KEY:?TYPESAFE_API_KEY missing}" echo "All live-test secrets present." - name: Run claude-settings enforcement live tests (DirectRoute) @@ -773,6 +790,17 @@ jobs: -m live -v --tb=short --strict-markers -ra -n 4 \ --junit-xml=tmp/junit-settings-bedrock.xml + - name: Run System One judge wire-contract live tests + # The only thing in CI that talks to TypeSafe. The unit tests mock the + # invoker, so nothing else catches a change to the answer shape the + # reduction assumes — notably the STRING level keys ("0", "1", ...) in a + # score answer's probabilities. `-n0`: three questions in one round trip, + # so there is nothing to parallelize. + run: | + .venv/bin/pytest tests/test_system_one_judge_live.py \ + -m live -n0 -v --tb=short --strict-markers -ra \ + --junit-xml=tmp/junit-system-one.xml + - name: Assert live tests actually ran (not silently skipped) # Parse JUnit XML for *passed* count, not collected count. Pytest collects # @pytest.mark.skipif-marked tests even when the predicate is True, so a @@ -798,11 +826,17 @@ jobs: return total - skipped - errors - failures p_settings_direct = passed("tmp/junit-settings.xml") p_settings_bedrock = passed("tmp/junit-settings-bedrock.xml") - print(f"Passed: settings(direct)={p_settings_direct}, settings(bedrock)={p_settings_bedrock}") + p_system_one = passed("tmp/junit-system-one.xml") + print( + f"Passed: settings(direct)={p_settings_direct}, " + f"settings(bedrock)={p_settings_bedrock}, system_one={p_system_one}" + ) if p_settings_direct < 1: sys.exit("test_claude_settings_enforcement_live.py (DirectRoute) reported zero PASSED tests") if p_settings_bedrock < 1: sys.exit("test_claude_settings_enforcement_live.py (BedrockRoute) reported zero PASSED tests") + if p_system_one < 1: + sys.exit("test_system_one_judge_live.py reported zero PASSED tests") PY - name: Run cost-budget smoke (max_usd → COST_BUDGET_EXCEEDED via DirectRoute) diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index 845f98d4d..70789dba4 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -35,6 +35,7 @@ Complete reference for defining evaluation tasks in Coder Eval. - [llm_judge](#llm_judge) - [agent_judge](#agent_judge) - [skill_triggered](#skill_triggered) + - [system_one_judge](#system_one_judge) - [Checker Context](#checker-context) - [Reference Solutions](#reference-solutions) - [Pre-Run Commands](#pre-run-commands) @@ -715,7 +716,7 @@ All criteria share these fields: **Scoring types:** - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `cli_called`, `classification_match`, `skill_triggered` - **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval` -- **Continuous** (0.0–1.0): `reference_comparison`, `commands_efficiency`, `llm_judge`, `agent_judge` +- **Continuous** (0.0–1.0): `reference_comparison`, `commands_efficiency`, `llm_judge`, `agent_judge`, `system_one_judge` **Task success:** all *gating* criteria must score >= their `pass_threshold`. A criterion with `weight: 0` is informational — it is still checked, stored, and @@ -1335,6 +1336,66 @@ Observed label is `"yes"` when either signal is found, else `"no"`. Expected lab **Typical pattern.** Label each dataset row with its true skill (`expected_skill`, `""` for negatives) and stack one `skill_triggered` criterion per skill against the same dataset — each gets its own confusion matrix from the same agent traces. This is the natural companion to a skill A/B experiment (skill plugin on vs. off); see the [A/B Experiment Guide](AB_EXPERIMENTS.md#recipe-ab-a-skill). +### `system_one_judge` + +Grade with a **System One model** ([TypeSafe's `jev`](https://docs.typesafe.ai/concepts/system-one)) instead of a text LLM. A System One model generates no text: it reads one state and answers a map of typed questions with calibrated probabilities, all in one round trip. The rubric you write **is** the grading schema, so there is no prompt to follow, no tool call to force, and no verdict to parse. + +```yaml +- type: "system_one_judge" + description: "Rubric grade of the refactor" + prompt: "The agent was asked to extract the retry loop into a helper." + files: ["src/client.py"] + questions: + helper_extracted: + type: noul + instructions: "Is the retry loop extracted into a named helper function?" + behaviour_preserved: + type: noul + instructions: "Does the refactor preserve the original retry semantics?" + weight: 2.0 + naming: + type: score + instructions: "How well does the helper's name describe what it does?" + criteria: ["opaque", "workable", "self-explanatory"] + leftovers: + type: noul + instructions: "Is any dead code left behind?" + expected: false +``` + +**Question types** + +| Type | What the model returns | How the rubric turns it into 0.0–1.0 | +| --- | --- | --- | +| `noul` | P(yes) | `expected: true` (default) scores P(yes); `expected: false` scores 1 − P(yes) | +| `choice` | the top option plus a distribution over all of them | `expected: