feat(criteria): add system_one_judge, a typed-rubric grader on a System One model (#192)

* feat(criteria): add system_one_judge, a typed-rubric grader on a System One model

A System One model (TypeSafe's jev) generates no text: it reads one state and
answers a map of typed questions with calibrated probabilities in a single round
trip. That removes the two things a text judge has to defend against — an
unparseable verdict and a tool call the model may decline to make — so the rubric
the author writes IS the grading schema.

The grade stays on our side: each question resolves to a value in [0,1] via its
own expected/values, and the criterion score is their weighted mean. That buys
determinism, a grade that can be recomputed from an archived transcript, and
findings that print the arithmetic per question instead of arguing it in prose.

A choice question's options may be written as a bare list when the names speak
for themselves; it widens to the option->description map the API wants, with null
descriptions, which the live endpoint accepts. Order is preserved and a repeated
option is a load-time error.

The judge reads the files it is given plus, opt-in per criterion, the agent's own
final message (include_agent_output), its tool-call trajectory
(include_tool_calls) and the simulated dialog (include_dialog) — so a rubric can
grade how the agent worked and whether its summary was honest, not only the
artifact left behind.

tasks/smoke_system_one_judge.yaml exercises all three primitives over all three
of those state slices, and runs in CI's smoke-pass bucket beside smoke_llm_judge.
tests/test_system_one_judge_live.py pins the external wire shape (notably the
STRING level keys in a score answer's probabilities) in the live-tests job, since
the unit tests mock the invoker and would keep agreeing with a stale contract.

Both need the TYPESAFE_API_KEY repo secret. A missing key escalates the row to
ERROR rather than scoring it 0.0, so each job preflights it and fails with a
message naming the key instead of reporting an unexplained tasks_errored.

Deliberately not shared with llm_judge: checker_context.api_route resolves a TEXT
judge model, which is not a substitute, and a transport failure escalates the row
rather than scoring an ungraded row 0.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(criteria): close scoring-integrity and credential-disclosure gaps in system_one_judge

Addresses PR review findings. Each was reproduced before it was fixed.

Scoring failed OPEN. json.loads accepts the NaN/Infinity tokens, so a non-finite
float arrives through an ordinary 200, and base_url may point at any gateway. NaN
is unordered, so max(0.0, min(1.0, nan)) is 1.0: a malformed answer graded as FULL
credit. An infinite question weight did the same through inf/inf in the weighted
mean. Every wire value is now checked with math.isfinite, _clamp maps a non-finite
input to 0.0, and weight is bounded and rejects inf/NaN. An unusable answer earns
no credit and says so in findings.

The detached-grading consent gate did not disclose this judge. embedded_commands()
had no system_one_judge arm, so `evaluate <run_dir>` on an untrusted run record
would read a host env var into a Bearer header and POST it to a recorded URL with
no prompt. Both halves are now disclosed by name.

The reference scrub ran after serialization. The state persists as json.dumps,
which escapes newlines, while scrub_reference matches raw file text — so every
multi-line reference survived into the archived transcript. Confirmed by a new
leak-canary test that failed before the fix. The state's strings are scrubbed
first, then serialized.

Also: a 200 with no usable `answers` escalates instead of scoring an ungraded row
0.0; argmax noul uses a strict > 0.5 so a coin flip no longer passes in both
directions; httpx2.InvalidURL escalates rather than escaping the retry loop to be
downgraded to 0.0; the bare-list choice form rejects non-string options, so YAML's
bare `yes`/`no` cannot become options named 'True'/'False'; the transcript records
the whole response, whose `model` field is the only record of which version a
floating alias actually served.

Parallel paths the change had missed: TYPESAFE_API_KEY joins the container env
allowlist (grading runs in-container, so a docker task ERRORed on every row), and
the $REFERENCE_DIR reference-consumer validator now covers this judge too.

Adds tests/test_judge_system_one.py — the retry/escalation gate had no hermetic
tests at all, since every checker test patches the invoker out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* style(criteria): use deferred annotations in system_one_judge so the TYPE_CHECKING import reads as used

CodeQL flagged `TurnRecord` as an unused import. It is not unused — it is used at
the `turn_records` parameter — but the annotation was a string literal, which
CodeQL does not resolve, so the only reference was invisible to it.

Rather than suppress the alert, adopt agent_judge.py's shape: `from __future__
import annotations` plus unquoted annotations. The names become real AST
references, so the import reads as used without deferring anything at runtime
that was not already deferred.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: akshaylive

.claude/notes/contracts.md

@@ -391,3 +391,69 @@ with a plain synchronous call, because offloading a single fast syscall only wid
 cancellation window. The cleanup itself is deliberately synchronous: a bare `await` inside
 `finally` is cancellable, and cancelling as that line is reached would skip cleanup and leak
 the copy with no reaper.
+
+## System One rubric scoring
+
+A System One model (TypeSafe's `jev`) does not generate text. It reads one state and answers
+a map of typed questions — `noul` (P(yes)), `choice` (a distribution over options), `score` (a
+distribution over ordered levels) — with calibrated probabilities. That inverts what a judge
+criterion has to defend against. There is no prose to parse, so the whole `submit_verdict`
+tool channel has no counterpart here: the response schema is fixed by the questions that were
+asked, and a model that answers off-schema is a wire fault, not a grading fault. It also means
+there is no system prompt to hold an identity, so the transcript's system-prompt slot carries
+the rubric as sent — that, not a persona, is what a reviewer needs to replay a grade.
+
+### The score is ours, not the model's
+
+The model reports a distribution; the grade is arithmetic we do on it. Each question resolves
+to a value in [0.0, 1.0] through the author's own `expected` / `values`, and the criterion
+score is their weighted mean. Keeping that reduction on our side buys three things a text
+judge cannot have: the same answers always produce the same grade, the arithmetic is printable
+(`findings` carries one line per question, with the value and the weight), and a rubric can be
+re-scored from an archived transcript without another call.
+
+`scoring: expected` weights every outcome by its probability, so a model that is genuinely
+torn lands mid-scale instead of being rounded into a confident-looking verdict — the point of
+a calibrated model. `argmax` exists for the case where partial credit is misleading rather
+than informative: a gate. The default is `expected` because discarding the confidence is the
+lossy choice and should be the one you ask for.
+
+A distribution that is absent, non-numeric or sums to zero falls back to the point answer
+(`choice` / `score` / `noul`) rather than grading as 0.0 — a broken `probabilities` block is a
+provider fault, and the point answer is still a real answer. A question that is *unanswered*,
+or answered with the wrong primitive, is the opposite case: it scores 0.0 at its full weight
+and names itself in `findings`, because the rubric asked something the grade depends on and
+dropping it would quietly inflate the mean.
+
+### What it does not share with `llm_judge`
+
+It does not read `checker_context.api_route`. The eval route resolves a TEXT judge model, and
+substituting one for a System One model is not a fallback, it is a different API. The
+credential comes from the env var *named* by `api_key_env`, so only the name is ever stored on
+the criterion or persisted into a run record. A transport failure raises
+`JudgeInfrastructureError` and escalates the row, rather than following `llm_judge`'s
+unconfigured-transport arm into a scored 0.0 — an ungraded row must not read as a failed one.
+
+A 200 response whose body carries no usable `answers` map is the same class of fault. Reducing
+it would score every question "no answer returned" and finalize the row as a graded 0.0, which
+reads as a failure the agent earned; it escalates instead.
+
+### Non-finite values fail CLOSED
+
+`json.loads` accepts the `NaN` and `Infinity` tokens, so a non-finite float arrives through an
+ordinary 200 — and `base_url` may point at any gateway. NaN is unordered, so the obvious clamp
+`max(0.0, min(1.0, x))` returns **1.0** for it: the naive reading of a malformed answer is FULL
+credit. Every wire value is therefore checked with `math.isfinite` before it is used (`_is_number`),
+`_clamp` maps a non-finite input to 0.0, and a question's `weight` is bounded and rejects inf/NaN
+so the weighted mean cannot become `inf/inf`. The rule is that an unusable answer never earns
+credit; it scores 0.0 and says so in `findings`.
+
+`argmax` uses a strict `> 0.5` for `noul`. At exactly 0.5 the agreement is 0.5 whichever way
+`expected` points, so `>=` passed a maximally uncertain answer in *both* directions.
+
+### The reference scrub runs before serialization
+
+`system_one_judge` persists its state as `json.dumps(state)`, which escapes newlines and tabs.
+`scrub_reference` matches raw file text, so scrubbing the *rendered* JSON silently misses every
+multi-line reference. The state's strings are scrubbed first, then serialized — a leak shipped
+exactly this way, so the leak-canary test uses a multi-line, quoted reference on purpose.

.env.example

@@ -31,6 +31,14 @@ LOG_TO_FILE=false  # Set to true to enable file logging
 # BEDROCK_MODEL="eu.anthropic.claude-sonnet-4-5-20250929-v1:0"
 # BEDROCK_SMALL_MODEL="eu.anthropic.claude-haiku-4-5"
 
+# TypeSafe System One (the system_one_judge criterion, model `jev`). NOT routed
+# through API_BACKEND — a System One model answers typed questions with
+# probabilities rather than text, so it is its own endpoint and its own key.
+# The criterion stores only this variable's NAME (api_key_env), never the token.
+# Unlike llm_judge, a missing key escalates the row to ERROR instead of scoring
+# it 0.0: an ungraded row must not read as a failed one.
+# TYPESAFE_API_KEY="<your_typesafe_api_key_here>"
+
 # Codex agent settings (requires the [codex] extra: `uv sync --extra codex`).
 # Only CODEX_API_KEY is read for auth (sent as a Bearer token). CODEX_BASE_URL
 # routes to a custom OpenAI-/responses-compatible endpoint; unset = standard

.github/workflows/pr-checks.yml

@@ -488,16 +488,21 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      # tasks_run for --tags smoke-pass. 8 task files (hello_date, dataset_example,
-      # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test,
-      # anti_cheat_reference, record_cli_responses); dataset_example fans out to 2
-      # inline rows, so 9 sub-tasks. If you add/remove a smoke-pass task or change
-      # the dataset row count, bump these.
+      # Grades smoke_system_one_judge. Not a Bedrock credential: the System One
+      # judge calls TypeSafe directly and ignores the run's API backend. A
+      # missing key escalates that task to ERROR rather than failing a criterion,
+      # so the preflight step below fails loudly instead.
+      TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+      # tasks_run for --tags smoke-pass. 9 task files (hello_date, dataset_example,
+      # smoke_llm_judge, smoke_agent_judge, smoke_system_one_judge, byod_smoke_test,
+      # agentless_smoke_test, anti_cheat_reference, record_cli_responses);
+      # dataset_example fans out to 2 inline rows, so 10 sub-tasks. If you
+      # add/remove a smoke-pass task or change the dataset row count, bump these.
       #
       # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not
       # match — the smoke-pass step names its path explicitly. Keep that in sync.
-      EXPECTED_SMOKE_PASS_RUN: "9"
-      EXPECTED_SMOKE_PASS_SUCCEEDED: "9"
+      EXPECTED_SMOKE_PASS_RUN: "10"
+      EXPECTED_SMOKE_PASS_SUCCEEDED: "10"
       # smoke-fail bucket: three tasks expected to fail.
       #   1. smoke_negative_path: file_contains criterion is unsatisfiable
       #      (sentinel-string regression detection for success-checker).
@@ -571,6 +576,15 @@ jobs:
       # record_cli_responses is the record_cli per-invocation-response probe and
       # is also driver: docker, so it needs that same image; it is flat in
       # tasks/, so the glob already matches it.
+      - name: Verify smoke secrets present
+        # smoke_system_one_judge grades through TypeSafe. Without the key the
+        # criterion raises JudgeInfrastructureError and the task lands in
+        # tasks_errored, which reads as "the harness broke" rather than "the
+        # secret is missing". Fail here, where the message says which.
+        run: |
+          : "${TYPESAFE_API_KEY:?TYPESAFE_API_KEY missing — needed by smoke_system_one_judge}"
+          echo "All smoke secrets present."
+
       - name: Run smoke-pass bucket (expect all to succeed)
         run: |
           .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \
@@ -697,6 +711,8 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
+      # The System One judge's own endpoint — unrelated to either route above.
+      TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
 
     steps:
       - name: Checkout code
@@ -746,6 +762,7 @@ jobs:
           : "${AWS_BEARER_TOKEN_BEDROCK:?AWS_BEARER_TOKEN_BEDROCK missing}"
           : "${AWS_REGION:?AWS_REGION missing}"
           : "${BEDROCK_MODEL:?BEDROCK_MODEL missing}"
+          : "${TYPESAFE_API_KEY:?TYPESAFE_API_KEY missing}"
           echo "All live-test secrets present."
 
       - name: Run claude-settings enforcement live tests (DirectRoute)
@@ -773,6 +790,17 @@ jobs:
             -m live -v --tb=short --strict-markers -ra -n 4 \
             --junit-xml=tmp/junit-settings-bedrock.xml
 
+      - name: Run System One judge wire-contract live tests
+        # The only thing in CI that talks to TypeSafe. The unit tests mock the
+        # invoker, so nothing else catches a change to the answer shape the
+        # reduction assumes — notably the STRING level keys ("0", "1", ...) in a
+        # score answer's probabilities. `-n0`: three questions in one round trip,
+        # so there is nothing to parallelize.
+        run: |
+          .venv/bin/pytest tests/test_system_one_judge_live.py \
+            -m live -n0 -v --tb=short --strict-markers -ra \
+            --junit-xml=tmp/junit-system-one.xml
+
       - name: Assert live tests actually ran (not silently skipped)
         # Parse JUnit XML for *passed* count, not collected count. Pytest collects
         # @pytest.mark.skipif-marked tests even when the predicate is True, so a
@@ -798,11 +826,17 @@ jobs:
               return total - skipped - errors - failures
           p_settings_direct = passed("tmp/junit-settings.xml")
           p_settings_bedrock = passed("tmp/junit-settings-bedrock.xml")
-          print(f"Passed: settings(direct)={p_settings_direct}, settings(bedrock)={p_settings_bedrock}")
+          p_system_one = passed("tmp/junit-system-one.xml")
+          print(
+              f"Passed: settings(direct)={p_settings_direct}, "
+              f"settings(bedrock)={p_settings_bedrock}, system_one={p_system_one}"
+          )
           if p_settings_direct < 1:
               sys.exit("test_claude_settings_enforcement_live.py (DirectRoute) reported zero PASSED tests")
           if p_settings_bedrock < 1:
               sys.exit("test_claude_settings_enforcement_live.py (BedrockRoute) reported zero PASSED tests")
+          if p_system_one < 1:
+              sys.exit("test_system_one_judge_live.py reported zero PASSED tests")
           PY
 
       - name: Run cost-budget smoke (max_usd → COST_BUDGET_EXCEEDED via DirectRoute)

docs/REPORT_SCHEMA.md

@@ -184,7 +184,7 @@ fields so subclass keys round-trip.
   `transcript_path` (a sibling `judge-N.yaml`, or `post-failure-judge-N.yaml` for
   diagnostic records). The full `transcript` is **stripped
   from `task.json`** — read it from the referenced file. Emitted by `llm_judge`,
-  `agent_judge`.
+  `agent_judge`, `system_one_judge`.
 
 ### Post-failure criterion evidence
 

docs/TASK_DEFINITION_GUIDE.md

@@ -35,6 +35,7 @@ Complete reference for defining evaluation tasks in Coder Eval.
   - [llm_judge](#llm_judge)
   - [agent_judge](#agent_judge)
   - [skill_triggered](#skill_triggered)
+  - [system_one_judge](#system_one_judge)
 - [Checker Context](#checker-context)
 - [Reference Solutions](#reference-solutions)
 - [Pre-Run Commands](#pre-run-commands)
@@ -715,7 +716,7 @@ All criteria share these fields:
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `cli_called`, `classification_match`, `skill_triggered`
 - **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval`
-- **Continuous** (0.0–1.0): `reference_comparison`, `commands_efficiency`, `llm_judge`, `agent_judge`
+- **Continuous** (0.0–1.0): `reference_comparison`, `commands_efficiency`, `llm_judge`, `agent_judge`, `system_one_judge`
 
 **Task success:** all *gating* criteria must score >= their `pass_threshold`. A
 criterion with `weight: 0` is informational — it is still checked, stored, and
@@ -1335,6 +1336,66 @@ Observed label is `"yes"` when either signal is found, else `"no"`. Expected lab
 
 **Typical pattern.** Label each dataset row with its true skill (`expected_skill`, `""` for negatives) and stack one `skill_triggered` criterion per skill against the same dataset — each gets its own confusion matrix from the same agent traces. This is the natural companion to a skill A/B experiment (skill plugin on vs. off); see the [A/B Experiment Guide](AB_EXPERIMENTS.md#recipe-ab-a-skill).
 
+### `system_one_judge`
+
+Grade with a **System One model** ([TypeSafe's `jev`](https://docs.typesafe.ai/concepts/system-one)) instead of a text LLM. A System One model generates no text: it reads one state and answers a map of typed questions with calibrated probabilities, all in one round trip. The rubric you write **is** the grading schema, so there is no prompt to follow, no tool call to force, and no verdict to parse.
+
+```yaml
+- type: "system_one_judge"
+  description: "Rubric grade of the refactor"
+  prompt: "The agent was asked to extract the retry loop into a helper."
+  files: ["src/client.py"]
+  questions:
+    helper_extracted:
+      type: noul
+      instructions: "Is the retry loop extracted into a named helper function?"
+    behaviour_preserved:
+      type: noul
+      instructions: "Does the refactor preserve the original retry semantics?"
+      weight: 2.0
+    naming:
+      type: score
+      instructions: "How well does the helper's name describe what it does?"
+      criteria: ["opaque", "workable", "self-explanatory"]
+    leftovers:
+      type: noul
+      instructions: "Is any dead code left behind?"
+      expected: false
+```
+
+**Question types**
+
+| Type | What the model returns | How the rubric turns it into 0.0–1.0 |
+| --- | --- | --- |
+| `noul` | P(yes) | `expected: true` (default) scores P(yes); `expected: false` scores 1 − P(yes) |
+| `choice` | the top option plus a distribution over all of them | `expected: <option>` scores that one option 1.0; `values: {option: 0.0–1.0}` gives partial credit per option |
+| `score` | a position on an ordered spectrum, plus a distribution over levels | levels ramp evenly from 0.0 (first) to 1.0 (last) unless `values:` overrides them |
+
+`choice` needs exactly one of `expected` or `values`. `score` takes 2–10 levels, ordered worst-first. Every question takes a `weight` (default 1.0).
+
+A `choice` question's `criteria` is a map of option to a description of when it applies, but when the option names speak for themselves you can write a bare list instead — it widens to that map with null descriptions, which the API accepts:
+
+```yaml
+exception_handling:
+  type: choice
+  instructions: "How does the function catch failures from requests.get?"
+  criteria: [none, bare_except, broad_exception, specific_timeout]
+  expected: specific_timeout
+  weight: 2.0
+```
+
+Option order is preserved as written, and a repeated option is a load-time error rather than a silently collapsed map.
+
+**What the judge reads.** By default the state is `prompt` plus the `files` you list. Three flags widen it, each off by default: `include_agent_output` adds the agent's own final message, `include_tool_calls` adds a summary of its tool-call trajectory, and `include_dialog` adds the multi-turn user/agent exchange (only meaningful in [simulation mode](#simulation)). Turning the first two on is what lets a rubric grade *how* the agent worked and whether its summary was honest, not just the artifact it left behind — see `tasks/smoke_system_one_judge.yaml` for a rubric that does both. `include_reference` (on by default) adds the reference solution. Each section is capped independently by `max_state_chars`.
+
+**Scoring** — the criterion score is computed by the harness, not the model: each question resolves to a value in [0.0, 1.0] and the score is their weighted mean. `scoring: expected` (default) weights every outcome by its probability, so a half-confident answer lands mid-scale; `scoring: argmax` reads only the top answer and discards the confidence. Either way the reduction is deterministic given the answers, and `findings` records the arithmetic per question so the grade is auditable line by line.
+
+**Credentials** — the bearer token comes from the env var named by `api_key_env` (default `TYPESAFE_API_KEY`); only the *name* is stored in the task and in run records. `base_url` (default `https://api.typesafe.ai/v1`) points the criterion at a gateway or a recording proxy. Unlike `llm_judge`, this criterion does **not** honour `checker_context.api_route` — a System One model is not interchangeable with a text model, so the eval route's judge model would be the wrong default.
+
+**When to reach for it over `llm_judge`** — a rubric with many small, repeated questions; a large dataset where a text judge's per-row cost dominates; or a grade you need to be reproducible and inspectable rather than argued in prose. Reach for `llm_judge` instead when the grade genuinely needs open-ended reasoning you cannot enumerate in advance.
+
+**Failure modes** — a transport failure escalates the row to `ERROR` (it is eval infrastructure, not agent quality) rather than scoring 0.0. A question the API leaves unanswered, or answers with the wrong primitive, scores 0.0 at its full weight and says so in `findings`.
+
 ## Checker Context
 
 `checker_context` carries task-authored config for the success-checking side, namespaced by reserved key. Currently the only recognized namespace is **`api_route`**:

evalboard/lib/pricing.generated.ts

@@ -69,6 +69,8 @@ export const PRICING: Record<string, Pricing> = {
     "gpt-5.6-luna": { inputPerMTok: 0.2, outputPerMTok: 1.2, cacheWritePerMTok: 0.2, cacheReadPerMTok: 0.02 },
     "gpt-5.6-sol": { inputPerMTok: 4.0, outputPerMTok: 20.0, cacheWritePerMTok: 4.0, cacheReadPerMTok: 0.4 },
     "gpt-5.6-terra": { inputPerMTok: 2.0, outputPerMTok: 12.0, cacheWritePerMTok: 2.0, cacheReadPerMTok: 0.2 },
+    "jev-1.13.0": { inputPerMTok: 0.042, outputPerMTok: 0.0, cacheWritePerMTok: 0.042, cacheReadPerMTok: 0.0 },
+    "jev-latest": { inputPerMTok: 0.042, outputPerMTok: 0.0, cacheWritePerMTok: 0.042, cacheReadPerMTok: 0.0 },
     "kimi-k2-7-code": { inputPerMTok: 0.95, outputPerMTok: 4.0, cacheWritePerMTok: 0.0, cacheReadPerMTok: 0.19 },
     "moonshotai.kimi-k2.5": { inputPerMTok: 0.72, outputPerMTok: 3.6, cacheWritePerMTok: 0.72, cacheReadPerMTok: 0.0 },
     "virtuoso-1-5": { inputPerMTok: 0.95, outputPerMTok: 4.0, cacheWritePerMTok: 0.0, cacheReadPerMTok: 0.16 },