feat(lint): replace the prose baseline with two self-adjusting rules

_ESSAY_BASELINE_WORDS was the weakest part of the gate. It was a single
tree-wide number, hand-edited eleven times in this work alone, that a reviewer
had to take on trust; it let growth in one file hide behind shrinkage in
another; and it said nothing at all about a file that did not exist yet.

It is gone. Two rules replace it, and neither has a number anyone maintains:

  own-line comments per file <= MAX(20, 0.15 * file length)
  no docstring over 150 words of PROSE

The comment budget is proportional, so deleting code takes its budget with it
and a new file is governed from its first commit. Own-line only: a trailing
`# noqa` is a directive and a per-member annotation on an enum is the contract
a dispatcher reads — counting either would push against documenting them. The
floor is what protects a constants module at one comment per constant, which is
where the tree's natural maximum sits.

The essay rule now counts prose, not structure: an Args/Returns/Raises block is
interface documentation, and counting it pushed exactly the docstrings that
document their contract best over the line. Exemptions are categorical rather
than numeric — an @AbstractMethod docstring IS the contract implementers read,
so the plugin SPI is covered by kind, and a new abstract method is covered
automatically. The old "at most two, by fiat" allowance is unnecessary: the
tree now has ZERO essays.

Getting there took 44 comment lines out of 7 files. Most came out by reflowing
two lines into one or cutting a section divider; the one real compression was a
22-line block in early_stop.py restating the floor bound that
orchestration.md already owns, and the distractor-exclusion rule it uniquely
held moved there rather than being dropped.

One reflow silently merged `# pyright: reportImportCycles=false` into the prose
line above it, which would have stopped pyright honouring it.
--assert-code-unchanged caught it. That is the second time the directive
multiset has earned its place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: uipreliga

.claude/notes/orchestration.md

@@ -236,7 +236,12 @@ trajectory continues. A `decide_within` timeout participates as an ordinary weig
 so a low-weight criterion's timeout that cannot doom the gate does not stop the run.
 
 A **pass-stop** fires once the `on_pass: stop` subset's FLOOR — worst case, every
-still-undecided member scores 0 — already meets the threshold.
+still-undecided member scores 0 — already meets the threshold. Criteria armed only on the
+FAIL side (distractors) are excluded from both the numerator and the denominator of that
+bound: they can never live-pass and exist only to guard the fail side, so folding them in
+would veto every pass-stop and penalise the bound for a criterion it was never scoped to
+cover. With no `on_pass: stop` criteria at all the bound is vacuous and returns nothing —
+there is no pass-stop to take, and the run continues to the cap.
 
 At the default threshold both bounds collapse exactly to "any single armed criterion's
 effective fail stops the run" and "every `on_pass: stop` criterion has live-passed".

CLAUDE.md

@@ -208,7 +208,7 @@ make evalboard-verify   # the JS half: tsc --noEmit + vitest + next build
 make docs-indexes      # README/docs index tables from the mkdocs nav (CE028)
 make plugin-reference  # the plugin's criteria reference from the models (CE033)
 
-make docs-budget       # prose budget report; fails `make verify` if the total grows
+make docs-budget       # per-file comment budget + docstring essay check (fails `make verify`)
 ```
 
 Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is
@@ -361,8 +361,11 @@ bandit, pre-commit, mcp
   code cannot say
 - **A docstring states the contract, not the history** — what a caller must know to call
   it correctly. Why the design is this shape belongs in `.claude/notes/`; what it used to
-  be belongs in git. `make docs-budget` reports the standing total and fails
-  `make verify` if it grows
+  be belongs in git. `make docs-budget` enforces two rules, both self-adjusting: a file's
+  own-line comments may not exceed `MAX(20, 0.15 × its length)`, and no docstring may
+  exceed 150 words of PROSE (an `Args:`/`Returns:`/`Raises:` block is structure, not
+  prose; an `@abstractmethod` is exempt because its docstring IS the interface contract).
+  There is no tree-wide total to hand-maintain — delete code and the budget shrinks with it
 
 ## Notes for AI Assistants
 

src/coder_eval/config.py

@@ -26,11 +26,9 @@
 ).decode("utf-8")
 
 
-# Load .env file with override so .env values always win over shell environment
+# override=True so .env always wins over the shell's possibly-stale credentials.
 load_dotenv(override=True)
 
-# For certain keys, we want .env values to take precedence over shell environment
-# because the shell may have outdated/different credentials
 env_values = dotenv_values(".env")
 for key in [
     "ANTHROPIC_API_KEY",
@@ -79,52 +77,43 @@ def __init__(self, *args: Any, **kwargs: Any) -> None:
         _reject_removed_default_knobs()
         super().__init__(*args, **kwargs)
 
-    # API Keys (for Claude Code agent only)
     anthropic_api_key: str | None = None
 
-    # Paths
     runs_dir: Path = Path("runs")  # Base directory for timestamped runs
 
-    # API Backend routing
     api_backend: ApiBackend = ApiBackend.DIRECT
 
-    # AWS Bedrock settings (used when api_backend == "bedrock")
     aws_bearer_token_bedrock: str | None = None
     aws_region: str | None = None
     bedrock_model: str | None = None  # Cross-region model ID
     bedrock_small_model: str | None = None  # Cross-region small model ID
 
-    # HAZARD: these map to the ANTHROPIC_* vars, but ONLY inside the SDK subprocess
-    # env. Deliberately NOT named anthropic_*, so the export loop below cannot leak
-    # ANTHROPIC_BASE_URL process-wide and redirect the judge's own client.
+    # HAZARD: these map to the ANTHROPIC_* vars ONLY inside the SDK subprocess env.
+    # NOT named anthropic_*, so the export loop cannot leak ANTHROPIC_BASE_URL
+    # process-wide and redirect the judge's own client.
     litellm_base_url: str | None = None
     litellm_auth_token: str | None = None
     litellm_model: str | None = None
     litellm_small_model: str | None = None
-    # Must point at the SAME file the proxy writes. When set and present, the harness
-    # joins each call's ACTUAL cost onto the turn; unset or missing => static pricing.
+    # Must point at the SAME file the proxy writes; unset or missing => static pricing.
     # Rationale: .claude/notes/reporting.md § Cost joining
     litellm_cost_log: str | None = None
 
     # CODEX_MODEL is the fallback when a task doesn't pin agent.model. For Azure set
-    # CODEX_API_VERSION too and use the deployment name as the model. CODEX_BASE_URL
-    # / CODEX_API_VERSION / CODEX_API_KEY are read via os.getenv in the agent.
+    # CODEX_API_VERSION too and use the deployment name as the model.
     codex_model: str | None = None
 
     # GEMINI_API_KEY is read from .env here so the export loop re-publishes it to
     # os.environ, where the SDK looks for it. ANTIGRAVITY_MODEL is the fallback.
     gemini_api_key: str | None = None
     antigravity_model: str | None = None
 
-    # Logging
     log_level: str = "INFO"  # Default log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
     log_to_file: bool = False  # Whether to enable file logging
 
-    # On by default via the baked-in connection string. TELEMETRY_ENABLED is the
-    # single canonical disable gate.
+    # On by default via the baked-in connection string, which any set value (env
+    # or .env) overrides. TELEMETRY_ENABLED is the single canonical disable gate.
     telemetry_enabled: bool = True
-    # Defaults to the embedded coder-eval resource; any set value (env or .env, via
-    # the aliases below) overrides it — pydantic-settings prefers env over default.
     telemetry_connection_string: str | None = Field(
         default=_DEFAULT_TELEMETRY_CONNECTION_STRING,
         validation_alias=AliasChoices(
@@ -180,8 +169,7 @@ def _validate_litellm_settings(self) -> None:
                 f"LiteLLM-endpoint routing is enabled but missing required settings: {', '.join(missing)}."
                 + " Please set them in your .env file."
             )
-        # Reject a malformed base_url here so the downstream preflight and
-        # environment_info get a well-formed absolute URL.
+        # Reject a malformed base_url so the preflight and environment_info get a well-formed URL.
         parts = urlsplit(self.litellm_base_url or "")
         if parts.scheme not in ("http", "https") or not parts.hostname:
             raise ValueError(
@@ -215,15 +203,12 @@ def validate_api_keys(self, agent_type: str) -> None:
             return
 
 
-# Global settings instance
 settings = Settings()
 
-# For external libraries that read os.getenv() rather than the Settings object.
-# Non-None values only, stringified.
+# For external libraries that read os.getenv(); non-None values only, stringified.
 for key, value in settings.model_dump().items():
     if value is not None:
         env_key = key.upper()
-        # Convert Path objects and other types to strings
         if isinstance(value, Path):
             os.environ[env_key] = str(value)
         elif isinstance(value, bool):

src/coder_eval/criteria/agent_judge.py

@@ -44,8 +44,7 @@
     path_uses_token,
 )
 
-# Not part of the public coder_eval.models surface, but the single source of truth
-# for both files.
+# Not part of the public coder_eval.models surface, but the single source of truth for both files.
 from coder_eval.models.criteria import (  # noqa: CE001
     JUDGE_SECURITY_IGNORE_FLOOR,
     _default_judge_agent_config,

src/coder_eval/evaluation/sub_agent.py

@@ -88,8 +88,7 @@ def __init__(
         # drop a user's own nested subdir of the same name.
         # Rationale: .claude/notes/contracts.md § The security floor
         self._reference_ignore_patterns = reference_ignore_patterns or []
-        # Runtime-only in-process MCP injection. NOT routed through ``sdk_options``
-        # -- ``mcp_servers`` is framework-owned.
+        # Runtime-only MCP injection, NOT via ``sdk_options`` -- ``mcp_servers`` is framework-owned.
         self._extra_mcp_servers = extra_mcp_servers or {}
         # Public so the criterion can read it after ``run_async()`` returns; absent
         # when the caller passed ``capture=None``.

src/coder_eval/isolation/docker_runner.py

@@ -653,8 +653,7 @@ async def run(self) -> EvaluationResult:
                 widened_workspace = await asyncio.to_thread(grant_container_access, self.grade_workspace, writable=True)
             argv = self._build_argv(input_dir, output_dir, container_name=container_name, image=image)
             logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv))
-            # Prime the heartbeat before the container starts so the
-            # watchdog never sees an initial stale state.
+            # Prime the heartbeat before the container starts so the watchdog never sees an initial stale state.
             heartbeat_path = output_dir / HEARTBEAT_FILENAME
             await asyncio.to_thread(heartbeat_path.touch)
             heartbeat_task = asyncio.create_task(_heartbeat_loop(heartbeat_path))
@@ -673,13 +672,11 @@ async def run(self) -> EvaluationResult:
                 returncode = await self._stream_container_output(proc, log_fh)
             finally:
                 heartbeat_task.cancel()
-                # Narrowed so a genuine KeyboardInterrupt / SystemExit from a
-                # parallel sibling still propagates.
+                # Narrowed so a genuine KeyboardInterrupt / SystemExit from a parallel sibling still propagates.
                 with contextlib.suppress(asyncio.CancelledError):
                     await heartbeat_task
                 await asyncio.to_thread(log_fh.close)
-                # Cancelled mid-flight: kill the container AND the docker CLI
-                # subprocess, best-effort.
+                # Cancelled mid-flight: kill the container AND the docker CLI subprocess, best-effort.
                 if proc.returncode is None:
                     await self._kill_container(proc, container_name)
 
@@ -906,8 +903,7 @@ def _assert_grade_honored(self, result: EvaluationResult, task_json: Path | None
         """
         if self.grade:
             return
-        # Keyed on EVIDENCE, not on the label: the question is not "what status is
-        # this" but "did it grade".
+        # Keyed on EVIDENCE, not on the label: the question is not "what status is this" but "did it grade".
         graded_anyway = bool(result.success_criteria_results) or result.weighted_score is not None
         if not graded_anyway and (
             result.final_status.is_execution_fact or result.final_status is FinalStatus.NOT_GRADED
@@ -1108,8 +1104,7 @@ def _prepare_task_dir_mount(self, staging: Path) -> None:
             return
         task_dir_copy = staging / "task_dir"
         shutil.copytree(source, task_dir_copy, ignore=ignore_patterns_and_symlinks(REFERENCE_COPY_IGNORE))
-        # Read-only like the reference copy: criteria read fixtures here, nothing
-        # legitimately writes them.
+        # Read-only like the reference copy: criteria read fixtures here, nothing legitimately writes them.
         grant_container_access(task_dir_copy, writable=False)
         self._task_dir_mount_src = task_dir_copy
 

src/coder_eval/orchestration/config.py

@@ -69,7 +69,6 @@ class BatchRunConfig(BaseModel):
         ),
     )
 
-    # Dataset sampling (for cheap smoke runs on dataset-backed tasks)
     max_rows: int | None = Field(
         default=None,
         ge=1,
@@ -85,7 +84,6 @@ class BatchRunConfig(BaseModel):
         ),
     )
 
-    # Replicate count override
     repeats: int | None = Field(
         default=None,
         ge=1,
@@ -104,7 +102,6 @@ class BatchRunConfig(BaseModel):
         ),
     )
 
-    # Logging
     verbose: bool = Field(default=False, description="Enable verbose (DEBUG level) logging for Docker output")
 
     # Docker WORKDIR alignment for the NON-docker-driver dispatch path — a host
@@ -121,15 +118,8 @@ class BatchRunConfig(BaseModel):
         ),
     )
 
-    # TODO(container-death-diagnostics): consider a run-level default resource
-    # cap. Containers run uncapped today (sandbox.limits.{max_memory_mb,
-    # max_cpus,max_pids} default to None -> _build_argv emits no --memory/
-    # --cpus/--pids-limit), so at --max-parallel=20 a single runaway task can
-    # pressure the whole host. An opt-in default cap is already expressible
-    # via the EXISTING layered sandbox config -- defaults.sandbox.limits.
-    # max_memory_mb in the experiment YAML, or `-D sandbox.limits.
-    # max_memory_mb=N` on `coder-eval run` -- both flow through
-    # resolve_all_tasks and are overridden by per-task limits. If a dedicated
-    # CLI knob is ever wanted, add it as the FIRST (lowest-priority) layer in
-    # _build_sandbox_layers so per-task limits win, and do NOT default it to
-    # a non-None value (would change behavior for existing configs).
+    # TODO(container-death-diagnostics): containers run uncapped today, so at a
+    # high --max-parallel one runaway task can pressure the host. An opt-in
+    # default is already expressible through the layered sandbox config; a
+    # dedicated CLI knob would go in as the LOWEST-priority layer, never
+    # defaulted to a value (that would change existing configs).

src/coder_eval/orchestration/early_stop.py

@@ -284,8 +284,6 @@ def for_task(cls, task: TaskDefinition) -> EarlyStopWatcher:
         )
         return cls(task.task_id, armed, max_turns=max_turns, gate_threshold=gate_threshold)
 
-    # --- StreamCallback -------------------------------------------------- #
-
     def on_event(self, event: StreamEvent) -> None:
         """Fail-open wrapper around ``_on_event_impl``: any unexpected exception
         anywhere in the round — the collector reduction included, not just the
@@ -361,8 +359,6 @@ def disarmed(self) -> bool:
         """True once a ``live_verdict`` raised and the watcher degraded to a full run."""
         return self._disarmed
 
-    # --- Stop rule -------------------------------------------------- #
-
     def _ceiling(self, verdicts: list[LiveVerdict]) -> float:
         """Best-case weighted score over the WHOLE armed set, given current verdicts.
 
@@ -502,28 +498,12 @@ def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None:
                 self._fire(reason, self._armed[candidate_index][0], tool_call_index=tool_call_index)
                 return
 
-        # Pass-stop: the on_pass=stop subset's own floor bound (worst case:
-        # every still-undecided member scores 0, weighted against only that
-        # subset's total weight) already meets ``gate_threshold`` — guaranteed
-        # regardless of what the rest of that subset still decides. Criteria
-        # armed only on the fail side (distractors) are excluded from both the
-        # numerator and the denominator: they can never live-pass and only
-        # guard the fail side above, so folding them in would veto every
-        # pass-stop and penalize this bound for a criterion it was never
-        # scoped to cover. At the default ``gate_threshold=1.0`` this requires
-        # every on_pass=stop criterion to actually be "pass" (any non-pass
-        # drops the floor below 1.0). The vacuous case (no on_pass=stop
-        # criteria at all) returns None — nothing to pass-stop on, the run
-        # continues to the cap.
-        #
-        # Recall deferral, mirrored from the fail-stop: the pass-stop is HELD
-        # while any pass-capable armed criterion OUTSIDE the on_pass=stop
-        # subset is still undecided (members of the subset are already priced
-        # into the floor). Cutting here would freeze a sibling
-        # ``on_pass: continue`` criterion's expected signal out of the
-        # trajectory — an unearned fail on the armed gate that a full run
-        # would not have produced. Once every such criterion decides (pass or
-        # fail), the still-satisfied floor fires the pass-stop on that round.
+        # Pass-stop: the on_pass=stop subset's FLOOR already meets
+        # ``gate_threshold``. Distractors are excluded from both the numerator and
+        # the denominator; no on_pass=stop criteria at all returns None. HELD while
+        # any pass-capable armed criterion OUTSIDE the subset is undecided --
+        # cutting there would freeze a sibling's expected signal out of the run.
+        # Rationale: .claude/notes/orchestration.md § The ceiling and floor bounds
         pass_stop_indices = [i for i, armed_pass in enumerate(self._pass_trigger) if armed_pass]
         outside_pass_capable_undecided = any(
             v == "undecided" and "pass" in pol and not armed_pass

src/coder_eval/path_utils.py

@@ -56,10 +56,9 @@ def write_text_atomic(path: Path, text: str) -> None:
     """Write ``text`` to ``path`` via a temp file + ``os.replace``.
 
     A plain ``write_text`` truncates first, so a crash mid-write leaves a
-    half-file — and a truncated ``task.json`` parses as *malformed*, which the
-    recovery paths read as "not complete", so ``--resume`` pays for the agent
-    again. One writer, so every producer of that file has the same crash
-    semantics.
+    half-file — and a truncated ``task.json`` parses as *malformed*, which
+    ``--resume`` reads as "not complete" and pays for the agent again. One
+    writer, so every producer has the same crash semantics.
 
     The temp file is opened ``O_CREAT | O_EXCL | O_NOFOLLOW`` under a name that
     is UNIQUE per call. ``O_NOFOLLOW`` closes a symlink-plant overwrite