UiPath
diff --git a/‎.claude/harness-candidates.md‎
Lines changed: 15 additions & 0 deletions b/‎.claude/harness-candidates.md‎
Lines changed: 15 additions & 0 deletions
diff --git a/‎.claude/shared/run-layout.md‎
Lines changed: 47 additions & 0 deletions b/‎.claude/shared/run-layout.md‎
Lines changed: 47 additions & 0 deletions
diff --git a/‎CLAUDE.md‎
Lines changed: 2 additions & 2 deletions b/‎CLAUDE.md‎
Lines changed: 2 additions & 2 deletions
diff --git a/‎README.md‎
Lines changed: 3 additions & 3 deletions b/‎README.md‎
Lines changed: 3 additions & 3 deletions
@@ -455,3 +455,18 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+## From the split-field / optimize-skill plan (2026-08-12)
+
+- [ ] **A run whose every task is skipped exits 0 — a green run of zero tasks.** When
+  `resolve_all_tasks` demotes every task to `skipped_tasks` (a load failure, `skip: true`,
+  or now a `--split` selector matching no labelled row), the run reports success: nothing
+  failed, so the exit gate in `cli/run_command.py` — which keys only on failed/errored tasks
+  and suite gates — passes. Verified directly: `coder-eval run <suite> --split holdou`
+  prints one yellow "1 task file(s) skipped" line and exits 0. This is pre-existing, but
+  `--split` makes it reachable by a one-character CLI typo rather than a broken file, and
+  the whole point of a holdout confirmation is that you trust its verdict. Not guarded, and
+  not a five-minute fix: making an all-skipped run non-green changes exit semantics for
+  every skipped-task path (including deliberate `skip: true` suites and tag filters that
+  match nothing), so it needs a decision about which of those should be fatal, plus tests
+  per case. A narrower option is to fail only when a CLI *selector* (`--split`, `--tags`)
+  eliminated everything, since that is unambiguously a user error rather than repo state.
@@ -14,6 +14,53 @@ runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}
 - `task.json.malformed` — present only on the docker degrade path: when an existing `task.json` fails to parse (schema skew from a stale `:latest` image, or a truncated/torn write), the docker runner moves the unparseable original aside to this sidecar and writes a synthetic `final_status=ERROR` `task.json` in its place. Diagnostic-only; `rglob("task.json")` consumers do not match it.
 - `task.log` — the human-readable task log; `artifacts/` — files the agent produced.
 
+## Suite rollups (dataset-backed tasks only)
+
+A task carrying `dataset:` fans out into one row-task per row and additionally writes a
+per-suite rollup:
+
+```
+runs/<run_id>/<variant_id>/<suite_id>/{suite.json, suite.md}
+```
+
+`<suite_id>` is the original (pre-fan-out) `task_id`. Nothing is written for a task
+without `dataset:`.
+
+`suite.json` carries the suite's pass counts plus `criterion_aggregates[]` — one entry per
+criterion that opted into across-row aggregation, each with:
+
+- `criterion_type`, and `description` (set when a task stacks several criteria of the same
+  type, e.g. one `skill_triggered` per skill — that is what distinguishes them);
+- `rows_total` and `rows_excluded` — the denominator and what was dropped from it. A row
+  that errored before criteria ran (a timeout, say) is **excluded** rather than scored, so
+  metrics are computed over `rows_total - rows_excluded`;
+- `metrics` — a **flat** name → float map. Classification-style criteria emit
+  `accuracy`, `macro_f1`, and per-label `precision.<label>` / `recall.<label>` /
+  `f1.<label>` (for `skill_triggered` the labels are `yes` / `no`). It also carries
+  `completion_rate` — the surviving fraction of the denominator above — which is an
+  ordinary metric, so `suite_thresholds: {completion_rate: 1.0}` gates a run whose sample
+  eroded;
+- `details` — `labels`, `per_label`, `confusion`, `total_pairs`;
+- `threshold_checks` + `passed`, from the criterion's `suite_thresholds`.
+
+Alongside the aggregates, `failed_samples[]` lists failed/errored rows **by `row_id`** with
+their failure reasons, capped at a fixed limit. It is the only place in `suite.json` that
+carries row identity — `metrics` and `details` are counts only — so a consumer that needs
+to know *which* rows failed reads `failed_samples`, then falls back to the per-row
+`<variant_id>/<suite_id>/<row_id>/<NN>/task.json` for anything past the cap.
+
+**Read `metrics`; never recompute a metric from `details.confusion`.** The criterion layer
+owns that arithmetic, including its division-by-zero convention, and a consumer that
+re-derives F1 will disagree with the gate the run already applied.
+
+**Rollups pool replicates.** The grouping key is `(variant_id, suite_id)` — the replicate
+index is *not* part of it. So `--repeats N` over an M-row suite writes **one** `suite.json`
+per variant, with `rows_total: N × M` and a single pooled confusion matrix. There is no
+per-replicate metric in that file. A consumer that needs replicate-to-replicate spread must
+invoke the run N times and read N run directories; `--repeats` cannot serve that purpose.
+(Contrast `experiment.md`'s `## Paired Comparison` block, which averages replicates per row
+before pairing — there `--repeats` is exactly the right tool.)
+
 **Scope-marker files** (used to detect what a given path represents):
 
 - `run.json` at the run root → **run scope**. If `experiment.json` (+ `experiment.md`) is also present → multi-variant experiment.
 
@@ -124,7 +124,7 @@ tests/                             # Test suite
 docs/                              # Documentation
 templates/                         # Sandbox template directories
 .claude-plugin/marketplace.json    # Makes this repo a Claude Code plugin marketplace (`/plugin marketplace add UiPath/coder_eval`); lists the one plugin below.
-plugins/coder-eval/                # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills/<name>/SKILL.md` × 6 (`/coder-eval:init`, `/coder-eval:check-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads (`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct from `run-layout.md`, which describes what is inside a run directory. Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. **Skill naming is verb-first imperative** — a skill is a command you issue (`/coder-eval:<name>`) and every one of them takes an action, so name it for the action: a bare verb where that is unambiguous (`init`, `analyze` — the object comes from the argument), otherwise `<verb>-<object>` (`lint-tasks`, `check-skill`). Never `<object>-<verb>`: `skill-check` was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`. `task` and `ci` predate the rule and stay — renaming a published skill breaks every user's muscle memory for no functional gain, since activation keys on the `description`, never the name. Distinct from `.claude/commands/`, which stays repo-local contributor tooling.
+plugins/coder-eval/                # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills/<name>/SKILL.md` × 7 (`/coder-eval:init`, `/coder-eval:check-skill`, `/coder-eval:optimize-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads (`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct from `run-layout.md`, which describes what is inside a run directory. Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. **Skill naming is verb-first imperative** — a skill is a command you issue (`/coder-eval:<name>`) and every one of them takes an action, so name it for the action: a bare verb where that is unambiguous (`init`, `analyze` — the object comes from the argument), otherwise `<verb>-<object>` (`lint-tasks`, `check-skill`, `optimize-skill`). Never `<object>-<verb>`: `skill-check` was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`, and `optimize-skill` was named that way rather than `skill-optimize` for the same reason — the rule is applied at authoring time now, not repaired later. `task` and `ci` predate the rule and stay — renaming a published skill breaks every user's muscle memory for no functional gain, since activation keys on the `description`, never the name. Distinct from `.claude/commands/`, which stays repo-local contributor tooling.
 action.yml                         # Published composite GitHub Action (coder-eval as a CI gate). release.yml's `release` job maintains its `version:` default; its `promote` job (gated on publish-pypi) moves the `v<major>` tag + cuts the Release, so nothing consumer-visible moves before the wheel is on PyPI. verify-published-action.yml then verifies the published composite (tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and nightly. Runbook: CONTRIBUTING.md § Releasing.
 ```
 
@@ -139,7 +139,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Single declarative merge resolver**: All five config layers merge through ONE engine (`orchestration/config_merge.py::resolve_root`) for the three `-D`-reachable roots (`agent`/`run_limits`/`sandbox`). Each field declares *how it merges* once, on the model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` lists and call the same `resolve_root`, so a field merges identically regardless of which layer supplied it (the unification invariant, enforced by `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare its strategy explicitly.
 - **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved `TaskDefinition` (`agent`/`run_limits`/`sandbox` roots), schema-validated with did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, lighter alias that does NOT route through that collision check — `--type` and `-D agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, and SDK options are `-D`-only.
 - **All core models importable from `coder_eval.models`** regardless of submodule
-- **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.<field>}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.)
+- **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.<field>}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row selection is filter-then-sample: CLI `--split <name>` (keep only rows whose `dataset.split_field` value matches, default field `split`) runs **first** and is orthogonal to the sampler win-order — a row is unlabelled when the field is absent/`null`/`""` and a task whose rows are all unlabelled passes through unfiltered; partial labelling drops the unlabelled rows. A *labelled* task with no matching row raises a `ValueError` listing the splits that exist — which `resolve_all_tasks` catches into `skipped_tasks` like any load failure, so a mistyped selector yields a zero-task run that still exits 0. Filtering before sampling is a correctness requirement: sampling first would leave an unpredictable (possibly zero) number of rows per split, destroying the tune/holdout comparison. Then sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.)
 - **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure.
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 
@@ -107,9 +107,9 @@ results — runs inside the agent:
 /plugin install coder-eval@coder-eval
 ```
 
-That adds six slash commands: `/coder-eval:init`, `/coder-eval:check-skill`,
-`/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze` and
-`/coder-eval:ci`. They drive the `coder-eval` CLI, so install it too
+That adds seven slash commands: `/coder-eval:init`, `/coder-eval:check-skill`,
+`/coder-eval:optimize-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`,
+`/coder-eval:analyze` and `/coder-eval:ci`. They drive the `coder-eval` CLI, so install it too
 (`uv tool install coder-eval`). See [Claude Code Plugin](docs/PLUGIN.md).
 
 ## Use as a GitHub Action