fix(trg): stop held-out cases from leaking and eval gates from failing open

crates/trg/docs/how-to/hillclimb-a-skill.md

@@ -18,6 +18,8 @@ split's verdict is `keep`; revert on `revert`, `suspected_overfitting`, or
 `inconclusive`. Never read a test-split run's transcript, prompt, or grading
 detail while deciding what to do next; only its aggregate pass rate, through
 `keep_or_revert` or an improvement bundle's `held_out` field, is fair game.
+Pass `--withhold-test-detail` to `iteration-summary` so that rule is enforced
+by the command's own output rather than by what you choose not to look at.
 
 ## Prerequisites
 
@@ -72,13 +74,17 @@ same iteration number and break that auto-detection on the next round.
 ## 4. Read the train split while you decide
 
 ```shell
-$ trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id>
+$ trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id> \
+    --withhold-test-detail
 ```
 
 `by_split.train`, its transcripts, and `grading.json` are fair game: read
 them to see whether the change did anything before spending a verdict on it.
-Leave `by_split.test` alone at this stage beyond noting whether it has data
-at all; its detail is not for reading here.
+`--withhold-test-detail` drops every held-out case id and assertion the
+document would otherwise carry, in the top-level stability lists as well as
+`by_split.test`, so there is nothing left to read there even by accident;
+its aggregate counts and headroom warning still show whether it has data at
+all.
 
 ## 5. Check for saturation
 
@@ -95,7 +101,8 @@ and [Reference: headroom warning](../reference/ai-skills-eval.md#headroom-warnin
 ## 6. Get the verdict
 
 ```shell
-$ trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id>
+$ trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id> \
+    --withhold-test-detail
 ```
 
 Once a previous round exists, `--previous` auto-detects it and the output

crates/trg/docs/reference/ai-skills-eval.md

@@ -388,13 +388,22 @@ trg ai skills eval iteration-summary <REPORT_DIR> [OPTIONS]
 | `--output-format` | enum | `text` | `text` prints a human-readable table; `json` prints the `iteration-summary.json` document on stdout |
 | `--fail-on` | enum list | *(none)* | Exit `1` when the keep-or-revert recommendation matches one of these. Comma-separated. Values: `revert`, `overfitting`. See [Exit codes](#exit-codes-2) and [Gating a hillclimb in CI](../how-to/run-in-ci.md) |
 | `--headroom-threshold` | proportion | `0.9` | The `with_skill` arm's Wilson lower bound is reported as saturated (see [Headroom warning](#headroom-warning)) at or above this proportion in `(0, 1]` |
+| `--withhold-test-detail` | flag | off | Drop test-split eval case ids and assertion text from the document, keeping only its aggregate counts, deltas, and intervals |
 
 `iteration-summary.json` reports `always_pass`, `always_fail`, and `helped_by_skill` for
 the whole report, and again per split under `by_split.train` and `by_split.test`. It also
 reports a `headroom` field, and again per split under `by_split.train.headroom` and
 `by_split.test.headroom`, the same shape and threshold [`eval benchmark`
 reports](#headroom-warning).
 
+`--withhold-test-detail` strips held-out case ids and assertion text from every list that
+would otherwise carry them: the top-level stability lists (`always_pass`, `always_fail`,
+`helped_by_skill`, `flaky_assertions`, timing and token outliers), `cross_iteration`, and
+`by_split.test` itself. Aggregate counts, pass rates, deltas, intervals, and the
+`keep_or_revert` verdict are unaffected, since a hillclimb round only ever needs those to
+decide whether to keep or revert a change. See [Hillclimb a
+skill](../how-to/hillclimb-a-skill.md#the-rule).
+
 ### Keep-or-revert verdict
 
 When `--previous` resolves to a prior iteration, `iteration-summary.json` carries a
@@ -415,8 +424,10 @@ into a verdict rather than left for the reader to eyeball:
   verdict for context.
 - `recommendation` is `keep` when the test split improved, `revert` when it regressed
   (checked before overfitting, since a held-out regression is reason enough on its own),
-  `suspected_overfitting` when train improved while test did not, and `inconclusive`
-  otherwise.
+  `suspected_overfitting` when train improved while a measured (`indistinguishable`) test
+  split did not, and `inconclusive` otherwise. A `no_runs` test split (no held-out cases
+  declared, or none selected) is silence about generalization, not a flat measurement, so it
+  never yields `suspected_overfitting` on its own, however much train improved.
 - `capped_by_saturation` is set to `test_split_saturated` when the test split's
   `with_skill` arm has cleared `--headroom-threshold`, so `recommendation` is not misread
   as an ordinary `inconclusive` or a clean `keep`: the split the verdict depends on most
@@ -751,9 +762,13 @@ scenario, with the same Wilson and Newcombe intervals `benchmark` reports, so
 a reviser knows the score without seeing what produced it. A suite that
 declares no `test` case gets `held_out.status: "no_test_cases"` instead, and
 `improvement.md` nudges toward declaring one, since every later overfitting
-check depends on a held-out set existing. Suite drift detection still
-compares hashes for a `test`-split case, but the bundle never lists that
-case's id among the added or removed ones.
+check depends on a held-out set existing. A suite that does declare `test`
+cases but whose selection (`--split train`, `--case`) did not run any of them
+gets `held_out.status: "declared_not_run"` with `declared_test_case_count`,
+instead of being reported as if the suite had no held-out set at all. Suite
+drift detection still compares hashes for a `test`-split case, but the bundle
+never lists that case's id among the added or removed ones, whether or not
+this run's selection covered it.
 
 ---
 
@@ -2151,7 +2166,11 @@ A narrowed run records what it covered under `suite.case_selection` in `report.j
       "tags": ["smoke"],
       "split": "test",
       "covered": 2,
-      "declared": ["analyze-refunds", "analyze-sales", "summarize-quarter"]
+      "declared": [
+        { "id": "analyze-refunds", "split": "test" },
+        { "id": "analyze-sales", "split": "test" },
+        { "id": "summarize-quarter", "split": "train" }
+      ]
     }
   }
 }
@@ -2166,9 +2185,13 @@ when the run covered every case the suite declares, however the selection was wr
 pattern that happens to match the whole suite narrowed nothing.
 
 `declared` names the whole suite the selection was taken from, since `dimensions.eval_cases`
-lists only what the run covered. Suite drift is diffed against `declared`, so a case a run
-skipped is not reported as one the suite lost, nor as one it gained the next time a run
-covers it.
+lists only what the run covered. Each entry also carries the split that case belongs to, so
+a case a narrowed run excluded still has a split to be judged by. Suite drift is diffed
+against `declared`, so a case a run skipped is not reported as one the suite lost, nor as
+one it gained the next time a run covers it. A `report.json` an older build wrote may still
+have `declared` as a bare list of case ids; that form is still read, with every case in it
+treated as having an unknown split rather than guessed as train or test, but it is never
+written by a current build.
 
 ## Measuring triggering
 

crates/trg/schemas/improvement-bundle.json.schema.json

@@ -339,6 +339,25 @@
           ],
           "type": "object"
         },
+        {
+          "description": "The suite declares `test`-split cases, but this run's selection (`--split train`,\n`--case`) did not cover any of them, so there is nothing to aggregate. Distinct from\n[`Self::NoTestCases`]: the suite is not missing a held-out set, this run just did not\nexercise it.",
+          "properties": {
+            "declared_test_case_count": {
+              "format": "uint",
+              "minimum": 0,
+              "type": "integer"
+            },
+            "status": {
+              "const": "declared_not_run",
+              "type": "string"
+            }
+          },
+          "required": [
+            "status",
+            "declared_test_case_count"
+          ],
+          "type": "object"
+        },
         {
           "properties": {
             "deltas": {

crates/trg/schemas/report.json.schema.json

@@ -120,9 +120,9 @@
           "type": "integer"
         },
         "declared": {
-          "description": "Every case ID the suite declared, covered by this run or not.",
+          "description": "Every case the suite declared, covered by this run or not.",
           "items": {
-            "type": "string"
+            "$ref": "#/$defs/DeclaredEvalCase"
           },
           "type": "array"
         },
@@ -196,6 +196,28 @@
       ],
       "type": "object"
     },
+    "DeclaredEvalCase": {
+      "description": "A case the suite declared, with the split it belongs to.\n\nA run narrowed by `--split train` never produces a run for a `test`-split case it\nexcludes, so this is the only record of that case's split a later reader (drift\ndetection, the held-out section) can consult without re-reading the suite off disk,\nwhich may no longer have the case at all.\n\n`split` is `None` when reading a `report.json` an older build wrote as a bare list of\ncase ids, before a declared case recorded its split at all. Such a case must never be\ncounted as train or test: guessing wrong here is indistinguishable from a real answer,\nso callers that need to know whether a case is safe to reveal treat an unknown split the\nsame as a test split, and callers that count test cases do not count it as one.",
+      "properties": {
+        "id": {
+          "type": "string"
+        },
+        "split": {
+          "anyOf": [
+            {
+              "$ref": "#/$defs/EvalSplit"
+            },
+            {
+              "type": "null"
+            }
+          ]
+        }
+      },
+      "required": [
+        "id"
+      ],
+      "type": "object"
+    },
     "DimensionsSection": {
       "properties": {
         "assertions": {
@@ -814,7 +836,7 @@
         },
         {
           "const": "incomplete",
-          "description": "At least one run recorded against this config has no known model, typically\nbecause no run has executed yet or because one fell back to a runner default.",
+          "description": "At least one run recorded against this config has no known model, typically\nbecause no run has executed yet or because one fell back to a runner default.\n\n`partial` is accepted as an alias so a `report.json` an older build wrote still\nloads; the schema and anything this process writes only ever emit `incomplete`.",
           "type": "string"
         }
       ]

crates/trg/schemas/scaling.json.schema.json

@@ -28,7 +28,7 @@
         },
         {
           "const": "incomplete",
-          "description": "At least one run recorded against this config has no known model, typically\nbecause no run has executed yet or because one fell back to a runner default.",
+          "description": "At least one run recorded against this config has no known model, typically\nbecause no run has executed yet or because one fell back to a runner default.\n\n`partial` is accepted as an alias so a `report.json` an older build wrote still\nloads; the schema and anything this process writes only ever emit `incomplete`.",
           "type": "string"
         }
       ]

crates/trg/skills/trg-eval-hillclimb/SKILL.md

@@ -81,7 +81,8 @@ a reason to run it twice.
 ## Step 4: read the train split, and only the train split, while you are still deciding
 
 ```shell
-trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id>
+trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id> \
+  --withhold-test-detail
 ```
 
 `by_split.train` and its transcripts are fair game: read them, read
@@ -91,9 +92,13 @@ change did anything at all before you spend a verdict on it.
 **Do not open a test-split run's transcript, workspace, or grading detail at
 any point in this loop.** The held-out split's job is to catch a change that
 learned the suite instead of the task, and it can only do that if nothing
-about a round's decisions was shaped by looking at it. If a number from
-`by_split.test` is not enough, that itself is information (see saturation,
-next), not a reason to look closer.
+about a round's decisions was shaped by looking at it. `--withhold-test-detail`
+is what makes that a rule the tool enforces rather than one you have to
+remember: without it, a held-out case id or assertion shows up in several
+places in the full document (the top-level stability lists, `by_split.test`,
+its headroom warning) even though you only meant to read the train split. If
+a number from `by_split.test` is not enough, that itself is information (see
+saturation, next), not a reason to drop the flag and look closer.
 
 ## Step 5: check for saturation before reading the verdict as ordinary
 
@@ -117,7 +122,8 @@ Continuing to tune the skill against a saturated test split only produces
 ## Step 6: get the verdict
 
 ```shell
-trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id>
+trg ai skills eval iteration-summary ./artifacts/my-skill/<report-id> \
+  --withhold-test-detail
 ```
 
 Once there is a previous round to compare against, `--previous` auto-detects

crates/trg/src/agentskills/cache.rs

@@ -1059,6 +1059,82 @@ mod tests {
         assert!(report_b.join("runs/run-001/workspace/outputs/final.md").is_file());
     }
 
+    /// `capture_status: "partial"` is a retired spelling of `incomplete` that an older build
+    /// left behind in a source report a cache pointer still names. Reading it back must not
+    /// be a cache miss: it aliases cleanly to `incomplete`, so the hit reuses the run exactly
+    /// as `exact_cache_hit_reuses_completed_run` does for a report the current build wrote.
+    #[test]
+    fn a_cache_hit_against_a_report_with_the_retired_partial_capture_status_still_reuses_the_run() {
+        let temp = tempdir().unwrap();
+        let out_dir = temp.path().join("out");
+        let report_a = out_dir.join("demo/report-a");
+        write_completed_run(&report_a, "run-001", "one", ScenarioKind::WithSkill);
+
+        let report_path = report_a.join("report.json");
+        let mut document: serde_json::Value = serde_json::from_str(&fs::read_to_string(&report_path).unwrap()).unwrap();
+        document["dimensions"]["model_configs"] = serde_json::json!([{
+            "id": "ci-default",
+            "capture_status": "partial",
+            "label": "ci-default",
+            "parameters": {},
+            "parameter_sources": {},
+            "extra": {}
+        }]);
+        fs::write(&report_path, serde_json::to_string_pretty(&document).unwrap()).unwrap();
+
+        let input = sample_key_input(ScenarioKind::WithSkill, "sha256:skill", FixtureHash::empty().as_str());
+        let key = CacheKey::from_input(&input);
+        record_completion(&out_dir, &key, &input, &report_a, "run-001").unwrap();
+
+        let report_b = out_dir.join("demo/report-b");
+        fs::create_dir_all(report_b.join("runs/run-001/workspace/outputs")).unwrap();
+        let mut run = RunRecord {
+            runner_model: None,
+            runner_model_source: None,
+            id: "run-001".to_string(),
+            eval_case_id: "one".to_string(),
+            eval_slug: "one".to_string(),
+            split: EvalSplit::Train,
+            scenario_id: ScenarioKind::WithSkill,
+            iteration: 2,
+            model_config_id: "ci-default".to_string(),
+            skill_revision_id: "current".to_string(),
+            attempt: 1,
+            failure_kind: None,
+            runner_invocations: 0,
+            status: "skipped".to_string(),
+            paths: RunPaths {
+                workspace: "runs/run-001/workspace".to_string(),
+                outputs: "runs/run-001/workspace/outputs".to_string(),
+            },
+            mirror_path: "iteration-2/eval-one/with_skill/".to_string(),
+            tool_grant: None,
+            artifacts: Vec::new(),
+            metrics: RunMetrics {
+                duration_ms: None,
+                exit_code: None,
+                total_tokens: None,
+                input_tokens: None,
+                output_tokens: None,
+                cached_tokens: None,
+                cost: None,
+            },
+            cache: None,
+            skill_integrity: None,
+            read_only_fixture_violations: Vec::new(),
+            warnings: Vec::new(),
+            mock_violations: Vec::new(),
+            case_score: None,
+        };
+
+        let pointer = lookup_exact(&out_dir, &key, &input).expect("the pointer's own key input is still fresh");
+        apply_cache_hit(&mut run, &key, &pointer, &report_b)
+            .expect("a retired 'partial' capture_status must not turn a cache hit into a miss");
+
+        assert_eq!(run.status, "completed");
+        assert!(run.cache.as_ref().unwrap().hit);
+    }
+
     /// A cache hit must not launder a run that failed its mock's `expect` guard into one
     /// that looks clean: grading only ever sees `mock_violations` and `mock-calls.jsonl`,
     /// so if a cache hit drops either, the second run of a suite passes where the first