feat(eval): Move 1 — trajectory capture + TraceProbe INSIGHT scoring on real SWE-bench tasks

docs/findings/0002-pack-cuts-search-loops.md

@@ -0,0 +1,181 @@
+# Finding 0002 — An OCH pack cuts a coding agent's shell-hunting and file re-reads (search loops don't fire on fix tasks)
+
+- Status: **Preliminary — first live run, 2026-07-08.** Six SWE-bench Verified
+  tasks × 5 runs/arm on Claude Code / Sonnet 5 via Amazon Bedrock. **4 tasks
+  produced a valid two-arm comparison; 2 (the largest repos) overflowed the
+  model context** — itself a finding, see below. A signal, not a benchmark.
+- Author: Bonk + Laith.
+- Instrument: `codehub code-pack --variance-probe --insight` (Move 1), scoring
+  each run's tool-call trajectory against the TraceProbe (arXiv:2607.06184)
+  structural anti-pattern detectors, on real SWE-bench Verified tasks.
+- Grounding: TraceProbe ran its 2,500 trajectories on **SWE-bench Verified**, so
+  these per-detector numbers are directly comparable to the paper's.
+
+## The headline
+
+Across the 4 tasks with a valid comparison, an OCH pack **suppressed every
+anti-pattern that fired and introduced none:**
+
+- **Shell-over-Tool: 1.9 → 0.65 firings/run** (mean Δ +1.25) — the pack stops the
+  agent from shelling `grep`/`cat`/`find` to reconstruct structure it was handed.
+  It cut this on all 4 tasks.
+- **Re-read Churn: 0.3 → 0.05 firings/run** (mean Δ +0.25) — fewer repeat reads of
+  the same file (fired on requests-1142).
+- **Search Loop: 0 → 0.** Never fired in either arm. SWE-bench fix tasks give the
+  agent a specific bug, so it does not run the ≥10-action open-ended search
+  stretches TraceProbe's Search Loop detects. On these tasks the pack's win is
+  *shell-hunting and re-reads*, not loops — the honest, narrower claim.
+- **Redundant Search: 0 → 0.** Same reason.
+
+This is the behavioral complement to Finding 0001's 2–4× token cut: the pack
+doesn't just cost fewer tokens, it changes what the agent *does* — less shelling
+out, fewer re-reads.
+
+## Second finding: a large-repo pack can overflow the model context
+
+The two pytest tasks **errored on every with-pack run** while their without-pack
+arms ran clean. Cause: the assembled pack for `pytest` is **~5.2 MB (≈1.3M
+tokens)** — past Claude's 1M window — so the model rejected it with "prompt is
+too long." This is not a probe bug; it is a real product boundary. The default
+`--budget` (100K tokens) bounds only the **AST-chunks** BOM item; `xrefs`,
+`skeleton`, and `file-tree` are unbounded and dominate a large repo's pack.
+**Action for OCH:** a whole-pack token ceiling (budget the assembled context,
+not just the chunks), and the probe should detect an oversized pack and skip the
+with-pack arm with a clear message rather than erroring all N runs. Tracked as a
+follow-up; the without-pack pytest arms still gave clean baseline trajectories
+(shell-over-tool 1.8–2.2/run, re-read churn up to 1.8/run — the behavior a
+fitting pack would target).
+
+## The question
+
+Finding 0001 showed an OCH pack cuts a coding agent's **token usage 2–4×** by
+letting it stop re-reading files to reconstruct structure. That is a *resource*
+number. This finding asks the *behavioral* question underneath it: does the pack
+change **what the agent does** — specifically, does it cut the anti-patterns
+TraceProbe found are the most stable failure-associated clues, chiefly the
+**search loop**?
+
+The hypothesis follows directly from OCH's thesis (hand the agent structure up
+front, it stops hunting): with the pack in context, the agent should run fewer
+long search/read stretches, re-read the same file less, and repeat fewer
+searches. If it does not — if the pack cuts tokens but not loops — that is a real
+result we publish too. **The number can come back null.**
+
+## What is measured
+
+For each task, the probe runs the agent N times with the pack and N times
+without, holding commit / instruction / agent / model fixed (the same two-arm
+design as Finding 0001). New in Move 1: every run's tool-call stream is captured
+and normalized to TraceProbe's nine-type action taxonomy, then scored against
+the four **structural** detectors — the ones whose predicates read only the
+action list, no LLM labeler:
+
+| Detector | Fires when (frozen predicate) |
+|---|---|
+| **Search Loop** | ≥10 consecutive search/read actions with no write and no validation command between them |
+| **Re-read Churn** | the same file is read ≥3× within a 10-action window with no intervening write |
+| **Redundant Search** | the same normalized query recurs ≥2× within a 10-action window |
+| **Shell-over-Tool** | a shell `cat`/`grep`/`rg`/`find` runs read/search work a structured tool covers |
+
+The report's headline is the **per-run `without − with` delta** for each
+detector: positive means the pack suppressed the anti-pattern.
+
+### The four semantic detectors are deliberately out of scope
+
+TraceProbe defines eight structural + four semantic detectors. The semantic ones
+(Phase Oscillation, and effect-labelled variants) need an LLM to label each
+action's phase/effect — the same judge-oracle dependency that kept Finding
+0001's headline on the directly-measured token number. Scoring them would import
+labeler noise into a number we want to be a pure function of the trajectory.
+They are a v2 concern.
+
+## Results
+
+Claude Code / Sonnet 5 on Bedrock, N=5 runs/arm. Per-run detector firings,
+`without pack → with pack`:
+
+| Task | tokenOverhead | Search Loop | Re-read Churn | Redundant Search | Shell-over-Tool |
+|---|---:|---|---|---|---|
+| `pallets/flask-5014` | 11.0× | 0 → 0 | 0 → 0 | 0 → 0 | **1.6 → 0.4** |
+| `psf/requests-1142` | 2.3× | 0 → 0 | **1.2 → 0.2** | 0 → 0 | **2.0 → 0.2** |
+| `psf/requests-1724` | 7.3× | 0 → 0 | 0 → 0 | 0 → 0 | **3.0 → 2.0** |
+| `psf/requests-1766` | 6.0× | 0 → 0 | 0 → 0 | 0 → 0 | **1.0 → 0.0** |
+| **mean (4 valid)** | 6.6× | 0 → 0 | 0.3 → 0.05 | 0 → 0 | 1.9 → 0.65 |
+| `pytest/pytest-10051` | — | \_context overflow\_ | | | (baseline 1.8/run) |
+| `pytest/pytest-10081` | — | \_context overflow\_ | | | (baseline 2.2/run) |
+
+Positive delta = the pack suppressed the anti-pattern. All 4 valid tasks agree in
+direction; the pack cut every anti-pattern that fired and introduced none. The
+two pytest tasks are the ~1.3M-token overflow (see "Second finding" above) —
+their without-pack baselines are shown for reference (the shell-hunting a fitting
+pack would target).
+
+**Codex arm:** not run. This Bedrock account exposes only `openai.gpt-oss-*`,
+not the `gpt-5.5` the Codex runner targets, so the run is Claude-only (Sonnet 5,
+the default agentic tier). The Codex arm is deferred to an account with the
+model.
+
+**Token overhead** ranged 2.3×–11×: the pack is injected fresh and uncached per
+run, so a small repo (flask) with a large-relative pack shows high overhead. On
+the `--cache-channel` path (Move 4), a stable pack prefix caches across runs on
+opt-in providers — not exercised here.
+
+**Assertion pass-rate: 0 in both arms, uninformative.** The `/tmp` clones did
+not have their Python deps installed, so the graded oracle could not run the
+tests. Exactly the fidelity limit called out below. The token + trajectory
+deltas are the trustworthy headline; the graded correctness number needs the
+Docker-image path (v2).
+
+## What this is / is NOT
+
+- **A behavioral complement to 0001, on graded tasks.** SWE-bench instances ship
+  their own tests, so correctness here is *graded*, not the eyeball judgment
+  0001 flagged.
+- **NOT leaderboard resolve-rate.** v1 grades on a `/tmp` clone with deps
+  installed; the probe runner does not reset the checkout between the N runs, so
+  treat the **token + trajectory deltas as the trustworthy headline** and the
+  assertion pass-rate as indicative. Per-run isolation (or SWE-bench's official
+  per-instance Docker images) is a v2 upgrade.
+- **NOT the four semantic detectors** (see above).
+- **NOT a claim the pack makes agents *smarter*.** TraceProbe is explicit that
+  structure helps by making navigation reproducible and disciplined; this
+  measures exactly that — less wasted search — not higher capability.
+
+## Reproduce
+
+```bash
+# 1. Get a SWE-bench Verified slice as instances.json (HF datasets), then:
+node packages/eval/scripts/swebench-to-tasks.mjs \
+  --instances /tmp/sb/instances.json --out-dir /tmp/sb/tasks \
+  --clone-root /tmp/sb/repos --limit 8
+
+# 2. Clone + install + analyze each repo (materializes the graph the pack needs):
+bash packages/eval/scripts/swebench-prep.sh /tmp/sb/tasks
+
+# 3. Run the probe with --insight (Claude on Bedrock; Sonnet 5 lane):
+CLAUDE_CODE_USE_BEDROCK=1 AWS_REGION=us-east-1 \
+  codehub code-pack --variance-probe /tmp/sb/tasks/<id>.task.json \
+  --insight --runs 10 \
+  --pack-tokenizer anthropic:claude-sonnet-5@2026-06-30 --json
+```
+
+The `--json` report carries per-arm `insight.total` / `insight.perRun` and the
+per-harness `insightDelta` (positive = the pack suppressed the anti-pattern).
+
+## Next
+
+1. **Fix the pack context ceiling** (highest priority — surfaced by this run).
+   Budget the *whole assembled pack* to a token ceiling, not just the AST
+   chunks, so a large repo's pack fits the target window; and have the probe
+   detect an oversized pack and skip the with-pack arm with a clear message
+   rather than erroring all N runs. The two pytest tasks failed only for this.
+2. **Widen the corpus.** Four valid tasks is a signal, not a benchmark. Add
+   tasks with genuinely open-ended exploration to test whether Search Loop
+   *ever* fires under a pack — the one detector still at zero.
+3. **Install deps in the checkouts** (or use SWE-bench's official per-instance
+   Docker images) so the assertion pass-rate becomes informative and we can
+   report graded correctness alongside the trajectory deltas.
+4. **Codex arm** on an account exposing a `gpt-5.x` Bedrock model, for the
+   agent-neutral claim.
+5. Add the SWE-bench Pro slice (harder, contamination-resistant).
+6. v2: the four semantic detectors behind an explicit judge opt-in.

packages/cli/src/commands/variance-probe.ts

@@ -73,6 +73,12 @@ export interface VarianceProbeArgs {
    * `auto` default. Defaults to `auto` when omitted.
    */
   readonly cacheChannel?: CacheChannel;
+  /**
+   * Score each arm's captured trajectory against the TraceProbe structural
+   * anti-pattern detectors and report the per-run without−with delta (Move 1).
+   * Off by default; the report shape is unchanged unless set.
+   */
+  readonly insight?: boolean;
   /**
    * Test seam — inject a fake pack-context assembler so unit tests don't need a
    * real analyzed repo + pack on disk. Receives the resolved tokenizer lane so
@@ -203,6 +209,7 @@ export async function runVarianceProbe(args: VarianceProbeArgs): Promise<Varianc
     packTokenizerId,
     ...(args.runs !== undefined ? { runs: args.runs } : {}),
     ...(args.harness !== undefined ? { harnesses: [args.harness] } : {}),
+    ...(args.insight === true ? { insight: true } : {}),
   };
 
   return runProbe(task, runnerFor, options);

packages/cli/src/index.ts

@@ -418,6 +418,13 @@ program
       "anthropic:claude-sonnet-5@2026-06-30 to author the pack for Sonnet 5's heavier tokenizer. " +
       "Recorded in the variance report so results attribute to a lane (Finding 0001 v2).",
   )
+  .option(
+    "--insight",
+    "With --variance-probe: also score each arm's captured tool-call trajectory against the " +
+      "TraceProbe (arXiv:2607.06184) structural anti-pattern detectors — search loops, re-read " +
+      "churn, redundant search, shell-over-tool — and report the per-run without−with delta " +
+      "(positive = the pack suppressed the anti-pattern; Move 1).",
+  )
   .action(async (path: string | undefined, opts: Record<string, unknown>) => {
     // Channel-aware cache-prefix enforcement (Move 4). Validated once here so
     // an unknown channel errors clearly before either path runs. Commander
@@ -446,6 +453,7 @@ program
         ...(typeof opts["packTokenizer"] === "string"
           ? { packTokenizer: opts["packTokenizer"] }
           : {}),
+        ...(opts["insight"] === true ? { insight: true } : {}),
         cacheChannel,
       });
       probeMod.printVarianceReport(report, opts["json"] === true);

packages/eval/README.md

@@ -38,6 +38,36 @@ Claude Code via `CLAUDE_CODE_USE_BEDROCK=1` + a `us.`-prefixed
 `amazon-bedrock` provider. AWS credentials and region are inherited from the
 operator's environment.
 
+## Trajectory scoring — INSIGHT anti-patterns (Move 1)
+
+Passing `--insight` also captures each run's **tool-call trajectory** (Claude
+Code `--output-format stream-json`; Codex `--json`), normalizes it to
+TraceProbe's ([arXiv:2607.06184](https://arxiv.org/abs/2607.06184)) nine-type
+action taxonomy, and scores it against four **structural** anti-pattern
+detectors — no LLM labeler:
+
+| Detector | Fires when |
+|---|---|
+| Search Loop | ≥10 consecutive search/read actions, no write or validation between |
+| Re-read Churn | same file read ≥3× in a 10-action window, no intervening write |
+| Redundant Search | same normalized query recurs ≥2× in a 10-action window |
+| Shell-over-Tool | `cat`/`grep`/`rg`/`find` shelled while a structured tool covers it |
+
+The report adds a per-harness `insightDelta` — the per-run `without − with` of
+each detector (positive = the pack suppressed the anti-pattern). The four
+*semantic* TraceProbe detectors (Phase Oscillation, …) need an LLM labeler and
+are deferred to v2 (the judge-oracle caveat).
+
+### Real tasks — SWE-bench Verified / Pro
+
+`scripts/swebench-to-tasks.mjs` turns a SWE-bench instances JSON into probe task
+files: the `problem_statement` becomes the instruction, and an `assertion`
+oracle applies the `test_patch` and runs the FAIL_TO_PASS + PASS_TO_PASS tests
+(so correctness is *graded*, unlike Finding 0001). `scripts/swebench-prep.sh`
+clones + installs + analyzes each repo. See `docs/findings/0002` for the full
+protocol and fidelity limits (v1 grades on a `/tmp` clone without per-run
+checkout isolation — token + trajectory deltas are the trustworthy headline).
+
 ## Determinism of the probe's own output
 
 The agent runs are nondeterministic by nature — that nondeterminism is exactly

packages/eval/scripts/swebench-prep.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# Clone + install + analyze each SWE-bench repo from a tasks dir's clones.json
+# (Move 1, Phase 0). Idempotent: skips a clone/analyze that already exists.
+#
+# Usage: bash packages/eval/scripts/swebench-prep.sh <tasks-dir>
+#
+# Reads <tasks-dir>/clones.json (produced by swebench-to-tasks.mjs), clones each
+# repo at its base_commit into the clone dest, best-effort installs Python deps,
+# then runs `codehub analyze` so the with-pack arm has a graph to pack from.
+#
+# ⚠️ Fidelity: this materializes ONE checkout per instance. The v1 probe runner
+# does not reset that checkout between the N runs, so the graded assertion
+# pass-rate is indicative; the token + trajectory deltas are the trustworthy
+# headline. Per-run isolation (or SWE-bench's official Docker images) is a v2
+# upgrade — see docs/findings/0002.
+set -euo pipefail
+
+TASKS_DIR="${1:?usage: swebench-prep.sh <tasks-dir>}"
+CLONES="${TASKS_DIR}/clones.json"
+[ -f "$CLONES" ] || { echo "no clones.json in $TASKS_DIR — run swebench-to-tasks.mjs first" >&2; exit 1; }
+
+# Resolve the codehub CLI: prefer a repo-local build, fall back to a global install.
+CODEHUB="${CODEHUB:-codehub}"
+
+# Emit the clone manifest as plain tab-separated rows in ONE node call, using
+# process.stdout.write (console.log can be ANSI-colorized by a shim, which would
+# corrupt the parsed fields). Iterate with `while read` — no per-field node
+# spawns, no arithmetic on a possibly-colorized count.
+node -e '
+const rows = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"));
+for (const r of rows) process.stdout.write([r.instanceId, r.cloneUrl, r.baseCommit, r.dest].join("\t") + "\n");
+' "$CLONES" > "$TASKS_DIR/.clones.tsv"
+
+echo "Preparing $(wc -l < "$TASKS_DIR/.clones.tsv" | tr -d ' ') SWE-bench repo(s) from ${CLONES}" >&2
+
+while IFS=$'\t' read -r id url commit dest; do
+  [ -n "$id" ] || continue
+  echo "── ${id}" >&2
+  if [ ! -d "$dest/.git" ]; then
+    git clone --quiet "$url" "$dest"
+  fi
+  git -C "$dest" fetch --quiet origin "$commit" 2>/dev/null || true
+  git -C "$dest" checkout --quiet --force "$commit"
+  git -C "$dest" reset --hard --quiet "$commit"
+
+  # Best-effort Python dep install (SWE-bench is ~94% Python). Non-fatal: a repo
+  # that needs a bespoke env still yields token/trajectory deltas; only its
+  # graded assertion needs the deps. Prefer uv, fall back to pip.
+  if [ -f "$dest/pyproject.toml" ] || [ -f "$dest/setup.py" ]; then
+    # Install the package + pytest into a per-repo uv venv so the assertion can
+    # import and test it. Best-effort — a repo needing a bespoke env still
+    # yields token/trajectory deltas; only its graded assertion needs deps.
+    ( cd "$dest" \
+      && uv venv --quiet .venv 2>/dev/null \
+      && uv pip install --python .venv --quiet -e . pytest 2>/dev/null ) </dev/null \
+      || echo "  (dep install skipped/failed — assertion may under-report; deltas still valid)" >&2
+  fi
+
+  # Analyze so the with-pack arm has a graph. --no-scan keeps it fast (findings
+  # aren't needed for the pack context); drop it if a task exercises SARIF.
+  # stdin from /dev/null so analyze never consumes the loop's manifest feed.
+  if [ ! -f "$dest/.codehub/store.sqlite" ]; then
+    "$CODEHUB" analyze "$dest" --no-scan >&2 </dev/null
+  fi
+done < "$TASKS_DIR/.clones.tsv"
+
+echo "Done. Run e.g.:" >&2
+echo "  CLAUDE_CODE_USE_BEDROCK=1 AWS_REGION=us-east-1 \\" >&2
+echo "    codehub code-pack --variance-probe ${TASKS_DIR}/<id>.task.json --insight --runs 10 --json" >&2