feat(status): show each nested memory file's sampled-session reach by Pantoria · Pull Request #209 · kunchenguid/backpass

Pantoria · 2026-10-15T15:48:21Z

Intent

Implement part (1) of issue #191: a nested memory file trains only on sessions whose every in-repo work path is under its directory, so in repos with cross-cutting sessions it may never reach minGapEvidence, and nothing told the user. Now backpass status and the run summary show, per nested file, how many sampled sessions were attributed to it, with a warning below minGapEvidence. The user (issue author, Pantoria) asked to implement their own ready-for-pr backpass issues, each in its own worktree and PR. Follow AGENTS.md invariants and VISION.md. No live model calls were made while implementing.
Scope decision: only part (1) (maintainer: restore, can ship alone). Part (2), the opt-in to ignore reads of configured memory files as work paths, is explicitly out of scope and left for a follow-up; commit says Refs #191, not Fixes. Design: counts come only from deterministic workedUnder attribution in nestedCorpora (src/commands/analyze.js), persisted to .backpass/nested/reach.json at attribution time so status stays read-only and cheap (no discovery/sampling); unmeasured files show 'not yet measured'. Proposal nested entries gain sampledSessions/minGapEvidence/belowMinGapEvidence; status --json gains nestedAttribution. Strictly report-only per the 'Presentation never gates' invariant - a test asserts the root edit is still proposed when a nested file is below the floor. Wording deliberately says 'no add, rewrite, or remove edit can reach this file' instead of the issue's 'cannot receive edits', because extract/move edits are exempt from the evidence floor. Accepted: status shows the count/floor saved at the last attribution, timestamped, not a fresh measurement.

What Changed

Each time sessions are attributed to nested memory files (nestedCorpora in src/commands/analyze.js), backpass now saves to .backpass/nested/reach.json how many sampled sessions each nested file can learn from, meaning sessions whose every in-repo work path is under that file's directory. The save has a timestamp and records the minGapEvidence floor and whether the file is below it. New helpers in src/nested.js build, write, read and format this record (nestedReach, writeNestedReach, readNestedReach, nestedReachWarning).
backpass status gains a "NESTED EVIDENCE REACH" section. It shows the count and floor saved at the last attribution, or not yet measured if no attribution has run, and warns when a file is below minGapEvidence, at which point no add, rewrite, or remove edit can reach it. status --json now includes a nestedAttribution object. status reads only the saved file and does no discovery or sampling.
In proposals, each nested entry now carries sampledSessions, minGapEvidence and belowMinGapEvidence. The run summary prints "N of M sampled session(s) worked only under dir/" and the same below-floor warning. All of this is report-only and never gates a run. README documents it, and tests cover the status output and the warning, including a check that the root edit is still proposed when a nested file is below the floor.

Refs #191. This covers part (1) only; the opt-in to ignore reads of memory files is left for a follow-up.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change only reports data: counts come from the existing deterministic workedUnder attribution and are saved to .backpass/nested/reach.json. That file sits beside the per-file state directories without clashing, since the raw-file cleanup only walks subdirectories. Nothing reads the count back to gate a run, and a test checks the root edit is still proposed when a nested file is below the floor, which matches every stated constraint in the intent.

Testing

I drove the real CLI end to end in four disposable monorepo setups: below the floor, at the floor, analyze only, and no nested config. In every setup only the model was stubbed. The run summary and status showed "1 of 3 sampled session(s) worked only under apps/api/" with the warning "below minGapEvidence (2); no add, rewrite, or remove edit can reach this file". At the floor ("2 of 4") there was no warning. The root edit was still proposed below the floor, so the count stays report-only. Before any run, status shows "not yet measured". Status does not change reach.json. A corrupt reach.json makes status warn and fall back to "not yet measured" without failing. With no nested config there is no reach section and no nestedAttribution key. The two targeted test files also pass. Every scenario passed, so the verdict is go.

Live validation: ✅ go - 9 of 9 scenarios driven live against the product

Scenario	Result	Live	Evidence
Before any run, backpass status lists the nested file as 'not yet measured' and writes no reach.json	✅ pass	live	sparse.txt: 'apps/api/AGENTS.md: not yet measured - run `backpass` or `backpass analyze`'; 'reach.json exists after status: false'
A full backpass run where the nested file gets 1 of 3 sessions prints the count and the below-minGapEvidence warning in the run summary	✅ pass	live	sparse.txt run summary; proposal.nested[0] has sampledSessions 3, minGapEvidence 2, belowMinGapEvidence true
Report-only: the root edit earned by the web sessions is still proposed while the nested file is below the floor	✅ pass	live	sparse.txt: proposal edit files = [["e1","add","AGENTS.md"]]
After the run, backpass status shows the timestamped reach section with the warning, and status --json carries nestedAttribution	✅ pass	live	sparse.txt: 'NESTED EVIDENCE REACH (sampled sessions at the last attribution, 2026-10-15 16:49; report-only)' plus the JSON nestedAttribution block
status reads the saved count without rewriting reach.json	✅ pass	live	sparse.txt: 'reach.json unchanged by status: true'
At the floor (2 of 4 sessions), the count is shown with no warning in both the run summary and status	✅ pass	live	floor.txt
backpass analyze alone records the count, and status then shows it	✅ pass	live	analyze.txt: 'reach.json exists after analyze: true'; status shows '1 of 2' with the warning
Adversarial: a corrupt or malformed reach.json does not make status fail; it falls back to 'not yet measured'	✅ pass	live	sparse.txt: both corrupt-file status runs exit 0 and show 'not yet measured'
A repo without nested memory files shows no reach section, no nestedAttribution key, and writes no reach.json	✅ pass	live	plain.txt: 'status --json has nestedAttribution key: false', 'reach.json exists: false'

Evidence: Below-floor run summary, status, status --json, and corrupt reach.json transcript


===== backpass status (before any run) (exit 0) =====
backpass-nested-repo-EdQHQ7 /st/tmp/backpass-nested-repo-EdQHQ7

BUDGET (always-loaded)
  AGENTS.md          [................................] 14 / 5,000 tok · 1 instructions
  apps/api/AGENTS.md [................................] 14 / 5,000 tok · 1 instructions · nested, loads under apps/api/

NESTED EVIDENCE REACH (report-only)
  apps/api/AGENTS.md: not yet measured - run `backpass` or `backpass analyze`

CACHE
  scan cache      0 file(s) fingerprinted
  evidence        0 ok · 0 skipped · 0 failed
  rejections      0 remembered

PROPOSAL
  none yet - run `backpass`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet

reach.json exists after status: false

===== backpass (full run, run summary) (exit 0) =====

proposal · backpass-nested-repo-EdQHQ7 · AGENTS.md · 1 edit(s) from 3 session(s) · interactive 3 · non-interactive 0
  budget [................................] 14 -> 28 / 5,000 tok
  evidence: 0 positive · 0 negative · 1 gap clusters
  nested apps/api/AGENTS.md · 1 of 3 sampled session(s) worked only under apps/api/ · budget [................................] 14 -> 14 / 5,000 tok
    below minGapEvidence (2); no add, rewrite, or remove edit can reach this file

  e1 ADD      Build the web bundle before opening a pull request. (+14 tok, 2 transcript(s))

  tier-1 tokens: not reported by pi
  tier-2 tokens: not reported by pi, claude

Review and apply with `backpass apply` (nothing has been written).

[stderr]
backpass v0.1.34 · backpass-nested-repo-EdQHQ7 · budget 5,000 tok · since all
· analyzing 3 transcript(s) with pi effort=medium at jobs=1
  3/3 analyzed
· apps/api/AGENTS.md is a nested memory file: 1 of 3 session(s) worked under apps/api/
· analyzing 1 transcript(s) with pi effort=medium at jobs=1
  1/1 analyzed
· AGENTS.md: [................................] 14 / 5,000 tok · 1 instructions
· evidence: 3 new · 0 cached · 0 too short · 0 failed
· apps/api/AGENTS.md (nested): [................................] 14 / 5,000 tok · 1 instructions
  evidence: 1 new · 0 cached · 0 too short · 0 failed
· synthesizing apps/api/AGENTS.md (nested) with claude (fake-model) effort=high
· synthesizing with claude (fake-model) effort=high

proposal.nested[0] = {"memoryFile":{"path":"apps/api/AGENTS.md","hash":"sha256:aab5b0f01dd0ba93","tokens":14},"dir":"apps/api","sessions":1,"sampledSessions":3,"minGapEvidence":2,"belowMinGapEvidence":true,"maxEdits":5,"edits":[]}
proposal edit files = [["e1","add","AGENTS.md"]]
reach.json = {
  "measuredAt": "2026-10-15T16:49:49.840Z",
  "files": [
    {
      "path": "apps/api/AGENTS.md",
      "dir": "apps/api",
      "sessions": 1,
      "sampledSessions": 3,
      "minGapEvidence": 2,
      "belowMinGapEvidence": true
    }
  ]
}


===== backpass status (exit 0) =====
backpass-nested-repo-EdQHQ7 /st/tmp/backpass-nested-repo-EdQHQ7

BUDGET (always-loaded)
  AGENTS.md          [................................] 14 / 5,000 tok · 1 instructions
  apps/api/AGENTS.md [................................] 14 / 5,000 tok · 1 instructions · nested, loads under apps/api/

NESTED EVIDENCE REACH (sampled sessions at the last attribution, 2026-10-15 16:49; report-only)
  apps/api/AGENTS.md: 1 of 3 sampled session(s) worked only under apps/api/
    below minGapEvidence (2); no add, rewrite, or remove edit can reach this file

CACHE
  scan cache      3 file(s) fingerprinted
  evidence        3 ok · 0 skipped · 0 failed
  gradients       3 session(s) · 0+ 0- · 1 gap cluster(s)
  rejections      0 remembered

PROPOSAL
  generated       2026-10-15T16:49:50.228Z
  edits           1
  review with `backpass apply`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet


===== backpass status --json (exit 0) =====
{
  "repo": "backpass-nested-repo-EdQHQ7",
  "budgets": [
    {
      "path": "AGENTS.md",
      "label": "AGENTS.md",
      "capTokens": 5000,
      "current": 14,
      "projected": 14,
      "delta": 0,
      "withinBudget": true,
      "over": 0,
      "utilization": 0.0028,
      "instructions": 1,
      "pointerTo": null,
      "separate": false
    },
    {
      "path": "apps/api/AGENTS.md",
      "label": "apps/api/AGENTS.md",
      "capTokens": 5000,
      "current": 14,
      "projected": 14,
      "delta": 0,
      "withinBudget": true,
      "over": 0,
      "utilization": 0.0028,
      "instructions": 1,
      "pointerTo": null,
      "separate": false,
      "nested": "apps/api",
      "missing": false
    }
  ],
  "nestedAttribution": {
    "measuredAt": "2026-10-15T16:49:49.840Z",
    "files": [
      {
        "path": "apps/api/AGENTS.md",
        "dir": "apps/api",
        "measured": true,
        "sessions": 1,
        "sampledSessions": 3,
        "minGapEvidence": 2,
        "belowMinGapEvidence": true
      }
    ]
  },
  "crossSurfaceDuplicates": [],
  "evidence": {
    "ok": 3,
    "failed": 0,
    "skipped": 0
  },
  "scanCacheEntries": 3,
  "hosts": {},
  "summary": {
    "analyzedSessions": 3,
    "totals": {
      "positive": 0,
      "negative": 0,
      "gapSightings": 3,
      "gapClusters": 1,
      "reportOnlyGapClusters": 0,
      "reportOnlyByReason": {
        "majorityOrchestration": 0,
        "belowFloorMixed": 0,
        "tooFewProjects": 0
      },
      "droppedGapSingletons": 0,
      "orchestrationGapSightings": 0,
      "instructionsWithNegatives": 0,
      "usedRawTranscript": 0,
      "crossSurfaceDuplicates": 0
    }
  },
  "proposal": {
    "generatedAt": "2026-10-15T16:49:50.228Z",
    "edits": 1
  },
  "rejections": 0,
  "skills": 0
}

reach.json unchanged by status: true

===== backpass status with corrupt reach.json (exit 0) =====
backpass-nested-repo-EdQHQ7 /st/tmp/backpass-nested-repo-EdQHQ7

BUDGET (always-loaded)
  AGENTS.md          [................................] 14 / 5,000 tok · 1 instructions
  apps/api/AGENTS.md [................................] 14 / 5,000 tok · 1 instructions · nested, loads under apps/api/

NESTED EVIDENCE REACH (report-only)
  apps/api/AGENTS.md: not yet measured - run `backpass` or `backpass analyze`

CACHE
  scan cache      3 file(s) fingerprinted
  evidence        3 ok · 0 skipped · 0 failed
  gradients       3 session(s) · 0+ 0- · 1 gap cluster(s)
  rejections      0 remembered

PROPOSAL
  generated       2026-10-15T16:49:50.228Z
  edits           1
  review with `backpass apply`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet

[stderr]
warn discarding corrupt state file .backpass/nested/reach.json: Expected property name or '}' in JSON at position 1 (line 1 column 2)


===== backpass status with malformed reach.json (files not array) (exit 0) =====
backpass-nested-repo-EdQHQ7 /st/tmp/backpass-nested-repo-EdQHQ7

BUDGET (always-loaded)
  AGENTS.md          [................................] 14 / 5,000 tok · 1 instructions
  apps/api/AGENTS.md [................................] 14 / 5,000 tok · 1 instructions · nested, loads under apps/api/

NESTED EVIDENCE REACH (report-only)
  apps/api/AGENTS.md: not yet measured - run `backpass` or `backpass analyze`

CACHE
  scan cache      3 file(s) fingerprinted
  evidence        3 ok · 0 skipped · 0 failed
  gradients       3 session(s) · 0+ 0- · 1 gap cluster(s)
  rejections      0 remembered

PROPOSAL
  generated       2026-10-15T16:49:50.228Z
  edits           1
  review with `backpass apply`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet

Evidence: At-floor run summary and status (no warning)


===== backpass (at floor: 2 api of 4) (exit 0) =====

proposal · backpass-nested-repo-5lpWLG · AGENTS.md · 2 edit(s) from 4 session(s) · interactive 4 · non-interactive 0
  budget [................................] 14 -> 28 / 5,000 tok
  evidence: 0 positive · 0 negative · 1 gap clusters
  nested apps/api/AGENTS.md · 2 of 4 sampled session(s) worked only under apps/api/ · budget [................................] 14 -> 28 / 5,000 tok

  e1 ADD      Build the web bundle before opening a pull request. (+14 tok, 2 transcript(s))
  e2 ADD      apps/api/AGENTS.md: Run the api contract tests before editing a handler. (+14 tok, 2 transcript(s))

  tier-1 tokens: not reported by pi
  tier-2 tokens: not reported by pi, claude

Review and apply with `backpass apply` (nothing has been written).

[stderr]
backpass v0.1.34 · backpass-nested-repo-5lpWLG · budget 5,000 tok · since all
· analyzing 4 transcript(s) with pi effort=medium at jobs=1
  4/4 analyzed
· apps/api/AGENTS.md is a nested memory file: 2 of 4 session(s) worked under apps/api/
· analyzing 2 transcript(s) with pi effort=medium at jobs=1
  2/2 analyzed
· AGENTS.md: [................................] 14 / 5,000 tok · 1 instructions
· evidence: 4 new · 0 cached · 0 too short · 0 failed
· apps/api/AGENTS.md (nested): [................................] 14 / 5,000 tok · 1 instructions
  evidence: 2 new · 0 cached · 0 too short · 0 failed
· synthesizing apps/api/AGENTS.md (nested) with claude (fake-model) effort=high
· synthesizing with claude (fake-model) effort=high


===== backpass status (exit 0) =====
backpass-nested-repo-5lpWLG /st/tmp/backpass-nested-repo-5lpWLG

BUDGET (always-loaded)
  AGENTS.md          [................................] 14 / 5,000 tok · 1 instructions
  apps/api/AGENTS.md [................................] 14 / 5,000 tok · 1 instructions · nested, loads under apps/api/

NESTED EVIDENCE REACH (sampled sessions at the last attribution, 2026-10-15 16:49; report-only)
  apps/api/AGENTS.md: 2 of 4 sampled session(s) worked only under apps/api/

CACHE
  scan cache      4 file(s) fingerprinted
  evidence        4 ok · 0 skipped · 0 failed
  gradients       4 session(s) · 0+ 0- · 1 gap cluster(s)
  rejections      0 remembered

PROPOSAL
  generated       2026-10-15T16:49:51.905Z
  edits           2
  review with `backpass apply`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet

Evidence: analyze-only run records reach, then status


===== backpass analyze (exit 0) =====

analyzed against AGENTS.md (1 instructions, 14 tok)
  2 newly analyzed · 0 cached · 0 skipped (too short) · 0 failed
analyzed against apps/api/AGENTS.md (nested, 1 instructions, 14 tok) from 1 session(s) under apps/api/
  1 newly analyzed · 0 cached · 0 skipped (too short) · 0 failed

  tier-1 tokens: not reported by pi

[stderr]
· analyzing 2 transcript(s) with pi effort=medium at jobs=1
  2/2 analyzed
· apps/api/AGENTS.md is a nested memory file: 1 of 2 session(s) worked under apps/api/
· analyzing 1 transcript(s) with pi effort=medium at jobs=1
  1/1 analyzed

reach.json exists after analyze: true

===== backpass status (exit 0) =====
backpass-nested-repo-umfxlO /st/tmp/backpass-nested-repo-umfxlO

BUDGET (always-loaded)
  AGENTS.md          [................................] 14 / 5,000 tok · 1 instructions
  apps/api/AGENTS.md [................................] 14 / 5,000 tok · 1 instructions · nested, loads under apps/api/

NESTED EVIDENCE REACH (sampled sessions at the last attribution, 2026-10-15 16:49; report-only)
  apps/api/AGENTS.md: 1 of 2 sampled session(s) worked only under apps/api/
    below minGapEvidence (2); no add, rewrite, or remove edit can reach this file

CACHE
  scan cache      2 file(s) fingerprinted
  evidence        2 ok · 0 skipped · 0 failed
  rejections      0 remembered

PROPOSAL
  none yet - run `backpass`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet

Evidence: Repo with no nested config: unchanged output


===== backpass (no nested config) (exit 0) =====

proposal · backpass-nested-repo-41R8F7 · AGENTS.md · 1 edit(s) from 4 session(s) · interactive 4 · non-interactive 0
  budget [................................] 14 -> 42 / 5,000 tok
  evidence: 0 positive · 0 negative · 2 gap clusters

  e1 ADD      Run the api contract tests before editing a handler. Build the web bundle before opening a pull request. (+28 tok, 4 transcript(s))

  tier-1 tokens: not reported by pi
  tier-2 tokens: not reported by pi, claude

Review and apply with `backpass apply` (nothing has been written).

[stderr]
backpass v0.1.34 · backpass-nested-repo-41R8F7 · budget 5,000 tok · since all
· analyzing 4 transcript(s) with pi effort=medium at jobs=1
  4/4 analyzed
· AGENTS.md: [................................] 14 / 5,000 tok · 1 instructions
· evidence: 4 new · 0 cached · 0 too short · 0 failed
· synthesizing with claude (fake-model) effort=high


===== backpass status (exit 0) =====
backpass-nested-repo-41R8F7 /st/tmp/backpass-nested-repo-41R8F7

BUDGET (always-loaded)
  AGENTS.md      [................................] 14 / 5,000 tok · 1 instructions

CACHE
  scan cache      4 file(s) fingerprinted
  evidence        4 ok · 0 skipped · 0 failed
  gradients       4 session(s) · 0+ 0- · 2 gap cluster(s)
  rejections      0 remembered

PROPOSAL
  generated       2026-10-15T16:49:53.492Z
  edits           1
  review with `backpass apply`

MODELS
  analysis        auto - 7 candidates, none probed yet
  synthesis       auto - 7 candidates, none probed yet

status --json has nestedAttribution key: false
reach.json exists: false

Evidence: Live driver script

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { execFileSync, spawnSync } from "node:child_process";
import { fileURLToPath } from "node:url";

/**
 * Nested memory files through the real CLI, the way a monorepo user runs them: a root
 * AGENTS.md and an `apps/api/AGENTS.md` named in `nestedMemoryFiles`, pi sessions that
 * worked either under `apps/api/` or under `apps/web/`, and a stand-in acpx that reports
 * one lesson per kind of session. `backpass` must feed the nested file only its own
 * sessions, land each lesson in the file that owns it, and hand both files to one
 * `backpass apply`; with nothing configured the run must stay the single-file run it was.
 */

const ROOT = "~/.no-mistakes/worktrees/8ce6865cb38e/01M503V10QDXF9V7GYQCNDCZN4";
const CLI = path.join(ROOT, "bin", "backpass.js");
const FAKE_LAVISH = path.join(ROOT, "test", "fixtures", "fake-lavish", "lavish-axi");

const API_QUOTE = "Please fix the api handler timeout in the orders route.";
const WEB_QUOTE = "Please fix the web checkout button styling.";
const API_GAP = "Run the api contract tests before editing a handler.";
const WEB_GAP = "Build the web bundle before opening a pull request.";

const ROOT_MEMORY = "# Monorepo memory\n\n- Use pnpm for every package script.\n";
const API_MEMORY = "# API memory\n\n- Handlers live in apps/api/src/routes.\n";

const binDir = fs.mkdtempSync(path.join(os.tmpdir(), "backpass-nested-bin-"));
const fakePi = path.join(binDir, "pi");
const fakeAcpx = path.join(binDir, "acpx");
fs.writeFileSync(fakePi, `#!${process.execPath}\nprocess.exit(0);\n`);
fs.chmodSync(fakePi, 0o755);

/**
 * The stand-in harness. Analysis reports the lesson whose quote is in the trace; the
 * synthesis edit turn appends every lesson the evidence it was shown carries; the annotate
 * turn claims those changes with the quotes and source labels that evidence issued.
 */
fs.writeFileSync(
  fakeAcpx,
  `#!${process.execPath}
const fs = require("node:fs");
const path = require("node:path");
const argv = process.argv.slice(2);
const at = (flag) => (argv.indexOf(flag) >= 0 ? argv[argv.indexOf(flag) + 1] : null);
if (argv.includes("config") && argv.includes("show")) {
  process.stdout.write(JSON.stringify({ agents: {} }) + "\\n");
  process.exit(0);
}
if (argv.includes("sessions") || argv.includes("set")) {
  if (argv.includes("new")) process.stdout.write("fake-session-id\\n");
  process.exit(0);
}
const promptFile = at("--file");
if (!promptFile) process.exit(0);
const prompt = fs.readFileSync(promptFile, "utf8");
const cwd = at("--cwd") || process.cwd();
const statePath = process.env.FAKE_STATE;
const state = fs.existsSync(statePath) ? JSON.parse(fs.readFileSync(statePath, "utf8")) : { log: [], shown: {} };
const save = () => fs.writeFileSync(statePath, JSON.stringify(state));
const lessons = ${JSON.stringify([
    { gap: API_GAP, quote: API_QUOTE },
    { gap: WEB_GAP, quote: WEB_QUOTE },
  ])};
if (process.env.FAKE_SHARED_GAP) lessons[1].gap = lessons[0].gap;

if (prompt.includes("You are auditing one past agent session")) {
  const audited = /## The memory file under audit: (\\S+)/.exec(prompt)[1];
  const gaps = lessons
    .filter((lesson) => prompt.includes(lesson.quote))
    .map((lesson) => ({
      mistake: "the session had to work it out by hand",
      proposedInstruction: lesson.gap,
      recurrenceRisk: "high",
      domain: "project",
      quote: lesson.quote,
    }));
  state.log.push({ turn: "analysis", audited, gaps: gaps.map((gap) => gap.proposedInstruction), alsoLoaded: prompt.includes("## Also loaded in this session") });
  save();
  process.stdout.write(JSON.stringify({ positive: [], negative: [], gaps }) + "\\n");
  process.exit(0);
}
if (prompt.includes("You are consolidating the gap ledger")) {
  process.stdout.write(JSON.stringify({ merges: [] }) + "\\n");
  process.exit(0);
}
if (prompt.includes("You are performing the synthesis step")) {
  const memory = /## Current memory file: (\\S+)/.exec(prompt)[1];
  const pairs = [...prompt.matchAll(/"([^"]+)" \\((pi · [^)]+)\\)/g)].map((m) => ({ text: m[1], source: m[2] }));
  const shown = lessons.filter((lesson, index) =>
    prompt.includes(":: " + lesson.gap) && lessons.findIndex((entry) => entry.gap === lesson.gap) === index);
  state.shown[memory] = shown.map((lesson) => ({
    ...lesson,
    evidence: pairs.filter((pair) => lessons.some((entry) => entry.gap === lesson.gap && entry.quote === pair.text)),
  }));
  const budgetCount = prompt.includes("The count is this file PLUS every skill's \\\`description:\\\` line;\\nskill bodies are free until triggered.")
    ? "surface"
    : prompt.includes("The count is this file alone") ? "file" : null;
  const nestedRule = prompt.includes("This run trains the nested memory file");
  state.log.push({ turn: "edit", memory, lessons: shown.map((lesson) => lesson.gap), budgetCount, nestedRule });
  save();
  const target = path.join(cwd, memory);
  let text = fs.readFileSync(target, "utf8");
  for (const lesson of shown) text += "- " + lesson.gap + "\\n";
  fs.writeFileSync(target, text);
  process.stdout.write("Added the lessons the evidence carries.\\n");
  process.exit(0);
}
if (prompt.includes("## Measured changes")) {
  const memory = /staging copy of (\\S+?)\\. Every change/.exec(prompt)[1];
  const changes = [...prompt.matchAll(/\\[(H\\d+):/g)].map((m) => m[1]);
  const shown = state.shown[memory] || [];
  state.log.push({ turn: "annotate", memory, changes });
  save();
  const edits = shown.length
    ? [{
        changes,
        kind: "add",
        title: shown.map((lesson) => lesson.gap).join(" "),
        rationale: "two sessions hit it",
        evidence: shown.flatMap((lesson) => lesson.evidence.map((pair) => ({ polarity: "negative", text: pair.text, source: pair.source }))),
      }]
    : [];
  process.stdout.write(JSON.stringify({ edits, verdicts: [], notes: [] }) + "\\n");
  process.exit(0);
}
process.exit(0);
`,
);
fs.chmodSync(fakeAcpx, 0o755);

function git(args, cwd) {
  execFileSync("git", args, { cwd, stdio: "ignore" });
}

/** A monorepo checkout with a root memory file and one nested one. */
function makeMonorepo({ config = null } = {}) {
  const dir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "backpass-nested-repo-")));
  const files = {
    "AGENTS.md": ROOT_MEMORY,
    "apps/api/AGENTS.md": API_MEMORY,
    "apps/api/src/routes/orders.ts": "export const orders = 1;\n",
    "apps/web/src/checkout.tsx": "export const checkout = 1;\n",
    ...(config ? { ".backpassrc.json": `${JSON.stringify(config)}\n` } : {}),
  };
  for (const [name, text] of Object.entries(files)) {
    fs.mkdirSync(path.dirname(path.join(dir, name)), { recursive: true });
    fs.writeFileSync(path.join(dir, name), text);
  }
  git(["init", "-q", "-b", "main"], dir);
  git(["config", "user.email", "test@example.com"], dir);
  git(["config", "user.name", "test"], dir);
  git(["add", "-A"], dir);
  git(["commit", "-q", "-m", "monorepo"], dir);
  return dir;
}

/** A pi session rooted at the repo that edits one file, named by a repo-relative path. */
function writeSession(home, id, cwd, { quote, file }) {
  const dir = path.join(home, ".pi", "agent", "sessions", id);
  fs.mkdirSync(dir, { recursive: true });
  const entries = [
    { type: "session", version: 3, id, timestamp: new Date().toISOString(), cwd },
    { type: "message", message: { role: "user", content: quote } },
    {
      type: "message",
      message: {
        role: "assistant",
        content: [
          { type: "text", text: "Reading the file first." },
          { type: "toolCall", id: `${id}-read`, name: "read", arguments: { path: file } },
        ],
      },
    },
    { type: "message", message: { role: "toolResult", toolCallId: `${id}-read`, content: "export const x = 1;" } },
    {
      type: "message",
      message: {
        role: "assistant",
        content: [{ type: "toolCall", id: `${id}-edit`, name: "edit", arguments: { path: file, edits: [] } }],
      },
    },
    { type: "message", message: { role: "toolResult", toolCallId: `${id}-edit`, content: "edited" } },
    { type: "message", message: { role: "user", content: "Now run the tests too." } },
    { type: "message", message: { role: "assistant", content: "Tests pass." } },
  ];
  fs.writeFileSync(path.join(dir, `${id}.jsonl`), `${entries.map((e) => JSON.stringify(e)).join("\n")}\n`);
}

/** Two sessions that worked under apps/api, two under apps/web. */
function makeCorpus(dir) {
  const home = fs.mkdtempSync(path.join(os.tmpdir(), "backpass-nested-home-"));
  writeSession(home, "api-one", dir, { quote: API_QUOTE, file: "apps/api/src/routes/orders.ts" });
  writeSession(home, "api-two", dir, { quote: API_QUOTE, file: "apps/api/src/routes/orders.ts" });
  writeSession(home, "web-one", dir, { quote: WEB_QUOTE, file: "apps/web/src/checkout.tsx" });
  writeSession(home, "web-two", dir, { quote: WEB_QUOTE, file: "apps/web/src/checkout.tsx" });
  return home;
}

function run(dir, home, args, extraEnv = {}) {
  const statePath = path.join(home, "fake-state.json");
  const result = spawnSync(
    process.execPath,
    [
      CLI,
      ...args,
      "--harness",
      "pi",
      "--since",
      "all",
      "--analysis-agent",
      "pi",
      "--synthesis-agent",
      "claude",
      "--synthesis-model",
      "fake-model",
      "--jobs",
      "1",
    ],
    {
      cwd: dir,
      encoding: "utf8",
      timeout: 60000,
      env: {
        ...process.env,
        HOME: home,
        USERPROFILE: home,
        PATH: `${binDir}${path.delimiter}${process.env.PATH}`,
        BACKPASS_ACPX_BIN: fakeAcpx,
        FAKE_STATE: statePath,
        NO_COLOR: "1",
        CI: "1",
        ...extraEnv,
      },
    },
  );
  const log = fs.existsSync(statePath) ? JSON.parse(fs.readFileSync(statePath, "utf8")).log : [];
  return { ...result, output: `${result.stdout}${result.stderr}`, log };
}

function status(dir, args = []) {
  const r = spawnSync(process.execPath, [CLI, "status", ...args], { cwd: dir, encoding: "utf8", env: { ...process.env, NO_COLOR: "1", CI: "1" } });
  return r;
}
const show = (title, r) => { console.log(`\n===== ${title} (exit ${r.status}) =====`); console.log(r.stdout); if (r.stderr.trim()) console.log("[stderr]\n" + r.stderr); };
const which = process.argv[2];
const readProposal = (dir) => JSON.parse(fs.readFileSync(path.join(dir, ".backpass", "proposal.json"), "utf8"));
const reachFile = (dir) => path.join(dir, ".backpass", "nested", "reach.json");

if (which === "sparse") {
  const dir = makeMonorepo({ config: { nestedMemoryFiles: ["apps/api/AGENTS.md"] } });
  const home = fs.mkdtempSync(path.join(os.tmpdir(), "bp-live-home-"));
  writeSession(home, "api-one", dir, { quote: API_QUOTE, file: "apps/api/src/routes/orders.ts" });
  writeSession(home, "web-one", dir, { quote: WEB_QUOTE, file: "apps/web/src/checkout.tsx" });
  writeSession(home, "web-two", dir, { quote: WEB_QUOTE, file: "apps/web/src/checkout.tsx" });
  show("backpass status (before any run)", status(dir));
  console.log("reach.json exists after status:", fs.existsSync(reachFile(dir)));
  const r = run(dir, home, []);
  show("backpass (full run, run summary)", r);
  const p = readProposal(dir);
  console.log("proposal.nested[0] =", JSON.stringify({ ...p.nested[0], budget: undefined }));
  console.log("proposal edit files =", JSON.stringify(p.edits.map((e) => [e.id, e.kind, e.file])));
  console.log("reach.json =", fs.readFileSync(reachFile(dir), "utf8"));
  const mtime = fs.statSync(reachFile(dir)).mtimeMs;
  show("backpass status", status(dir));
  show("backpass status --json", status(dir, ["--json"]));
  console.log("reach.json unchanged by status:", fs.statSync(reachFile(dir)).mtimeMs === mtime);
  // adversarial: corrupt reach.json
  fs.writeFileSync(reachFile(dir), "{not json");
  show("backpass status with corrupt reach.json", status(dir));
  fs.writeFileSync(reachFile(dir), JSON.stringify({ measuredAt: "x", files: "nope" }));
  show("backpass status with malformed reach.json (files not array)", status(dir));
} else if (which === "floor") {
  const dir = makeMonorepo({ config: { nestedMemoryFiles: ["apps/api/AGENTS.md"] } });
  const home = makeCorpus(dir);
  const r = run(dir, home, []);
  show("backpass (at floor: 2 api of 4)", r);
  show("backpass status", status(dir));
} else if (which === "analyze") {
  const dir = makeMonorepo({ config: { nestedMemoryFiles: ["apps/api/AGENTS.md"] } });
  const home = fs.mkdtempSync(path.join(os.tmpdir(), "bp-live-home-"));
  writeSession(home, "api-one", dir, { quote: API_QUOTE, file: "apps/api/src/routes/orders.ts" });
  writeSession(home, "web-one", dir, { quote: WEB_QUOTE, file: "apps/web/src/checkout.tsx" });
  const r = run(dir, home, ["analyze"]);
  show("backpass analyze", r);
  console.log("reach.json exists after analyze:", fs.existsSync(reachFile(dir)));
  show("backpass status", status(dir));
} else if (which === "plain") {
  const dir = makeMonorepo();
  const home = makeCorpus(dir);
  const r = run(dir, home, []);
  show("backpass (no nested config)", r);
  show("backpass status", status(dir));
  const j = JSON.parse(status(dir, ["--json"]).stdout);
  console.log("status --json has nestedAttribution key:", "nestedAttribution" in j);
  console.log("reach.json exists:", fs.existsSync(reachFile(dir)));
}

Evidence: Run summary excerpt

nested apps/api/AGENTS.md · 1 of 3 sampled session(s) worked only under apps/api/ · budget [....] 14 -> 14 / 5,000 tok
below minGapEvidence (2); no add, rewrite, or remove edit can reach this file

e1 ADD Build the web bundle before opening a pull request. (+14 tok, 2 transcript(s))

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 8 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Before any run, `backpass status` lists each nested file as 'not yet measured' and writes no state	✅ pass	live	nested-reach-cli-transcript.txt section 'backpass status (before any run)'; reach.json exists after status-only: false
A full `backpass` run shows each nested file's 'N of M sampled session(s) worked only under dir/' and warns only for the file below minGapEvidence	✅ pass	live	nested-reach-cli-transcript.txt 'backpass (full run)': api 1 of 4 + warning, web 2 of 4 without a warning
Report-only: the root edit is still proposed while a nested file is below the floor	✅ pass	live	proposal.json edits e1 (AGENTS.md) and e2 (apps/web) with api belowMinGapEvidence:true; proposal nested[] has sampledSessions/minGapEvidence/belowMinGapEvidence
Adversarial: a cross-cutting session (reads apps/api, edits apps/web) counts toward neither nested file	✅ pass	live	Out of 4 sessions, api reach is 1 and web reach is 2, so the cross-one session is excluded from both
After a run, `backpass status` shows the saved, timestamped counts and the warning without re-measuring	✅ pass	live	'backpass status' section shows the header with timestamp 2026-10-15 15:47; reach.json mtime unchanged by status: true
`backpass status --json` exposes nestedAttribution with measuredAt and per-file fields	✅ pass	live	'backpass status --json' section: nestedAttribution.files entries carry measured, sessions, sampledSessions, minGapEvidence and belowMinGapEvidence
`backpass analyze` alone records reach that status then shows	✅ pass	live	repo 2 sections: analyze then status shows 'apps/api/AGENTS.md: 3 of 4 sampled session(s)' with no warning
A repo with no nested files configured shows no reach section and no nestedAttribution key	✅ pass	live	'backpass status (no nested configured)' section; json has nestedAttribution: false

node --test --test-name-pattern="nested|status" test/nested-cli.test.js test/status.test.js (12 pass, including the new CLI floor-warning test and the status reach test)
node $EV/drive.mjs $EV: real CLI driver (helpers copied from test/nested-cli.test.js) running backpass status, backpass, backpass status, backpass status --json, and backpass analyze in disposable repos and HOMEs
Adversarial: a cross-cutting pi session that reads an apps/api file and edits an apps/web file, checked to count toward neither nested file
Checked that status neither creates nor modifies reach.json (existence before any run; mtime after a run)
Checked a repo with no nested files configured: no NESTED EVIDENCE REACH section and no nestedAttribution JSON key

✅ No issues found.

Live validation: ✅ go - 9 of 9 scenarios driven live against the product

Scenario	Result	Live	Evidence
Before any run, backpass status lists the nested file as 'not yet measured' and writes no reach.json	✅ pass	live	sparse.txt: 'apps/api/AGENTS.md: not yet measured - run `backpass` or `backpass analyze`'; 'reach.json exists after status: false'
A full backpass run where the nested file gets 1 of 3 sessions prints the count and the below-minGapEvidence warning in the run summary	✅ pass	live	sparse.txt run summary; proposal.nested[0] has sampledSessions 3, minGapEvidence 2, belowMinGapEvidence true
Report-only: the root edit earned by the web sessions is still proposed while the nested file is below the floor	✅ pass	live	sparse.txt: proposal edit files = [["e1","add","AGENTS.md"]]
After the run, backpass status shows the timestamped reach section with the warning, and status --json carries nestedAttribution	✅ pass	live	sparse.txt: 'NESTED EVIDENCE REACH (sampled sessions at the last attribution, 2026-10-15 16:49; report-only)' plus the JSON nestedAttribution block
status reads the saved count without rewriting reach.json	✅ pass	live	sparse.txt: 'reach.json unchanged by status: true'
At the floor (2 of 4 sessions), the count is shown with no warning in both the run summary and status	✅ pass	live	floor.txt
backpass analyze alone records the count, and status then shows it	✅ pass	live	analyze.txt: 'reach.json exists after analyze: true'; status shows '1 of 2' with the warning
Adversarial: a corrupt or malformed reach.json does not make status fail; it falls back to 'not yet measured'	✅ pass	live	sparse.txt: both corrupt-file status runs exit 0 and show 'not yet measured'
A repo without nested memory files shows no reach section, no nestedAttribution key, and writes no reach.json	✅ pass	live	plain.txt: 'status --json has nestedAttribution key: false', 'reach.json exists: false'

node drive.mjs sparse (evidence driver): status before any run, then a full backpass run with 1 api session and 2 web sessions, then status, status --json, and status with corrupt and malformed reach.json
node drive.mjs floor: full backpass run with 2 api sessions and 2 web sessions (at the floor), then status
node drive.mjs analyze: backpass analyze alone writes reach.json, then status
node drive.mjs plain: run with no nested memory files configured, then status and status --json
node --test test/nested-cli.test.js test/status.test.js

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

greptile-apps · 2026-10-15T15:51:11Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable defects were found.

_{Reviews (2) · Last reviewed commit: "feat(status): show each nested memory fi..." · Reviewed by Greptile}

A nested file trains only on sessions whose every in-repo work path stays under its directory, so in repos where sessions cross directories it may never reach minGapEvidence, and a run that proposes nothing there looked the same as a file with no gaps. Each attribution now records, per nested file, how many sampled sessions worked only under its directory (.backpass/nested/reach.json). The run summary and `backpass status` print that count and warn below minGapEvidence, when no add, rewrite, or remove edit can reach the file. `status --json` reports it as nestedAttribution, and the proposal's nested entries carry sampledSessions, minGapEvidence, and belowMinGapEvidence. The count comes from deterministic attribution and is report-only. Refs kunchenguid#191 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kunchenguid · 2026-10-16T04:40:45Z

Speaking as Kun's firstmate:

Merged as 8bb390b (squash). Thank you, @Pantoria — clear report-only nested reach on status / run summary, solid live evidence (9/9), and a clean no-mistakes attestation.

contract-class: restore. Part (1) of #191: Failure-is-loud already requires a run that proposes nothing to explain why; showing each nested file's sampled-session reach (and warning below minGapEvidence) restores that visibility without gating. Report-only; extract/move wording kept accurate. Part (2) of #191 (opt-in ignore memory-file reads as work paths) stays open for a follow-up.

VISION (per rule) @ pre-merge f24dbab / tip now 8bb390b:

Evidence is the only currency — aligns (deterministic workedUnder counts only).
The human owns the weights — aligns (no write / review-gate change).
Nothing the model says is taken on faith — aligns (no model arithmetic; measured attribution).
The budget is the constraint — aligns (no always-loaded growth).
It reads what you already have and owns nothing — aligns.
Failure is loud and named — aligns (names "below floor" vs "no gaps").
Scope — aligns (status / proposal presentation only; Presentation never gates).

CI green (ubuntu + macos build-and-test, generated-files guard, no-mistakes). Live validation already in the PR — no further ask. Refs #191 (does not close; part 2 remains).

Pantoria mentioned this pull request Oct 15, 2026

Nested memory files: show when a file cannot collect evidence under the all-paths attribution rule #191

Closed

Pantoria force-pushed the feat/191-nested-attribution-count branch from 70b7fe3 to 1fde44a Compare October 15, 2026 16:50

kunchenguid merged commit 8bb390b into kunchenguid:main Oct 16, 2026
5 checks passed

github-actions Bot mentioned this pull request Oct 15, 2026

chore(main): release backpass 0.1.35 #215

Merged

Navigation Menu

Repository navigation

feat(status): show each nested memory file's sampled-session reach - #209

feat(status): show each nested memory file's sampled-session reach#209
kunchenguid merged 1 commit into
kunchenguid:mainfrom
Pantoria:feat/191-nested-attribution-count

Pantoria commented Oct 15, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

kunchenguid commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Pantoria commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

greptile-apps Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Uh oh!

Uh oh!

kunchenguid commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pantoria commented Oct 15, 2026 •

edited

Loading

greptile-apps Bot commented Oct 15, 2026 •

edited

Loading