diff --git a/docs/dogfooding.md b/docs/dogfooding.md index 2c39ca7..331da38 100644 --- a/docs/dogfooding.md +++ b/docs/dogfooding.md @@ -65,6 +65,339 @@ Config layers are `defaults ← user ← project ← env`, so an environment var beats the file (`PI_BGRUN_SHOW_COMPLETED=0`, `PI_BGRUN_MAX_LOG_BYTES=0`). To drop the setup, delete `$CONFIG_DIR/pi-bgrun.json`; nothing else depends on it. +## How these numbers are produced + +Every number below comes from a session transcript, not from an impression of +what the tool does. The method is fixed here so a later run is comparable to +these ones — and because a fixture we can tune is also a fixture we could tune +in our own favour, it is fixed *before* the runs, predictions included. + +### The protocol + +| element | definition | +|---|---| +| arms | **vanilla**: `pi -ne -ns` plus the provider extension. **bgrun**: the same, plus `-e extension/index.ts`. **vanilla-hinted** (control, below) | +| model | one provider and model per comparison (`bifrost-openai/fireworks/deepseek-v4.1-flash` here) | +| prompt | identical within a pair except the mechanism clause: `Run '' in this repo and report the failure details.` vs `…as a background job with bgrun, and report the failure details once it finishes.` | +| sessions | **live** for every bgrun arm: a one-shot `pi -p` exits before the wake is delivered, so those runs need a human present. Vanilla arms block and report on their own, so they are scripted | +| samples | 3 per arm, reported as median **and** range. A single run is stated as such | +| metrics | from the transcript: suite executions, blocked-on-output, idle, session wall, context chars, per-call (wait, size), the exact commands, and whether the failure diagnostic reached context at all | +| re-derive | `bun scripts/measure-sessions.ts ` — the committed profiler, so any table here can be recomputed from the transcripts | + +The **vanilla-hinted** control matters: it asks vanilla pi to redirect the run to +a file and grep it. That is the technique bgrun implements, hand-rolled by the +agent, and it is the strongest baseline in the set — if bgrun cannot beat it, the +tool is a convenience rather than a capability. + +### Pre-registered expectations + +Written down before the runs, including the cells where the tool is expected to +*lose*. Cells that contradict their prediction are findings, not failures to +explain away. + +| cell | scenario | predicted winner | of what, and what would falsify it | +|---|---|---|---| +| R1 | the repo's own suite: green, 224 tests, ~22s, ~340 lines | **vanilla** | nothing blocks and the output is small; bgrun adds a job, a wake and a read. Falsified if the digest makes the bgrun session cheaper overall | +| R2 | red, failure **78–90 lines from the end** | **vanilla** | a `tail -40` window hits it, one call, no wait — the pytest/jest shape | +| R3 | generated suite at the **standard density**: ~180s, **694** lines, failure 65% through | **bgrun** | the failure sits ~250 lines from the end, so no `head`/`tail` window reaches it and vanilla buys a second full run. The long-suite row measured below used the *verbose* density and is being re-run under this one | +| R4 | **same volume, no time**: generated with `DUMMY_SLEEP_MS=20`, `DUMMY_LINES_PER_TEST=8` | **bgrun** on completeness, **vanilla** on cost | isolates volume from duration. If one or two greps find the failure, vanilla is cheaper and bgrun's advantage is time, not size | +| R5 | **fail-fast**: `DUMMY_FAIL_FAST=1`, the run collapses at the failure | **vanilla** | when a failure ends the work early the blocking cost is small, and the wake is pure overhead | +| C1 | start the long suite, then do an unrelated ~60s task in the same session | **bgrun** | overlap is the actual niche: wall ≈ max(180, 60) against 240. **Falsified if the agent waits anyway** — a plausible outcome worth measuring | +| C2 | three long suites at once | **bgrun** | a foreground agent serialises them (≈3 × 180s); jobs run concurrently | +| C3 | restart the session mid-run | **bgrun** | the log outlives the session; a foreground run's output dies with it | +| C4 | "which part is it on right now?" | **bgrun** | `bgtail` answers mid-run; a foreground command cannot be asked | +| D1 | a weaker or faster model | **unknown** | hypothesis: if bgrun's value is reliability (no guessing, no re-running), it should grow as the agent weakens. If the advantage holds only for the strong model, that story is wrong | + +C1–C4 are dialogues rather than fixtures: no generated suite can express them, +because the thing under test is what the session does *while* a job runs and what +survives afterwards. + +### The fixture, and why it is shaped that way + +`bun run bench:make-suite` generates the long, noisy, failing suite +(`scripts/make-dummy-suite.ts`). It is generated rather than committed because +`bun test` discovers `*.test.ts` anywhere in the tree and does **not** respect +`.gitignore` (measured) — a committed fixture would run inside every developer's +bare `bun test` and cost them minutes. So it defaults outside the repo, and the +script refuses to write inside it, fatally, because it clears its output +directory before writing and could otherwise delete tracked files. + +What it builds, and why each property is the thing under test: + +- **Long**: 300 tests over 10 files, ~180s by default. Every test awaits one + chain shared through `globalThis`, so wall-clock is `sleep × tests` whatever + the runner's scheduling does — bun 1.3.6 happens to run them sequentially, and + a concurrent scheduler would otherwise collapse a naive per-test sleep. +- **Noisy**: each test logs a tunable number of lines. The default is **realistic** + (2, against bun's own ~1.5 lines per test); `DUMMY_LINES_PER_TEST=8` reproduces + the verbose density the earliest measurements here used, which matters when + comparing against them. Measured: **694** output lines at the default density, + **2,494** at `=8`. +- **Failing in the middle**: one planted test asserts a marker pair at 65% of the + run, so no `head`/`tail` window reaches it and the failure's *position* is under + test rather than its existence. The 65% is measured, and now explained: bun 1.3.6 + discovers these files in a **non-lexical** order (`2,3,1,8,9,10,5,4,6,7`, + measured twice), so `part-05` runs seventh of ten while `DUMMY_FAIL_FILE=5` + reads as 45%. +- **Honest failure shape**: the failure block looks like a real assertion failure + (assertion, expected vs received, stack naming the generated file and line, and a + footer — ~19 lines) rather than a two-line stub, so the output volume a red run + really produces is not understated. +- **Knobs for the negative cells**: `DUMMY_FAIL_FAST=1` ends the run at the + failure, which is the regime where a background job should have nothing to + offer; `DUMMY_FILES`/`DUMMY_TESTS_PER_FILE`/`DUMMY_FAIL_FILE`/`DUMMY_FAIL_STEP` + move the size and the failure's position. Measured at smoke scale: fail-fast + finished in **4.09s** against **6.29s**, i.e. it stops at the fraction of the run + that had already completed before the failure in bun's order — 65%, not the 45% + a lexical order would give. + +The script validates its own fixture: the planted failure must exist, or the +generated run would silently invalidate the measurement it was made for. + +### What this method cannot see + +- It is one machine, one checkout, one model, and mostly n=3. The spread across + runs is large enough that single numbers are anecdotes. +- Context is counted in characters of transcript text, not billed tokens. +- The fixture is synthetic. Real suites fail in ways this one does not: flaky + tests, skips, retries, parallel workers, enormous stack traces. +- Live sessions make bgrun's cells expensive to reproduce — a reviewer cannot + re-run them unattended, which is itself a property of the tool worth naming. +- The metrics split into two kinds, and only one is stable: **executions and + blocked time are the mechanism's** (the agent cannot block on a job that runs + detached), while **context and wall-clock are the agent's** (it chooses what to + read and when to look). Stable columns are evidence about `bgrun`; unstable + ones are evidence about the agent's use of it, and the reports below say which + is which. + +### Running a cell by hand + +The vanilla cells are scripted (`pi -ne -ns -e -p --session-dir -n