Repository navigation
Epic: apr-dogfood v3.0 — coverage is the organizing gate (17.1% of 830 features, 27 of 28 binaries at zero) #2582
Description
owner: aprender-w5404 (re-owned 2026-10-19 by aprender-triage-worker at the cop's word: previous owner session absent; picks up children only as batch constituents after its current timed task)
Tracking epic for the apr-dogfood v3.0 rewrite. The four artifacts exist and are
verified; this issue exists so the reasoning survives them.
The finding that drove the rewrite
A four-part surface audit gave the gate suite a denominator it never had:
| metric | value |
|---|---|
| features covered | 142 / 830 = 17.1% |
| binaries with any coverage | 1 of 28 — all 142 are in apr |
| largest uncovered binary | aprender-orchestrate (184 features), then aprender-test-cli (39), pv (38) |
| coverage, quality band 1–2 | 72.5% |
| coverage, quality band 5–6 | 4.3% |
| live ledger defect (quality ≤ 4) and no gate | 44 features, all in apr |
verified_hardware=UNKNOWN |
427 rows |
confidence=low and uncovered |
204 rows |
Read the bands correctly. 644 features scored 6 because no ledger finding exists
and no gate covers them. That is not "fine" — it is unlooked-at. Where gates
actually look, they find defects at 72.5%. The 4.3%/72.5% split is not evidence that
the uncovered surface is healthy; it is evidence that nobody has looked.
That inverts the organizing principle. v2.0 was 19 gates in search of things to check.
v3.0's organizing gate is coverage itself (Phase 2): the CSV is the denominator, the
runtime enumeration must reconcile against it in both directions, and coverage may
never decrease.
Artifacts
| file | destination | size |
|---|---|---|
SKILL.md v3.0 |
.claude/skills/apr-dogfood/SKILL.md |
631 lines (v2.0 was 828) |
surface_audit.csv |
docs/audits/surface_audit.csv |
831 lines = 830 rows + header, 8 cols, evidence path per row |
dogfood.sh |
scripts/dogfood.sh |
1147 lines, ~24 gates, receipt-emitting |
invariance.py |
scripts/invariance.py |
136 lines, transport-invariance |
Integrity verified on receipt: zero U+FFFD replacement characters, line counts match
stated values exactly.
Constraints that must not be violated
- Do not re-derive Tier 1 P1–P12 or Tier 2 T2.1–T2.9. v3.0 carries them by
reference only — its author read the headings and falsifier IDs of v2.0's Gates
2–18, not their bodies. Copy those bodies verbatim. A re-derived body silently
loses the specific mutation each was hardened against, which is the entire reason
they are worded as they are. Splice points are marked in-file at lines 164, 346–347,
and the caveat at 607–609 (delete the caveat once satisfied, or a completed merge
looks permanently outstanding). - Every line reference is UNVERIFIED until checked at HEAD. Two were already stale
on arrival (below). If one has moved, fix the reference, do not delete the claim. - Every number is a snapshot. Recompute from the CSV and from HEAD. If the reading
disagrees, the reading wins, and the discrepancy is recorded rather than passed
through. - The G2.3 threshold table is deliberately blank. Measure today's values, commit
them as the baseline, ratchet from there. Do not pick a round number. - Ticket first (
pmat work add) → branch → PR → greenci / gate.pvcontract in the
same PR as the implementation. pmat query, not grep.- The skill is read-only in operation. apr-dogfood audits; it never modifies files.
Defects become issues and tickets.
Verification ledger — claims checked at HEAD (4bbfeb0)
| claim | attached | HEAD | verdict |
|---|---|---|---|
serve run --backend has no value_parser |
true | serve_commands.rs:73 is exactly #[arg(long, value_name = "BACKEND")], 0 BACKEND_VALUES in the file |
✅ VERIFIED |
path src/commands/serve_commands.rs |
— | actual: src/serve_commands.rs |
|
commands_enum.rs:110 is the backend decl |
— | 110 is profile: bool; real decl is :154 #[arg(… value_parser = BACKEND_VALUES)], field :155, const :10 |
|
extended_commands.rs:85 |
field | correct; parser at :84 |
✅ VERIFIED |
serve/types.rs:57 consumes it |
true | pub backend: Option<String> |
✅ VERIFIED |
apr eval --task: 10 arms, 2 documented |
8 undocumented | exactly 8 — humaneval, mbpp, code, contamination, compare, verify, correlation, human, plan (arms at dispatch_analysis.rs:1383-1419; help at extended_commands.rs:138) |
✅ VERIFIED |
two apr bins duplicated by hand |
already diverged | src/bin/apr.rs:9-10 delegates to apr_cli::cli_main(); apr-cli/src/main.rs re-implements the prologue; dhat #[global_allocator] exists only at apr-cli/src/main.rs:10-12 |
✅ VERIFIED |
Flagging staleness is a success, not a failure. Two of four G1.3 references had moved
and the substantive claim survived both — which is the ledger working.
Work
- G1.3 —
apr serve run --backendaccepts any string.apr runandapr chat
pinBACKEND_VALUES; serve does not, and the value is consumed. The constant's own
doc-comment says it exists to prevent "a run whose throughput number was taken
through a backend the caller did not ask for" — reproduced on the serve path. The
doc also claims it covers "apr run / apr chat": there are three declaration
sites, not two. Separately:apr benchhas no--backendat all, despite
being the throughput command. - G1.4 — 8 reachable, undocumented
--taskarms. - G0.5 — the two
aprbin targets are the same program by hand-duplication, not
by sharing, and have already diverged. - CUDA doc falsehood —
dispatch.rs:180-182tells the user only the root crate's
feature chain can enable the CUDA path. False:apr-cli/Cargo.toml:89defines
cudadirectly andqwen-story-daily.yml:63builds the nightly CUDA binary via
-p apr-cli. - G2.2 — reconcile the denominator, both directions, and silently reconcile
neither. In-binary-not-in-CSV means the ledger is stale; in-CSV-not-in-binary means
the surface shrank unnoticed. - G2.3 — measure and commit
contracts/apr-dogfood-coverage-v1.yaml(author it).
Every value cites the command that produced it. This is the item that turns v3.0
from a document into a ratchet. - Coverage gate — blocking. Fails when coverage drops below baseline, or when a
quality-≤-4 feature has no gate and no waiver. - Enumerate the 44 — each needs an open issue or an explicit written waiver
before the next release.
Registered mutations
Each must be observed RED, with the pre-fix GREEN captured as before-evidence in the
PR body, plus a discrimination check (RED for the mutation, GREEN for a no-op rebuild
of the same commit — a gate that fires on both is not measuring what it claims).
| gate | mutation → RED |
|---|---|
| G2.1 freshness | backdate the CSV behind HEAD |
| G2.2 reconciliation | delete one CSV row for a live command |
| G2.3 floors | flip one in_dogfood_skill from yes to no |
Prove each mutation ENGAGED before trusting its verdict. A sed that silently fails
to match reads exactly like "the gate is fine". This has bitten twice today — once where
a replacement hit a comment quoting the literal rather than the code.
Anti-vacuity — a lesson paid for today, applies directly here
The multi-platform dogfood gate (#2573) held its host list and its anti-vacuity floor
as literals in the same file. One commit editing both to a smaller matrix exited 0 —
measured. The original mutation proved the floor fires; it never proved the floor could
not be lowered.
Root cause, from five rounds on the competitive-parity ledger:
Any state the author writes and the gate reads can be moved in the same commit.
The resolution is that the comparand lives on protected main — git show origin/main:<file> — which a PR cannot rewrite in its own commit. The coverage
baseline must be read from origin/main, not from the working tree.
The #2332 name-collision hazard — preserve the frontmatter comment
Without an explicit name:, the skill takes its name from its directory, collides with a
user-scope ~/.claude/skills/dogfood/, and never appears in the session's skill
listing — it cannot be invoked and nothing warns. Edits look effective and change
nothing that runs. That is exactly what happened when #2357 hardened Gates 1 and 13.
Open design question — escalate, do not default
The four-host benchmark gate (lambda-labs, gx10, mini, intel) has no obvious CI
home: lambda-labs was retired as a runner 2026-05-10 and marked do-not-revive; mini
is cowork-first with selective CI; intel is the clean-room runner the benchmark would
contend with. Four options in APR-BENCH-RFC-001 §7.3 (paiml/infra#259). Pick one,
record the rationale in the ticket, and if the ledger-based option is chosen, ship the
staleness arm — otherwise the gate measures declared state rather than resolved
state.
Definition of done, per item
- Merged to
mainvia greenci / gate - A gate exists that would have caught the original gap
- The named mutation applied and observed RED, pre-fix GREEN captured in the PR body
pvcontract in the same PR- Discrimination confirmed — GREEN on a no-op change
- Any doc claim the change invalidates updated in the same PR
Related
- paiml/infra#259 —
APR-BENCH-RFC-001, the four-host benchmark gate - feat(release): dogfood every supported platform before a cut, and prove it happened #2573 — multi-platform dogfood gate (Gate 12) and its anti-vacuity fix
- Cross-platform dogfood: aarch64 Linux and arm64 macOS verified — Q4_K GEMV is scalar on ARM, OOM guard fails open on macOS #2566 — cross-platform dogfood findings (aarch64, macOS)
Folded sub-tickets (TRIAGE-100, 2026-10-14)
These open sub-tickets were folded here as checklist rows and closed as not planned so the open count stays at or under 100. Reopen one when its work resumes; an open PR for it reopens it.
- The dogfood release gate is unrunnable for every sibling repo whenever aprender is not on main #2701 — The dogfood release gate is unrunnable for every sibling repo whenever aprender is not on main (branch:
fix/2701-dogfood-canon-ref,fix/2701-dogfood-canon-ref-port) - SafeTensors serve route drops
ignore_eosandseed: apr-cli's ChatCompletionRequest has no such fields #2853 — SafeTensors serve route dropsignore_eosandseed: apr-cli's ChatCompletionRequest has no such fields (branch:fix/2853-safetensors-ignore-eos,fix/2853-safetensors-ignore-eos-seed)
Activity
noahgift commentedon Aug 22, 2026
Clustering lands — the denominator now has structure, and the zeros are subsystems
830 rows clustered at k=14. Every number recomputed from the CSV and verified exact.
Nine clusters at exactly 0%, covering 468 features. The zeros are not scattered — they
are whole subsystems: every Banco HTTP route, the entire data pipeline, all of
pv, thewhole agent stack.
What this changes about the work
"The 44" is one work package, not 44 tickets. Cluster 10 holds 36 of 44; the rest
are six strays. That reframes it from a scattered backlog into a bounded domain.
Per-cluster is a better ratchet than per-binary, and the reason is concrete:
aprender-orchestrate's 184 features split into three genuinely different subsystems —Banco HTTP (95), agent stack (56), Pacha secrets (17) — which have different failure modes
and should be gated separately. A binary-level number hides all three behind one figure.
Adopting the proposed G2.3 addition: no cluster may sit at 0% at
--release. Thatconverts nine zeros into nine bounded tickets rather than one unbounded aspiration.
Two zeros that are qualitatively different, filed separately
contracts-pv, 76 features, 0% covered, 0 UNKNOWN. All hardware-verified,none gated. The tool that enforces provable contracts everywhere else proves nothing
about itself — recursive self-application failing on the most load-bearing binary in the
stack. Three defects found in it this week alone (pv validate accepts competitor: THIS-DOES-NOT-EXIST, demand_score: 99999, intake_status: banana — 0 errors #2555, fix/pv schema #2554, fix(pv): pin pv to a HEAD build — the release receipt was citing pv 0.49.0 #2552), which is the
expected rate for 76 ungated features, not bad luck.
orchestrate-pacha-secrets, 17 features, 0%, 17/17 UNKNOWN.keygen,encrypt,decrypt,sign. Every other zero cluster is uncovered surface; this oneis uncovered crypto, where failures are silent by construction — a verifier that
accepts everything passes its positive test perfectly. I would gate this first among the
nine, ahead of the larger
http-orchestrate-banco, precisely because its failures arequiet.
Method notes worth preserving
k=14 from the elbow; silhouette climbs to 0.26 at k=44 without settling, meaning the
structure is hierarchical, not naturally k-partitioned. The ledger independently
arrived at 37 root-cause clusters, so k≈37 is the finer-grained alternative if the two
should line up.
The module-stem signal is the one that matters — it is the only input that is not a
restatement of the feature name. Transport prefix kept deliberately: a route and a
subcommand sharing a noun are different features with different failure modes.
cluster_labelis the durable key;cluster_idis incidental and stable only for a fixedinput.
cluster.pyis deterministic (random_state=0) so the assignment is reproduciblewhen the surface moves.
fix(apr-cli): the two `apr` bins were one program by hand-copying, an…
ignore_eosandseed: apr-cli's ChatCompletionRequest has no such fields #2853ignore_eosandseed: apr-cli's ChatCompletionRequest has no such fields #2853noahgift commentedon Oct 19, 2026
Triage evidence 2026-10-19 17:15Z: all children named for this epic are closed: #2701 #4861 #2853 #1293. The epic body lists no open child issue. Owner label is owner:aprender-a7 (session not live). Closing is left to the owner.