Skip to content

Epic: apr-dogfood v3.0 — coverage is the organizing gate (17.1% of 830 features, 27 of 28 binaries at zero) #2582

Open
9 of 10 issues completed

Description

@noahgift

owner: aprender-w5404 (re-owned 2026-10-19 by aprender-triage-worker at the cop's word: previous owner session absent; picks up children only as batch constituents after its current timed task)

Tracking epic for the apr-dogfood v3.0 rewrite. The four artifacts exist and are
verified; this issue exists so the reasoning survives them.

The finding that drove the rewrite

A four-part surface audit gave the gate suite a denominator it never had:

metric value
features covered 142 / 830 = 17.1%
binaries with any coverage 1 of 28 — all 142 are in apr
largest uncovered binary aprender-orchestrate (184 features), then aprender-test-cli (39), pv (38)
coverage, quality band 1–2 72.5%
coverage, quality band 5–6 4.3%
live ledger defect (quality ≤ 4) and no gate 44 features, all in apr
verified_hardware=UNKNOWN 427 rows
confidence=low and uncovered 204 rows

Read the bands correctly. 644 features scored 6 because no ledger finding exists
and no gate covers them. That is not "fine" — it is unlooked-at. Where gates
actually look, they find defects at 72.5%. The 4.3%/72.5% split is not evidence that
the uncovered surface is healthy; it is evidence that nobody has looked.

That inverts the organizing principle. v2.0 was 19 gates in search of things to check.
v3.0's organizing gate is coverage itself (Phase 2): the CSV is the denominator, the
runtime enumeration must reconcile against it in both directions, and coverage may
never decrease.

Artifacts

file destination size
SKILL.md v3.0 .claude/skills/apr-dogfood/SKILL.md 631 lines (v2.0 was 828)
surface_audit.csv docs/audits/surface_audit.csv 831 lines = 830 rows + header, 8 cols, evidence path per row
dogfood.sh scripts/dogfood.sh 1147 lines, ~24 gates, receipt-emitting
invariance.py scripts/invariance.py 136 lines, transport-invariance

Integrity verified on receipt: zero U+FFFD replacement characters, line counts match
stated values exactly.

Constraints that must not be violated

  1. Do not re-derive Tier 1 P1–P12 or Tier 2 T2.1–T2.9. v3.0 carries them by
    reference only
    — its author read the headings and falsifier IDs of v2.0's Gates
    2–18, not their bodies. Copy those bodies verbatim. A re-derived body silently
    loses the specific mutation each was hardened against, which is the entire reason
    they are worded as they are. Splice points are marked in-file at lines 164, 346–347,
    and the caveat at 607–609 (delete the caveat once satisfied, or a completed merge
    looks permanently outstanding).
  2. Every line reference is UNVERIFIED until checked at HEAD. Two were already stale
    on arrival (below). If one has moved, fix the reference, do not delete the claim.
  3. Every number is a snapshot. Recompute from the CSV and from HEAD. If the reading
    disagrees, the reading wins, and the discrepancy is recorded rather than passed
    through.
  4. The G2.3 threshold table is deliberately blank. Measure today's values, commit
    them as the baseline, ratchet from there. Do not pick a round number.
  5. Ticket first (pmat work add) → branch → PR → green ci / gate. pv contract in the
    same PR as the implementation.
  6. pmat query, not grep.
  7. The skill is read-only in operation. apr-dogfood audits; it never modifies files.
    Defects become issues and tickets.

Verification ledger — claims checked at HEAD (4bbfeb0)

claim attached HEAD verdict
serve run --backend has no value_parser true serve_commands.rs:73 is exactly #[arg(long, value_name = "BACKEND")], 0 BACKEND_VALUES in the file ✅ VERIFIED
path src/commands/serve_commands.rs — actual: src/serve_commands.rs ⚠️ STALE path
commands_enum.rs:110 is the backend decl — 110 is profile: bool; real decl is :154 #[arg(… value_parser = BACKEND_VALUES)], field :155, const :10 ⚠️ STALE line
extended_commands.rs:85 field correct; parser at :84 ✅ VERIFIED
serve/types.rs:57 consumes it true pub backend: Option<String> ✅ VERIFIED
apr eval --task: 10 arms, 2 documented 8 undocumented exactly 8 — humaneval, mbpp, code, contamination, compare, verify, correlation, human, plan (arms at dispatch_analysis.rs:1383-1419; help at extended_commands.rs:138) ✅ VERIFIED
two apr bins duplicated by hand already diverged src/bin/apr.rs:9-10 delegates to apr_cli::cli_main(); apr-cli/src/main.rs re-implements the prologue; dhat #[global_allocator] exists only at apr-cli/src/main.rs:10-12 ✅ VERIFIED

Flagging staleness is a success, not a failure. Two of four G1.3 references had moved
and the substantive claim survived both — which is the ledger working.

Work

  • G1.3 — apr serve run --backend accepts any string. apr run and apr chat
    pin BACKEND_VALUES; serve does not, and the value is consumed. The constant's own
    doc-comment says it exists to prevent "a run whose throughput number was taken
    through a backend the caller did not ask for"
    — reproduced on the serve path. The
    doc also claims it covers "apr run / apr chat": there are three declaration
    sites, not two. Separately: apr bench has no --backend at all, despite
    being the throughput command.
  • G1.4 — 8 reachable, undocumented --task arms.
  • G0.5 — the two apr bin targets are the same program by hand-duplication, not
    by sharing, and have already diverged.
  • CUDA doc falsehood — dispatch.rs:180-182 tells the user only the root crate's
    feature chain can enable the CUDA path. False: apr-cli/Cargo.toml:89 defines
    cuda directly and qwen-story-daily.yml:63 builds the nightly CUDA binary via
    -p apr-cli.
  • G2.2 — reconcile the denominator, both directions, and silently reconcile
    neither. In-binary-not-in-CSV means the ledger is stale; in-CSV-not-in-binary means
    the surface shrank unnoticed.
  • G2.3 — measure and commit contracts/apr-dogfood-coverage-v1.yaml (author it).
    Every value cites the command that produced it. This is the item that turns v3.0
    from a document into a ratchet.
  • Coverage gate — blocking. Fails when coverage drops below baseline, or when a
    quality-≤-4 feature has no gate and no waiver.
  • Enumerate the 44 — each needs an open issue or an explicit written waiver
    before the next release.

Registered mutations

Each must be observed RED, with the pre-fix GREEN captured as before-evidence in the
PR body, plus a discrimination check (RED for the mutation, GREEN for a no-op rebuild
of the same commit — a gate that fires on both is not measuring what it claims).

gate mutation → RED
G2.1 freshness backdate the CSV behind HEAD
G2.2 reconciliation delete one CSV row for a live command
G2.3 floors flip one in_dogfood_skill from yes to no

Prove each mutation ENGAGED before trusting its verdict. A sed that silently fails
to match reads exactly like "the gate is fine". This has bitten twice today — once where
a replacement hit a comment quoting the literal rather than the code.

Anti-vacuity — a lesson paid for today, applies directly here

The multi-platform dogfood gate (#2573) held its host list and its anti-vacuity floor
as literals in the same file. One commit editing both to a smaller matrix exited 0 —
measured. The original mutation proved the floor fires; it never proved the floor could
not be lowered.

Root cause, from five rounds on the competitive-parity ledger:

Any state the author writes and the gate reads can be moved in the same commit.

The resolution is that the comparand lives on protected main — git show origin/main:<file> — which a PR cannot rewrite in its own commit. The coverage
baseline must be read from origin/main, not from the working tree.

The #2332 name-collision hazard — preserve the frontmatter comment

Without an explicit name:, the skill takes its name from its directory, collides with a
user-scope ~/.claude/skills/dogfood/, and never appears in the session's skill
listing
— it cannot be invoked and nothing warns. Edits look effective and change
nothing that runs. That is exactly what happened when #2357 hardened Gates 1 and 13.

Open design question — escalate, do not default

The four-host benchmark gate (lambda-labs, gx10, mini, intel) has no obvious CI
home
: lambda-labs was retired as a runner 2026-05-10 and marked do-not-revive; mini
is cowork-first with selective CI; intel is the clean-room runner the benchmark would
contend with. Four options in APR-BENCH-RFC-001 §7.3 (paiml/infra#259). Pick one,
record the rationale in the ticket, and if the ledger-based option is chosen, ship the
staleness arm
— otherwise the gate measures declared state rather than resolved
state.

Definition of done, per item

  1. Merged to main via green ci / gate
  2. A gate exists that would have caught the original gap
  3. The named mutation applied and observed RED, pre-fix GREEN captured in the PR body
  4. pv contract in the same PR
  5. Discrimination confirmed — GREEN on a no-op change
  6. Any doc claim the change invalidates updated in the same PR

Related

Folded sub-tickets (TRIAGE-100, 2026-10-14)

These open sub-tickets were folded here as checklist rows and closed as not planned so the open count stays at or under 100. Reopen one when its work resumes; an open PR for it reopens it.

Activity

  1. noahgift commented on Aug 22, 2026

    @noahgift
    MemberAuthor

    Clustering lands — the denominator now has structure, and the zeros are subsystems

    830 rows clustered at k=14. Every number recomputed from the CSV and verified exact.

    id label n cov q≤4 uncov UNKNOWN
    5 http-orchestrate-banco 95 0.0% 0 95
    4 data-pipeline 76 0.0% 2 40
    6 contracts-pv 76 0.0% 0 0
    2 orchestrate-agent-stack 56 0.0% 0 56
    12 test-harness 49 0.0% 0 38
    8 rag-eval 44 0.0% 0 24
    7 qa-cgp 37 0.0% 0 26
    11 simulation 18 0.0% 0 9
    13 orchestrate-pacha-secrets 17 0.0% 0 17
    9 profiling 44 2.3% 0 29
    0 entrypoint-and-mcp 99 9.1% 0 85
    10 apr-core-commands 109 34.9% 36 7
    1 apr-lint-diag 66 83.3% 2 1
    3 http-apr-serve 44 88.6% 4 0

    Nine clusters at exactly 0%, covering 468 features. The zeros are not scattered — they
    are whole subsystems: every Banco HTTP route, the entire data pipeline, all of pv, the
    whole agent stack.

    What this changes about the work

    "The 44" is one work package, not 44 tickets. Cluster 10 holds 36 of 44; the rest
    are six strays. That reframes it from a scattered backlog into a bounded domain.

    Per-cluster is a better ratchet than per-binary, and the reason is concrete:
    aprender-orchestrate's 184 features split into three genuinely different subsystems —
    Banco HTTP (95), agent stack (56), Pacha secrets (17) — which have different failure modes
    and should be gated separately. A binary-level number hides all three behind one figure.

    Adopting the proposed G2.3 addition: no cluster may sit at 0% at --release. That
    converts nine zeros into nine bounded tickets rather than one unbounded aspiration.

    Two zeros that are qualitatively different, filed separately

    Method notes worth preserving

    k=14 from the elbow; silhouette climbs to 0.26 at k=44 without settling, meaning the
    structure is hierarchical, not naturally k-partitioned. The ledger independently
    arrived at 37 root-cause clusters, so k≈37 is the finer-grained alternative if the two
    should line up.

    The module-stem signal is the one that matters — it is the only input that is not a
    restatement of the feature name. Transport prefix kept deliberately: a route and a
    subcommand sharing a noun are different features with different failure modes.

    cluster_label is the durable key; cluster_id is incidental and stable only for a fixed
    input. cluster.py is deterministic (random_state=0) so the assignment is reproducible
    when the surface moves.

  2. added this to the 0.68.0 milestone on Sep 10, 2026
  3. added
    automationBot-maintained dashboard, not a defect — skip in triage
    P1High priority
    on Sep 11, 2026
  4. modified the milestones: 0.68.0, 0.69.0, 0.72.0 on Sep 16, 2026
  5. modified the milestones: 0.72.0, 0.73.0 on Oct 14, 2026
  6. noahgift commented on Oct 19, 2026

    @noahgift
    MemberAuthor

    Triage evidence 2026-10-19 17:15Z: all children named for this epic are closed: #2701 #4861 #2853 #1293. The epic body lists no open child issue. Owner label is owner:aprender-a7 (session not live). Closing is left to the owner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityautomationBot-maintained dashboard, not a defect — skip in triageepicEpic — multi-story umbrellaowner:aprender-a7owner sessionsize:MK̂ 3-4 h

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions