Skip to content

EPIC S1 standing: Gates That Cannot Lie #3863

Open
10 of 13 issues completed
Open

Description

@noahgift

owner: aprender-a7 (re-owned 2026-10-09 by clerk aprender-3e for cop aprender-13: no live session works its open children (#3745 #3844 #2876 #4645 #2878))

Spec: APR-EPIC-001 §2 · Epic: S1 · Train: standing · Successor at cut: —

Theme

Gates That Cannot Lie

done_when

vacuous PASS = 0, env-death-as-code-failure = 0, hand-enumerated universes = 0 (each is a ratchet that only falls).

Budget

At most 4 open children (docs/roadmaps/epics.yaml). Repo-wide cap 100 open issues; andon at 95.

Absorbs

#2876, #2877, #2878, #2879, #2503

Prior body

Standing epic. Not a milestone. Items accumulate here; each release bus pulls the highest-EV subset it can finish, and the rest stay. Some land in 0.70.0, some in 0.71.0, and an item that keeps losing the EV comparison is telling us something true about its value.

Why this epic exists

0.69.1 was held by a single missing green witness, and the investigation that found it turned up a pattern that is bigger than any one ticket: we have gates that cannot fail. Not gates that are wrong — gates that are structurally incapable of going red, reporting green forever while the thing they name goes unchecked.

Counted during the 0.69.1 batch, each one measured rather than reviewed:

# the green that cannot go red measured
1 ladder verdict ignores 3 of 4 verbs failed serve probe → row still green: True
2 contract falsification bindings 563 of 4,810 entries (11.7%) produce a reference anything checks
3 tests that have never executed 1,380 (#3809 124, #3810 1,256) + one dark since 2026-04-22
4 README gated in a region, boundary invisible cookbook count wrong by 5.4× (341 vs 1,825)
5 coverage gate skipped, and a skip is tolerated by construction reverse-coverage → Skipped { reason: "no --binding or --crate-dir provided" }; lint/mod.rs:607 is g.passed || g.skipped

Plus the ones already closed that establish the pattern is not new: check_package_includes.sh reporting OK: All 0 include!() files and unable to fail (CLAUDE.md records it); qwen-story-daily capturing tee's status so its fail-the-job step was unreachable, three green runs proving nothing (#2336); make publish's POST-PUBLISH VERIFICATION doing the same (#2360).

The unifying defect is not carelessness — it is that a green is indistinguishable from a working gate without a red. Every instance above was written by someone competent, reviewed, and shipped. What separates the good ones from the theatre is a single artifact: a proof that the gate has been observed to fail on a planted fault.

How EV is computed here

Not vibes. Three measured inputs, then a division.

EV  =  (blast_radius × fire_probability)  ÷  cost
  • blast_radius — 4 ships wrong behaviour to users · 3 lets a bad release tag · 2 blocks/delays a good release · 1 gives a wrong local answer
  • fire_probability — measured (observed firing) · armed (will fire on the next instance of a known-recurring bug) · latent-bounded (has a measured threshold) · latent-unbounded (no known threshold)
  • cost — hours, and an item with no cost estimate cannot be ranked, which is how it gets prioritised: estimate it or it waits

Two rules that override the arithmetic:

  1. Every fix ships with its own RED. A planted fault, the observed failure, and the restore proven byte-identical. A fix without a demonstrated red does not close its item — it just moves the theatre one level up. (5e's mutant silently failed to apply tonight and stayed green; they caught it and the lesson is in the commit: a mutation must be proven to have landed.)
  2. Extending a guard's scope requires re-mutating in the new scope (fix(release): the POST-PUBLISH verification could not fail, and the guard never scanned the Makefile #2360). The old proof does not transfer, and the re-mutation must target the new input class specifically.

Seeded items

item blast fire cost EV bus
1. ladder green must read verbs 3 measured 0.5h highest 0.70.0
5b. reverse-coverage is skipped and a skip passes 3 measured 2h high 0.70.0
2. falsification test: binds to nothing (#3858) 2 measured 8h high 0.70.0
3. dark tests (#3809, #3810, crux-A-01) 4 armed 6h high 0.70.0
5a. coverage 88.78% against a ≥95% target 2 armed large medium 0.71.0
4. README numeric claims outside the gated region 1 measured 3h medium 0.71.0

Item 1 is first and it is half an hour. The ladder's verdict is
cap_ok and golden_output and all(ran and not fallback and not escaped_special) — every field from apr run. chat, code and serve are recorded in the receipt and never read. #3828 widened the probe from one verb to four precisely because a live defect walked through when only run was probed (/api/chat could not reach the Qwen3.5 hybrid while every gate stayed green) — and the verdict still cannot see the verb that would have caught it. Its mutation proof: flip one verb red on a rung that is otherwise green, so the rung's colour changes for that reason and nothing else can be credited with it.

Item 5 is two problems and they separate. (5a) the number: 88.78% line (786448/885829) against min_coverage = 95.0 aspirational and COV_FLOOR := 88 enforced — a 7-point gap and a floor one point below the measurement, so coverage can fall 0.78% before anything complains. (5b) the gate: reverse-coverage skips for want of --binding/--crate-dir, and lint/mod.rs:607 computes passed = gates.iter().all(|g| g.passed || g.skipped) — a skip is tolerated by construction, so the gate contributes nothing and cannot. 5b is cheap, 5a is a programme; do not let 5a's size hide 5b.

This one has a personal case study attached: the 0.69.1 dogfood coverage extractor scraped COV_THRESHOLD := 95 out of the Makefile banner and fabricated a passing number, returning a threshold constant in all three test cases including 88% (the floor) on a real 87.82% miss. A measurement that returns the threshold it is compared against is the purest form of this epic's defect.

Candidate pool — triage into a bus, don't let it rot

Found during the same batch, unranked until someone estimates cost:

Intake rule, so this does not become a graveyard

An item enters with: the measurement (not the suspicion), the blast radius and fire probability from the table above, and a cost estimate. An item without a cost cannot be ranked and will sit here until someone gives it one — that is the feature.

An item leaves when its fix ships with its own observed RED. Nothing closes on "looks right."

Refs #3687, #3809, #3810, #3852, #3857, #3858, #3859, #3860, #3861, #3862

Parked

Closed over budget (APR-EPIC-001 rule 5). Reopen = pull (rule 7).

Superseded epics

Activity

  1. added
    P1High priority
    epicEpic — multi-story umbrella
    on Sep 22, 2026
  2. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Added: #3532's surviving half — a 5 GB model taking 23.1 GB, gated by nothing

    Half of #3532 closed tonight by measurement, and the other half is an epic item.

    #3532 reports two things about qwen3-8b Q4_K_M on CUDA sm_89: apr qa golden_output Empty, and 23.1 GB peak VRAM for a 5 GB model. The 0.69.1 lambda ladder run (sm_89 = RTX 4090 = lambda, apr 0.69.1 (87d9d5484)) measured:

    [ OK ] qwen3-8b-q4km   qa cap+golden pass, backends cpu,cuda honoured
    

    So the golden_output Empty half is fixed — by the FP8 per-token/per-channel scaling fix (0c85942dd), d8's F16+IQ4_XS GEMV kernels, and #3850's refusal-instead-of-silent-Q4_K, all folded in this batch. That half of #3532 should close against this receipt.

    The VRAM half is untouched and nothing gates it. A 4.6× memory multiplier is the reason a model "works" on a 24 GB card and is unusable on a 16 GB or 8 GB one. It is invisible to every gate we have, because the ladder asks did it produce correct output and never at what cost:

    item blast fire cost EV bus
    6. peak-VRAM has no ceiling anywhere (#3532) 3 measured 4h high 0.70.0

    Blast radius 3 rather than 4 because it does not ship wrong output — it ships a model that will not load on smaller silicon while every gate reports green. Two corroborating measurements from the same batch: the UD 4B hybrid reads 9.8–9.9 tok/s against a 10 tok/s floor on yoga's 4060 and 162 tok/s on the same card for the pure-F16 0.5B, which is memory pressure on the smaller GPU showing up as a throughput coin-flip on the only gate that can see it. A VRAM ceiling would name the cause directly instead of letting it surface as a flaky throughput row.

    On whether #3532 blocks the tag. It does not, and the reasoning matters because the operator rule is absolute: ALL Q4_K models must work on CUDA on gx10 AND lambda before any release, with no optional clause. Tonight qwen3-8b works on lambda — measured, not asserted. A model that works while consuming 4.6× the memory it should is a defect, not a violation of that rule. It goes in the epic; it does not hold the bus.

    Two process notes attached to this item, because they are the reason it was nearly mis-ruled:

    1. qwen3-8b Q4_K_M on CUDA sm_89 (24 GB): 23.1 GB peak VRAM for a 5 GB model, apr qa golden_output Empty output — optional ladder rung red since 0.68.1 #3532 was filed as an "optional ladder rung red since 0.68.1". "Optional" is exactly how P0 release-process: the train can publish with "most models working" — the model gate is a hand-picked ladder with OPTIONAL rungs, a skip reads as pass, and the dogfood can be inherited. Every Q4_K model × {lambda, gx10} × CUDA must be green or NOTHING ships #3712 describes a red model getting through — by being left off the list or marked required: false — and check_model_ladder.sh now REFUSES required: false on a Q4_K rung for that reason. The ticket's own framing predates the refusal that exists to prevent it.
    2. Its "red since 0.68.1" status was stale by two releases and nobody knew, because no gate re-measures a ticket's premise. A closed-by-measurement path — re-run the claim, close on the receipt — is itself an epic candidate: we are carrying an unknown number of P1s that describe fixed behaviour.

    Refs #3532, #3712

  3. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Added: two detector gaps for the instrument errors themselves (#3864, #3865)

    Filed by aprender-5e during the 0.69.1 batch. Neither arms a guard — both are tickets for detectors that do not exist, deliberately not written mid-release, which is the correct call and the second override rule of this epic applied to itself.

    item blast fire cost EV bus
    7. rc=$? after a pipeline — the CLAUDE.md #1 defect, undetected (#3864) 1 measured ×3 tonight 6h high 0.70.0
    8. unbraced "$var:path" in a git-ref position (#3865) 1 measured 4h medium 0.70.0

    Blast radius 1 — these produce wrong local answers, not shipped defects. Their EV is high anyway, and the reason is the frequency term: #3864 was committed three times tonight, by two different agents, every one of whom knows the rule by name. check_no_pipe_into_grep_q.sh is its narrow cousin (a pipe into grep -q under pipefail); the general form cmd | other; rc=$? has no detector at all.

    The instance that shows why discipline is not the fix. c7 hit #3864 inside the very command that was verifying 5e's zsh trap: git cat-file -e … | head -1; echo "rc=$?" printed rc=0 for both a valid and an invalid ref, so the first run reported that the braced and unbraced forms agreed — the exact opposite of the real finding. It was caught only because one form had printed fatal: and the other had not, contradicting the identical rc.

    That is the method this whole epic is currently relying on: noticing that chatty output disagrees with a number. It works, it caught roughly a dozen instrument errors tonight, and it fails silently the moment the command is quiet. Every catch tonight came from an internal inconsistency in the output, never from suspicion about the method. That is not a process we should keep depending on.

    #3865 already cost something: a false ABSENT report to this session about contracts/apr-model-capability-v1.yaml, which was present. zsh reads :c in "$r:contracts/…" as a modifier, so the ref became origin/release/0.69.1-batch-2ontracts/…, git cat-file -e returned non-zero, and the loop printed ABSENT — a sound predicate pointed at a mangled ref. An exit-code test does not protect you from testing the wrong thing. The ticket carries the measured prefix table, with crates/ and contracts/ flagged as both silently corrupting and both starting with c, which is this repo's specific bad luck (8 of 10 common prefixes corrupt silently).

    Both tickets say the hard part is the must-NOT-match set, not the detector — #3865 must tell a git-ref position from an ordinary colon-bearing string, #3864 must not flag rc=$? after a non-pipeline. That is rule 7 of Verification Discipline (guard regexes ship a case table; the apr-invocation patterns were wrong five times and every one was caught by the table, none by review), stated in the ticket rather than discovered by whoever picks it up.

    One detail worth keeping from #3865: it flags "$r:$p" as a must-match row even though that form currently works, because a form that is correct by accident survives review and breaks on a refactor. Same reasoning as item 1's note that a green is indistinguishable from a working gate without a red.

    Refs #3864, #3865

  4. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Added — item 9: both halves of the tensor-layout defence are declared and neither operates

    Found by aprender-5e. This is now the strongest item on the epic, and the only one where a copy of the false claim ships to crates.io.

    contracts/tensor-layout-v1.yaml — the file CLAUDE.md names SOURCE OF TRUTH for LAYOUT-001/002, the bug class it says "has occurred 100+ times" — asserts a compile-time impossibility. Verbatim from type_enforcement.compiler_guarantee:

    "It is IMPOSSIBLE to construct AprTransformer with unvalidated data because: 1. AprTransformer fields are ValidatedEmbedding, not Vec … 4. Inner data field is private - cannot be accessed without validation. This is Poka-Yoke: the mistake (using unvalidated data) is physically impossible."

    The code, crates/aprender-serve/src/apr_transformer/mod.rs:407:

    #[derive(Debug, Clone, Serialize, Deserialize)]
    pub struct AprTransformer {
        pub config: AprTransformerConfig,
        pub token_embedding: Vec<f32>,      // claim 1 says ValidatedEmbedding
        ...

    Claim 1 false (it is Vec<f32>). Claim 4 false (it is pub). Claim 2 defeated independently by Deserialize — the struct constructs from serialized bytes with no constructor at all. ValidatedEmbedding has zero hits in every file under crates/aprender-serve/src/apr_transformer/. The type still exists elsewhere; it is simply not what AprTransformer is made of. The five enforcement_locations are also pre-monorepo paths, all five absent as written.

    It ships

    Four copies, all four containing physically impossible twice:

    ./contracts/tensor-layout-v1.yaml
    ./contracts/aprender/tensor-layout-v1.yaml
    ./crates/apr-cli/contracts/tensor-layout-v1.yaml                              <-- SHIPS
    ./crates/aprender-contracts-staging/contracts/aprender/tensor-layout-v1.yaml
    

    The root include list carries crates/aprender-core/contracts/ but not the workspace-root contracts/, so this reads as internal-only at first glance. It is not: apr-cli/Cargo.toml has include: None, so cargo packages everything git-tracked under its crate root. That is the identical mechanism that made #3857 a false positive (apr-cli packages all 16 model-families/*.yaml) — the accident that made #3857 harmless is what makes this one ship.

    Both halves, and they do not reference each other

    half mechanism state
    runtime 4 × layout_contract_enforce fns — incl. enforce_embedding_contract, documented "MANDATORY … prevents garbage inference output" zero production callers (#3852)
    compile-time type_enforcement.compiler_guarantee describes a struct that does not have those types

    And no copy of tensor-layout-v1.yaml names any of the six enforce_*/validate_* functions, so the two declared defences are not even aware of each other.

    The conclusion is narrow and it is the sharpest statement of this epic's thesis: their silence has never been evidence about any model. Every green read from that direction was read from nothing.

    Explicitly not claimed — that this explains #3791 (HHHH…) or #3804 (CUDA gibberish). Those are GPU-path defects; this is load-path and type-level. The narrow claim is what makes it credible.

    Ranking and the split

    item blast fire cost EV bus
    9a. correct the false claim + the 5 stale paths 3 measured 0.5h highest pre-tag
    9b. actually make AprTransformer hold validated types 4 armed large high 0.70.0

    9a is pre-tag and doc-only. A missing guarantee makes someone check; a false one makes them stop checking — the same asymmetry as CLAUDE.md's "a shadowed artifact is worse than a missing one." Anyone debugging garbage output who reads this will eliminate unvalidated-embedding as a hypothesis on the strength of a Poka-Yoke claim that stopped being true. Zero behavioural risk to fix the text; real harm to ship it.

    9b is not a release-night change — implementing type-level validation on the load path, on release night, on the bug class that has recurred 100+ times, is exactly the trade this epic exists to refuse.

    The proof for 9a is inverted, and that is the finding

    There is no gate to turn red, so the demonstration is the opposite: pv validate and pv lint both pass on this contract, today, with the false claim in it. Prose in compiler_guarantee is unvalidated by construction, so a contract can assert a compile-time impossibility that the compiler flatly contradicts and every gate stays green. That generalises well past this file and belongs with item 2.

    Refs #3852, #3857, #3860

  5. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Two items for this epic, from the #3769 cookbook-pin work (aprender-45):

    #3866 — a must-RED self-test row returns GREEN. check_readme_claims.sh --self-test is rc=1 on a pristine tree (6/7 rows). The failing row is row 4, which exists specifically to prove the guard discriminates under --exact, and it returns rc=0.

    Mechanism, in three layers: the script has two contract counters measuring different populations (census.json .n_files = 1831 vs contract_files_only over a git archive = 1832; a plain find gives a third figure, 1885). It attributes that gap to "UNCOMMITTED working tree vs merge tree" on a tree that is clean. And census.json's git_sha — the one field that would settle whether the census is stale — is null, so "the census is stale" is an unverifiable guess.

    The consequence: row 4 builds its fixture as cc - 1 = 1831, which coincidentally equals the other counter, so the uncommitted-tree allowance accepts it and exits 0. Run the same scenario by hand against a fixture built from the census counter and it goes RED correctly. The row is disarmed only while the two counters differ by exactly one — so it will flip back to green on its own and hide whatever it was meant to catch.

    Contains the smaller item too: the stale/undatable contracts/census.json (1831 vs HEAD's 1832, no git_sha). Filed as one ticket because the census gap is the cause of row 4's disarming rather than an independent defect; happy to split if you'd rather have two rows.

    Not fixed: row 4 needs someone to establish what it used to catch first. "Make it red again" without that is how you get an over-strict row that gets deleted the next time it fires.

    #3867 — README.md is gated in a REGION and the boundary is invisible. The scope defect behind #3769. Enumerating the file takes two minutes and finds three more drifted claims, two of them inside the same At HEAD table as the gated ones (Book CLI chapters 112→113, Book lib chapters 71→72), plus a tree diagram claiming "82 crates total" which matches neither the gated 79 nor the 84 directories. Requires an enumerator whose anti-vacuity control is written first — one that finds zero claims and reports "all 0 gated" is check_package_includes.sh reborn.

  6. added a commit that references this issue on Sep 22, 2026
  7. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Added — item 10: 27% of contracts' assertions never reach the compiled code

    Measured by aprender-c7 while answering the pv codegen resolution question. This is the purest instance of the epic's thesis on the board, because it is not one gate that cannot fail — it is the contract system's entire runtime-assertion layer, silently 73% wired.

                              contracts represented
    committed apr-cli generated_contracts.rs      293
    regenerated from root today                   401
    

    108 contracts' assertions are missing from the compiled crate. Not cosmetic: macro_rules! 2606 vs 3525, debug_assert! 678 vs 810.

    And it is stratified — five different vintages, none current:

    crate contracts represented
    aprender-data / -distribute / -graph 165
    aprender-orchestrate 277
    aprender-present-core 278
    apr-cli 293
    aprender-compute / aprender-core 294
    available today 401

    Each crate's generated_contracts.rs was regenerated at some past moment and frozen there. They are tracked artifacts — which contradicts aprender-db/build.rs:65's own comment calling them gitignored — produced by a manual command, and no check_* guard asserts that any of them is current.

    So contracts are authored, pv validated, pv linted, scored, and their preconditions simply never reach the code for 27% of the corpus. Every green from that direction was read from a frozen snapshot of an older contract set.

    item blast fire cost EV bus
    10a. guard: regenerate to temp, assert byte-equality 3 measured 3h high 0.70.0
    10b. reconcile the five vintages / decide tracked-vs-generated 3 measured 6h high 0.70.0

    The guard shape already exists in this tree and is proven: #3077's doc test regenerates and compares (gpu_support_doc::the_committed_doc_matches_the_capability_gate), and 45's release-section renderer and c7's capability mirror both use it with mutants in both directions. 10a is that pattern applied to generated_contracts.rs. Deliberately not built mid-release — a new guard on release night is the trade this epic refuses.

    The good news, and it removes the emergency from #3860

    The layout contract has NOT forked into compiled code. Answered by observation rather than by reading the path join:

    $ cd crates/apr-cli && pv codegen contracts/ -o /tmp/from_crate.rs
      No contracts with preconditions/postconditions/invariants found.   <- NO FILE WRITTEN
    $ pv codegen contracts/ -o /tmp/from_root.rs    # repo root
      Lean theorems: 1309 · Generated: 1,102,837 bytes
    

    The crate-local contracts carry no pre/postconditions or invariants, so they cannot be the source of any generated_contracts.rs — not "probably weren't." So the 154 divergent crate-local copies are dead weight, the fix is deletion or an enforced mirror, and #3860 is cleanup rather than a correctness hazard. Severity down an order of magnitude from where it was scoped.

    Method note worth keeping: c7's first attempt at this question used falsification IDs as the discriminator — they appear in neither generated file, so it returned 0 hits for everything. A sound predicate pointed at the wrong input, which is tonight's dominant failure mode (items 7 and 8). The planted-difference / regenerate-and-compare test cannot fail that way, which is why observation was the required method rather than the preferred one.

    Also closed: the ont ratchet regression (#3863 item, resolved)

    unanchored_but_bindable 298 → 297, ratchet PASS, verified by running it rather than reasoning about it. The disjunct was names a file via ^[[:space:]]*(file|path|source_file):, matching two path: keys nested inside op_implementation rows — the contract's evidence, not its subject. c7 declared the real subject with a top-level entity: block instead of deleting the two lines, and said why: deleting measured evidence to make a number go down is the move this epic exists to refuse.

    The predicate over-match stays here as a candidate: any contract with an evidence table naming source files counts as "bindable" forever. That is a guard-regex case-table question (rule 7 — the apr-invocation patterns were wrong five times, every one caught by the table and none by review), so it needs a must-match/must-not-match set, not a patch.

    Refs #3077, #3860

  8. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Item 5b, measured in full — reverse-coverage has THREE defects and cannot arm

    aprender-55 re-ran #2797's premise. The mirror gate now exists (pv lint Gate 7: reverse-coverage). It cannot arm.

    A) --binding contracts/aprender/binding.yaml   skipped: "no --binding or --crate-dir provided"
    B) --crate-dir crates/aprender-core            skipped: "no --binding or --crate-dir provided"
    C) BOTH                                        runs: 208/4344 bound (68.3%) -> Unknown(NotArmed)
    D) --binding /nonexistent.yaml                 accepted silently, lint proceeds
    
    1. The skip message says or; the code means and. An operator who follows the instruction — supply either flag — gets the identical line back telling them neither was provided.
    2. Even when it computes, it is NotArmed. Row C produces a number and the run's own summary still reads 11 armed; not armed: reverse-coverage. The one gate that answers the ticket's question gates nothing by construction.
    3. Nothing passes the flags anywhere. Zero pv lint invocations across .github/workflows/, scripts/ and the Makefile pass --binding or --crate-dir. check_contract_test_binding.sh runs pv lint contracts --strict-test-binding with no binding. There is no path, PR-time or otherwise, on which this gate runs.
    4. A --binding pointing at a nonexistent path is accepted without error.

    And the number it exists to produce cannot be read: 208/4344 bound (68.3%). 208/4344 is 4.8%. The fraction and the percentage on one line disagree by more than an order of magnitude — possibly two denominators printed as one statistic. Not asserting which is right; asserting that as printed it is unusable.

    This raises 5b's cost estimate from 2h and splits it: the or/and bug and the flag plumbing are small; deciding what the gate should arm on, and fixing a statistic that contradicts itself, are not.

    Item 11 — pv score understates every contract by a letter grade, corpus-wide (#2704)

    pv score contracts/decision-tree-v1.yaml                      0.66 (C)  D5 Binding 0.00 "(no binding registry)"
    pv score contracts/decision-tree-v1.yaml --binding <registry>  0.86 (B)  D5 Binding 1.00
    

    Same contract, same commit, one flag: C becomes B. And the amplification the ticket does not state:

    pv lint WITHOUT --binding :  Gate 3: score OK (1831 contracts, mean=0.48)
    pv lint WITH    --binding :  Gate 3: score OK (1831 contracts, mean=0.48)
    

    The flag does not reach the lint score gate. So the corpus-wide score our own release gate publishes is the no-binding scoring, applied to all 1,831 contracts. (Not claimed: that D5 is the sole cause of 0.48 — that needs the per-dimension breakdown. Claimed: the gate does not honour --binding, and on a bound contract that costs a full grade.)

    item blast fire cost EV bus
    5b. reverse-coverage cannot arm (4 defects + an unreadable statistic) 3 measured 2h + design high 0.70.0
    11. pv score / lint score gate ignore --binding 2 measured 3h medium 0.70.0
    12. cargo_classify.sh wired to nothing (#2775) 1 measured 1h medium 0.71.0
    13. #[contract] fails open in 4 places, doc says the opposite (#2699) 3 measured 6h high 0.70.0
    14. SmoothL1Loss detached from autograd — silent zero gradient (#3034) 4 measured 2h high 0.70.0

    Item 14 is the highest blast radius on this epic. crates/aprender-core/src/nn/loss.rs maps to a raw Vec<f32> then builds a fresh Tensor, so anything trained with that loss receives zero gradient from it, silently. That ships wrong behaviour to users and nothing reports it.

    Item 13's module doc still states the opposite of its code: "Missing env var = compile error" against const #const_name: Option<&str> = option_env!(#env_key);, which yields None. Plus read_contract_assertions conflating "no contract" with "no data" via .unwrap_or(0), and make_env_key still private so every producer re-derives the key. Same family as item 9 — a doc asserting an impossibility the compiler contradicts. Note for whoever takes it: :356 documents that a missing binding emits a compile-time warning, i.e. fail-open by design, so four defects sit beside a fifth that is policy. That distinction matters to the fix.

    Item 15 — the ticket board's open/closed state does not measure the tree

    15 premises re-run. All four retirements — #3532, #3314's anonymous half, #2526, #2828 — were fixed under OTHER work, and not one was closed by the ticket that described it. Zero decayed on their own. Four for four.

    So the board measures who wrote a comment, not what is true of the tree — and #3314 proves it in both directions at once: closed while still mostly true, half-fixed by a ticket that never referenced it.

    item blast fire cost EV bus
    15. nothing re-measures a ticket's premise 2 measured ×4 8h high 0.71.0

    The sweep's product is therefore not a stale-ticket cull. It is a list of fixes that shipped without their ticket noticing, plus a longer list of premises that are simply still true (11 of 15).

    Fourth override rule (from 55): an empty result is a claim about the instrument until proven otherwise

    Four times tonight an empty result nearly became a finding, and on #2828 it would have produced a wrong finding rather than a missing one: a grep for run-\$\{GITHUB_RUN_ID\}-guards returned 0 because the workflow uses ${{ github.run_id }}. Also: zsh not word-splitting an unquoted $a in an arg matrix produced four empty rows that read as clean negatives, and git show "$R:crates/..." fused ref and path via :c three separate times.

    Refs #2526, #2630, #2699, #2704, #2775, #2797, #2828, #3034, #3314, #3320

  9. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Five more override rules, all earned in one session

    These are not items to fix. They are the rules that produced every finding on this epic, and each one was learned by someone getting it wrong first.

    5. Compiling is not wiring — and one string count is uninterpretable

    aprender-c7's first build of the /v1/completions fix was dead code. The insertion script died partway: the include landed, the call never did. cargo build clean, clippy silent, fmt clean — and the probe still returned 503. "Still 503, so the diagnosis was wrong" was right there to be reported.

    What named it was not a test:

    strings target/release/apr | grep -c 'cmpl-qwen35'          -> 0
    strings target/release/apr | grep -c '<the arm error string>' -> 2
    

    The arm's error string present and its discriminator absent is only possible if the code is compiled and unreachable.

    The pair is the instrument. One count is uninterpretable — 0 could mean not compiled, optimised out, or never written. Two counts, one on a path that exists either way and one that exists only if reachable, localise the boundary. This generalises to every "I applied the fix and nothing changed" moment, of which this session had four: an unapplied sed mutant, a grep-shell-function mutant table that changed nothing while reporting KILLED, a coverage extractor that returned its own threshold, and this.

    6. A mutant that fails to BUILD is a kill wearing a kill's clothes

    aprender-55's harness distinguishes three outcomes where most tables have two: KILLED, NOOP (the mutation did not apply) and NOBUILD (it applied and the crate will not compile). A NOBUILD turns the control red for free and reads exactly like a kill.

    Found live: a planted deletion removed the condition of an if let, leaving if let Some(r) = with no expression. The right mutation was a runtime bypass — substituting Option::<T>::None for the call — which tests the thing actually under test rather than the compiler.

    7. Assert the field, never the position

    A control that explicitly asserted non-empty text still passed the empty-text mutant, because it asserted on the last line of the response and the last line became the id line once the text collapsed. A value assertion pointed at a position degrades into a shape assertion in precisely the case it exists to catch. Parse the structure and read the field.

    8. A threshold is not the whole hazard — find the other variable

    The ladder's zero-width-space check fails open on a match: producer | grep -q under pipefail takes the producer's status, and grep -q closes the pipe on first match, so the producer dies with SIGPIPE and the if goes false because the check succeeded. Measured threshold: ~10,000 lines.

    Reproducing it failed — until the match moved from the last line to the first:

                  match FIRST                match LAST
    n=1000        old=true                   old=true
    n=10000       old=FALSE  [141 0]         old=true
    n=200000      old=FALSE  [141 0]         old=true
    

    With the match last, the producer has already written everything before the consumer exits, so the defect never fires at any size. The hazard is not the line count — it is how much output remains after the match. Both framings agree at n=1 and diverge everywhere else, and the realistic shape (an escaped character in an early prompt line of a verbose run) is the failing one. "Could not reproduce, fix is precautionary" was one keystroke away.

    9. A correct verdict with a wrong reason survives review

    Two instances in one ladder run (#3872). A refusal reporting GGML type 1 for a tensor that is IQ3_XXS (18) — with 1 being in the whitelist, so the message is self-refuting against the predicate that emitted it — and a BackendUnavailable (rc=14) asserting a host property that twenty rows in the same receipt refute.

    Both verdicts were right. Both reasons were wrong, and wrong in the expensive direction: one pointed at a whitelist that had just been correctly extended, the other at the CUDA installation. Nobody audits the reason on a red row, because red is what they expected.

    And the one that is a work item, not a rule

    The ladder computes the cause and throws it away. apr qa reported Tensor Contract ✗ FAIL (27 tensors failed data-quality) on the -st.apr row, in the same process, from the same file. model_ladder.sh:351 records only capability_match + golden_output, so the receipt carries gibberish (fragment "NavController" repeats 3+ times) — the symptom — and drops the diagnosis. qa_rc: 5 is the tell that qa failed for more than the recorded gate.

    item blast fire cost EV bus
    16. receipt records the symptom and discards the computed cause 2 measured 0.5h highest 0.70.0

    One more key out of JSON the ladder already parses turns an observation into a diagnosis. Cheapest item on this epic.


    Session tally, for calibration

    Roughly a dozen instrument-or-author errors across five sessions, against about six real defects. The errors were made by people who knew the rules by name — the $?-through-a-pipe rule was violated three times in one night by two agents who can each cite its ticket number.

    Every single catch came from output visibly contradicting itself — a histogram with no F16 in it, an rc that disagreed with twenty sibling rows, a printed count that contradicted its own rows, two string counts that could not both be true. Not one came from anyone doubting the method. That is the argument for this epic, and it is also the reason none of the rules above are "be more careful."

  10. added 3 commits that reference this issue on Sep 22, 2026
  11. noahgift commented on Sep 22, 2026

    @noahgift
    ContributorAuthor

    Three more shapes, and two of them are new kinds

    10. A gate whose SUBJECT does not exist

    aprender-d8's first re-aim of the parity refusal guard scanned ggml type ids 0..=39 for "a type the GPU has no kernel for" and went GREEN — on id 4, which is Q4_2, one of eight ids upstream REMOVED (ggml_type.rs::REMOVED_IDS = [4,5,31,32,33,36,37,38]).

    The refusal genuinely fires on it. So the row would have asserted, permanently and truthfully, that parity refuses a type no GGUF on earth can contain.

    Every other instance on this epic has a real subject and a wrong claim. This one had a correct claim about nothing. It was caught only because d8 probed which type it picked rather than trusting the green — a green with no subject is indistinguishable from a green with a subject, from the outside.

    The fix generalises: the guard now asks the production refusal which type to use, over GgmlType::ALL ("every LIVE variant"), so it self-heals — give the current pick a kernel and it moves to the next uncovered type, and it fails only when every live type has one, with a panic that says to delete it. A guard that names its own example outlives the example; one that derives it does not.

    11. A fix that is correct, proven in both directions, and INSUFFICIENT

    d8 fixed the golden gate's hardcoded stop token (item below), proved it red in both directions across all four local GGUFs, and kept measuring. apr qa still failed, and its displayed text was truncated so the output looked shorter than before:

    apr run --max-tokens  64  ->  3.62s,  300 chars
    apr run --max-tokens 512  -> 22.11s, 1410 chars, cut off mid-word
    

    512 of budget, 512 consumed. "The fix landed and the symptom shrank" was available and wrong. The second cause: golden_prompt_for keys the chat template on general.architecture → "llama" → Llama-2 [INST], while tinyllama-1.1b-chat-v1.0's GGUF declares Zephyr (<|user|>). Same binary, same model, one variable:

    [INST]   (what the gate does)      22.11s  1410 chars  [S][INST] loop, no "Paris"
    <|user|> (what the GGUF declares)   2.45s   170 chars  "The capital of France is Paris."   <- PASSES
    

    tinyllama is a correct model being asked the wrong question by the gate, twice, in the same defect class both times: a constant keyed on architecture where the model carries the answer in its own metadata.

    12. An assignment naming a sha only one worktree holds — and it fails QUIETLY

    The cop assigned another host a commit that existed only in a local worktree. c7's first fetch on gx10 reported the sha NOT FOUND while leaving the worktree at an unrelated commit — so a driver checking the fetch's exit code but not the resulting HEAD would have built and measured the wrong tree, produced a well-formed receipt, and reported it under the assigned sha. apr_bin.sh would have pinned correctly to the wrong commit, and nothing downstream checks the tree.

    Same family as #3784 ("the receipt's identity is the binary that ran") reached from the other side: not a stale binary, but a stale tree the binary honestly reports.

    Rules: push a sha before assigning it to another host, or say "pull from my worktree" — and a fetch that reports NOT FOUND is a hard stop, because the worktree it leaves behind is a plausible-looking wrong answer.

    And the one that was a tag blocker for two hours

    apr qa's golden gate hardcoded SpecialTokens::qwen2() — eos_id = 151645 — as stop tokens for every model. tinyllama declares vocab=32000, eos=2, and 151645 is not a token id that exists in a 32000 vocabulary, so the stop condition was not late or wrong but unreachable. Perfect correlation across the local cache, with a mechanism that explains it:

    REACHABLE    eos=151645  vocab=151936  qwen3moe   Qwen3-Coder-30B-A3B-Instruct-Q4_K_M
    REACHABLE    eos=151645  vocab=151936  qwen3      Qwen3-1.7B-Q4_K_M
    UNREACHABLE  eos=2       vocab=32000   llama      tinyllama-1.1b-chat-v1.0.Q4_K_M
    

    The release's first-tool-for-any-issue gate was structurally Qwen-only, and it silently mislabels every non-Qwen model it is ever pointed at. It also explains the observation that started the investigation: both legs hardcode the same wrong constant, which is why GPU and CPU output were byte-identical — read at the time as "the backends agree, so the defect is elsewhere", when it was the signature of a shared upstream cause.

    item blast fire cost EV bus
    17. golden gate's chat template keyed on arch, not the model's declared template 3 measured 6h high 0.70.0
    18. ten more SpecialTokens::qwen2() call sites in apr-cli 2 armed 4h medium 0.70.0
    19. IQ4_NL needs elements-per-block as a TYPE PROPERTY, not a match arm 4 measured 8h high 0.70.0

    Item 19 is a scoping correction worth reading. IQ4_NL was scoped (by the cop) as "a small addition against an existing sibling, since the lookup table already exists." It is not. iq_dispatch.rs is built on IQ_BLOCK_ELEMS = 256 hardcoded into the matvec arithmetic (blocks_per_row = in_dim.div_ceil(256), col0 = b*256), and IQ4_NL is 32 elements / 18 bytes. Adding IQ4_NL => Some(18) to the block-size table would make blocks_per_row = in_dim/256 and silently under-read each row 8× — garbage, not an error. That is resolve_qtype().unwrap_or(Q4K) (#3850) one layer along: a plausible wrong answer where a refusal belongs, caught before the arm was written rather than after it shipped.

  12. 80 remaining items

  13. modified the milestones: 0.71.0, 0.73.0 on Oct 14, 2026
  14. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into this epic's checklist from #2876 (same theme; text kept verbatim; owner owner:aprender-a7 keeps the work).

    Original text of #2876

    Mechanism. An arm, validator or gate is handed an input that carries nothing to judge — an empty band set, a zero comparator median, a profiler with zero bricks, a method string with no vocabulary, a workload label bound to nothing, a proof count with a 12× margin — and answers PASS (or a Python traceback) because the loop body never runs, the predicate is vacuously true, or the check tests presence rather than value. The opposite polarity is often armed (a missing denominator FAILS loudly), which is what makes each instance a gap rather than a design.

    The one structural fix. Every gate declares, per arm, the input shape it needs and REFUSES by name when the shape is absent — SKIP (precondition: …) or FAIL, never a silent pass; a validator refuses a non-positive or empty statistic on either side with the same sentence; self-tests carry an empty-input row for every arm (PP-066 §9: 'poka-yoke — refusal table derived, never authored'). PP-066 C0 names the first instance (#2830) as a release criterion; the class is what C0's polarity rule generalises.

    Instances (one line each):

    Rows. C0 (required check readable/strict; #2830's polarity), C4 (post-publish parity block on four hosts), master row 7 (JOIN), master row 15 (gx10 shakedown).

    Members (parented here): #2730, #2732, #2735, #2756, #2816, #2830

    PP-066 triage 2026-09-05 (issue #2873); ledger docs/audits/pp-066-triage-2026-09-05.md.

    Folded sub-tickets (TRIAGE-100, 2026-10-14)

    These open sub-tickets were folded here as checklist rows and closed as not planned so the open count stays at or under 100. Reopen one when its work resumes; an open PR for it reopens it.

    Merged in (TRIAGE-100, 2026-10-14)

    From #5307 — Gate honesty: checks that pass with nothing measured or judge the wrong commit

    Folded P0 and operator-worded rows (TRIAGE-100, 2026-10-14)

    These tickets were folded here by the cop ruling of 2026-10-14 (sub-rows of this master). Each keeps its priority and its owner. The operator text each carried is quoted below exactly as written; if an owner objects, the ticket is reopened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityepicEpic — multi-story umbrellaowner:aprender-a7owner session

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions