Skip to content

F-PROVE-007 asserts 297 compile-time proofs against a tree that generates 578, and never executes #2732

Closed as not planned

Description

@noahgift

Assimilated from #2728 · child of #2706 (APR-PERF-GATE-001, the receipt rule)

Behaviour

f_prove_007_proof_count_is_exactly_297 asserts the tree contains exactly 297 compile-time proofs. The tree generates 578. The assertion has never fired, because the file it lives in is compiled out by default — and the only two targets that would compile it in report success unconditionally.

Two independent failures: it does not run, and it would fail if it did.

Reproduction

It does not run

$ cargo test -p aprender-core --test falsification_spec_v10_tests f_prove_007
rc=0
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

crates/aprender-core/tests/falsification_spec_v10_tests.rs:16 — #![cfg(feature = "model-tests")] gates all 138 tests in the file.

It would fail if it did

$ cargo test -p aprender-core --features model-tests --test falsification_spec_v10_tests f_prove_007
rc=101

thread 'f_prove_007_proof_count_is_exactly_297' panicked at
crates/aprender-core/tests/includes/falsification_spec_v10_contract_model.rs:259:5:
assertion `left == right` failed: F-PROVE-007: Expected 297 proofs, found 578
  left: 578
 right: 297

Ground truth, counted directly out of the build directory:

$ grep -c "const _: () = assert!" \
    /mnt/nvme-raid0/targets/aprender/release/build/aprender-core-*/out/model_families_generated.rs
578    # identical across all generated copies

Nothing that would run it can report failure

model-tests appears nowhere in .github/. It appears twice in the Makefile, and both targets exit 0 no matter what happens:

$ make test-spec
rc=0
🔬 Running spec structural tests...
✅ Spec tests complete          # ran zero tests

$ make test-model
rc=0
  ❌ f_ollama_001 FAILED
  ❌ f_ollama_002 FAILED
  ...  (all 10)
✅ Model tests complete

Two compounding causes:

  1. Neither passes -p aprender-core (Makefile:113, Makefile:121). From the workspace root, cargo test --features model-tests resolves the feature against the root aprender facade, which does not define it:
    error: the package 'aprender' does not contain this feature: model-tests.
    No test is ever compiled, let alone run.
  2. .ONESHELL: (Makefile:20) with no set -o pipefail and no set -e. The whole recipe runs as one bash script, so the recipe's exit status is its last command — which in both targets is a decorative @echo "✅ ...". The cargo … 2>&1 | grep "test result:" pipeline already discards cargo's rc=101 in favour of grep's status, and even grep's non-zero is then overwritten by the trailing echo.

make test-model additionally prints ❌ … FAILED ten times and still exits 0 — the failure is visible on stdout and invisible to the exit code.

This is the feedback_verification_discipline rule 1 and the pass-grep class, in the one lane that would have caught the stale constant.

Which number is wrong

Not established here, and the fix should not assume. CLAUDE.md and #2728's H0-064 both claim 297; build_codegen.rs emits 578. Either the generator gained proofs nobody re-baselined, or the constant was aspirational. Whichever it is, the constant must be derived or explicitly re-baselined with a dated note — not silently bumped to 578 to make the test green. Bumping it to match the current output reproduces the original defect one commit later.

RED-turning mutation to register

  1. Un-dark it. Fix Makefile:113,121 to pass -p aprender-core, and make both recipes fail-closed: capture ${PIPESTATUS[0]} (or drop the pipe) and drop the unconditional trailing echo "✅", or add set -euo pipefail at the top of the ONESHELL recipe.
    Mutation: point the recipe at a test known to fail. make test-spec must exit non-zero. It currently exits 0 — so run this mutation first, against the current Makefile, to confirm the target is presently unable to red.
  2. Re-baseline F-PROVE-007 to the decided value, with the derivation recorded next to the constant.
  3. Mutation on the assertion: change the expected count by ±1. cargo test -p aprender-core --features model-tests --test falsification_spec_v10_tests f_prove_007 must go RED. Revert; confirm GREEN.
  4. Discrimination: the corrected test must pass on an unmodified tree — a gate that flags everything is as broken as one that flags nothing.
  5. Wire the structural (non-model, non-GPU) subset into a lane that actually runs. Note the standing trap: a suite gated only in the Makefile is invisible to ci.yml.

Why this belongs to #2706

F-PROVE-007 is a receipt for a claim the project publishes: "297 compile-time algebraic invariant proofs" (CLAUDE.md, #2728 H0-064, SHOWCASE-BRICK-001 §16). The receipt has never been checked, disagrees with the artifact by 281, and the lane that would check it ends in echo "✅". That is the epic's thesis in miniature.

Activity

  1. noahgift commented on Aug 28, 2026

    @noahgift
    ContributorAuthor

    Correction: the 297 half was already fixed, and the replacement is also inadequate

    This issue was filed against d6c6c6f8f — the stale main checkout, which is 58 commits behind origin/main. On origin/main the literal is gone: #2631 (51bfcb189) replaced it, and the repro command gives rc=0, not 101. My apologies; that is the third finding today traced to reading the stale tree.

    What survived is real: the function was still named ..._proof_count_is_exactly_297, and the file is still #![cfg(feature = "model-tests")] — dark by default. The darkness half is fixed separately (the Makefile omits -p aprender-core, so --features model-tests resolved against a facade that does not declare it and no test had ever compiled).

    297 vs 578: neither is wrong, and a literal is the wrong shape

    The archived spec that published 297 states its own basis — "297 total across 8 families, 24 size variants" — and its per-class table sums to exactly 297 at that size. The registry has since grown to 16 families / 47 variants and the same generator emits 578. Per-variant density is unchanged: 297/24 = 12.4, 578/47 = 12.3. Nothing stopped being proved; more models arrived.

    Decomposing today's file against the generator's rules predicts 578 exactly: 9 unconditional × 47 = 423, + 47 Vaswani + 46 Ainslie + 31 RoPE-even + 31 RoPE-maxpos. Both numbers are snapshots of a moving population, so any literal is wrong again at the next family YAML. The gate now enumerates the variants the generated file itself declares and requires each to carry each of the nine unconditional proofs.

    #2631's replacement does not hold either

    count >= KNOWN_FAMILIES.len() * 3 is 578 >= 48 — a 12x margin. Measured: deleting one unconditional proof from build_codegen.rs drops 578 → 531, and >= 48 stays GREEN. It would tolerate the generator losing eight of its nine unconditional proofs.

    A second live defect, found only by mutation

    find_generated_file scanned the shared <target>/debug/build/ directory. I verified independently: that directory currently holds 401 aprender-core-<hash>/out/ copies, and the walk returns whichever it hits first.

    The first mutation run came back GREEN because the gate read a months-old OUT_DIR that still contained the deleted proof. It now resolves env!("OUT_DIR") — the same mechanism family_registry.rs:464 uses to include! the file — which also un-darkens F-PROVE-003/004/005, and cross-checks the file's KNOWN_FAMILIES against the compiled-in list so a foreign artifact is named rather than tolerated.

    This is the shared-target-dir hazard reaching a test: worktrees do not isolate it here (target_directory = /mnt/nvme-raid0/targets/aprender for every worktree).

    tree gate result
    unmodified new F-PROVE-007 GREEN (discrimination)
    proof deleted new gate, old finder GREEN — the finder bug
    proof deleted new gate, env!("OUT_DIR") RED — "47 of 423 unconditional proofs are missing"
    proof deleted old families * 3 gate GREEN (531 ≥ 48)
    restored F-PROVE-001..007 GREEN (8 passed)
  2. added this to the 0.69.0 milestone on Sep 10, 2026
  3. added
    automationBot-maintained dashboard, not a defect — skip in triage
    P1High priority
    on Sep 11, 2026
  4. modified the milestones: 0.69.0, 0.72.0 on Sep 17, 2026
  5. noahgift commented on Sep 25, 2026

    @noahgift
    ContributorAuthor

    Parked by APR-EPIC-001: over S1 budget (rank 19/163). Reopen = pull. Applied by aprender-d1 under a 3-lane quorum receipt (handoff/balance-quorum-rev4.json).

  6. modified the milestones: 0.72.0, backlog on Sep 25, 2026
  7. added
    parkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityautomationBot-maintained dashboard, not a defect — skip in triageparkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions