Skip to content

Every src file must compile: widen the #3809 dark-file guard to the whole tree (138 orphan .rs files, 53,211 lines) #4700

Closed as not planned

Description

@noahgift

Proposed by la-0.73 (cop-inbox handoff/la-0.73-propose-tickets.md §S5), verbatim:

aprender: every src file must compile; widen the #3809 dark-file guard to the whole tree (S5, new, from item v; sent 2026-10-04T06:12Z). At 316dee2, 138 of the 8,264 crates/*/src .rs files (53,211 lines, in 12 packages) are reached by no mod, include! or #[path] from any package root, so nothing compiles them. scripts/check_src_test_files_wired.sh sees only the 42 in its baseline: it looks only at files with a test attribute, and counts a file as declared when any file of its crate names it. Live files point into dark ones (contracts/realizar/binding.yaml:250-252 marks paged_attention_into implemented; apr-model-capability-v1.yaml:81 cites a dark gelu_host; a falsify test in contract_tests.rs reads the dark file named at :621 as text). Scope: widen the guard to every src file, re-mutated in the new scope (Rule 4) with a case table (Rule 7), with a baseline that only shrinks; delete a first batch; re-point the live references. S3 can land first or fold into the first batch. Evidence: la-73/3999-runs-everywhere-drafts @e4e2cbc3dc docs/lookahead/0.73/ticket-bodies-side-fixes.md S5, orphan_census.py (self-test 31/31, 13 mutants RED) and orphan-census-316dee2cd4.tsv.

Update 2026-10-04T06:28Z (@4f077d121f): 27 of the 138 orphans (15,241 lines) are copies of a compiled file. 24 have the same text and 3 have the same text up to // comments and whitespace (orphan_census.py twin column; self-test 33/33, 18 mutants RED). S5's first batch deletes these 27, which need no decision.

Update 2026-10-04T07:23Z (@a658cacd34): fn_census.py compares each unit of the other 111 orphans (fns, tests, items, impl headers, the code outside them) with the compiled units by text. Covered: 20 files (7,334 lines); every unit has a same-text compiled copy, so they are S5's second batch and need no decision. Stale: 31 files (10,229 lines); 73 of their 347 units differ from the compiled namesake, so diff first. Unique: 60 files (20,407 lines); 826 units have no compiled copy (636 tests, 169 fns, 21 other), so port or drop each. The 27 copies all come out covered. Self-test 24/24, 24 mutants RED; the 138 rows match at aca6f2d. The repo body (ticket-bodies-side-fixes.md S5) carries the per-file verdicts.

Train 0.73, under epic #3999 (E6 0.73: Runs Everywhere). Owner: la-0.73. Minted by the cop's clerk.

Update 2026-10-04T09:26Z (@316dee2cd4): S6 is now a checklist item here. It was drafted as a separate ticket (cop-inbox handoff/la-0.73-propose-tickets.md §S6) but never sent; it belongs here because #4700 is where reduces.rs gets deleted. Branch la-73/4700-dark-src-batch1 @5b70349457 has no PR yet, because paiml/aprender is over its open-PR cap. It holds two commits:

  • f3a0a48, batch 1: deletes 27 copies (40,766 lines). That is 26 of the 27 above plus crates/aprender-test/src/generated_contracts.rs, a 25,651-line copy under a virtual manifest. gguf/cuda/cache.rs stays until the dark loading.rs goes, because loading.rs include!s it (:347).

  • 5b70349: widens the guard to every crates/*/src file, transitively (scripts/lib/src_reach.py, with a 53-row case table). Its shrink-only baseline lists 113 files, identical to a census of its parent, and each of 14 mutations turns its self-test red.

  • S6: port reduces.rs's PMAT-374 capture fix to the compiled capture sites, then delete reduces.rs. paiml/realizar de21747e (2026-03-26) changed the capture to open in ThreadLocal mode instead of Global, and to synchronize the stream when end_capture fails. In aprender-serve, that fix exists only in the dark crates/aprender-serve/src/cuda/executor/layers/reduces.rs (:19, :31-38). The compiled sites in that dir still open Global, and return on an end_capture error without a sync:

    • par-121.rs:37/:50, try_batched_graph_capture, reached under BATCHED_GRAPH=1 (crates/aprender-serve/src/gguf/cuda/generate_batched_streaming.rs:461 → :498 → batched_forward.rs:496 → :576);
    • prefill.rs:1106/:1119, try_prefill_graph_capture, reached under PREFILL_GRAPH=1 (prefill.rs:27 → :178);
    • par-062.rs:17/:29, try_graph_capture (:6), which nothing calls.

    The default decode graph is not affected: it is built node by node, with no stream capture (manual_graph.rs:74-75). aprender-train's two capture sites already use ThreadLocal. Global should be avoided for the reason GPU-ORD-4 records (crates/aprender-gpu/src/driver/cuda_tests/cuda_graph_tests.rs:40, :78): while a Global capture is open, every legacy-stream synchronous call anywhere in the process is illegal, and an unsafe call from another thread invalidates the capture. Done when: both opt-in sites capture in ThreadLocal mode and sync on an end_capture error; try_graph_capture and reduces.rs are deleted; on an sm_89 host, each opt-in path gives the same tokens before and after the change. This needs a GPU host. Only reduces.rs waits on it; the other dark files do not.

Update 2026-10-04T09:29Z (@316dee2cd4): the stale files' decisions, as checklist items. They come from an analysis run at 08:26Z by unit_history.py and dark_fix_lines.py; the per-pair cites are in ticket-bodies-side-fixes.md S5, stale bullet, on la-73/3999-runs-everywhere-drafts. That analysis follows the 72 units of the 31 stale files whose text differs from their compiled namesake, and in 35 of them only the compiled copy moved. 18 (file, commit) pairs add lines that no compiled holder has, and 9 of those pairs are code that has never compiled in this repo. Each of the 9 needs a decision before its file goes. S6 above is one (PMAT-374 capture); the other 8 are below.

  • Decode event (PMAT-283, realizar 921eece0): the only call of init_decode_event (crates/aprender-serve/src/cuda/executor/workspace.rs:240) is in the dark layers/reduces.rs (:50), and its reader decode_event_complete (workspace.rs:252) has no caller. Decide whether the compiled decode path records the event, or both fns go.
  • Fused QKV (Contract: Shannon entropy bounds #288, realizar 8e2f6900): the only dispatch of fused_qkv_hw_dp4a_q4k_gemv_into (crates/aprender-serve/src/cuda/executor/q4k_mwv_gemv.rs:527) is in the dark cuda/executor/layers/rope.rs (:111), so that kernel has never run in this repo. It is a speed change, not a fix. Either port the dispatch with a GPU A/B, or drop it on purpose.
  • Contract checks (realizar 2ee35521, dafc2fa5, bb8461e3; PMAT-495), 5 pairs: five macros from crates/aprender-serve/src/generated_contracts.rs are called only from dark files, so their checks have never compiled. Call each one from the compiled namesake, or record in its contract that nothing calls it.
    • contract_pre_cuda_graph_guard! (:11810): called from the dark reduces.rs:15; the compiled namesakes are S6's capture sites.
    • contract_pre_attention_sublayer! (:20371) and contract_post_attention_sublayer! (:20393): called from forward_block_traced in the dark gpu/scheduler/forward_from_model.rs (:190, :348); the compiled namesake is gpu/scheduler/gpu_forward_pass.rs:184.
    • contract_pre_hybrid_block! (:20495): called from is_hybrid_attention in the dark safetensors/safetensors_config.rs (:466); the compiled namesake is safetensors/safetensors_parser.rs:529.
    • contract_pre_tensor_count! (:24280): called from tensor_count in the dark safetensors/tensor.rs (:158); the compiled namesake is safetensors/shard.rs:157.
  • Argmax (the release: merge back v0.69.3 into main #4338 merge-back, 5fa13fe): it moved gpu_argmax and batched_gpu_argmax onto ensure_kernel_module, but only in the dark reduces.rs. ensure_kernel_module (cuda/executor/module_key_guard.rs:47) carries the GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759 debug-build module-key check. The compiled copies in layers/par-062.rs (:124, :263) still compile their 4 modules inline (:149, :158, :303, :310), so that check never sees them. Port the 4 calls; release behaviour does not change.
  • ALiBi slope doc (ce39f30, PMAT-858): in the commit that deletes layers/a_li_bi.rs, fix crates/aprender-serve/src/layers/scaled_rope.rs:327, which still says 2^(-8h/n). The compiled layers/alibi.rs has the right 2^(-8(h+1)/n), in its doc (:27) and in its code (:53).

The other 9 pairs need no decision: 5 were ported in other words and 4 are hygiene, and the deleting commit names where each fix went. Of the 2 units that never had the same text as their compiled copy, one is the old copy of a fix the compiled copy has (scatter_single_kv_to_batched, PMAT-044), and the other is the compiled fn with its parts moved into helpers (aprender-cbtop analyze). Neither needs a decision.

Activity

  1. added this to the 0.73.0 milestone on Oct 8, 2026
  2. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  3. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Moved: this sub-ticket is now a row in #5338 (Backend portability: correct kernels and config on every CPU, wgpu and CUDA target), split out of #5313 by theme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions