Repository navigation
FLAKE-0: aprender-compute lib test binary SIGSEGV mid-run (1 of 19 CI runs in 7 days; blis tests in flight) #4891
Description
What happened. On the 0.70.2 release PR (#4890, head b11fd23), CI run 37595776609, job 112707927917 (workspace-test-shard (1), tier full), the step Compute tests (cargo test -p aprender-compute --lib) died mid-run: running 3518 tests, 384 ok, then
process didn't exit successfully: `/workspace/target/debug/deps/trueno-bab453617dce039c` (signal: 11, SIGSEGV: invalid memory reference)
The step tolerates a SIGSEGV at exit only after test result: ok.. This crash came before any result line, so the tolerance did not apply and the required workspace-test went red.
How often. In the last 7 days the step ran 19 times in ci.yml: 18 green, and this one crash (counted by aprender-a7, 2026-10-07). Three local runs of the same tree in the CI image: 3 of 3 green, 3514 passed each, no SIGSEGV. Main's merge queue runs the quick tier, which skips this step, so main gives no signal either way.
Last tests to finish before the crash. libtest prints only completions, so the crashing test is one still in flight. The last five completions were all in blis:
blis::norms::tests::test_rmsnorm_zero_inputblis::microkernels::codegen::tests::test_codegen_bcast_b_64x6_correctnessblis::norms::tests::test_rmsnorm_unit_gamma_normalized_rmsblis::prepacked::tests::test_prepack_emptyblis::prepacked::tests::test_prepack_small
The release commit changes no code against 4098007: version bump, ladder scope entry, certification, notes.
FLAKE-0 (operator, verbatim: "zero tolerance flaky test, you disable and fix, or remove"). Find the test in flight, for example --test-threads=1 in a loop, or a sanitizer or valgrind run. Then disable it with #[ignore = "FLAKE-0 #<this issue>"] and fix the memory fault, or remove the test. Safe Rust cannot segfault, so treat this as unsafe code (or a native library it calls) reading or writing out of bounds until shown otherwise.
Owner: aprender-a2 (FLAKE-0). Milestone 0.71.0. The 0.70.2 decision on this run (one rerun, or hold) is the operator's, and is recorded on #4890.
Agent: aprender-a3
🤖 Generated with Claude Code
Activity
noahgift commentedon Oct 7, 2026
Kernel log at the crash: this is an out-of-bounds read, not infrastructure (2026-10-07).
At 09:02:15Z the runner host's kernel logged:
blis::prepackedis the name of the crashing thread. libtest names each test thread after its test path, and Linux cuts thread names to 15 characters, so the crashing test's path starts withblis::prepacked.error 4is a user-mode read of an unmapped page, and the address is page-aligned: a read that ran off the end of a large (mmap-backed) buffer. A smaller over-read lands on mapped heap and returns garbage without crashing, which fits 1 crash in 19 runs.ipis in the test executable, not a shared library.The green rerun (attempt 2) does not clear it. Under operator ruling C325, CI on the 0.70.2 release commit does not count while this stands.
Finding it. valgrind memcheck or AddressSanitizer on
cargo test -p aprender-compute --lib blis::prepackedreports an over-read every time, whether or not it crashes ("Invalid read ... 0 bytes after a block of size N"). Fix the read; no#[ignore]goes into a release commit.Owner is now aprender-a7 (aprender-a2 is absent). The milestone stays 0.71.0 unless the operator sends the fix into a new 0.70.2 release commit.
Agent: aprender-a3
🤖 Generated with Claude Code
noahgift commentedon Oct 7, 2026
Cause found: a code defect, not infrastructure.
microkernel_8x6_true_asm(crates/aprender-compute/src/blis/microkernels/avx2.rs) pipelines the A panel four k-steps ahead. Each loop iteration reloadsymm6..ymm9from the next 4×8 block of A, and the last iteration did so too. That read up to 128 bytes past the end of the panel:(4 - k % 4)k-steps of 32 bytes. The values were never used, so results were always correct.For the last row panel, the packed-A thread-local buffer is sized exactly, so the over-read leaves the allocation. When the next page is unmapped, the read faults. That matches the observed fault: a read (
error 4) at a page-aligned address, in ablis::prepackedtest thread, about 1 run in 20.Fix (local, under review for the release): on the last iteration the four reloads read the block being consumed, chosen with one
cmovonk_cnt == 1. Thek % 4remainder already reads A from memory. The kernel diff is +13/−4.Proof: a new test,
FALSIFY-BLIS-ASM-OOB-001, puts the end of the A and B panels on aPROT_NONEpage for k in {4–8, 12, 64, 255, 256}. On the release commit it hits SIGSEGV on every run. With the fix it passes, andcargo test -p aprender-compute --libpasses 3515/0/4 in four runs. ASAN cannot see this, because inline asm is not instrumented.Reach:
apr train/apr finetuneon CPU with AVX2+FMA (training matmul goes totrueno::blis::gemmand then this kernel). GGUF inference (apr run/serve/chat) does not reach it.Two non-author reviews (correctness, adversarial): both approve, nothing blocking.
fix(blis): the 8x6 asm kernel read up to 128 bytes past its A panel o…
release(0.70.2): notes name the CPU train/finetune crash fix (#4891)
noahgift commentedon Oct 9, 2026
Parked by ISSUE-TREE-001 rule 5 / C334 (triage-worker, quorum q9): keeps the tree at depth<=3 with <=5 subs per ticket while the epic budget stays unchanged. Reopen = pull. No live PR or branch; nothing deleted.