You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nextest: 5 aprender-serve lib tests SIGKILLed by 16G local cargo-memcap — local runs false-red (heavy-memory group; mega test allocates 34GB) #4343
Five aprender-serve lib tests get SIGKILLed by the 16G local cargo-memcap and falsely fail local nextest runs
On lambda-labs, cargo nextest run --profile ci -p aprender-serve -p apr-cli --lib reports 5 failed with SIGKILL (cargo-memcap: KILLED — this job hit its 16G memory cap, infra#930). None of the five fails an assertion. All five pass in CI's workspace-test, which has no memcap. The result is a false red on every local readiness sweep (seen on the 0.69.4 readiness run, batch/0.69.4-epic-4249 @ cc47da2).
Measured peak RSS, one test at a time (/usr/bin/time -f %M, debug build, lambda)
~34.4 GB by design (gpu/tests/streaming.rs:36: 32 L × 32768 pos × 32 H × 128 D × K+V × f32)
no, SIGKILLed 3/3 even with --test-threads 1
With the full suite at -j 8 they overlap each other and the rest of the suite, so any two of them together exceed the cap.
Proposal
Add a heavy-memory test group in .config/nextest.toml with max-threads = 1 and threads-required = "num-test-threads", following the existing serial-timing pattern. Put the four tests that fit under 16G in it. Each then runs alone and passes under the cap; measured: 4/4 pass with --test-threads 1.
The mega test can never pass under a 16G cap. Options:
(a) Keep it in CI only: gate it behind an env var or #[ignore] plus a CI --run-ignored line. The CI wiring means a workflow edit, which needs a check-in.
(b) Make it assert the memory formula without allocating: memory_bytes() is computed from the dimensions, so a lazy or zero-sized-backing constructor would test the same property.
(c) Add a local nextest profile whose default-filter excludes it, and have the memcap message point at that profile.
(b) is preferred: the test asserts arithmetic, and allocating 34 GB to check a product of five integers is waste.
Done when
LAMBDA_MEMCAP on (default): cargo nextest run --profile ci -p aprender-serve --lib on lambda reports 0 SIGKILL failures.
CI's workspace-test still runs all five tests, or runs their replacement assertions.
Folded into #5309 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Activity
noahgift commentedon Oct 14, 2026
Folded into #5309 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Agent: aprender-clerk
noahgift commentedon Oct 14, 2026
Row moved: #5309 was split into named masters (TRIAGE-100 master naming), so this ticket's checklist row is now on #5337.
Agent: aprender-clerk