Skip to content

nextest: 5 aprender-serve lib tests SIGKILLed by 16G local cargo-memcap — local runs false-red (heavy-memory group; mega test allocates 34GB) #4343

Closed as not planned

Description

@noahgift

Five aprender-serve lib tests get SIGKILLed by the 16G local cargo-memcap and falsely fail local nextest runs

On lambda-labs, cargo nextest run --profile ci -p aprender-serve -p apr-cli --lib reports 5 failed with SIGKILL (cargo-memcap: KILLED — this job hit its 16G memory cap, infra#930). None of the five fails an assertion. All five pass in CI's workspace-test, which has no memcap. The result is a false red on every local readiness sweep (seen on the 0.69.4 readiness run, batch/0.69.4-epic-4249 @ cc47da2).

Measured peak RSS, one test at a time (/usr/bin/time -f %M, debug build, lambda)

test peak RSS alone under 16G?
gguf::tests::tests_14::test_parity018a_dequantized_weight_cache_production 6.5 GiB yes
gpu::streaming_kv::kv_contract_tests::falsify_kv_001_memory_formula 6.1 GiB yes
gpu::tests::gpu_compute::test_ultra_long_context_memory_bound 7.8 GiB yes
gpu::tests::gpu_compute::test_super_long_context_memory_bound 15.5 GiB barely
gpu::tests::gpu_compute::test_mega_long_context_memory_bound ~34.4 GB by design (gpu/tests/streaming.rs:36: 32 L × 32768 pos × 32 H × 128 D × K+V × f32) no, SIGKILLed 3/3 even with --test-threads 1

With the full suite at -j 8 they overlap each other and the rest of the suite, so any two of them together exceed the cap.

Proposal

  1. Add a heavy-memory test group in .config/nextest.toml with max-threads = 1 and threads-required = "num-test-threads", following the existing serial-timing pattern. Put the four tests that fit under 16G in it. Each then runs alone and passes under the cap; measured: 4/4 pass with --test-threads 1.

  2. The mega test can never pass under a 16G cap. Options:

    • (a) Keep it in CI only: gate it behind an env var or #[ignore] plus a CI --run-ignored line. The CI wiring means a workflow edit, which needs a check-in.
    • (b) Make it assert the memory formula without allocating: memory_bytes() is computed from the dimensions, so a lazy or zero-sized-backing constructor would test the same property.
    • (c) Add a local nextest profile whose default-filter excludes it, and have the memcap message point at that profile.

    (b) is preferred: the test asserts arithmetic, and allocating 34 GB to check a product of five integers is waste.

Done when

  • LAMBDA_MEMCAP on (default): cargo nextest run --profile ci -p aprender-serve --lib on lambda reports 0 SIGKILL failures.
  • CI's workspace-test still runs all five tests, or runs their replacement assertions.

Activity

  1. added this to the backlog milestone on Sep 24, 2026
  2. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into #5309 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.

    Agent: aprender-clerk

  3. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Row moved: #5309 was split into named masters (TRIAGE-100 master naming), so this ticket's checklist row is now on #5337.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions