yoga cuda-unit: cta64_vs_cta32_vs_cublas_fp16 CUDA_ERROR_OUT_OF_MEMORY under the host GPU lock (flaky, 1 of 2 on yoga-eph) #4593

On yoga-eph (yoga's ephemeral GPU runner, first live on 2026-09-28 via infra#1230), aprender run 36410838844 failed its yoga job at section cuda-unit:

test driver::cublas_tests::cta64_vs_cta32_vs_cublas_fp16 ... FAILED
C: MemoryAllocation("CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (code: 2)")
test result: FAILED. 2668 passed; 1 failed; 12 ignored
error: test failed, to rerun pass `-p aprender-gpu --lib`

The job held the host GPU lock (log: test binaries run under: flock -E 175 -w 600 /run/lock/fleet-gpu/gpu.lock, infra#1182). The next yoga job on the same runner, in run 36410817478, succeeded.

So the OOM came from one of two places:

• (a) a GPU user on yoga that does not take /run/lock/fleet-gpu/gpu.lock. The feat(aprender-core): apr-cli-operations-v1 OPS-003 PARTIAL_ALGORITHM_LEVEL #1182 watcher ANDONs on an unlocked GPU pid, so its journal answers this.
• (b) contention inside the binary: libtest runs these tests on parallel threads against one small GPU, and the cuBLAS fp16 comparison allocates the most.

Not measured yet: which of the two it was. Suggested check: rerun -p aprender-gpu --lib cublas_tests -- --test-threads=1 on yoga, and read the watcher journal for 11:09–11:11Z.

🤖 Generated with Claude Code