You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
FLAKE: intermittent SIGSEGV in libcuda during gdn device tests (aprender-gpu --lib) #4956
Owner: aprender-cd. Kind: flaky test (moved off the PR path to the nightly CUDA lane; fix or remove).
What happened. CI run 37903771339, job yoga (section cuda-unit), on #4927 head 4715d5e: the aprender-gpu --lib test binary died with SIGSEGV (exit 139) at 08:22:44Z. The kernel log puts the fault inside libcuda.so (driver 595.99.02), in a test thread whose name is truncated to kernels::gdn::d.
Why it is a flake, not the code. A/B on the same runner and the same driver: run 37907342562, job 113747683686 (another sha, 09:00–09:17Z) passed cuda-unit, these tests included. crates/aprender-gpu has the identical tree id (547f69e4f3) on both shas and on main 9259972. The PR differs only in version strings.
Candidates. These are the 6 kernels::gdn::d* device tests that the control run reported ok and the crashed run never finished. They are now #[ignore] on the PR path and run by the nightly CUDA lane (cuda-nightly.yml, ada-yoga):
Activity
test(gpu): move six gdn device tests to the nightly CUDA lane (flake #…
fix(gpu): the device-filling pressure test took no claim, so a neighb…
release: 0.70.3 — streaming tool calls (paiml#4927)
fix(gpu): a test filling the device let neighbours' allocations OOM i…
fix(gpu): a shared cubin temp file let a reader load a partial cubin (p…
fix(gpu): cuBLAS tests synchronized the context inside another test's…
test(gpu): run the six gdn device tests in the CUDA unit job again (p…