Skip to content

FLAKE: intermittent SIGSEGV in libcuda during gdn device tests (aprender-gpu --lib) #4956

Closed

Description

@noahgift

Owner: aprender-cd. Kind: flaky test (moved off the PR path to the nightly CUDA lane; fix or remove).

What happened. CI run 37903771339, job yoga (section cuda-unit), on #4927 head 4715d5e: the aprender-gpu --lib test binary died with SIGSEGV (exit 139) at 08:22:44Z. The kernel log puts the fault inside libcuda.so (driver 595.99.02), in a test thread whose name is truncated to kernels::gdn::d.

Why it is a flake, not the code. A/B on the same runner and the same driver: run 37907342562, job 113747683686 (another sha, 09:00–09:17Z) passed cuda-unit, these tests included. crates/aprender-gpu has the identical tree id (547f69e4f3) on both shas and on main 9259972. The PR differs only in version strings.

Candidates. These are the 6 kernels::gdn::d* device tests that the control run reported ok and the crashed run never finished. They are now #[ignore] on the PR path and run by the nightly CUDA lane (cuda-nightly.yml, ada-yoga):

  • decode_attention::gdn_decode_attention_device_tests::gdn_decode_attention_indirect_is_bit_identical_to_direct
  • decode_attention_split::gdn_decode_attention_split_device_tests::gdn_decode_attention_split_matches_unsplit
  • delta_rule_scan::gdn_chunk_scan_device_tests::gdn_chunk_scan_is_bitwise_per_token_{0_8b,9b_grouped,27b_grouped}_shape
  • delta_rule_scan::gdn_chunk_scan_device_tests::gdn_chunk_scan_one_token_is_one_per_token_step

To close:

  • Reproduce: loop the 6 tests with the default test threads until SIGSEGV, then name the one test.
  • Fix it (race or driver bug, with an upstream reference) or remove it. Then drop the #[ignore] and its nightly step.
  • Check every other runner that executes these device tests, on its own driver version, for the same fault in its kernel log.
  • If the nightly is red or unmeasured for 7 days, it stops until the owner brings it back.

Refs #4927

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions