Skip to content

jax gate: test_jax_slowrot segfaults on CIT when test_flow_reuse_default.py is collected first (pre-existing on rift_O4d) #300

Description

@oshaughnessy-junior

Found while running the jax gate locally for the 2026-09-08 integration landing. Not introduced by that batch: it reproduces on the untouched base rift_O4d at 21772f5.

Reproducer (CIT, ~/.cache/jaxci_venv: python 3.13, jax 0.9.2 CPU; seen on ldas-grid, ldas-pcdev11 and citlogin6):

cd MonteCarloMarginalizeCode/Code
PYTHONPATH=$PWD JAX_PLATFORMS=cpu OMP_NUM_THREADS=1 python -X faulthandler -m pytest -v -k test_rotation_path_a     test/jax/test_jax_slowrot.py test/jax/test_flow_reuse_default.py

Result: SIGSEGV (exit 139) inside test_rotation_path_a, no Python frame printed by faulthandler. The same command without test_flow_reuse_default.py passes, and so does the file alone (3 passed). Collecting test_nuts_phimarg.py, test_tvals_grid_convention.py or test_angle_marg_smoke.py instead does not trigger it. test_flow_reuse_default.py loads the driver script through importlib at collection time, which is the suspected global-state change (lal/lalsimulation or jax config) that the slowrot precompute later trips on.

Effect on .travis/test-jax.sh: any shard whose FILES put test_flow_reuse_default.py after test_jax_slowrot.py dies at ~7% on these hosts. GitHub's runner (pip jax) has not shown it.

Logs: /scratch/richard.oshaughnessy/yolo_20260908/logs/probe_*.log (six probes, base and merged).

Related, same session: a single long pytest process on CIT hits vm.max_map_count=65530 after ~150 JIT-heavy tests (measured 61k maps at 7.5 GB RSS) and dies with SIGABRT in jax compile; this is why the full gate aborts at ~57-85% here and is unrelated to any branch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions