Repository navigation
ci: mutants-cuda never runs its scope ("Failed to connect to bus", no measured.txt) #4895
Closed as not planned
Description
The mutants-cuda section of ci.yml (ci/sections.yml:3401) fails on PR #4890 at b11fd23, in both attempts of run 37595776609 (jobs 112708920123 and 112728461402).
What the log shows:
- the cuda-only lib test set is derived (cuda binary 18085 tests, default 16293, cuda-only 1795);
- the mutants step then prints
Failed to connect to bus: No medium foundand fails after ~690 s; - the gate step finds no
mutants-cuda-gate/measured.txt(ci/sections.yml:3490): "the shard never ran its scope".
So this is a measurement that never ran, not a surviving mutant. It does not block the 0.70.2 release: the release reads only ci / gate and workspace-test, which are green on b11fd23 (attempt 2).
Owner: aprender-wmainred. Done when mutants-cuda writes measured.txt and is green on main, or is re-ruled, within 7 days of the v0.70.2 tag. Related: #4715.
Activity
noahgift commentedon Oct 7, 2026
Same job on the 0.70.2 merge-back, #4909 (run 37676897108, head
fd31b82a85; its tree is identical tov0.70.2):systemd-runcannot reach the user bus, so nomeasured.txtis written. That is 2 of 2 runs on this tree, so the cause is structural.aprender-servecuda lib test binary was killed with SIGKILL (exit 137) atgpu::tests::imp_1004a::test_parity_120b_full_layer_cached. The same section passed on the same tree at 12:50Z (run 37623855045: 1781 passed). The sections run at the same time, so memory pressure is a candidate cause, not a named one.gatetreats this job as advisory, so it does not block the merge-back. No rerun was made.Agent: aprender-a3
noahgift commentedon Oct 8, 2026
#4912 removes the
mutants-cudasection from the PR and merge-queue path. It cannot block a merge, so it is LAB.After #4912 merges, nothing runs it:
mutants-nightly.ymlcovers the CPU mutants only. This ticket is now its one LAB ticket. To bring it back, give it a nightly home on an idle GPU host, outside the PR, merge-queue and release jobs, and fix the scope failure above first. It never comes back to the PR path.noahgift commentedon Oct 8, 2026
Update 2026-10-08 (infra triage): the scope step does not fail for lack of linger. The job runs inside the ephemeral runner container, which is unprivileged, has a private cgroup namespace and no /run/user, so
systemd-run --userhas no user bus to reach; host-side linger is already on. The fix is in aprender: run mutants-cuda where a user bus exists (for example the yoga-gpu runner) or change the scope step. Since #4912 (merged 2026-10-08 01:03Z) the job is off the PR and merge-queue path, so this is LAB work for its nightly home.noahgift commentedon Oct 14, 2026
Folded into #4715 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Agent: aprender-clerk