Skip to content

0.70.2 T42: determinism holds an ARM64 runner waiting for an X64 artifact that cannot come #4747

Closed

Description

@noahgift

Measured 2026-10-04, read-only: all 6 determinism reds (#4623 #4677 #4705 #4707 #4716 #4724) are X64 artifact-wait timeouts, with 0 hash mismatches. ARM64 was built every time, and each wait held an ARM64 clean-room runner for about 3665 s. Cost: about 6 runner-hours of the two-runner ARM64 pool, and 6 false reds that each need a full re-run.

Fix (signed by the operator): determinism starts only after x86-main concludes, so its wait for the X64 raster drops from 3600 s to 0. A late or missing X64 raster is RED, never a skip, where it used to wait.

Status: built; review passed. In the CI lane after T43.

K^ 3. Owner aprender-a7k. Refs #4678.

Activity

  1. noahgift commented on Oct 6, 2026

    @noahgift
    ContributorAuthor

    Closing: the bar is met on main since #4755 (acd5031, merged 2026-10-05 04:44Z).

    Read-only sample: the last 40 ci.yml runs after the merge.

    • 23 determinism jobs ran, and all 23 are green. On the ARM64 runner each took 67–76 s (median 70 s). Before the fix, each X64 artifact wait held that runner for about 3665 s.
    • None of the 23 failed on an artifact wait; before the fix, 6 of 6 reds were such timeouts.
    • 10 more determinism jobs were cancelled before a runner picked them up (0 s), so they held no ARM64 runner.
    • Ordering: in run 37430767012, x86-main completed at 08:12:01Z and determinism started at 08:12:04Z.
    • In 7 runs x86-main failed and determinism was still green, which is correct. In the two checked (37414554056, 37406965027), the X64 raster had been uploaded early (determinism-X64 at 05:06:37Z and 03:13:28Z), and only the later "wait for every one to finish" step failed.
    • Not seen live yet: a missing X64 raster giving a fast RED. That path is proven only by the guard self-test with planted inputs (check_determinism_wait.sh --selftest 23/23, mutants 10/10) and by if: ${{ !cancelled() }}.
    run event ARM64 time
    37405652999 pull_request 67 s
    37412136923 pull_request 67 s
    37412165676 merge_group 67 s
    37408334484 merge_group 68 s
    37413257498 pull_request 68 s
    37406965027 pull_request 69 s
    37430767012 merge_group 69 s
    37434689547 push 69 s
    37408602581 merge_group 70 s
    37415220939 push 70 s
    37418280084 push 70 s
    37427811679 pull_request 70 s
    37430348357 push 70 s
    37408814946 pull_request 71 s
    37412885803 merge_group 71 s
    37415222035 merge_group 71 s
    37415223723 merge_group 71 s
    37418420322 push 71 s
    37408603699 merge_group 72 s
    37414602114 merge_group 72 s
    37426900756 merge_group 74 s
    37409837217 pull_request 75 s
    37414554056 pull_request 76 s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions