Repository navigation
LAB: Nightly Train printed NOT RELEASABLE with every lane not_measured (late start judged a head no producer ran on) #4963
Description
LAB (C324): this check cannot block a merge, a tag or a publish.
nightly-train.ymlruns on a schedule. It is not a required check.scripts/release/nightly_train.shsays "Report-only: this script blocks nothing and no release script reads it".- I checked that claim against the tree at
92599723e6. No release script and no other workflow reads the train's line or its exit code.release_ready.shandcontracts/release-ready-v1.yamlonly use the train's lane names asproducer:labels.
What happened
Nightly Train run 37924118002, job train, step "RELEASABLE or NOT RELEASABLE for main's head" printed:
NOT RELEASABLE: deep-examples, not_measured (+13 more) [C=92599723e6 pin=unpinned]
All 28 lanes in reds.tsv are not_measured. None is red.
Cause (read from the run, not inferred)
- The run started late, after main had moved.
- The cron is
45 4 * * *, but this scheduled run was created at 11:30:11Z. - At 04:45Z main's head was
e0cb1372d2, committed 2026-10-08 20:39Z. That is the commit the night's producers ran on. 92599723e6landed on main at 08:05Z. By 11:30Z the train judged that commit, which no producer had run on yet. So 22 lanes read "no run on C; newest run … on e0cb137". Of the other 6, 3 have no producer on main, 1 (qwen-story) has never run on main, and 2 are listed below.- This was the train's first scheduled run:
history.tsvhas one row. Why GitHub created the run 6h45m late is not measured.
- The cron is
- Some lanes would not be measured even with an on-time start:
- 3 verdict lanes have "no nightly producer on main yet":
dogfood,readinessandassets. deep-examples(verdict lane): its newest run, 37558444740, is ona719e1aa28; about 299 h lost.mutants(info lane): its newest run, 37426972044, is onece9e563bd; about 269 h lost.
- 3 verdict lanes have "no nightly producer on main yet":
So with today's design the verdict would have been NOT RELEASABLE at 04:45Z as well. A late start only turned every lane into not_measured.
For the owner (pick one; no change was made here)
- When a scheduled run fires late, judge the head the producers last ran on, or print that the run fired after main moved. Today's line names
deep-examplesas the first failure, which reads like a defect in that lane. - The
deep-exampleslane has not been measured on main for about 12 days. dogfood,readinessandassetshave no producer on main yet.
Refs #4686
keep-open: LAB ticket per C324; it closes when the train prints a line whose lanes were measured on the head it names.
Activity
noahgift commentedon Oct 9, 2026
Owner: aprender-wprodmodels.
Plan for the late start: when the verdict producers' newest scheduled runs sit on a commit that main has since moved past, the train judges that commit and prints main's head beside it. A late start then reads the evidence the night produced, instead of turning every lane into
not_measured.That alone does not turn the train green.
deep-exampleshas no recent run on main, anddogfood,readinessandassetsstill have no producer. Those are producer work, separate from this fix.noahgift commentedon Oct 9, 2026
Nightly Train run 37924118002: the 14 not_measured verdict rows, by cause
Read from the run's bundle and the producer jobs' logs (tail and grep only).
The train judged C =
92599723e6. Every producer's newest run on main was one0cb1372d2. The train starts late every day: since 10-05 every Nightly Train run was created between 11:19Z and 11:49Z, against its 04:45Z cron. By then main had moved past the commit the nightlies ran on, so 11 rows readno run on C. The fix on this ticket's branch (a848f53552) judges the commit the night ran on. The last column says what each row reads when the train judgese0cb1372d2.Setup / runner
modelsjob was skipped. The relay printedMODELS-NIGHTLY RELAY not_measured at e0cb1372d: no bundle for e0cb1372d on nightly-evidence. Thenightly-evidencebranch does not exist on origin: the GPU host has never published a bundle. The fix is the host unit, not code.curl: (22) The requested URL returned error: 403reading the paiml/infra clean-room runs with theINFRA_ACTIONS_READsecret, then printednot_measured — paiml/infra clean-room runs were not read. The cause is the credential's scope or validity. Not touched here.disabled_manually. Its newest run is ona719e1aa28(10-07), so it can never be on C.Own code
a848f53552on this ticket.green — all 1 job(s) of b2-gpu at C succeeded.SMOKE RED: 2 of 29 bins.pvandpv-satprintv0.70.2+no-gitwith no sha (reasonversion-no-sha). The other 27 bins print(e0cb1372d2). pv's version string does not carry the build sha. Tracked in #4896.summary pass=597 fail=29 timeout=5 needs-args=30 needs-hardware=2 needs-data=316. 14 of the 29 fails are examples the harness should class as needs-args or needs-hardware. 8 need arguments:qwen35_parity,qwen35_prefill_parity,think_ab,rex,ssc_eval,ssc_preflight,check_layer4,constrain_mask_overhead. 4 need CUDA:test_gemv_correctness,profile_cuda_trainer,bench_gemv,test_tc_attention. 1 needs a tty:calculator_tui. 1 needs a LAN host:test_mac_worker. The other 15 fails (parity_035–040,debug_q4k/q6k_controlled,qa_chat,qa_serve, …) and the 5 timeouts (servers and TUIs killed at 180 s) were not read further.main is 0.70.2 and crates.io already has 0.70.2: --verify would resolve the registry copy (the gate's documented false red); main must be bumped first (BLD-002 R3). The fix is main's version bump.no nightly producer on main yetCount. Setup/runner: 2 rows (models, cleanroom-cpu), plus half of deep-examples (disabled). Own code: 11 rows (6 late start, deep-bins-smoke, publish-dryrun, 3 with no producer), plus the other half of deep-examples (its harness and examples). With the late-start fix alone, the train would print
NOT RELEASABLEon deep-bins-smoke (red) and 7 not_measured rows.Agent: aprender-wprodmodels
noahgift commentedon Oct 9, 2026
Triage: this is the one ticket for Nightly Train. Newest run 37924118002 (2026-10-09 11:30Z, sha 9259972, schedule) failed; the runs of 10-06, 10-07 and 10-08 failed too (37456774718, 37613335130, 37771142959). Not a required check on main (required: ci / gate, workspace-test). Milestone set to 0.71.1; it stays unlinked until #4715 has a free slot. — aprender-triage-worker
noahgift commentedon Oct 10, 2026
Nightly Train failed again on 2026-10-10 at 10:47Z:
NOT RELEASABLE: deep-examples, not_measured (+13 more) [pin=unpinned]. Same LAB class as this ticket; tracking here, no new ticket. Still needs an owner label.noahgift commentedon Oct 12, 2026
Branch
fix/4963-lanes-rows-test@227e79d0e05691be8a1aaf9d43f4e5bfb8c50949 (branch-only, no PR).Cause of the red. Run 38248921564 failed in the train's "case table and mutants" step, before any verdict:
nightly_lanes_rows_test.shshowed 64/91 rows. The train table had moved on and the test still pinned the old version. #5142 madeworkflow_dispatcha night, and C324 moved deep-bins-smoke, cleanroom-cpu and publish-dryrun to LAB. The broken rows:The mutant kills were hollow while the table was red.
Fix (test only; the train is untouched)
~.planted_push_never_reaches_verdictandpush_alone_is_not_measured. New rowdispatch_is_a_night.lab_red_never_names_the_lane.BROKE preflight/baselineand 42/43.Measured (local)
Late start. Already fixed on main by #5086, so
wprod/4963-late-start@a848f53552 is superseded.