You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Models nightly ladder producer failed with exit 3 #5099
Evidence from the lambda user journal (aprender-wprodmodels, 2026-10-12). Times CEST.
10-10 23:00 (the run this ticket was filed for): crux, ladder and publish all exited 3 with the same refusal: MODELS-NIGHTLY REFUSED: a registry token is reachable (env, CARGO_HOME or $HOME/.cargo): this producer holds none.
The ladder's exit 3 is the same cause as #5098 and #5100, not a separate defect.
Since then the guard passes: 10-11 06:00 publish ran its check and printed nothing pending; 10-11 21:46 crux printed green at 812dbe3eb ... models_t1 GO (CRUX smoke) on lambda and gx10, and publish at 22:45 printed PUBLISHED nightly-evidence a85548be1. Today no credentials file exists under the cargo home and the user manager sets no CARGO_* variable (checked by stat and name only).
10-11 23:00 (the latest night): crux and ladder exited 3 again, with a different and correct refusal: another models_t1.sh is running (a release's models step shares the GPU hosts): ... models_t1.sh 0.71.0 cc71bde02.... The release held the GPU hosts, so the nightly correctly stood down.
Side observation, not exit 3: on 10-11 the ladder was stopped by SIGTERM 41 s and 11 s after starting (19:22:33, 22:45:28), each time after a crux run that was not started by the timer (18:21, 21:46). The journal does not name who sent the stop.
Fix: none in this repo. The token guard and the concurrency guard both did their job. The units and the wrapper that start these runs live in the infra repo, not here, so there is no aprender branch to open. Leaving the ticket open for the owner to close.
Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Activity
noahgift commentedon Oct 12, 2026
Evidence from the lambda user journal (aprender-wprodmodels, 2026-10-12). Times CEST.
10-10 23:00 (the run this ticket was filed for): crux, ladder and publish all exited 3 with the same refusal:
MODELS-NIGHTLY REFUSED: a registry token is reachable (env, CARGO_HOME or $HOME/.cargo): this producer holds none.The ladder's exit 3 is the same cause as #5098 and #5100, not a separate defect.
Since then the guard passes: 10-11 06:00 publish ran its check and printed
nothing pending; 10-11 21:46 crux printedgreen at 812dbe3eb ... models_t1 GO (CRUX smoke) on lambda and gx10, and publish at 22:45 printedPUBLISHED nightly-evidence a85548be1. Today no credentials file exists under the cargo home and the user manager sets no CARGO_* variable (checked by stat and name only).10-11 23:00 (the latest night): crux and ladder exited 3 again, with a different and correct refusal:
another models_t1.sh is running (a release's models step shares the GPU hosts): ... models_t1.sh 0.71.0 cc71bde02.... The release held the GPU hosts, so the nightly correctly stood down.Side observation, not exit 3: on 10-11 the ladder was stopped by SIGTERM 41 s and 11 s after starting (19:22:33, 22:45:28), each time after a crux run that was not started by the timer (18:21, 21:46). The journal does not name who sent the stop.
Fix: none in this repo. The token guard and the concurrency guard both did their job. The units and the wrapper that start these runs live in the infra repo, not here, so there is no aprender branch to open. Leaving the ticket open for the owner to close.
noahgift commentedon Oct 14, 2026
Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.
Agent: aprender-clerk