LAB: dogfood-nightly has never been green (run 38155045677) #5199

Activity

1. noahgift commented on Oct 12, 2026

   

   noahgift

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Read the failing step of run 38155045677. Its own stdout is empty, so these rows come from the run's dogfood-receipt artifact (dogfood-nightly.log, 79 lines). The verdict is NO-GO, with 9 red rows. Each row below is one failing gate.

   Workflow defect, measured, separate from the 9 rows. The step runs under the default bash -e shell, and set -uo pipefail does not turn -e off. So bash scripts/dogfood.sh ... > log 2>&1; rc=$? exits the step the moment dogfood is red. The tail, the receipt copy, the tree-clean check and the R5 negative control (--receipt-only must refuse a nightly receipt) never run. Repro: bash -e -c 'set -uo pipefail; false; rc=$?; echo reached' prints nothing and exits 1. Fix shape: rc=0; bash scripts/dogfood.sh ... || rc=$?, plus the same for the R5 call. Without it, a red night never exercises the negative control.

   Runner setup: the gate never measured the code (4). Each needs the runner provisioned, not a code change.

   1. tokenizer_parity exit 2: REFUSE, the pinned llama.cpp build has no llama-tokenize binary.
   2. dogfood_comparator_env_tests exit 1: the pinned llama.cpp does not resolve (scripts/llama_bin.sh rc 1).
   3. dogfood-use: no model was exercised, because the runner's models directory is empty.
   4. check_model_ladder (scoped crux-smoke): one named host has no CRUX receipt from the release binary.

   Code or tree (5).
   5. check_tokenizer_nonascii_parity: aprender-serve --test tokenizer_nonascii_4005 fails; this is the open #4005 non-ASCII tokenization difference.
   6. float16_greedy_parity exit 1. Detail is in the run's worklog, which the artifact does not carry. To be re-measured.
   7. surface_audit_bins_gate: 29/30 binaries. [[bin]] aprender-ci-tools has no row in docs/audits/surface_audit.csv.
   8. package_tarball_build exit 2: rustc failed compiling apr_cli from the packaged tarball. Root cause not yet read.
   9. pmat-comply CB-200: head 618 > base 616. Two definitions below min_grade were added between v0.70.3 and this head. Fix them; no baseline gets raised.

   (The dogfood-gates row is the roll-up of rows 1, 2 and 4 to 8: 7 of 18 declared gates are red.)

   Next: the step fix first, so the next red night still exercises R5. Then 7 and 9, which are small, then 8. I'll re-measure 6 on the next night. Rows 1–4 go to runner provisioning. No gate is loosened and no row is skipped.
2. noahgift

   added a commit that references this issue on Oct 12, 2026

   fix(deep-nightly): a red doctest run ended the step at `; rc=$?` with…

   ...

   7109d9a
3. noahgift commented on Oct 12, 2026

   

   noahgift

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Update on two rows:

   • CB-200: no longer red on main. Measured scripts/cb200_head_vs_base.sh --base v0.71.0 --head HEAD at main 8c12fca: ok CB-200 head 613 <= base 618 (HEAD vs v0.71.0; one scanner, one run: pmat 3.42.0). The red night measured 618 > 616 against v0.70.3. No fix is needed on the branch.
   • Workflow -e defect: fixed on branch fix/5199-dogfood-nightly-reds. The dogfood step had 3 ; rc=$? sites. deep-nightly.yml's doctest step had the same defect, and it is fixed on the same branch. The PR opens in its queue slot.
4. noahgift

   added this to the 0.71.1 milestone on Oct 12, 2026
5. noahgift

   added a parent issue on Oct 14, 2026

   • B2: root-cause fixes for red nightlies on main #4715
6. noahgift

   modified the milestones: 0.71.1, backlog on Oct 14, 2026
7. noahgift commented on Oct 14, 2026

   

   noahgift

   on Oct 14, 2026

   ContributorAuthor

   More actions

   Folded into #4715 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it. Pushed branch: fix/5199-dogfood-nightly-reds.

   Agent: aprender-clerk
8. noahgift

   removed a parent issue on Oct 14, 2026

   • B2: root-cause fixes for red nightlies on main #4715
9. noahgift

   closed this as not plannedon Oct 14, 2026