Repository navigation
0.70.2 T18: RQ-2 the full model ladder runs nightly on main's head; release day reads it for the same H #4701
Description
Row T18 of aprender-a7's 0.70.2 PROPOSE-TICKET list (cop-inbox handoff/0702-propose-tickets.md, 08:24Z section, from C299 "all recommended"; ranking in handoff/0702-delay-rank.md), verbatim:
- T18 RQ-2 ladders nightly. The full model ladder on both GPU hosts runs each night on main's head as a nightly lane.
Release day reads the result for the same H and does not run it again. First measure E2: the ladder (~160 min)
plus full CRUX must fit the 6 h night on the slower GPU host, so full CRUX there must take ≤200 min.
Lane first. The release-day switch comes only after 3 green nights (P4), and it changes what release day
accepts, so it needs sign-off. −160 min per release. Serves B5, B1. K̂ 5.
Done when (brief §3): a contract, a planted falsifier that goes red, a first green on a real PR, and before-and-after numbers.
Train 0.70.2, P0. Owner: aprender-a7. Minted by the cop's clerk under C297 §4 / C298.
Folded sub-tickets (TRIAGE-100, 2026-10-14)
These open sub-tickets were folded here as checklist rows and closed as not planned so the open count stays at or under 100. Reopen one when its work resumes; an open PR for it reopens it.
- 0.69.4: run the #4250 serve batched-prefill test where the Qwen3.5 model exists (it is a model-absent SKIP in ci.yml cuda-unit) #4251 — 0.69.4: run the 0.69.4 G0: serve-level test that fails unless apr serve prefills Qwen3.5 through the batched path (#3596 countermeasure) #4250 serve batched-prefill test where the Qwen3.5 model exists (it is a model-absent SKIP in ci.yml cuda-unit) (branch:
PMAT-4251-serve-prefill-nightly) - 0.70.2 P0: Qwen2.5-0.5B-Instruct-f16.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) #4661 — 0.70.2 P0: Qwen2.5-0.5B-Instruct-f16.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) — done when: the ladder row is OK on lambda and gx10. (branch:
fix/4661-chat-eog-stop-set,fix/4661-cuda-serve-gibberish) - 0.70.2 P0: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) #4662 — 0.70.2 P0: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) — done when: the ladder row is OK on lambda and gx10. (branch:
fix/4662-moe-serve-runon) - 0.70.2 P0: Qwen3.5-0.8B-IQ4_XS.gguf golden_output thinking_on: measure the thinking budget for this model (known failure in 0.70.1) #4663 — 0.70.2 P0: Qwen3.5-0.8B-IQ4_XS.gguf golden_output thinking_on: measure the thinking budget for this model (known failure in 0.70.1) — done when: the ladder row is OK on lambda and gx10.
- 0.70.2 P0: Qwen3.5-0.8B-UD-IQ2_XXS.gguf golden_output: GPU forward falls back to CPU (known failure in 0.70.1) #4664 — 0.70.2 P0: Qwen3.5-0.8B-UD-IQ2_XXS.gguf golden_output: GPU forward falls back to CPU (known failure in 0.70.1) — done when: the ladder row is OK on lambda and gx10. (branch:
build-kaizen/4664-golden-refuse,fix/4664-iq2xxs-gpu-fallback) - 0.70.2 P0: ladder fit gate (#4016) refuses every .apr model (llama-fit-params cannot read .apr; exit 1 is read as does-not-fit) #4668 — 0.70.2 P0: ladder fit gate (Process rule: llama.cpp fit (llama-fit-params) is the required placement gate for every model certification/test until apr has a parity fit tool #4016) refuses every .apr model (llama-fit-params cannot read .apr; exit 1 is read as does-not-fit) (branch:
build-kaizen/4668-apr-fit) - Chat routes stop only on the declared EOS; completions stop on every end-of-generation marker (#4661) #4810 — Chat routes stop only on the declared EOS; completions stop on every end-of-generation marker (0.70.2 P0: Qwen2.5-0.5B-Instruct-f16.gguf cuda serve /api/chat: gibberish output (known failure in 0.70.1) #4661) (branch:
fix/4810-chat-eog-stop) - model_ladder: GPU ownership check reads the ladder's own apr as FOREIGN when $APR is under a directory symlink (CONTENDED, exit 75) #4876 — model_ladder: GPU ownership check reads the ladder's own apr as FOREIGN when $APR is under a directory symlink (CONTENDED, exit 75)
- model_ladder: a CPU serve probe held the GPU lock past its probe and stalled the same ladder's GPU leg (cause open) #4877 — model_ladder: a CPU serve probe held the GPU lock past its probe and stalled the same ladder's GPU leg (cause open)
- Models nightly ladder producer failed with exit 3 #5099 — Models nightly ladder producer failed with exit 3
- check_model_ladder: unarmed cells judge is not_measured on the nightly full ladder (#4667) #5192 — check_model_ladder: unarmed cells judge is not_measured on the nightly full ladder (0.70.2 P0: capability cells were not measured for 0.70.1 (~45 min/model x 26 models = 15-20 h per host) #4667)
Folded P0 and operator-worded rows (TRIAGE-100, 2026-10-14)
These tickets were folded here by the cop ruling of 2026-10-14 (sub-rows of this master). Each keeps its priority and its owner. The operator text each carried is quoted below exactly as written; if an owner objects, the ticket is reopened.
-
0.70.2 P0: capability cells were not measured for 0.70.1 (~45 min/model x 26 models = 15-20 h per host) #4667 — 0.70.2 P0: capability cells were not measured for 0.70.1 (~45 min/model x 26 models = 15-20 h per host) (P1 owner:aprender-wcensus)
For 0.70.1 the ladder ran without
--cellson lambda and gx10 (operator ruling C279, path A). The measured rate on cc4463f was about 45 min per model with 26 models, which is 15-20 GPU-hours per host. That does not fit a release window. -
0.70.2: test_analyze_workflow writes into the source tree (EACCES on host); 4 Qwen3.5 model tests time out on host and skip in CI #4669 — 0.70.2: test_analyze_workflow writes into the source tree (EACCES on host); 4 Qwen3.5 model tests time out on host and skip in CI (P1 owner:aprender-wcensus)
Found by the 0.70.1 release-commit test runs (operator ruling C283.4: the clean-container run decides for 0.70.1; this ticket carries the host-run failures to 0.70.2).
Activity
fix(release): the night reader could GO past a red it never read (#4701…
25 remaining items