Repository navigation
EPIC: release train 0.69.1 — every capability ORACLE-PROVEN (CRUX cells) on gx10 and lambda; no defer #3710
Description
EPIC: release train 0.69.1, every capability ORACLE-PROVEN on gx10 and lambda
Revised 2026-09-23 by the cop (aprender-6a), after the plan-mode design review (comment 5790578906) and the CRUX-design quorum (#3957, do-not-implement-as-written, 2/2). The original body is kept at the bottom.
Binding operator rules (verbatim)
- 2026-09-21: "you must ensure all models Q4_K CUDA work; the end. on gx10 and lambda-labs"
- 2026-09-23: "no defer; from now forward we practice toyota way and oracle approach of crux with capabiltiy report style from Alfredo". A cell is ORACLE-PROVEN or RED, with no third state. No waivers.
- 2026-09-23: "1. fold in MoE 2. fold in 88% coverage". The
*A3B*deferral is retired, and coverage ≥ 88 is required. - 2026-09-23: "CRUX is critical here, as we need quorum our style on them, as we could test "the happy path""
- Scope also covers every Qwen 2.5/3.5 that fits on either host, in any quant, IQ* / BF16 / .apr included (the "don't leave behind" rule), plus vLLM added to the CRUX comparators.
State
v0.69.0 is tagged (225b2a9ab) and on GitHub. It is NOT on crates.io and never will be. 0.69.1 supersedes it. Integration branch: release/0.69.1-batch-2.
done_when: a CELL REPORT, not a pass line
Every cell (model, format, quant, host, backend, verb, thinking) → number + oracle + positive control, all measured with the binary built from the release commit. verb ∈ {run, chat, code, serve (incl. streaming)}, host ∈ {lambda sm_89, gx10 sm_121}, backend ∈ {cuda, cpu}.
- Every held model's cell is PROVEN on both hosts. Zero RED.
inventory.deferredis empty. - Oracle, per 0.69.1 gate hardening (no GPU): sha-bound receipts, missing-field=FAIL, verb output oracles, CRUX judge strength, DEFER exits 2 #3957 as revised by quorum:
- (a) SAME-QUANT differential: apr vs the ggml family on the identical GGUF, greedy.
- (b) GROUND TRUTH: pre-certified prompts with verifiable answers (
<answer>X</answer>; code executed against unit tests; multi-turn state recall). hf/vLLM bf16 are the answerability control, not a vote. - Independent votes are ggml, hf and vllm. A split at gate time is RED.
- Controls: a positive control per (host, verb, thinking), and a planted-wrong negative control per verb, which must go RED.
- Kernels: every GPU-whitelisted type is proven at EVERY (k,n) shape its models use, with real bytes, |err|/Σ|w||x| ≤ 1e-5, planted faults RED. The CPU decoder is itself proven bit-exact vs gguf-py on every tensor.
-
.apr: tensor-chain to the source file AND runtime greedy equality with the source. - Gate integrity: receipts are bound to the release-commit sha; a missing field = FAIL; the judge reads every per-verb verdict; DEFER exits non-zero.
- Coverage ≥ 88% measured on the release commit, with the floor unchanged.
- Pre-publish dogfood GO on the release commit; clean-room, B2-GPU and assets green; crates.io publish (the operator is asked before
make publish).
Blocker rows
| Row | Ticket | Owner | State |
|---|---|---|---|
| Gate hardening (F1-F8) + CRUX judge | #3957 | aprender-6c [8b6b78] | in progress (F4a) |
| CRUX producer: prompt sets, code verb, hf/vllm rows | #3962 | aprender-83 (after #3952) | queued |
| vLLM engine driver | #3952 | aprender-83 | in progress (installs on both hosts) |
| IQ2_XXS CUDA GEMV | #3950 | aprender-6c [3ada9a] | in progress |
| IQ2_S CUDA GEMV | #3953 | aprender-f5 | device A/B |
| IQ3_XXS CUDA GEMV | #3963 | aprender-6c [3ada9a] after #3950 | queued |
| Q2_K CUDA GEMV | #3960 | aprender-f5 after #3953 | queued |
| aprender-core IQ dequantizers | #3947 / PR #3958 | aprender-9b | FOLDED fc942f6 (3 PASS) |
| BF16 CUDA | #3908 | aprender-3a | PROVEN on sm_89 AND sm_121 |
| Serve teardown + health wait | #3943 | aprender-3a | fixed; 27B re-run queued on gx10 |
| MoE qwen3moe CUDA cells | #3714 / #3817 | fold: cop; cells: aprender-70 after #3951 | queued |
| tinyllama degenerate output | #3914 | — | CLOSED: fixed by 350c557 (template), llama.cpp oracle |
| Coverage ≥ 88 | #3839 | aprender-9b after #3958 | queued |
| False provenance banner / envelope | #3955 | aprender-3a (after #3965) | queued |
| cuda-feature tests red on base | #3956 | aprender-19 [337148] | in progress |
| qa --json: skipped reads as passed:true | #3965 | aprender-3a | in progress |
| thinking-ON leg discards used_gpu; empty think passes | #3961 | aprender-6c [8b6b78] (inside #3957) | queued |
| Ollama bypasses the GPU lock | #3964 | cop (keep_alive=0 APPLIED, infra#918); sampler: aprender-f5 | partly done |
| GPU whitelist admission per shape | #3968 | aprender-dd | queued |
| gpu-q waiter spins when its ticket vanishes | #3966 | aprender-dd | queued |
| IQ4_XS device A/B (0.8B-IQ4_XS = RED-MODEL, operator (a)) | #3951 / PR #3967 | aprender-70 | PROVEN; fold pending |
| Dedupe IQ decoders (serve onto trueno_quant) | #3959 | — | after the GEMV rows |
| F2 GPU validation fails OPEN | #3973 | aprender-3a | queued (priority) |
| Jidoka audit: dogfood verdicts / qa perf gates / validate grade / missing must-REDs | #3974 | unowned — next free worker | open |
| Early full ladder at release head (find apr REDs early) | — | aprender-cf | in progress |
| CRUX producer: code + serve verbs | #3962 | aprender-19 [c630e1] | in progress |
| Empty think block (#3948) + ON-leg used_gpu (#3961) | #3948/#3961 | aprender-6c [8b6b78] | quorum: DNIAW with changes; queued |
| ptxas all emitted kernels | #3970 | aprender-19 [337148] after #3956 | queued |
| ONT-4c5 capability-cell extractor + arming rule | #3972 | aprender-6c [8b6b78] after #3957 | design-only |
Sequence
- Gate fixes that need no GPU (0.69.1 gate hardening (no GPU): sha-bound receipts, missing-field=FAIL, verb output oracles, CRUX judge strength, DEFER exits 2 #3957 F1-F4), then fold golden_output thinking-ON judge passes an EMPTY <think></think> — the leg cannot tell reasoning from skipping it #3948 and re-derive the true RED set.
- Oracle (0.69.1 gate hardening (no GPU): sha-bound receipts, missing-field=FAIL, verb output oracles, CRUX judge strength, DEFER exits 2 #3957 F6 + 0.69.1 BLOCKER: CRUX producer — per-verb prompt sets (incl. hard prompts) + code verb + hf/vllm quorum rows #3962) before any further whitelist flip.
- In parallel: kernels (IQ2_XXS (ggml 16) has no CUDA GEMV — 0.69.1 release blocker #3950 → IQ2_S (ggml 22) has no CUDA GEMV — 0.69.1 release blocker #3953 → IQ3_XXS → Q2_K (ggml 10) has no CUDA GEMV — 0.69.1 release blocker #3960, landing in that order), BF16 gx10, MoE cells, tinyllama, coverage.
- One full sweep of both hosts on the release commit, then the cell report, then the tag.
Original body (2026-09-21)
Operator, 2026-09-21, verbatim: "you must ensure all models Q4_K CUDA work; the end. on gx10 and lambda-labs", preceded by: "STOP THE LINE, you will ensure this works before .69".
State: v0.69.0 is tagged (225b2a9ab) and released on GitHub (16 assets). It is NOT published to crates.io and won't be: its ladder had qwen3-8b-q4km red on lambda (golden_output "Empty output"; green on gx10). 0.69.1 carries the fixes.
done_when
- a Q4_K sweep of EVERY Q4_K model file on lambda and gx10 (
apr qacapability+golden,apr run --gpuwith no CPU fallback), measured with the 0.69.1 release-commit binary: 0 red on both hosts. Sweep receipts committed. -
contracts/model-capability-ladder-v1.yamlencodes that universe: every Q4_K model on the fleet is a required rung (norequired: falseQ4_K rung), socheck_model_ladder.shenforces the rule on every future train - per-host ladder receipts for 0.69.1 committed in the bump PR (the step 0.69.0 missed, autopilot's INHERITED dogfood GO writes no receipt that check_publish_preflight.sh R5 accepts — v0.69.0 stopped at the publish preflight 43 min after tagging #3708)
- the real pre-publish dogfood (not inherited) GO on the release commit; clean-room, B2-GPU and assets green; crates.io 74/74
Rows
- qwen3-8b-q4km on lambda CUDA: aprender-fd
- further reds from the full sweep (running 15:2xZ): aprender-37
- ladder contract universe + receipts in the bump: release-tooling lane (aprender-f0, with autopilot's INHERITED dogfood GO writes no receipt that check_publish_preflight.sh R5 accepts — v0.69.0 stopped at the publish preflight 43 min after tagging #3708)
Activity
noahgift commentedon Sep 21, 2026
Scope set by the operator (verbatim, 2026-09-21 ~15:45Z)
Earlier the same day: "you must ensure all models Q4_K CUDA work; the end. on gx10 and lambda-labs" · "fix our release process that you think it is ok to publish models with 'most working'; this is p0 tire fire".
0.69.1 release bar (replaces the done_when above where it's stricter)
For every Q4_K model in each host's measured inventory × {lambda, gx10} × CUDA:
apr run --gpuanswers the golden prompts correctly, with no CPU fallbackapr chaton CUDA (today it hard-skips CUDA for every qwen35 model, apr chat hard-skips CUDA for every qwen35 model and prints a banner citing #3090 as unimplemented — #3090 shipped a day earlier, and the corrected replacement has zero callers #3595), multi-turn, correct answersapr serveloads the model (apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571) and/v1/chat/completionsreturns the correct answer, streaming and non-streamingapr code(the coding-agent verb) completes a scripted edit-and-verify task with the model as its backendKnown gaps against this bar: #3714 (qwen3moe has no CUDA path) · #3595 (chat skips CUDA on qwen35) · #3571 (serve can't load Qwen3.5, only layer 1 landed) · #3693 (long-context corruption) · #3711 (a GPU golden error reads as a skip) · lambda Q4_K reds (qwen3-8b, qwen2.5-coder-0.5b/7b).
noahgift commentedon Sep 21, 2026
All hands (operator, verbatim): "ask every claude code agent working on projects that would use apr to give you their user interace requirements and this get all hands on deck with .69". Consumer requirements intake: #3716 (asked of RAH, paiml-implement, arbiter/infra, apex and rmedia). Owners on the 0.69.1 path: qwen3-8b lambda (aprender-fd) · qwen2.5-coder lambda (aprender-37) · #3714 qwen3moe CUDA (aprender-eb) · #3711 GPU-error-as-skip (aprender-0e) · #3712 matrix measuring (aprender-62) · #3708/#3712 T-1/T-4 wiring (aprender-f0) · unowned: #3595 chat on CUDA, #3571 serve,
apr codeon Qwen3.5, #3693 long context, #3715 SHACL shape.noahgift commentedon Sep 21, 2026
Owners updated (all hands, 4 new workers from the operator): #3714 qwen3moe CUDA → aprender-eb · #3595 chat on CUDA + #3571 serve → aprender-c7 · #3715 SHACL release-readiness → aprender-97 · #3693 long context +
apr codeon Qwen3.5 → aprender-c3. Unchanged: fd (qwen3-8b lambda), 37 (qwen2.5-coder lambda), 0e (#3711), 62 (#3712 measuring), f0 (#3708 / #3712 wiring). Every 0.69.1 row now has an owner.noahgift commentedon Sep 21, 2026
Context rung, derived from MEASURED consumer traffic (RAH/quorum, paiml-implement-c9 on #3716, counted with Qwen3.5's own tokenizer rebuilt from the GGUF): over 113 real quorum briefs the p50 ≈ 20k tokens, p90 ≈ 60k, max ≈ 148k (the max's diff alone is 120,745). The garbled #3693 lane read 13,495. Output p99 over 365 verdicts fits 4,096 tokens. Arbiter, apex and rmedia-88 are still to report.
Proposed gate (cop recommendation; the operator confirms or amends):
noahgift commentedon Sep 21, 2026
Rows added from consumer requirements (#3716): #3720 response contract (weights digest, seed determinism, empty = error, refused ≠ did-not-run) → aprender-fd after qwen3-8b · #3568 JSON-schema-constrained decoding (moved from 0.70) → aprender-0e after #3711 · #3718
prompt_tokensinrun --json(P1, unowned). Host added to the matrix: intel x86_64 CPU for run/serve (rmedia is pinned to intel on measured cause GH-562). Qwen3-8b lambda root cause (aprender-fd): a gate defect, not a kernel defect. The golden test prompted Qwen3 in thinking mode, and 545 think tokens on x86 vs <512 on aarch64 left the answer empty after stripping. The fix builds golden prompts with production's no-think template. Fleet: gx10 disk 99% → 88% after the cop reclaimed 60 GB of idle root-owned CI caches (operator-authorized).noahgift commentedon Sep 21, 2026
Owners: #3596 batched CUDA prefill for Qwen3.5 → aprender-3e (new). Rule rev 2 in force after gx10's 15:56Z global OOM killed 18 CI containers (three concurrent interactive apr runs on GB10 unified memory): interactive GPU runs use
flock /tmp/apr-gpu.lock choom -n 1000 -- apr …. #3712 decision: the refusal lands with qwen3-8b kept atrequired: falseplus a note (so T-2 can't pass while it's broken and PRs aren't blocked); the flip torequired: truelands in the same 0.69.1 batch as fd's fix plus a green lambda receipt. Unowned: #3718 (prompt_tokens/prefill_ms) and the intel-CPU column of the matrix.noahgift commentedon Sep 21, 2026
Owners: #3718 (
prompt_tokens/completion_tokens/prefill_ms, serveusage) plus the intel-CPU column (run/serve on intel x86_64 for rmedia) → aprender-a8. Every 0.69.1 row now has an owner.noahgift commentedon Sep 21, 2026
Bar raised by the operator (verbatim, 2026-09-21 ~16:15Z)
Context: overdo it. Each Qwen3.5 model must be CORRECT (output matches the CPU reference, quoted identifiers intact, no silent corruption) at every rung up to the model's own declared context length (GGUF
*.context_length, read from the file, not picked here). That's well past the consumers' measured max (RAH/quorum 148k tokens; rmedia's plan 128k). Required rungs per model: 4k · 8k · 20k (consumer p50) · 60k (p90) · 148k (max) · the declared context length. VRAM is not an excuse on its own: the hybrid's Gated DeltaNet layers carry fixed-size state and only the full-attention layers grow a KV. Where memory truly can't hold a rung on a host, the refusal names the arithmetic, and it's a FAIL for the release unless the operator waives that cell by name.Verbs dogfooded in BOTH thinking modes: run, chat, serve, code, each with thinking ON and OFF:
--think/--no-think; serve takes it per request), and--helpdocuments it.Consumer latency still applies: a correct answer at 148k that takes an hour isn't usable, so time-to-first-token per rung is recorded on both hosts (#3596 batched prefill).
noahgift commentedon Sep 21, 2026
Operator decision (verbatim, 2026-09-21 ~16:40Z): "a", answering: "(a) Waive the 27B lambda cells above 98k, covered by gx10 running the 27B at the full 262k (recommended) · (b) KV-cache quantization so the 27B fits on the 4090 · (c) keep them as release failures".
How it's encoded (a rule, not a hand-listed exemption):
noahgift commentedon Sep 21, 2026
Owners: #3725 long-context decode (site 4: decode attention reading f16 and f32 KV) → aprender-3e [8f56c1] (a new session; a second live session is also named aprender-3e [44388e], owner of #3596, so refs are used in routing). #3733 (
apr inspect --jsondrops qwen35.* keys, blocking the gate) → aprender-a8, first in lane. Every 0.69.1 row now has an owner.noahgift commentedon Sep 21, 2026
Owners: #3739 CRUX inference dogfood (apr vs llama.cpp vs ollama on the same GGUF, per verb) → aprender-76 (new). #3563 comparator pin (split A→C→B; A = resolve + prove on both hosts, including building d1d3c3396 on gx10) → aprender-f0 after #3462. Every 0.69.1 row has an owner.
noahgift commentedon Sep 21, 2026
Close-step obligation (operator, verbatim, 2026-09-21): "will message alfredo via slack when it is out". 0.69.1 is not closed until Alfredo Deza has a Slack DM with the crates.io + GitHub release links, sent after all crates are published and the release is marked non-pre-release. The draft and a fallback (push-notify Noah with the draft) are in the cop's ledger. Cop: aprender-04.
noahgift commentedon Sep 21, 2026
Close-step obligation 2 (operator, verbatim, 2026-09-21): "and that arbiter will also send release message to #arbiter channel". Measured: arbiter can't post a release announcement today (slk-post-v1 forces in-thread answers; no release sensor). Filed as paiml/infra#898: a top-level post derived from the published release + crates.io read-back, once per release, first-green dry-run on v0.68.2. The 0.69.1 close step verifies that post in #arbiter, and the Slack DM to Alfredo.
noahgift commentedon Sep 21, 2026
Intel CPU column (rmedia's host, GH-562): Qwen3.5 Q4_K_M,
apr run+apr serve, CPUMeasured by aprender-a8 on 2026-09-21, 17:13–17:48Z.
Setup
intel(Xeon W-3245, 32 threads), reached over the fleet SSH mesh from lambda.nice -n 19and one model at a time. Host load average was 100–130 from the CI pool, so no timings are reported.apr 0.69.0 (52f43da71), i.e. origin/main before batch-1. sha256fbc1165a13fd2967…, built on lambda (generic x86_64, glibc 2.35 on both hosts) and HEAD-checked withscripts/apr_bin.sh.bd258782, 2Baaf42c8b, 4B00fe7986, 9B03b74727.rungolden (greedy: 2+2, capital of France, spider legs)rungreedy determinism (×2, token-identical)runseeded-sampled determinism (T 0.8, top-k 40, same seed ×2)serve(/v1/chat/completions)Notes on each column
Sampled determinism. Each row carries two controls, and both held on all four models:
A first pass without
--top-kwas void:apr rundefaults to--top-k 1, so--temperaturealone decodes greedily (filed separately by the cop). Those rows were discarded, not reported.0.8B golden is an apr defect, not the model. The pinned llama.cpp
d1d3c3396, found throughscripts/llama_bin.sh, reproduces the wrong answers only with apr's scaffold:<think>\n</think>\n)<think>\n\n</think>\n\n)The source is
chat_template_qwen3_nothink.rs:44, andchat-template-v1.yamlasserts the same wrong literal. Tracked as #3755. The larger models tolerate the malformed scaffold.Serve. Every Qwen3.5 size exits before binding its port (rc 8):
<unk>is fixed in batch-1 #3707; the hybrid route is #3571. So rmedia's plannedservesurface does not exist for Qwen3.5 at 52f43da.Re-measure. This column should be re-run on the 0.69.1 release commit, once batch-1, #3571, #3720 and #3755 are in. The binary, the harness (
~/a8-matrix/intel-matrix.sh,intel-sampled.sh) and all four models are staged on intel for that.noahgift commentedon Sep 23, 2026
Cop: design review run in Claude plan mode (read-only, headless
claude -p --permission-mode plan), atrelease/0.69.1-batch-2@1fee8ef98. The cop re-read the citations for #1, #2, #3, #5, #6 and #8 against the tree, and all six match: the DEFER branch never sets rc;apr_sha_driftis referenced only in the yaml;qa_rc not in (0, None)andsha_ok is False; 3948 is not an ancestor of HEAD; COMPARATORS has no vllm; the whitelist admits 23.Status of #7: vLLM (#3952) is installable on BOTH lambda and gx10. The driver
scripts/crux_engine_vllm.shis in progress, stacked on #3781. Caveat: vLLM 0.30 cannot load a local GGUF, so vLLM rows compare against the bf16 source (repo+revision) and must say so.Design review findings (Plan mode, 2026-09-23)
*A3B*is still deferred, so both qwen3moe files pass the gate unproven. This breaks "no defer". It also contradicts our own verdict lattice, which maps DEFER to Unknown(NotRun), i.e. exit 2.check_model_ladder.sh:324-327(the DEFER branch never sets rc) ·model-capability-ladder-v1.yaml:155·ontology/verdict.rs:201PMAT-3714-r2-on-v2); or (b) take the files out of the inventory and report #3817's refusal-by-name as an explicit RED cell.apr_sha_driftkey in the contract is read by no code. It excuses cut28c24207a, but HEAD has 4 later commits, including the BF16 kernel and a golden ON-leg change, which the declaration itself says it does not cover.check_model_ladder.sh:265·git grep apr_sha_driftmatches only the yaml ·release_inputs.rs:65(--receipts-commitlets the caller choose the sha)apr_shais not the cut commit. Delete the prose-only key. Re-measure both hosts on the release commit.qa_rcpasses, a missingsha_okpasses, and a skippedcapability_matchpasses on a CPU-only claim.check_model_ladder.sh:120(not in (0, None)),:340(is False),:89check_model_ladder.sh:145-163·model_ladder.sh:672,723runuses, plus a planted wrong answer that must go RED.5103e07deis only onorigin/PMAT-3948-empty-think-block;merge-baseconfirms it is not in HEADcrux_inference_judge.py:326-327, 655-661, 766;parse_apr(:206) never strips<think>crux_inference_judge.py:56listsllama.cpp, ollama, hf, llamafiledtype.rs:50-60,73· #3947 · #3950 step 3 · #3953 step 3 ·e64fc7496dequantizeon every tensor of the real file, per type, with a planted-fault control, plus llama.cpp CRUX on the model. Settle IQ4_XS before porting it into aprender-core.dtype.rs:39,73·e64fc7496message line 70 · #3953 table · #3947 censusdogfood.sh:613Recommended sequencing
Needs an OPERATOR ruling (these re-open earlier operator rulings, so a quorum cannot decide them): finding 1, A3B deferral vs "no defer"; finding 11, coverage deferral #3839. Everything else goes to workers or quorum.
noahgift commentedon Sep 23, 2026
OPERATOR RULING, 2026-09-23 (verbatim): "1. fold in MoE 2. fold in 88% coverage"
This supersedes the 2026-09-22 MoE deferral and the earlier coverage deferral. Both now block 0.69.1:
PMAT-3714-r2-on-v2@6f6fa9a83(parity 92/92 at cosine 1.0 on lambda and gx10). The*A3B*key comes out ofinventory.deferredin the same fold. The ladder cells (run, chat, serve on CUDA, on both hosts) must be measured on the release commit. Parity alone does not prove apr run --gpu on qwen3moe Q4_K_M silently falls back to CPU (rc 14) instead of refusing by name — 0.69.1 must DECLARE it does not support qwen3moe on CUDA #3817 (apr run --gpufalling back to CPU) fixed.--skip) is in scope, because that fix is how the number becomes real.[-]EPIC: release train 0.69.1 — every Q4_K model works on CUDA on gx10 and lambda (supersedes 0.69.0 on crates.io)[/-][+]EPIC: release train 0.69.1 — every capability ORACLE-PROVEN (CRUX cells) on gx10 and lambda; no defer[/+]noahgift commentedon Sep 23, 2026
OPERATOR RULING, 2026-09-23 (verbatim): "c". The question was option (c): Qwen3.5-35B-A3B-UD-IQ4_XS.gguf (general.architecture
qwen35moe, a hybrid SSM MoE, on lambda) stays in the report as an explicit RED cell, "no CUDA path, 0.70.0", that does not block 0.69.1.F10 (for #3957): the
RED-UNSUPPORTEDverdict. It is the sibling of F9's RED-MODEL, with the same anti-laundering rule:RED-UNSUPPORTED, is never green, and is never counted as passing. It does not fail the release gate only if that same sweep observes apr REFUSING BY NAME:apr run --gpuexits with the named refusal fromcapability::no_cuda_forward_reason(qwen35moe arm), with no fallback to CPU and no output. A refusal that is not observed on the sweep, or a run that produces output instead, is plain RED.RED-UNSUPPORTED+ the observed refusal text, so the capability report says plainly that apr does not run this model on CUDA.The 0.70.0 work (a qwen35moe CUDA forward) gets its own ticket.
noahgift commentedon Sep 23, 2026
OPERATOR RULING, 2026-09-23 (verbatim): "B, I'll open 2 more workers".
Plan B: code freeze + one test run. Target release ~18:00–19:00 CEST.
make publish.noahgift commentedon Sep 23, 2026
FREEZE PLAN (cop, 14:10 CEST). At 15:00: the freeze sha is announced, and every non-sweep GPU job on gx10 and lambda stops. The final sweep runbook is /mnt/nvme-raid0/apr-release/final_sweep.sh (sha256 53962fed, owner aprender-3a):
check_model_ladder.sh --receipts … --cut-commit <freeze> --crux evidence/crux/0.69.1/<freeze>/.The ladder and CRUX never share a card in one phase (the serve /health stall would read the lock queue as a stall). Operator decision pending: CRUX scope, full (~10–15 h) vs known-answer only (~1.5–2 h).
noahgift commentedon Sep 23, 2026
❄️ FREEZE:
release/0.69.1-batch-2@c619dddd40367b06d17cec234301e412f6705457(15:30 CEST)Verified on the merged tree before the freeze: fmt clean; ladder self-test pass; CRUX judge 151/0; check_guards_are_wired pass; root
cargo check --release --features cudarc 0; the raw-route and stream-UTF-8 must-REDs pass.From now on, nothing lands except fixes for REDs that the final sweep itself finds. All non-sweep GPU work on lambda and gx10 stops.
Final sweep (runbook /mnt/nvme-raid0/apr-release/final_sweep.sh):
Pending operator decisions, which affect only how the judge reads the CRUX cells: (1) CRUX coverage, every held model vs the 3 certified models with ladder coverage for all; (2) CRUX depth, full admitted vs known-answer only (default: known-answer only).
noahgift commentedon Sep 23, 2026
❄️ FREEZE v2:
release/0.69.1-batch-2@4f81bdea7bdc17da7a4b22823ef000671b78a435(~17:00 CEST)The first freeze (c619ddd) sweep found real REDs; this freeze carries their fixes, and nothing else:
Verified on this tree: fmt, ladder self-test, ladder write-errors, guards_are_wired, CRUX judge 153/0, serve/code, root
cargo check --release --features cuda, and the #3723 tests. Lambda CRUX rehearsal on c619ddd: 124/140 GREEN, and every RED was a finding fixed here.Re-sweep: the same runbook with REQUIRE_ANCESTORS="186e56932 5c18bb1" and CRUX_DIR=evidence/crux/0.69.1/4f81bdea7/. Publish ETA ~20:30 CEST. #4022 is NOT included: it's pending the operator's direct word.
noahgift commentedon Sep 23, 2026
0.69.1 operator rulings (2026-09-23 ~15:45Z), recorded by the release cop aprender-cf (6a handed off)
The operator, asked 4 release questions with recommendations, answered verbatim: "recommended for all questions".
git diff --name-only 4f81bdea7 Xtouches ONLYcrates/aprender-mcp/**,crates/apr-cli/Cargo.toml,Cargo.lock(no dependency name/version changes) anddocs/**;fix(ladder): crux-smoke keys its thinking modes off the certification…
noahgift commentedon Sep 24, 2026
triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): closed — train epic: v0.69.1 tagged d8a6df5 and published; unmet done_when rows are carried by the CRUX/ladder rows in 0.70 (#3739 family, #4045). Open PR / live branch: PR#3736. Reopen if a done_when row is found unmet and is not carried elsewhere.