Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
bf60d83
fix(cells): quorum-2 findings — refuse vacuous --only and failed rung…
noahgift Sep 28, 2026
c89e944
fix(cells): the producer deadlocked on its own apr serve GPU lock (#3…
noahgift Sep 28, 2026
82ee483
feat(cells): de-claim Qwen3 on GB10 for 0.70 -- only those cells leav…
noahgift Sep 28, 2026
12841fd
feat(serve): warn [KNOWN-ISSUE #4590] for Qwen3 on sm_12x; release-no…
noahgift Sep 28, 2026
afd25aa
test(cells): rebuild green-cells-declaimed from the #4016 green-cells…
noahgift Sep 28, 2026
41d88f0
feat(release): wire release-readiness-v1 at T-1 and T-4, report-only …
noahgift Sep 25, 2026
9091cbb
fix(release): report mode records a could-not-judge and continues (#3…
noahgift Sep 25, 2026
f6cc936
test(release): report-mode rows for every could-not-judge branch (#3715)
noahgift Sep 25, 2026
0875b36
fix(#3715): de-claim Qwen3 on gx10 through DATA, not a pv change (#4590)
noahgift Sep 28, 2026
c017020
fix(#3715): release-readiness ENFORCES at T-1/T-4; models_t1 measures…
noahgift Sep 28, 2026
83fb0de
fix(#3715): no tag without "#3715 ENFORCE PASS" for this version+commit
noahgift Sep 28, 2026
cd6a9b0
fix(#3715): pass the candidate surface to pv -- without it no cell is…
noahgift Sep 28, 2026
398aef4
fix(release-readiness): the prompt certification is not a crux receip…
noahgift Sep 28, 2026
35ddb8c
fix(release): models_t1 readiness rows ran without the release apr (#…
noahgift Sep 28, 2026
71d8ced
fix(#3715 R10): delete the report-only arm -- enforce is the only mod…
noahgift Sep 28, 2026
69a0334
G2 (#4590): withdraw every GB10 dense claim through data — qwen2@gx10…
noahgift Sep 28, 2026
60a61bc
feat(ladder): a de-claimed (host, arch) the ladder still claims is RE…
noahgift Sep 28, 2026
5b12603
docs(#3715): release-note warning — qwen3moe on GB10 is not claimed f…
noahgift Sep 28, 2026
e59329f
fix(D1): the qwen3moe de-claim said lambda does not hold qwen3moe — i…
noahgift Sep 28, 2026
2205cfd
docs(D1): CHANGELOG said no other host holds qwen3moe — lambda holds …
noahgift Sep 28, 2026
976013b
merge split/ont10-rest-0.70 into #4603 — conflicts resolved to this b…
noahgift Sep 29, 2026
616fd14
merge origin/main into #4603 (roadmap.yaml: kept PMAT-4510 + PMAT-459…
noahgift Sep 29, 2026
9efceaa
merge split/ont10-rest-0.70 @ 9a29a7d19f into #4603 (stack base advan…
noahgift Sep 29, 2026
33b0325
docs(#3715 B1): receipt §8 describes THIS diff; README/CHANGELOG say …
noahgift Sep 29, 2026
dd89791
evidence(#3715 B1): Q0 quorum AGREED at 33b032506a (opus-5-5, haiku-4…
noahgift Sep 29, 2026
0b53d1f
merge split/ont10-rest-0.70 @ 2953777888 into #4603 (#4588 reverts)
noahgift Sep 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Known issues
- **Qwen3 MoE (Qwen3-30B-A3B, Qwen3-Coder-30B-A3B) on GB10 is not claimed for 0.70 (#3715); planned for 0.70.1.**
On GB10 (sm_121) every verb (run, chat, serve, code) times out at 600 s on the 20k-token rung. The 0.69.1
pre-load refusal is gone in 0.70.0, so these models now load and run, but too slowly to be claimed. The
release gate does not owe the qwen3moe x GB10 cells (`cells.declaimed` in
`contracts/model-capability-ladder-v1.yaml`), and the ladder gate is RED if any rung or long-rung
representative still claims them. Withdrawn, not waived: lambda holds both files and still owes every
qwen3moe cell.
- **GB10 dense models: prompt processing unbatched (11.4 tok/s); planned for 0.70.1 (#4590).** Widens the Qwen3 entry below to every dense model on GB10 (operator ruling 2026-09-28
16:15Z): the serial prefill is the sm_12x default for every non-hybrid arch, so dense Qwen2.5 on GB10
is de-claimed too (`cells.declaimed`: qwen2 and qwen3 on gx10). Output is correct; only prefill speed
is affected. Withdrawn, not waived: lambda still owes every dense cell, gx10 still owes every Qwen3.5 cell.
- **Qwen3 on GB10 (sm_121, and the sm_12x family) is correct but very slow, and is not claimed for
0.70 (#4590).** Dense Qwen3 on sm_12x prefills one token at a time, the sm_12x default, because the
generic batched prefill corrupts the KV cache there (FALSIFY-CPU-GPU-009). Measured on 0.70.0
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,15 @@ A non-`pass` cell names the gates the receipt records, which is not always the c

<!-- RELEASE_MATRIX_END -->

> **Known issue — Qwen3 MoE (30B-A3B) on GB10: not claimed for 0.70; planned for 0.70.1 (#3715).** Every verb
> times out at 600 s on the 20k-token rung on GB10 (sm_121), so qwen3moe on GB10 is withdrawn from this
> release's claims (`cells.declaimed`).

> **Known issue — GB10 dense models: prompt processing unbatched (11.4 tok/s); planned for 0.70.1 (#4590).** Output is correct; only prefill speed is affected. On GB10 (sm_121) every dense
> model (Qwen2.5, Qwen3) is withdrawn from this release's claims (`cells.declaimed` in
> [`contracts/model-capability-ladder-v1.yaml`](contracts/model-capability-ladder-v1.yaml)); Qwen3.5 on
> GB10 and every model on other hosts are still claimed and gated.

### Command surface

| Stage | Commands |
Expand Down
26 changes: 24 additions & 2 deletions contracts/model-capability-ladder-v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -236,11 +236,32 @@ ladder:
correct but 27-34x slower than Qwen3.5 on GB10 sm_121 (8k think-on: Qwen3-1.7B 967 s, Qwen3-8B
2163 s vs Qwen3.5-9B 63-80 s; serial prefill, the sm12x default). 0.70.1 P0 ships native sm_121
SASS and re-tests batched prefill on GB10. lambda still owes every Qwen3 cell.
# G2 (operator ruling 2026-09-28 16:15Z): R1 was Qwen3 only; this widens it to EVERY dense arch gx10 holds.
# Serial prefill is the default for every arch at cc>=120 (select_prefill_path), not only Qwen3.
- arch: qwen2
host: gx10
issue: 4590
until: "0.70.1"
why: >-
GB10 dense models: prompt processing unbatched (11.4 tok/s); fixed in 0.70.1 (#4590). Correct output,
prefill speed only: sm_12x prefills every dense arch one token at a time (select_prefill_path). Withdrawn,
not waived: lambda still owes every Qwen2 cell, and gx10 still owes every Qwen3.5 (hybrid) cell.
# #3715 H4 (cop 2026-09-28 20:26Z, scope = qwen3moe on gx10 only; STAGED, branch-only, operator rules 07:30Z):
# arch-level DATA is clean here -- no named rung and no long-rung representative is qwen3moe.
- arch: qwen3moe
host: gx10
issue: 3715
until: "0.70.1"
why: >-
Qwen3-30B-A3B (and Coder-30B-A3B) on GB10 sm_121: every verb hits the 600 s cell timeout at the
20k rung (0d H4 19:51Z, 12+12 cells). The 0.69.1 pre-load refusal is gone in 0.70.0-dev, so the
cells run and time out. Withdrawn, not waived: lambda holds both files and still owes every
qwen3moe cell; gx10 still owes every Qwen3.5 (hybrid) cell. Operator D1 APPROVED 2026-09-28 20:10Z.
long_rungs_for:
families: [qwen35] # every consumer's family: the full rung set, through declared
representatives: # one file per other arch the fleet holds
qwen2: qwen2.5-coder-7b-instruct-q4_k_m.gguf # lambda + gx10
qwen3: Qwen3-8B-Q4_K_M.gguf # lambda + gx10
qwen2: qwen2.5-coder-7b-instruct-q4_k_m.gguf # lambda (gx10 de-claimed, #4590)
qwen3: Qwen3-8B-Q4_K_M.gguf # lambda (gx10 de-claimed, #4590)
# qwen3moe: REMOVED for 0.69.1 (#3817). Not a fit question — this build has no CUDA
# forward for the architecture at all (#3714, lands 0.70.0), so `apr run --gpu` refuses
# pre-load by name and there is no GPU cell to owe. The CPU path is unchanged and works.
Expand All @@ -251,6 +272,7 @@ ladder:
arch: qwen2
gguf: qwen2.5-1.5b-instruct-q4_k_m.gguf
sha256: 6a1a2eb6d15622bf3c96857206351ba97e1af16c30d7a74ee38970e434e9407e
hosts: [lambda] # gx10 DE-CLAIMED for 0.70 (#4590, cells.declaimed, G2); restore to every host in 0.70.1
backends: [cpu, cuda]
required: true
note: "control rung — the architecture every other gate already exercises; if this is red the host is broken, not the model"
Expand Down
113 changes: 113 additions & 0 deletions crates/aprender-contracts-cli/tests/ont_release_readiness.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1157,3 +1157,116 @@ fn names(v: &serde_json::Value) -> Vec<&str> {
.map(|a| a.iter().filter_map(serde_json::Value::as_str).collect())
.unwrap_or_default()
}

// ── G2 (#4590, operator ruling 2026-09-28 16:15Z): a GB10 dense model is WITHDRAWN, not waived ───────────
//
// scripts/model_ladder.sh moves an arch `cells.declaimed` names for the host into the receipt's
// `declaimed_inventory[]`; pv reads `inventory[]` only, so the withdrawn (host, model) owes nothing. Withdrawn
// is not waived: every other host that holds the model still owes every cell of it. And a claim of it on GB10
// (the item back in gx10's `inventory[]`, unmeasured) is RED at the enforce-mode gate.

const DENSE_SHA: &str = "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd";

fn dense_item() -> Value {
serde_json::json!({
"file": "tiny-qwen2-dense.gguf", "sha256": DENSE_SHA, "bytes": 4, "arch": "qwen2",
"quant": "q4_k_m", "context_length": 10000, "owes_long_rungs": false, "thinking_modes": ["off"],
"chat_template_sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
})
}

fn dense_cells(t: &Path, host: &str) -> Vec<String> {
read(&t.join("cells.json"))["cells"]
.as_array()
.expect("cells")
.iter()
.filter(|c| c["host"] == host && c["model_sha256"] == DENSE_SHA)
.filter_map(|c| c["id"].as_str().map(str::to_string))
.collect()
}

/// The green base plus a dense model lambda HOLDS and MEASURES (inventory, kernel dispatch, tokenizer parity,
/// one passing row per derived cell) and gx10 holds WITHDRAWN (`declaimed_inventory[]`, as model_ladder.sh writes it).
fn withdrawn_on_gx10() -> tempfile::TempDir {
let t = green();
edit(&models(t.path(), "lambda"), |v| {
v["inventory"]
.as_array_mut()
.expect("inventory")
.push(dense_item())
});
edit(&kernels(t.path(), "lambda"), |v| {
let mut d = v["dispatch"][0].clone();
d["sha256"] = DENSE_SHA.into();
d["file"] = "tiny-qwen2-dense.gguf".into();
v["dispatch"].as_array_mut().expect("dispatch").push(d);
});
edit(&tokenizer(t.path()), |v| {
let mut r = v["rows"][0].clone();
r["sha256"] = DENSE_SHA.into();
r["file"] = "tiny-qwen2-dense.gguf".into();
r["family"] = "qwen2".into();
v["rows"].as_array_mut().expect("rows").push(r);
});
edit(&models(t.path(), "gx10"), |v| {
v["declaimed_inventory"] = Value::Array(vec![dense_item()])
});
synthesize(t.path());
t
}

#[test]
fn a_withdrawn_gb10_dense_model_owes_nothing_there_a_planted_claim_is_red_and_lambda_still_owes_it()
{
let t = withdrawn_on_gx10();
let r = gate(t.path(), &[]);
assert_eq!(
r.code,
0,
"withdrawn on gx10, measured on lambda: GREEN\n{:#}",
json_of(&r)["findings"]
);
assert!(
!dense_cells(t.path(), "lambda").is_empty(),
"withdrawn is not waived: lambda owes the dense model's cells"
);
assert!(
dense_cells(t.path(), "gx10").is_empty(),
"gx10 owes no cell of a withdrawn model"
);

// PLANTED CLAIM: gx10's receipt claims the dense model (back in inventory[]) with no row measured -> RED.
edit(&models(t.path(), "gx10"), |v| {
let item = v["declaimed_inventory"]
.as_array_mut()
.expect("declaimed_inventory")
.remove(0);
v["inventory"].as_array_mut().expect("inventory").push(item);
});
let red = gate(t.path(), &[]);
assert_eq!(
red.code,
1,
"a planted GB10 dense claim is RED\n{}",
show(&red)
);
let v = json_of(&red);
let msgs: Vec<&str> = v["findings"]
.as_array()
.expect("findings")
.iter()
.filter_map(|f| f["message"].as_str())
.collect();
assert!(
msgs.iter()
.any(|m| m.contains("/gx10") && m.contains("tiny-qwen2-dense")),
"the planted GB10 dense claim is what went red:\n{}",
msgs.iter().take(40).copied().collect::<Vec<_>>().join("\n")
);

// WITHDRAWN != WAIVED: lambda dropping one dense row is RED naming that cell.
let t = withdrawn_on_gx10();
let id = dense_cells(t.path(), "lambda").remove(0);
drop_row(t.path(), "lambda", &id);
assert_red_naming(&gate(t.path(), &[]), &named(&id));
}
31 changes: 31 additions & 0 deletions docs/audits/impl-PMAT-3715-receipt.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,3 +145,34 @@ Measured on this branch's head:
Known and not fixed here, because it is apr's gap and not the producer's: `apr code --output-format json` carries
no backend field (`crates/aprender-orchestrate/src/agent/code_envelope.rs` `envelope()`). So every `code` cell
is red and T-1 STOPs until that field lands. This is why the branch is on the FINAL path, not rc.1 (cop, 2026-09-28).

## 8. PR #4603, `0d/3715-b1-main` (base `split/ont10-rest-0.70` @ 9a29a7d19f): B1 enforces at T-1/T-4

Sections 4, 6 and 7 describe earlier branches. Section 7 in particular is the `0d/3715-cells-at-t1` diff, which
touched no Rust and no contract. **This** diff (`git diff origin/split/ont10-rest-0.70...HEAD`, 25 files) does touch
Rust (`crates/aprender-contracts-cli/tests/ont_release_readiness.rs`, a test) and one contract
(`contracts/model-capability-ladder-v1.yaml`, data only: `cells.declaimed`). It:

- deletes the report-only arm of `release_readiness.sh` (R10); `report` is refused as env value and as a committed default;
- makes `autopilot.sh` `cut_tag()` and `check_publish_preflight.sh` R8 refuse a `WARN R8` row and any rc 0 that lacks
`#3715 ENFORCE PASS version=V commit=MC` for this version and HEAD;
- adds the ladder RED for a de-claimed (host, arch) the ladder still claims (`check_model_ladder.sh`, planted-RED cases);
- withdraws the GB10 dense claims through data (`cells.declaimed`: qwen3moe, qwen2 on gx10), with the README and CHANGELOG warning.

**De-claim is not a waiver.** Ruling, operator 2026-09-28 16:15Z (via the cop, `cop-inbox/handoff/ruling-gx10-1615.md`):
"every GB10 dense claim is withdrawn via DATA (gx10 inventory), with a warning in the notes and README, and the
enforce-mode gate turns RED on any GB10-dense claim ('withdrawn ≠ waived')". The gate has no waiver input; a claim
that returns needs receipts at the 0.70.1 pin (G5). The control rung `qwen2.5-1.5b` is `hosts: [lambda]` for the same
reason: it is a dense GB10 claim on the withdrawn set. Wording in README and CHANGELOG is "planned for 0.70.1", not "fixed".

Measured at head `9efceaa024` (intel for cargo; scripts on lambda, load 8):

| command | result |
|---------|--------|
| `bash scripts/release/release_readiness.sh --selftest` | 30/30 rows, incl. `mode_has_no_report_arm` |
| `bash scripts/check_release_models_t1.sh --self-test` | PASS 41 rows |
| `bash scripts/check_model_ladder.sh --self-test` | 153 cases, 0 bad |
| `bash scripts/check_publish_preflight.sh --selftest` | 57/57 rows |
| `cargo test -p aprender-contracts-cli --test ont_release_readiness` | see the PR thread (re-run at the final head) |

Blocked on the gate pv: 0.69.2 has no `--surface`; the chain is pv 0.69.4, pin, B1, rc.1.
Loading
Loading