Skip to content

batch(0.69.5): EPIC #4249 patch train + #4313 f16 prefill — 0.69.5 RC - #4318

Closed
noahgift wants to merge 77 commits into
mainfrom
batch/0.69.4-epic-4249
Closed

noahgift wants to merge 77 commits into
mainfrom
batch/0.69.4-epic-4249

Conversation

@noahgift

@noahgift noahgift commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

The 0.69.4 epic batch (EPIC #4249), following the operator's batching rule (verbatim, 2026-09-24): "individual PR is too costly for release canidates and even epics, we should actively batch like PRs together". Every 0.69.4 ticket lands here as one commit (Refs/Closes #N per commit), and the batch gets one quorum and one CI run. Do not open a separate PR for a 0.69.4 ticket: push a commit to batch/0.69.4-epic-4249, with no force-push and one commit per ticket. It is a draft until the batch is ready.

Commits

Commit Ticket What
0590d50 #4290 The F2 receipt key was {model_sha256, apr_version, device}, and every build sharing a version string shared receipts (rc.1, rc.2, final, dev builds). The key gains build_id = exe-sha256:<hex> of the running executable. An exe that cannot be hashed never skips and never writes a receipt. Schema 1→2; a schema-1 file parses and re-validates as a schema mismatch.
64b20f8 #4316 Qwen3.5 CUDA forward_single allocated a fresh device residual per token (GpuBuffer::from_host). The model now owns one hidden_dim residual, allocated at construction and refilled with copy_from_host_async on the executor's execution stream. It is lent to forward_from_residual and restored on every path, refusals included.

#4290 evidence (lambda, local)

  • cargo test -p aprender-serve --lib f2_receipt passes 21/21. The 7 new tests include the case table: same version with a different build → BuildIdMismatch; two +no-git builds of one version → BuildIdMismatch; the identical build → Skip; an unhashable exe → BuildUnidentified.
  • Planted mutant. With the build_id comparison disabled (if false && …), which reverts to a version-only key: 2 FAILED, exactly the two F2 receipt key uses CARGO_PKG_VERSION, so rc.1/rc.2/final (and every dev build) of one version share receipts: a forward-path change skips the CPU reference check #4290 falsifiers. The file was restored by a trap; porcelain was clean afterwards.
  • cargo clippy -p aprender-serve --lib --features cuda -- -D warnings is clean. The receipted caller in forward_qwen35.rs is cuda-gated.
  • Why not the git sha: a dirty dev build shares its HEAD's sha, and the release assets report +no-git. Hashing the binary costs one streamed pass over the exe, once per process, next to the existing full sha256 of a 2.5 GB model.

#4316 evidence (lambda RTX 4090, Qwen3.5-0.8B-Q4_K_M + 4B)

  • qwen35_cuda_ suite: 17 passed, 0 failed, 0 skipped (118.9 s), serialized under /tmp/apr-gpu.lock. This includes qwen35_cuda_forward_single_matches_cpu_logits_end_to_end unchanged and the 4B argmax end-to-end.
  • The new qwen35_cuda_residual_pointer_is_stable_across_tokens_and_refusals checks: the device pointer is identical across 3 tokens (GPU argmax == CPU on each), after an OOV refusal, and after a position-limit refusal.
  • cargo clippy -p aprender-serve --lib --features cuda -- -D warnings is clean.

Closes #4316
Closes #4290
Refs #4249
keep-open: #4249 is the 0.69.4 epic and stays open until its patch train ships.
ont-delta: none — a runtime cache key change with no contract or ontology surface

🤖 Generated with Claude Code

0.69.5 fold (cop / operator 2026-09-25: "yes, make it .69.5")

…on string (#4290)

f2_receipt::decide skipped the CPU reference forward whenever a stored
receipt's {model_sha256, apr_version, device} matched. apr_version is
CARGO_PKG_VERSION, so v0.69.3-rc.1, rc.2 (which rewrites the generation
loop, #4263), the final tag and every dev build on release/0.69.3 all
shared receipts: rc.2 would skip F2 on a receipt rc.1 wrote.

The key gains build_id = "exe-sha256:<hex>" of the running executable
(streamed, hashed once per process). Not the git sha: a dirty dev build
shares its HEAD's sha and the release assets report +no-git. An executable
that cannot be hashed gets an empty id, which never skips and is never
written. Schema 1 -> 2; build_id is serde(default) so a schema-1 file
parses and re-validates as a schema mismatch.

Case table in f2_receipt_tests.rs: same version + different build ->
BuildIdMismatch; two +no-git builds of one version -> BuildIdMismatch;
identical build -> Skip; unhashable exe -> BuildUnidentified.

Closes #4290

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of allocating per token

forward_single called GpuBuffer::from_host for every token: one device
allocation plus a blocking copy on the hot decode path. The model now owns
a hidden_dim residual buffer, allocated once at construction, and refills it
with copy_from_host_async on the executor's execution stream. The buffer is
taken, lent to forward_from_residual, and restored on every path, refusals
included.

Test: qwen35_cuda_residual_pointer_is_stable_across_tokens_and_refusals
checks that the device pointer stays the same across 3 tokens, an OOV refusal
and a position-limit refusal, with GPU argmax == CPU argmax on each token.

Closes #4316
Refs #4249

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift noahgift changed the title batch(0.69.4): EPIC #4249 patch train — one commit per ticket (first: #4290 F2 receipt build key) batch(0.69.4): EPIC #4249 patch train — one commit per ticket (#4290 F2 receipt build key, #4316 CUDA residual reuse) Sep 24, 2026
noahgift and others added 3 commits September 24, 2026 18:23
….4 port of rc.2 8be3565)

These have no caller anywhere in the workspace on this tree either:

- apr-cli ChatSession::generate_gguf (#[allow(dead_code)], ~90 lines): the
  bespoke OwnedQuantizedModel generate_with_cache loop.
- apr-cli generate_and_print_fallback (no-inference cfg): unreachable!() stub
  with no caller.
- aprender-serve gguf/inference/cached/sequence.rs: never included by any
  module, byte-identical to single_true_batched_owned.rs.

No behaviour change. Verified: fmt, clippy -D warnings (apr-cli +
aprender-serve lib), cargo test -p apr-cli --lib chat 263 pass.

Refs #4266

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sts that only fed them

generate_with_smallvec (forward/batched.rs) and
generate_with_adaptive_attention (cached/single_true_batched_owned.rs) had no
caller outside their own tests on this tree: no src caller, no example, no
contract/doc/script reference. Deleted with the 6 tests that existed only to
call them (batch_tests.rs x4, imp_115a 117f, imp_121a 121c).

No behaviour change. Verified: fmt, clippy -p aprender-serve --lib -D
warnings, cargo test -p aprender-serve --lib imp_115a imp_121a batch_tests 81
pass. (clippy --tests is red on this tree in untouched files —
falsification_crux_c_34.rs, cancel_scope_2376.rs — none in the edited files.)

Refs #4266

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0.69.3 G1 showed 9B SKIPPED. These are the refs owed before 0.69.4
acceptance: apr 0.69.3-rc.1 (7ff50ec) temp-0 ids for the pinned 9B GGUF
(sha256 03b74727…) on the G1 prompts, every cell ran=gpu fell_back=false.

- rc == base 0.69.1 id-for-id at 850 and 4k. Base at 32k was stopped
  (per-token prefill, ~40 min GPU hold, rc.2 acceptance queued behind it).
- llama.cpp on the GGUF's own template (thinking off, same prompt-token
  counts 1006/4133/31848): identical at 32k; diverges at 850 pos 13 and 4k
  pos 2, both measured near-ties (top-3 within 0.08 nats; 0.117 nats, apr's
  pick is llama's #2) - margins.json.

Refs #4261, #4249

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=4318 head=b3ff63c6c2dd1d13d07b004cc94658a31f9c3681 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

noahgift and others added 20 commits September 24, 2026 21:54
…ratchet); name run.sh's wall-clock DET002 (#4290, #4261)

guard-tree RED on #4318: f2_receipt.rs::decide cyclomatic 9 / cognitive 26, new vs the comparand.
guard-cargo RED: bashrs DET002 on the #4261 harness's `date +%s`, whose wall time IS the measurement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ops at EOS/<|im_end|> and honours top_p (#4265)

#4278 (d65a915) landed on release/0.69.3 only. main, batch/0.70.0 and
batch/0.69.4-epic-4249 still built the APR CPU GenerateConfig with
stop_tokens: vec![] and top_p: 0.9 (handler_apr_cpu_completion.rs:139,213).
Generation therefore ran to max_tokens and ignored the request's top_p.

This cherry-pick resolves three conflicts against the newer base:
- apr_cpu_generate_config drops `seed`. The #3760 field is release-only;
  GenerateConfig on this base has no seed and already carries `cancel`.
- build_cpu_sse_stream keeps this base's 3-arg signature. The 4th arg is
  release-only. top_p is still threaded into spawn_cpu_streaming_task.
- roadmap.yaml is re-aggregated from the carried PMAT-4269 fragment.

Evidence: the 8 tests_apr_cpu_gen_config_4265 tests pass. A mutant that makes
apr_cpu_stop_tokens return vec![] fails 3 of them. Clippy -D warnings is clean
on apr-cli --lib.

(cherry picked from commit d65a915)
Refs #4265

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ur top_p (#4334)

chat.rs and simple.rs built their own GenerateConfig with stop_tokens: vec![]
and top_p: 0.9, so every SafeTensors reply ran to max_tokens and ignored the
request's top_p (the #4265 defect on a sibling path). Both now go through
apr_cpu_generate_config with the tokenizer's EOS + chat-turn-end ids
(tokenizer_info_stop_tokens, now shared with the APR CPU path), and strip the
stop id the loop ended on from the reply.

Refs #4265 #4249

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ealth handler; the route is :170 (0.69.4 dogfood G2.1)

This branch changes serve/safetensors.rs, which the ledger cites, so G2.1 required the row to be re-audited. On inspection the row was already stale on main: :164 is inside /health, and /tensors is at :170 on main and here. The single-server /tensors row is corrected; check_dogfood_coverage: run PASS, self-test PASS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…terals removed (#4085)

CI run 36057175909 on batch/0.69.4-epic-4249 failed guard-cargo on
handler_gpu_completion.rs:364-365 ("100+ tok/s CPU, 500+ tok/s GPU",
"800+ tok/s (2.8x Ollama)"). These are not new claims: #4265's forward-port
pushed the lines down 2 from their baselined :362/:363. Per #4085 they are
removed, not re-keyed; the comment points at docs/BEATS.md, where the
measured Ollama comparison lives.

Baselines only shrink: claim_literal 4 removed, perf_claim_citation 7
removed (stale entries, --update with 0 new hits; diffs are removals only).
Both guards rc 0, check_baseline_ratchets PASS, cargo fmt -p apr-cli clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ME/models}, not a user home

check_hardcoded_paths.sh (guard-tree, CI 36066752831) flagged 3 shipped
/home/noah/models paths added by 9e72256. Same file resolves for the
original runner; APR_MODELS overrides on other hosts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#4295)

apr_cpu_stop_tokens walked the whole tokenizer.json vocab on every request. The set is now derived when AprServerState is built (derive_apr_cpu_stop_tokens) and held as Arc<[u32]>; a request reads it. a_request_reads_the_stop_set_derived_at_load changes the tokenizer after the state is built and requires the load-time set.

Ported onto batch/0.69.4-epic-4249 by aprender-x4233: the one conflict is the vocab scan, which #4334 factored into tokenizer_info_stop_tokens there. derive_apr_cpu_stop_tokens now calls it on the load-time tokenizer.

Pmat-Ticket: PMAT-4295
Refs #4295

(cherry picked from commit deb9ff4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4-epic-4249 — main wins on 0.69.3-line code

Resolutions (the batch-only 0.69.4 commits survive on main's structure):
- serve SafeTensors (chat.rs, simple.rs): main's #4269 Session path (st_cpu_generate)
  now takes #4334's gen_config — tokenizer stop set + request top_p — instead of
  hardcoding top_p 0.9 / stop [0].
- handlers.rs: main's apr_cpu_generate_config (QuantizedGenerateConfig, token 0 in
  the stop set, pub(crate)); batch's #4295 load-time stop set and #4334 helpers kept.
- qwen35 cuda: main's #3596 bounded state + prefill_attention, batch's #4316
  persistent residual; both tests kept.
- f2_receipt: both sides bumped schema to 2 for different reasons (#3596, #4290);
  now 3, so neither side's schema-2 receipt skips validation.
- roadmap.yaml: regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…last turn's checkpoint

Scripted forward: an identical prompt, a re-rendered turn 2 and turn 3 must
resume at the checkpoint (3 RED); a dropped state, a lost checkpoint, a
prompt diverging before it and a score must not reuse it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 664d1cf)
(cherry picked from commit fd2112b)
…nt-state checkpoint (#4214, #4274)

Qwen3.5's Gated DeltaNet state cannot be rolled back, so a prompt that did not
strictly extend the state (the same prompt again, #4214; serve's re-rendered
chat history, #4274) prefilled from position 0.

The prompt is now prefilled in two spans, and the forward keeps a copy of the
state between them, before the prompt's last <|im_start|> (else before its
last token). A later prompt that repeats the copied tokens restores the
conv/ssm state (CPU clone; device D2D on the compute stream), truncates KV to
the copied rows and prefills from there. The copy is dropped whenever the
positions under it are rewritten: a reallocation, a GPU fallback, a forward
from below it (score, a diverging prompt) or a failed forward.

CPU: 21/21 session tests, incl. the 0.8B model: identical repeat resumes at the
generation header and a re-rendered turn 2 at turn 1's, token-identical to the
one-shot at temp 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 629ef71)
(cherry picked from commit 7d1b3af)
…eration header

The checkpoint is taken at the last <|im_start|>, so turn 1 is two batched
prefills and turn 2's suffix two more. Measured on lambda RTX 4090 under gpu-q:
the counts were 2 and 4, and the tokens are still identical to the one-token path
and to the one-shot decode.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 082586f)
(cherry picked from commit 0010866)
…ybrid (#4254)

On the v0.69.1 CPU asset serving Qwen3.5-4B, /health said compute_mode "cpu"
while /v1/effective-config said compute_class "unknown", backend_loaded [],
build_features [..., "gpu"], quantization null, parameter_count null and
build_commit null. Each field now reads what the process actually has:

- backend_loaded / compute_class count the resident Qwen3.5 session (cuda when
  it is on the GPU, cpu otherwise), the same predicate /health reads.
- build_features drops "gpu"/"cuda" when the CLI that launched the server was
  built without wgpu/cuda: realizar compiles wgpu by default, but apr serve
  cannot dispatch to it, and --list-devices already said so.
- the hybrid's model source carries a quantization (dominant_quantization's
  rule, read off the GGUF tensor table: modal qtype over attn_output/FFN
  projections) and a parameter count (sum of tensor elements). The dense
  CPU and CUDA paths carry the parameter count too.
- build.rs reads HEAD from .git directly (worktrees, packed-refs) when
  `git rev-parse` is unavailable, before falling back to .git-sha.

Tests: dispatchable_build_features rows; a real-model Qwen3.5-0.8B row that
both routes report cpu on one state (mutating the session read out of
backend_loaded reproduces the #4254 body exactly and fails it); the qwen35
serve source carries Q4_K and a 0.5-1.5B parameter count.

Refs #4254, #4252 (PMAT-4252)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 144c5e9fb372bbd5a272dbf9b39cd4bebd046c77)
(cherry picked from commit 1203d03)
…; the device argmax drops a checkpoint it writes under

Reconciles cf20f6f52 (#4274, explicit `keep` from the chat backend) onto the
#4214 automatic checkpoint (7d1b3af), which already covers #4274: it splits
every prompt before its last <|im_start|>, so a re-rendered history resumes
without the backend predicting a boundary. Keeping both would have left two
checkpoint systems, so cf20f6f52's code is not taken; its real-template test is.

- qwen35_chat_backend_tests: second_chat_turn_resumes_from_the_first_turns_checkpoint_4274
  (0.8B, thinking on and off, official template): turn 2 does not extend turn 1,
  reuses > 0 tokens, and answers what a fresh full re-prefill answers.
- session: advance_and_choose's forward_greedy branch (release-line #4263) now
  drops the checkpoint when it prefills from under it, like step() does.
  Falsifier a_device_argmax_from_below_the_checkpoint_drops_it; mutant
  (condition -> false) turns it RED.
- runtime: truncate carries the same used_underscore_binding allow as its
  sibling (clippy -D warnings).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9eb442e)
…pe, and says SCOPED (#4086)

At 0.69.1 the release gate judged the operator's recorded emergency scope
(`--scope crux-smoke`, satisfied on both hosts) while the dogfood ran the same
check_model_ladder.sh with no arguments -- the full ladder -- and went RED: two
gates, one release, opposite verdicts, shipped as known-red #4086.

check_model_ladder.sh, called with no --scope, now applies the scope RECORDED for
the release being judged (crux_smoke_scope.recorded_scope over
ladder.emergency_scopes). It is never inferred: no entry for this version = the
full ladder; two entries or a nameless one = RED, judging neither. It is SAID:
a `SCOPED: <name> -- ...` line precedes the scope's own verdict, and
`--scope none` forces the full ladder on a scoped release (the nightly).
MODEL_LADDER_CRUX_DIR is the receipts seam for a caller that cannot pass --crux.

dogfood.sh's classify_declared carries a gate's `SCOPED:` line into its row
note, green or red (the scope NAME only: mark() keeps 200 chars and a red row's
reason must survive), so a scoped pass never reads as a bare `exit=0`.

Must-RED: with NO recorded scope, CRUX smoke receipts that would satisfy a scope
do not turn the gate green -- the full ladder is judged and is RED (e2e row).
Tables: select (6 rows, 4 module mutants), e2e dispatch (7 rows, 4 script
mutants), dogfood note (4 rows, 3 mutants); all mutants killed. Real data: the
no-argument call at cut d8a6df5 on the published 0.69.1 receipts is SCOPED and
green; at this branch's HEAD it is RED (receipts are not from this binary).
Contract: FALSIFY-MCL-017.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5122280)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 708c6bf)
… (lane 1 .err is agy idle narration)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit b1a765d)
…'s backend flag

#3797 done_when names both mutants; the F6 table had neither. Each is killed by
the S1 row 'serve green' (m1 merges nonstream+stream into one cell; m2 leaves
backend_verified None). Baseline CRUX_NO_MUTANTS=1: 129 ok, 0 broke.

Refs #3797

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 6ab17f9)
noahgift and others added 2 commits September 25, 2026 18:25
… the CPU (#4443)

90f8ec7 doubles MIN_CAPACITY until a turn fits, clamped to the GGUF
context (262144 for Qwen3.5), so a 32782-position turn asked for 65536.
On Qwen3.5-27B Q4_K_M / RTX 4090 that would not allocate, and the whole
session fell back to the CPU: p32k ran past a 900 s timeout, where
llamafile 0.10.6 takes 15.6 s on the same card (CRUX #4223 cells).

A failed oversized allocation now retries at exactly the turn and says so;
only a turn that does not fit at all reaches the CPU fallback.

Refs #4443

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y is RED (#4443)

Refs #4443

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

v0.69.5-rc.1 dogfood verdict: independent install verification (#4186; operator: release candidate ONLY, no promotion): aprender-36, 2026-09-25T16:40Z

Verdict: GREEN (0 of 5 hosts RED after round 1). Tag v0.69.5-rc.1 = 5660b08.

Checked on each host, read-only: the apr on PATH prints 0.69.5 and is bound to the tag. The binding is either the sha in --version or, when the asset prints +no-git (#4256), a sha256 that matches a sha-verified release asset's apr. Then one CRUX smoke: greedy "2+2" on qwen2.5-coder-1.5b-instruct-q4_k_m; the answer must be 4 and fell_back must be false. intel runs --backend wgpu.

host backend state PATH apr --version sha256 binding smoke why
lambda-labs cuda GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) 4bfdacd25c3c --version sha 5660b08 text='4' ran=gpu fell_back=False -
gx10 cuda GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) d39bb0b661f2 --version sha 5660b08 text='4' ran=gpu fell_back=False -
yoga cuda GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) 4bfdacd25c3c --version sha 5660b08 text='4' ran=gpu fell_back=False -
intel wgpu GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) a1de0a826fdb --version sha 5660b08 text='4' ran=gpu fell_back=False -
mini default GREEN /Users/noahgift/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) 3d036e1acda1 --version sha 5660b08 text='4' ran=cpu fell_back=False -

Asset apr binaries (sha256, asset, .sha256 check):

3d036e1acda1f9fc22306d2b9a4813d9238c0b1bea098e1d91c9896a4a1ca1a7	apr-v0.69.5-rc.1-aarch64-apple-darwin-cpu.tar.gz	ok
912576ed7ace8eb5fdb3e229529853f9fb43e33b11284e4a9e449fce7b041496	apr-v0.69.5-rc.1-aarch64-unknown-linux-gnu-cpu.tar.gz	ok
d39bb0b661f2ba41219a11456eb55c84c9601316c4cb5459d4094d836c6eddfd	apr-v0.69.5-rc.1-aarch64-unknown-linux-gnu-cuda.tar.gz	ok
cdbc566d5313cc00b10f8078d1a82be57f0c38d995d17a34567182a5d2d8439b	apr-v0.69.5-rc.1-x86_64-unknown-linux-gnu-cpu.tar.gz	ok
d7dc9fcd6960a3c3bbc2b722f51a3ea4a01dfa19b2552db60fefc7c7f5fba4bc	apr-v0.69.5-rc.1-x86_64-unknown-linux-gnu-cuda.tar.gz	ok
a1de0a826fdb4dcee690d268f203e844089f4a9a84542323df20f8bfedaf9dc9	apr-v0.69.5-rc.1-x86_64-unknown-linux-gnu-wgpu.tar.gz	ok

Note (not RED): lambda-labs and yoga run apr sha256 4bfdacd25c3c…, which matches NO release asset (x86_64-cuda asset = d7dc9fcd…). They are tag-bound by --version (5660b08) only, i.e. a local build from the tag commit, not the published asset. gx10/intel/mini run the byte-identical published asset.

Verifier: rc1_verify.sh; positive control vs v0.69.3 discriminated (intel RED on a non-tag build). Rules out: wrong commit on any host, fallback, wgpu→cpu on intel. Does not rule out: asset-vs-local-build divergence on lambda/yoga.

@noahgift

Copy link
Copy Markdown
Member Author

AMENDED 16:43Z: all 5 hosts now run sha-verified release ASSET bytes

Per cop: lambda-labs and yoga had a local tag build (sha 4bfdacd…, matching no asset). I installed the x86_64-unknown-linux-gnu-cuda asset on both. It passed sha256sum -c and landed with sha d7dc9fcd…; each old binary is kept at ~/.cargo/bin/apr.local-4bfdacd. Then I re-ran the full verifier. The asset runs fine on both hosts; no release defect.

Verdict: GREEN (0 of 5 hosts RED after round 1). Tag v0.69.5-rc.1 = 5660b08.

Checked on each host, read-only: the apr on PATH prints 0.69.5 and is bound to the tag. The binding is either the sha in --version or, when the asset prints +no-git (#4256), a sha256 that matches a sha-verified release asset's apr. Then one CRUX smoke: greedy "2+2" on qwen2.5-coder-1.5b-instruct-q4_k_m; the answer must be 4 and fell_back must be false. intel runs --backend wgpu.

host backend state PATH apr --version sha256 binding smoke why
lambda-labs cuda GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) d7dc9fcd6960 --version sha 5660b08 text='4' ran=gpu fell_back=False -
gx10 cuda GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) d39bb0b661f2 --version sha 5660b08 text='4' ran=gpu fell_back=False -
yoga cuda GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) d7dc9fcd6960 --version sha 5660b08 text='4' ran=gpu fell_back=False -
intel wgpu GREEN /home/noah/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) a1de0a826fdb --version sha 5660b08 text='4' ran=gpu fell_back=False -
mini default GREEN /Users/noahgift/.cargo/bin/apr apr 0.69.5-rc.1 (5660b08) 3d036e1acda1 --version sha 5660b08 text='4' ran=cpu fell_back=False -

Asset apr binaries (sha256, asset, .sha256 check):

3d036e1acda1f9fc22306d2b9a4813d9238c0b1bea098e1d91c9896a4a1ca1a7	apr-v0.69.5-rc.1-aarch64-apple-darwin-cpu.tar.gz	ok
912576ed7ace8eb5fdb3e229529853f9fb43e33b11284e4a9e449fce7b041496	apr-v0.69.5-rc.1-aarch64-unknown-linux-gnu-cpu.tar.gz	ok
d39bb0b661f2ba41219a11456eb55c84c9601316c4cb5459d4094d836c6eddfd	apr-v0.69.5-rc.1-aarch64-unknown-linux-gnu-cuda.tar.gz	ok
cdbc566d5313cc00b10f8078d1a82be57f0c38d995d17a34567182a5d2d8439b	apr-v0.69.5-rc.1-x86_64-unknown-linux-gnu-cpu.tar.gz	ok
d7dc9fcd6960a3c3bbc2b722f51a3ea4a01dfa19b2552db60fefc7c7f5fba4bc	apr-v0.69.5-rc.1-x86_64-unknown-linux-gnu-cuda.tar.gz	ok
a1de0a826fdb4dcee690d268f203e844089f4a9a84542323df20f8bfedaf9dc9	apr-v0.69.5-rc.1-x86_64-unknown-linux-gnu-wgpu.tar.gz	ok

Every host's PATH apr sha256 now equals a sha-verified asset: lambda/yoga d7dc9fcd (x86_64-cuda), gx10 d39bb0b6 (aarch64-cuda), intel a1de0a82 (x86_64-wgpu), mini 3d036e1a (aarch64-darwin-cpu). Rules out: wrong commit, non-asset bytes, fallback, wgpu→cpu on intel.

noahgift and others added 3 commits September 25, 2026 19:49
Quorum on #4318 (gemini lane): the tests exercised grown_capacity and
allocate_with_exact_retry directly, so reverting either call site in
ensure_capacity stayed green.

- cpu_a_turn_past_min_capacity_is_allocated_with_headroom: a real CPU
  session with min_capacity 16 allocates 32 for a 20-position turn,
  keeps it for 30, doubles to 64 for 33.
- gpu_headroom_that_will_not_allocate_keeps_the_turn_on_the_gpu (cuda):
  a test-only allocation ceiling makes the headroom fail; the turn stays
  on the GPU at exactly 20 positions with the #4443 notice. Control: a
  ceiling below the turn moves it to the CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g resolved

Round 1 (3c4e192..d1bb4d9): sonnet PASS, haiku PASS, gemini FAIL. ensure_capacity
had no test driving it. c13a233 (aprender-0d) adds a CPU test through reserve() that
goes RED under a call-site mutant. Round 2 (d1bb4d9..c13a233): sonnet-5, haiku-4-5
and gemini-3.1-pro-high all PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
version_ok stripped `-rc.N` from the tag and required the bare crate version
as a word, on the premise that "an rc tag's -rc.N is not in the crate
version". stamp_rc_version.sh has since made it so: v0.69.5-rc.1 prints
`apr 0.69.5-rc.1 (5660b08)`, which names the right tag AND the right commit,
and the check FAILed it. Accept the full `X.Y.Z-rc.N` or the bare `X.Y.Z`.

Case table +5 rows: a stamped rc passes; another rc's stamp, an rc stamp under
a final tag, +no-git and a wrong sha still fail. Reverting only the comparator
turns the stamped-rc row red. The real CI inputs: rc -> 0, and stable
(`apr 0.69.3 (v0.69.3+no-git)`) stays 1. That one is a true red on the
published v0.69.3 asset (#4110; the build.rs fix is on this PR) and clears
when a final built with its sha is published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Member Author

Delta quorum dded9c7ea...a95de3f2e (1 file, scripts/tests/install_e2e_container.sh): AGREED 3/3 PASS, lanes gemini-3.1-pro-high, gemini-3.8-flash-high, gemini-3.7-flash-high; author claude-opus-5-5 (non-author family).

🤖 Generated with Claude Code

@noahgift

Copy link
Copy Markdown
Member Author

Closing as superseded by main's #4439 lineage (rc.1 was already cut from it). Cop-approved.

Audit (read-only, 2026-09-26, full table in the cop handoff batch-audit.md (b)):

Kept for reference: the branch batch/0.69.4-epic-4249 is not deleted, and there is a local tag archive/pr-4318 at b3ff63c6c.

@noahgift noahgift closed this Sep 26, 2026
auto-merge was automatically disabled September 26, 2026 07:23

Pull request was closed

noahgift added a commit that referenced this pull request Sep 26, 2026
Quorum on #4318 (gemini lane): the tests exercised grown_capacity and
allocate_with_exact_retry directly, so reverting either call site in
ensure_capacity stayed green.

- cpu_a_turn_past_min_capacity_is_allocated_with_headroom: a real CPU
  session with min_capacity 16 allocates 32 for a 20-position turn,
  keeps it for 30, doubles to 64 for 33.
- gpu_headroom_that_will_not_allocate_keeps_the_turn_on_the_gpu (cuda):
  a test-only allocation ceiling makes the headroom fail; the turn stays
  on the GPU at exactly 20 positions with the #4443 notice. Control: a
  ceiling below the turn moves it to the CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c13a233)
(cherry picked from commit 402c695)
noahgift added a commit that referenced this pull request Sep 26, 2026
…ratchet); name run.sh's wall-clock DET002 (#4290, #4261)

guard-tree RED on #4318: f2_receipt.rs::decide cyclomatic 9 / cognitive 26, new vs the comparand.
guard-cargo RED: bashrs DET002 on the #4261 harness's `date +%s`, whose wall time IS the measurement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

(cherry picked from commit 9e72256)
backport note: PARTIAL: run.sh DET002 naming applied; f2_receipt.rs key_mismatch split SUPERSEDED (it splits the build_id comparison car does not have; car's decide keys 3 fields)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 26, 2026
… (0.69.4 #4318 cuda-unit red)

cuda-unit on #4318 (f230832) failed gpu_every_serve_route_prefills_through_the_batched_prefill
with left 2, right 1. Its log shows the chat request DID prefill batched, twice:
14 tokens from position 0, then 7 from position 14. e6004d7 (#4214) splits a
prompt at its checkpoint (before a chat's last <|im_start|>, else before its last
token) and updated the session test to 2 and 4; the #4250 serve test kept
"one per request".

Expected per route, from Session::prepare_prompt + Qwen35Session::checkpoint_at:
- chat, fresh: history span + generation-header span = 2
- chat again, streamed: restores request 1's checkpoint, one span = 3
- /v1/completions, raw: one span up to the last token, then a one-token step
  the counter never sees = 4
Per-token serve adds 0 everywhere, so #3596 stays RED. The messages keep their
"prefilled its prompt one token at a time" prefix, which the cuda-nightly RED
control greps.

Refs #4250 #4214 #4318

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9bc06c3)
noahgift added a commit that referenced this pull request Sep 26, 2026
…per-machine model path

guard-cargo: #4110 inserted FALSIFY-VERSION-005 before -004 in
apr-version-traceability-v1, a new "test ID gap" violation that took
contract_data_integrity to 445 against its 444 ceiling. Put 005 after 004.
validate_contracts: 10/10 pass; pv validate: 0 errors.

guard-tree: #4254's effective-config test hardcoded
/home/noah/models/Qwen3.5-0.8B-Q4_K_M.gguf, taking out-of-workspace
fixture paths 85 -> 86. The path now comes from APR_QWEN35_08B_GGUF (skip
names the variable when it is unset). check_test_fixture_paths: 85/85 PASS.
With the variable set, the test ran and passed (1 passed, no SKIP).

Refs #4318

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

(cherry picked from commit deb10aa)
backport note: PARTIAL: effective_config_route_pp2.rs per-machine model path applied; contract ID reorder SUPERSEDED (car already lists FALSIFY-VERSION-005 after -004, pointing at aprender-build-sha emit_tests)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 26, 2026
…nreachable)

Unsigned; the CI signer signs. Guard accepts a throwaway-key-signed copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 01c71e6)
noahgift added a commit that referenced this pull request Sep 26, 2026
…fix, degraded: same-family (#4318)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 02d4618)
noahgift added a commit that referenced this pull request Sep 26, 2026
…s sha (#4318)

- install_e2e_container.sh: resolve apr by walking PATH inside the container and assert the
  resolved path is tester's ~/.local/bin/apr (guard-cargo flagged 3 unpinned apr refs).
- binary-release.yml build-apr-darwin: pass APR_GIT_SHA_OVERRIDE (9-hex HEAD), as the linux
  cuda/cpu lanes already do, so the asset is refused by asset_version_check.sh if +no-git.

The e2e red on 'apr 0.69.3 (v0.69.3+no-git)' is the published v0.69.3 asset (built before
59135e4 added the override); it clears when an override-built tag is published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

(cherry picked from commit 696576e)
backport note: non-workflow part applied; WITHHELD for 36: .github/workflows/binary-release.yml hunks (this session may not edit .github/workflows)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift added a commit that referenced this pull request Sep 26, 2026
SC2106 misread the { ...; break; } brace group as a subshell: exit 0/exit 1 from the $(...)
instead, which also makes a missing apr a non-zero status. SC2154: APR is assigned in the
su shell, invisible to the linter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 6281d9f)
noahgift added a commit that referenced this pull request Sep 26, 2026
…hrs SEC001, #4318)

guard-cargo's bashrs-gate failed on 'eval "$oldng"' at :223. Record the state with
shopt -q and restore it with shopt -s/-u. Self-test PASS; bashrs-gate PASS locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e819301)
noahgift added a commit that referenced this pull request Sep 26, 2026
…xity, roadmap aggregate

- kernel-fusion-v1.yaml: FUSION-006/007/010 call_site pins moved +10 lines
  (381/384/387) after da38177 (#4273) grew kernels_generate_gemm_cuda.rs;
  fusion_call_site_guard_3985 read a match arm, not the kernel call.
- session.rs: Session::generate's prompt refusals move to admit_prompt, so
  generate drops from cognitive 26 (NEW over 25 vs main after #4325's
  cancel poll) to 20. Same errors, same order.
- roadmap.yaml regenerated with make roadmap-aggregate (16 lines of drift).

Pmat-Ticket: PMAT-4318
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

(cherry picked from commit b02807d)
backport note: PARTIAL: session.rs generate complexity split + roadmap applied; FUSION call_site pins kept as car's (they pin car's own kernels_generate_gemm_cuda.rs lines)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@noahgift
noahgift deleted the batch/0.69.4-epic-4249 branch October 9, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment