[ashhart]

Draft until it has run on an M3 Ultra. Not yet measured on a device.

This changes replies. Every Qwen3.8-27B prompt longer than one token on M1-M4 moves from the simdgroup prompt kernel to the decode path's attention arithmetic, at every width, not only past 128 rows. M5, the GQA16 and 16-over-4 shapes, and decode and draft-verify rounds don't change.

What will qualify it: byte equality with main can't, since the arithmetic differs. The check is that a prompt's state equals the state the same tokens give when fed as decode rows, at 2, 128, 129, 512 and 2,048 rows; then replies compared with main with the difference stated, and prompt speed at 1k, 8k and 32k. It overlaps #2106 on the same kernels, so that lands first.

What this changes

Qwen3.8-27B on M1-M4 previously executed at most 128 prompt rows even when its frame held more. Validated single chains now follow the admitted frame through canonical tail attention, up to 2,048 rows. Trees and mixed streams retain 128 rows; M5 and other attention geometries keep their existing selection.

The tail retains decode's half probabilities, per-row key chunks, reductions and merge. Main's simdgroup prompt uses different arithmetic, so pre-M5 prompt states can change from main. The byte replacement is against main's one-row decode tail. Performance and model qualification remain pending.

• Issue: Refs Qwen3.8-27B on M1-M4 Macs: prompt passes capped at 128 rows without tensor units; 4K prefill 109 tok/s on an M4 Pro against 0.6.6's 130 #2089, prompt-width stage. The long-history decode gap remains open.

Checks on this head

• Lean: 425 findings on base and head, zero new; changed source files at most 600 lines. Formatting and full-tree/outgoing-history privacy checks pass.
• Focused host graph: 98/98 steps and 188/188 tests pass. Final graph including every ragged width 1-2048 and the updated Metal runner build: 61/61 steps and 33/33 tests pass. Three actual planner mutations are caught and restored.
• Plain zig build -Dtarget=aarch64-linux-gnu -j1 --summary all: 31/31 steps pass.
• zig build test-cuda-host -j1 --summary all: initially 4576/4604 tests passed, 19 skipped, 9 failed. After disk headroom recovered, the two affected binaries reran with the same seed: 316 passed/1 skipped/0 failed, and 946 passed/3 skipped/1 failed. All eight recall failures cleared; the inherited HTTPS tunnel assertion remains. The 4 GiB write floor and server tests are unchanged.
• CI: not run here. Actual Metal execution and checkpoint tests are assigned to the maintainer's device run.

Receipt

• Environment: Zig 0.17.0 on a Mac with the Metal toolchain; CPU checks and a real contract-runner build only.
• Scope: pre-M5 27B GQA6 chains, including the 12/2-head rank shape. Projections, GDN, admission and native scheduling sources are unchanged. The shader changes only the two K/V physical-address selections.
• Exactness: host frame/address/keep checks pass. Prompt-against-decode state, drafted/plain, resumed/fresh and concurrent/solo checkpoint equality remain unchecked.
• Decode and cold-prefill speed: unmeasured. Compare candidate frames 128/256/512/1024 on identical cold prompt IDs before interpreting a speed change against main's different simdgroup prompt arithmetic.
• Weights or drafter licence: n/a; none added or changed.

Kernel contract

• Eight registrations: canonical chain bytes versus frozen main one-row tail/merge, and independent FP64 attention, gated and ungated, at 24/4 and 12/2 heads, D=256.
• Cases reach 129/256/512/1024/2048 rows, ragged tiles and the 8192 chunk transition. The FP64 cases include a long chain; host-checked cancellation distinguishes FP16/BF16 sums in all three contract orders.
• Two actual-source wrong-address controls must fail shapes. Their GPU execution is unchecked.
• Tail and merge launches are in the live/offline collector. Runtime-only Metal descriptor policy does not invent M1/M2 compiler measurements.
• Metal SUMMARY and exit status: unchecked; no device execution here. M1-M5 device receipts are still owed. The masked-P.V NaN repair is outside this patch.

Checklist

• Title, description and stack match this commit.
• Host red/green regression, exhaustive width sweep and restored source faults are recorded.
• Comments are single lines; no private paths, machine names or tool attribution were added.
• Runtime execution stays in the existing lane rounds.
• Metal byte replacement and checkpoint equality pass on affected chips.
• Decode and prefill performance meet the agreed gate.

[ashhart]

The M3 Ultra run is in: Qwen3.8-27B MLX 4-bit, one stream, plain and with the DFlash2 drafter, six runs per arm. The base is #2293's head and the new arm is this branch on top of it, so both builds report their prompt pass widths.

mode	prompt	prefill tok/s, base → new	time to first token	decode tok/s
plain	2,048	372.7 → 373.6 (+0.3%)	5.50 → 5.48 s	34.5 → 34.4
plain	16,384	346.3 → 265.9 (−23.2%)	47.3 → 61.6 s (+30%)	29.6 → 29.6
DFlash2	2,048	368.5 → 369.3 (+0.2%)	5.56 → 5.55 s	78.9 → 69.2
DFlash2	16,384	345.8 → 265.4 (−23.2%)	47.4 → 61.7 s (+30%)	52.0 → 64.7

• The widths are real: the new build runs a 16K prompt as eight 2,048-row passes (tensorfold.prefill_widths), where main runs 128-row passes.
• Within each build, replies are exact: repeats, drafted against plain, and the chunk and window cases all match.
• Against main, replies differ, as expected: the first difference is at token 89 for the 2K prompt and token 32 for the 16K prompt. DFlash2's decode numbers move with the replies (−12.3% at 2K, +24.3% at 16K), so they aren't a like-for-like comparison.

So wider passes on decode's attention arithmetic are slower than main's simdgroup prompt kernel at 128 rows, by a quarter at 16K, and they change replies. That's not a trade worth making, so I'm closing this draft. The prompt-width stage of #2089 needs a faster wide prompt kernel, not this route, and the long-history decode gap stays open there. The prompt pass widths themselves stay in #2293.