Repository navigation
lanes: Drafted passes held windows to a target that reads them itself - #1378
BlivionIaG wants to merge 1 commit into
Conversation
With Drafted.held_on_target set, Drafted.verify gives the target its windows with Window.held as the round loop made them, so a target that reads held drafts from the drafter on the device skips the host readback. A window of a stream with fillers still gets host tokens. The flag is off by default; without it verify runs as before and the new check is not evaluated. drafted_held_test.zig checks the three cases: held windows pass through, a stream with fillers gets its pending token as host tokens, and with the flag off the drafter's drafts are read back as host tokens.
|
Thanks @BlivionIaG. This one reviewed clean on its own: default-off, no source defect found in its commit. It lands after a device check showing unchanged tokens and a per-family A/B. Please qualify the actual asynchronous consumer before anything enables it. |
|
Thanks @ashhart. The consumer of |
Branch
series/lanes-held, 1 commit on main9a8d654ef:9a8d654ef. Sits on main. Part of #1374.What this changes
Drafted.held_on_target(default false). When a target reads held drafts from the drafter on the device, Drafted.verifygives it the windows with
Window.heldas the round loop made them, so the host does not wait to read the drafts back.A stream with fillers still gets host tokens. With the flag off, verify runs as before; the new check is not evaluated.
Nothing on main sets the flag. It lands with the LFM2 DSpark drafter PR, which copies held drafts on the GPU. It is a
separate PR so the lane-core change can be reviewed on its own.
Receipts at this head
Head
9a8d654efis unchanged by the review and mergeable with main. The consumer ofDrafted.held_on_targetis the LFM2 Metal backend only: it reads the drafter's held picks on the GPU throughheld_sourceandlfm2_draft_idsafter an unwaited hold. No other family sets it.Device check, M4 Pro, the DSpark stack head (bf16 drafter):
exact.py8 of 8 drafted equal"draft": false;bench_concurrent --alone --serialat 1, 4, 8 and 16 streams, same and distinct prompts, 0 unequal, 0 failed; sampled (temperature 1.0, seeded) distinct prompts at 4 and 8 streams equal. Speed in those cells is not valid (other builds ran).Test that fails before the change (
drafted_held_test.zig, run with the flag kept but the pass-through removed):242 pass, 1 fail: "held_on_target passes held windows through ..." expected 3, found 0. With the change: 243/243.Host gates on this branch (fresh cache; main: lanes 242/242, server-cpu 792/795 (3 skipped), cuda-host 2021/2033
(12 skipped), golden 322/335 and 26/26 equal with 0 unexpected, lean 433 problems):
Lean: 433 problems, 433 on main, 0 new (
tools/zig/lean_new.py). The 433 are inherited.A/B on current main (other families and builds)
M4 Pro, LFM2.5-24B-A2B 4-bit, no drafter, 4 streams, a different prompt on each, greedy 256 tokens, 4 alternated rounds, a fresh server per cell, a quiet machine (load under 2, no Zig build), every build merged with main
8c36359d5; aggregate tok/s by round:Every build is within 0.5% of main and every reply token-equal to main's. An earlier single-round run showed #1378 at -6% and #1379 at -4%; four rounds showed nothing, so those were session effects.
M5 Ultra, Qwen3.8-27B + DFlash2, 4 streams, distinct prompts, greedy 256 tokens, 6 rotated rounds, 3 reps a cell,
token_shaequal across all four binaries.main2is8c36359d5built a second time in a separate worktree. Median of the 3 reps, tok/s:6539b72be)1 stream: 296.0, 296.5, 296.0 and 295.8. The two builds of identical source differ as much as the lane head does, and the spread inside a cell is 304 to 333, so the gap follows the build or the session, not the code. An earlier session had reported 1.8% to 2.7% lower at this cell; I did not reproduce it.
The 6-round table above includes the held head; the other families are in #1382.
Not measured
Related PRs
Sits on main. Merges cleanly with #1379 and #1380. The DSpark drafter PR (#1391) sits on it.