[BobClawblaw]

What this changes

With --drafter, a llama request that stands aside (its drafts cost more than they land) probes the drafter after 64 plain rounds, then after twice as many each time, up to 1,024. On a lone stream those probes were never verified. The lane core asks the proposer once to look ahead and again to verify with the context a token longer, and Request.propose reset the stand-aside count and doubled the gap when it answered the look-ahead. The verify ask was then no longer a probe and the round went plain, so every probe paid its drafter steps and a request that stood aside never measured its acceptance again.

The probe now stays armed (probing) until a round verifies its drafts, and round() resets the count and the gap there. It keeps the one plain round of the lone stream's look-ahead; if the round after that verifies nothing too (the lane core can trim a priced proposer's drafts to nothing through the copy plan), the probe disarms and its gap doubles, as a probe cost on main. Only zig/src/families/llama/drafter.zig changes, plus two tests.

• Issue: Fixes llama --drafter: a request that stands aside never re-measures, its probes are spent on the look-ahead ask #2168
• Stacked on: main (a26fdae91)
• atc overlap --json: no overlap for this change. Coverage was incomplete (pullFilesUnavailable, timeLimit), so I checked the file lists of all open pull requests by hand: none touches zig/src/families/llama/drafter.zig or its test.

Checks on this head

• Head commit: e7e4db398 (40c97348f plus the disarm after a second plain round)
• On e7e4db398, M5 Ultra, macOS, Zig 0.17.0: zig fmt --check, zig build test, test-kernel-contract and test-golden exit 0; lean 425 on the base and on this head, 0 new.
• On 40c97348f: also test-server-security test-pull-security (24/24), fuzz-cortheon, check-release-paths and dev-tools exit 0.
• test-native on the base this was first checked on (b331731e6) passes except main's own crash in the Metal Qwen27 draft-sources test, fixed by native tests: the Qwen27 lanes' facts read a dense model's config (test-native crash) #2122.
• The new test, llama drafter: a probe stays armed until a round verifies its drafts, then its gap doubles, walks a probe through a plain round (still armed) and a verified one (gap doubled, count reset); it does not build without the change. The second, llama drafter: a probe whose drafts the lane core cuts disarms after one more plain round, its gap doubled, runs two plain rounds after arming and expects !probing, the stand-aside count reset and the gap doubled.

Served: bf16 Nanbeige4.3-33B with Nanbeige4.1-3B drafting

On this head (e7e4db398)

One stream (--parallel 1, 16,384-token window), a fresh server per run, the eight requests below, 3 passes of main (a26fdae91) with --drafter, this head with --drafter and main without it, with no other GPU work during any pass.

All 72 replies are byte-identical per case across the three builds and passes. chat2, decode tokens a second (drafter rounds, accepted/offered drafts):

pass	main + drafter	this head + drafter	main, no drafter
1	18.68 (221, 158/337)	18.55 (221, 158/336)	12.92
2	18.46 (222, 157/336)	17.91 (221, 158/336)	11.74
3	11.86 (375, 4/11)	17.15 (241, 138/290)	12.38

Main stood chat2 aside only in pass 3, and stayed at plain speed; this head came back to drafting. The other seven cases are level: the six chat and story requests average 14.73 on main with the drafter, 14.83 on this head and 12.38 without it; copy runs 85.2-86.9 with the drafter on both.

Earlier, on 40c97348f

One stream (--parallel 1, 16,384-token window), fresh server each time, eight requests in order: greedy chat, story and copy, four seeded sampled chats and stories at 0.7, then the greedy chat again (chat2). Main (b331731e6) with --drafter against this change with --drafter, and main without it.

Replies. Every reply is byte-identical across main with the drafter, this change with the drafter and main without it, in every pass (8 cases, 3 passes, and 3 more passes in an earlier run).

The case this fixes is chat2: the sampled requests before it can stand the stream aside, and on main it then stays at the plain speed. Tokens a second, decode:

chat2	main + drafter	this change + drafter	main, no drafter
2 Nov, the same change on the stack before it landed (2 fresh servers each)	13.61, 13.55	16.70, 19.14	13.51, 13.71
4 Nov, on main, the quietest pass	12.40	18.58	10.39

The other seven cases are level within noise in both runs.

The latest run is noise, not signal. The Mac was shared while I measured: plain decode, normally about 13.5 tokens/s, ran at 6-10 in the last three passes, and in those passes main did not stand aside on chat2 either (rounds and accepted drafts: main 226 and 153/283, 235 and 144/275, 219 and 160/305; this change 226 and 153/286, 222 and 157/311, 225 and 154/286). So those passes show no regression, with the same replies and the same drafting where both draft, but not the fix itself. The speeds above come from the passes where plain decode ran near its normal rate.

Receipt

n/a: the change is in the drafter's stand-aside bookkeeping; no kernel or served bit changes.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @BobClawblaw, the diagnosis is right: the look-ahead ask took the probe, so the verify ask a round later came back plain, and a request that stood aside never measured its acceptance again. Moving the reset into round() fixes that, and the byte-identical replies across all eight cases are what we'd want to see. One change before it lands. probing now clears only when a round verifies drafts, and aside doesn't move while drafts are given, so an armed probe drafts on every ask until something verifies. The lane core can still cut a priced proposer's drafts: they become a copy plan, and allocate can trim it to zero when the stream's copy rate is low, which is likely for a request that stood aside. Then every round pays the drafter's steps for nothing, where on main a probe cost one ask before its gap doubled. Could you keep the one plain round between the look-ahead and the verify, but disarm and double the gap if the round after that verifies nothing too? A test with two plain rounds after arming (expect !probing and the gap doubled) would pin it. The rest is ready.

[BobClawblaw]

Thanks @ashhart. Done in e7e4db398: the probe keeps the one plain round of the lone stream's look-ahead, and if the round after that verifies nothing too, it disarms, its gap doubled and the stand-aside count reset, the cost a probe had on main. A round that verifies still resets as before.

The new test, a probe whose drafts the lane core cuts disarms after one more plain round, its gap doubled, runs two plain rounds after arming and expects !probing, the count at 0 and the gap at 16 (from 8). On this head zig build test, test-kernel-contract, test-golden and fmt exit 0, lean 0 new. The served A/B on bf16 Nanbeige (replies against main with and without the drafter, and chat2) is still to run on this head; I'll post it here when it is in.

[ashhart]

Thanks @BobClawblaw, that's the bound: one plain round for the look-ahead, then it disarms with its gap doubled, and
the new test pins the trimmed case. The code is ready. It lands once your served A/B on this head is in.

[BobClawblaw]

Thanks @ashhart. The served A/B on e7e4db398 is in: bf16 Nanbeige4.3-33B with Nanbeige4.1-3B drafting, M5 Ultra, one stream (--parallel 1, 16,384-token window), a fresh server per run, the same eight requests as before, 3 passes of main (a26fdae91) with --drafter, this head with --drafter, and main without it. No other GPU work ran during any pass (the script waited for the Mac to be idle and would have repeated a pass that overlapped other work; none did).

Replies. All 72 replies are byte-identical per case across the three builds and three passes: one hash per case, greedy and seeded sampled alike.

chat2, decode tokens a second, with the drafter's rounds and accepted/offered drafts:

pass	main + drafter	this head + drafter	main, no drafter
1	18.68 (221, 158/337)	18.55 (221, 158/336)	12.92
2	18.46 (222, 157/336)	17.91 (221, 158/336)	11.74
3	11.86 (375, 4/11)	17.15 (241, 138/290)	12.38

In passes 1 and 2, main never stood the stream aside on chat2, so the two builds draft the same rounds and run level. Pass 3 caught the bug on main: the stream stood aside and offered 11 drafts in 375 rounds, at plain speed. This head stood aside and came back to drafting (290 drafts offered), at 17.15.

The other seven cases are level: mean decode of the six sampled and greedy chat/story requests 14.73 on main with the drafter, 14.83 on this head, 12.38 without it; copy 85.2-86.9 on both with the drafter.

The description is updated to this head.

[ashhart]

Thanks @BobClawblaw for the A/B, and especially for making the script wait for an idle Mac: that's what makes pass 3 readable. All 72 replies match per case, the seven other cases are level, and chat2 came back to drafting where main stood aside. Ready to land, and it's in the queue.

[ashhart]

Thank you @BobClawblaw. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @BobClawblaw! This is in TensorFold 1.2.3, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.3