[BobClawblaw]

What

--drafter for the llama family: a smaller Llama on the same tokenizer proposes a stream's next tokens, and the served model verifies them in its windows, so replies keep the model's bits (builds on #1415). It was brought up with Nanbeige/Nanbeige4.1-3B drafting for Nanbeige4.3-33B, both served from their bf16 releases as published. The two share the tokenizer exactly (166,100 ids, merges, added tokens, bos and eos); only their vocab_sizes pad differently. The 3B is admitted as a drafter only: serving it on its own is refused by name. --drafter-bits 4 and quantized drafter checkpoints are refused by name too: the drafter is the bf16 release, nothing prepared at load.

How

zig/src/families/llama/drafter.zig is the lane host's priced copy proposer (LaneHost.proposers). A round asks it for up to 15 tokens after the stream's context. The window verifies them row by row and keeps the prefix the model's own draws agree with, as it keeps any copy draft. There is no lane-core change and no new kernel: the drafter runs the family's own forward on the served model's queue (Model.loadOn) and in its residency set.

• One budget. The streams' caches and the drafter's share one ledger (budget.Ledger). A stream's admission (budget.admitStream) checks its peak first (its caches, its drafter cache's growth and the old state's copy), kept drafter caches shedding first; trims the kept prompt states to what that peak leaves, freeing a restore entry that can't stay beside it; and only then allocates. A freed drafter cache keeps its tokens for the next request sharing its prefix, counted, and gives way first.
• Drafts. Greedy steps chained on the GPU through token slots (a step's argmax is the next step's row), so one proposal is one chain of command buffers. A chain stops at an eos.
• Depth. It follows measured rates: the stream's decayed acceptance, a chained step's wall time, and a line fitted to the round's time by its window's rows. A stream alone drafts the depth with the most expected tokens a millisecond; with several streams in a round, each drafts the depth with the most tokens a millisecond for the whole round (pricing.depthShared), nothing only where drafting costs the round more than it lands.
• Start and stop. Drafter.start owns the model from the call and frees the drafter if its warm-up fails; the host frees the drafter before the backend whose ledger it gives back to.
• Check mode. tf-llama-run <model> <tokens> draft <drafter>,<n> decodes greedily through the drafter's windows and prints the tokens with the rounds and drafts kept.

Measured (M5 Ultra, the bf16 33B and the bf16 3B, 28 Oct 2026, one model process on the Mac)

Identity. On eight chat prompts (128 tokens, thinking on and off, English and Chinese) and after a 1,500-token prompt (256 tokens), the drafted greedy tokens equal the plain greedy tokens exactly: 21.1-40.7 tok/s drafted against 15.8-16.1 plain, 1.73-4.54 tokens landing a round, a chained bf16 step 8.5-8.9 ms. The recipe doc has each prompt.

Two streams. On fresh servers (--context 32768 --parallel 2 --prompt-cache-gib 8), eight new prompts sent two at a time (greedy, 256 tokens, thinking off) decode together at 30.1-40.9 tok/s (36.1 on average) with --drafter, against 26.6-30.1 (29.0) without; every reply's token ids are equal. Streams joining, leaving, queued, refused (HTTP 400) and cancelled each return the ids of their plain solo run, a reply alone after a cancel is unchanged, and shared rounds keep drafting (199 drafts offered, 125 kept).

Receipt

• Environment: TensorFold at this branch (main e4cc630 plus these commits), Zig 0.17.0, one Apple M5 Ultra (256 GB unified, macOS 27, tensor units). Nanbeige/Nanbeige4.3-33B @ c1093c501cc6460427c0ac634db7223d0450c0ee with --drafter Nanbeige/Nanbeige4.1-3B @ 9c4555c37921c0982af8fffeee1e0a6e6b953c4f (--drafter-bits 0), LLAMA_LOAD_LIMIT_GB=150.
• Tests run: On these commits over main ea01ae1: zig build test and test-kernel-contract exit 0 on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's head; on llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415) #1417's head, which carries llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0; zig fmt --check is clean, the lean check finds 0 new problems (426 on main and on each head), llama_teacher_judge_test.py passes, and tf-kernel-contract --only llama on the M5 Ultra gives SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. Rebased onto main e4cc630 for this push (its lane-host commits change only TENSORFOLD_PREFILL_SHARE, off by default, and a flag only GLM's CUDA backend reads): both heads rebuilt, zig build test exits 0 on both and test-kernel-contract on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's, fmt is clean, lean finds 0 new, the bf16 1,500-position logits are bit-identical to the judged run on both, and a drafted greedy chat equals the plain one.
• Not run: the harness quality suite; the harness's serving plans with the bf16 drafter (they passed with an earlier 4-bit drafter, whose drafts change no reply either); CUDA, HIP, M1-M4.

Checks

zig build test covers the depth rule and its shared-round pricing, the cost fit, the acceptance history, the drafter role, the roll call, the one budget (two drafted 240,000-token requests that each fit a 100 GiB budget alone are refused together), admission's order with an allocation recorder (kept prompt states trimmed before anything is made, a failed growth giving every byte back) and a failed warm-up freeing the drafter. Without --drafter, the served path is #1415's: the 1,500-position logits are bit-identical to its judged run.

[ashhart]

Drafted equals plain per the receipts, and there's no lane-core change or new kernel. That's good. Two things:

• The default --drafter-bits 4 refuses a bf16 drafter and needs a llama_convert.py --bits 4 conversion, so the default depends on llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416. Please make bf16 the default, or accept MLX's layout once llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416 does.
• Shared rounds. With the drafter, two-stream greedy drops from 31.0 to 27.8 tok/s (chat) and from 34.1 to 29.5 (code) on an M5 Ultra by the PR's own numbers. Yet the recipe keeps the drafter on. Either skip drafts in shared rounds when they cost more than they land, or don't recommend the drafter above --parallel 1.

[BobClawblaw]

Thanks. Pushed 5d4a3af (one commit, on #1416's new head):

--drafter-bits 4. We took the second route: with #1416 reading MLX's layout in place, the default now serves a published MLX 4-bit drafter as it is on the Hub, mlx-community/Nanbeige4.1-3B-4bit @ 4d6c41d88f1416777bbda005aad92798e56e5ba9. There is no conversion and nothing depends on llama_convert.py, which is gone. --drafter-bits 0 serves the bf16 release. With the published 4-bit drafter, drafted greedy tokens equal plain greedy tokens on all nine check prompts, at 23.2-46.4 tok/s against 16.0-16.3. The bf16 drafter gives the same tokens on two of them, at 8.5 ms a step against 4.0-4.2 ms.

Shared rounds. Both of your options, in turn:

1. While more than one stream holds a drafter cache, the drafter proposes nothing (Drafter.sharing), so shared rounds decode undrafted. On fresh servers (bench_concurrent.py, 128 greedy tokens, 8 runs each), two drafted streams give 31.2-31.3 tok/s together on chat against 31.3 without --drafter, and 31.4-31.6 on code against 34.4. On the harness ruler they give 31.9-32.5 against 28.0-31.8.
2. Code is still 8% behind, because a server without --drafter keeps the lane core's own copy drafts and shared rounds give those up. Getting them back would mean copying the lane core's copy policy into the family, so instead the recipe no longer recommends the drafter above --parallel 1. Its drafted command is now --parallel 1, and the doc says a server for several streams leaves the drafter off. The harness's quick and speed plans pass on that server (one stream 99.5-116.8 tok/s on the ruler).

One correction to the old receipt: the 87-117 tok/s single-stream figures from bench_openai.py and bench_concurrent.py --alone overstate ordinary text. Those tools send the same prompt more than once, and a freed drafter cache keeps its tokens for a request sharing its prefix, so a greedy repeat drafts the earlier reply back whole. On a fresh server one stream gives 30.0-32.8 tok/s on bench_concurrent.py's code and 24.9-25.3 on its chat, against 20.2 and 16.4. The PR body now quotes those.

Also: the 3B is admitted as a drafter only (config.zig's qualified list has a role). Serving it on its own is refused by name.

On these commits over main a1e6d34: zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security, test-pull-security all exit 0; zig fmt --check is clean; the lean check finds 0 new problems (428 on main and on the head); tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. After the rebase, the Fibonacci prompt's drafted tokens still equal the plain ones. Main then took MOSS-Image (#1457), DFlash2 and MTP fixes for other families, a GLM CUDA log message and an opt-in prompt-cache rule for CUDA families (off by default): none of it in this family's files or the paths it serves through. Rebased onto main 15a0b2a for this push, the stack's top head (#1417's, which carries these commits) reran the build, tf-kernel-contract --only llama, test-kernel-contract, zig fmt --check and the bf16, quant and drafted bit-identity checks, and all pass.

[ashhart]

Thanks @BobClawblaw. The drafter's default now reads the published 4-bit layout, or bf16 with an explicit width of 0 (zig/src/native/llama.zig:158), and the recipe pins both checkpoints.

Three asks:

1. One memory budget. Startup counts both drafter caches (zig/src/native/llama.zig:94), then hands the whole remaining budget to the target (:110). Runtime admission doesn't count the drafter's live and kept caches, its growth copies or kept prompt states. On the production formulas, with a 100 GiB cache budget, context 262,144 and two streams, startup passes, yet two admissible 240,000-token requests together exceed the budget by 25.3 GiB. Please admit the target and the drafter against one budget, and add a host test where two requests that each fit alone are refused together.
2. Shared rounds. Thanks for taking up the shared-round point. Our suggestion was to skip a shared round's drafts when they cost more than they land; the head skips them in every shared round (zig/src/families/llama/drafter.zig:141, :345), so drafting stops as soon as a second stream joins. Our recipe rules count that as a drafts-off fallback, which is a bug for an advertised drafted path (docs/recipes/SKILL.md, the concurrency rules). Please keep drafting on for every admitted stream and price each shared round, then check drafted against plain and concurrent against solo as streams join, leave, are refused and are cancelled.
3. Warm-up failure leaks the drafter. If warm-up fails (zig/src/native/llama.zig:117), the drafter made just before it is never freed: its errdefer comes after, at :121. Please register the cleanup before the warm-up, and add a test that fails it.

[BobClawblaw]

Thanks. Pushed b652456 (one commit on #1416's new head):

1. One memory budget. The backend's admission now runs on one ledger (budget.Ledger) for every cache the load holds. That covers the streams' caches, the drafter's slots (taken or kept) and a slot's growth copy while it lives. When a stream is admitted (backend.prefillFn), its drafter slot is made or grown to the stream's capacity in the same check (Drafter.slotsHook: growth, prepare, shed). If the two don't fit beside what is held, kept drafter slots give way first; if they still don't fit, the stream waits (error.LaneMemory) until another finishes. Slots no longer grow during decoding, so nothing allocates outside admission. Kept prompt states fit in what the ledger leaves. The host test (llama budget: two drafted requests that each fit alone are refused together, and pass in turn) takes the 33B's and the 3B's configs: a 100 GiB budget, a 262,144-token context and two streams pass startup. One 240,000-token request fits, the second waits (together they would pass the budget by 24.5 GiB on these formulas), and after the first ends the second fits beside the first's kept slot. growth has its own test (the new cache plus the old state's copy, nothing when the slot holds enough).

2. Shared rounds are priced, and drafting stays on. pricing.depthShared gives each stream in a shared round the depth with the most tokens a millisecond for the whole round. Each stream's drafts land 1 + p + ... + p^k tokens, for a window of all the streams' rows plus every stream's chained steps, against the round undrafted. It returns 0 only for a round where drafting costs more than it lands. Shared rounds don't feed the fitted line of a round's time by its rows. Checked on a drafted --parallel 2 server (--context 32768), greedy, token ids from return_token_ids:

• A drafts alone; B joins after A's first 40 tokens (shared rounds); C arrives while two streams hold the server and queues; D starts and its client disconnects after 20 tokens (cancelled); E sends a prompt past the context and is refused with HTTP 400. A, B and C each return the token ids of their plain solo run ("draft": false), and A alone after the cancel is unchanged.
• Two streams from the first token to the last: both equal their plain solo runs, and the shared rounds offered 189 drafts and kept 127.
• Speed on fresh servers, eight new prompts sent two at a time (greedy, 256 tokens): two drafted streams decode together at 34.4-42.4 tokens/s (38.4 on average) against 26.6-30.0 (29.0) without --drafter, all eight replies' ids equal to the undrafted ones.
• bench_concurrent.py --levels 1,2 --alone --serial on a fresh drafted server: every reply equals its request alone and undrafted (alone 48/48, serial 12/12). Its speeds aren't quoted, because its repeated prompts let a freed slot draft the earlier reply back.

3. Warm-up failure. Drafter.start owns the model from the call: if init fails it frees the model, and the drafter it makes is freed if its warm-up fails (started, with the cleanup registered before the warm-up). The host frees the drafter before the backend, whose ledger the drafter's slots give back to. The test llama drafter: a warm-up that fails frees the drafter it was made for makes the warm-up fail and checks the drafter is freed. tf-llama-run's draft mode uses Drafter.start and Drafter.prepare too.

The harness's quick plan on the drafted --parallel 1 server, this build (its ruler with no other model loaded): structured 18/18 with drafted == plain 12/12, tools 5/5, sampling 9/9, turns resumed 5,493 of 5,649 tokens (== cold), soak 60/60, drafts 12/12, concurrent 12/12, streaming 4/4, multi-turn 8/8, stop 3/3, thinking 10/10, copy 2/2, max_tokens 3/3. Its ruler gives 113.5 / 110.7 / 109.2 / 101.8 tokens/s on one stream (texts the drafter continues almost whole).

The doc's drafter section is rewritten for the budget and the priced rounds. --parallel 2 drafts both streams.

CI on these commits over main 03d08f1 (the pushed heads are rebased onto d0c9dc9, which adds only a CI workflow time limit): on #1417's head, which carries all three, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0, and on #1415's and #1416's heads zig build test and test-kernel-contract exit 0; zig fmt --check is clean and the lean check finds 0 new problems (428 on main and on each head); llama_teacher_judge_test.py passes; tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems (21 kernels on #1415's head). Rebuilt on the rebased heads, the bf16 release's 1,500-position logits are bit-identical to the judged run on all three, both quants' to theirs, and a drafted greedy chat (128 tokens) equals the plain one. The pushed heads differ from the ones CI ran on only in comments and docstrings (the lean check's one-line rule); fmt, lean, the judge's tests, the build and zig build test were rerun on them.

[ashhart]

Thanks @BobClawblaw. On b6524565, the drafter's cache and growth copies are in the common ledger (budget.zig:24), shared rounds are priced against their undrafted cost instead of skipped (pricing.zig:60), and the drafter's cleanup is registered before warm-up, with a failed-warm test.

One ordering problem is left. backend.prefillFn reserves the target and calls the drafter's prepare (zig/src/families/llama/backend.zig:216-218) before room.fit (:224) trims the prompt store. But prepare has already allocated the new cache and the old-state copy (drafter.zig:220, :226). Retained prompt snapshots can still fill the old budget, so the peak can pass it. Worked through with this PR's own formulas at an 8 GiB budget, old drafter cache, prompt states, new cache and copy reach 9.61 GB, about 1 GB over, before the trim runs. Please trim or reserve that peak before allocating, keep a protected restore entry or refuse safely, and add an admission test with retained prompt states and an allocation recorder that checks the peak, including a failed prepare.

The checker and float64 requests on #1415 and #1416 also apply to this stack.

[BobClawblaw]

Thanks, that ordering was wrong. Pushed 50fe245. This PR now builds on #1415 alone: we've withdrawn #1416 (the quants; the reasons are there), so the drafter is served from its bf16 release as published.

Admission before allocation. budget.admitStream now runs a stream's admission in one order, and backend.prefillFn goes through it:

1. the stream's peak (its caches, plus its drafter slot's growth with the old state's copy) is checked against the ledger, kept drafter slots shed first, LaneMemory if it still doesn't fit;
2. the room that peak leaves is computed, and a restore entry that can't stay beside the peak is freed (the stream prefills cold) — otherwise it stays protected;
3. the kept prompt states are trimmed to that room (room.fit, never evicting the protected entry);
4. only then is the stream's cache reserved and the drafter's prepare allowed to allocate; after the cache is made, the store gets back what the copy and the old slot gave up.

The host tests drive the same function with a fake store, drafter slot and allocation recorder that tracks the most the ledger and the kept states hold at once:

• your shape at an 8 GiB budget (slot 1, kept states 3 with a 1 GiB restore entry, a 3 GiB stream, a slot growing to 2 with a 0.5 copy): allocating before the trim would reach 9.5 GiB; the recorded peak is 8.0, and the protected entry stays;
• a restore entry too big to stay beside the peak is dropped, and the peak stays inside;
• a failed prepare gives every byte back (the ledger returns to the old slot alone) with the peak inside;
• a peak that can't fit at all waits before anything is trimmed or made.

The drafter without #1416. --drafter takes Nanbeige/Nanbeige4.1-3B @ 9c4555c37921c0982af8fffeee1e0a6e6b953c4f as published (bf16, 7.33 GiB); --drafter-bits 4 and quantized drafter checkpoints are refused by name before anything is served ("a Llama drafter is served from its bf16 release as published"). Measured again on this head, one model process on the Mac:

• tf-llama-run ... draft: on eight chat prompts (128 tokens, thinking on and off) and after a 1,500-token prompt (256), the drafted tokens equal plain greedy exactly, at 21.1-40.7 tok/s against 15.8-16.1 (1.73-4.54 tokens a round; a chained bf16 step 8.5-8.9 ms).
• Served, two streams (--parallel 2), fresh servers, eight new prompts two at a time: 30.1-40.9 tok/s together (36.1 on average) against 26.6-30.1 (29.0) undrafted, every reply's ids equal.
• The streams scenario (join, leave, queue, refuse with HTTP 400, cancel) returns each reply's plain solo ids, and shared rounds keep drafting: 199 drafts offered, 125 kept.

CI on these commits over main ea01ae1: zig build test and test-kernel-contract exit 0 on #1415's head; on #1417's head, which carries #1415, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0; zig fmt --check is clean, the lean check finds 0 new problems (426 on main and on each head), llama_teacher_judge_test.py passes, and tf-kernel-contract --only llama on the M5 Ultra gives SUMMARY 21 kernels: 21 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. Rebased onto main e4cc630 for this push (its lane-host commits change only TENSORFOLD_PREFILL_SHARE, off by default, and a flag only GLM's CUDA backend reads): both heads rebuilt, zig build test exits 0 on both and test-kernel-contract on #1415's, fmt is clean, lean finds 0 new, the bf16 1,500-position logits are bit-identical to the judged run on both, and a drafted greedy chat equals the plain one.