Repository navigation
Conversation
|
Thanks @nuk3s. We've reviewed this and it looks right: rank 0 sending the grammar bits over the plan link, and each rank masking its own vocabulary columns, is the shape we'd want. It needs a TP=2 run on two GB10s before it lands, because the shared chat change (a grammar's stop token ending an ignore_eos reply) reaches every engine. That run is ours, and our DeepSeek pair harness needs updating for current main first, so this won't make the 1.2.0 cut. We'll post the numbers here when it has run. Nothing is needed from you. |
|
Thanks @nuk3s, this is a fix people are waiting for: DeepSeek-V4.1 refuses structured output on CUDA today, so tool benchmarks fall back to unconstrained replies. We reviewed it against main, and three things are needed before it lands:
Once those are in, we'll gate it and land it with your credit. |
|
@nuk3s, main has moved and this conflicts with it. Could you rebase onto current main, and answer the two 1 Nov asks ( |
DeepSeek-V4.1's LaneHost set no .structures, so the server answered response_format, the guided_* fields and required or named tool calls with HTTP 400 (ashhart#847). The lane backend now passes each window's grammar bits to the target in Segment.masks, and before a prompt's pick it passes the first token's bits from the stream's own grammar (Target.first_mask). first_masked checks the prompt's pick against the bits the lane core asks for. With TF_DSV41_TAIL_JOIN, a joined prompt's first draft keeps only a count of held drafts, so the first round's plan has held drafts, and windows.constrain walks them with the grammar through Backend.held. Lanes.heldChain gives them: the held chain's drafts, the same tokens that `verify` puts in a window's held rows. Only constrain reads `held` here (dsv41 sets no context_copies). On the GPU target, grammar_gpu.zig fills the Forward.grammar hook. Before each forward's picks, rank 0 stages the masked rows and sends their row ids and bits to the other ranks on the plan link (op_grammar). Before a greedy or keyed pick reads "w.logits", every rank copies those rows to pinned memory, sets the logits of disallowed tokens in its vocabulary part to -inf and writes them back. Each run of adjacent masked rows is one copy each way. The pinned buffer (64 rows of the rank's part) is allocated at load, so a pick allocates nothing after rank 0 sends its op. A request without a grammar, and a row that allows every token, send and copy nothing. Tree windows give each chain row its tree row's bits; row windows stage once per forward. A nucleus row (top_p without top_k) whose grammar masks a rank's whole vocabulary part gave that rank the statistics (-inf, NaN), and keepCount sent the row to the whole-row gather. keepCount now leaves such a rank out of the sum: every logit it holds is -inf, so it adds 0. No finite sum changes, and a row without a grammar never has such a rank. With ignore_eos the server kept every end token in the reply's text, so a grammar's stop token came back at the end of the content: '{"key": "value"}<｜end▁of▁sentence｜>' on the GPUs. The lane core already ends the reply on it. The server's Generation now follows the request's grammar, from the token after `</think>` with thinking on. Under it an end token is the grammar's stop token, so it ends the reply as without ignore_eos: finish reason stop, and out of the text and the stream. The commit loop moves the grammar's phase token by token, so each end token is judged at its own phase. The stream still scans the whole landed round, and eos_phase.zig replays the phase from the round's start: an end token in the reasoning and the grammar's stop can land in one round, and judged at the round's last phase the first one would end the stream. End tokens before the grammar starts, and all end tokens without a grammar, stay in the text. draft/lanes_grammar_test.zig runs the lane engine over the CPU twin with a test grammar: greedy and keyed replies equal the twin's masked serial decode, drafted, with drafts off and four at once. A third test runs the prompts in pieces with a `tail` that leaves the last row to the first round, as the GPU target does, and joins them with one draft call or one each. A sampling_test.zig case masks one rank's whole part of two nucleus rows: the picks equal one rank's and the whole rows', and no row is gathered whole (2 before the keepCount change). grammar_gpu_test.zig covers the staged rows, the plan-link message, a rank's mask and the copy runs. hf_eos_http_test.zig sends a json_object chat request with ignore_eos on the DeepSeek-V4.1 test tokenizer, thinking off and on, whole and streamed, in rounds of every size from one token to the whole reply. Each round's reasoning and content stream with it: judged at the round's last phase, the whole-reply round streams its reasoning in 2 pieces, not 1.
The structured output section gains a line for dsv41-cuda beside GLM's CUDA backend: rank 0 sends each pick's grammar rows to the other ranks, and every rank masks its own vocabulary columns before the pick. The ignore_eos paragraph says that a grammar's stop token still ends the reply, out of the text.
The followers take the mask bits rank 0 sends, so op 28 has no bind, step or release sub-op any more. The note says so, and it no longer claims 29-31.
The new `grammar` phase covers one pick's masked rows: the copy out, the host mask and the copy back. With TF_DSV41_PHASES=sync it syncs at both ends, so with drafts off its ms_avg is the cost of one masked row.
…rallel walk The lanes grammar test now also runs its cases with first-position siblings (TF_DSV41_TREE 2-4) on the CPU twin, four at once and alone: every reply equals its masked serial decode, and the twin counts the tree windows it ran (more than 0 in each run). The test grammar gains an optional early stop and a grammar_budget hook that keeps only the stop token once a longer reply could not end within the draws left. With the budget armed and max_new below the full length, the drafted, "draft": false, tree, four-at-once and solo replies all end on the stop token within max_new and equal the serial decode under the same budget. The same grammar without the bound runs past max_new. The test grammar's matchers are independent, so it sets parallel_walk, and four constrained streams take the lane core's worker pool (ashhart#1455). The four-at-once case now checks that the pool ran.
… bits The lane core calls Backend.first_masked after the prompt pass to draw the first token among the grammar's bits. DeepSeek's lanes draw it in the pass itself: before `prefill`, `finish` or `join` they take the stream's first-token bits (Constraint.next), and the target draws the prompt's last row among them. lanes_first.zig now states this contract and first_masked checks it. The grammar takes no token between that call and Engine.drawFirst, so drawFirst passes the same bits. first_masked compares them word for word with the bits the pass drew under (error.FirstMaskChanged when they differ, and when the pass drew without bits), checks that the pick is among them (error.FirstNotMasked), and returns the pass's pick as `first` does. It runs no forward, writes no KV row and calls no drafter. Before, it only checked the pick against the bits it was given. The pass's bits are a copy in the stream's St (first_bits), freed with it. The pick stays in the pass, and is not drawn in the callback from a kept row as Gemma 4's CUDA backend does, because on DeepSeek it is the TP=2 forward's own pick: rank 0 sends the row's bits on the plan link and every rank masks its part of the vocabulary first. A later draw would need its own plan-link op and a logits row kept per slot on every rank. With TF_DSV41_TAIL_JOIN the first token comes from the first round's window, under the lane core's mask for its pending row, so first_masked is not called; when the target runs the tail row anyway (`tail` returns its pick), it is. lanes_first_test.zig drives the backend over the CPU twin with a target that runs prompts as the GPU target does: whole, in pieces, after a prefix that `begin` puts in place as a cache hit does, and joined with the tail row run. On each path, under a grammar that refuses the twin's unmasked first token, greedy and keyed: the pick equals the masked serial decode's first token; the KV rows, taps and drafter rings equal the unmasked pass's, so the bits changed only the pick; first_masked leaves them and the lanes' state as they were and adds one handle; other bits are refused with error.FirstMaskChanged, and the state stays the same. That includes bits one token away from the pass's that still allow its pick: with the pick-only check the test fails, because it takes them. A target that draws the pick without the bits is refused with error.FirstNotMasked. Through the lane engine, three replies at once whose prompts came whole, in pieces, cached, joined or tail-run equal their masked serial decode.
TF_DSV41_TREE resolves each tree window as chain windows (branches.zig), and its counts reached only the gates. Rank 0 now logs them as it logs piece and replay runs, at the first tree window, at the first one with a row its grammar narrows and at every 256th: "tree windows: N (M under a grammar), C chains, R rows run again". M counts a window only when a row's bits refuse a token, so a grammar that allows everything at that point does not count. A structured-output run with trees on can then show that tree windows ran under its grammar.
9bc9a4c to
facad85
Compare
|
Thanks @ashhart. I rebased onto main The GPU receipts ran on my two Sparks on Status: two receipts are still open. The tree receipt with the binding
Run 2 has no Two of the run 1 failures were bugs in my test tool,
I fixed the tool. The nonce is now a system message, so it comes right after BOS and ahead of the tools, and the binding-budget check takes The json_object split at token 28 (run 1, The run 2 miss beside the structured rows is the same kind of near tie, on a plain prompt. The plain reply in one of 3 passes was The #1664 gate numbers (the grammar checks, drafted vs undrafted with and without Host tests, x86_64 Linux, Zig 0.17.0, main
The one fail on main is Heads-up on open PRs: #1720 and #1722 each add a One fix to the PR body: the "0.03 ms per masked forward" line was wrong. It compared two replies of different lengths (399 and 565 tokens). I took it out. The copy cost in my #1664 reply replaces it. |
|
Thanks @nuk3s: both asks are answered (rebased; EOS judged at each token's own grammar phase, with a test; the first_masked contract written and tested), and the unconstrained path is main (no staged rows means no link send or copy). It lands after our TP=2 run on two GB10s, which waits on the DeepSeek kernel contracts already queued there. |
What this changes
Closes #1664. Follow-up to #847, as @ashhart asked there.
On
main, DeepSeek-V4.1 answers every structured request with HTTP 400:response_format, theguided_*fields, andtool_choice: "required"or a named function. Its lane host never sets.structures(the flag that tells the server it can run a grammar), and nothing fills the forward's grammar hook.This branch wires dsv41 on CUDA into main's libtfgrammar path, with drafts on. Rank 0 (the process that serves HTTP) sends each pick's grammar bits to rank 1 over the plan link it already uses to drive rank 1. Each rank sets the blocked tokens in its half of the vocab to -inf before the pick. No new kernel. The message is op 28 by itself, and ops 29-31 stay free.
Testing on two DGX Sparks turned up two more bugs, and both fixes are in here. With prompt joins on (
TF_DSV41_TAIL_JOIN), every grammar request with drafts on failed withHeldDraftsUnreadable, because dsv41 had noBackend.held. Now it has one (Lanes.heldChain). Withignore_eos, the grammar's stop token ended up in the text:{"key": "value"}<｜end▁of▁sentence｜>. Now, under a grammar, that token ends the reply withfinish_reasonstopand stays out of the text.For that
chat.zigfix, the commit loop moves the grammar phase one token at a time. The stream keeps main's scan of the whole landed chunk and replays the phase from the chunk's start (eos_phase.zig). So an end token in the reasoning and the grammar's stop can land in one chunk, and each one is judged at its own phase.The first token: the lanes take the stream's first-token bits before the prompt pass, and the target draws the prompt's last row under them.
first_maskedthen returns that pick only if the lane core hands it the same bits (lanes_first.zig), and it runs no forward and no draft.Heads-up on open PRs that touch the same lines: #1720 and #1722 each add a
Targetvtable entry iniface.zigand aTimedTargetline inphases_timed.zignext to mine. #1722 also adds a line inLanes.prefill. #1892 adds a line after theLaneHost.initcall I changed. Each conflict is a line or two on both sides, so keep both.Receipt
facad850on main8d8e2d74. The GPU runs used5eb46486on main45fcab6c, the same 7 commits before the last rebase (git range-diffchanges only commit 1, in the two files that conflicted). 2x DGX Spark (GB10), TP=2 over RoCE, driver 580.178.04, CUDA driver API 13.0,Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw@45ee5d0f. No MLX or PyTorch.token_shaagainst"draft": false. With a grammar andTF_DSV41_TAIL_JOIN=1, json_object, json_schema and a sampled json_schema are equal drafted, with"draft": falseand 4 at once. That holds on the first run, and on all 3 passes of a second run with drafts priced from the cost tables (TF_DSV41_DRAFT_TIMED=0) and one fixed cost table. In the second run the json_object prompt without a grammar is equal on all 3 passes too. A plain reply beside 3 structured ones equals its solo reply on the first run and in 2 of 3 passes on the second. The miss parts at token 3 on the rivers prompt, where main's own replies change between passes. WithTAIL_JOIN=0, first run only: json_schema and the sampled json_schema are equal, and json_object splits at token 28 in 2 of 4 concurrent replies, where the mask allows both tokens. Cached against fresh: 8/8 withTAIL_JOIN=1(second run, 2416 to 2736 cached tokens), 8/8 with 0 (first run), and 4/4 on prompts of 9631 to 9660 tokens that run in pieces. Required tools 3/3,ignore_eos2/2, a sampled reply that differs from greedy 2/2, draft trees 6/7 (the miss is my tool's finish check on the bindinggrammar_budget; the fixed check has no GPU run yet). Without a grammar, 10 of 15 rows give main's tokens on every pass. The other 5 use the rivers prompt, whose replies part at token 3, and main's replies on it change between passes too. Still open: theTAIL_JOIN=0json_object group and the fixed tree check. Both reruns failed the engine's pool check at boot.tools/bench_openai.py: I didn't runbench_openai.py. Without a grammar, on the first run, rank 0's decode tok/s (itsdoneline) on the same 565-token list reply was 112.0 on main and 111.7 on this branch with drafts on, 45.0 and 45.2 with"draft": false. Copy cost per masked row, from thegrammarphase ofTF_DSV41_PHASES=syncwith drafts off: 0.047 ms on rank 0 over 1197 rows.bench_cache.py(it sends noresponse_format). Time to first token on the first run (rank 0'sdoneline) of a cold 32768-token prompt: 12.6 s on main, 12.6 s on this branch. A cold 8626-token chat prompt: 3.5 s and 3.5 s. A request without a grammar sends and copies no mask.8d8e2d74with and without this branch.zig build test6419 pass, 45 skip, 0 fail (main 6415 pass, 45 skip, and 1 fail that passed on a rerun: a qwen3_5 disk test this branch doesn't touch).test-dsv41-draft138/138 (main 129/129),test-dsv41-device-host182/182 (main 178/178),test-eos-policy-host34/34 (main 31/31),test-server-cpu951/951 (main 948/948). GPU, TP=2: the structured-output check script 19/19, and the receipts above.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.SUMMARYline is in the receipt. (n/a: no new or changed kernel.)How it works, the GPU settings and the other test results
How the mask gets to each rank:
op_grammarwith the masked row ids and their bits before each forward's picks: about 16 KB per masked row (4,040 words of 32 bits), 64 rows max.tf_grammar_maskfromsample.cubecause it takes bf16 and dsv41'sw.logitsare fp32. A small kernel inglue.cucould replace the round trip if you'd rather have that.</think>).TF_DSV41_TREE) give each chain row its tree row's bits. Trees need one slot, since row windows refuse them. Rank 0 logstree windows: N (M under a grammar)at the first tree window, the first one with a row its grammar narrows and every 256th.The first token under a grammar (
lanes_first.zig):prefill,finishorjoin, the lanes take the stream's first-token bits fromConstraint.nextand keep a copy in the stream's state. The target draws the prompt's last row under them.Engine.drawFirst, sodrawFirstasks for the same bits.first_maskedcompares them word for word (error.FirstMaskChangedif they differ), checks the pick (error.FirstNotMasked), and returns the pass's pick asfirstdoes.l.last), because on dsv41 the pick is part of the TP=2 forward. A later draw would need a new plan-link op and a kept row per slot on both ranks.TF_DSV41_TAIL_JOINthe first token comes from the first round's window under the lane core's mask for its pending row, sofirst_maskedisn't called.Constraint.nextper grammar request, so main'sgrammar_fillscounts two fills for a dsv41 first token.Two smaller changes:
serve/sampling.zig: a nucleus row (top_pwithouttop_k) whose grammar blocks one rank's whole half of the vocab gave that rank the stats (-inf, NaN), since exp(-inf - (-inf)) is NaN. SokeepCountreturned null and the row fell back to gathering the full row. NowkeepCountskips that rank. All its logits are -inf, so it adds 0 and no finite sum changes. The token was already right on the gather path; this only drops the gather.TF_DSV41_PHASES: agrammarphase times one pick's masked rows (copy out, host mask, copy back). With=syncand drafts off, itsms_avgis the cost of one masked row. Main's host grammar timing (fa598714) times the matcher, so the two don't overlap.GPU runs:
5eb46486and main45fcab6cbuilt for aarch64 (Cortex-A725, sm_121, ReleaseFast), each arm on a fresh boot with jayleaton'sprod-zig-flash17env (TF_DSV41_TAIL_JOIN=1, 4 slots, the prompt cache on): main for the plain reference, the branch for the checks and the receipts. Then the same env withTF_DSV41_TAIL_JOIN=0(exactness, cached, a prompt in pieces), withTF_DSV41_PHASES=sync(the copy cost), and with one slot andTF_DSV41_TREE=3(trees). Then a second run withTF_DSV41_DRAFT_TIMED=0(main's default) and one cost table read at boot (TF_DSV41_CALIB_FILE), the same table on main and the branch, 3 passes on one boot per setting: the plain set on both builds, the exactness set on the branch (main runs only its group without a grammar), and the cached receipt on the branch, all withTAIL_JOIN=1. ItsTAIL_JOIN=0and tree boots failed the engine's pool check (PoolDoesNotFit).Other host results:
test-dsv41-cuda-host357/357,test-shared-lane-regressions-host316 pass, 1 skip,test-tool-required-host26/26,test-grammar-error-host14/14,test-grammar-time-host14/14,test-grammar-parallel-host23/23,test-grammar-metrics-host18/18,fuzz-cortheon19/19,test-draft-dsv41-host1/1,test-server-security19/19,test-pull-security9/9, andtest-goldenpasses.grammar_budgetthat ends the reply insidemax_new, and with four streams on the lane core's parallel grammar walk (Lanes: grammar mask fills for many constrained streams run serially on the lane thread #1455). Every reply equals its masked serial decode. The new tests are intest-dsv41-draftandtest-dsv41-device-host;zig build testgains 3.5eb46486with one change taken out: withoutheldChain, 2 joined-prompt tests fail withHeldDraftsUnreadable(test-dsv41-draft136/138). Without thekeepCountchange, thesampling_test.zigcase gathers 2 rows whole where it expects 0 (137/138). Without the grammar inchat.zig, the EOS test gets{}<｜end▁of▁sentence｜>instead of{}(test-eos-policy-host33/34). With the stream judging every token at the chunk's last phase, the thinking +ignore_eostest streams the whole-reply chunk's reasoning in 2 pieces instead of 1. Withfirst_maskedchecking only the pick, thelanes_first_test.zigcase takes bits one token away from the pass's (137/138), and without the pick check it takes a pick drawn without the bits (137/138).compile-dsv41-serverpasses.lean_check.py: 424 problems on the base and on the branch, 0 new.zig fmt --checkandgit diff --checkare clean.