Repository navigation
Conversation
The lane backend fills Stream.prompt_rows, so the server stops refusing prompt_logprobs for GLM. A prompt chunk's rows go through each rank's head columns 128 at a time; the rows are made whole on every rank with the broadcasts a sampled draw already uses (Sampler.gather), and rank 0 runs sample.cu's tf_logprob_rows on them. Rank 0's prefill and chunk steps carry flag 1 so the other ranks join each pass. The pass's buffers are made at the first request that asks; if a rank cannot make them, every rank takes the verify window's buffers, 8 rows a pass. A pass with prompt rows starts from position 0, as Nemotron's does.
The lane backend gives first_masked and held, so the server serves response_format and required tool calls for GLM. Only rank 0's lane core holds the grammar, and only rank 0's tokens are read (it sends them on), so rank 0 alone masks: a verify or shared step under a grammar carries flag 1, every rank gathers its draw's rows whole (Sampler.gather, as a sampled draw does), and rank 0 runs sample.cu's tf_grammar_mask on them, then tf_draw, or greedy's rule (tf_glm_top1 over the whole row: the largest value, then the lowest id, as the per-rank merge picks). The first token is drawn again from the prompt pass's last head row by a new redraw step. The grammar rows go to host-mapped words before the step is sent, so a refusal never strands the other ranks mid-broadcast. Draft draws stay unmasked.
|
Thanks @Bizuayeu for the TP=2 and TP=3 schema, seeded-sampling and drafted/plain comparisons. We're taking this with fixes for 1.1.1. The first constrained draw must use its own sequence's retained hidden row, including with MTP off, because another prompt can overwrite shared logits before that draw. We also fixed follower-result initialization and mask-buffer ownership on mapping failure. Held drafts still come from the active MTP or DFlash path. The repaired combination needs generated schema/tool tests with real grammar support, mixed constrained/plain streams, cached resumes, separate prompt passes and MTP disabled. Your original receipts are useful evidence, but they do not yet qualify those repairs on the combined engine. Posted by Ash's AI Assistant Synthena |
|
Thanks. Once the combined 1.1.1 branch is up, we'll run these against it on GB10 at TP=2 and TP=3 (TP=3 still with rank 0's NCCL id handed over in a file until #636), with
|
2 participants
What this changes
Part of GLM-5.3-Flash on the native CUDA engine (#593, where the three stacked PRs for GLM's ranks are summed up): replies that always parse against their schema.
GLM-5.3-Flash's CUDA lanes give
first_maskedandheld, so the native server enforcesresponse_format,guided_*andstructured_outputson GLM instead of answering HTTP 400. Stacked on #1016 (it uses that PR'sSampler.gather).Only rank 0's lane core holds the grammar, and only rank 0's tokens are read (it sends them with the next step), so rank 0 alone masks. A verify or shared step under a grammar carries flag 1: every rank gathers its draw's rows whole, the grouped broadcasts a sampled draw already uses, and rank 0 runs
sample.cu'stf_grammar_maskon them, thentf_draw, or greedy's rule astf_glm_top1over the whole row (the largest value, then the lowest id, which is what the per-rank pairs' merge picks). The first token is drawn again from the prompt pass's last head row by a newredrawstep. The grammar rows go to host-mapped words before the step is sent, so a refusal never leaves the other ranks inside a broadcast. Draft draws stay unmasked;heldreturns the drafts the lanes already hold on the host.zig/tests/cuda/glm_lanes_tests.zigtakes the newverifyargument.Receipt
38aa9875, this PR's32141eb6with the next PR's server-side tool-call style on top (the code of itsf8e47ac6, before this PR's docs commit), on GLM-5.3-Flash on CUDA: prompt_logprobs on /v1/completions #1016, Zig 0.17.0, CUDA 13.3, NCCL 2.30.7, DGX Spark (GB10, sm_121), one GPU a host over RoCE, checkpointnvidia/GLM-5.3-Flash-NVFP4at423acf37,libtfgrammarfrom xgrammar v0.2.8 (zig build grammar); TP=2 with--parallel 2 --context 131072.{"ok": true}) gives the same text and the sametoken_shaunder the grammar, so the masked whole-row pick equals the per-rank merge. Ajson_schemarequest gives the same text with"draft": false, the same text beside a plain request (two streams, the plain one keeping itstoken_sha), and at temperature 0.7 with a seed the same text twice; thinking on, the JSON follows the thinking.tool_choice: "required"(with the next PR) gives oneget_weathercall with{"city":"Osaka"}andfinish_reasontool_calls.token_sha09a15f0582e7before and after); a constrained draw adds one gather of its rows.38aa9875against GLM-5.3-Flash on CUDA: prompt_logprobs on /v1/completions #1016's head (5a6ad9ed) with fresh caches:zig build test-cuda-host855/855 (854/854 at the base; the new test),zig build test787/794 with 7 skipped (767/774 at the base, 0 failures in both),zig build,zig build native,zig build grammar,zig fmt --check zig/src/families/glm,git diff --check, lean: 517 problems, 517 on the base, 0 new.--context 1048576, one stream: the same checks pass, with TP=2's texts andtoken_shavalues for the schema, same-as-plain,"draft": false, seeded sampling,tool_choice: "required"and plain checks (the thinking-on reply is valid JSON with another population figure).Checklist
"draft": false; concurrent equals solo (two streams).tf_grammar_mask,tf_drawandtf_glm_top1are existing ones).