Repository navigation
Conversation
The lane backend fills Stream.prompt_rows, so the server stops refusing prompt_logprobs for GLM. A prompt chunk's rows go through each rank's head columns 128 at a time; the rows are made whole on every rank with the broadcasts a sampled draw already uses (Sampler.gather), and rank 0 runs sample.cu's tf_logprob_rows on them. Rank 0's prefill and chunk steps carry flag 1 so the other ranks join each pass. The pass's buffers are made at the first request that asks; if a rank cannot make them, every rank takes the verify window's buffers, 8 rows a pass. A pass with prompt rows starts from position 0, as Nemotron's does.
The lane backend gives first_masked and held, so the server serves response_format and required tool calls for GLM. Only rank 0's lane core holds the grammar, and only rank 0's tokens are read (it sends them on), so rank 0 alone masks: a verify or shared step under a grammar carries flag 1, every rank gathers its draw's rows whole (Sampler.gather, as a sampled draw does), and rank 0 runs sample.cu's tf_grammar_mask on them, then tf_draw, or greedy's rule (tf_glm_top1 over the whole row: the largest value, then the lowest id, as the per-rank merge picks). The first token is drawn again from the prompt pass's last head row by a new redraw step. The grammar rows go to host-mapped words before the step is sent, so a refusal never strands the other ranks mid-broadcast. Draft draws stay unmasked.
tool_choice "required" and a named function refused every GLM template: its calls are <tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value> ...</tool_call>, neither Qwen3-Coder's XML nor a JSON object. The probe call's markup now picks a third style, and the call's grammar is the name after <tool_call>, the arguments in xgrammar v0.2.8's glm_xml (the same tags; a string value written as it is, others as JSON, as the template writes them), then </tool_call>, with no blanks between, as the template has none.
Owner
|
Thanks @Bizuayeu for the required Posted by Ash's AI Assistant Synthena |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Part of GLM-5.3-Flash on the native CUDA engine (#593, where the three stacked PRs for GLM's ranks are summed up): a forced tool call whose arguments are inside the tool's schema.
tool_choice: "required"and a named function were refused for every GLM template: its calls are<tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value>...</tool_call>, neither Qwen3-Coder's XML nor a JSON object, sotool_grammar.stylefound no style. The probe call's markup now picks a third style,glm, and a call's grammar is<tool_call>with the name, the arguments in xgrammar v0.2.8'sglm_xmlstyle (the same tags), then</tool_call>, with no blanks inside, as the template writes none. Stacked on #1017, which makes GLM enforce a grammar on CUDA.Receipt
f8e47ac6on GLM-5.3-Flash on CUDA: structured output (first_masked and held) #1017, built and tested as38aa9875(the same code, before the docs commits beneath it), the same machines, checkpoint andlibtfgrammaras there; TP=2 with--parallel 2 --context 131072.tool_choice: "required"with one offered tool (get_weather, a requiredcitystring) and the prompt "What is the weather in Osaka?" gives one call,{"city":"Osaka"},finish_reasontool_calls, parsed by the server's GLM call parser.zig build test-cuda-host855/855 andzig build test(0 failures) at38aa9875; the style and grammar tests inzig/src/server/tool_grammar.zigcover the GLM markup.--context 1048576, one stream: the required call isget_weather({"city":"Osaka"})again,finish_reasontool_calls.parallel_tool_callswith several calls in one reply; Metal (GLM's Metal engine does not enforce grammars).Checklist
"draft": falseunder the grammar (GLM-5.3-Flash on CUDA: structured output (first_masked and held) #1017's receipt); this change only builds the grammar.glmstyle, and its grammar names the tool after<tool_call>with theglm_xmlstyle. Before this change that markup had no style, which is the refusal above.