Repository navigation
server: GLM tool calls stream their arguments, and one-call streams match the whole reply (upstream #1975, #1396) - #2
Merged
Merged
Conversation
… MLP, group counters through grownCounters, the replacements in the kernel contract - The 27B's fixed plans and exl3-plan-f64 are gone: every projection takes plan()'s launch (Weight.sk/wk stay). - reserveGroups runs after the layer's MLP is set; a host test checks a dense layer's gate/up group decodes 1-16 rows without allocating. - A group's counters come from main's grownCounters (ashhart#1475) with its members' summed bytes; scratch() never zeroes. - contract_exl3.zig registers the mid-M kernels, linear_wc, the groups and rot_group in gates 1-4 as byte-equal replacements of linear_body (rot_in), each with a planted control; host tests for groupable, wcPick and planOf.
… in the kernel contract - contract_exl3 registers the mid-M kernels at 3 and 5 bits too, each with a control, and its served shapes reach every form plan() gives a width: both row passes, 4 and 8 warps, split and unsplit K, every linear_wc tile. The groups gain 4 warps unsplit. - exl3.pick is matmul's dispatch alone. A host test sweeps K and N to 16384 through it and groupable, and fails on a form no registration's gate-3 shapes run. A family's fixed plan keeps the linear. - Gate 4's stress weights are all-ones states, one weight at every width (0xaaaa decodes to two at 3 and 5 bits).
…ig stays under 600 lines on main ashhart#1475's grownCounters and its test took exl3.zig past the lean check's 600 lines; the variant tables, wcShared, wcPick and mpgIndex move as they are (exl3.zig re-exports what other files name).
…s.zig stays under 600 lines
…ations and the linear
The drafter's trimmed-vocabulary head runs through the target's quant kernels, whose lane pipelines compile from source on first use. Its shape is unique, so it compiled inside the first draft round after a prompt. Backend.init now asks Kernels.prepareDecode for that pipeline; the lookup moves into laneEntry with the same key, so every later launch selects the same pipeline as before.
Exercise the actual preparation, cache lookup and lane encoder with a recording Metal boundary. Check that short and long target prompts leave first head launches on the prepared decode entry, while long prompt projections retain their own key and use the entry's row capacity. Cover preparation rollback, reuse and no-op guards, and run the regression in the default CPU test graph as well as test-qwen27-quant-host. The supplied head preparation already preserves main's long key and max_rows dispatch; keep that production change intact. Co-authored-by: kieranscotney <346361909+kieranscotney@users.noreply.github.com>
… and running replies TENSORFOLD_PREFILL_ORDER=srpt: before each long prompt step the lane host compares each running reply's estimated time left (its token budget over measured tokens a round and round time) with the prompt's (rows left over measured time a row). When a reply has less left, the step owes the replies nine times its time, so the prompt keeps at least a tenth. Unset, nothing changes.
… first, at most the floor otherwise; 0 stays whole)
Adds starlight/, a Go module with three binaries: starlight (gateway, control plane, admin console, SSO, SCIM and metrics), starlight-node (enrols a machine and supervises its engines and rank groups) and tensorfold-connect (desktop sign-in and a local endpoint for tools). The engine is untouched: Starlight drives tensorfold serve through its documented flags, key files and routes. .github/workflows/starlight.yml tests the module and runs only for changes under starlight/. The README points to it.
hashicorp/mdns keeps updating an entry it has already sent as more answers arrive, so reading entries while the query ran raced (seen on Linux CI). A test sends an entry and keeps changing it.
An Admitter built without a Waker made one on first use, unguarded, so concurrent admissions raced on the field (seen on Linux CI). The server always passes one; the fallback now runs under sync.Once, and a test admits concurrently on a fresh Admitter.
…seHeader, so bytes match each shape
…LASHNEXT_ATTN_LANES (ashhart#1739) Squashed from ashhart#1739's branch, which merges main in; its eight commits replayed onto main conflicted where the merge had already resolved them. The tree is the PR merged with current main.
ashhart#1739 moved fn_attn_merge's loop into fn_attn_merge_body, which the kernel calls; the Qwen4 CUDA attention host instrument extracted only the kernel, so test-qwen4-host, test-qwen4-gdn-host and test-qwen4-projection-bodies stopped compiling. Extract the body too, as inline host code.
…; nvfp4_slice.shapes checks without reading bytes
…s shards open instead of reparsing them (ashhart#1873)
…6 activation dtype docs/api.md's Decisions section named the MLX server and CUDA GLM, but only Flash Next on one CUDA GPU, LFM2 on CUDA and Gemma 4 on XPU score today; every other engine returns HTTP 400. glm/config.zig now refuses tensorfold_activation_dtype unless it is absent or bfloat16, naming the key and the value. A float32 checkpoint was silently served in bf16 before. The host test parses the real config absent and with bfloat16, and refuses float32 and float16.
core/rope_freq.zig computes 1 / f32(pow64(theta, f32(2i)/f32(dim))), the transformers words, and its proportional form gives a prefix of pairs then zeros. lfm3's Rotary.inverseFrequencies keeps main's () [64]f32 signature and now calls proportional; spark25's inverseFrequencies and the GLM vision tower's rotary call the helper directly, keeping their output slices and zero tails. Host tests compare each copy's old expression with the new words word for word; finite values cannot move. The Metal core root exports rope_freq for the tower and Spark2.5's Metal build. D237's rebase keeps main's proportional mode (Gemma 4 CUDA), drops the duplicate rope_freq.zig and core/root.zig hunk, and puts the GLM old-against-new test in the helper's own test file.
Under the share the prompt cache holds pool pages, so a balanced idle pool logged (UNEVEN): Pool.idle() already checks free + kept == total, free == total holds only for the copy store. Spark 25's form from 004655c.
…ing the race The concurrent writers-and-readers test read the table straight after each save and failed on a null load. On Windows a load can miss there: std.Io's Atomic.replace documents a window in which operations on the destination return AccessDenied while a replacement settles, and the cache's Win32 reader returns a pending or refused open as a silent miss, which cost_cache_windows_test.zig already requires. A miss is not a torn table, but loadAt's null could not tell the two apart, so the test treated both as failures and flaked on windows-latest. readAt now reports missing, refused or the value; loadAt keeps its results on every platform. The race counts each read: no refused or incomplete table anywhere, no miss off Windows, and one whole table once the writers have joined. Two planted races make it deterministic: a replace window that refuses every other open after a thread's own rename (0/300 pass with the old checks, 300/300 now) and a publisher that renames half-written tables (every read refused). Receipt (macOS arm64): the unplanted test passes 300/300 before and after; zig build test fails only the HTTPS-tunnel test (2 steps, same test); zig build dev-tools passes; zig fmt --check is clean; lean has 424 problems on both base and head, 0 new. Both test binaries compile for x86_64-windows-gnu with the host retargeted; they were not run on Windows.
…aders with core Loader.addShard took its allocation from the length word and indexed the JSON without checks, so a malformed header panicked. It now reads the length with headerSize and the header with parseHeaderOptions, leaving out a dtype or rank core does not name, as before; offsets are checked against the file and the fd is closed when the header is refused. xpu/safetensors.zig parses with parseHeader and keeps its error names, Tensor and the header's dtype spelling.
mv.metal runs the router, indexer, compressor and head matvecs as lane chains over K with one butterfly, so the bits never depend on the row count. glue_tmp.metal is the layer's phase-1 glue (norms, act_quant, RoPE, FP4, pools, candidates, hc and MoE glue), the reference bits the registered glue kernels are checked against.
…und trips, embed, join, mean) glue.metal holds the layer's small launches in one stated order each: RMSNorm with fp32 or bf16 weights, RoPE, act_quant and fp4_act_quant round trips, the embedding, the MoE join and the hc mean. Registered with the contract against float64, and bit-equal to the phase-1 glue.
…I root never reached
index.metal scores FP4 queries against FP4 keys in one stated order with causal and block masks, topk512.metal keeps each row's top 512, cand.metal picks the candidate blocks as a byte mask, and route.metal takes the top-k of sqrt(softplus(logit)) + bias for the backbone and the drafter. Each is registered with the contract.
compress.metal pools the ratio-2 and ratio-4 compressed KV and round-trips it through FP4; engram.metal dequantises the FP8 Engram rows, projects them, gates them and adds them to the residual. Registered with the contract against float64 and bit-equal to the phase-1 glue where they replace it.
…lded decode glue hc.metal runs hc_mixes with Sinkhorn, hc_pre and hc_post in fixed orders; fused.metal and fold.metal split or fold neighbouring glue launches at the same bits. Registered with the contract against float64 and bit-equal to the phase-1 glue.
… by stage against float64 The Metal package over the family's neutral host files: the kernels' pipelines, release weights (copied or mapped from the shards), a slot's KV regions, RoPE tables, scratch and one layer's encoders (attention with the indexer and compressor, MoE, hyper-connections). test-dsv41-metal checks every stage of every layer kind against float64 and rows at once against one at a time; test-dsv41-metal-real runs one release layer under TF_DSV41_REAL.
…d the DSpark taps forward.zig runs the stack for 1-16 rows of one or several slots in one launch: Engram before its layers, the hc mean into the taps, the head. model_weights loads a whole checkpoint, its shards mapped zero-copy where aligned. The tiny model's stages are checked against float64 and R rows in a launch against one at a time.
The prompt runs the encoder layers over every row in chunks, then the decoder's suffix cone: layer L of the decoder computes only the rows the prompt's end depends on (kept + (39 - L) x 127), with the bits of running every row. ced gains the cone's rules and the snapshot replay; tests check chunk sizes, the cone against every row, resumes and the grouped experts against the decode rows.
The drafter on Metal: its weights, the ingest of committed rows, the propose launches for several slots at once and the pass's plan. Its stages are checked against float64, R rows ingested at once against one at a time, and several slots in one pass against each alone.
serve_lanes.Model is what a backend gives the lanes (target, pass, shape, costs, hooks) and Served runs it under LaneHost; calib_lanes holds the draft-cost tables. The CUDA engine (serve_engine) is unchanged and keeps its own lane half; the Metal backend in the next part is the first to serve through this module.
With parallel_tool_calls false, a streamed reply now carries the same content and finish reason as the whole reply in the cases ashhart#1396 lists that still reproduced on main: - item 1: a rejected whole envelope in any case or NAME: spelling (<foo:tool_call>, <TOOL_CALL>, <Tool_Call>) stayed in the whole reply's content but was dropped from the stream. The one-call text filter now holds the prose after a hidden closed envelope, and the finish sends the reply's own content past what streamed. - item 2: the stream's end-of-reply prose took the cut-call branch in one-call mode, which the whole reply never does, so a rejected envelope before a cut call and the cut prose's trailing newline were lost. endProse now uses finishedProse in one-call mode. - item 4 already agrees on main; it is kept as a regression case, and the trailing-newline variant that did differ in one-call mode is fixed by the finishedProse/kept change. tool_single_stream_test.zig drives every case whole and streamed, one byte a round and batched.
…hhart#1975) A GLM call (<tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value>...) used to abort the streamer and go out in one piece at the finish. Its header now goes out once the name is whole and names an offered tool, then its string values as JSON-escaped pieces (holding back only a partial </arg_value>) and its typed values as compact JSON at their closer. tool_glm_stream.zig mirrors tool_parse.glmCall's reading, valueKey and repairGlmKeys included, over resumed indexes, so the joined pieces equal the whole reply's arguments byte for byte. DSML calls are unchanged. Two stream/whole differences found on the way, same bar: - after a call had streamed, an envelope the streamer could not stream set it to abort, so every later call was missing from the stream; it now parses that envelope at its closer instead, and aborts only while nothing streamed. - typed XML values streamed with Python's ", " separators while the whole reply's arguments are compact.
Owner
Author
|
The same work is now proposed upstream as two separate pull requests, one per issue: ashhart#2049 (issue ashhart#1396, one-call stream parity) and ashhart#2050 (issue ashhart#1975, GLM argument streaming), both rebased onto upstream main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streamed chat replies with tools now match the whole (non-streamed) reply in more cases, and GLM tool calls stream their arguments while the model writes them. The work comes from upstream ashhart/TensorFold issues ashhart#1975 and ashhart#1396. I picked those two after going through all 121 open upstream issues and keeping the ones a single DGX Spark can test. Every check below ran on that Spark (GB10, aarch64, Ubuntu 24.04, Zig 0.17.0).
What this changes
server: one-call streams send the whole reply's contentparallel_tool_calls: false, the stream dropped a rejected envelope such as<foo:tool_call>,<TOOL_CALL>or<Tool_Call>that the whole reply keeps in its content. It also dropped a rejected envelope before amax_tokenscut, and the cut prose's trailing newline.server: GLM tool calls stream their arguments as they are written<tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value>…) aborted the streamer, so the whole call went out in one piece at the finish.</arg_value>, and typed values go out as compact JSON at their closer. The joined pieces equal the whole reply's arguments byte for byte.<function=other>, a GLM or JSON payload,<|tool_call>,<NAME:tool_call>) aborted it, so every later call was missing from the stream. The whole reply still returned them. D1/D2: typed XML values streamed as["a", "b"]while the whole reply sent["a","b"].Nothing changes for requests without tools, for whole replies' content, or for the model's tokens. These are parser and stream changes only, so no inference path is affected.
DSML (the other half of ashhart#1975) is left as it is, on purpose. Streaming a DSML invoke before its block closes would break the existing containment contract in
stream_dsml_test.zig("unfinished and malformed DSML never emit their complete nested call"): an unclosed block, or a closed one whose invoke never closes, must stream no call. DSML blocks also parse all or nothing (tool_dsml.calls): one malformed or unoffered invoke rejects the whole block, so a call already streamed from an earlier invoke can't be taken back. That trade-off is a maintainer decision. Upstream also has its own DSML streamer for DeepSeek-V4.1's served engine (zig/src/families/deepseek_v41/serve/tool_stream.zig), which is related.maincfcbecb0b4691ddf971ebfb56140e9b3478e8e34.atc overlap --json(baseupstream/main): 14 open upstream PRs share files. 9 merge clean, 4 couldn't be tested (pullFilesUnavailable), and 1 conflicts: upstream PR dsv41: serve structured output with the lane core's grammar masks ashhart/TensorFold#1665 (dsv41 structured output) inzig/src/server/chat.zig. Both PRs touch the sameGenerationlines, which is close to the file's 600-line limit. The fix is a small rebase and I haven't attempted it here because this PR targets the fork.atc board,atc overlap) and posted nothing. Claim Streamed chat: GLM and DSML tool calls arrive in one piece, only XML calls stream their arguments ashhart/TensorFold#1975 and Streamed chat: four more cases where the stream and the whole reply disagree (single-call envelopes, max_tokens cuts, DSML whitespace) ashhart/TensorFold#1396 (atc claim) before proposing these commits upstream. Streamed chat: four more cases where the stream and the whole reply disagree (single-call envelopes, max_tokens cuts, DSML whitespace) ashhart/TensorFold#1396 had an earlier claim, a release and a takeover offer, and ATC showed no live claim on 3 Nov.Steps taken
main(Python 0.3.4.1,bb4b4a358). The fork has no unique commits, and itsmainis an ancestor ofupstream/main. I resetcard/t_14cd2239toupstream/maincfcbecb0b, following the fork rule "a fix that still matters is re-made on a fresh branch fromupstream/main", and read upstream'sAGENTS.md,CONTRIBUTING.md,docs/atc/SKILL.mdanddocs/recipes/SKILL.md.atc --repo ashhart/TensorFold --json, 3 Nov 11:40 UTC), then classified each issue by the hardware its check needs. The full table is at the end. 33 can be tested on one Spark, 46 only partly, and 42 not at all (TP=2/3/4, Metal, HIP, XPU, Vulkan, RTX-only). Of the 33, 13 have no claim and no open PR. I picked the two that (a) are free, (b) are host-testable, so they don't need the Spark's GPU, which is serving a model I must not stop (docs/recipes/SKILL.md: "Do not stop somebody else's server"), and (c) have a bar the maintainer already set ("stream and whole reply send the same content, arguments and finish reason in every case"). The other free ones need a model loaded on the busy GPU (Flash Next CUDA on GB10: NV_ERR_NO_MEMORY in request state/KV allocations on v1.2.0 and main ashhart/TensorFold#1856, Living Weights: the Qwen3.8-27B CUDA learner's gradient is 1.6-1.8x too large at the last prompt row ashhart/TensorFold#1806, Living Weights on Qwen3.6-35B-A3B on CUDA ashhart/TensorFold#1802, Living Weights: conflicting facts, updates and forgetting (measurements + proposal) ashhart/TensorFold#678, Living Weights: unrelated-reply drift and recall after lessons (follow-up to #646) ashhart/TensorFold#1389), nvcc 13.2 where this Spark has 13.0 (RFC: qualify nvcc 13.2 for the release fatbins (native FP4 decode, prompt16 −4.5 %, bit-equal) ashhart/TensorFold#1448), the moss_image engine (moss_image: serve pictures over POST /v1/images/generations ashhart/TensorFold#1573), or a maintainer decision first (RFC: --profile stable | fast-exact | fast, one table for the opt-in switches ashhart/TensorFold#1660, RFC: one load-time fit for every affine-4 draft head, weighted by diag(H) (Flash-Next: affine-4 head near bf16's acceptance, 7.8 % less time) ashhart/TensorFold#1870, glm: the released Python engine renders the--no-thinkingprompt 6-7 tokens shorter than the checkpoint template; native renders it verbatim ashhart/TensorFold#1911, Studio: carry #941 to #944 onto main as reviewed increments ashhart/TensorFold#1204).cfcbecb0binto a scratch directory on the Spark and used the Zig 0.17.0 already installed there (zig version→0.17.0). The local tree was rsynced over for every run. Nothing on the Spark was installed, stopped or reconfigured.main; results are in the table below.openai.runwith a fake byte-token engine, batched and one character per round) for every case in Streamed chat: four more cases where the stream and the whole reply disagree (single-call envelopes, max_tokens cuts, DSML whitespace) ashhart/TensorFold#1396 and some suspects:Writecontent fed 3 bytes at a time) gave GLMheaders 0, argument deltas 0, state abort.tool_glm_stream.zig(tool_stream.zigwould otherwise pass 600 lines), and the one-callTextFilter/endProse/keptfix. Each new test was confirmed red on the old code first.</arg_key>unconditionally, whilerepairGlmKeysrepairs only when closers outnumber openers, so the stream could invent an argument. The counter-example<tool_call>lookup<arg_key>a<arg_key>b</arg_key><arg_value>1</arg_value> w</arg_key><arg_value>2</arg_value></tool_call>is now a test that failed without the guard. The fix: repair only when the counts seen so far sayrepairGlmKeyswould, with the counts carried incrementally so it stays linear. Two simplifications were applied too (oneglmStephelper, onesendValuehelper).Checks on this head
Head commit:
3fda03c56(parent6c30fd845, basecfcbecb0b). Machine: DGX Spark GB10 (sm_121), Linux 7.0.0-1019-nvidia aarch64, Zig 0.17.0, Python 3.12.3. Each run used an empty local.zig-cache. The Spark's GPU stayed on its existing server for the whole run and these steps don't use it..github/workflows/no-gpu.yml)cfcbecb0bzig build test --test-timeout 10mzig build test-kernel-contractzig build test-goldenzig build test-server-security test-pull-securityzig build fuzz-cortheonzig build test-cuda-host(CUDA host subset)zig build(plain, every target)zig build test-server-cpu(server unit tests)zig build test-tool-parse-host test-tool-visibility-http-host test-tool-required-host test-goldenzig fmt --check build.zig zigpython3 -m unittest discover -s tools/tests -p "test_bench*.py"python3 -m unittest discover -s tools/zig -p flashnext_batch_serve_test.pypython3 -m unittest discover -s tools/zig -p build_options_test.pytools/zig/lean_check.py+lean_new.pygit diff --check¹ The
test-cuda-hosttotals count only run steps that executed in that invocation. Some of them share runs with the precedingteststep, so the count moves between invocations. Every skip on both sides is an environment skip:libjpeg-turbo 3.1.4.1 not found/libwebp 1.6.0 not foundpixel cases. There were 0 failures on both sides.New tests: 12 total, each red before its fix except the regression-only cases noted.
zig/src/server/tool_glm_stream_test.zig(8 tests):<in the name) send nothing;closeCallpath;tool_glm_cut;max_tokenscut inside a value, a zero-argument call, a key without a value.zig/src/server/tool_single_stream_test.zig(4 tests), every Streamed chat: four more cases where the stream and the whole reply disagree (single-call envelopes, max_tokens cuts, DSML whitespace) ashhart/TensorFold#1396 case whole vs streamed in one-call mode, batched and per character: 1a–1d, 2a–2c, 4a/4b with and without a trailing newline in both modes, and an accepted call whose prose streams before it.ashhart#1975 probe (2,000-character
Writecontent, fed 3 bytes at a time):abort, the call goes out whole at the finish)Not run, and why:
unicodedata16.0.0; the Spark has 3.12, and this change doesn't touch the tables or their generator./v1/messagesend-to-end test. Anthropic'sinput_json_deltais built from the same argument deltas (anthropic.zigdrain), so GLM calls now produce many of them instead of one.Receipt
n/a for inference. The change is in the HTTP server's tool-call parsing and streaming and never reaches the model or its tokens.
"draft": false, resume and concurrency exactness are unaffected for the same reason.Known limitations
These cases are left as they are, documented so a maintainer can decide.
</tool_call>written inside a GLM<arg_value>isn't special-cased. The XML path has the same limitation: the whole reply's parser ends the envelope there, while the stream has already sent part of the value.<arg_key>K</arg_key>with no value yet),closeCall's strict reading appends no closers. The stream keeps the call it opened and the whole reply returns none. XML has the same case for an end token inside<parameter=.repairGlmKeysdecides over the whole payload: it gives up on all repairs at a wordless lone</arg_key>, and repairs only when closers outnumber openers. A malformed call with two problems, where later text changes that decision, can still differ. The single-problem case is guarded and tested. This is noted in the module header.net: -13 lines possible): droptypedSchemaintool_stream.zig; addkeyAt/pendingTaghelpers intool_glm_stream.zig; set.countedin theGlmreset and drop the clamp.Reviewed by GLM-5.3 (2 rounds: round 1 changes needed, fixed; round 2 approve).
Checklist
All 121 open upstream issues, by whether one DGX Spark can test them (ATC board snapshot, 3 Nov 2026)