Repository navigation
Python engine features to carry into the native engine #840
Description
These features shipped in the Python engine, from 0.2.0 through 0.6.6, and the people who wrote them are a big part of why it worked as well as it did. Thank you. The native engine should carry each of them, with your name on it.
A comment on a row claims it. If a row stays unclaimed, the maintainers port it and put your Co-authored-by line on the commit. docs/credits.md names the Python work each of these rows carries.
return_token_ids(feat: read EXL3 checkpoints on CUDA (any codebook, 1 to 8 bits, mixed-K packs) #42, @vcruz305).zig/src/server/chat.zig. The ids equal the reply's tokens and its token SHA.- End ids from
generation_config.json(fix: Qwen3.8 Flash Next and 27B on EXL3 checkpoints stop at the end of the chat turn #69, @vcruz305).zig/src/server/hf_text.zig. An EXL3 pack stops on its own end id. Other checkpoints' end ids stay as they are. --name-priority ID=background(feat(cuda): --name-priority ID=background, a default priority by the model id a request asks for #445, @philip-pentatonic).zig/src/server/cli.zig. Replies are unchanged. Background requests wait as 0.6.6's do.--mtp-confidence(feat: a quantized KV cache for Flash Next on CUDA (--kv-dtype int8 or int4) #47, @vcruz305).zig/src/families/nemotron/cuda_drafts.zig. Drafts only propose, so replies are unchanged. Tok/s at least 0.6.5's.TF_GLM_CLEAR_THINKING(fix(server, glm): chat_template_kwargs.thinking heard as enable_thinking; GLM-5.3 keeps earlier turns' reasoning (clear_thinking false) #236, @MiaAI-Lab).zig/src/server/hf_text.zig. GLM-5.3-Flash is served. The first-revision template keeps earlier reasoning unless this is set.tensorfold plan(feat(cli): tensorfold plan — the budget arithmetic before weights load #281, @akol1).zig/src/cli/cli.zig. The verdict equals the serve startup budget for each model and RAM class.--decode-share(qwen4_exp CUDA: feed row_s from stand-alone passes so --decode-share sizes them #248, @simon-lin88).zig/src/core/lane_host.zig. While a prompt fills, replies keep moving. The PR measured about 1.33 s to about 111 ms on Flash Next NVFP4.- Spill of evicted states, and
--checkpoint-slotson Macs (feat: spill evicted conversation prefixes to disk, and --checkpoint-slots #68, @gilby).zig/src/core/prompt_cache.zig. A state read back from disk resumes exactly as fresh. /v1/decisionsscores (feat: /v1/decisions scores choice, score, and yes/no from next-token logits #127, @mikolaj92, and Flash Next CUDA: /v1/decisions scores labels from the prompt's last logits, the questions filling in one pass #232, @Mirrdhyn).zig/src/core/engine_api.zig. Label scores equal 0.6.6's. A second decision on a 6,753-token shared instruction, 20.4 s to 2.05 s on an M3 Ultra.- GLM fp32 activations (glm5_next: opt-in float32 activations (tensorfold_activation_dtype) #118, @feni6).
zig/src/families/glm/forward.zig. An opt-in setting: more precise, and slower. - Ampere sm_80 (build: admit Ampere (sm_80) via TENSORFOLD_MIN_CAPABILITY, keep NVFP4/cluster floors #464, @liumorrisclaw).
zig/build/cuda.zig. Kernel checks pass on sm_80. The README line that the A100 is not supported yet changes, and FP8 paths trap below sm_89.
There is no pressure on timing. If you would like to bring yours across, claim it here and we will review it with you; either way, your name goes on it.
Activity
liumorrisclaw commentedon Oct 14, 2026
Claiming the Ampere sm_80 row (originally #464). @ashhart — I'll port it into
zig/build/cuda.zig: kernel checks passing on sm_80, the README's "A100 not supported yet" line updated, and FP8 paths trapping below sm_89. PR to follow.liumorrisclaw commentedon Oct 14, 2026
The sm_80 row is ported: #846 carries it into the native engine (release fatbins, README, and the ModelOpt floor's refusal by name). Review away.
ashhart commentedon Oct 14, 2026
Hello @liumorrisclaw,
Thank you for claiming the Ampere sm_80 row, and for #846 so soon after.
#846's device gate on a GA100 meets the row's bar: all 34 kernels build for compute_80, every release fatbin carries the sm_80 image, and the kernel checks pass on the card. The README line changes, and the ModelOpt floor now names its refusal. One docs line is left (packaging/README.md, noted on #846). The row moves when #846 lands, with your name on the commit.
The other rows are still unclaimed.
Thank you again.
CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor na…
CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor na…
CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor na…
CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor na…
Boscoeuk commentedon Oct 24, 2026
Claiming the
TF_GLM_CLEAR_THINKINGrow (#236, @MiaAI-Lab's), with @MiaAI-Lab's Co-authored-by line on the commit.On main (
6244c620) the native server gives a template onlyenable_thinkingandreasoning_effort(zig/src/server/hf_text.zig:352), so GLM-5.3'sclear_thinkingis never set: neitherTF_GLM_CLEAR_THINKING=1nor a request'schat_template_kwargs.clear_thinkingreaches it.The port:
TF_GLM_CLEAR_THINKING=1setsclear_thinking: truefor the server, a request'schat_template_kwargs.clear_thinking(a bool) wins over it, and without either the prompt renders byte for byte as it does now. Tests will render the GLM-5.3 template both ways. I'll check it on a Mac againstGLM-5.3-Flash-MLX-4bit-MTPwith a two-turn conversation, and post the result in the PR.javierbits commentedon Oct 25, 2026
Claiming the
return_token_idsrow (#42, @vcruz305's), with @vcruz305's Co-authored-by line on the commit.On main (
faccd11c) the native server already acceptsreturn_token_idsand passes it through (zig/src/server/responses_translate.zig), but nothing reads it: the response'stensorfoldblock carriestoken_sha/token_sha256and no ids.The port, in
zig/src/server/chat.zig: the request flag ridesPrepared/Generationlike the other response-side options, and when it is set the runtime block gainstoken_ids— the reply's tokens, the very slicetokenSha/tokenSha256hash, so the ids and the sha cannot disagree. Both response paths already stamptensorfold(the whole body and the streaming final chunk), so streamed replies pick the ids up with no streamer change, and without the flag replies are byte for byte as they are now.Tests: a host test drives the scripted engine, asserts
token_idsequals the scripted tokens and re-hashes totoken_sha256, with the control that notoken_idsappears without the flag. Then a GB10 CUDA run against a served checkpoint, posted in the PR.atc check 840reportedclaimed: false, claimants: 0before this comment.javierbits commentedon Oct 25, 2026
The
return_token_idsrow is ported: #1290, with the host suite and the GB10 CUDA receipt (the pinnedmlx-community/Qwen3.5-2B-MLX-4bit@93760be4checkpoint, ids re-hashed against an independent SHA-256) in the description.jschmied commentedon Oct 25, 2026
Claiming the CUDA part of the "Spill of evicted states" row (#68, @gilby's), with @gilby's Co-authored-by line on the commits.
On main (
674fb9e05),--learn-conversationswrites kept states to disk on CUDA only for Nemotron (native/cuda_disk.zig). The CUDA qwen3_5 family (27B and 35B), Flash Next, GLM, Gemma 4, Dory and Kolibri-1 refuse it (cuda_refusal.zig:47).The plan is the CUDA qwen3_5 family first, then Flash Next, one PR each:
--checkpoint-slotson Macs is not part of this claim.server: return_token_ids puts the reply's tokens in the response's te…
server: GLM-5.3's template hears clear_thinking, from TF_GLM_CLEAR_TH…
javierbits commentedon Oct 25, 2026
Claiming the
--name-priority ID=backgroundrow (#445, @philip-pentatonic's), with @philip-pentatonic's Co-authored-by line on the commit.On main (
84e0d560) the native server has--alias,model_ids(served name first, then aliases) andapi.Request.background, but no per-name default:chat.preparecomputes background from the request's ownpriorityfield or a session title only. I also checked the end-ids row while picking:33a10ee0already carries it, so I left it alone.The port:
--name-priority ID=background(repeatable) parses inzig/src/server/cli.zigwith #445's refusals — anything butID=background(the word case- and whitespace-insensitive) and an id that is neither--namenor an--alias; the native server has no separate MLX line, so that one refusal does not apply. The ids rideServer.Config.background_ids, andchat.prepareapplies the default as #445's table does: a request with nopriorityof its own whose model id is named is background, an explicit priority wins, a session title is background either way. Replies are unchanged.Tests on the CPU fake-engine HTTP harness mirror #445's five-row table (named id without priority → background; other id → not; explicit background → background; explicit non-background on a named id → not; explicit background on a plain id → background), plus the parse and validation refusals.
zig build teston this Linux aarch64 box; the GB10 is not needed.javierbits commentedon Oct 25, 2026
Claiming the
tensorfold planrow (#281, @akol1's), with @akol1's Co-authored-by line on the commit.On main (
84e0d560) the native checkpoint CLI (zig/src/cli/cli.zig) offersmodels/info/pull/mergebut noplan: the budget arithmetic lives only inside the server's startup admission, which names the memory budget ("free less a reserve") just when it refuses. I'll port #281's shape —tensorfold plan MODEL [--memory-gb GIB] [--ram GB]...answering from config.json and the local checkpoint before any download: per-RAM-class budget = min(allowance × RAM, ceiling), room = budget − process reserve, fits/does-not-fit per class, exit 1 with the smallest-fitting budget named as a hint. #281 is Mac/MLX-only; on this host the same math maps onto the native server's admission budget (the reserve behind TENSORFOLD_MEMORY_RESERVE_GIB), so the row's bar holds: the verdict equalstensorfold serve's startup budget for each model and RAM class. Verification is host-only: the Zig test suite green, the lean gate clean, and a plan-vs-serve comparison on the local Qwen3.5-2B checkpoint — no GPU needed.13 remaining items
prompt cache: --spill-gib writes a state RAM evicts to disk, and --ch…
Changelog: --spill-gib and --checkpoint-slots on Macs (#840)
gilby commentedon Oct 26, 2026
Claiming the Flash Next shared-session part of the spill row (#68): #1349 extends 47f397f's
--spill-gibto an explicit--parallelabove one on Macs, which still refused it.--learnand--learn-conversations(write behind) stay refused there.lanes: --decode-share sizes a filling prompt's chunks beside live rep…
lanes: --decode-share sizes a filling prompt's chunks beside live rep…
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
ashhart commentedon Oct 28, 2026
Thanks @gilby. #1349 landed as 846c3e9 and shipped in 1.1.7, so Flash Next's shared sessions on Macs (an explicit
--parallelabove one) now spill evicted states with--spill-gibtoo.Where the rows stand on main, as of 1.1.8:
return_token_ids(server: return_token_ids carries the reply's tokens in the response's tensorfold block #1290) and--name-priority(server: --name-priority ID=background gives a served id a background default #1317), thanks @javierbits;TF_GLM_CLEAR_THINKING(server: GLM-5.3's template hears clear_thinking, from TF_GLM_CLEAR_THINKING or the request (#840) #1300), thanks @Boscoeuk;tensorfold plan(cli: tensorfold plan, the serve startup budget before weights load (#840) #1319, @javierbits, with @akol1's co-author line); and Ampere sm_80 (CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor named at its refusal #846, thanks @liumorrisclaw, in 1.0.12). Our 25 Oct list had sm_80 as still open, which was wrong: CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor named at its refusal #846 carried it.--checkpoint-slots(feat: spill evicted conversation prefixes to disk, and --checkpoint-slots #68): on Macs for Qwen3.8-27B, Nemotron, Flash Next and GLM-5.3-Flash (47f397f, with @gilby's co-author line, and flashnext: shared sessions spill evicted states to disk (--spill-gib) #1349), and on CUDA for Nemotron, the Qwen3.5 hybrids and Flash Next (cuda qwen3_5: --learn-conversations keeps the 27B's and 35B's states on disk (#840) #1358, cuda flashnext: --learn-conversations keeps Flash Next's states on disk (#840) #1359 and cuda: serve --spill-gib and --checkpoint-slots for Nemotron, the Qwen hybrids and Flash Next (#840, split from #1359) #1413, thanks @jschmied, in 1.1.8). Other CUDA families and HIP don't take--spill-gibyet.generation_config.json(fix: Qwen3.8 Flash Next and 27B on EXL3 checkpoints stop at the end of the chat turn #69) and--mtp-confidence(feat: a quantized KV cache for Flash Next on CUDA (--kv-dtype int8 or int4) #47).--decode-share(qwen4_exp CUDA: feed row_s from stand-alone passes so --decode-share sizes them #248), lanes: --decode-share sizes a filling prompt's chunks beside live replies (#840) #1327 by @javierbits./v1/decisionsscores (feat: /v1/decisions scores choice, score, and yes/no from next-token logits #127, Flash Next CUDA: /v1/decisions scores labels from the prompt's last logits, the questions filling in one pass #232) and GLM fp32 activations (glm5_next: opt-in float32 activations (tensorfold_activation_dtype) #118, which needs a Mac to verify).Thank you all for bringing these across.
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
lanes: gate --decode-share's fill cap on per-family chunk invariance (a…
lanes: gate --decode-share's fill cap on per-family chunk invariance (#…
ashhart commentedon Oct 31, 2026
#1327 has landed, thanks @javierbits, with @simon-lin88's co-author line. The native server now takes
--decode-share X(0 to under 1) instead of refusing it, on the CUDA, Metal and HIP lane servers: while a long promptfills beside running replies, the replies get X of each fill step's time. Without the flag nothing changes. The flag
can also cap a filling prompt's chunks at 512 rows, but that stays off for every model until it is shown to give the
same tokens at any cut.
Where the rows stand now:
return_token_ids(server: return_token_ids carries the reply's tokens in the response's tensorfold block #1290),--name-priority(server: --name-priority ID=background gives a served id a background default #1317),TF_GLM_CLEAR_THINKING(server: GLM-5.3's template hears clear_thinking, from TF_GLM_CLEAR_THINKING or the request (#840) #1300),tensorfold plan(cli: tensorfold plan, the serve startup budget before weights load (#840) #1319), Ampere sm_80 (CUDA: Ampere sm_80 in the release fatbins, with the ModelOpt floor named at its refusal #846), spill of evicted states and
--checkpoint-slots(feat: spill evicted conversation prefixes to disk, and --checkpoint-slots #68's row), and--decode-share(lanes: --decode-share sizes a filling prompt's chunks beside live replies (#840) #1327; the 512-row cap still needs each family's token check).
generation_config.json(fix: Qwen3.8 Flash Next and 27B on EXL3 checkpoints stop at the end of the chat turn #69) and--mtp-confidence(feat: a quantized KV cache for Flash Next on CUDA (--kv-dtype int8 or int4) #47)./v1/decisions(zig/src/server/decisions.zig). We'll check feat: /v1/decisions scores choice, score, and yes/no from next-token logits #127's and Flash Next CUDA: /v1/decisions scores labels from the prompt's last logits, the questions filling in one pass #232's scores against it and strike them if they match.