dsv41 Zig: load the EXL3 2.8 bpw pack (bf16 router weights and confidence head, one source a tensor) by Elevatormusic · Pull Request #38 · Elevatormusic/TensorFold

Elevatormusic · 2026-10-22T05:14:31Z

What this changes

The Zig dsv41 loader now loads the DeepSeek-V4.1-Flash-1017 EXL3 2.8 bpw pack. A header comparison against the 2.9 bpw pack found these differences:

Tensor	2.9 bpw	2.8 bpw	Zig before	Zig now
`ffn.gate.weight` (43)	F16	BF16	load fails (`UnsupportedConversion`)	narrowed to the f16 the router kernel reads
`ffn.gate.bias` (43)	F16	F32	read as f32	read as f32 (copied)
`attn.indexer.weights_proj.weight` (8)	F16	BF16	read as f32	read as f32 (exact widening)
`engram.q_weight` / `k_weight` (4)	F32	BF16	read as f32	read as f32 (exact widening)
`mtp.2.markov_head.head.weight`	F16	BF16	converted to f16 at load	converted to f16 at load
`mtp.2.confidence_head.proj.weight`	F16	BF16	uploaded as stored; the draft code reads it as f16, so the bf16 bytes would be misread	converted to f16 at load, like the markov head

Changes:

checkpoint.readFrom: a bf16 tensor under an f16 key is narrowed with narrowBfloat (round to nearest even, as the Python loader's .to(torch.float16) does).
weights.zig: markovHead is now halfAtLoad, used for the markov head and the confidence head. The cache key stays the checkpoint's own dtype, so no rank cache is invalidated.
Shards.open: a tensor name in two shards with another dtype or shape fails with error.DuplicateTensor. Example: the dtype-override file that the Python A/B view keeps next to the pack. The 1017 packs repeat the Engram q / k tables in two shards with the same dtype and shape (the 1017-headers fixture shows it). Such a pair loads as before: the later shard is used.

No change was necessary for these:

Expert bit widths. The 2.8 pack has routed experts at 2 to 5 bits and shared experts at 5 or 6 bits; the 2.9 pack has 2 to 3 and 4 to 5. The loader already keeps a width for each expert matrix, and the kit's experts_cb.cubin has the mma, rows and cp kernels for the ranges [2, 10], [2, 16] and [8, 8] (k2, half-bits).
The 11 plain BF16 tensors that only the 2.8 pack has (attn.compressor.wkv / wgate, attn.indexer.wk in layers 2, 8, 14 and 20). Their EXL3 forms are also in the pack, and both engines load the EXL3 form.
The kit and the token map. The 43 image routing biases in native.safetensors equal the kit's gate_bias_vl (F32, difference 0); the tokenizer and chat template are the same files.

Receipt

Environment: TensorFold dsv41-zig plus this branch, in dsv41-zig-dev @ 32894bad, image tensorfold-dsv41-zig:32894bad (the same image ID sha256:89812ac5… on both nodes). Zig 0.17.0. Two NVIDIA GB10 nodes at TP=2. DeepSeek-V4.1-Flash-1017 EXL3 2.8 bpw, from a view without the Python dtype-override file, and its own rank cache (built by tf-dsv41-rank-cache in 288 s).
Tests run: the new unit test in checkpoint.zig (bf16 router weights read as f16, f32 bias copied, a repeated name with the same dtype loads, a repeated name with another dtype fails); zig build test-cuda-host at 32894bad: rc 0. zig fmt --check on the changed files: passed. The first host-test run at 977e3487 failed one test: the strict duplicate rule rejected the 1017 fixture packs. That is what made the rule depend on dtype and shape.
Bring-up window on the GB10 pair, 2.8 pack:
- ready after 34 s; smoke test (17 × 23 = 391) passed;
- so_check 16/16;
- recipe suite 30 of 31 (the one fail is router/chat_tools_stream, which calls the LAN router, not this engine);
- grammar check 14/14 two times;
- 0 round faults; both ranks exit 0 at the stop.
Decode speed: kit_bench set a, 512 tokens, median of 3: 85.0 tok/s at 1 stream (84.9 to 85.2); concurrent set b, 4 streams, 384 tokens: 122.4 tok/s in all.

Memory (MemAvailable):

	Head	Worker
Idle after 120 s, Zig	6.43 GiB	6.47 GiB
Idle, Python 1.1.1 on the same pack (median)	6.47 GiB	6.80 GiB
Low point over the window (suite and 4-stream decode), Zig	5.34 GiB	5.69 GiB
Low point over long 4-stream runs, Python 1.1.1 on the same pack	4.46-4.87 GiB	4.78-5.49 GiB
Under 4 streams, Zig on the 2.9 pack (earlier soak)	about 1.7 GiB

Prompt speed (release campaign, window C, with dsv41: special-token text in request strings is plain text #39): cold 32K prefill, median of 3, Zig / Python 1.1.1 on the same pack: 2,161 / 2,004 tok/s (+7.8%), 2,242 / 2,066 (+8.5%), 2,487 / 2,268 (+9.7%) on three prompt sets; cached_tokens 0. Decode: 85.3 / 87.1 tok/s at 1 stream (97.9%), 123.0 / 123.3 at 4 streams (99.8%).
Release fault campaign on this pack (2026-10-22, images 32894bad and 2307b3f3): agentic tool-call load 505 requests, 0 lost; needle found 7/7 up to 1,038,664 tokens (low point head 3.25 GiB); prefix cache on and off pass; two 65-min soaks, about 2,500 requests each, 0 reference mismatches, head memory flat (4.4-4.7 GiB). Faults it found are in other PRs: F1 (dsv41: special-token text in request strings is plain text #39) and F3 (dsv41: a rank whose peer dies ends with exit 75 (peer link watch) #40).
The 2.9 pack still serves with this image (2307b3f3): ready after 26 s, smoke and so_check pass, both ranks exit 0.
Not run: Metal (dsv41 is CUDA only).

Checklist

Every token still goes through the lane rounds.
No precision traded for speed: the conversions match the Python loader's casts.
No GPU kernel change.
Prompt processing is no slower than the last release: not measured yet on this pack.
Platforms: CUDA on GB10 at TP=2. Metal not run.
New tests fail before the change (bf16 to f16 had no conversion), pass after it.
Comments and docstrings are one line (the CI lean check passes).

…idence head, one source a tensor) The 2.8 bpw pack holds ffn.gate.weight in bf16 (the 2.9 pack: f16). The reader now narrows a bf16 tensor to the f16 the router kernel reads, as the Python loader's .to(torch.float16) does. The DSpark confidence head is converted to f16 at load like the markov head, because the draft code reads it as f16. Shards.open fails on a tensor name that is in two shards (for example a dtype-override file next to the pack), so a view with two sources for a name cannot load silently.

…ffers The 1017 packs repeat the Engram q / k tables in two shards with the same dtype and shape (the 1017-headers fixture shows it). Such a pair loads as before: the later shard is used. A name in two shards with another dtype or shape (a dtype-override file next to the pack) still fails the load. The message is a warning, so a test that expects the failure does not log an error.

chatgpt-codex-connector · 2026-10-22T05:14:40Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-24T18:43:34.769618Z	`07f1204`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Elevatormusic · 2026-10-22T05:14:47Z

@codex review

Elevatormusic · 2026-10-22T07:25:30Z

@codex review

chatgpt-codex-connector · 2026-10-22T07:29:15Z

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: e027f415e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…very pull request) zig fmt 0.17.0 changes only layout and @intFromEnum to @backingInt. The Zig format check fails on these five base-branch files in every open dsv41-zig pull request, also where the pull request does not touch them.

…ments are one line (lean check)

Elevatormusic · 2026-10-24T18:42:05Z

@codex review

chatgpt-codex-connector · 2026-10-24T18:43:32Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 07f1204fa1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Elevatormusic · 2026-10-24T19:15:30Z

Merged on the Codex 👍 and green checks. Follow-up (needs a GPU window): measure prefill speed (cold 32K/128K TTFT and prefill tok/s) on the 2.8 bpw pack against the last release. Scheduled by the main session after the EAGLE chain.

Elevatormusic added 2 commits October 21, 2026 19:27

Elevatormusic added the type: feature new code (feature, tool, add-on) label Oct 22, 2026

Elevatormusic mentioned this pull request Oct 22, 2026

dsv41: special-token text in request strings is plain text #39

Merged

7 tasks done

Elevatormusic mentioned this pull request Oct 24, 2026

zig: zig fmt on five dsv41-zig base files (Zig format check fails on every PR) #43

Merged

9 tasks done

Elevatormusic added 2 commits October 24, 2026 11:28

dsv41: the 2.8 bpw pack test moves to checkpoint_test.zig; loader com…

88f5f36

…ments are one line (lean check)

Merge fix/zig-fmt-base (#43): zig fmt on five base files

07f1204

Elevatormusic merged commit 1759496 into dsv41-zig Oct 24, 2026
6 checks passed

Elevatormusic deleted the dsv41-zig-28pack branch October 24, 2026 19:14

Navigation Menu

Repository navigation

dsv41 Zig: load the EXL3 2.8 bpw pack (bf16 router weights and confidence head, one source a tensor) - #38

dsv41 Zig: load the EXL3 2.8 bpw pack (bf16 router weights and confidence head, one source a tensor)#38
Elevatormusic merged 5 commits into
dsv41-zigfrom
dsv41-zig-28pack

Elevatormusic commented Oct 22, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Elevatormusic commented Oct 22, 2026

Uh oh!

Elevatormusic commented Oct 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 22, 2026

Uh oh!

Elevatormusic commented Oct 24, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 24, 2026

Uh oh!

Uh oh!

Elevatormusic commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Elevatormusic commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Receipt

Checklist

Uh oh!

chatgpt-codex-connector Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Elevatormusic commented Oct 22, 2026

Uh oh!

Elevatormusic commented Oct 22, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 22, 2026

Uh oh!

Elevatormusic commented Oct 24, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 24, 2026

Uh oh!

Uh oh!

Elevatormusic commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Elevatormusic commented Oct 22, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 22, 2026 •

edited

Loading