[gilby]

What this changes

Main's deepseek_v41 family reads the release checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 as published: FP8 E4M3 blocks with 32x32 E8M0 scales, MXFP4 routed experts and the FP8 Engram table, beside the EXL3 trellises; Engram row shards open on macOS, and tf-dsv41-engram-pack writes the release's Engram tables as row shards with a manifest beside them: the release tensors each table came from (file, offset, length, dtype, shape), each table's SHA-256 and a seeded sample of rows. tf-dsv41-engram-pack manifest writes one for tables packed earlier, after checking the sampled rows against the release shards, and engram_manifest.check() is what a backend calls at boot to refuse tables that no longer match the release (the Metal server's boot check comes with its part). The CUDA engine refuses a release checkpoint right after reading its config, with a message that says to pack it to EXL3 or serve it on Metal.

Part 1 of 18 of the stack for #1676; stacks on main.

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: TensorFold 0f2ce63 on main 2761c24 (native Zig 0.17.0; no MLX or PyTorch); Apple M3 Ultra (60-core GPU), 96 GB, macOS; one DGX Spark (GB10) for the CUDA host tests (build and host tests only, no GPU run); checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a: readers and admission only; no served path changes.
• Decode speed before and after, from tools/bench_openai.py: measured on the last PR of the stack.
• Prompt speed before and after: the cold first request's ttft_s from tools/bench_cache.py --require-cold on a long prompt, or "n/a" with the reason when the change cannot reach prompt processing: measured on the last PR of the stack.
• Tests run, with pass, fail and skip counts: zig build test test-dsv41-shared dsv41-engram-pack -Dcpu=apple_m1: exit 0, 586/586 steps succeeded; test-dsv41-shared includes the four new manifest tests (a good table passes and parses back; one flipped byte in a sampled row, re-sharded source tensors and a missing manifest are each refused by name); on the GB10, zig build test-dsv41-cuda-host test-dsv41-shared: exit 0, 18/18 steps, 436 pass, 0 fail, 1 skip, including the refusal test (cuda_format.zig); python3 tools/zig/lean_check.py: 4817 files, 424 problems, the same count as main (4807 files, 424). A CHANGELOG.md Unreleased line is included. Zig's summary counts only the test binaries it ran in that invocation, so the step count is the comparable figure.
• Not run, and why: CUDA on a GPU: the change only parses and refuses a config; Metal on M1, M2, M4 and M5: one M3 Ultra only.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt.
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thank you @gilby. This is a big, careful piece of work, and splitting it into one-job parts with contract records for each kernel makes it much easier to review. This reply covers the whole stack, #1705 to #1722, for #1676.

We'll review in this order: the release-format readers (#1705), then the kernel parts, then the decode forward, then serving. Images come after the text path lands. It won't make 1.2.0, which we're cutting in the next few hours, but it's first in line after that.

A few requests before landing:

• Rebase: please rebase onto current main. It conflicts in registry.zig, config.zig and serve_engine.zig. Main's core/contract/registry.zig is already at 596 lines, so the new entries will need it split past the 600-line limit.
• Kimi K3 (dsv41 metal: MXFP4 routed experts for decode rows over Kimi's plan, down and combine #1707): this part changes Kimi K3's shipped expert kernel (kimi/experts.metal, kimi_common.h). Could the scale handling move into a DeepSeek-only copy? If it has to stay shared, we'll need Kimi K3 token equality against main's serial decode, plus decode and prompt speed against main.
• Contract coverage: the phase-1 glue kernels (glue_tmp.metal) and fused.metal run in the served decode path, so they need contract records and planted controls of their own.
• Default test step: please keep the DeepSeek Metal device tests out of the default zig build test and give them their own step. Host gates run that step on Macs whose GPUs are busy with other work.
• CUDA (dsv41: the served engine's lane half as a backend-neutral module #1719): this changes the CUDA served path for the EXL3 checkpoints people already run. We'd prefer the Metal lane target not to depend on it. If it stays, we'll need CUDA token equality against main's serial decode, drafts on and off, unchanged cached_tokens, and speed against main, on a GB10.
• dsv41: read the release checkpoint's FP8 blocks, MXFP4 experts and Engram tables, and Engram rows on Darwin #1705 on CUDA: a release checkpoint now gets past config parsing on CUDA and is refused only later. Please refuse it early with a clear message and a test.
• The full model: the real-layer float64 test, which was skipped in every part; a comparison against DeepSeek's reference implementation on the first tokens of a few prompts; a drafts-off decode number; and cold prefill at a few lengths. Per-part speed isn't needed. The release needs a 512 GB Mac, which we don't have, so we'll rely on your runs for these, posted on the PR that turns serving on.
• Test counts: could you say why the zig build test pass counts move between parts (170, 3,352, 216, 1,993)?

Thanks again. We're keen to get this in.

[gilby]

Thank you for the careful review, and congratulations on 1.2.0. Since you review bottom-up and main moves quickly, I've rebased only this part for now: #1705 is now one commit on current main (acea3b5). I'll rebase each later part onto main when you reach it, rather than rebasing all 18 at once and doing it again a day later. All the changes below are done and tested together on main in one tree, and each part will carry its share.

Rebase and registry.zig: #1705 itself doesn't touch the registry. For the kernel parts, the family's records chain in one new file (core/contract/dsv41_metal_records.zig), so registry.zig grows by three lines (599) instead of past the limit. config.zig now builds on main's parseWith / parseMetadata / checkpoint_metadata: the release format is admitted with a format field (exl3 or release_fp8) and the asset-metadata path is unchanged. One test of main's changes meaning: "asset metadata preserves release checks…" expected the release config to fail with MissingField (its quantization_config has no version). With the release format admitted it now parses, so the test checks .format == .release_fp8 and its name says so. Please say if you'd rather keep Config.parse refusing it and admit the release through a separate entry point.

#1705 on CUDA: the CUDA engine now refuses a release checkpoint right after reading its config, before any pack, weights or launch, with: "DeepSeek V4.1 CUDA: this checkpoint stores the release's own FP8 blocks and MXFP4 experts; the CUDA engine executes EXL3 trellises, so pack the checkpoint to EXL3 (or serve it on Metal) first". The test (cuda_format.zig, in test-dsv41-cuda-host) parses the EXL3 fixture (admitted) and the release fixture (refused by name). This is in this push.

Kimi K3 (#1707): DeepSeek now has its own copy of the expert plan, chains and combine (deepseek_v41/experts_kimi.metal, kernels renamed tf_dsv41_route_plan / tf_dsv41_xp_combine*), with the full-range E8M0 scale in the copy. kimi/experts.metal and kimi_common.h are byte-identical to main. The DeepSeek contract still passes 68 of 68 kernels with every control red.

Contract coverage for glue_tmp.metal and fused.metal: every kernel in both files now has planted controls (non-finite read as 0, a lost or swapped term, an fp16 total, a write past the end) that compile away at TF_CTL 0, and contract records (14 + 21) that pass the limits check on M1 to M5. The unused g_actq, g_swiglu and g_join are deleted. The gate suites for these kernels are being finished and will be in the glue parts when you reach them.

Default test step: the DeepSeek Metal device suites (layer and served) are out of zig build test; they run from test-dsv41-metal, test-dsv41-metal-serve and test-dsv41-metal-vision, and test-dsv41-metal-all runs all three. The family's host checks stay in test.

CUDA (#1719): the Metal lane target no longer depends on it. serve_engine.zig and the other CUDA served-path files are byte-identical to main; the lane host modules the Metal target uses are Metal-only, and no CUDA root imports them. Part 15 will shrink to that, so the CUDA served path for EXL3 checkpoints doesn't change.

Engram tables: tf-dsv41-engram-pack writes a manifest beside the tables (the release tensors each table came from with file, offset, length, dtype and shape; each table's SHA-256; a seeded sample of rows). At boot the Metal server reads the sampled rows from the release shards and refuses to start, with a message naming the cause, if the manifest is missing, the shards moved or changed, a table is corrupt, or a sampled row differs; TF_DSV41_ENGRAM_VERIFY=full also checks every byte. The manifest writer comes to this part in its next push, once its command for tables packed before it is finished; the boot check goes with the Metal server part.

The full model (release checkpoint on an M3 Ultra, 512 GB; the full receipts go on the serving PR):

• Real-layer float64 test (TF_DSV41_REAL, release layer 2): 4 of 4 tests, every stage within its bound (e.g. wq_b worst 7.798e-3 against a bound of 7.874e-3), quantization points bit-equal.
• DeepSeek's reference inference/model.py on CPU (its kernels replaced by float32/bfloat16 stand-ins), greedy, same token ids: an 18-token code prompt matches for all 8 tokens (margins 1.8 to 21.7 logits) and a 17-token arithmetic prompt matches. A 201-token prompt differs at its first token: the reference picks A over The by 0.69 logits and the native engine picks the reference's second choice. A length sweep agrees at 96, 112 (margin 0.23) and 128 tokens, so we read it as summation order flipping a near tie, not a bug; that isn't proof, because the native server returns no logprobs to compare probabilities.
• Decode with drafts off: 35.0 tok/s code and chat, one stream (DSpark on: 46.8 code / 43.1 chat; four streams 78.4 / 76.0 total).
• Cold prefill: 591 tok/s at 9,513 tokens, 746 at 32,768, 750 at 38,472 and 153,210, about 580 at ~912,000.

Test counts: the moving numbers were what zig build test printed for the test binaries that actually ran in that invocation. Steps cached from the previous part's build weren't run again, so they weren't counted: a part that changed a widely imported file re-ran most binaries (3,352, 1,993), one that didn't re-ran few (170, 216). The device suites in the default step added to that. From now on the receipts give the step count, which doesn't depend on the cache, beside the tests that ran; this part's is 576/576 on current main.

[gilby]

#1705 is rebased on current main (6d3fa01), still one commit, and now carries the Engram manifest writer I mentioned: tf-dsv41-engram-pack pack writes the manifest beside the tables, tf-dsv41-engram-pack manifest writes one for tables packed earlier after checking the sampled rows against the release shards, and engram_manifest.check() is the call a backend makes at boot (the Metal server's boot check comes with its part). Four new tests in test-dsv41-shared: a good table passes and parses back; a flipped byte in a sampled row, re-sharded source tensors and a missing manifest are each refused by name. The receipt in the description is updated: 586/586 steps on the M3 Ultra, 18/18 on the GB10 (including the CUDA refusal test), lean check 424 problems, the same as main. The only conflict was the CHANGELOG.md Unreleased list; both lines are kept.

[ashhart]

Thanks @gilby, this answers everything for the first three parts. The early CUDA refusal with its test, DeepSeek's own copy of the expert kernels (Kimi's files are untouched), and the records kept out of registry.zig are just what we hoped for. #1705 is gating to land now; #1706 and #1707 follow once the dsv41 and kimi kernel contracts pass on our M3 Ultra. On Config.parse: admitting the release with a format field is fine; keep the test as you changed it. For the later parts we'll look for the glue_tmp and fused contract records, the test-dsv41-metal* steps, and the CUDA served path byte-identical to main, as you describe.

[ashhart]

Thank you @gilby. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @gilby! Reading DeepSeek-V4.1's release checkpoint as published, with the checked Engram pack, is in 1.2.1: https://github.com/ashhart/TensorFold/releases/tag/v1.2.1