Repository navigation
DeepSeek-V4.1 at four ranks: 2,304-wide experts do not split into 128-aligned shards #1202
Description
The limitation
At tensor-parallel world 4, d1018b31 (1.1.1) refuses the DeepSeek-V4.1 family before any weights are read, and
current main (da275a36) still does, with the same checks at the same lines.
Two checks require a rank's share of a width to be equal across ranks and a whole number of 128-column blocks:
zig/src/families/deepseek_v41/block_admission.zig:5(decode admission):
if (c.expert_width % o.world != 0 or (c.expert_width / o.world) % 128 != 0) return error.InvalidBlock;zig/src/families/deepseek_v41/plan.zig:270(plan.build):... or c.vocab % (128 * o.world) != 0) return error.BadWorld;
The default config (config.zig) has expert_width = 2304 and vocab = 129280:
| width | 128-column blocks | per rank at world 2 | per rank at world 4 |
|---|---|---|---|
| expert, 2,304 | 18 | 9 blocks (1,152) | 4.5 blocks (576): refused, InvalidBlock |
| vocabulary, 129,280 | 1,010 | 505 blocks (64,640) | 252.5 blocks: refused, BadWorld |
A whole-block split exists at world 4: 5, 5, 4, 4 blocks for the experts (640, 640, 512, 512 columns) and 253, 253,
252, 252 for the vocabulary.
The RoCE transport has a separate two-rank limit (zig/src/fabric/tp/roce.zig:148,
if (b.world != 2) return error.Unsupported;). It is out of scope here.
Reproduction
No GPU or checkpoint needed. Zig 0.17.0 (.zig-version), clean checkout of d1018b31 or da275a36. Append to
zig/src/families/deepseek_v41/state_test.zig:
test "world 4 with 2304-wide experts is refused" {
const adm = @import("block_admission.zig");
const cfg: @import("config.zig").Config = .{};
const Opts = struct { world: u32, expert_rows: i64 = 64, dspark_rows: i64 = 64, branch_rows: i64 = 0, prefill_rows: i64 = 0, image_rows: i64 = 0, expert_topp: ?f32 = null };
try adm.options(&cfg, Opts{ .world = 2 }, &.{}, 1);
try std.testing.expectError(error.InvalidBlock, adm.options(&cfg, Opts{ .world = 4 }, &.{}, 1));
}zig build test-dsv41-forward
Result, the same on both commits: 101 of 101 pass (100 without the test). World 2 is admitted; world 4 returns InvalidBlock. With the
expectation removed, the world-4 call fails at block_admission.zig:5 with error.InvalidBlock.
Question
Is it acceptable for a rank's share of a width to be a run of whole 128-column blocks that is not equal across
ranks (rank 0 [0, 5), rank 1 [5, 10), rank 2 [10, 14), rank 3 [14, 18) for the experts), with every
emitter and sampler reading the rank's own [lo, len) instead of width / world? Or would you rather the width be
padded to a multiple of 128 * world?
Under the first option an even split keeps its offsets and lengths, so every two-rank shape is unchanged, and no
128-column block is cut. I have a three-commit branch for it (the [lo, len) helper; the emitters and samplers; the
plan and admission checks). It has host tests only; no four-rank token or speed run has been made. I will open it as
one pull request, or follow the padding route instead, depending on your answer.
Activity
ashhart commentedon Oct 22, 2026
/atc claim
flashosophy commentedon Oct 22, 2026
/atc propose pair
ashhart commentedon Oct 22, 2026
/atc accept 6351172198
ashhart commentedon Oct 22, 2026
Thanks for offering to pair on this, @flashosophy. We're writing the host-side fix now, along the whole-block split you described, and we'll share it as a draft pull request so you can run the four-rank cells on your setup.
flashosophy commentedon Oct 22, 2026
Thanks for taking it on. Two things that may save you time.
A reference implementation. Our version of the whole-block split is on
neko-legends/TensorFold:pr/admission-nonaligned-width(fc9e3d3a, three commits ond1018b31). It merges cleanly onto currentmain. With it,zig build test-dsv41-forwardpasses 106 tests (100 on cleand1018b31). The six new tests cover the split arithmetic (checked,ofOrEven,widest, the refusals), the uneven four-rank expert and head tile, and admission of each rank's whole-block shard on a synthetic pack. Use any of it or none of it. We'll run your draft either way.What we've already seen on four GB10s. We booted that branch at world 4, merged with our RoCE transport branch, on the EXL3 2.9 bpw pack of DeepSeek-V4.1-Flash. Admission now passes on every rank, and the shard probe reports the uneven split:
All four ranks came up over RoCE with no panics. To be clear about what this doesn't show:
a 1024/2048/4096-row segment is not on the prefill path at any position (block_prefill: Unsupported)). That's a separate limitation. At world 4 a rank holds 16 attention heads, and the prefill attention path requires a multiple of 32. It's unchanged in v1.1.2. We'll open it as its own issue, with a fix, rather than widen this one.When your draft is up we'll run the same cells on it: boot, the shard probe, and a decode receipt, and post the output here. We can also add a world-1 run beside the four-rank one, which would cover the "unverified" note on the docs page.
Plan four-rank DeepSeek prefill through the existing 32-head core
ashhart commentedon Oct 24, 2026
Fixed on
main(e8faf00) by #1206: four-rank DeepSeek-V4.1 shares whole 2,304-wide expert blocks instead of splitting them, with #1205's scratch fix. Device qualification at four ranks is still owed.- Synthema, Ash's AI Assistant.