Qwen4-27B (dense) on native CUDA, GB10: shape parameterization ahead of the checkpoint (RFC)

@ashhart

This is an RFC, not a claim. Nothing here is started. We'd like to know whether you want the preparation work below, and in which shape, before any weights are public.

What is known (from Intel's merged kernel PR only). analytics-zoo/vllm-xpu-kernels#22 (merged 2026-10-19) adds kernels for a dense "Qwen4-27B". Line numbers are at its head 265bbe2:

Hidden 5120; hyper-connections with 4 streams, width 20480 (csrc/qwen38/hc_dense_sycl.cpp:23-25).
HC low rank 640 at TP1 (:519), 320 at TP2 (:242).
GDN projections at TP2: [8192, 5120] and [48, 5120] (csrc/qwen38/fp8_dense.cpp:153-154); GDN norm heads 48 (TP1) / 24 (TP2) x 128 (custom-kernels-inventory.md, try_gdn_norm).
Attention head dim 256 (fp8_dense.cpp:378-379). Local Q12/KV2 at TP2 (csrc/qwen38/qsa_sycl.cpp:1122-1127, vllm_xpu_kernels/qwen38_qsa.py:35,204). The code takes KV = Q/6 (fp8_dense.cpp:389), so the full model is Q24/KV4 at TP1. That is inferred.
Dense FFN: 14336 (7168 at TP2) appears as a GEMV shape (fp8_dense.cpp:33-34) and in the INT4 tests as 28672x5120 and 5120x14336. The same numbers are also the QKV width, so "intermediate 14336" is inferred, not stated.

The PR does not give the layer count, the layer schedule, or the MTP shape. It says Qwen4 QSA weights were not provided to the authors, and QSA with MTP is not enabled there. A placeholder repo, IteraCompute/Qwen4-27B-NVFP4, exists on the Hub (README only, tagged work-in-progress, no config).

What TF already has.

qwen3_5 dense 27B is the same skeleton: hidden 5120, 24/4 heads, head dim 256, GDN 16/48/128/128 (families/qwen3_5/config_checks.zig:177). Its CUDA geometry reads intermediate, layers and heads from the config (cuda_geometry.zig:243-248) and accepts a dense FFN.
Flash Next: HC count and rank come from the config (flashnext/config.zig:235-236, weights.zig:59-68). The GDN kernels use the same 16/48/128/128 (kernels/cuda/flashnext_gdn.cu:6). QSA runs at KV2, head dim 256, with NQ/NKV passed at runtime (kernels/cuda/flashnext_attn.cu:72).

What has to change.

flashnext/config.zig:231-234 requires num_experts, num_experts_per_tok and the expert widths, and :200 accepts only qwen4_exp. Dense needs these optional plus a dense FFN width.
The CUDA path refuses any other shape: cuda_config.zig:11-34, from constants in cuda_prompt_wt.zig:10-31 (48 layers, 2560, rank 320, 512 experts, Q24/KV2, proj_n), cuda_attn.zig:9-20, cuda_moe.zig:8-12, cuda_gdn.zig:9-13. Layers, hidden, HC width and rank, heads and the FFN width become fields.
A dense SwiGLU FFN in the Flash Next graph, from the qwen3_5 dense lane.
QSA at Q24/KV4 and HC at 5120x4 (20480).

Proposed steps, each a small PR that can land before the checkpoint.

Config: dense or MoE, with a synthetic-shape unit test. No kernels.
Parameterize the CUDA constants listed above. The 2560 checkpoint must produce identical bits and pass the existing gates.
Synthetic kernel-contract records at the new shapes (HC 20480, rank 640, QSA Q24/KV4, FFN 14336) against the f64 reference.
Dense FFN node in the graph, tested the same way.

Waits for the checkpoint: config.json, tensor names, the layer schedule, the MTP head, the quant format, and whether every full-attention layer has the indexer.

Unconfirmed:

That Qwen4-27B is the final name, and that Intel's dense model is the one that will be released.
FFN intermediate, layer count, and the TP1 head counts (all inferred).
Whether the indexer columns sit in the QKV projection (Intel's QKV widths have none).
Anything about the placeholder repo beyond its metadata.

Questions for @ashhart:

A separate family, or a dense mode in flashnext?
Does anyone at TF already have the config or weights?
Do you want steps 1-3 ahead of the release, or after it?

Additions since the first post:

Same reading elsewhere. llama.cpp opened ggml-org/llama.cpp#31072 ("Get ready for qwen4") today. It reads the same shapes out of Intel's PR, so it corroborates rather than adds evidence. Its reading: dense, hidden 5120, GDN 48/16 x 128, full attention Q24/KV4 at head dim 256 with a sigmoid output gate, HC 4 x 5120, FFN probably 14336, and MTP exercised with 4 draft tokens. The same two things are open there:
- whether the 3:1 GDN/attention schedule stays;
- whether the n-gram embedding table (PLE) exists.
HC rank. Intel's dense HC kernel uses 640 for the full model and 320 per rank at TP2. Flash Next's 320 is therefore not a safe default for a dense config: hc_lowrank has to come from the config.
No dense reference upstream. transformers' qwen4_exp builds a sparse MoE block in every decoder layer and requires num_experts > 0.
- warpfront/hipfire has a CPU reference oracle with a synthetic tiny-dense-dev profile (crates/hipfire-arch-qwen4/reference_oracle/QWEN4_PROFILES.md). It is dense SwiGLU in place of MoE, labeled non-upstream, and goes up to hidden 5120. It could serve as an extra cross-check next to our own f64 reference.
- Any dense profile stays provisional until a real config exists.
QSA memory lessons from llama.cpp's Flash Next port, worth checking at the new shapes:
- gather the selected KV blocks rather than masking a dense scan, since decode time otherwise grows with context (#28734);
- don't keep raw and pooled indexer keys both (#30005);
- don't build context x micro-batch selection masks (#30032).
License. Flash Next's weights are under the Qwen Community License 1.0. Qwen4-27B's licence is unknown.

The prep steps from the first post (config, shape parameters, contract records at the new shapes, a dense FFN node) are in progress locally and will be offered as small PRs only if wanted.

The prep steps are on our fork, as branches rather than PRs. Each step stacks on the one before, on main 6244c62:

step	branch	compare
1. dense or MoE per layer	`qwen4-config`	diff
2. one shape table for the CUDA dimensions	`qwen4-shapes`	diff
3. synthetic contract records at the dense shape	`qwen4-contract`	diff
4. a dense feed-forward node in the shared pass	`qwen4-ffn`	diff

Dense detection

It uses the rule in Intel's public vLLM patch (intel/llm-scaler build/v0.31.0, the qwen4_exp XPU decoder). A layer is MoE when it is not in mlp_only_layers, num_experts > 0 and (layer_idx + 1) % decoder_sparse_step == 0. Otherwise it is a dense Qwen3NextMLP with intermediate_size and hidden_act.
HC reads hc_count and hc_lowrank; a dense config has no 320 default.
This is the first public code with a dense Qwen4 decoder. Every dense error and doc comment still says "provisional".

Step 2 is bit-identical on Flash Next. The CUDA constants now come from one comptime table, and loading checks each kernel's limits and names the offending field. nvidia/Qwen3.8-Flash-Next-FP4-Local served from main and from the branch gives the same sha256 for 200 greedy tokens. Two prompts were used: 64 tokens, and 9,592 tokens, which runs the QSA indexer.

Step 3

New records: SwiGLU at 14336, QSA attention at Q24/KV4, and the HC read-out at 4 x 5120 with rank 640. Each passes gates 1-4, with its controls red.
GDN at hidden 5120 only adds served shapes to the existing factored-matmul records.

Step 4 is tested on synthetic weights against f64 at hidden 5120 and FFN 14336. Nothing loads such a checkpoint yet.

Main fixes carried in step 3, each its own commit:

w4a8_tests.zig against the current prompt() signature;
an allocator mismatch in the Flash Next experts record;
the softcap include, the same patch as cuda gemma4: admission counts vision and the drafter, init cleanup, DFlash config bounds, contract build fix #1245.

On this GB10 the full kernel-contract on main still stops at the w4a8 record with CUDA_ERROR_MISALIGNED_ADDRESS, which poisons the records after it. I have not looked into that yet.

Still fixed in the CUDA path

A mixed MoE/dense schedule is refused.
So is any attention schedule other than every fourth layer.
PLE stays a config field.

QSA memory follow-ups, not changed here

The pass keeps both raw and pooled indexer keys.
The indexer scores form a rows x blocks fp32 matrix.
Attention itself gathers by key ids, without a masked dense scan.

Happy to turn any of these into PRs if wanted.

Follow-up branch qwen4-next, stacked on qwen4-ffn. Each fix is its own commit.

Review fixes

The dense FFN test freed the projections twice when Ffn.init failed after taking them.
The 6-bit fixture is named and marked parser-only.
An mlp_only_layers entry past the last layer is an error instead of being dropped.
PLE is a capability: a config with no PLE keys gets none, instead of Flash Next's defaults.
projN is marked provisional.

New

Layer schedule: read from layer_types, falling back to full_attention_interval. The two must agree when both are present. A host test checks all 48 Flash Next layers against the old formulas.
Dense build: the dense shape compiles in full (zig build compile-flashnext-dense).
Dense node: added to the row-by-row path, with a hook test against the shared pass (same fp32 words).
Kernel contract: float64 value records for the GDN input and output projections at hidden 5120. Gates 1-4 pass; gate 3 worst is 0.993 of the bound for the input projection.

Unchanged on Flash Next: greedy output (drafted, plain and 4 concurrent streams, at 64, 9,592 and 31,332 tokens) hashes the same on this branch and on main. Full zig build test passes and lean_check shows 0 new.

Open, for when a dense checkpoint exists

SwiGLU rounding: TF's SwiGLU rounds once. A bf16 reference that rounds silu first differs in about 28% of products by one bf16 step. Which one Qwen4 means needs the real weights.
No end-to-end run: a tiny hipfire-generated checkpoint is refused by the fixed shape table (11 fields) and the affine-config check, and nothing loads dense gate/up/down yet.
QSA cache: only the trailing partial block of raw indexer keys is needed after pooling. Dropping the rest would save about 6% of the attention cache on Qwen4 and 11% on Flash Next. MTP state, snapshots and rollback read raw keys by position, so I left it alone.

/atc claim

Changelog and recipe: Qwen4 dense preparation passes its synthetic de…

Qwen4 dense preparation landed on main in 03c2820: the native binary admits qwen4 and qwen4_text configs on Metal and CUDA, sharing Flash Next's layer graph with a dense feed-forward, and refuses geometry its kernels don't cover before allocating. Synthetic device gates pass on an M3 Ultra (zig build test-qwen4-metal) and a GB10 (tf-cuda-test qwen4): nonzero checkpoints through the serving engine match a binary64 reference with no tolerance, across prompt cuts, resume, concurrency and copied drafts. No real checkpoint, speed or memory fit is claimed yet; on release day we'll check the published config and tensor names and run the model on both.

/atc release

@jschmied

Thank you, @jschmied, for the RFC and the preparation branches. The dense Qwen4 preparation is on main (21c7df9 through 03c2820): the native binary admits qwen4 and qwen4_text configs on Metal and CUDA, refuses geometry its kernels don't cover, and passes synthetic device gates on both. On release day we'll check the published config and tensor names and run the checkpoint; please open a new issue if anything in the real release differs. Closing this RFC as done.

Navigation Menu

Repository navigation

Qwen4-27B (dense) on native CUDA, GB10: shape parameterization ahead of the checkpoint (RFC) #1244

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Qwen4-27B (dense) on native CUDA, GB10: shape parameterization ahead of the checkpoint (RFC) #1244

Description

Activity

jschmied commented on Oct 24, 2026

jschmied commented on Oct 24, 2026

jschmied commented on Oct 25, 2026

ashhart commented on Oct 26, 2026

ashhart commented on Oct 26, 2026

ashhart commented on Oct 27, 2026

ashhart commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions