Skip to content

xpu: LFM3-1.7B on Intel GPUs, Iris Xe first (draft, stacked) - #1696

Draft
jschmied wants to merge 19 commits into
ashhart:mainfrom
jschmied:lfm3-xpu
Draft

jschmied wants to merge 19 commits into
ashhart:mainfrom
jschmied:lfm3-xpu

Conversation

@jschmied

Copy link
Copy Markdown
Contributor

Closes #1666 once its base lands. Draft: it carries the commits of the PRs below it until they merge, so the diff shrinks to the last seven commits (from c682749c0 on).

Stacked on:

What it adds

  • Kernels without DPAS: the short conv, QK norm with partial RoPE, gated GQA attention over a ring, the sandwich norm and SwiGLU, each with an FP64 op test and a planted control.
  • The LFM3 forward on the Intel GPU:
    • prefill in windows, decode, and the conv state and KV rings;
    • weights loaded through names.zig, MLX 4-bit group 64 as stored;
    • a checkpoint test against ref64.py's float64 dumps.
  • Serving: xpu-native serves LFM3 through the lane host. --parallel streams run on their own sequences, and a prompt cache keeps states.
  • Exact speed work:
    • decode fuses w1 + w3 + SwiGLU, the four attention matvecs, and RoPE with the KV store;
    • prompts run on nem_prow_gen12 in windows of 128 rows.
  • Docs: the Iris Xe row in docs/recipes/xpu-arc.md.

Checks (Iris Xe, WSL2)

  • Forward against float64: every layer of 3 prompts is at 0.96–1.05× the bf16 gap. The 64 greedy tokens match the CUDA engine (lfm3 CUDA: serve the block FP8 export weight-only and the MLX 4-bit export as stored (stacked on #1508) #1634, MLX 4-bit) on all 3 prompts, and match again after every speed change, at windows of 128 and 64, with each fusion on and off.

  • Streams: at --parallel 1, 2 and 4, 5 concurrent replies are byte-equal to solo, in two runs each.

  • Prompt cache: a resumed second turn is byte-equal to the cold one, at 1k and 8k tokens. TTFT is 0.56 s against 13.25 s at 1k, and 0.80 s against 122.6 s at 8k.

  • Speed, server, --context 16384:

    before the speed work after
    decode 14.8–15.8 tok/s 16.7–16.9 tok/s
    TTFT at 1k 13.1–14.2 s 11.7–12.2 s
    TTFT at 8k 130.8–131.3 s 116.3–117.7 s
  • Spark (GB10): lean reports no new findings; zig build test, the contract, golden, security and fuzz steps pass; git diff --check is clean.

Overlaps

atc overlap reports file conflicts with:

I'll rebase onto whichever lands first.

jschmied and others added 19 commits October 30, 2026 15:30
zeDriverGet and zeDeviceGet were called once with a preset count of 8. On WSL2 with an Iris Xe (loader 1.16, NEO 23.43) the loader left the count as asked and filled the rest with null handles, and zeDeviceGetProperties on a null handle crashed. Both enumerations (context.enumerate and Gpu.open) now query the count first, clamp it to the array, and skip null handles.
…allback for its two DPAS products

ocloc refuses intel_sub_group_bf16_bf16_matrix_mad_k16 for tgllp. With TF_XPU_NO_DPAS, nem_attn_dec.cl computes the same 8x16 by 16x16 bf16 product with sub-group shuffles and fp32 FMAs. -Dxpu-dpas (default off for tgllp, rkl, adl-*, dg1, gen12lp devices) passes it to every SPIR-V image's ocloc options; other devices build as before.
kernels.dpas carries -Dxpu-dpas. Without it nem_attn_dec builds from its fallback but Nemotron leaves a_dec null and decodes through attn_partial/attn_merge, which measured faster on Iris Xe (12.0 against 57.3 ms at 131k keys; the fallback spills 128 KiB of registers).
…_prow gets a 4 x 4 tile image for Gen12LP

The Qwen3.8-27B runner needs DPAS, 2D block loads and nGEN, so `zig build xpu-native -Dxpu-device=tgllp`
could not build. With -Dxpu-dpas=false the server is built without it and lists qwen3_5 only when it is
built; a bmg build is unchanged and still carries it.

nem_prow.cl's PR and PO become overridable; the new image nem_prow_t4 (-DPR=4 -DPO=4) spills about
25 registers on Gen12LP where the 8 x 8 pdense spills about 210. The 14 existing bmg images stay
byte-equal to main.
…a prompt cache needs free memory

Under WSL2 (Intel Iris Xe 0x9a49 over /dev/dxg, compute runtime 23.43.27642, Level Zero loader 1.16)
zesInit(0) segfaults inside libze_intel_gpu.so.1. tensorfold-native died at startup in ze.Gpu.memory(),
which the lane host called for the prompt cache budget even when no cache is made. The lane host now
reads free memory only for drafted engines, the only ones that keep a cache.

zesDriverGet, zesDeviceGet and zesDeviceEnumMemoryModules ask for the count first, clamp it and skip
null handles, as the core enumeration does.
Prepare text and VL 4/6/8-bit checkpoints with revision-pinned header inspection and native metadata validation. Inference remains unregistered pending operator and model qualification.

Refs ashhart#1491
…kends' loaders

names.zig names every module of the text decoder with its path; moduleName() builds a full tensor
name in either namespace (bare `model.` or wrapped `language_model.model.`), detect() picks the
namespace from the embedding a header holds. It has no dependency on the safetensors parser, so a
backend's loader can import it where core already owns that parser. weights.zig re-exports it, and
schema() and validate() read the same table, so a backend that loads by these names cannot drift
from the admission check. A test checks every module name of both namespaces against the schema.
Without torch (or with --numpy) the forward runs on numpy: MLX affine and bf16 weights are dequantized
exactly; NVFP4 and FP8 stay torch-only and are refused by name. --bf16-points rounds where the Intel GPU
engine stores bf16 (embedding rows and RoPE cos/sin too, attention P and logits never). dump writes the
float64 hidden state after every layer and check replays an engine's greedy replies teacher-forced. The
collect and miss-table CLI is unchanged.
…, gated GQA attention over a ring, sandwich norm, SwiGLU) and their FP64 op tests
…state and KV rings) loading by names.zig, its checkpoint test against ref64.py's float64 dumps
…pt projections on nem_prow_t4 without DPAS, Iris Xe measurements
…less what the process allocated

Under WSL2 the GPU is a WDDM adapter (/dev/dxg, no /dev/dri). NEO 23.43's zesInit walks
/dev/dri/renderD128..191, then opens /dev/dxg and faults inside libze_intel_gpu.so.1 (SIGSEGV at
0x100000000, an AVX-512 string compare in libc on a bad pointer), with or without
ZES_ENABLE_SYSMAN. With ZES_ENABLE_SYSMAN=1 set before zeInit, zesInit returns
ZE_RESULT_ERROR_UNINITIALIZED instead; Sysman has no WDDM backend either way.

Gpu.memory() now asks Sysman only where /dev/dri exists (or /dev/dxg does not). Elsewhere free and size
are the core total (zeDeviceGetMemoryProperties, asked with room for 8 modules: NEO answers a count
query with 0), and the lane host's freeMemory subtracts this process's allocations as before, so a
drafted engine's prompt cache is sized on WSL too. tf-xpu-memory-test asks free memory the way the
lane host does, before and after a 512 MiB allocation.
…ompt cache of kept states

The lane adapter gets a third mode for engines with sequences but no draft head: each stream binds its
own sequence, the prompt runs in planned 64-row chunks, rounds are one-row windows of up to
--parallel streams one after another, and kept prompt states go to the prompt cache. Nemotron's modes
are unchanged.

LFM3's per-stream state (the conv rows of 13 layers, the KV slots of 11, the position) moves out of
the layers into a sequence the model binds. A kept state is the conv rows, the full layers' slots up
to the mark and the sliding rings whole, copied on the device. Concurrent replies at --parallel 1, 2
and 4 equal the solo ones byte for byte; a resumed second turn equals a cold one.
… and RoPE with the KV store

lfm3_mv.cl takes attn.cl's qmv_row verbatim (same compile options, so the same contraction) and runs
w1 and w3 of a row with their SwiGLU in one launch, and q, k, v and the gate in one; lfm3_rope_store
normalises and rotates the q and k heads and stores k and v in one launch, through the same per-head
function lfm3_qk_norm_rope now calls. Every value is rounded as before: the greedy tokens of the three
reference prompts are unchanged. Decode on Iris Xe: 15.2 to 17.2 tok/s (LFM3_FUSE selects the fusions
for A/B timing).
A sweep of nem_prow's PR and PO over {2, 4, 8} on Iris Xe (LFM3-1.7B prefill, 1,024 tokens, windows of
32, 64 and 128 rows) puts 8 prompt rows by 4 outputs a sub-group first: about 8 % above 4 x 4, while
PO 8 spills (8 x 8: about 210 registers, 4 x 8: about 575) and runs 2 to 10 times slower. The image is
renamed for what it is for; every pdense output sums as before, so the bits do not change. The 14
existing bmg images stay byte-equal to main.
On Iris Xe the prompt projections take the 8 x 4 tile image and the server cuts prompts into windows
of 128 rows: 1,024-token prefill 78 to 85 tok/s against 4 x 4 tiles in windows of 64. The greedy
tokens of the three reference prompts are unchanged at windows of 128 and of 64.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LFM3-1.7B on the XPU backend (Intel GPUs, Iris Xe first)

2 participants