Repository navigation
Conversation
zeDriverGet and zeDeviceGet were called once with a preset count of 8. On WSL2 with an Iris Xe (loader 1.16, NEO 23.43) the loader left the count as asked and filled the rest with null handles, and zeDeviceGetProperties on a null handle crashed. Both enumerations (context.enumerate and Gpu.open) now query the count first, clamp it to the array, and skip null handles.
…allback for its two DPAS products ocloc refuses intel_sub_group_bf16_bf16_matrix_mad_k16 for tgllp. With TF_XPU_NO_DPAS, nem_attn_dec.cl computes the same 8x16 by 16x16 bf16 product with sub-group shuffles and fp32 FMAs. -Dxpu-dpas (default off for tgllp, rkl, adl-*, dg1, gen12lp devices) passes it to every SPIR-V image's ocloc options; other devices build as before.
kernels.dpas carries -Dxpu-dpas. Without it nem_attn_dec builds from its fallback but Nemotron leaves a_dec null and decodes through attn_partial/attn_merge, which measured faster on Iris Xe (12.0 against 57.3 ms at 131k keys; the fallback spills 128 KiB of registers).
…_prow gets a 4 x 4 tile image for Gen12LP The Qwen3.8-27B runner needs DPAS, 2D block loads and nGEN, so `zig build xpu-native -Dxpu-device=tgllp` could not build. With -Dxpu-dpas=false the server is built without it and lists qwen3_5 only when it is built; a bmg build is unchanged and still carries it. nem_prow.cl's PR and PO become overridable; the new image nem_prow_t4 (-DPR=4 -DPO=4) spills about 25 registers on Gen12LP where the 8 x 8 pdense spills about 210. The 14 existing bmg images stay byte-equal to main.
…a prompt cache needs free memory Under WSL2 (Intel Iris Xe 0x9a49 over /dev/dxg, compute runtime 23.43.27642, Level Zero loader 1.16) zesInit(0) segfaults inside libze_intel_gpu.so.1. tensorfold-native died at startup in ze.Gpu.memory(), which the lane host called for the prompt cache budget even when no cache is made. The lane host now reads free memory only for drafted engines, the only ones that keep a cache. zesDriverGet, zesDeviceGet and zesDeviceEnumMemoryModules ask for the count first, clamp it and skip null handles, as the core enumeration does.
Prepare text and VL 4/6/8-bit checkpoints with revision-pinned header inspection and native metadata validation. Inference remains unregistered pending operator and model qualification. Refs ashhart#1491
…kends' loaders names.zig names every module of the text decoder with its path; moduleName() builds a full tensor name in either namespace (bare `model.` or wrapped `language_model.model.`), detect() picks the namespace from the embedding a header holds. It has no dependency on the safetensors parser, so a backend's loader can import it where core already owns that parser. weights.zig re-exports it, and schema() and validate() read the same table, so a backend that loads by these names cannot drift from the admission check. A test checks every module name of both namespaces against the schema.
Without torch (or with --numpy) the forward runs on numpy: MLX affine and bf16 weights are dequantized exactly; NVFP4 and FP8 stay torch-only and are refused by name. --bf16-points rounds where the Intel GPU engine stores bf16 (embedding rows and RoPE cos/sin too, attention P and logits never). dump writes the float64 hidden state after every layer and check replays an engine's greedy replies teacher-forced. The collect and miss-table CLI is unchanged.
…, gated GQA attention over a ring, sandwich norm, SwiGLU) and their FP64 op tests
…state and KV rings) loading by names.zig, its checkpoint test against ref64.py's float64 dumps
…pt projections on nem_prow_t4 without DPAS, Iris Xe measurements
…less what the process allocated Under WSL2 the GPU is a WDDM adapter (/dev/dxg, no /dev/dri). NEO 23.43's zesInit walks /dev/dri/renderD128..191, then opens /dev/dxg and faults inside libze_intel_gpu.so.1 (SIGSEGV at 0x100000000, an AVX-512 string compare in libc on a bad pointer), with or without ZES_ENABLE_SYSMAN. With ZES_ENABLE_SYSMAN=1 set before zeInit, zesInit returns ZE_RESULT_ERROR_UNINITIALIZED instead; Sysman has no WDDM backend either way. Gpu.memory() now asks Sysman only where /dev/dri exists (or /dev/dxg does not). Elsewhere free and size are the core total (zeDeviceGetMemoryProperties, asked with room for 8 modules: NEO answers a count query with 0), and the lane host's freeMemory subtracts this process's allocations as before, so a drafted engine's prompt cache is sized on WSL too. tf-xpu-memory-test asks free memory the way the lane host does, before and after a 512 MiB allocation.
…ompt cache of kept states The lane adapter gets a third mode for engines with sequences but no draft head: each stream binds its own sequence, the prompt runs in planned 64-row chunks, rounds are one-row windows of up to --parallel streams one after another, and kept prompt states go to the prompt cache. Nemotron's modes are unchanged. LFM3's per-stream state (the conv rows of 13 layers, the KV slots of 11, the position) moves out of the layers into a sequence the model binds. A kept state is the conv rows, the full layers' slots up to the mark and the sliding rings whole, copied on the device. Concurrent replies at --parallel 1, 2 and 4 equal the solo ones byte for byte; a resumed second turn equals a cold one.
… and RoPE with the KV store lfm3_mv.cl takes attn.cl's qmv_row verbatim (same compile options, so the same contraction) and runs w1 and w3 of a row with their SwiGLU in one launch, and q, k, v and the gate in one; lfm3_rope_store normalises and rotates the q and k heads and stores k and v in one launch, through the same per-head function lfm3_qk_norm_rope now calls. Every value is rounded as before: the greedy tokens of the three reference prompts are unchanged. Decode on Iris Xe: 15.2 to 17.2 tok/s (LFM3_FUSE selects the fusions for A/B timing).
A sweep of nem_prow's PR and PO over {2, 4, 8} on Iris Xe (LFM3-1.7B prefill, 1,024 tokens, windows of
32, 64 and 128 rows) puts 8 prompt rows by 4 outputs a sub-group first: about 8 % above 4 x 4, while
PO 8 spills (8 x 8: about 210 registers, 4 x 8: about 575) and runs 2 to 10 times slower. The image is
renamed for what it is for; every pdense output sums as before, so the bits do not change. The 14
existing bmg images stay byte-equal to main.
On Iris Xe the prompt projections take the 8 x 4 tile image and the server cuts prompts into windows of 128 rows: 1,024-token prefill 78 to 85 tok/s against 4 x 4 tiles in windows of 64. The greedy tokens of the three reference prompts are unchanged at windows of 128 and of 64.
…les, with streams and the prompt cache
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1666 once its base lands. Draft: it carries the commits of the PRs below it until they merge, so the diff shrinks to the last seven commits (from
c682749c0on).Stacked on:
ecdc0f8cd);ref64.py, numpy path (a66a2bf75).What it adds
names.zig, MLX 4-bit group 64 as stored;ref64.py's float64 dumps.xpu-nativeserves LFM3 through the lane host.--parallelstreams run on their own sequences, and a prompt cache keeps states.nem_prow_gen12in windows of 128 rows.docs/recipes/xpu-arc.md.Checks (Iris Xe, WSL2)
Forward against float64: every layer of 3 prompts is at 0.96–1.05× the bf16 gap. The 64 greedy tokens match the CUDA engine (lfm3 CUDA: serve the block FP8 export weight-only and the MLX 4-bit export as stored (stacked on #1508) #1634, MLX 4-bit) on all 3 prompts, and match again after every speed change, at windows of 128 and 64, with each fusion on and off.
Streams: at
--parallel1, 2 and 4, 5 concurrent replies are byte-equal to solo, in two runs each.Prompt cache: a resumed second turn is byte-equal to the cold one, at 1k and 8k tokens. TTFT is 0.56 s against 13.25 s at 1k, and 0.80 s against 122.6 s at 8k.
Speed, server,
--context 16384:Spark (GB10): lean reports no new findings;
zig build test, the contract, golden, security and fuzz steps pass;git diff --checkis clean.Overlaps
atc overlapreports file conflicts with:xpu/.I'll rebase onto whichever lands first.