MQ4-Lloyd (qt=52): per-tensor Lloyd codebook in the MQ4V2 container, -15% KLD at 4 bpw; mq4-pro prefill fix; AWQ rmsnorm rewrite by Kaden-Schutt · Pull Request #768 · warpfront/hipfire

Kaden-Schutt · 2026-09-15T09:38:05Z

What

MQ4L tiers (mq4l-xt / mq4l / mq4l-pro, quant type 52 MQ4G256V2Lloyd): the MQ4V2 tiers re-quantized with a per-tensor 16-level Lloyd-Max codebook instead of the uniform 4-bit grid. Byte-compatible MQ4V2 container (136 B groups, AWQ sidecars, identical size) plus a 64 B <tensor>.lloyd_levels sidecar per 4-bit tensor. Levels are constrained to centered E4M3-representable values so the FP8 prefill path and the f16 decode path share one exact codebook; the loader folds the centering into the group zero-points.

WT2 KLD, 24 chunks, gfx1201, prefill scoring, Q8 KV (same protocol as the HF model card):

tier	uniform MQ4V2 (AWQ'd)	MQ4L	Δ
xt	0.048190	0.040826	−15.3%
base	0.040519	0.034639	−14.5%
pro	0.033540	0.028404	−15.3%

Speed, gfx1201 (R9700), same card interleaved A/B, warm, HIPFIRE_VERIFY_GRAPH=0: xt uniform 36.49–36.54 tg / 1410–1415 pp512 / 1276–1281 pp8192; mq4l-xt 36.14–36.17 / 1395–1400 / 1264–1267 (−1.0% / −1.2%). Shipped-xt master reference on the same card: 36.55 / 1417 / 1284 (this branch: 36.58 / 1412 / 1280 — no uniform-path change).

Runtime coverage: gfx1201 prefill (FP8-WMMA LUT twins) + decode (fused scalar LUT kernels); gfx1100/gfx1151 decode (T1 admission). gfx11 batched prefill fails closed for now — the F16-WMMA-LUT route was gated and abandoned (+61% vs F16 base on the XTX; docs/plans/2026-09-15-lloyd-gfx11-port.md); the MMQ byte-LUT route is planned (docs/plans/2026-09-15-lloyd-gfx11-mmq-lut.md) as the next PR.

Fixes found on the way

mq4-pro batched prefill was incoherent on gfx1201 master (079eb152d): gated_norm_f32_batched / sigmoid_mul_f32 rewrite a stable batched scratch without invalidating the pointer-keyed F16 x-cache; the Q8 out_proj (gemm_q8_0_residual_wmma_gfx12) consumed the previous layer's activations. Masked while the F16 MQ4V2 launchers evicted the single-slot cache; exposed once all three FP8 prefill paths became default (gfx1201: FP8-WMMA MQ4v2 prefill GEMMs (+28% pp512), opt-in; GDN permlanex16 fix #757). 1-chunk WT2 KLD 6.42 → 0.019; 24-chunk 0.0335. Cherry-pickable.
AWQ rmsnorm-rotate rewrite: fused_rmsnorm_mq_rotate_awq is now derived from fused_rmsnorm_mq_rotate.hip (-DHIPFIRE_RMSNORM_AWQ) instead of a stale LDS-staged fork: 18.6 → 13.3 µs at K=5120 on gfx1201, +2.5% decode on every AWQ'd MQ4V2 artifact (fresh xt 35.6 → 36.5). Removes the gfx1100-only _direct variant and HIPFIRE_GFX1100_AWQ_NORM_DIRECT.
FP8-LUT residual odd-tail padding seeded from Y (residual semantics); qkvza/gate_up/qkv LUT select keyed on the wave-uniform slab start row (SGPR; launchers require m % 16 == 0).

Hugging Face

hipfire-models/qwen3.8-27b: mq4l-xt / mq4l / mq4l-pro added; mq4-xt re-issued with its AWQ sidecars (the prior upload lacked them: KLD 0.057 → 0.048; new sha256 80e7c624…, pinned in AGENTS.md §5); mq4 and mq4-pro byte-identical to the fresh v2 tiers, untouched. Model card updated.

Validation

KLD ladder above; 1-chunk prefill-vs-per-token agreement on every kernel change.
scripts/serve_harness.py battery on mq4l-xt (serve path, --speculation off): 5/5 stop, runaway/empty/attractor/retrieval_miss 0, decoded text read.
Greedy generation on mq4l vs mq4: coherent, correct.
cargo test --lib on rdna-compute / hipfire-dispatch / hipfire-runtime / hipfire-quantize / hipfire-arch-qwen35; crate maps + coverage tests; rustfmt on changed files.

Known

Intermittent bench runs land in an eager (non-graph) state (~29 tok/s flat) regardless of cache warmth; HIPFIRE_AR_GRAPH_TRACE=1 shows capture+replay on re-run. Harness artifact under investigation; not caused by this branch (seen on shipped-xt/master arms too).

…id-mul producers gated_norm_f32_batched and sigmoid_mul_f32 rewrite a stable batched scratch (dn_normed_batch / fa_attn_out_batch) every layer without dropping the pointer-keyed ensure_fp16_x cache. The Q8 residual WMMA consumer (gemm_q8_0_residual_wmma_gfx12, i.e. the mq4-pro tier's out_proj) then reuses layer L's converted activations at layer L+1. The F16 MQ4V2 GEMM launchers happened to evict the single-slot cache between layers; with all three gfx1201 FP8 prefill paths default-on (#757) nothing evicts it, so mq4-pro batched prefill went incoherent: WT2 KLD 6.42 -> 0.019 (1 chunk), 24-chunk 0.0335 after the fix. Any single FP8 flag off masked it. Producers now call invalidate_x_caches_for(out), matching the gemv.rs producer contract.

…book in the MQ4V2 container Quantizer: --format mq4v2-lloyd emits qt=52 tensors (byte-identical 136 B MQ4V2 groups) plus F32[16] <tensor>.lloyd_levels sidecars: a per-tensor 16-level Lloyd-Max codebook, constrained to centered levels {7.5+e : e finite E4M3} so the FP8 prefill path and the f16 decode path share one codebook exactly. AWQ sidecars compose as for qt=44. Runtime: loader rewrites group zero-points (zp' = f16(zp + 7.5*sc)) so both paths decode w = sc*C[q] + zp'; lloyd_lut.rs derives the E4M3 [u32;4] and f16 [u32;8] LUTs. Kernels: LUT twins of the FP8 WMMA prefill family (-DHIPFIRE_FP8_LUT_ARG), of the MQ4V2 GEMV/residual decode kernels and of the fused gate_up/qkv/qkvza decode kernels (-DHIPFIRE_MQ4G256V2_LUT), selected by the WEIGHT dtype; every route fails closed on a missing LUT. Dispatch: three Fused*Mq4G256V2Lloyd keys guarded on lloyd_lut_f16, the SwiGLU-residual variant admits qt=52 through the LUT residual kernel. Fixes found on the way: - FP8-LUT residual wrapper: odd-tail padding copied an uninitialized temp over Y (dropped the residual stream on every odd-sized prefill chunk). - Lloyd qkvza FP8: per-slab codebook select keyed on the wave-uniform slab start row (SGPR) instead of the lane row; re-admits the S2BT8 geometry (launcher requires all four m % 16 == 0, fails closed otherwise). - fused_rmsnorm_mq_rotate_awq is now derived from fused_rmsnorm_mq_rotate (-DHIPFIRE_RMSNORM_AWQ): same prefetch/reduction tree, 13.3 us vs the retired LDS-staged fork's 18.6 us at K=5120 on gfx1201 (+2.5% decode on every AWQ'd MQ4V2 artifact). Removes the gfx1100-only _direct variant and HIPFIRE_GFX1100_AWQ_NORM_DIRECT. WT2 KLD (24 chunks, gfx1201, prefill scoring, q8 KV): xt 0.0482 -> 0.0408, base 0.0405 -> 0.0346, pro 0.0335 -> 0.0284 vs the AWQ'd uniform tiers. Decode gfx1201 xt: 36.2 vs 36.5 tok/s uniform; prefill pp512 1374 vs 1412. Signed-off-by: Kaden Schutt <151092359+Kaden-Schutt@users.noreply.github.com>

…ne, docs, CHANGELOG Kernel: gate_up and qkv FP8-LUT symbols now select the per-tensor codebook once per 16-row slab keyed on the wave-uniform slab start row (as qkvza already did), replacing the per-lane candidate evaluation (mq4_pick2/3). Host launchers fail closed unless every projection m is a multiple of 16. pp512 gfx1201 per-kernel vs uniform: gate_up +1.8% (was +3.0%), qkv +2.0% (was +7%), total prefill -1.6% (was -6.4%). 1-chunk WT2 KLD prefill unchanged (L-base 0.020986). CI: crate maps regenerated (quant_mq4v2_lloyd.rs, lloyd_lut.rs), coverage_tests anchors for MQ4G256V2Lloyd and the three Fused*Lloyd keys. Docs: QUANTIZATION.md qt=52 inventory, quant-formats/mq4-v2.md §11 Lloyd variant, CHANGELOG Unreleased, stale references to the retired AWQ rmsnorm fork rewritten. Plan: docs/plans/2026-09-15-lloyd-fp8-prefill-gap.md. Signed-off-by: Kaden Schutt <151092359+Kaden-Schutt@users.noreply.github.com>

…on on gfx1151 T0 (gate-first): gemm_mq4g256v2_residual_wmma.hip gains a -DHIPFIRE_MQ4G256V2_LUT arm (LDS-staged 16-entry f16 codebook, decode w = sc*C[q] + zp'), symbol gemm_mq4g256v2_residual_wmma_gfx11_lloyd, launcher gemm_mq4g256v2_residual_wmma_gfx11_lloyd (base tile, fail-closed off gfx11 wave32 WMMA / k%256). Uniform symbol's preprocessed output unchanged. Throwaway differential: examples/tmp_lloyd_gfx11_diff.rs (to be deleted before merge). Gate criteria and the rest of the port: docs/plans/2026-09-15-lloyd-gfx11-port.md. T1: mq4g256v2_scalar_fusion_ok admits gfx1151 (fused Lloyd decode kernels are scalar wave32); gfx1150/1152 refusal tests; stale fused_qkv.rs comment. Signed-off-by: Kaden Schutt <151092359+Kaden-Schutt@users.noreply.github.com>

…fx1151 decode admission T0 gate on the XTX: F16-LUT residual twin +61% vs the uniform F16 base tile (N=384), +71% vs the production MMQ route (N=512); criterion was +10%. Correctness passed (rel-L2 2.7e-4 vs LUT GEMV). Kernel arm, const, launcher and throwaway reverted; plan annotated. Lloyd tiers stay gfx12-only for batched prefill (fail-closed elsewhere); gfx1151 fused decode admission (T1) retained. Signed-off-by: Kaden Schutt <151092359+Kaden-Schutt@users.noreply.github.com>

… KLD, rustfmt on changed files Registry: files qwen3.8-27b.mq4l{-xt,,-pro}, tags qwen3.8:27b-mq4l{-xt,,-pro}, WT2 KLD in descs, MQ4 DFlash sidecar pairing, mq4-xt re-issued as the AWQ'd artifact (KLD 0.048, 14.99 GB). Docs/CHANGELOG follow the naming. rustfmt applied to every .rs changed vs master (CI's changed-file gate). Signed-off-by: Kaden Schutt <151092359+Kaden-Schutt@users.noreply.github.com>