[jason-fxz]

The CPU MoE executor's nvfp4 W4A8 dots were compute bound per core. With weights in L2, main's AVX-512 VNNI kernel does 13.4 GB/s per core, about what each of 15 cores needs to read 189 GB/s, and its AVX-VNNI (256-bit) kernel 9.3 GB/s. This PR keeps the int8 math and rewrites the data path of both kernels with about 3x fewer instructions per weight, so both reach the memory read bandwidth.

Part of #671.

Changes

• Activations are quantized once per token into chunks of 8 blocks (16 K each), laid out as [even-K int8 | odd-K int8]. One 64-byte weight load then splits into low and high nibbles with two pshufb that line up with the activations, with no interleave per row.
• The e2m1 LUT carries a +12 bias, so the weight is the unsigned vpdpbusd operand as is (no sign trick). -12 * sum(a) per lane is computed with the quant and seeds the accumulator.
• e4m3 block scales decode with a shift into fp16 plus vcvtph2ps, 16 (AVX-512) or 8 (AVX2) per instruction; the 2^-8 left by the shift is folded into the final scale.
• Both activation quants (the layer input and the down input) run with AVX-512 / AVX2 instead of a scalar lround loop, with the same int8 values and scales: 4.2-4.7x faster. At 2048x512, top-8, bs = 4 they took about 9% of an executor step.
• The W4A8 kernel follows FREETOKEN_CPU_MOE_ISA (main ran the 512-bit kernel at any tier). CPUs without VNNI keep the fp32 W4A16 path unchanged.
• test_cpu_decode_nvfp4_matches_dequant_then_gpu runs each FREETOKEN_CPU_MOE_ISA tier (scalar, avx2, avx512; a tier the host lacks is skipped), so an AVX-512 host also checks the 256-bit kernel. It uses I = 752 (47 blocks of 16), so the down dots run their partial-chunk and scalar tails.

Results

Xeon Platinum 8559C (Emerald Rapids, 16 cores, KVM guest), 15 CPU MoE threads, RTX PRO 6000 Blackwell Server Edition, driver 595.91.07; main = ee43b2a. Read ceiling (AVX-512 stream, 15 threads): 189 GB/s.

One dot, K = 2048, GB/s of weights + scales, main -> PR: AVX-512 VNNI 13.4 -> 34.4 (1 core, L2) and 169 -> 184 (15 cores, DRAM); AVX-VNNI 9.3 -> 25.5 and 94 -> 180.

ft bench bw --model <checkpoint>, CPU-MoE GB/s, median of 4 runs (AVX2 tier: 3):

layer kind	AVX-512 tier	AVX2 tier (FREETOKEN_CPU_MOE_ISA=avx2)
2048x512 top8 (Qwen3.6-35B-A3B)	113 -> 149 (1.32x)	84 -> 147 (1.74x)
2560x640 top10 (Qwen3.8-Flash-Next)	118 -> 145 (1.23x)	90 -> 148 (1.64x)
2816x704 top8 (Gemma-4-26B-A4B)	138 -> 188 (1.36x)	100 -> 182 (1.81x)

With the faster CPU side the hybrid backend fetches 19-26% of the misses over PCIe instead of 29-32%.

Hybrid decode, aggregate tok/s of bs concurrent streams, main -> PR, median of 2 server starts x 2 rounds:

checkpoint, --moe-cache-rate, tier	bs=1	bs=2	bs=3	bs=4
nvidia/Qwen3.6-35B-A3B-NVFP4, 0.1, AVX-512	136 -> 150 (1.10x)	180 -> 203 (1.13x)	201 -> 235 (1.17x)	258 -> 303 (1.17x)
nvidia/Qwen3.6-35B-A3B-NVFP4, 0.25, AVX-512	155 -> 168 (1.08x)	212 -> 239 (1.13x)	252 -> 300 (1.19x)	317 -> 381 (1.20x)
nvidia/Gemma-4-26B-A4B-NVFP4, 0.1, AVX-512	107 -> 119 (1.11x)	151 -> 170 (1.12x)	152 -> 187 (1.23x)	218 -> 244 (1.12x)
nvidia/Gemma-4-26B-A4B-NVFP4, 0.25, AVX-512	126 -> 129 (1.02x)	189 -> 210 (1.11x)	226 -> 264 (1.17x)	295 -> 351 (1.19x)
RadixArk/Qwen3.8-Flash-Next-NVFP4, 0.1, AVX-512	63.2 -> 65.3 (1.03x)	95 -> 102 (1.07x)	102 -> 113 (1.10x)	138 -> 153 (1.10x)
RadixArk/Qwen3.8-Flash-Next-NVFP4, 0.25, AVX-512	69.0 -> 71.3 (1.03x)	112 -> 118 (1.05x)	134 -> 144 (1.08x)	180 -> 195 (1.08x)
nvidia/Qwen3.6-35B-A3B-NVFP4, 0.1, AVX2	128 -> 148 (1.16x)	166 -> 201 (1.21x)	177 -> 231 (1.30x)	232 -> 297 (1.28x)
RadixArk/Qwen3.8-Flash-Next-NVFP4, 0.1, AVX2	59.8 -> 65.3 (1.09x)	89 -> 102 (1.15x)	91 -> 112 (1.22x)	125 -> 153 (1.23x)

Each side benched its own ft bench bw profile first. The AVX2 rows run main with FREETOKEN_CPU_MOE_ISA=avx2 FREETOKEN_CPU_MOE_NO_AVX512VNNI=1 (its AVX-VNNI kernel) and the PR with FREETOKEN_CPU_MOE_ISA=avx2. Server: ft serve --model <checkpoint> --moe-backend hybrid --moe-cache-rate <r> --max-running-requests 4 --cuda-graph-max-bs 4 --cache-type naive; client: bs streamed /v1/chat/completions requests (AIME-25 problems 0..bs-1, thinking on, greedy, ignore_eos, 512 tokens), one warm round, throughput over the window where all streams decode. Spread across server starts is 1-2%. Qwen3.8-Flash-Next gains least: its ~15 ms step is mostly GPU work.

Behavior changes

• FREETOKEN_CPU_MOE_ISA=avx2 on a CPU with AVX-512 VNNI but no AVX-VNNI (Ice Lake-SP, Zen 4) runs the fp32 W4A16 kernel instead of the 512-bit W4A8 one; =scalar runs the scalar fp32 kernel.
• A ft bench bw profile from main still loads, since the ISA labels are unchanged, but it understates the CPU side. Rerun ft bench bw to get the lower fetch fraction.

Testing

• pytest tests/moe/test_cpu_moe.py tests/moe/test_cpu_moe_fp8_block.py tests/moe/test_mixed_expert_formats.py: 53 passed, 3 skipped (HIP only). The nvfp4 tests also pass with FREETOKEN_CPU_MOE_NO_VNNI=1 and with FREETOKEN_CPU_MOE_NO_AVX512VNNI=1. A wrong chunk offset in the 256-bit scalar tail fails the three avx2 cases of the parity test.
• Both dots against a float64 W4A8 reference, K = 48..5120, with positive and with random-sign e4m3 scales: max relative error 1.8e-7 (main: 3.6e-7). The SIMD quants give the same int8 values and scales as main's scalar quant, also for all-zero blocks and for blocks whose absmax is below ~3.7e-37 (both quantize to zeros).
• Not run on a CPU that has AVX-VNNI but no AVX-512 (Alder Lake and later), where the 256-bit kernel is the default; here it runs at FREETOKEN_CPU_MOE_ISA=avx2, which the parity test now covers on AVX-512 hosts too.