Deprecate XNNPACK fp32 variants that have a faster, smaller quantized twin #1456

From the v0.10.0 benchmarks in #1363 (my SM-S948B estate, Bartek's Pixel 10 / iPhone SE 3 / iPhone 17 runs).

17 XNNPACK fp32 variants have a quantized XNNPACK twin that is faster and smaller, and 15 LLM bf16/fp16 variants have one too. They are candidates for deprecation once output quality is checked. A further 63 fp32 variants have no quantized twin at all and need one exported first.

Speed and size are from SM-S948B. median ms is the whole pipeline, MB is download size. No accuracy data exists yet for any row here.

Tier A: deprecate after a quality check (24)

Vision (9)

Segmentation is 777 MB of fp32 against 206 MB of int8. Needs mIoU. Style transfer needs a visual diff only.

LLMs (15)

Every non-quantized LLM variant in the registry has a quantized twin in the same family, so all 15 qualify. Sizes are the real .pte content-length on Hugging Face at v0.10.0. Only 2 of the 15 were benchmarked: the bf16 tier was abandoned mid-run because the first one took 34 min of device time.

bf16 (12), 49.8 GB against 18.2 GB

qwen2-5-0-5-b-xnnpack-bf16 ran at 0.5 tok/s against 27.1 for its own 8da4w twin, measured minutes apart on the same device, and it emitted all 63 tokens so it is not a truncated generation. The one fp16 pair costs only 2.2x, so 58x is not the price of precision: bf16 is off the fast kernel path. Treat this as a defect, not a tradeoff. The other 11 bf16 rows are unmeasured, but they share the export path, so the finding very likely generalizes.

fp16 (3), 6.6 GB against 2.0 GB

fp16 is a weaker case than bf16 and should be judged on quality, not speed. 2.2x slower for 3x the download is a defensible tradeoff if the output is measurably better, which nobody has checked.

Tier B: probable, smaller margin (8)

The whisper .en models need WER on a held-out set. int8 is already the iOS default for them, so the fp32 build is opt-in traffic.

The two 8da4w embedders look mis-exported. 414 MB against 487 MB fp32, and 396 MB against 544 MB. A 4-bit weight variant that is only 1.2x to 1.4x smaller suggests most of the model was not quantised. Worth checking before treating 8da4w as the replacement.

Keep fp32 (4)

Quantising these buys nothing and only risks accuracy.

Quantize first: 63 fp32 variants with no quantized twin (12.3 GB)

Priority:

1. whisper multilingual tiny / base / small. whisper-small is the slowest non-LLM model in the estate at 2121 ms and 1133 MB. The .en twins already have int8, so the export path is known.
2. yolo26 + rfdetr + fastsam. 38 variants, 3.6 GB, and the entire detection / instance-segmentation / keypoint offering is fp32 only.
3. mpnet embedders. 437 MB each, fp32 only on XNNPACK. A Vulkan int8 exists but Vulkan is Android only.

Evidence caveat

The fp32-vs-quantized head-to-head exists on SM-S948B only. None of Bartek's three tables contain an xnnpack-fp32 row for any family that also has an XNNPACK quantized twin, so no second device confirms these ratios. That absence is itself signal that fp32 is not what those platforms default to.

iOS is not stranded by a removal: on iPhone 17 CoreML fp16 runs deeplab-v3-resnet50 at 103.1 ms and candy at 63.2 ms, both well ahead of XNNPACK int8 on the same device.

Proposed rule

Not "remove every non-quantized XNNPACK model", because 63 of them are the only build that exists for their model and 5 more are better off as fp32. Instead:

An unquantized build (fp32, fp16, bf16) does not ship where a quantized twin on the same backend is at least as fast and materially smaller, and where output quality has been shown to hold.

Steps

• Accuracy comparison against fp32 per candidate (mIoU for segmentation, WER for whisper, cosine similarity for embedders, visual diff for style transfer)
• Check HF download counts per variant so anything with real traffic gets a deprecation window
• Re-check the 8da4w text embedder exports
• Deprecate in the registry with a console warning for one minor, then remove
• Root-cause the bf16 kernel path before deprecating on speed alone; if bf16 is fixable it changes the argument for those 12 rows
• Export quantized variants for the priority families above