Repository navigation
Add native Metal support for LFM3-1.7B and LFM3-VL-2B MLX 4/6/8-bit checkpoints #1491
Description
Goal and current state
Add native Metal support for LiquidAI LFM3-1.7B and LFM3-VL-2B on Apple Silicon, using the official MLX affine 4-, 6-, and 8-bit checkpoints.
The implementation is on BlivionIaG/TensorFold:feature/lfm3-vl-support, in commit aa3d4c265. It implements text generation, native image input, shared lane verification, prompt snapshots, concurrent requests, and cancellation recovery. It preserves stored quantization, BF16 activation boundaries, and FP32 reductions.
This is an experimental candidate tested on an M4 Pro. Maintainer agreement on model-fidelity and performance gates remains open; this issue does not claim release qualification.
Architecture
flowchart LR
T[Text tokens] --> E[Embedding rows]
I[Image pixels] --> V[LFM3 vision encoder and projector]
V --> E
E --> D[Shared LFM3 decoder]
D --> L[Existing lane engine]
L --> O[Generated tokens]
Both models use the same 24-layer decoder geometry: width 2048, FFN width 7680, 16 query heads, 4 KV heads, and head dimension 128. The port implements seven-tap gated convolution, full attention, 256-token sliding attention, per-head attention gates, sandwich RMS norms, and the checkpoint's partial-RoPE formula.
The VL model adds its own 14-layer hybrid vision encoder, axial RoPE, spatial convolution, and projector. It uses the existing native image interfaces and content-aware prompt cache. It does not add a second scheduler or a Python inference path.
Checkpoint handling
- All six official exports were inspected at pinned revisions, including actual safetensors headers.
- Text 4/6/8-bit and VL 6/8-bit use affine groups of 64.
- VL 4-bit uses groups of 32 and an 8-bit embedding override.
- All 138 vision/projector tensors are BF16 in each VL export.
- The loader checks tensor names, shapes, dtypes, byte ranges, and module-specific widths before executing kernels.
The recipe records the exact revisions and reproduction commands: LFM3 native Metal bring-up.
Validation
Environment: Apple M4 Pro, 48 GB RAM, macOS 26.5.1, Zig 0.17.0, MLX 0.32.4.
- All six checkpoints: 186 served checks passed, with zero failures or skips. Plain/drafted and concurrent 1/2/4-stream outputs matched complete token hashes. Warm repeats reused 896 of 910 prompt tokens and matched fresh output.
- All six checkpoints: 60 cancellation/recovery checks passed. Fifteen actual HTTP cancellations were confirmed by the server metric, followed by 30 plain/drafted responses matching fresh token hashes.
- VL at all three widths: red and blue images, both multi-image orders, concurrent images, and corrupt-PNG rejection passed.
- 1.7B 4-bit: all 68 default real-weight exactness cases passed. These cover prompt chunks, sliding-window seams, partial acceptance, unequal shared streams, snapshots and reset.
- Metal operators: all 27 LFM3 variants passed their declared gates and limits. All seven planted controls were red as intended. Dense/projector stress cases distinguish FP32 accumulation from reduced precision.
- Repository tests: 7,231 passed, 45 skipped. Host contract/golden/fuzz checks passed, with no unexpected server-golden differences. Native and default builds passed.
- Lean: 428 inherited findings, zero new findings.
The complete global GPU runner cannot finish on M4 because an unrelated Gemma case requires M5 tensor units. Its compiler preflight was exercised with a complete measured rotary receipt; no guard was bypassed. The targeted LFM3 and control runs are not described as a passing global suite.
The independent text reference matched all seven next-token choices in the initial public prompt, with byte-identical embeddings. On the 256-patch synthetic vision fixture, patch projection and embedding RMS were byte-identical; final projected rows had relative RMS difference 0.026 against the independent MLX reference. This observation is not an accepted fidelity threshold.
Raw receipts remain local. The committed reference, exactness, served, and contract tools reproduce the checks.
Remaining qualification
- Agree the independent model-fidelity and served-performance acceptance gates.
- Expand teacher-forced fidelity checks, especially image-conditioned text, nonsquare images, and long positions.
- Run the optional long exactness suite, maximum image budgets, and other Apple GPU generations.
- Strengthen accumulation witnesses for nonlinear operators whose BF16 outputs hide reduced-precision sum error.
Video input and logprobs are not implemented. Structured output needs the native grammar library. JPEG/WebP input needs the matching decoder libraries; JPEG pixel fixtures were unavailable in this test environment.
Detected additive conflicts with PRs 1492, 1235, 1457 and local worktrees were removed by relocating registrations without changing model arithmetic. The final ATC scan reports no detected conflicts; some merge coverage remains unknown under its reported myPullRequestsLimit.
Activity
BlivionIaG commentedon Oct 27, 2026
/atc claim
feat(lfm3): add strict MLX checkpoint admission checks
jschmied commentedon Oct 27, 2026
I'd like to take the CUDA side of LFM3, the same way #1492 brings your LFM2.5 work (#1377) to CUDA. It would stack on your branch and follow its config and tensor admission (
8192e5437), so both backends share one admission and one family definition. I'd start the engine only once your Metal text decoder is in place, so the CUDA port follows your choices instead of guessing them.Most of it exists on CUDA already: the MLX affine linears, the norms,
qk_rows, and paged attention at head dim 128. The new parts would be the 7-tap gated conv (#1492's conv kernel is 3-tap; I'll make the tap count a template parameter), the sliding-window path, the per-head attention gates and, later, the vision tower. Meanwhile I'm preparing locally: a float64 reference for the text decoder checked against the officialmodeling_lfm3.py, and the kernel variants that don't depend on the family code.If you'd rather keep CUDA in this issue or have it split out, either works for me.
feat(lfm3): implement native Metal text and vision inference
BlivionIaG commentedon Oct 27, 2026
/atc release
jschmied commentedon Oct 27, 2026
If the CUDA side is free, I'd take it here (or in a separate issue, if you'd rather keep this one Metal-only). I'd start with LiquidAI's NVFP4 and FP8 text checkpoints, which map onto TF's existing CUDA NVFP4 and block-FP8 linears, then the MLX affine exports, reusing #1492's kernels. If you're continuing the Metal port, I'll keep following your admission (
8192e5437) so both backends share one family definition.BlivionIaG commentedon Oct 27, 2026
Feel free to do so
jschmied commentedon Oct 27, 2026
/atc claim
jschmied commentedon Oct 27, 2026
Thanks. Claiming the CUDA side here. NVFP4 text comes first, weight-only, then the MLX affine exports and FP8, and VL after that. It follows your admission (
8192e5437); I'll add the NVFP4/FP8 tensor kinds to the shared schema so Metal refuses those exports cleanly.BlivionIaG commentedon Oct 28, 2026
/atc presence ff75668d6288fb62abae5fe779ef60f69c56e0db57bcd26d98886f36c07d18b9
ashhart commentedon Oct 28, 2026
A status note: the gates are answered on #1502 (fidelity against an FP64 forward of the dequantized checkpoint, with every top-1 difference listed; speed against mlx-lm, or mlx-vlm for VL, at each width). The work is now split as asked: admission in #1513, the text decoder in #1514 and VL in #1502 on top, with @jschmied's CUDA side in #1508 on the same admission.
feat(lfm3): add strict MLX checkpoint admission checks
cwschroeder commentedon Oct 29, 2026
@BlivionIaG @jschmied, we would like to help with the Metal text port's remaining long-context qualification on an M3 Ultra.
Proposed scope, on an agreed pinned candidate:
The initial target would be the text slice in #1514. Which head and checkpoint width would be most useful to qualify first? We can start with 4-bit and extend to 6/8-bit after the first matrix. This offer covers independent qualification and resulting agreed fixes; your decoder and CUDA work keep their current owners.
cwschroeder commentedon Oct 29, 2026
/atc propose pair
BlivionIaG commentedon Oct 29, 2026
@cwschroeder Yes, please proceed with independent M3 Ultra qualification of the Metal text slice in #1514, pinned to
bdfd263c33178ed9a67322c0d709d6322ec03f32. Start withLiquidAI/LFM3-1.7B-MLX-4bitat the checkpoint revision recorded indocs/recipes/lfm3-native.md, then extend to 6/8-bit as available. Please retain full token hashes, failures and unrun cells, and coordinate any shared-file fixes before editing.We are currently running long-context qualification on an M5 Ultra: VL 6/8-bit at 32k/64k, text 4/6/8-bit at 131072 and 262128 prompt tokens, and VL 4/6/8-bit at 127984 prompt tokens, with 16 output tokens and repeated fresh-process runs. The matrix is still in progress; these are planned cells, not completed results.
ATC's acceptance preview returns
notAuthorizedfor our account because the issue claim is held by @jschmied. @jschmied, please accept pairing proposal 6487991897 for this independent qualification scope. Metal and CUDA implementation ownership stays unchanged.jschmied commentedon Oct 29, 2026
/atc accept 6487991897
feat(lfm3): add strict MLX checkpoint admission checks