LFM3-Omni-2B on Metal and CUDA: speech in, streamed text and speech out (gate and split) #1627

Proposal

Serve LiquidAI's LFM3-Omni-2B (speech in, streamed text and 24 kHz speech out) natively on Metal and CUDA as a standalone family, families/lfm3_omni, in the shape of Drama 3's standalone engine. Before opening pull requests, this asks to agree on the gate and the split.

The family doesn't depend on the LFM3 text series (#1508, #1513, #1514): one generic Liquid stack drives the model's four networks, all in bf16 as released:

• the speech encoder (LFM2 layers) and its projector;
• the backbone, with interleaved text and audio tokens;
• the depthformer audio head;
• the LiquidCodec decoder.

It's built as zig build lfm3-omni-generate. lfm3-omni-generate --serve HOST:PORT serves:

• POST /v1/audio/chat, taking a base64 WAV. It streams server-sent events: text deltas, 16-bit 24 kHz PCM chunks, then a done event with timings. Without streaming, it returns the whole reply as one JSON.
• GET /health.

It serves one request at a time.

Gate we propose

Parity against the checkpoint's own code (transformers with liquid-audio) on the same machine, with 16 synthetic utterances (generated voices only), greedy, stage by stage. The figures are Metal on an M5 Ultra against MPS bf16, then CUDA on a GB10 against CUDA bf16:

• Precision: as in d1-2B (LFM3-VL decisions) on native CUDA: bf16 text decoder, LFM3-VL tower, set attention #1581, we propose "no further from an fp32 run than the reference's own bf16 is". On Metal against MPS fp32, the encoder and the prefill rows sit exactly where the reference's bf16 sits (0.999995 and a 0.99988 median).
• Text and code misses are near-ties: every miss is within 2 bf16 ulps of the reference's pick. 15% of the reference's own greedy codebook picks are exact bf16 ties, so free-running codes diverge after a frame or two, and identical audio codes aren't reachable on either backend.
• Quality instead of identical audio: UTMOS22 and word error between the reply's speech and its own text:

• Streamed vs whole codec decode: 34–43 dB SNR, not bit-identical. The first one-frame chunk takes a different product kernel than the eight-frame chunks; using one product for every chunk size would make it exact, if you want that.
• Kernel contract: every new Metal and CUDA kernel registered against a host replay of the reference's rounding points, with a planted control each. This isn't done yet.
• Lean checker and host tests: zero problems added over the base; host tests for the family.

Latency

The same 16 turns through the server, with a streaming client, median:

• Machines were shared: both had other GPU work running during these measurements, so we alternated runs and took two-pass medians. We'll redo them on an idle machine for the receipts.
• Metal: prompt products run on the M5 tensor units. Earlier chips fall back to a simdgroup GEMM that is correct but slow (about 1.6 s of prefill) and untested on that hardware.
• CUDA: the backbone step is bandwidth-bound (2.9 GB of bf16 weights per step). There are CUDA graphs for the backbone step and the speech frame, a few-row GEMV, and mel on the GPU.

Questions

1. Split. Would you take two pull requests (Metal first, then CUDA), or three (the shared family with host tests, then each backend)?
2. Server. Is a standalone tool with its own /v1/audio/chat fine for now, as with Drama 3? Or would you rather have it as a route of tensorfold-native serve (OpenAI input_audio in, audio out)?
3. Exactness of the streamed codec. Should streamed chunks equal a whole decode bit for bit, at a small cost in first audio?
4. Gate. Is the stage table above, plus UTMOS and WER on the replies, the right shape?

Work in progress

drowzeys/TensorFold branches lfm3-omni-cuda and lfm3-omni-metal. They currently sit on our d1-2B work (#1581); for pull requests we'll rebase the family alone onto main. Licence note: the weights are under the LFM Open License v1.0, which allows commercial use only for organisations under US$10M annual revenue. The recipes say so.