Ling-3.0-tiny on a Qualcomm board: a Ling family on Metal first, then an ARM CPU backend #1247

Activity

1. jschmied commented on Oct 25, 2026

   

   jschmied

   on Oct 25, 2026

   Contributor

   More actions

   Thanks for the pointer. I ran Ling-3.0-tiny on #874's CUDA path, and q_lora_rank 256 was the only config value it refused (plus null SwiGLU limit lists). The change is a 256-wide RMS-norm instance for q_a_layernorm; the q projections are ordinary lane matmuls. The rest of tiny's config (24 layers in groups of 4, 6 MLA and 18 KDA, dense layer 0, 128 experts in 8 groups of 16 keeping 4, 16 heads, hidden 1536) already went through the existing kernels.

   What #874's CUDA path now covers for tiny, on one GB10 (bf16, TP=1, 9a98e35f):

   • Whole model, served through the lane core. Logits of 480 real tokens are 1.05e-2 (rel L2) from a float64 forward on the same routing, as far as the HF model's own bf16 rounding is (1.04e-2); the transformers model itself is 3.3e-2 from float64. Greedy chat matches transformers token for token over 64 tokens. Chunked == whole, drafted == plain, resumed == fresh, and concurrent == solo at 1, 2 and 4 streams.
   • 76.6 tok/s decode at one stream, 149-157 tok/s aggregate at four, 1.74 s for an 8,189-token prompt. These are GB10 numbers; they are not comparable with your board's 20 tok/s (CPU) or 14 tok/s (NPU).
   • Float64 reference (ling_forward.py, reads the sharded checkpoint), a logits dump for external references (ling-ids-logits), tiny's config and index names as fixtures, and a synthetic tiny-shaped checkpoint writer (ling-synth --tiny) for bf16, FP8 and NVFP4. These are what a Metal or Arm port can check against without the 15.8 GB.

   What it does not cover:

   • No Metal. The CUDA kernels (KDA, MLA with RoPE and head gate, group-limited routing) are CUDA C; your plan's eight Metal PRs are still needed, and the fp64 reference and fixtures above would serve them.
   • No Arm CPU, Adreno or Hexagon backend.
   • The board has about 14 GB visible and tiny in bf16 is 15.8 GB. ling: Ling-3.1-flash on CUDA at TP=1 on #804's kernels (draft, stacked on #804) #874 reads bf16, block FP8 and ModelOpt NVFP4. The 4/6/8-bit expert formats you list, and a 4.8 GB Q4-class file, are not read by it; EXL3 waits on EXL3 checkpoints on the native engine (tracker) #832. The official FP8 tiny has not been run on real weights (only synthetic tiny-shaped files).
   • bf16 experts have no grouped kernel; the speed above comes from running each expert on just the rows that picked it, which needs a host sync per layer. A Metal or NEON port would want a real grouped kernel.
   • The server needs possessive-quantifier support in the tokenizer's regex, which main has since 79ba3f58c.
   • ling: Ling-3.1-flash on CUDA at TP=1 on #804's kernels (draft, stacked on #804) #874 is a draft with conflicts against main; the tiny changes are five commits on top of it, on ling-tiny.
2. ashhart commented on Oct 27, 2026

   

   ashhart

   on Oct 27, 2026

   Owner

   More actions

   Thanks, @ThinkOffApp, and @jschmied for running tiny through the CUDA path. Yes to Ling-3.0-tiny on Metal: an MIT licence and a float64 reference are what a new family needs. Please build it as one family package that CUDA can share, starting from the config, weight map, float64 reference and fixtures on jschmied's ling-tiny branch, with credit, rather than writing them again, and keep each PR to one job with its gate. Weights are read as published (bf16, the official FP8, or an existing MLX quantisation); we don't ship conversions. The board backend (Arm CPU, Adreno, Hexagon) and Living Weights on Ling are separate proposals for after Ling serves on Metal. Your first PR is the next step.