Skip to content

docs: the 27B recipe names its native CUDA backend and which formats it reads #1146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jschmied wants to merge 1 commit into from
Closed

docs: the 27B recipe names its native CUDA backend and which formats it reads #1146

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/recipes/qwen3.8-27b.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

The native engine serves `TensorFold/Qwen3.8-27B-MLX-4bit` on Metal, qualified on M5 Max and M3 Ultra, from
`zig/src/families/qwen27/`. It drafts with DFlash2 (`zig/src/families/qwen27/dflash/`) only when `--drafter` names
the drafter; without it the model decodes plain. It has no CUDA backend yet.
the drafter; without it the model decodes plain. On CUDA the native engine serves the MLX affine checkpoints (2, 3,
4, 5, 6 and 8 bits in groups of 64) from `zig/src/families/qwen3_5/cuda_*.zig`, with DFlash2 drafts and
`--kv-cache fp8`; it reads no NVFP4 or EXL3 checkpoint yet.

```bash
tensorfold pull TensorFold/Qwen3.8-27B-MLX-4bit
Expand Down Expand Up @@ -83,6 +85,10 @@ quality comparisons are TBD [release-0.3.5].

## CUDA

This section is the Python engine's (0.6.6). The native CUDA engine reads only MLX affine checkpoints so far: the
NVFP4 and EXL3 readers below have not been ported to it (1.1.0 carries the shared EXL3 module of #832, which the
27B does not read through yet).

On CUDA the 27B serves NVFP4 and EXL3 checkpoints, and the MLX 4-bit checkpoint as the portable option: the same
files a Mac serves, and the only format two ranks read. `tensorfold serve` loads the checkpoint you name; it picks
none by itself. Use the [CUDA container setup](https://github.com/ashhart/TensorFold/blob/v0.6.6/RUNBOOK.md#nvidia-gpus)
Expand Down
Loading