CRUX parity spike: apr serve vs ollama on Qwen3.5 (0.8B/2B/4B/9B/27B) — measure gap, name techniques, file fixes #4220

Activity

1. noahgift

   added this to the 0.69.3 milestone on Sep 24, 2026
2. noahgift commented on Sep 24, 2026

   

   noahgift

   on Sep 24, 2026

   MemberAuthor

   More actions

   #4220 (ollama CRUX) part 1: source-level techniques, no GPU numbers yet

   The GPU stays serialised until 0.69.3 G1/G2 and RC acceptance finish, so this part reads source only.

   • ollama: latest release v0.34.4 (b2da9e468af2). lambda runs 0.34.2, which is not the latest; the upgrade goes through infra-8d/forjar-d3 before part 2.
   • ollama's CUDA engine: llama.cpp b11081 (161755f29e41), per LLAMA_CPP_VERSION. ollama starts llama-server as a subprocess (llm/llama_server.go:351-445) and patches only the loader and clip (llama/compat/001-llama-cpp-hooks.patch). So the kernel-side techniques below are llama.cpp's; the llama.cpp CRUX worker will find the same kernels. What is specific to ollama is its launch defaults.
   • apr: origin/chore/0.69.1-merge-back @ eed4a959a.

   0. Baseline caveat: the 149 tok/s decode ran a different GGUF, with MTP speculative decoding on

   The LQW-001 baseline ran ollama on the registry qwen3.5:4b (blob sha256:81fb60c7…). I parsed that GGUF's header: it carries 15 mtp.* tensors. The unsloth GGUF that apr ran (00fe7986…) carries 0.

   ollama switches MTP drafting on automatically for such a blob. hasMTPDraft/hasLegacyQwenMTPDraft return true for qwen35 when mtp.* tensors exist (llm/llama_server.go:836-851). Then EnableMTP=true (:900-907), and it adds --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-backend-sampling (:806-821). The default DraftNumPredict: 4 is at api/types.go:1147.

   So 149 vs 87 compares apr without speculation against ollama with MTP speculation on a different file. The same-GGUF row in part 2 will be ollama without MTP; the headline ratio for #4218 has to name which one it uses. ollama's defaults also include -b/-ub 512 (api/types.go:1144 → llm/llama_server.go:593-595), --flash-attn auto (:600-624) and cache_prompt: true (:2262).

   Techniques ollama/llama.cpp use that apr lacks, ranked by gain ÷ effort

   Not listed on purpose: fused conv1d+SiLU and in-place GDN state. apr already has both (forward_qwen35_cuda.rs:1336, :1392).

   #	Technique	Where in ollama / llama.cpp	apr today	Est. gain	Effort	Owner
   1	Batched prefill: one GDN launch per layer per ubatch (512 tokens), state held in registers across the token loop, plus MMQ GEMMs and flash attention on the 8 full-attention layers	ggml-cuda/gated_delta_net.cu:63 (token loop inside the kernel), :169-215 (launch: grid H×seqs); ubatch 512 from ollama api/types.go:1144. llama.cpp has no chunked GDN either (:180 TODO)	forward_single is the only entry point (forward_qwen35_cuda.rs:1480), so prefill is N decode steps. That fits the 77 tok/s prefill ≈ 87 decode	~100× prefill (8.2k vs 77)	M–H, in flight	#3596 (f5), no new ticket
   2	CUDA graph replay of the decode step	ggml-cuda.cu:4420-4440 (graph per key), :2595 (update check), :4185-4400 (capture and replay)	No graph on the qwen35 path, although aprender-gpu/src/driver/graph.rs exists. About 546 launches per token: 24 GDN layers × 16 (8 GEMV, gates, conv, delta, gated-norm, swiglu, 2 add, 2 norm) plus 8 attention layers × 20, plus norm and lm_head. GpuBuffer::from_host allocates a new device buffer every token (:1506), which blocks capture and adds a cudaMalloc per token	1.2–1.4× decode (launch overhead at ~3–5 µs × 546 ≈ 1.6–2.7 ms of the 11.5 ms token); the persistent embedding buffer is nearly free	M	#4215 (infra-76)
   3	Prefix reuse through recurrent-state checkpoints	llama-server: n_ctx_checkpoints = 32 per slot (common/common.h:629), create_checkpoint (tools/server/server-context.cpp:2309-2365, snapshot via llama_state_seq_get_data_ext(..., PARTIAL_ONLY) at :2363); ollama always sends cache_prompt: true (llm/llama_server.go:2262)	No prefix reuse: TTFT pays the full prompt every turn	TTFT on multi-turn: baseline "cached 37k" vs none	M	#4214 (infra-76)
   4	MTP speculative decoding with GDN state rollback	ollama auto-enable (§0); MTP block src/models/qwen35.cpp:36-101; the GDN kernel writes K rollback snapshots (gated_delta_net.cu:147-154), fused into the cache scatter (ggml-cuda.cu:2752-2800, :3454-3466); spec replay from checkpoints server-context.cpp:56,257,1246	None. The unsloth GGUF has no MTP head anyway; this needs a GGUF that carries mtp.* (the ollama blob does)	typically 1.5–2× decode at temp 0 (to be measured on the ollama blob with and without MTP)	H	new child: proposed to cf, not inside #4215
   5	Quantized-activation GEMV (MMVQ, q8_1 + dp4a) on decode	ggml-cuda.cu:1868 ggml_cuda_should_use_mmvq; mmvq.cu, quantize.cu	apr pins float GEMV on qwen35 (forward_qwen35_cuda.rs:562-568) because its own dp4a path was "catastrophic through the recurrence". llama.cpp runs MMVQ through the same recurrence, so apr's divergence is an apr quantization defect, not something inherent to the model	uncertain: decode at 87 tok/s is overhead-bound, not bandwidth-bound (4B Q4_K ≈ 2.7 GB ⇒ ~370 tok/s bandwidth ceiling on a 4090). Measure after #2	M	#4030 / #4215 (infra-76): numerics first
   6	Fused RMSNorm+mul+RoPE (+set_rows) on the attention layers	ggml-cuda.cu:2700, :3229-3249	separate per_head_rmsnorm_into + gdn_partial_neox_rope_into (2+2 launches per attention layer)	small (~32 launches per token)	L	#4215, folds into #2

   Not a gap vs ollama: apr downloads the full 248k-float logits vector every token (forward_qwen35_cuda.rs:1549, about 1 MB). llama.cpp does the same by default (common/common.h:296 backend_sampling=false; ollama turns backend sampling on only for the draft). A device-side argmax would be a "beat" item, not a parity item.

   Part 2 (after the GPU frees up)

   Same-GGUF table in #4218's row format: engine, version, GGUF sha, GPU, n≥3 with spread. Rows: prefill at 850/4k/32k, decode, TTFT, on ollama v0.34.4 against the unsloth 00fe7986… via a Modelfile FROM, plus a second ollama row on the MTP blob with and without draft_num_predict=0, to size technique 4. Correctness first: temp 0 token-identity, and a trace line proving the GPU path. Sources used here: ollama v0.34.4 and llama.cpp b11081 shallow clones under /mnt/nvme-raid0/tmp/crux4220/.
3. noahgift

   modified the milestones: 0.69.3, 0.70.1 on Sep 25, 2026
4. noahgift commented on Sep 25, 2026

   

   noahgift

   on Sep 25, 2026

   MemberAuthor

   More actions

   #4220 (ollama CRUX) part 2: same-GGUF, same-host table

   Setup. lambda, RTX 4090 (driver 580.119.02), under flock /tmp/apr-gpu.lock. One foreign compute app (936 MiB, containerised) was resident the whole time. Measured 2026-09-25 15:59–16:20 UTC (17:59–18:20 Madrid).

   • GGUF: unsloth Qwen3.5-4B-Q4_K_M.gguf, sha256 00fe7986ff5f6b46…, 0 mtp.* tensors.
   • apr: 0.69.5-rc.1 (5660b0877), the fleet-installed rc binary (sha256 4bfdacd2…), running serve run --gpu-layers all --context-length 40960.
   • ollama: v0.34.4 (latest). The same file goes in through a Modelfile FROM, with /api/chat, think:false and num_ctx 40960.
   • llama.cpp: llama-bench b11050, -ngl 99 -fa 1, 3 reps.

   GPU path proven:

   • apr: Backend: GPU (CUDA, NVIDIA GeForce RTX 4090…) [qwen35 hybrid forward] and gpu-layers: requested=all resolved=32 total=32, 0 fell_back, plus a batched prefill … attention cuBLAS f32 trace line for every request.
   • ollama: ollama ps read 100% GPU.

   Prompt tokens are identical across engines: 30 / 863 / 4082 / 32734. That means the same chat template and the same tokenizer output.

   metric (4B)	apr serve	ollama, same GGUF	apr ÷ ollama	llama-bench, same GGUF
   prefill 850	5389 tok/s (trace), TTFT 218 ms	5765 (server prompt_eval)	0.93	10267 ± 1319
   prefill 4k	7798 tok/s (trace), TTFT 582 ms	10166	0.77	11490 ± 326
   prefill 32k	3178 tok/s (trace), TTFT 10.43 s	12365	0.26	10099 ± 55
   decode, 128 tok	84.0–102.5 tok/s (SSE, 6 runs over 2 sessions)	172–201 (server eval; client SSE agrees within 2%)	~0.5	168.6 ± 5.6 (tg128)
   TTFT, 30-token prompt	44–66 ms	~110 ms prompt_eval, plus a 2.2–2.5 s model reload on every request (load_duration)	apr ahead	—

   Caveats:

   • ollama reloaded the model on every request under this harness (load_duration 2.2–2.5 s even with the Modelfile and request num_ctx equal). Its TTFT column therefore uses the server's own prompt_eval_duration, which excludes load. That is fair to ollama.
   • A first run under heavy host load read llama-bench tg128 at 96.8 ± 4.2. It is discarded. Decode here is launch-bound, so CPU contention moves it.

   The MTP blob does not explain the old 149 baseline. On ollama's registry qwen3.5:4b, which auto-enables MTP drafting, decode was 152–161 tok/s. That is slower than ollama on the unsloth file without MTP (172–201). On this prompt MTP is not a win, so #4245 (MTP) drops in the gain ÷ effort ranking.

   Correctness at temp 0:

   • Each engine is self-identical across 3 runs.
   • apr and ollama agree for the first 266 characters (~55 tokens), then diverge: ollama …standing where the stone met the sky, apr …standing on the very same granite planks.
   • One prompt is an anecdote, so no cause is named. The next step is to read the top-2 logit margin at the divergence token (near-tie vs defect). apr's float GEMV vs llama.cpp's q8_1 MMVQ activations is the obvious numeric difference.

   Ranking, updated by measurement (gain ÷ effort)

   1. Decode ~0.5×. This is the largest fair gap, and it sits in launch and host overhead: the 4B Q4_K bandwidth ceiling is ~370 tok/s.
      • The allocation-free decode token plus device argmax (apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215 pieces) is in perf(qwen35-cuda): DP4A requant staleness (#4258) + argmax first-of-ties (#4259) + allocation-free decode plumbing (#4215) — replaces #4322 #4332 #4439 (armed).
      • The next lever is CUDA graph replay of the decode step (apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215).
   2. Long-context prefill: 0.26× at 32k. apr's prefill rate falls 7.8k→3.2k tok/s from 4k to 32k while llama.cpp holds ~10k, because the full-attention layers run cuBLAS f32 with a materialised score matrix. New child qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442 (flash-attention prefill).
   3. Prefix reuse (apr serve (0.69.1) has no prompt-prefix reuse on the qwen35 CUDA path: an identical ~850-token prompt costs 11.0 s on both runs (ollama: 8.2k tok/s uncached, 37k cached) #4214). Unchanged; it matters for multi-turn TTFT only.
   4. Short-prompt prefill is at parity or ahead: 0.93× at 850 tokens, and TTFT at 30 tokens is ahead.
   5. MTP (qwen35 MTP speculative decoding with GDN state rollback #4245): demoted, see above.

   Receipts: /mnt/nvme-raid0/tmp/crux4220/p2/ on lambda.

   • bench.sh, serve_probe.py and ollama_probe.py are the harness.
   • out-4b-r2/ holds the table run.
   • out-4b-ident/ holds the identity and reload-attribution run.
5. noahgift

   added a parent issue on Sep 25, 2026

   • EPIC E2 0.73: Verbs Are Fast #3598
6. noahgift

   modified the milestones: 0.70.1, 0.71.0 on Sep 25, 2026
7. noahgift

   added

   P1High priorityHigh priority

   on Sep 25, 2026
8. noahgift commented on Sep 25, 2026

   

   noahgift

   on Sep 25, 2026

   MemberAuthor

   More actions

   #4220 part 3: the same-GGUF table at every size (0.8B, 2B, 4B, 9B)

   Setup as in part 2: apr 0.69.5-rc.1 (5660b0877, sha256 4bfdacd2…), ollama 0.34.4, and llama-bench b11050 (-ngl 99 -fa 1). Each size runs from the same GGUF file (lqw/models/Qwen3.5-*-Q4_K_M.gguf) on one RTX 4090, under flock /tmp/apr-gpu.lock. Prompts are 863, 4,082 and 32,734 tokens, the same counts on both engines. apr prefill rates come from the [qwen35] batched prefill trace line. ollama's come from prompt_eval_count / prompt_eval_duration; wall clock minus load_duration agrees within 5%. Every apr run logged gpu-layers … resolved=N total=N (backend=cuda) with no fallback.

   size	prefill 850: apr / ollama	prefill 4k: apr / ollama	prefill 32k: apr / ollama	decode tok/s: apr / ollama	llama-bench pp32k / tg128
   0.8B	21,086 / 8,115 (2.60)	25,233 / 34,392 (0.73)	10,913 / 35,014 (0.31)	270–274 / 429–455 (0.60)	33,678 / 593
   2B	9,703 / 7,800 (1.24)	9,485 / 17,570 (0.54)	9,140 / 26,128 (0.35)	178–184 / 306–326 (0.58)	11,975 / 181
   4B, run 2	5,389 / 5,765 (0.93)	7,798 / 10,166 (0.77)	3,178 / 12,365 (0.26)	84 / 184–193 (0.46)	10,099 / 169
   4B, run 3	4,595 / 5,898 (0.78)	5,236 / 10,869 (0.48)	3,269 / 12,589 (0.26)	55–60 / 154–181 (0.36)	8,302 / 150
   9B	5,255 / 5,067 (1.04)	4,625 / 8,328 (0.56)	per-token fallback: first token at 732.7 s (#4450)	40–64 / 132–134 (0.41)*	8,435 / 141

   * The 9B decode ran after the fallback had stuck (see below), so it is not a clean decode number.

   Readings

   • Short prompts: apr is at parity or ahead at every size. Time to first token on a 30-token prompt was 37–66 ms, against ollama's 93–145 ms of prompt_eval time. That excludes ollama's model reload of 2.2–2.5 s per request in this harness.
   • Long prompts: the gap grows with length at every size, from 0.5–0.8 at 4k to 0.26–0.35 at 32k. The 4B's collapse at 32k reproduced across two runs (3,178 and 3,269). Tracked in qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442. The trace shows apr already has a flash prefill path ("flash (f16 inputs, f32 accumulation)"), but it chose attention cuBLAS f32 on every run here. qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442 should first measure what flash does at 32k before writing a new kernel.
   • Decode is 0.36–0.60× ollama at every size. This is the biggest fair gap, and perf(qwen35-cuda): DP4A requant staleness (#4258) + argmax first-of-ties (#4259) + allocation-free decode plumbing (#4215) — replaces #4322 #4332 #4439 plus apr serve (0.69.1) decodes Qwen3.5-4B Q4_K_M at ~87 tok/s on RTX 4090 vs ollama 0.34.2 at ~149 tok/s (0.58x) on the same box #4215 target it.
   • 9B, new defect perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450: with --context-length 40960, the perf(prefill): MMQ int8 prefill matmul + LM head on the last token only (0.69.5) #4313 fp16 prefill weights (13.2 GB) leave about 400 MiB free. The 512-row prefill workspace misses and the session prefills one token at a time. It stays that way: afterwards, 23-token prompts also go one token at a time. The workspace is sized by the end position, not by the tokens being prefilled, and nothing tries a smaller chunk.
   • Caveats on noise: the host was under load from other sessions. llama-bench itself moved 10–20% between the 4B runs, and apr's 4B decode fell from 84 to 55–60. Treat single cells as ±30%. The ratios that repeat across sizes (decode about 0.4–0.6, 32k about 0.3) are the findings.
   • Not run: 27B (16.7 GB of Q4_K). Its fp16 prefill copy would be about 54 GB, so perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450 will decide how it behaves. It is queued after perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450.

   Evidence: /mnt/nvme-raid0/tmp/crux4220/p2/out-{0p8b,2b,4b-r2,4b-r3,9b}/: provenance.txt (sha256 of binary and GGUF, versions, foreign GPU processes), apr.jsonl, ollama.jsonl, llama-bench.json and serve.log. table.py builds this table from them.
9. noahgift commented on Sep 25, 2026

   

   noahgift

   on Sep 25, 2026

   MemberAuthor

   More actions

   #4220 part 3 follow-up: 9B decode rerun. Part 3 flagged the 9B decode figure (0.41×) as contaminated. This rerun does not give a clean replacement. It shows the 9B decode figure depends on host load.

   The rerun used a fresh apr serve (0.69.5-rc.1 5660b0877, same GGUF, --context-length 8192, RTX 4090) and ran the decode probes before any long prompt, so the #4450 sticky state could not form. Decode = (completion−1)/(e2e−TTFT), 128 tokens.

   run	load1	apr decode tok/s (per probe)	median	÷ ollama 132.0
   part 3 (after 850/4k/32k)	—	40.4, 63.9, 54.1	54.1	0.41
   this rerun	67 on 48 cores	36.7, 37.4, 32.5, 38.2, 37.7	37.4	0.28

   llama-bench tg128 on the same file: 140.6 tok/s. A foreign 936 MiB GPU process (pid 3650689, outside our namespace) was resident for the whole rerun.

   Reading: apr's 9B decode swings 0.28–0.41× with host load, while the GPU-bound comparators do not. That is consistent with an eager, launch-bound decode step (#4233's opt-in CUDA-graph decode is on batch/0.70.0, not in rc.1). A quiet-host run with QWEN35_CUDA_GRAPH=1 on a 0.70.0 binary is what gives a citable 9B decode number. Until then treat 0.28–0.41 as a range, not a point.

   The TTFT side is unchanged from part 3: 850 → 685 ms and 4k → 992 ms (prefill 5138 tok/s from the serve log). Raw data: /mnt/nvme-raid0/tmp/crux4220/p2/out-9b-dec/.
10. noahgift commented on Sep 25, 2026

    

    noahgift

    on Sep 25, 2026

    MemberAuthor

    More actions

    #4220 p5 — 9B decode, eager vs QWEN35_CUDA_GRAPH=1 on the 0.70.0 car binary (apr-m0694)

    Binary apr 0.70.0 (99c6c61b4), RTX 4090, Qwen3.5-9B-Q4_K_M.gguf, ctx 8192. Order was A/B/A (eager, graph, eager), with a fresh server per arm, under the GPU lock. Each arm got an 8-token warmup, then 3× 128-token decode on /v1/chat/completions. Measured 2026-09-25 21:45–21:52Z.

    arm	decode tok/s (3 × 128)	mean
    eager A	35.9 / 34.8 / 33.3	34.7
    graph	26.5 / 27.8 / 38.0	30.8
    eager B	32.5 / 35.2 / 36.2	34.6

    • The graph engaged. The serve log shows [aprender#4233] qwen35 decode graph: captured 626 kernels at position 30 and then replay launched (626 kernels, position 31).
    • Tokens are identical. All 3 decode texts are byte-equal across eager A, graph and eager B.
    • No speedup at this load. The graph arm's spread (26.5–38.0) covers the eager arms, so the graph is not faster here. Host load was not quiet (load average 92 → 21 → 17 → 67 across the arms), so this is a range, not a point.
    • Weakens "launch-bound". If 9B decode were launch-bound, collapsing 626 launches into one replay should have shown up even through the noise. The ~35 tok/s floor (ollama 132, llama-bench tg128 140.6) more likely sits in host-side per-token work or the kernels themselves. The next step is apr profile / nsys on a quiet host, not the graph flag.

    Artifacts: /mnt/nvme-raid0/tmp/crux4220/p2/out-9b-graph/ (graph9b.sh, apr.jsonl, serve-*.log, provenance.txt).
11. noahgift

    added sub-issues on Oct 4, 2026

    • Release gate: apr serve throughput vs llama.cpp on the SAME GGUF (Qwen3.5 0.8B/2B/4B/9B/27B) — a regression ratchet that blocks the tag, plus a published parity target #4218
    • qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442
    • perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450
12. 16 remaining items

    Load more
13. noahgift

    removed sub-issues on Oct 14, 2026

    • serve: Qwen3.5 chat turn 2 re-prefills from position 0; the session extend never engages via /v1/chat/completions #4274
    • 0.70.x: prefill kernel parity — llama-bench pp ≥0.8x llama.cpp (today 0.19–0.20x) #4376
    • qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442
    • perf(serve): qwen35 prefill falls to one token at a time when the 512-row workspace misses — 9B 32k TTFT 733 s, and it sticks (#4220 child) #4450
    • serve: KVCache::new and OptimizedKVCache::new commit num_layers-1 full layers at construction (the form #4778 removes from StreamingKVCache) #4783
    • client-driven Ollama arm (apr test llm bench + client sha256) before any ratio vs Ollama #5033
    • perf(serve): Qwen3.5 host build double-holds weights on unified memory (f32 embed + fp16 prefill prewarm), so the 4B Q4_K_M cannot fit 8 GB #5258
    • perf(serve): GPU decode on 8 GB unified-memory ARM64 CUDA runs 0.27-0.39x ollama on the same Qwen3.5 GGUF #5259
    • perf(serve): Qwen3.5 GPU decode at 0.15-0.27x of ollama on 8 GB unified-memory Orin; batched prefill falls back to one token #5301
    • fix(qa): ollama_parity cannot measure on unified memory, ollama stays resident while apr plans its GPU load #5302
14. noahgift commented on Oct 14, 2026

    

    noahgift

    on Oct 14, 2026

    MemberAuthor

    More actions

    CRUX parity sweep, apr serve vs ollama, Qwen3.5 0.8B–27B, GB10 (gx10, sm_121, 122 GB unified), 2026-10-14

    Same harness and method as the 2026-10-12 RTX 4090 sweep, on the second GPU class.

    Setup

    • apr 0.71.2 (98a370669 = main @ 98a3706), built --features cuda on the host, binary sha256 9310d5df….
    • ollama 0.40.4, the latest release (gh api repos/ollama/ollama/releases/latest → v0.40.4).
    • Same Q4_K_M GGUF per size for both engines. 0.8B/2B/4B/9B sha256 match the copies used elsewhere in this sweep; 27B sha256 84b5f7f1…. Prompt tokens 869 / 4087 / 32740 on both.
    • Temp 0, ctx 40960. Every size ran under the host GPU lock with no foreign compute app on the GB10 (provenance per size). Host load 2–6.
    • Harness: scripts/crux_ollama_parity.sh @ 9991ba6, unmodified. Rates are measured as in the 10-12 comment (ollama server-side, apr serve-log prefill lines body+suffix, decode = median of 3 × 128 client-stamped).

    Prefill / decode tok/s, ratio = apr ÷ ollama

    size	p850 apr / ollama	ratio	p4k apr / ollama	ratio	p32k apr / ollama	ratio	decode apr / ollama	ratio	TTFT p850 / p4k / p32k apr (ms)
    0.8B	4938 / 6828	0.72	5830 / 11973	0.49	3322 / 13246	0.25	120 / 263.3	0.46	214.2 / 733.7 / 10033.1
    2B	4281 / 5583	0.77	4666 / 8434	0.55	2768 / 8967	0.31	66.9 / 139.7	0.48	243.1 / 913.6 / 12007.3
    4B	2336 / 3310	0.71	2373 / 4089	0.58	1227 / 3890	0.32	35 / 67.2	0.52	404.5 / 1760.2 / 27026.7
    9B	1552 / 2328	0.67	1661 / 2805	0.59	1025 / 2782	0.37	21.3 / 39.9	0.53	596.8 / 2493.2 / 32237.9
    27B	140 / 814	0.17	89 / 905	0.10	83 / 848	0.10	7.2 / 13.1	0.55	6260.6 / 46073 / 396791.7

    (apr TTFT in the last column; ollama TTFT includes a model load because of keep_alive 0, so its server prefill rate is the fair figure.)

    GPU path, from the serve trace on every size: Backend: GPU (CUDA, NVIDIA GB10, 122502 MB VRAM); gpu-layers requested=all resolved=N total=N (24/24 … 64/64); every prefill is batched prefill … attention cuBLAS f32; 0 fell_back/per-token lines.

    Correctness, temp 0, 128-token decode prompt: no size is byte-identical. First differing byte: 0.8B 10, 2B 49, 4B 226, 9B 13, 27B 136. Every output is a coherent continuation. One prompt per size, so these are candidate near-ties, not a named defect.

    What GB10 adds to the 4090 picture

    1. 27B prefill: the fp16 prewarm still does not fit, even with 122 GB unified. The serve log: fp16 prefill weights NOT prewarmed: 46445 MiB needed, 32361 MiB free; prefill uses f32. The result is 0.10–0.17× on prefill and a 397 s TTFT at 32k. The dense fp16 copy is 2.8× the 16.7 GB Q4_K_M file. Bigger memory does not fix cause Feature Request: Decision Tree & Random Forest for Classification Tasks #1; a quantized-weight prefill GEMM (or a streamed per-layer dequant) does. Side note: on unified memory the "free" figure leaves out reclaimable page cache, so the check also under-reports the room it has.
    2. Short-prompt prefill is below parity on GB10 (0.67–0.77× at 850) where it was 1.0–2.0× on the 4090. The prewarmed fp16 path that wins on the 4090 does not win on GB10.
    3. Long-context prefill falls with length (0.25–0.37× at 32k on 0.8B–9B) while ollama's rate rises: the same attention cuBLAS f32 signature as the 4090 (qwen35 batched prefill: flash attention for the full-attention layers — 32k prefill 3.2k tok/s vs llama.cpp 10.1k (0.26× ollama) #4442: flash path exists, not chosen).
    4. Decode is ~0.5× on every size on both GPUs (0.46–0.55× here). Effective weight bandwidth at 27B: apr 7.2 tok/s × 16.7 GB ≈ 120 GB/s vs ollama 13.1 × 16.7 ≈ 219 GB/s, against GB10's ~273 GB/s: about 44% vs 80% of peak, the same split as the 4090 (37% vs 79%). This is per-byte GEMV efficiency, not launch overhead, and not 4090-specific.

    The CPU (intel, 4 pinned threads, 0.8B–9B) table follows when its run finishes. Technique ranking and child tickets follow after that.

    Agent: aprender-wmainred
15. noahgift

    modified the milestones: 0.72.0, 0.73.0 on Oct 14, 2026
16. noahgift commented on Oct 14, 2026

    

    noahgift

    on Oct 14, 2026

    MemberAuthor

    More actions

    Folded into the checklist of its epic #3598 (full original text kept there, verbatim). Owner owner:la-0.71: the work is unchanged and continues from the epic checklist; reopen if you want it tracked separately. Refs #3598.
17. noahgift

    closed this as not plannedon Oct 14, 2026