Repository navigation
CRUX verb parity: chat, serve and perf receipts against llama.cpp and ollama #5336
Copy link
Copy link
Closed as not planned
Closed as not planned
Copy link
Labels
Milestone
Description
Scope: The CRUX comparison slices: every apr verb mapped to its llama.cpp/ollama counterpart, the chat and serve slices through one client, per-cell speed receipts with bootstrap intervals, and a serve test that proves batched prefill is used.
Why it matters: Parity claims need the same inputs through all three tools; without it a gap cannot be named.
Done when: Each CRUX slice runs apr, llama.cpp and ollama on the same GGUF and prompts in CI and publishes a receipt per cell with n>=5 and a confidence interval.
Epic: #3999. Folded under triage rule 7b (2026-10-14): each row below was its own issue, closed as not planned and kept here as a sub-ticket. Reopen an issue when its work starts or its PR opens.
Sub-tickets
- CRUX verb correspondence (#3739 amendment): every leaf of the derived apr surface maps to its llama.cpp/ollama counterpart or
comparator: none+ reason #3774 CRUX verb correspondence (CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739 amendment): every leaf of the derived apr surface maps to its llama.cpp/ollama counterpart orcomparator: none+ reason (milestone 0.73.0)- accept: - Exit 75 means every engine of that cell is a refused row.
- CRUX slice 3 (#3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn #3795 CRUX slice 3 (CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn (milestone 0.73.0)
- accept: the outcome in the title, with a receipt
- CRUX slice 4 (#3739): the serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming, plus hf/llamafile servers #3797 CRUX slice 4 (CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739): the serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming, plus hf/llamafile servers (milestone 0.73.0); branch
feat/3797-crux-serve-verify- accept: the outcome in the title, with a receipt
- CRUX-PERF: crux-perf-receipt-v1 — per-cell T0 speed vs competitor, n>=5 bootstrap CI, falsifiers F1-F5 (G2/G3/G5) #4464 CRUX-PERF: crux-perf-receipt-v1 — per-cell T0 speed vs competitor, n>=5 bootstrap CI, falsifiers F1-F5 (G2/G3/G5) (milestone 0.73.0)
- accept: the outcome in the title, with a receipt
- 0.69.4 G0: serve-level test that fails unless apr serve prefills Qwen3.5 through the batched path (#3596 countermeasure) #4250 0.69.4 G0: serve-level test that fails unless apr serve prefills Qwen3.5 through the batched path (Time-to-first-token on Qwen3.5-4B-Q4_K_M is 14-16s vs llama.cpp's 1.8-2.9s on the same box (7.7x); apr's decode rate is unmeasurable with shipped tooling #3596 countermeasure) (milestone 0.71.0); branch
feat/4250-nightly-wire- accept: the outcome in the title, with a receipt
Reactions are currently unavailable
Activity
noahgift commentedon Oct 22, 2026
Tracked in #3598 (triage 2026-10-22).
noahgift commentedon Oct 22, 2026
tracked in #3999