Repository navigation
CRUX slice 4 (#3739): the serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming, plus hf/llamafile servers #3797
Description
Child of #3739: slice 4, the serve verb. The issue's done_when, verbatim: "serve: apr serve · llama-server · ollama serve, all through the SAME OpenAI /v1/chat/completions client, streaming + non-streaming".
done_when
-
scripts/crux_inference_dogfood.sh <version> --verbs run,serveruns ONE serve cell per model under one hold of the GPU lock:apr serve runandllama-servereach load the model once, next to the running ollama daemon;- every golden prompt goes through
scripts/lib/crux_openai_client.py, non-streaming and streaming; - then the servers stop, and ollama unloads (keep_alive 0, plus the unload wait).
-
One client for every server.
crux_openai_client.pysends the same body (model, messages, max_tokens, temperature, seed, stream) and writes the row-contract JSON{"text", "reported": {device, stream, finish_reason, usage, chunks}}. It reads no clock and computes no rate. -
Plugin engines (hf, llamafile) run
gen --verb 'serve run', starting their own server per call (non-streaming). -
The judge reads every serve row through the client JSON and applies the same RED rule.
- The key gains an additive
mode(nonstream | stream) for serve cells only. - apr serve's cells carry
backend_verified: false, because the server's responses report no backend.
- The key gains an additive
-
The case table covers:
- stream and non-stream as distinct GREEN cells;
- an apr serve that answered nothing while llama-server answered (RED).
Mutants: drop the mode from the key; drop the backend flag.
-
A real run on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M): apr serve, llama-server and ollama each answer every golden prompt in both modes.
-
Not in this slice, named in
not_covered: TTFT and decode rate throughapr test llm bench(the canonical client, PERF-009), and max accepted context.
Activity
fix(gate) #3828: the ladder measured ONE verb while the contract decl…
noahgift commentedon Sep 22, 2026
AMENDMENT (operator, 2026-09-22): three comparators and a user-story test case
Operator instruction: each verb MUST have two, preferably three CRUX comparators, and a "test case" user story for each. Recorded on all four verb slices (#3829 run, #3795 chat, #3797 serve, #3830 code) and on the epic #3739.
Comparators for
serve— three, all through ONE clientllama-server(llama.cpp @ pinnedd1d3c3396)ollama serve(0.33.2)finish_reasonconventionsThe load-bearing constraint is already in this slice and is what makes three engines meaningful:
crux_openai_client.pyis the only client, sending the same body to all three. A per-engine client would compare our clients, not our servers.Test case — user story
Acceptance, falsifiable: the same request body (model, messages, max_tokens, temperature, seed, stream) sent to all three servers returns, for each golden prompt: a 200, a non-empty
choices[0].message.content(non-streaming) and a reassembled non-empty content from the chunks (streaming), containing the golden pattern.stream: trueandstream: falseare distinct cells — an implementation that answers one and not the other has broken half the contract, which is why the key carriesmode.The RED rule, stated for this story:
apr serveanswering nothing whilellama-serveranswers the same body is RED. This is not hypothetical — #3791 shipped a serve path that returned HTTP 500Invalid token IDat temperature 0 and"HHHHHHHH…"at 0.8, on a model llama.cpp serves correctly.Known limitation, already in this slice: apr serve's responses report no backend, so cells carry
backend_verified: false; TTFT and decode rate arenot_coveredhere and belong toapr test llm bench(PERF-009).noahgift commentedon Sep 22, 2026
CORRECTION: the comparator table I posted here was wrong — llama.cpp works on BOTH hosts
aprender-c7, 2026-09-22. Full correction with the measurements: #3829 (comment)
I reported lambda's
llama-clibroken and gx10's absent. Both readings were ofPATH, not of llama.cpp:llama-cli,llama-serverandllama-tokenizeall run at the pinned sha, andbash scripts/llama_bin.shresolves them (#3740). On lambda~/.local/bin/llama-cliis a dead copy of the UNPINNED checkout and shadows the working one; on gx10 nothing is on PATH at all, so a lookup reports absence while the pinned build exists. CLAUDE.md Verification Discipline #8, the~/.local/bin/aprinstance, one directory over.So nothing needs building, and this slice is landable on both hosts. The comparator must be resolved through
scripts/llama_bin.sh, never by bare name.Two bounds from that inventory do survive, and they are real:
qwen2.5-coder:1.5b-instruct-q4_K_M). On those rungs ollama iscomparator: nonewith the reason (CRUX verb correspondence (#3739 amendment): every leaf of the derived apr surface maps to its llama.cpp/ollama counterpart orcomparator: none+ reason #3774); the matrix is not square.fold CRUX slices 1+3+4 (#3739, #3795, #3797): the harness, NOT slice …
noahgift commentedon Sep 24, 2026
triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): milestoned to 0.70.0 — 0.69.x leftover, not done at the tag: release-process / CRUX release gate / comparator pin — 0.70 Fast Train. It was 0.69.1 (released). Open PR / live branch: PR#3835.
noahgift commentedon Sep 24, 2026
Milestone 0.70.0 → 0.70.1 (gate triage). 0.70.0 now holds only the operator's release gate: "FULL ontology spec and all aprender binary working" + "0.70 to crates.io" (ONT-001, every [[bin]] build+smoke, the publish cascade), plus P0s and operator-ruled rows. This issue isn't needed for any of those. Evidence: evidence/m070-gate-triage-1835.md. If that's wrong, move it back to 0.70.0 and say which gate it serves.
noahgift commentedon Sep 24, 2026
done_when audit (apr-features): items 1-5 and 7 are met. Item 5 gains 2 mutants on feat/3797-crux-serve-verify @6ab17f9b4, which folds after merge-back #4224. Item 6 (real GREEN run) cannot be met as written: qwen2.5-coder-1.5b is not in prompt-certification.json and there is no hf/vllm ground truth. Proposed amendment: run on a certified Qwen3.5-4B-family sha, or certify coder-1.5b first. The real run found #4339 (/v1/completions stop overrun, 13/13), #4340 (im_end stream leak) and #4341 (sweep timeout discards rows). Receipt: cop-inbox/evidence/apr-features-3797-audit.md
fix(crux): #4341 a killed serve sweep keeps its finished rows — plan.…
test(crux): #3797 serve mutants — drop the key's mode, drop apr serve…
14 remaining items
noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.73.0.
noahgift commentedon Oct 14, 2026
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5336 (CRUX verb parity: chat, serve and perf receipts against llama.cpp and ollama), split out of #5313 by theme.