PMAT-3797: CRUX serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming

docs/roadmaps/entries/PMAT-3797.yaml

@@ -0,0 +1,22 @@
+- id: PMAT-3797
+  github_issue: 3797
+  item_type: task
+  title: 'CRUX slice 4 (#3739): the serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming, plus hf/llamafile servers'
+  status: in_progress
+  priority: critical
+  assigned_to: aprender-76
+  created: 2026-09-21T23:55:00Z
+  updated: 2026-09-22T00:40:00Z
+  spec: null
+  acceptance_criteria:
+  - 'Issue #3797 done_when 1, verbatim: "`scripts/crux_inference_dogfood.sh <version> --verbs run,serve` runs ONE serve cell per model under one hold of the GPU lock: `apr serve run` and `llama-server` each load the model once, next to the running ollama daemon; every golden prompt goes through `scripts/lib/crux_openai_client.py`, non-streaming and streaming; then the servers stop, and ollama unloads (keep_alive 0, plus the unload wait)." Teardown kills every server before waiting, and waits on those pids only. The first real run hung here: a bare `wait` after the first kill blocked on the other server.'
+  - 'done_when 2, verbatim: "One client for every server. `crux_openai_client.py` sends the same body (model, messages, max_tokens, temperature, seed, stream) and writes the row-contract JSON `{\"text\", \"reported\": {device, stream, finish_reason, usage, chunks}}`. It reads no clock and computes no rate." MEASURED against the three servers on lambda (cpu): non-stream and stream both answer, and the usage is transcribed (apr serve prompt_tokens 18; llama-server and ollama 36).'
+  - 'done_when 3/4, verbatim: "Plugin engines (hf, llamafile) run `gen --verb ''serve run''` ... The judge reads every serve row through the client JSON and applies the same RED rule. The key gains an additive `mode` (nonstream | stream) for serve cells only. apr serve''s cells carry `backend_verified: false`." Per aprender-97 (#3715), the correspondence ENTRY for `serve run` declares `modes: [nonstream, stream]`, with per-engine modes (llama.cpp and ollama both, hf and llamafile nonstream), so pv never names a verb.'
+  - 'done_when 5, MEASURED: `bash scripts/check_crux_inference_judge.sh` gives 44 ok, 0 broke. The serve rows: stream and non-stream are distinct GREEN cells with apr serve''s backend unverified; an apr serve that answered nothing (connection refused) while llama-server answered is RED. Mutants: dropping the mode from the key breaks 1; dropping the serve backend flag breaks 1.'
+  - 'done_when 6, MEASURED on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, `--verbs run,serve`), rerun after the teardown fix and finishing unaided: PASS, 9 cells (3 run + 6 serve), 9 GREEN. apr serve, llama-server@d1d3c3396 and ollama 0.33.2 each answered every golden prompt in both modes. No server process and no ollama import was left behind. apr serve answers 2+2 with "4" from an 18-token prompt, where llama-server and ollama send 36 and answer "2 + 2 equals 4.": apr serve does not inject the GGUF template''s default system prompt. It is recorded; the byte-equal tmpl rows against HF are what will judge it.'
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'Refs #3739 (slice 4; not a closing keyword for the parent). Closes #3797 when merged. STACKED on crux-3739-chat (#3796), so the diff to judge is `origin/crux-3739-chat...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the serve cell, the `serve` verb token mapped to key `serve run`, emit_gen/cell_result/rows_for carrying verb and mode), scripts/lib/crux_openai_client.py (new), scripts/lib/crux_inference_judge.py (serve parsing, the `mode` key part), scripts/check_crux_inference_judge.sh (serve rows), evidence/crux/verb-correspondence.yaml (`modes` on the serve run entry, regenerated from the same v1.1 surface, sha 207c3297), and this fragment. NOT in this slice and named in `not_covered`: TTFT and decode rate through `apr test llm bench` (the canonical client, PERF-009, the next increment), and max accepted context. The client computes no rate: check_no_competing_harnesses stays count=0.'

docs/roadmaps/roadmap.yaml

@@ -20043,3 +20043,25 @@ roadmap:
   labels:
   - kind:code
   notes: 'Refs #3739 (slice 3; not a closing keyword for the parent). Closes #3795 when this merges. STACKED on crux-3774-verb-correspondence (#3789), so the diff to judge is `origin/crux-3774-verb-correspondence...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the chat verb), scripts/lib/crux_pty_chat.py (new), scripts/lib/crux_inference_judge.py (chat parsing), scripts/check_crux_inference_judge.sh (chat rows), scripts/crux_inference_prompts.json (the chat prompt; the golden drift check covers only the golden rung), and this fragment. FOUND while building it: #3794, where `apr chat --no-gpu` still uploads the GGUF to CUDA (VRAM held outside the GPU lock; generation honours force_cpu, the load path does not) and apr chat reports no machine-readable backend. The pty helper starts only the argv it is given and reads no clock for a metric (its per-turn timeout is a bound), so the competing-harness count stays 0.'
+- id: PMAT-3797
+  github_issue: 3797
+  item_type: task
+  title: 'CRUX slice 4 (#3739): the serve verb — apr serve, llama-server and ollama through ONE OpenAI client, streaming + non-streaming, plus hf/llamafile servers'
+  status: in_progress
+  priority: critical
+  assigned_to: aprender-76
+  created: 2026-09-21T23:55:00Z
+  updated: 2026-09-22T00:40:00Z
+  spec: null
+  acceptance_criteria:
+  - 'Issue #3797 done_when 1, verbatim: "`scripts/crux_inference_dogfood.sh <version> --verbs run,serve` runs ONE serve cell per model under one hold of the GPU lock: `apr serve run` and `llama-server` each load the model once, next to the running ollama daemon; every golden prompt goes through `scripts/lib/crux_openai_client.py`, non-streaming and streaming; then the servers stop, and ollama unloads (keep_alive 0, plus the unload wait)." Teardown kills every server before waiting, and waits on those pids only. The first real run hung here: a bare `wait` after the first kill blocked on the other server.'
+  - 'done_when 2, verbatim: "One client for every server. `crux_openai_client.py` sends the same body (model, messages, max_tokens, temperature, seed, stream) and writes the row-contract JSON `{\"text\", \"reported\": {device, stream, finish_reason, usage, chunks}}`. It reads no clock and computes no rate." MEASURED against the three servers on lambda (cpu): non-stream and stream both answer, and the usage is transcribed (apr serve prompt_tokens 18; llama-server and ollama 36).'
+  - 'done_when 3/4, verbatim: "Plugin engines (hf, llamafile) run `gen --verb ''serve run''` ... The judge reads every serve row through the client JSON and applies the same RED rule. The key gains an additive `mode` (nonstream | stream) for serve cells only. apr serve''s cells carry `backend_verified: false`." Per aprender-97 (#3715), the correspondence ENTRY for `serve run` declares `modes: [nonstream, stream]`, with per-engine modes (llama.cpp and ollama both, hf and llamafile nonstream), so pv never names a verb.'
+  - 'done_when 5, MEASURED: `bash scripts/check_crux_inference_judge.sh` gives 44 ok, 0 broke. The serve rows: stream and non-stream are distinct GREEN cells with apr serve''s backend unverified; an apr serve that answered nothing (connection refused) while llama-server answered is RED. Mutants: dropping the mode from the key breaks 1; dropping the serve backend flag breaks 1.'
+  - 'done_when 6, MEASURED on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, `--verbs run,serve`), rerun after the teardown fix and finishing unaided: PASS, 9 cells (3 run + 6 serve), 9 GREEN. apr serve, llama-server@d1d3c3396 and ollama 0.33.2 each answered every golden prompt in both modes. No server process and no ollama import was left behind. apr serve answers 2+2 with "4" from an 18-token prompt, where llama-server and ollama send 36 and answer "2 + 2 equals 4.": apr serve does not inject the GGUF template''s default system prompt. It is recorded; the byte-equal tmpl rows against HF are what will judge it.'
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'Refs #3739 (slice 4; not a closing keyword for the parent). Closes #3797 when merged. STACKED on crux-3739-chat (#3796), so the diff to judge is `origin/crux-3739-chat...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the serve cell, the `serve` verb token mapped to key `serve run`, emit_gen/cell_result/rows_for carrying verb and mode), scripts/lib/crux_openai_client.py (new), scripts/lib/crux_inference_judge.py (serve parsing, the `mode` key part), scripts/check_crux_inference_judge.sh (serve rows), evidence/crux/verb-correspondence.yaml (`modes` on the serve run entry, regenerated from the same v1.1 surface, sha 207c3297), and this fragment. NOT in this slice and named in `not_covered`: TTFT and decode rate through `apr test llm bench` (the canonical client, PERF-009, the next increment), and max accepted context. The client computes no rate: check_no_competing_harnesses stays count=0.'

evidence/crux/verb-correspondence.yaml

@@ -12,7 +12,8 @@ engines:
 - ollama
 - hf
 - llamafile
-rule: 'every leaf of the surface has exactly one entry; every entry states every engine, as a mapping or {none: <reason>}. A mapped engine is a required :CruxCell (#3715/#3745 S2 reads this file).'
+rule: 'every leaf of the surface has exactly one entry; every entry states every engine, as a mapping or {none: <reason>}. A mapped engine is a required :CruxCell (#3715/#3745 S2 reads this file). An entry
+  with `modes` owes at least one :CruxCell per declared mode; an entry without owes rows with no mode.'
 verbs:
 - key: attn-parity-lint
   comparators:
@@ -2251,25 +2252,38 @@ verbs:
       none: a pre-flight capacity plan (VRAM budget, roofline); it runs no model
 - key: serve run
   judged_by: golden
+  modes:
+  - nonstream
+  - stream
   comparators:
     llama.cpp:
       binary: llama-server
       args:
         --port: --port
         --context-length: -c
+      modes:
+      - nonstream
+      - stream
     ollama:
       subcommand: serve
       args: {}
+      modes:
+      - nonstream
+      - stream
     hf:
       api: transformers serve
       args:
         --port: --port
+      modes:
+      - nonstream
     llamafile:
       binary: llamafile
       args:
         <none>: --server
         --port: --port
         --context-length: -c
+      modes:
+      - nonstream
 - key: shard
   comparators:
     llama.cpp:

scripts/check_crux_inference_judge.sh

@@ -387,7 +387,7 @@ case "$got" in "False degenerate output"*) ok "hf's degenerate '!!!!' cannot vou
 # C1-C3. the chat verb (#3739 slice 3): judged on the FINAL turn; apr's backend is
 #        recorded as unverified until apr chat reports one (#3794).
 apr_chat_out() { # apr_chat_out <dir> <pid> <reply 1> <reply 2> — apr chat's transcript shape
-  printf 'Detected ChatML chat template\nYou: [3.9s, ~2 tok/s]\nAssistant: %s\nYou: [5.9s, ~1 tok/s]\nAssistant: %s\nYou: \nGoodbye!\n' "$3" "$4" > "$1/apr-$2.out"
+  printf 'Detected ChatML chat template\nYou: \nAssistant: %s\nYou: \nAssistant: %s\nYou: \nGoodbye!\n' "$3" "$4" > "$1/apr-$2.out"
   : > "$1/apr-$2.err"
 }
 chat_json() { # chat_json <dir> <engine> <pid> <reply 1> <reply 2> — the pty helper's JSON
@@ -416,6 +416,39 @@ ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" ""
 run_judge "$d"; GOT_RC=$?
 expect "an apr chat transcript with no Assistant turn is no answer (RED)" "$d" 1 $CP RED
 
+# S1-S3. the serve verb (#3739 slice 4): every server through the ONE OpenAI client,
+#        nonstream and stream as distinct cells (the key's additive `mode`).
+serve_json() { # serve_json <dir> <engine> <pid> <mode> <text|""> [error] — crux_openai_client.py's output shape
+  python3 -c 'import json,sys
+t = sys.argv[5] or None
+d = {"text": t, "reported": {"device": "fixture", "stream": sys.argv[4] == "stream", "finish_reason": "stop" if t else None, "usage": None, "chunks": 0}}
+if len(sys.argv) > 6: d["error"] = sys.argv[6]
+json.dump(d, open(sys.argv[1], "w"))' "$1/$2-$3-$4.json" "$2" "$3" "$4" "$5" "${@:6}"
+}
+serve_row() { # serve_row <manifest> <engine> <pid> <mode> <rc> <json>
+  python3 - "$@" <<'PY2'
+import json, sys
+m, eng, pid, mode, rc, o = sys.argv[1:7]
+open(m, "a").write(json.dumps({"kind": "gen", "engine": eng, "prompt_id": pid, "rc": int(rc), "stdout": o, "stderr": None,
+    "refused": None, "model_sha256": "%s", "host": "fixture", "verb": "serve run", "thinking": "off", "backend": "gpu",
+    "mode": mode}) + "\n")
+PY2
+}
+d=$(newcase serve_green); control_green "$d"
+for mode in nonstream stream; do
+  serve_json "$d" apr $P $mode "4"; serve_json "$d" llama $P $mode "2 + 2 equals 4."
+  serve_row "$d/manifest.jsonl" apr $P $mode 0 "$d/apr-$P-$mode.json"; serve_row "$d/manifest.jsonl" llama.cpp $P $mode 0 "$d/llama-$P-$mode.json"
+done
+run_judge "$d"; GOT_RC=$?
+got=$(python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print(sorted((c["key"].get("mode"), c["verdict"], c["engines"]["apr"].get("backend_verified")) for c in r["cells"] if c["key"]["verb"]=="serve run"))' "$d/receipt.json" 2>/dev/null)
+[ "$GOT_RC" = 0 ] && [ "$got" = "[('nonstream', 'GREEN', False), ('stream', 'GREEN', False)]" ] && ok "serve nonstream and stream are distinct GREEN cells; apr serve's backend unverified" || broke "serve green: rc $GOT_RC '$got'"
+d=$(newcase serve_apr_refused); control_green "$d"
+serve_json "$d" apr $P nonstream "" "URLError: [Errno 111] Connection refused"; serve_json "$d" llama $P nonstream "2 + 2 equals 4."
+serve_row "$d/manifest.jsonl" apr $P nonstream 3 "$d/apr-$P-nonstream.json"; serve_row "$d/manifest.jsonl" llama.cpp $P nonstream 0 "$d/llama-$P-nonstream.json"
+run_judge "$d"; GOT_RC=$?
+got=$(python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print([c["verdict"] for c in r["cells"] if c["key"]["verb"]=="serve run"])' "$d/receipt.json" 2>/dev/null)
+[ "$GOT_RC" = 1 ] && [ "$got" = "['RED']" ] && ok "an apr serve that answered nothing while llama-server answered is RED" || broke "serve apr refused: rc $GOT_RC '$got'"
+
 # D1-D4. tok: raw-text ids, BYTE-EQUAL or RED.
 d=$(newcase tok_equal); control_green "$d"
 printf '{"tokens": [9707, 11, 1879]}\n' > "$d/apr-tok.json"; printf '{"tokens": [9707, 11, 1879]}\n' > "$d/hf-tok.json"