Repository navigation
CRUX slice 3 (#3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn #3795
Description
Child of #3739: slice 3, the chat verb. The issue's done_when, verbatim: "chat (multi-turn): apr chat · llama-cli conversation · ollama run multi-turn". Design and measurements: #3739 issuecomment-5768519742.
done_when
-
scripts/crux_inference_dogfood.sh <version> --verbs run,chatruns a multi-turn conversation per model through every engine. Achatrequest withoutrundeclines, because the positive control lives inrun.- Prompt:
chat-arith-2turn, the user turns "What is 2+2?" then "Now multiply that by 3.". It is judged on the FINAL reply (expect_any ["12"], so the first turn's answer must carry over), and every reply is recorded.
- Prompt:
-
How each engine is driven:
- apr:
apr chat, one user turn per stdin line. - llama.cpp and ollama:
scripts/lib/crux_pty_chat.py, a stdlib pseudo-terminal driver, because they don't take turns from a pipe. Measured: llama-cli loops on empty prompts, and ollama reads the pipe as ONE prompt. The driver waits for each CLI's prompt marker, sends one turn, and writes the row-contract JSON{"text", "turns", "reported": {"device"}}. - hf and llamafile:
gen --verb chatwith the user turns (infra-3c, a3eecfe).
- apr:
-
The judge scores the final turn with the same RED rule.
apr chatreports no backend (apr chat --no-gpu still uploads the GGUF to CUDA (VRAM held outside the GPU lock), and apr chat reports no machine-readable backend #3794), so apr's chat cells carrybackend_verified: falseand the receipt'snot_coverednames it. The run-verb token-parity field isn't reported as measured for chat. -
The case table covers:
- a GREEN chat;
- apr losing the first turn's answer while llama.cpp carries it (RED);
- an apr transcript with no
Assistant:turn (RED); - the record of every turn, the unverified backend, and no run-verb parity.
A mutant of each rule turns the table RED.
-
A real run on lambda (cpu lane, qwen2.5-coder-1.5b) with apr, llama.cpp@pin and ollama 0.33.2: every engine's two turns are recorded.
Activity
noahgift commentedon Sep 22, 2026
AMENDMENT (operator, 2026-09-22): three comparators and a user-story test case
Operator instruction: each verb MUST have two, preferably three CRUX comparators, and a "test case" user story for each. Recorded on all four verb slices (#3829 run, #3795 chat, #3797 serve, #3830 code) and on the epic #3739.
Comparators for
chat— three (this slice already names them; naming them as the required set)d1d3c3396)crux_pty_chat.pyollama runmulti-turn (0.33.2)crux_pty_chat.pygen --verb chatTest case — user story
Acceptance, falsifiable: the FINAL reply contains
12. Not the first — the first turn's answer must survive into the second turn's prompt, which is exactly what a dropped or re-templated history destroys. Every turn is recorded, so a failure shows which turn lost the thread.The RED rule, stated for this story: if llama.cpp carries the answer across and
apr chatdoes not, the cell is RED. Both failing is recorded, not RED — the comparator is the oracle, not the truth.Known limitation, already in this slice:
apr chatreports no backend (#3794), so apr's chat cells carrybackend_verified: false. The story above is judged on the answer; which device produced it is not yet verifiable on this verb, andnot_coveredsays so.noahgift commentedon Sep 22, 2026
CORRECTION: the comparator table I posted here was wrong — llama.cpp works on BOTH hosts
aprender-c7, 2026-09-22. Full correction with the measurements: #3829 (comment)
I reported lambda's
llama-clibroken and gx10's absent. Both readings were ofPATH, not of llama.cpp:llama-cli,llama-serverandllama-tokenizeall run at the pinned sha, andbash scripts/llama_bin.shresolves them (#3740). On lambda~/.local/bin/llama-cliis a dead copy of the UNPINNED checkout and shadows the working one; on gx10 nothing is on PATH at all, so a lookup reports absence while the pinned build exists. CLAUDE.md Verification Discipline #8, the~/.local/bin/aprinstance, one directory over.So nothing needs building, and this slice is landable on both hosts. The comparator must be resolved through
scripts/llama_bin.sh, never by bare name.Two bounds from that inventory do survive, and they are real:
qwen2.5-coder:1.5b-instruct-q4_K_M). On those rungs ollama iscomparator: nonewith the reason (CRUX verb correspondence (#3739 amendment): every leaf of the derived apr surface maps to its llama.cpp/ollama counterpart orcomparator: none+ reason #3774); the matrix is not square.fold CRUX slices 1+3+4 (#3739, #3795, #3797): the harness, NOT slice …
noahgift commentedon Sep 24, 2026
triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): milestoned to 0.70.0 — 0.69.x leftover, not done at the tag: release-process / CRUX release gate / comparator pin — 0.70 Fast Train. It was 0.69.1 (released). Open PR / live branch: PR#3835.
noahgift commentedon Sep 24, 2026
Milestone 0.70.0 → 0.70.1 (gate triage). 0.70.0 now holds only the operator's release gate: "FULL ontology spec and all aprender binary working" + "0.70 to crates.io" (ONT-001, every [[bin]] build+smoke, the publish cascade), plus P0s and operator-ruled rows. This issue isn't needed for any of those. Evidence: evidence/m070-gate-triage-1835.md. If that's wrong, move it back to 0.70.0 and say which gate it serves.
PMAT-3795: CRUX chat verb — a two-turn conversation per model; apr ch…
noahgift commentedon Sep 24, 2026
Landed on
batch/0.70.0(ba75da2..cb09def, fast-forward) by aprender-d1, on the cop's assignment. aprender-1c confirmed it had no claim.The work already existed but had never been pushed. Local branch
crux-3739-chatheld 12 commits from 3 days ago, in worktree76-3774, untouched since 09-21. I rebased all 12 onto the batch as a stack:runverb, first receiptscomparator: none+ reason #3774: verb correspondence (253 leaves), hf and llamafile judgesapr chatis fed on stdin; llama.cpp and ollama through a pty driver. The verdict is on the final turn.I added one commit: 4 bashrs false positives suppressed, each with its reason. The release bashrs gate is back to the 7 errors the batch already had.
Verified:
check_crux_inference_judge.sh: 42 ok, 0 broke. This includes the chat cases: final turn carries turn 1 → GREEN; apr drops turn 1 → RED; no Assistant turn → RED.Not verified: I did not run the live dogfood against real engines (that needs the GPU queue and the models). The
evidence/crux/0.69.0/*receipts are 3 days old: lambda cpu PASS, lambda gpu RED.Still unpushed:
crux-3739-serve(#3797, the serve verb), one commit on the #3774 tip.noahgift commentedon Oct 14, 2026
P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.73.0.
noahgift commentedon Oct 14, 2026
merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5336 (CRUX verb parity: chat, serve and perf receipts against llama.cpp and ollama), split out of #5313 by theme.