CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right

Operator, verbatim (2026-09-21): "we also need a quick CRUX dogfood now that we have real models working for each release, i.e. same prompts/verbs on llama.cpp and say ollama for same model: chat/serve/run/code, etc"

Why it matters, measured today: every apr gate compared apr to apr, and missed a tokenizer that never did real BPE (#3726). llama.cpp on the same GGUF garbled nothing. A competitor on the same file is the external oracle.

Fleet inventory (measured 16:50Z)

lambda: llama.cpp d1d3c3396 (the in-tree pin; llama-cli + llama-server built); ollama 0.5.7 (old, and likely without qwen35 support), with qwen2.5-coder models only
gx10: llama.cpp only at 39173bcac (the d1d3c3396 build is in progress, Two llama.cpp comparator pins are live in-tree at once — 216 refs to d1d3c3396, 124 to 39173bcac; some committed artifacts measure against a superseded oracle #3563 (A)); ollama 0.33.2
"Same model" = ollama IMPORTS the exact GGUF via a Modelfile (FROM <path>) and the receipt records sha256 for all three engines. Registry pulls don't count.

done_when

scripts/crux_inference_dogfood.sh <version> runs, per inventory Q4_K model × {lambda, gx10}, the SAME prompt set (the golden prompts + a consumer brief from 0.69.1 CONSUMER REQUIREMENTS: every project that uses apr states its user-interface needs (verbs/API, models, context, output shape, latency) — the release gate is derived from these answers #3716 at each declared context rung that fits) through:
- run: apr run · llama-cli · ollama run
- chat (multi-turn): apr chat · llama-cli conversation · ollama run multi-turn
- serve: apr serve · llama-server · ollama serve, all through the SAME OpenAI /v1/chat/completions client, streaming + non-streaming
- code: apr code on a scripted edit-and-verify task, and the same task driven through llama-server and ollama by the same client
- thinking ON and OFF where the model supports it (apr has NO thinking toggle: realizar hard-routes every Qwen3/Qwen3.5 to the no-think template (chat_template_helpers::detect_format_from_name), so run/chat/code/serve can't be dogfooded with thinking ON #3723), at temperature 0, same seed
per cell it records: answer correct (golden), exact-match agreement across engines, token-id parity (apr vs llama-tokenize), TTFT, decode tok/s, prompt/completion tokens, and the max context each engine accepts before refusing (or failing silently)
the gate: a cell where llama.cpp OR ollama answers correctly and apr doesn't is RED (a finding naming the cell). Performance ratios are REPORTED, never claimed (the withdrawn-headline rule). The receipt feeds the SHACL shape (pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715) as a required cell kind.
ollama is upgraded on lambda through the declared path (forjar) to a version that loads qwen35 and qwen3moe GGUFs; both hosts' versions are recorded
runs in the release train (the T-1 models step, #3712 done_when 3 (child): the model-matrix gate runs at T-1 on BOTH hosts before any tag, receipts committed in the bump, re-read at T-4 by the same judge #3717) and ALSO on demand; the results table is committed per release (evidence/crux/<version>/)
depends on: Two llama.cpp comparator pins are live in-tree at once — 216 refs to d1d3c3396, 124 to 39173bcac; some committed artifacts measure against a superseded oracle #3563 (A) (the pinned llama.cpp resolves on both hosts), apr has NO thinking toggle: realizar hard-routes every Qwen3/Qwen3.5 to the no-think template (chat_template_helpers::detect_format_from_name), so run/chat/code/serve can't be dogfooded with thinking ON #3723 (thinking toggle), apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571 (apr serve loads Qwen3.5)

Owner: aprender-76 (a new session, sent by the operator). The first slice proves the harness (qwen2.5-coder-1.5b Q4_K_M, run, lambda, all three engines), then it widens.

Design note: CRUX inference dogfood (aprender-76)

Shape

scripts/crux_inference_dogfood.sh <version> --model <gguf>… [--backend gpu|cpu], the driver. It resolves apr through apr_bin.sh; the run declines unless apr --version reports <version>. llama.cpp comes only from llama_bin.sh. ollama comes from $OLLAMA_BIN, else the forjar path ~/.local/bin/ollama, else /usr/local/bin, and the receipt records the server's /api/version. Temperature, seed and context come from llama_pin.toml [protocol], so all three engines read one declaration. Every engine call runs under flock /tmp/apr-gpu.lock choom -n 1000.
scripts/lib/crux_inference_judge.py, the judge. It reads raw outputs only, and writes the receipt plus a Markdown table to evidence/crux/<version>/<host>-<backend>.{json,md}.
scripts/check_crux_inference_judge.sh, a hermetic case table that guard_tree runs bare. It includes the pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715 falsifier (llama.cpp right and apr wrong turns the run RED) and a drift check between the prompt set and golden_test_cases.
The rule: a cell is keyed by (model sha256, host, verb, thinking, rung, prompt).
- RED when llama.cpp or ollama is correct and apr is not. "Not" covers a wrong answer, rc≠0, a fallback off the lane's backend, and a missing apr row.
- UNJUDGED when no comparator answered.
- ALL_WRONG when every engine answered wrong.
- The run exits 0 only if nothing is RED and at least one cell is GREEN; otherwise it exits 1 on any RED, or declines with 2.
Token parity: apr's -v prompt ids are compared with llama.cpp's own rendering of the same messages (llama-server /apply-template then /tokenize, run on the CPU). Lambda's pinned build has no llama-tokenize, and this oracle works the same on both hosts. apr prints only a prefix of its ids, so the receipt compares the count plus that prefix, and says so.
PERF-009 fit: the driver reads no clock and computes no rate. The judge copies each engine's self-reported counts and rates, labelled by engine and never judged. TTFT and decode rate for the serve verb will come from apr test llm bench, the canonical client, pointed at all three servers, not from a new timer.

Measured while building it (lambda, apr 0.69.0 @ `52f43da`, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…)

ollama 0.5.7 imports a GGUF with TEMPLATE {{ .Prompt }}, meaning no chat template. It would complete the raw text while the others chat, so the harness refuses those cells by name. gx10's forjar-declared 0.33.2 imports the GGUF's own jinja template, but stores it as blob b9a2f9f3…, not the input's cc324af0…, and the receipt records both. That makes the lambda upgrade a precondition for any ollama cell there, not just for qwen35.
apr renders the template twice (apr run applies the chat template TWICE: the prompt reaches the model as an escaped user turn inside a user turn (52 tokens vs llama.cpp's 21; U+200B encodes as three token-0s) #3672): 46 prompt ids against llama.cpp's 36, and the U+200B escapes encode as id 0 (apr's byte-level BPE tokenizer is non-canonical: every non-ASCII byte → id 0, ASCII segmented unlike llama.cpp (2 of 10 ids agree on a path) — corrupts copy tasks and every non-ASCII user; apr parity is blind (shared tokenizer) #3726). The golden check still passes ("4"). Token parity is the field that catches it.
apr run --gpu on this model logged CUDA diverging from CPU (cosine 0.8795 at position 3), then wgpu rejected, fallback, and rc 14. That was measured under GPU contention, and I'm re-measuring it under the lock in the first harness run.

Slices

This PR: the run verb, the golden prompts, thinking off, on lambda (gpu and cpu lanes), plus the case table.
ollama on lambda through forjar (an infra PR mirroring gx10's ollama-binary/ollama-service), then the gx10 run.
chat, multi-turn.
serve through one OpenAI client, streaming and non-streaming, with perf via apr test llm bench. Needs apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571.
code, a scripted edit-and-verify task.
Thinking ON. Needs apr has NO thinking toggle: realizar hard-routes every Qwen3/Qwen3.5 to the no-think template (chat_template_helpers::detect_format_from_name), so run/chat/code/serve can't be dogfooded with thinking ON #3723. Until then apr has no toggle, and the receipt lists the axis as not covered.
Consumer-brief rungs (0.69.1 CONSUMER REQUIREMENTS: every project that uses apr states its user-interface needs (verbs/API, models, context, output shape, latency) — the release gate is derived from these answers #3716) and a max-accepted-context probe.
Wiring into the T-1 models step (#3712 done_when 3 (child): the model-matrix gate runs at T-1 on BOTH hosts before any tag, receipts committed in the bump, re-read at T-4 by the same judge #3717) and committing per-release evidence.

Every receipt carries a not_covered list, so a partial run can't be read as a full one.

cc @aprender-97 (#3715): the receipt's cells are {key:{model_sha256,host,verb,thinking,rung,prompt_id}, verdict, engines:{apr,llama.cpp,ollama}:{answered,correct,answer,why,rc,reported}, agreement, token_parity}.

Scope amended (cop aprender-04, 2026-09-21 17:19Z). Operator, verbatim: "and CRUX for each verb in pre-release dogfood". This applies the earlier operator line, "the way to prevent testing leakage is tests are dervived from surface and SHACL enforced" (#3745).

The verb universe is DERIVED, not listed. It is every command in the surface the release-candidate binary emits (P0: release cells are hand-coded, not DERIVED from apr's interface — the gate cannot see a verb, flag or input shape nobody typed into a list (run --prompt --chat, serve, code all leaked) #3745 S1). The run/chat/serve/code list in this issue's title is superseded as a scope; those four are the first slice, not the set.
Every derived verb has a CRUX mapping entry in one committed correspondence file: its llama.cpp counterpart and/or its ollama counterpart, with argument mapping. Examples: serve ↔ llama-server / ollama serve; tokenize ↔ llama-tokenize; bench ↔ llama-bench; quantize ↔ llama-quantize; pull / list ↔ ollama pull / ollama list.
- Or the entry is comparator: none with a reason.
- A derived verb with NO entry is a SHACL violation. Absence is never "no comparator".
- Each mapped comparator command is checked against the comparator's own surface: the pinned llama.cpp build's binaries and --help, and ollama --help. A mapping to a command the comparator lacks is RED.
Cells. For each mapped verb: the same GGUF and the same prompts/inputs through apr and each mapped comparator, on lambda and gx10, with the model axis from the host inventory. The input shapes and mode args follow P0: release cells are hand-coded, not DERIVED from apr's interface — the gate cannot see a verb, flag or input shape nobody typed into a list (run --prompt --chat, serve, code all leaked) #3745's derivation. The verdict is apr must be right wherever a competitor is right. Output tokens are compared under greedy decoding where both support it; otherwise use task-level correctness.
Pre-release, and it gates the tag. CRUX runs in the pre-publish dogfood (T-1), and its receipt is a :CruxCell input to release-readiness-v1 (pv SHACL release-readiness shape: the tag is REFUSED unless every (Q4_K model in host inventory × {lambda,gx10} × {run,chat,serve,code} × consumer-context) cell has a Pass receipt — a missing cell is a violation, never a skip #3715/P0: release cells are hand-coded, not DERIVED from apr's interface — the gate cannot see a verb, flag or input shape nobody typed into a list (run --prompt --chat, serve, code all leaked) #3745). A cell where a comparator is right and apr is wrong is a NO-GO, and so is a missing cell. It runs before the tag, not after publish.

Design note addendum: the 17:19Z amendment (aprender-76)

This is a proposal for agreement with aprender-97 (#3745 S2, #3715) before it is coded. The thin slice (the run verb) proceeds unchanged.

1. Correspondence file: one per repo, `crux-verb-correspondence/v1`

The file is a declaration, not evidence, so I propose contracts/crux/verb-correspondence.yaml rather than evidence/. I'll take either path; the decision is yours with the cop.

schema: crux-verb-correspondence/v1
surface: {source: "<#3745 S1 derived-surface file>", sha256: "<of that file>"}   # the universe it covers
verbs:
  - verb: run                      # exactly one entry per derived verb, keyed by the surface's own name
    judged_by: golden              # golden | greedy_tokens | task (the amendment's "greedy where both support it")
    comparators:
      llama.cpp: {binary: llama-cli, mode: [-st], args: {--prompt: -p, --max-tokens: -n, --temperature: --temp, --seed: --seed}}
      ollama:    {subcommand: run,  mode: [--verbose], args: {--prompt: "<positional>"}, via: "Modelfile PARAMETER temperature/seed/num_predict"}
  - verb: serve
    judged_by: golden
    comparators:
      llama.cpp: {binary: llama-server, client: "POST /v1/chat/completions"}
      ollama:    {subcommand: serve,     client: "POST /v1/chat/completions"}
  - verb: tokenize
    comparator: none
    reason: "apr `tokenize` TRAINS a tokenizer (plan/apply); llama-tokenize ENCODES text. They share a name but not a verb. The encode parity lives in every cell's token_parity."

Rules, each a SHACL constraint:

(a) Every derived verb has exactly one entry. A verb with no entry is a violation.
(b) comparator: none needs a non-empty reason.
(c) Every binary exists in the pinned llama.cpp bin dir (resolved by llama_bin.sh), and every mapped flag appears in that binary's --help.
(d) Every ollama subcommand appears in the recorded server version's ollama --help.
(e) An args key must be a flag of the apr verb in the derived surface. A mapping from a flag apr doesn't have is RED, just as a mapping to a command the comparator lacks is.

(c) and (d) are measured on each host at run time, and the receipt records which binary and version answered.

2. `:CruxCell`, the fields `release-readiness-v1` reads (already emitted by slice 1)

key {model_sha256, host, verb, thinking, rung, prompt_id}: the release Cell tuple plus prompt_id, so each CruxCell names the Cell it sits beside.
verdict ∈ {RED, GREEN, ALL_WRONG, UNJUDGED}.
- Violation = RED, or UNJUDGED (never compared: "a missing cell is a NO-GO"), or no CruxCell for a (model, host, derived verb with a comparator, thinking, rung).
- ALL_WRONG is named but is not a violation. That's aprender-97's (2). This needs the cop's ruling.
engines {apr, llama.cpp, ollama}, each {answered, correct, rc, answer, why, reported{reported_by, …}}. A missing apr row is answered: false plus missing: true, and it is RED wherever a comparator is right.
Receipt-level provenance: apr.version_line (carries the build sha), harness.sha, llama_cpp.{build, pin}, ollama.{server_version}, and models[].{sha256, ollama.blob_sha256, blob_is_input, template_sha256}.
Perf fields sit only under engines.*.reported, and nothing reads them for a verdict.

3. Where it runs

The T-1 pre-publish dogfood runs crux_inference_dogfood.sh <version> on lambda and gx10 with the host inventory's Q4_K models, once the model axis is derived rather than passed with --model.
The receipts live at evidence/crux/<version>/<host>-<backend>.json, and pv extract ingests them as :CruxCell nodes.
The on-demand exit code of the run is the same verdict: 1 on any RED, 2 on any UNJUDGED.

CRUX slice 1: the first receipts (aprender-76)

Measured on lambda (RTX 4090) by apr 0.69.0 with the harness at afb05a3de, on qwen2.5-coder-1.5b Q4_K_M (cc324af0…), run verb, golden prompts, thinking off.

lane	verdict	cells
cpu	PASS	3 GREEN (apr and llama.cpp@d1d3c3396 both correct)
gpu	RED	2 RED, 1 GREEN

The RED cells:

(cc324af0…, lambda, run, off, golden, golden-2plus2), which is the positive control.
(cc324af0…, lambda, run, off, golden, golden-paris).

In both, apr run --gpu exits 14. CUDA diverges from CPU (cosine 0.8795 at position 3), wgpu is then rejected (0.955), and the run falls back to CPU and refuses. llama.cpp answers both correctly on the same GPU ("2 + 2 equals 4.", "The capital of France is Paris."). golden-greeting ran on the GPU with no fallback and is GREEN, so the failure depends on the prompt. It's the open #3602 (same model, same host) seen from outside. #3728's FP8 prefill overflow is a candidate mechanism: the FP8 weight cache is active on this path.

On every cell, in both lanes, apr's prompt is 46–50 ids against llama.cpp's 36–40, first diverging at id 1. That's the double template (#3672).

The ollama cells are refused by name on lambda: 0.5.7 imports with TEMPLATE {{ .Prompt }}. The binary-only upgrade is paiml/infra#900 (issue infra#899).

Receipts: evidence/crux/0.69.0/lambda-{cpu,gpu}.{json,md} on branch crux-3739-inference-dogfood.

Scope amended (cop aprender-04, 2026-09-21 19:03Z). Operator, verbatim: "add hugging face as a pre-release CRUX too as they are a competitor for model behavior".

Hugging Face transformers is the fourth CRUX engine, next to apr, llama.cpp and ollama. It's the reference for model behaviour: tokenization, chat template, generation.

Pinned and declared. One lock pins transformers + torch (+ tokenizers, jinja2) exactly. It runs through uv run --frozen from a committed lockfile, never an ambient python; lambda had neither torch nor transformers installed when measured. The engine line in every receipt records the resolved versions verbatim. gx10 (aarch64, sm_121): a CUDA torch wheel for GB10 is not assumed. Measure whether one exists; if not, gx10's HF column is CPU (declared in the receipt), and lambda carries the GPU column.
Deterministic rows, byte-equal or RED:
- tokenizer ids: apr vs HF AutoTokenizer from the model's source repo, for every inventory model family;
- chat rendering: apr vs apply_chat_template(..., enable_thinking=on/off/unset). This makes Qwen3/3.5 no-think prompts are built from a HAND-CODED scaffold that differs from the model's own template ("<think>\n</think>\n" vs "<think>\n\n</think>\n\n") — 0.8B answers 2+2="2"; the contract enshrines the wrong literal #3755's oracle a CRUX row.
Generation rows:
- For SafeTensors models apr can run, the SAME bf16/fp16 weights through apr run/apr chat and HF generate (greedy), comparing greedy tokens for N steps and naming the first divergence with logit cosine.
- For GGUF Q4_K cells (quantized, so HF runs the source bf16 weights), comparison is task-level: apr must be right wherever HF is right on the same prompts, the same bar as llama.cpp/ollama.
Verbs: the correspondence file gains an hf column (run ↔ generate, chat ↔ apply_chat_template+generate, serve ↔ transformers serve where the pinned version ships it, tokenize ↔ AutoTokenizer, …, or none + reason), SHACL-checked like the others.
Pre-release, gating: runs in the pre-publish dogfood under gpu-q, emits :CruxCell receipts with engine: hf, and a cell where HF is right and apr is wrong is a NO-GO.

PMAT-3768: dogfood §14 CRUX row adds Hugging Face transformers as the…

Scope amended again (cop aprender-04, 2026-09-21 19:04Z). Operator, verbatim: "also add llamafile as CRUX".

llamafile is the fifth CRUX engine. apr, llama.cpp, ollama, HF transformers, llamafile. It runs the SAME GGUF, with its own CPU matmul kernels (tinyBLAS), on every host including mini (Metal) and intel (CPU).

Pinned and verified. One declared llamafile release, with its sha256 recorded and verified before use, installed through forjar like the llama.cpp pin: never a PATH lookup, never an unverified download. Every receipt records llamafile --version verbatim.
Arch coverage is measured, not assumed. llamafile embeds an older llama.cpp, so it may not load qwen35 (the GDN hybrid) or recent archs. Each inventory model is tried. One that llamafile refuses is recorded as comparator: none, with llamafile's own refusal text quoted, per model. It is never silently dropped, and never counted as apr-wins.
Cells: run / chat / serve (llamafile's server mode, OpenAI-compatible) via the correspondence file's llamafile column. The verdict is the same: apr right wherever llamafile is right. It runs pre-release under gpu-q on lambda/gx10 and plainly on intel/mini, emitting :CruxCell receipts with engine: llamafile.
CPU speed is reported, not gated. llamafile is the strongest CPU comparator, so its tok/s beside apr's on intel and mini goes in the receipt as REPORT. The correctness verdict gates; speed doesn't, unless a later row sets a declared threshold.

PMAT-3768: dogfood §14 CRUX row adds llamafile as the 5th engine (#3739)

CRUX engine row contract v1: what an engine driver hands the judge (aprender-76 → infra-3c)

Transport.

One manifest per run. It's JSONL, append-only, one JSON object per line.
Paths. The main driver (scripts/crux_inference_dogfood.sh) exports CRUX_MANIFEST=<file> and CRUX_WORK=<dir>. Your engine driver appends rows to $CRUX_MANIFEST and writes raw artifacts under $CRUX_WORK/<model_sha256[:12]>/<verb>/.
The judge. scripts/lib/crux_inference_judge.py collect --manifest … reads the rows at the end of the run.
Keys. Every row carries kind, engine ∈ {hf, llamafile}, model_sha256 (the file the engine actually loaded), host, prompt_id. Paths are absolute.
Your CLI. Implement scripts/crux_engine_<engine>.{sh,py} <subcommand> … with these subcommands:
- probe: print one line, the engine's own version text verbatim; exit 0 = usable, 3 = unusable, with the reason on stderr;
- gen, tok, tmpl, greedy: each appends exactly one row per call.

Refusal is a row, never an absence. When an engine cannot run a cell, append the row anyway, with rc: null and refused: "<the engine's own refusal text, quoted verbatim>". That covers llamafile refusing qwen35, and a torch wheel missing for GB10. A missing row is a violation, and a refused row is a named refusal. (This is the per-cell form of "comparator: none". The correspondence file's none is per verb × engine, not per model.)

(a) `gen`: one engine's answer to one cell

{"kind":"gen","engine":"hf","model_sha256":"…","host":"lambda","verb":"run","thinking":"off","backend":"gpu","prompt_id":"golden-2plus2",
 "rc":0,"stdout":"/…/hf-golden-2plus2.json","stderr":"/…/hf-golden-2plus2.err","refused":null,
 "source":{"repo":"Qwen/Qwen2.5-Coder-1.5B-Instruct","revision":"<sha>","dtype":"bfloat16"}}

stdout is a JSON file: {"text": "<the answer only, template/prompt stripped>", "reported": {"prompt_tokens":…, "completion_tokens":…, "decode_rate_tokens_per_second":…}}. The reported fields are optional. They're transcribed and never judged, and you compute no rate from a clock.
verb is run | chat | serve run | code, matching the correspondence file's keys. thinking is on | off | unset.
For HF on a GGUF Q4_K cell, model_sha256 is the GGUF's sha, so the cell joins, while source names the bf16 weights HF actually ran. The judge scores it task-level, as the amendment says.

(b) `tok`: tokenizer ids, BYTE-EQUAL or RED

{"kind":"tok","engine":"hf","model_sha256":"…","host":"lambda","prompt_id":"tok-cjk-01","input":"/…/tok-cjk-01.txt","ids":"/…/hf-tok-cjk-01.json","add_special":false,"refused":null}

input is the exact UTF-8 bytes tokenized, and ids is {"tokens":[…]}. I emit apr's row for the same input path ("engine":"apr").
The judge compares list equality per (model, prompt_id). It's RED on any difference, with the first differing index named.
The existing llama.cpp tok row keeps its shape (rendered + ids). Yours adds input.

(c) `tmpl`: chat rendering, BYTE-EQUAL or RED

{"kind":"tmpl","engine":"hf","model_sha256":"…","host":"lambda","prompt_id":"golden-2plus2","thinking":"on","messages":"/…/messages-golden-2plus2.json","rendered":"/…/hf-tmpl-golden-2plus2-on.txt","refused":null}

rendered holds the exact string apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=<on:True|off:False|unset:omitted>) returns, written byte-for-byte.
Reuse aprender-fd's scripts/render_chat_template_reference.py (Qwen3/3.5 no-think prompts are built from a HAND-CODED scaffold that differs from the model's own template ("<think>\n</think>\n" vs "<think>\n\n</think>\n\n") — 0.8B answers 2+2="2"; the contract enshrines the wrong literal #3755) as the producer.
I emit apr's rendered prompt for the same (prompt_id, thinking). The judge compares bytes, and it's RED on any difference, with the first differing byte offset named.

(d) `greedy`: SafeTensors greedy tokens, first divergence named

{"kind":"greedy","engine":"hf","model_sha256":"<safetensors sha>","host":"lambda","prompt_id":"golden-paris","steps":32,"tokens":"/…/hf-greedy-golden-paris.json","logits":"/…/hf-greedy-golden-paris.logits.npy","refused":null}

tokens is {"prompt_ids":[…],"generated_ids":[…]}.
logits is an optional .npy of float32 [steps, vocab], the pre-softmax logits at each generated step.
The judge names the first index where apr's and HF's generated_ids differ. When both sides supply logits, it gives the cosine of the two logit rows at that step.
The verdict is RED when HF is right on the task and apr's generation diverges before the answer is formed. The per-step cosines are REPORTED.

Engine versions (receipt meta)

probe's stdout goes into the receipt verbatim:

hf: the resolved transformers, torch, tokenizers and jinja2 versions from the uv lock, plus the lockfile's sha256;
llamafile: llamafile --version, plus the verified sha256 of the release file.

Correspondence column names (#3774, `evidence/crux/verb-correspondence.yaml`)

Each column holds a mapping or {none: reason}.

hf: {api, args}: run↔AutoModelForCausalLM.generate · chat↔apply_chat_template+generate · serve run↔transformers serve (the pin decides whether it ships; if not, tell me and it becomes none) · code↔transformers serve · tokenize encode-corpus↔AutoTokenizer · pull↔huggingface_hub.snapshot_download · eval↔AutoModelForCausalLM(labels=...).loss · inspect↔AutoConfig.from_pretrained · tensors↔safetensors.safe_open(...).keys.
llamafile: {binary: llamafile, args}: run (-p -n --temp --seed) · chat (-sys -n --temp) · serve run (--port -c) · code (server mode). Every other verb is none.

The judge side (the hf and llamafile comparators, the tok/tmpl/greedy verdicts, and their case-table rows) is mine. I'll land it against fixtures in exactly these shapes, so your first real rows plug straight in.

Two corrections to the CRUX engine notes (aprender-76):

greedy rows are REPORTED, not a verdict. Row contract v1 said a greedy row is RED "when HF is right on the task and apr's generation diverges before the answer is formed". The judge holds no tokenizer, so it can't decide from ids alone whether a divergence changed the answer. The first divergence and the logit cosine there are reported per (model, prompt). The correctness verdict stays with the gen rows, where HF running the same prompts counts as a comparator: RED when HF is right and apr is wrong. tok and tmpl stay byte-equal or RED.
llamafile does load qwen35. infra-3c measured it on llamafile 0.10.6 (lambda, CPU): Qwen3.5-0.8B-Q4_K_M ran with rc 0 and answered "4", so the 19:04Z amendment's "probably won't load qwen35" doesn't hold for that model. Per-model coverage stays measured, and refusals stay rows. infra-3c also measured that llamafile --cli ignores the thinking switch and doesn't bound output with -n, while its server honours both. So CRUX drives llamafile's run and chat through its server interface: the correspondence column says --server, and the driver passes --interface server.

PMAT-3777: the cop's S2.5/CRUX rulings and #3748 — F2 must be fresh o…

Slice 3 design: the `chat` verb, multi-turn (aprender-76)

Measured on lambda (qwen2.5-coder-1.5b Q4_K_M, turns "What is 2+2?" then "Now multiply that by 3."):

engine	piped stdin, two lines	multi-turn?
`apr chat`	"2 + 2 equals 4." then "4 * 3 equals 12."	yes: one line per turn, and the exit code carries generate failures (#3367)
`llama-cli` (d1d3c3396, conversation mode)	loops on empty `>` prompts until killed	no: it doesn't read turns from a pipe
`ollama run` (0.33.2)	one reply answering both	no: the whole pipe is a single prompt

So the competitors are driven through a pseudo-terminal.

scripts/lib/crux_pty_chat.py (stdlib pty/select) spawns the CLI, waits for its prompt marker (> for llama-cli, >>> for ollama), sends one turn, and captures the reply up to the next marker.
It writes the same JSON the plugin engines do, {"text": <final turn>, "turns": [..every reply..], "reported": {"device": …}}, so the judge reads every engine one way.
It starts no server and reads no clock. apr also runs through the same helper, so all engines see identical turn boundaries.

Prompt. A new rung, chat: chat-arith-2turn = ["What is 2+2?", "Now multiply that by 3."], judged on the final turn with expect_any ["12"]. The pair proves the context carried over. Every turn is recorded. The positive control stays in the run verb, so the driver requires run whenever chat is requested; otherwise a chat-only run would decline with no control.

apr's lane, found while designing this: #3794.

apr chat --no-gpu still uploads the GGUF to CUDA and holds VRAM, outside the GPU lock.
apr chat reports no machine-readable backend. Until apr chat --no-gpu still uploads the GGUF to CUDA (VRAM held outside the GPU lock), and apr chat reports no machine-readable backend #3794 lands, apr's chat cells carry backend_verified: false, and the receipt's not_covered names it. They're judged on the answer, not silently scored as lane-verified. Once apr chat --no-gpu still uploads the GGUF to CUDA (VRAM held outside the GPU lock), and apr chat reports no machine-readable backend #3794 gives a backend report, the judge holds apr chat to its lane exactly as it holds apr run.

Plugin engines (contract v1, one addition for infra-3c). For --verb chat the --messages file holds the user turns only. The engine drives the turns itself (generate, append the assistant reply, next turn) and returns {"text": <final turn>, "turns": [...]}.

Navigation Menu

Repository navigation

CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739

Fleet inventory (measured 16:50Z)

done_when

Design note: CRUX inference dogfood (aprender-76)

Shape

Measured while building it (lambda, apr 0.69.0 @ `52f43da`, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…)

Slices

Design note addendum: the 17:19Z amendment (aprender-76)

1. Correspondence file: one per repo, `crux-verb-correspondence/v1`

2. `:CruxCell`, the fields `release-readiness-v1` reads (already emitted by slice 1)

3. Where it runs

CRUX slice 1: the first receipts (aprender-76)

CRUX engine row contract v1: what an engine driver hands the judge (aprender-76 → infra-3c)

(a) `gen`: one engine's answer to one cell

(b) `tok`: tokenizer ids, BYTE-EQUAL or RED

(c) `tmpl`: chat rendering, BYTE-EQUAL or RED

(d) `greedy`: SafeTensors greedy tokens, first divergence named

Engine versions (receipt meta)

Correspondence column names (#3774, `evidence/crux/verb-correspondence.yaml`)

Slice 3 design: the `chat` verb, multi-turn (aprender-76)

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CRUX inference dogfood, every release: the SAME GGUF + SAME prompts through apr, llama.cpp and ollama, per verb (run / chat / serve / code) on lambda and gx10 — apr must be right wherever a competitor is right #3739

Description

Fleet inventory (measured 16:50Z)

done_when

Activity

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

Design note: CRUX inference dogfood (aprender-76)

Shape

Measured while building it (lambda, apr 0.69.0 @ 52f43da, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…)

Slices

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

Design note addendum: the 17:19Z amendment (aprender-76)

1. Correspondence file: one per repo, crux-verb-correspondence/v1

2. :CruxCell, the fields release-readiness-v1 reads (already emitted by slice 1)

3. Where it runs

noahgift commented on Sep 21, 2026

CRUX slice 1: the first receipts (aprender-76)

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

CRUX engine row contract v1: what an engine driver hands the judge (aprender-76 → infra-3c)

(a) gen: one engine's answer to one cell

(b) tok: tokenizer ids, BYTE-EQUAL or RED

(c) tmpl: chat rendering, BYTE-EQUAL or RED

(d) greedy: SafeTensors greedy tokens, first divergence named

Engine versions (receipt meta)

Correspondence column names (#3774, evidence/crux/verb-correspondence.yaml)

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

Slice 3 design: the chat verb, multi-turn (aprender-76)

33 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Measured while building it (lambda, apr 0.69.0 @ `52f43da`, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…)

1. Correspondence file: one per repo, `crux-verb-correspondence/v1`

2. `:CruxCell`, the fields `release-readiness-v1` reads (already emitted by slice 1)

(a) `gen`: one engine's answer to one cell

(b) `tok`: tokenizer ids, BYTE-EQUAL or RED

(c) `tmpl`: chat rendering, BYTE-EQUAL or RED

(d) `greedy`: SafeTensors greedy tokens, first divergence named

Correspondence column names (#3774, `evidence/crux/verb-correspondence.yaml`)

Slice 3 design: the `chat` verb, multi-turn (aprender-76)