PMAT-3774: CRUX verb correspondence (253 leaves × 4 engines) + the judge/driver side of the hf and llamafile engines

docs/roadmaps/entries/PMAT-3774.yaml

@@ -0,0 +1,30 @@
+- id: PMAT-3774
+  github_issue: 3774
+  item_type: task
+  title: 'CRUX verb correspondence (#3739 amendments): every leaf of the derived apr surface maps to its llama.cpp / ollama / hf / llamafile counterpart, or `{none: reason}` per engine'
+  status: in_progress
+  priority: critical
+  assigned_to: aprender-76
+  created: 2026-09-21T19:20:00Z
+  updated: 2026-09-21T22:00:00Z
+  spec: null
+  acceptance_criteria:
+  - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.'
+  - 'done_when 2, verbatim: "Every entry has `comparators` with ALL FOUR engine keys (`llama.cpp: {binary, args}`, `ollama: {subcommand, args}`, `hf: {api, args}`, `llamafile: {binary, args}`), each either that mapping or `{none: <non-empty reason>}`. An entry with at least one mapping has `judged_by` ∈ golden | greedy_tokens | task." MEASURED: all 253 entries state all 4 engines; 16 verbs map at least one engine (run, chat, serve run, code, bench, quantize, pull, list, rm, tokenize encode-corpus, eval, inspect, tensors, devices, modelfile parse, import); 237 are `none` on every engine, each with a reason.'
+  - 'done_when 3, verbatim: "Measured at authoring and recorded, each with a negative control that fails: every mapped llama.cpp `binary` exists in the pinned build (d1d3c3396) and every mapped flag appears in its `--help`; every ollama `subcommand` and flag appears in `ollama --help` (0.33.2); every mapped apr flag is an arg of that verb in the surface." MEASURED: 7/7 llama.cpp binaries and 10/10 flags ok on gx10''s pinned build; 7/7 ollama subcommands plus `create --quantize` ok on lambda''s 0.33.2; every mapped apr flag is present in its verb''s surface args. The controls `llama-cli --frobnicate`, bin `llama-frobnicate` and `ollama frobnicate` all FAIL.'
+  - 'done_when 4, out of scope, verbatim: "the `hf` and `llamafile` mappings are declared here but verified against their pins by the engine worker''s rows, since the pins don''t exist yet; the SHACL reading is aprender-97''s S2; the cells for newly mapped verbs are later #3739 slices."'
+  - 'Added to #3774 at ~19:30Z (the cop: "You own the correspondence file columns (`hf` and `llamafile`) and the judge side (#3774)"), verbatim: "`scripts/lib/crux_inference_judge.py` treats `hf` and `llamafile` as comparators: RED when either is right and apr is wrong." MEASURED by `bash scripts/check_crux_inference_judge.sh`: 33 ok, 0 broke. New rows cover: hf right with apr wrong is RED; a llamafile refusal is quoted and, beside a right llama.cpp, the cell is GREEN; hf output that is not the contract JSON, or JSON with no `text`, is no oracle.'
+  - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.'
+  - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_<hf|llamafile>.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.'
+  - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.'
+  - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-3} -- bash <cell>` (the cop''s priority classes: CRUX development = 3, the train''s T-1 run passes 1) when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".'
+  - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.'
+  - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' 
+  - 'Also measured (~21:45Z): a plugin engine is held to its lane as apr is. An hf/llamafile gen row must carry `reported.device` matching its lane (`cpu*` on the cpu lane, anything else on gpu), and a missing device is no answer. This came out of the integration run, where HF on the cpu lane reached the 4090 outside the GPU lock (infra-3c confirmed it and fixed their side in 9c806f008). Rows: hf on the CPU in the gpu lane, and hf with no device, are both UNJUDGED. Each mutant breaks 1. Cells also carry `positive_control` (aprender-97''s additive field for pv). The table is 38 ok, 0 broke.'
+  - 'INTEGRATION, lambda cpu lane, with infra-3c''s REAL engine drivers (PMAT-3778 @ e719fc11a, copied in untracked for the run and removed after): the hf probe was recorded verbatim (transformers 5.17.0, torch 2.14.0+cu130, lock sha256 be29a4f4…); hf gen rows were appended and judged; the absent llamafile install was refused by name ("probe exit 3: llamafile is not installed at … (forjar resource crux-llamafile, paiml/infra#906)"). It FOUND that the hf engine answered `!` × 64 on all 3 prompts (transformers left lm_head untied; reported to infra-3c), and that the golden greeting pattern scored it correct: #3782 and the degenerate rule above.'
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, evidence/crux/hf-sources.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.'