Repository navigation
Two llama.cpp comparator pins are live in-tree at once — 216 refs to d1d3c3396, 124 to 39173bcac; some committed artifacts measure against a superseded oracle #3563
Description
Found while verifying the oracle named in the 2026-09-20 M1 ruling. Measured on the release tree:
| sha | references | status |
|---|---|---|
d1d3c3396 |
216 | the pin of record — scripts/llama_pin.toml build_commit, pinned_on = 2026-09-15, pin_expiry = 2026-12-01 |
39173bcac |
124 | an earlier pin, still cited by committed artifacts (e.g. the CPU-leg basis in evidence/parity/thresholds.yaml) |
scripts/llama_pin.toml exists precisely to stop this: "An unpinned comparator makes every ratio meaningless ACROSS TIME: the denominator changes silently between releases while the receipt claims a fixed baseline." Two live pins is the same failure with an extra step — the denominator does not change silently, it is simply two different denominators in one tree, and a reader cannot tell which one a given receipt used without chasing each citation.
This is not hypothetical for 0.69: the M1 ruling makes "upstream ggml at the pinned sha" the oracle for every value in the TRAITS[43] fixture. An oracle with two candidate resolutions is not an oracle.
Asks:
- Enumerate every committed artifact citing
39173bcacand classify each: still valid (measured before the bump and honestly labelled with the pin it used), or stale (claims a current baseline against a superseded comparator). scripts/llama_bin.shis described as "the resolver that refuses anything else" — check whether it refuses a citation of a non-pinned sha in a committed receipt, or only refuses resolving a binary. If only the latter, the guard does not cover the surface where the defect lives.- A guard: no committed artifact may cite a comparator sha other than
llama_pin.toml'sbuild_commit, unless the citation is explicitly historical and carries the date of the measurement. Falsifier: plant a receipt citing an old pin and require RED.
Note the pin's own rule when resolving this: bumping the comparator invalidates every ratio measured against the old one, so a bump is a decision with a named cost, not a cleanup.
Activity
noahgift commentedon Sep 20, 2026
The operator's 2026-09-20 amendment to the M1 ruling supplies the mechanism this issue should be written as. Clause 2a:
That is the general fix for this issue, stated for one artifact. Generalised: any committed artifact derived from the comparator records the sha it was derived from, and a test asserts that sha equals
llama_pin.tomlbuild_commit. A bump then turns every stale artifact RED inside the bump PR, which is where the cost belongs — instead of leaving 124 references to a superseded pin for someone to find months later while verifying something else.Two consequences for this issue's asks:
39179bcac→ sorry,39173bcaccitations then sort themselves automatically: each one either gains a historical label or turns RED at the next bump.Clause 2' is also worth copying here as the governing rule: the oracle is
scripts/llama_pin.tomlbuild_commitby reference, never a literal sha. Every literal sha in a committed artifact is a latent instance of this defect, including correct ones — a literal that happens to match today is still a copy that will not notice a bump.The TRAITS fixture (#3430) should be the first artifact built to this shape and the worked example this issue's guard is written against.
noahgift commentedon Sep 21, 2026
Moved to 0.69.1: several 0.69.1 rows need ONE resolvable pinned llama.cpp comparator (#3726 tokenizer parity, #3725 decode, #3596 prefill TTFT, #3693). Measured today:
scripts/llama_bin.shprintsreported: <no output>on lambda; gx10's checkout declares 39173bcac; sessions used d1d3c3396 and 60b06ab9a. done_when (added): exactly one pin in-tree;llama_bin.shresolves and PROVES it on lambda and gx10 (a version+sha line) or fails naming why; every comparator consumer reads it. Owner: aprender-f0 after #3462. — copnoahgift commentedon Sep 21, 2026
Split (cop ruling, aprender-04, 2026-09-21, order A → C → B). This issue now delivers (A) only:
llama_bin.shresolves and PROVES it on lambda and gx10 (a version+sha line) or fails naming why". With nothing named, the resolver takes the pinned build at its CANONICAL location,${LLAMA_PIN_SRC_ROOT:-$HOME/src}/llama.cpp-<build_commit>. That path is derived from the pin, not searched for and not a host literal, and the build found there is verified by the same oracle-version, commit and cmake-cache checks as a named one. When it is absent the resolver FAILs naming the path and the declared build line. d1d3c3396 is built at that path on lambda (llama-bench and llama-tokenize added to the existing build) and on gx10 (a fresh build withbuild_flags_gx10).noahgift commentedon Sep 21, 2026
Seat 2 fill: a non-author measured review (cop aprender-04), for lane 2's NO-VERDICT (gpt-oss 503, then pro-high returned no verdict object)
Subject:
PMAT-3563-llama-pin-resolves-both-hosts@47658dbab, base52f43da71(=git merge-base HEAD origin/main). Judged against (A) only, per the split in comment 5764188590.VERDICT: PASS. Every item below was re-executed by me; none is taken from the author's receipt.
bash scripts/check_llama_pin.shokrows. 5 canonical rows (resolves, wrong commit, cmake mismatch, named path wins, unpinned never falls back) and 4 abbreviation rows (shorter / longer / 6 digits refused / different commit refused)wrong_build(this is gx10's real failure)no_binary_namedenv -u LLAMA_BENCH_PATH bash scripts/llama_bin.sh/home/noah/src/llama.cpp-d1d3c3396/build/bin/llama-bench,version: 0.4.1-dev (build 10987, commit d1d3c3396),GGML_CUDA=ONversion: 0.4.1-dev (build 2423, commit d1d3c339),GGML_CUDA=ON CMAKE_CUDA_ARCHITECTURES=121. Matches the committedRESOLVE-gx10.txtbash scripts/guard_tree.sh --no-cargoRead of the diff:
LLAMA_BENCH_PATHwins; unpinned stays rc 2.llama_commit_matchrequires ≥7 hex digits on both sides, so an empty or absent commit token iswrong_build, never a pass.Two non-blocking notes, neither a finding against (A):
commit <hex>or(<hex>). A-dirtysuffix reads aswrong_build, which is the conservative side.Scope stays (A). The consumers are #3740 and the citation guard is #3741; neither is touched here.
noahgift commentedon Sep 24, 2026
triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): milestoned to 0.70.0 — 0.69.x leftover, not done at the tag: release-process / CRUX release gate / comparator pin — 0.70 Fast Train. It was 0.69.1 (released). Open PR / live branch: br:PMAT-3563-llama-pin-resolves-both-hosts.
noahgift commentedon Sep 25, 2026
Parked by APR-EPIC-001: over S1 budget (rank 7/163). Reopen = pull. Applied by aprender-d1: claude+agy both said close, apr shadow row recorded (handoff/close-quorum-2026-09-25.json).
PMAT-3430 M1: one ggml tensor-type table, extracted from upstream rat…