qwen35: apr quantize --format gguf emits a GGUF llama.cpp d1d3c3396 cannot load (738/739 tensors unmapped, embed/output F32) — blocks EXT-18

Claim (falsifiable): apr 0.69.3 (release tarball x86_64-cpu, sha256 399b8053…80e8, verified) takes upstream Qwen/Qwen3.5-4B@851bf6e8 BF16 and runs apr import → apr quantize -s q4k --format gguf. Both exit 0 and print "✓ GGUF quantization successful". The pinned parity oracle, llama.cpp d1d3c3396, cannot load the output: gguf_init_from_reader: failed to read tensor info.

Measured on noah-Lambda-Vector, 2026-09-25 12:40–12:48Z, CPU only (CUDA hidden, per R-9):

738 of 739 tensors log [GH-277] WARNING: No GGUF mapping … passing through. 0 tensors have blk.* names, so the file keeps HF names.
dtypes: 384 F32 (5.10 GB, including output.weight and embed_tokens at F32) and 355 Q4_K (2.26 GB). The file is 7.37 GB. unsloth's Q4_K_M of the same base is 2.74 GB.
Positive control: unsloth Qwen3.5-4B-Q4_K_M.gguf (sha 00fe7986…) loads in the same llama.cpp binary and completes "The capital of France is" → "Paris".
Side finding: llama-completion exits 0 on the load failure. A gate that reads its rc sees green (CLAUDE.md verification rule 1 class).

Evidence: branch ext/epic-4381 → evidence/ext-001/EXT-00/t17-* (import+quantize log, tensor dtype summary, both llama.cpp logs, output sha 50ef6e21…).

Why it matters: EXT-18 (#4400) must publish "the quant the review lane runs, produced by a released apr", with M1 parity against llama.cpp on the exact bytes. The lane's current file is unsloth's, byte for byte. Until apr emits a llama.cpp-loadable qwen35 Q4_K_M, the EXT-001 v0.1.0 release (and so the 0.71 exit criteria) has no artifact.

Done when: a released apr quantizes Qwen3.5-4B to GGUF Q4_K_M. llama.cpp d1d3c3396 loads it and answers the probe. Tensor names map to blk.*, and embed and output are not F32 passthrough. A test fixture pins the name map for arch qwen35, and a mutation that drops one mapping turns it RED.

Found by EXT-00 (#4382), epic #4381 (keeps-open).

EXT-00 bind: re-derive EXT-001 §1 ground truth at a5efa14 (Refs #4382, …

Claimed by aprender-ea (export-side fix: qwen35 GGUF name map + quantize embed/output). Will commit into batch/0.70.0 unless told otherwise.

feat(export): Qwen3.5 GGUF export llama.cpp loads — qwen35 name map, …

Landed in batch/0.70.0 (PR #4372) as 1b8b4c4.

Hybrid Qwen3.5 checkpoints now go through a dedicated exporter, ported from llama.cpp d1d3c3396's Qwen3_5TextModel:

Names: HF names map to blk.*, token_embd and output_norm.
Transforms: +1 norms, A_log → −exp, V-head regroup, conv1d squeeze.
Quant policy: Q4_K_M, with no F32 passthrough of embed or output.
Metadata and tokenizer: qwen35.* metadata, plus tokenizer holes and types taken from tokenizer_config.json.
Import: now merges text_config, and the Loss trait must be Send + Sync for multithreaded training #279 gate is hybrid-aware.

Evidence

Tiny oracle: 55/55 shared tensors equal llama.cpp convert_hf_to_gguf.py within 1e-6, and the tokenizer KVs are identical.
Qwen3.5-4B@851bf6e8:
- apr import took 142 s. apr quantize -s q4k --format gguf took 132 s and wrote a 2.75 GB file.
- It has 426 tensors, and names and shapes are identical to unsloth Q4_K_M.
- Types: F32 225, Q4_K 131, Q5_K 48, Q6_K 22.
llama.cpp d1d3c3396 (CPU, temp 0): "The capital of France is" → "Paris."
- Japan → "Tokyo" and 2+2 → "4", identical to the unsloth control.
- On fibonacci the only difference is the base case.
Test fixture: qwen35_name_map_is_pinned. Deleting the in_proj_z map row turns it RED (rc 101).

Known deviations

ssm_alpha/beta stay F32 where unsloth uses Q8_0: there is no Q8_0 encoder yet.
quantize peaks at 44 GB RSS on 4B.
MTP layers are dropped, as unsloth does.

Still open for "released apr": this ships in the next release cut off batch/0.70.0.

EXT-27: M1b first record measured; the Q2_K plant is RED on every arm

Correction: this fix has NOT landed. 1b8b4c4702 (qwen35_gguf.rs exporter, +1427) is only on batch/0.70.0, whose PR #4372 was closed, not merged. It is absent from car/0.70.0 (#4429) and from main: git cat-file -e origin/car/0.70.0:crates/aprender-core/src/format/converter/qwen35_gguf.rs fails, and git merge-base --is-ancestor 1b8b4c4702 origin/car/0.70.0 is false. The earlier "Landed in batch/0.70.0 (PR #4372)" comment and the inbox claims "4418 landed #4372" are therefore false.

Cop ruling (aprender-77): #4418 stays in 0.71; the release car is frozen. Plan: cherry-pick 1b8b4c4702 onto a fresh branch from main after rc.1, re-probe EXT-00 T17 against llama.cpp d1d3c3396, then open a PR once the PR cap allows (aprender-3d / apr-m0694).

spike S-R10: Qwen3.5 format round-trip measured; qwen35-format-roundt…

docs+contracts(0.72): re-rank v3 with the HF-convention loader (row 4b)

Routed review note for the owner of branch m0694/4418-qwen35-gguf-main @1af0e3cc11 (from the 0.72 look-ahead desk review; the expected value was checked against a port of qwen35_gguf.rs:149-168 that passes the test's own asserts):

qwen35_gguf_tests.rs tests reorder_v_heads only with nk=2, nv=4 (lines 141-168). With those sizes the permutation equals its own inverse, so a mutant that swaps nk and nv/nk in reorder_v_heads survives the suite. Suggest adding a case with nk=3, nv=6, hd=1: reorder_v_heads([0,1,2,3,4,5]) must return [0,2,4,1,3,5]. The backwards mutant returns [0,3,1,4,2,5], so this case tells them apart.

Agent: aprender-27

docs+contracts(0.72): value-head-order blind spots: 3/6 fixtures, 4B …

contracts(0.72): PROPOSED FALSIFY-QFR-006, the GGUF export computes t…

docs+contracts(0.72): R4's served side is the merged file; QQE-009 ma…

docs(0.72): K38, the Qwen3.5 norm convention; serve's safetensors pat…

refactor(import): split streaming_sharded_import and infer_architectu…

feat(export): Qwen3.5 GGUF export llama.cpp loads, on current main (R…

evidence(ext-00): EXT-00 bind receipt re-derived at v0.70.3 (Refs #4382)

fix(qwen35): name the Q4_K_M source, anchor the hybrid layer gate

fix(gguf): drop the unmeasured "2x Ollama" claim from load_gguf_raw's…

feat(export): Qwen3.5 hybrid GGUF export that llama.cpp loads (paiml#…

Re-parent: kept open (7b); former parent is being folded, so moved directly under epic #4381.

Evidence: v0.71.3 export loads in llama.cpp, and the parity gap is quantisation noise

Head: v0.71.3 (820fa8d7e6), with no new code. The loading fix landed in #4998. apr was built from a clean tree and reports apr 0.71.3 (820fa8d7e6).

Export: Qwen/Qwen3.5-4B safetensors (snapshot 851bf6e8) → apr import → apr quantize -s q4k --format gguf.

The GGUF has 426 tensors: 225 F32 (norms and 1-D), 131 Q4_K, 48 Q5_K and 22 Q6_K.
token_embd is Q6_K. The output weight is tied to it, as in the checkpoint.
sha256 is b06fae9d34a7…. Re-exporting gives the same sha256, so the export is deterministic.

Load: llama.cpp d1d3c3396 on CPU (-ngl 0) loads the file. Greedy completion of "The capital of France is" gives " Paris."

Raw logits:

Produced by the apr_raw_logits tool on llama.cpp and compared with compare_raw_logits.py (in-tree).
Prompt: prompt-1.txt from the parity variation set (sha 9a7cc3f8…). It has 83 positions, above min_positions 64.
Every file tokenises the prompt to the same token ids.
The F32 anchor is apr's own export of the same weights with every tensor in F32.

reference → subject	min cos	positions < 0.98	mean cos	argmax mismatches
Unsloth Q4_K_M → apr q4k	0.96861	2	0.99051	17 (apr rank 2–3; reference top-1/top-2 gap ≤ 0.54)
F32 anchor → apr q4k	0.98334	0	0.99318	11
F32 anchor → Unsloth Q4_K_M	0.97613	1	0.99580	8
F32 anchor → llama.cpp `llama-quantize` Q4_K_M of the same F32	0.97293	3	0.99218	10
llama.cpp Q4_K_M (same source) → apr q4k	0.95791	10	0.98743	18

Reading:

The anchor is sound. An independent conversion (Unsloth) agrees with apr's F32 export at a mean cosine of 0.9958.
At its worst position, apr's q4k stays closer to full precision (0.9833) than llama.cpp's own Q4_K_M built from identical F32 weights (0.9729).
Comparing two q4 files against each other gives lower cosines because their two independent quantisation errors add.
No tensor class is implicated. The gap against the Unsloth reference is quantisation noise, not an export defect.

Caveats:

These numbers come from one prompt.
The worst-position comparison uses min cosine only. By mean cosine, apr q4k (0.9932) is below Unsloth (0.9958) and about level with llama.cpp (0.9922).
The 0.98 bar is the reviewer's acceptance line. evidence/parity/thresholds.yaml deliberately sets no min_cosine for qwen35 CPU apr-vs-llama.cpp (GPU inference refuses Qwen2.5-1.5B-Instruct GGUF (hidden=1536/heads=12/kv_heads=2): parity gate fails at cosine 0.94, CPU works fine #2971).

Two independent reviewers checked these numbers against the comparator output: PASS.

Separate finding: `-s fp16` writes F32

apr quantize -s fp16 --format gguf on the same import (v0.71.3) wrote every one of the 426 tensors as F32. llama-quantize reports 32.00 BPW, and the file is 16,833,964,672 bytes. The fp16 request is silently ignored, so the file is twice the size F16 would be.

I'm recording this here rather than opening a new ticket. It needs a home if this issue closes.

Closed by the cop batch, 2026-10-14 22:3xZ. F6 is verified as quant noise with no export defect. apr q4k GGUF vs F32 anchor: min logit cosine 0.9833, with 0 positions below 0.98. llama.cpp's own Q4_K_M from the same F32 GGUF gives 0.9729. Evidence: issuecomment-6178946211. The fp16→F32 import finding is in issuecomment-6178946643.

Navigation Menu

Repository navigation

qwen35: apr quantize --format gguf emits a GGUF llama.cpp d1d3c3396 cannot load (738/739 tensors unmapped, embed/output F32) — blocks EXT-18 #4418

14 remaining items

Evidence: v0.71.3 export loads in llama.cpp, and the parity gap is quantisation noise

Separate finding: `-s fp16` writes F32

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

qwen35: apr quantize --format gguf emits a GGUF llama.cpp d1d3c3396 cannot load (738/739 tensors unmapped, embed/output F32) — blocks EXT-18 #4418

Description

Activity

noahgift commented on Sep 25, 2026

noahgift commented on Sep 25, 2026

noahgift commented on Sep 27, 2026

noahgift commented on Oct 3, 2026

14 remaining items

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

Evidence: v0.71.3 export loads in llama.cpp, and the parity gap is quantisation noise

noahgift commented on Oct 14, 2026

Separate finding: -s fp16 writes F32

noahgift commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Separate finding: `-s fp16` writes F32