Skip to content

docs(embeddinggemma): quantization is bf16, not Q4_1 - #693

Open
Aitschend3251 wants to merge 1 commit into
ROCm:mainfrom
Aitschend3251:docs-embeddinggemma-quantization
Open

docs(embeddinggemma): quantization is bf16, not Q4_1#693
Aitschend3251 wants to merge 1 commit into
ROCm:mainfrom
Aitschend3251:docs-embeddinggemma-quantization

Conversation

@Aitschend3251

Copy link
Copy Markdown

docs/docs/models/embeddinggemma.md line 14 declares Q4_1. The shipped model is not quantised.

What the repo itself says. src/model_list.json gives embed-gemma/300m:

"details": { "format": "NPU2", "family": "embed-gemma", "quantization_level": "none" }

It is the only entry in that file carrying none, and the docs page is the only place Q4_1 appears for this model.

What the artefact says. model.q4nx is 615,197,168 bytes for a ~308M-parameter model — 2.0 bytes per parameter, i.e. bf16 despite the .q4nx extension. Genuine 4-bit weights would be near 0.5 bytes/parameter. Verified independently on two machines and two FLM builds (v0.9.45 and v1.0.3), and a header parse reports 316 of 316 tensors as BF16; the same parser finds 197 quantised tensors in the Qwen3-1.7B-NPU2 container, so that is a finding rather than a parser artefact.

Why bf16 and not none. docs/docs/models/smolvla.md already uses - **Quantization:** bf16, so this follows existing usage on the docs side while agreeing with model_list.json.

Why it is worth fixing. This line sent #661 down a dead end: the opening report named coarse quantisation as the likely cause of an embedding defect, on the strength of this line, and two of us spent time ruling it out. The likely origin looks innocent — the v1.0.3 release notes upgrade the Qwen3.5 family and Qwen3.6-MoE "from Q4_1 to Q4_K", so Q4_1 was a real value for the LLM containers; it just does not describe this one.

One-line change, docs only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant