docs: model-size guidance for scheme selection, language alignment for calibration data - #3025
docs: model-size guidance for scheme selection, language alignment for calibration data#3025spped2000 wants to merge 2 commits into
Conversation
…ent to dataset selection choosing-scheme.md organizes schemes by GPU capability only, which answers 'what can my GPU run' but not 'what can my model absorb'. Adds a short section with paired measurements showing 4-bit cost growing sharply as parameter count falls (~30B W4A4 within noise; ~7B -2.4 knowledge and -10.2 instruction-following, both significant), the practical rule to prefer W4A16 below ~10B, and the observation that activation and weight quantization fail on different task types. choosing-dataset.md has no guidance on the language of calibration data and every listed dataset is English. Adds a Language alignment subsection with the mixed-set recommendation, a short snippet for building one, and a note to evaluate in the target language. Signed-off-by: spped2000 <spped2000@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: spped2000 <spped2000@gmail.com>
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
mergify
Bot
added
documentation
Merge Protections🔴 2 of 2 protections blocking · waiting on 👀 reviews
🔴 Require one maintainer reviewWaiting for any of
This rule is failing.All PRs must have at least one approving review from a maintainer before merging.
🔴 Require two reviewsWaiting for
This rule is failing.PRs labelled "two-reviews" must have at least two approving reviews before merging.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the documentation to include guidelines on language alignment during calibration and how model size affects the choice of quantization schemes (specifically comparing W4A4 and W4A16). The review feedback suggests correcting a spelling inconsistency ("behaviour" to "behavior") for regional consistency and refining the speed comparison between W4A16 and W4A4 to accurately reflect compute throughput differences at larger batch sizes.
| A mixed set is usually enough: half target-language, half English preserves | ||
| English behaviour while covering the target language's activation ranges. In |
There was a problem hiding this comment.
The rest of the documentation consistently uses American English spelling (e.g., 'optimization', 'quantization'). To maintain consistency, 'behaviour' should be spelled as 'behavior'.
| A mixed set is usually enough: half target-language, half English preserves | |
| English behaviour while covering the target language's activation ranges. In | |
| A mixed set is usually enough: half target-language, half English preserves | |
| English behavior while covering the target language's activation ranges. In |
| - **Below roughly 10B parameters, prefer weight-only (W4A16) over W4A4**, or | ||
| step up to FP8. Re-running the same 7B with W4A16 recovered the knowledge | ||
| and instruction-following losses entirely (−0.7 pt and −3.7 pt, neither | ||
| significant) at the same size and speed. |
There was a problem hiding this comment.
While W4A16 and W4A4 share the same 4-bit weight footprint (leading to similar memory bandwidth limits at batch size 1), they do not have the same compute speed. W4A4 can leverage 4-bit Tensor Cores (e.g., NVFP4 on Blackwell) for significantly higher compute throughput at larger batch sizes or longer contexts, whereas W4A16 is limited by 16-bit activation math. To avoid misleading users, it is better to qualify the speed comparison.
| - **Below roughly 10B parameters, prefer weight-only (W4A16) over W4A4**, or | |
| step up to FP8. Re-running the same 7B with W4A16 recovered the knowledge | |
| and instruction-following losses entirely (−0.7 pt and −3.7 pt, neither | |
| significant) at the same size and speed. | |
| - **Below roughly 10B parameters, prefer weight-only (W4A16) over W4A4**, or | |
| step up to FP8. Re-running the same 7B with W4A16 recovered the knowledge | |
| and instruction-following losses entirely (−0.7 pt and −3.7 pt, neither | |
| significant) at a similar memory footprint (though without the compute speedups of W4A4 at larger batch sizes). |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…d claim - behaviour -> behavior for consistency with the rest of the docs. - 'same size and speed' overstated the comparison: W4A16 matches W4A4 only where decode is bandwidth-bound. At larger batch or longer context W4A4 can use 4-bit tensor cores and wins on compute throughput. Qualified. Signed-off-by: spped2000 <spped2000@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: spped2000 <spped2000@gmail.com>
|
Thanks — both points taken, pushed:
Also adding the |
|
Note: I cannot add the |
1 participant
Two documentation gaps I ran into while quantizing Thai models, each backed by paired measurements.
1.
choosing-scheme.md— model size, not just GPUThe scheme table answers what can my GPU run. It does not answer what can my model absorb, and that turned out to matter more than hardware for us. Same recipe (NVFP4 W4A4 via
QuantizationModifier), same 512-sample calibration set, same machine, paired per-item against each model's own BF16 source with exact McNemar:Re-running the same 7B as W4A16 recovered both (−0.7 and −3.7 pt, neither significant) at the same footprint and speed — hence the added rule of thumb to prefer weight-only below ~10B.
The two halves of W4A4 also fail on different task types, which the two arms isolate cleanly: dropping activation quantization fixed instruction following and knowledge, but mathematical reasoning was unchanged (−5.4 vs −5.6 pt, both significant) — that damage comes from the 4-bit weights and weight-only does not rescue it.
2.
choosing-dataset.md— language alignment"Domain alignment" covers general/instruct/code, and all four suggested datasets are English. There is currently no guidance that calibration data language matters, even though calibration is what sets the activation scales. Added a short subsection: use a mixed set (we used 50/50 target-language/English, 512 samples), a snippet for building one, and a note to evaluate in the target language too.
Framing
These are one lab's numbers on one model family (Thai-capable Qwen2/Qwen3 derivatives, NVIDIA GB10). I wrote both sections as a starting prior with an explicit "validate on your own workload" caveat rather than as universal rules — happy to soften or cut anything that reads as over-claiming. All runs were pre-registered before measurement and the artifacts are public: https://github.com/spped2000/thaillm-nvfp4-dgx-spark
Disclosure: prepared with assistance from Claude (Anthropic). Every figure quoted comes from paired runs on the hardware described.