[lut-b] data-free round-to-nearest example - #3018
Conversation
Signed-off-by: Brian Dellabetta <bdellabe@redhat.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new example script demonstrating LUTB quantization on Llama 3 and updates the observer initialization logic in calibration.py to prevent errors when observer is None. The review feedback recommends saving the model before dispatching it in the example script to avoid saving incomplete weights, and suggests refining the dynamic quantization check in calibration.py to correctly handle DynamicType.LOCAL and avoid potential AttributeError exceptions.
| ) | ||
|
|
||
| if args is not None and args.dynamic is not True: | ||
| if args is not None and args.dynamic is not True and observer is not None: |
There was a problem hiding this comment.
The check args.dynamic is not True does not account for other dynamic quantization types such as DynamicType.LOCAL (which is checked on line 149). If args.dynamic is DynamicType.LOCAL, this check evaluates to True, which incorrectly initializes a static observer for dynamic quantization.
Additionally, note that if args can be None (as implied by the args is not None check), the function would have already raised an AttributeError on line 62 (observer = args.observer) and line 65 (args.observer). It is highly recommended to add an early return if args is None: at the beginning of the function to prevent this potential crash.
| if args is not None and args.dynamic is not True and observer is not None: | |
| if args is not None and args.dynamic not in (True, DynamicType.LOCAL) and observer is not None: |
|
related -- vllm-project/vllm#51880 |
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews
🔴 Require one maintainer reviewWaiting for any of
This rule is failing.All PRs must have at least one approving review from a maintainer before merging.
|
1 participant
Prerequisite
SUMMARY:
Adds an example for pure round-to-nearest quantization using the LUT-B compressor and QDQ pathway, targeting only MLP layers at the moment.
TODOs:
TEST PLAN:
output of example: