[Tests] Nightly lmeval tests - #3004
Conversation
…stale configs - Change w4a4_nvfp4 and w4a16_actorder_weight cadence from weekly to nightly - Add FP8 block quantization config with nightly cadence (calibrated thresholds) - Disable fp8_static_per_tensor and w4a16_actorder_group by moving to disabled_configs/ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the cadence of several evaluation configurations from "weekly" to "nightly" and introduces a new configuration file fp8_block.yaml. Feedback was provided regarding the new configuration file, which is missing the dataset_id and dataset_split fields required by the evaluation pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dsikka
left a comment
dsikka
left a comment
There was a problem hiding this comment.
FYI, based on the benchmarking we’ve done + conversations with Michael, fp8 static per tensor may have better performance than dynamic per token and should maybe be a path we focus on more
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
I would make two changes:
- Just remove the disabled one
- Keep the configs in the same
configsfolder - just add both cadences, weekly and nightly and update the cadence handling logic to consider multiple options so that we can be sure that the tests run both during a nightly run or a weekly run
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
kylesayrs
changed the title
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews
🔴 Require one maintainer reviewWaiting for any of
This rule is failing.All PRs must have at least one approving review from a maintainer before merging.
|
2 participants
Purpose
Co-requisites
Changes
Testing
configs (h100-solo): 1h 35m 18s
vl_configs (h100-solo): 47m 21s
distributed_configs (h100-duo): 29m 15s