feat: add Kimi-K3 model definition and quantization example by kylesayrs · Pull Request #2994 · vllm-project/llm-compressor

kylesayrs · 2026-08-02T19:06:37Z

Summary

Adds vendored Kimi-K3 model implementation (KimiK3ForConditionalGeneration) with configuration, vision processing, tokenization, and encoding support
Adds quantization example that loads a quantization config from pretrained, adds ignore patterns for residual projections and routed experts, and passes the modified config as a quantization_config loading argument
Adds _apply_attn_res to default tracing ignore list for Kimi-K3 compatibility with the sequential pipeline

Test plan

Verify example script runs with a small Kimi-K3 model (e.g., inference-optimization/Kimi-K3-0.18B)
Verify tracing succeeds with _apply_attn_res in ignore list
Verify the vendored model definition loads and generates text correctly

🤖 Generated with Claude Code

coderabbitai · 2026-08-02T19:06:44Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: ee585514-0910-4ff9-8655-e141fc947e9f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-08-02T19:06:44Z

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

mergify · 2026-08-02T19:07:12Z

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

gemini-code-assist

Code Review

This pull request introduces support for the Kimi-K3 model, adding its configuration, XTML encoding helpers, multimodal and vision processors, model architectures, and a tokenizer, along with a quantization example. The review feedback identifies a critical bug in modeling_kimi_k3_linear.py where the optimized inference path is hardcoded to False. Additionally, it highlights several improvement opportunities, such as replacing unsafe assert statements used for runtime input, file, and dependency validation with standard if checks that raise appropriate exceptions, and defensively using .get() and type checks to prevent potential KeyError or TypeError crashes.

brian-dellabetta

overall lgtm, couple questions. when you have empty checkboxes in your PR summary, does that mean PR is still WIP or that you've added each of those?

brian-dellabetta · 2026-08-03T16:59:46Z

do you have a record at all of what you had to change to get this to work? In case we need to modify later?

Added a diff record!

sorry, where is the diff? is it a commit?

mergify · 2026-08-10T19:46:17Z

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

Add vendored Kimi-K3 model implementation (KimiK3ForConditionalGeneration) with configuration, vision processing, tokenization, and encoding support. Add quantization example that loads a quantization config from pretrained, adds ignore patterns for residual projections and routed experts, and passes the config as a loading argument. Also adds _apply_attn_res to default tracing ignore list for Kimi-K3 compatibility with the sequential pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

…files Each kimi_k3 file now has a comment at the top documenting implementation differences from the original files at moonshotai/Kimi-K3 on Hugging Face. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mergify · 2026-08-10T19:51:25Z

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

…, remove formatting-only comments - Rename modeling_kimi_k3_linear.py back to modeling_kimi_linear.py to match the upstream HuggingFace filename - Remove the re-export shim that was in modeling_kimi_linear.py - Remove "Differences: formatting only" comments from files with no substantive changes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mergify · 2026-08-10T20:00:04Z

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

dsikka · 2026-08-11T18:01:20Z

+from llmcompressor.modifiers.quantization import QuantizationModifier
+from llmcompressor.utils import load_context
+
+MODEL_ID = "moonshotai/Kimi-K3"


can we include in key models: https://docs.vllm.ai/projects/llm-compressor/en/latest/key-models/

mergify · 2026-08-12T17:54:29Z

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require one maintainer review	👀 reviews

🔴 Require one maintainer review

Waiting for any of

approved-reviews-by=HDCharles
approved-reviews-by=brian-dellabetta
approved-reviews-by=dsikka
approved-reviews-by=kylesayrs
approved-reviews-by=yiliu30

This rule is failing.

All PRs must have at least one approving review from a maintainer before merging.

kylesayrs requested review from brian-dellabetta and dsikka as code owners August 2, 2026 19:06

mergify Bot added the quality-failed label Aug 2, 2026

gemini-code-assist Bot reviewed Aug 2, 2026

View reviewed changes

brian-dellabetta reviewed Aug 3, 2026

View reviewed changes

kylesayrs requested a review from yiliu30 as a code owner August 10, 2026 19:45

mergify Bot removed the quality-failed label Aug 10, 2026

mergify Bot added the quality-failed label Aug 10, 2026

kylesayrs and others added 3 commits August 10, 2026 15:49

Apply suggestions from code review

63f8033

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

docs: add comments noting differences from upstream Kimi-K3 modeling …

566f0e1

…files Each kimi_k3 file now has a comment at the top documenting implementation differences from the original files at moonshotai/Kimi-K3 on Hugging Face. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

kylesayrs force-pushed the kimi-k3-example branch from f849c0d to 566f0e1 Compare August 10, 2026 19:50

mergify Bot removed the quality-failed label Aug 10, 2026

mergify Bot added the quality-failed label Aug 10, 2026

mergify Bot removed the quality-failed label Aug 10, 2026

mergify Bot added the quality-failed label Aug 10, 2026

dsikka reviewed Aug 11, 2026

View reviewed changes

Uh oh!

Conversation

kylesayrs commented Aug 2, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Aug 2, 2026

Uh oh!

mergify Bot commented Aug 2, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brian-dellabetta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

brian-dellabetta Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

kylesayrs Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

brian-dellabetta Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 10, 2026

Uh oh!

mergify Bot commented Aug 10, 2026

Uh oh!

mergify Bot commented Aug 10, 2026

Uh oh!

dsikka Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 12, 2026

Merge Protections

🔴 Require one maintainer review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Aug 2, 2026 •

edited

Loading