Skip to content

feat: support selector-local mixed quantization settings - #7

Open
rdzhu225 wants to merge 1 commit into
flagos-ai:mainfrom
rdzhu225:feature/per-selector-quantization
Open

feat: support selector-local mixed quantization settings#7
rdzhu225 wants to merge 1 commit into
flagos-ai:mainfrom
rdzhu225:feature/per-selector-quantization

Conversation

@rdzhu225

Copy link
Copy Markdown
Collaborator

Summary

  • add one-pass mixed quantization through repeated selector-local --select clauses
  • reuse the existing activation-bits, strategy, group-size, scale-dtype, and chunk-size vocabulary per selector
  • keep weight formats explicit (int4 / int8) so future fp4 / fp8 formats remain unambiguous
  • export heterogeneous compressed-tensors groups with per-group pack-quantized / int-quantized formats and the official mixed-precision top-level format when required
  • extend manifest generation, validation, DSV4 target classification, documentation, recipes, and end-to-end coverage

Interface

flagos-compressor quantize \
  --input /path/to/DeepSeek-V4-Flash \
  --output /path/to/DeepSeek-V4-Flash-Mixed \
  --select moe=int4 activation-bits=16 strategy=group group-size=32 \
  --select attention=int8 activation-bits=8 strategy=channel scale-dtype=fp32 \
  --n-candidates 8 \
  --backend cuda

The existing homogeneous interface remains unchanged:

--select moe --select attention \
--bits 4 --activation-bits 16 --strategy group --group-size 32

Selector-local and homogeneous selectors cannot be mixed in one command. In selector-local mode, activation-bits and strategy are required, so W8A16 groupwise, W8A16 per-channel, and W8A8 per-channel are distinguishable.

Scope

Selector-local mixed quantization currently uses the checkpoint-only MSE path. GPTQ, AWQ, and AutoRound are rejected explicitly in this mode. Runtime-specific DeepGEMM integration is intentionally outside this change.

Validation

  • ruff check .
  • local: 136 passed, 1 skipped
  • H20 flagos_quant_infer / flagos: 137 passed
  • real DSV4 W4A16 MoE + W8A8 attention dry-run: 33,924 groupwise INT4 actions, 241 channelwise W8A8 actions, 0 unmatched
  • real DSV4 W8A16 granularity dry-run: 33,924 groupwise INT8 actions, 241 channelwise INT8 actions, 0 unmatched
  • full DSV4 artifact validation: 46 shards, 103,109 tensors, 33,924 INT4 tensors, 241 INT8 tensors; all 44 wo_a weights stored as INT8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants