Support Qwen3.6 hybrid-attention blockwise calibration

[dogwood-flo]

Summary

• Extended onecomp/utils/blockwise.py calibration handling to support Qwen3.6's hybrid decoder, which mixes GatedDeltaNet linear_attention layers with regular full_attention layers (the same layer_types scheme as transformers' Qwen3_5 architecture), alongside the existing Gemma-style full_attention/sliding_attention mixed-attention handling.

• Added dedicated test coverage in tests/onecomp/utils/test_blockwise.py for the new hybrid-attention logic, which previously had no tests.

Details

Qwen3.6 support (GatedDeltaNet hybrid linear-attention / full-attention layers)

• Extended blockwise calibration handling in onecomp/utils/blockwise.py to support Qwen3.6's hybrid decoder, which mixes GatedDeltaNet linear_attention layers with regular full_attention layers (same layer_types scheme as transformers' Qwen3_5 architecture)
  • Hybrid layer-type detection now also recognizes the linear_attention + full_attention mix (is_qwen35_like_hybrid), alongside the existing Gemma-style full_attention + sliding_attention mix (is_gemma_like_mixed_attention); has_mixed_types is true when either applies
  • Added _create_linear_attention_mask(), ported from transformers' Qwen3_5Model._update_linear_attn_mask: returns None for cached-decode or no-padding forwards, otherwise the boolean padding mask
  • _compute_per_type_attention_masks() now dispatches to a Qwen-hybrid mask-creator mapping (linear_attention -> _create_linear_attention_mask, full_attention -> create_causal_mask) instead of the Gemma pair (create_causal_mask / create_sliding_window_causal_mask) when the model's layer types match the hybrid set

Tests

• Added test coverage for the Qwen3.6 hybrid-attention support above, which previously had no dedicated tests: _get_block_layer_type's linear_attn fallback and priority order, _create_linear_attention_mask's cached/no-padding/padding branches, the mask-creator dispatch added to _compute_per_type_attention_masks for {"linear_attention", "full_attention"} layer-type sets (with a regression guard for the existing Gemma-style {"full_attention", "sliding_attention"} path), and end-to-end hybrid-layer-type detection in get_blocks_and_inputs (tests/onecomp/utils/test_blockwise.py)

Validation

Verified the implementation with the following script:

from onecomp import GPTQ, CalibrationConfig, ModelConfig, Runner, setup_logger
import torch

# Set up logger (output logs to stdout)
setup_logger()

# Prepare the model
model_config = ModelConfig(
    model_id="Qwen/Qwen3.6-27B",
    device="cuda",
    dtype="bfloat16",
)

# Configure the quantization method
gptq = GPTQ(wbits=4)

# Configure the runner
runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    qep=True,
    calibration_config=CalibrationConfig(max_length=512, num_calibration_samples=128),
)
# NOTE: The calibration settings above are kept compact so the demo runs
# fast and may be insufficient for real quantisation.  For higher quality,
# prefer the CalibrationConfig() defaults
# (max_length=2048, num_calibration_samples=512).

# Run quantization
runner.run()

# Calculate perplexity
# Set True for models to evaluate, False returns None
original_ppl, dequantized_ppl, quantized_ppl = runner.calculate_perplexity(
    original_model=True, dequantized_model=False, quantized_model=True
)

# Display perplexity
print(f"Original model perplexity: {original_ppl}")
print(f"Dequantized model perplexity: {dequantized_ppl}")
print(f"Quantized model perplexity: {quantized_ppl}")

model, tokenizer = runner.create_quantized_model()

device = torch.device(model_config.device)
model = model.to(device)
model.eval()

print(f"Direct model type  : {type(model).__name__}")
print(f"Direct model device: {next(model.parameters()).device}")

prompt = "Fujitsu is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=32,
        do_sample=False,
    )

new_token_ids = output_ids[0, inputs["input_ids"].shape[1]:]
generated = tokenizer.decode(new_token_ids, skip_special_tokens=True)

print(f"\nPrompt   : {prompt}")
print(f"Generated: {generated}")