Skip to content

feat: add linear attention masking support for Qwen3.5 models#32

Open
dogwood-flo wants to merge 11 commits into
FujitsuResearch:develop/v1-3-0from
dogwood-flo:feature/qwen36_27b
Open

feat: add linear attention masking support for Qwen3.5 models#32
dogwood-flo wants to merge 11 commits into
FujitsuResearch:develop/v1-3-0from
dogwood-flo:feature/qwen36_27b

Conversation

@dogwood-flo

Copy link
Copy Markdown

Summary

  • Add a function to Blockwise.py that applies masking compatible with linear attention.
  • Detect Qwen3.5 attention modules and apply the appropriate masking logic.

Validation

Verified the implementation with the following script:

import torch

from onecomp import (
    GPTQ,
    CalibrationConfig,
    ModelConfig,
    Runner,
    load_quantized_model,
    setup_logger,
)

setup_logger()

MODEL_ID = "Qwen/Qwen3.6-27B"
SAVE_DIR = "./Qwen3.6-27B_gptq4"

model_config = ModelConfig(model_id=MODEL_ID, device="cuda:0")

gptq = GPTQ(
    wbits=4,
    groupsize=128,
)

runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    qep=True,
    calibration_config=CalibrationConfig(max_length=512, num_calibration_samples=128),
)

runner.run()

runner.save_quantized_model(SAVE_DIR)
print(f"\nQuantized model saved to: {SAVE_DIR}")

model, tokenizer = load_quantized_model(SAVE_DIR, torch_dtype=torch.bfloat16)
print(f"Loaded model type : {type(model).__name__}")
print(f"Loaded model device: {next(model.parameters()).device}")

prompt = "Fujitsu is"
inputs = tokenizer(prompt, return_tensors="pt").to(next(model.parameters()).device)

with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)

generated = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(f"\nPrompt   : {prompt}")
print(f"Generated: {generated}")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants