Skip to content

Add Qwen3.5-MoE architecture support and fix Qwen3-VL-MoE linear attention weights - #696

Open
alchernysh wants to merge 2 commits into
arcee-ai:mainfrom
alchernysh:add-qwen3-5-moe-architecture
Open

Add Qwen3.5-MoE architecture support and fix Qwen3-VL-MoE linear attention weights#696
alchernysh wants to merge 2 commits into
arcee-ai:mainfrom
alchernysh:add-qwen3-5-moe-architecture

Conversation

@alchernysh

@alchernysh alchernysh commented Jul 3, 2026

Copy link
Copy Markdown

Summary

  • Add linear attention (gated delta net) weights to the Qwen3-VL-MoE architecture — hybrid layers use linear_attn instead of self_attn on most layers, and these weights were missing from the definition, so mergekit silently dropped them during merges.
  • Add architecture support for Qwen3.5-MoE (Qwen3_5MoeForConditionalGeneration), including a visual module for the vision tower + merger. Without it, this multimodal model's model.visual.* tensors were silently dropped from merge output entirely.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fcdaa9a. Configure here.

]
}
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing deepstack merger weights

High Severity

The new visual module lists patch embed, vision blocks, and the main merger, but not model.visual.deepstack_merger_list.*. Qwen3.5 MoE multimodal vision uses DeepStack like Qwen3-VL; mergekit only writes planned weights, so those tensors are omitted from merge output.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit fcdaa9a. Configure here.

@alchernysh

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant