Add Qwen3.5-MoE architecture support and fix Qwen3-VL-MoE linear attention weights - #696
Add Qwen3.5-MoE architecture support and fix Qwen3-VL-MoE linear attention weights#696alchernysh wants to merge 2 commits into
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fcdaa9a. Configure here.
| ] | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Missing deepstack merger weights
High Severity
The new visual module lists patch embed, vision blocks, and the main merger, but not model.visual.deepstack_merger_list.*. Qwen3.5 MoE multimodal vision uses DeepStack like Qwen3-VL; mergekit only writes planned weights, so those tensors are omitted from merge output.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit fcdaa9a. Configure here.
|
I have read the CLA Document and I hereby sign the CLA |


Summary
linear_attninstead ofself_attnon most layers, and these weights were missing from the definition, so mergekit silently dropped them during merges.Qwen3_5MoeForConditionalGeneration), including avisualmodule for the vision tower + merger. Without it, this multimodal model'smodel.visual.*tensors were silently dropped from merge output entirely.