Motivation
Qwen3.5 dense and MoE multimodal models can run frozen image GSPO, but their adapters do not expose the common configure_multimodal_training() contract. The generic tower/projector CLI options therefore cannot unfreeze the Qwen vision tower and merger or assign them independent LR schedules. Model-side trainability controls and checkpoint verification remain the missing pieces.
Proposed feature
Implement the shared multimodal training contract for Qwen3.5 dense and MoE adapters:
- classify the vision transformer as the tower and the vision merger/projector as the projector group;
- keep both media groups frozen by default;
- support
--unfreeze-mm-tower and --unfreeze-mm-projector/--unfreeze-mm-merger;
- apply independent base LR, minimum LR, decay steps, and decay style to each group;
- set
requires_grad and train/eval mode consistently;
- include unfrozen media parameters in rollout policy synchronization;
- preserve text, tower, and projector weights in standalone checkpoints.
Acceptance criteria
- Qwen3.5 dense and MoE adapters expose the common multimodal training hook.
- Frozen image GSPO behavior and metrics remain unchanged.
- Unfrozen image GSPO produces nonzero tower/projector gradients with the configured LRs.
- Frozen checkpoints preserve original media tensors; unfrozen checkpoints contain changed media tensors.
- Save/reload/serve round trips preserve the trained tower and projector values.
- Unit tests cover parameter classification, freeze state, optimizer groups, and unsupported/missing media modules.
Optimizer and scheduler state restoration is outside this issue; the required checkpoint round trip covers model weights.
Related work
Motivation
Qwen3.5 dense and MoE multimodal models can run frozen image GSPO, but their adapters do not expose the common
configure_multimodal_training()contract. The generic tower/projector CLI options therefore cannot unfreeze the Qwen vision tower and merger or assign them independent LR schedules. Model-side trainability controls and checkpoint verification remain the missing pieces.Proposed feature
Implement the shared multimodal training contract for Qwen3.5 dense and MoE adapters:
--unfreeze-mm-towerand--unfreeze-mm-projector/--unfreeze-mm-merger;requires_gradand train/eval mode consistently;Acceptance criteria
Optimizer and scheduler state restoration is outside this issue; the required checkpoint round trip covers model weights.
Related work