Skip to content

feat(qwen3.5): add multimodal tower and projector training controls #494

Description

@xsuler

Motivation

Qwen3.5 dense and MoE multimodal models can run frozen image GSPO, but their adapters do not expose the common configure_multimodal_training() contract. The generic tower/projector CLI options therefore cannot unfreeze the Qwen vision tower and merger or assign them independent LR schedules. Model-side trainability controls and checkpoint verification remain the missing pieces.

Proposed feature

Implement the shared multimodal training contract for Qwen3.5 dense and MoE adapters:

  • classify the vision transformer as the tower and the vision merger/projector as the projector group;
  • keep both media groups frozen by default;
  • support --unfreeze-mm-tower and --unfreeze-mm-projector/--unfreeze-mm-merger;
  • apply independent base LR, minimum LR, decay steps, and decay style to each group;
  • set requires_grad and train/eval mode consistently;
  • include unfrozen media parameters in rollout policy synchronization;
  • preserve text, tower, and projector weights in standalone checkpoints.

Acceptance criteria

  • Qwen3.5 dense and MoE adapters expose the common multimodal training hook.
  • Frozen image GSPO behavior and metrics remain unchanged.
  • Unfrozen image GSPO produces nonzero tower/projector gradients with the configured LRs.
  • Frozen checkpoints preserve original media tensors; unfrozen checkpoints contain changed media tensors.
  • Save/reload/serve round trips preserve the trained tower and projector values.
  • Unit tests cover parameter classification, freeze state, optimizer groups, and unsupported/missing media modules.

Optimizer and scheduler state restoration is outside this issue; the required checkpoint round trip covers model weights.

Related work

Metadata

Metadata

Assignees

Labels

area/checkpointIssues or PRs related to checkpoint save/load and model weight I/Oarea/modelsIssues or PRs related to model adapters (llama, qwen3, bailing, gemma4, etc.)kind/featureCategorizes issue or PR as related to a new feature

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions