Add Gemma 4 text model builder support (dense gemma-4-12b-it + 26B-A4B MoE)

[Thiago Pereira Rocha (thpereir)]

Summary

Adds model-builder support for Google's Gemma 4 text models, producing GenAI-compatible ONNX:

• Dense gemma-4-12b-it (Gemma4UnifiedForConditionalGeneration) via a new Gemma4Model.
• MoE gemma-4-26B-A4B-it (Gemma4ForConditionalGeneration) via a new Gemma4MoEModel, including the parallel dense-MLP-alongside-experts topology and the float (MoE) / quantized (QMoE) op-type split.

Both inherit from the existing Gemma3Model and override only the hooks that differ:

• Per-layer-type nested RoPE parameters ({"full_attention": {...}, "sliding_attention": {...}}) instead of the flat rope_parameters["rope_type"] form — handled via a make_rope_init override.
• Gemma4 keeps a parallel dense MLP alongside the experts, so intermediate_size is pinned to the dense size while the expert size is tracked separately as moe_intermediate_size.

Scope / relationship to prior work

This is the text builder layer. It complements the already-merged Gemma 4 C++ runtime / multimodal processor. It supersedes the non-functional scaffolding stub in #2088 (which prints "not yet end-to-end functional" and raises NotImplementedError) — here the dense and MoE text paths build end-to-end and are unit-tested. See #2062 for the architecture background.

Tests

• test/python/models/test_gemma4_builder.py — dense Gemma4 builder.
• test/python/models/test_gemma4moe_builder.py — MoE builder incl. moe_op_type (MoE vs QMoE) and dense/expert intermediate-size wiring.

All 16 tests pass locally.

Follow-ups (separate stacked PRs)

• 2-bit (uint2) QMoE support.
• 4-bit (Quark/AWQ int4) expert generalization.

Draft: opening for early review while the stacked quantization PRs are prepared.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds Python model-builder support for Google Gemma 4 text models (dense + MoE), integrating new Gemma4 builder classes into the dispatcher and extending unit tests to validate Gemma4-specific RoPE/KV-cache geometry and MoE wiring.

Changes:

• Introduces Gemma4Model and Gemma4MoEModel builders with Gemma4-specific RoPE cache handling, per-layer KV-cache geometry, and (for MoE) parallel dense+expert FFN construction.
• Wires Gemma4 architectures into create_model() dispatch and exports the new builders from the package.
• Adds focused unit tests for dense Gemma4 and Gemma4 MoE builder structure/parity, plus updates QMoE weight tests to include Gemma4 builders.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/python/models/test_gemma4moe_builder.py	New unit tests covering Gemma4 MoE builder (attrs/op selection, parallel FFN combine, router path, per-expert scale folding).
test/python/models/test_gemma4_builder.py	New unit tests covering dense Gemma4 builder (per-layer KV shapes, RoPE cache parity vs HF reference).
test/python/builder/test_qmoe_weights.py	Extends builder CLI test allowlist to include Gemma4 builders.
src/python/py/models/builders/gemma.py	Implements Gemma4Model and Gemma4MoEModel builder logic (RoPE, KV cache geometry, layer scalar, MoE FFN).
src/python/py/models/builders/base.py	Ensures MoE op-type is consistently available under both moe_attrs["op_type"] and moe_attrs["moe_op_type"].
src/python/py/models/builders/init.py	Exports the new Gemma4 builder classes.
src/python/py/models/builder.py	Adds dispatcher branches for Gemma4 architectures and emits warnings about precision and unsupported towers.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.