Export compact GatedDeltaNet state updates for Qwen3.5/3.8 - #2453
Merged
Conversation
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/qwen_gdn_production_export
branch
from
August 27, 2026 07:42
0ffd128 to
a9a4300
Compare
Tianlei Wu (tianleiwu)
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Python model builder to export Qwen3.5/3.8 hybrid linear-attention layers using packed varlen contrib ops and to optionally emit compact state-transition updates (conv values + GatedDeltaNet capsules) for speculative tokens, while keeping the existing state_groups schema unchanged.
Changes:
- Add builder support for
VarlenCausalConvWithStateand packedGatedDeltaNetwith optional compact state-update outputs. - Introduce Qwen3.5/3.8 extra options
linear_attn_opandstate_update_capacity, including validation and I/O shape/type rewiring for paged exports. - Extend docs and tests to cover compact state-update behavior and the removal of legacy checkpoint outputs.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/python/builder/test_decoder_state_groups.py | Adds targeted unit tests validating packed varlen op wiring, compact-update enable/disable behavior, and option validation. |
| src/python/py/models/README.md | Documents compact state updates for Qwen3.5/3.8 and how to enable them via --extra_options. |
| src/python/py/models/builders/qwen.py | Implements Qwen builder option parsing/validation, configures compact-update I/O, and routes linear-attention layers to packed/dense GatedDeltaNet paths. |
| src/python/py/models/builders/base.py | Adds builder emission helpers for VarlenCausalConvWithState, dense GatedDeltaNet, and packed GatedDeltaNet with compact capsule outputs. |
| src/python/py/models/builder.py | Updates CLI --extra_options help text to describe linear_attn_op, state_update_capacity, and updated state_window semantics. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
4 tasks
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/qwen_gdn_production_export
branch
from
August 27, 2026 17:25
b6decc8 to
f3f59d9
Compare
Contributor
Author
|
/azp run Integration Tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
Author
|
/azp run Integration Tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/qwen_gdn_production_export
branch
from
August 28, 2026 01:40
f3f59d9 to
ebbfd4c
Compare
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/qwen_gdn_production_export
branch
from
August 28, 2026 06:22
64bd219 to
f556520
Compare
Tianlei Wu (tianleiwu)
requested review from
Baiju Meswani (baijumeswani) and
kunal-vaishnavi
August 28, 2026 16:13
Baiju Meswani (baijumeswani)
previously approved these changes
Aug 28, 2026
Separate Qwen layer construction from the generic GatedDeltaNet emitter, document compact state options, and cover the disabled ABI.
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/qwen_gdn_production_export
branch
2 times, most recently
from
August 28, 2026 18:44
e8a70d1 to
14d9c2c
Compare
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/qwen_gdn_production_export
branch
from
August 28, 2026 18:45
14d9c2c to
adc6898
Compare
Baiju Meswani (baijumeswani)
approved these changes
Aug 28, 2026
Tianlei Wu (tianleiwu)
enabled auto-merge (squash)
August 28, 2026 19:44
Tianlei Wu (tianleiwu)
added a commit
that referenced
this pull request
Aug 30, 2026
## Summary Replace dense recurrent checkpoint tensors with compact state-update replay for dynamically batched hybrid models. The engine now owns fixed convolution and GatedDeltaNet state beside paged KV cache, captures packed per-token updates, and replays only an accepted prefix into inactive state banks before atomically publishing both cache types. ## Stack - Stacked on #2453, which exports compact Qwen3.5/3.8 `value` and `capsule` outputs. - Depends on microsoft/onnxruntime#32282 and microsoft/onnxruntime#32290. - MTP scheduling and drafter orchestration remain outside this PR. ## Key Changes - Parse explicit `paged_kv` and `fixed` decoder state groups, including compact convolution `value` and GDN `capsule` update bindings. - Add a fixed-state pool with per-request ownership, double-buffered publication, compact prefix replay, and CPU/CUDA provider implementations. - Add composite paged/fixed reservations with validation and preparation before the allocation-free publication boundary. - Bind packed variable-length inputs and fixed state together through `HybridDecoderIO`, including optional packed `position_ids`. - Resolve paged cache tensor names from explicit manifest bindings while retaining legacy name-template synthesis for models without `state_groups`. - Add synthetic hybrid/composite fixtures and focused manifest, reservation, replay, transaction, and end-to-end Engine coverage. ## Memory Impact The runtime no longer requires dense per-token recurrent checkpoint outputs. Speculative rollback data is bounded by `state_update_capacity` and represented as compact convolution values or packed GDN transition capsules. ## Testing - `test/python/builder/test_decoder_state_groups.py`: 24 passed - CPU Debug `engine_unit_tests`: 256 passed - CUDA Debug `engine_unit_tests` on H200: 261 passed - Paged and hybrid Engine Python tests across CPU/CUDA: 32 passed - CUDA compact convolution/GDN partial-prefix replay: passed ## Checklist - [x] Tests added and updated - [x] CPU and CUDA replay paths validated - [x] No MTP/drafter scheduling included - [x] Existing paged-only Engine behavior preserved --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VarlenCausalConvWithStateandGatedDeltaNetA_log/dt_biasfor native Qwen gate arithmetic, withoutarithmetic_modeLinearAttentiondefaultThe existing
state_groupsschema is unchanged in this PR. Parsing compact update metadata and replaying these transitions is handled by the stacked runtime PR #2454.Dependencies
Testing
python3 -m pytest test/python/builder/test_decoder_state_groups.py test/python/builder/test_qwen_vl.py -q(50 passed)python3 -m pytest test/python/builder -q(493 passed, 3 skipped)