Skip to content

(WIP) perf(kimi3): enable batched AttnRes runtime fusion - #1146

Draft
panditsa wants to merge 1 commit into
lightseekorg:sanket/wip-k3-sharded-finalizefrom
panditsa:sanket/wip-k3-attnres-runtime
Draft

(WIP) perf(kimi3): enable batched AttnRes runtime fusion#1146
panditsa wants to merge 1 commit into
lightseekorg:sanket/wip-k3-sharded-finalizefrom
panditsa:sanket/wip-k3-attnres-runtime

Conversation

@panditsa

@panditsa panditsa commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • allow the backend-selected fused AttnRes combine for supported multi-token decode batches
  • hoist next-layer MLP/attention partials whenever the projection kernel reports support
  • join the producer stream before gfx950 M<16 reductions that consume AttnRes scratch

Validation result

This WIP is not ready to land: activating the batched fusion regresses end-to-end decode even though the isolated kernel in #1142 is faster.

4K/1K TP8/EP1 CUDA graphs, median of three:

M Equivalent parent tok/s Head tok/s Change Parent TPOT Head TPOT
2 103.64 93.99 -9.3% 18.621 ms 20.308 ms
4 185.64 172.06 -7.3% 20.380 ms 21.459 ms

The parent figures are the exact #1144 head. #1145 only selects its sharded tail at M>=8, so M=2/4 are unchanged at the #1146 parent boundary.

The likely issue is orchestration rather than fused-kernel latency: joining the producer stream and consuming AttnRes scratch removes overlap that the prior split path retained. Runtime activation should remain gated until the dependency/stream schedule is redesigned and reprofiled.

Tests

  • batched runtime-dispatch test passes and confirms the fused Iris combine is selected when supported
  • (WIP) perf(comm): extend fused AttnRes through M16 #1142 covers world-size-8 M=1/2/4/8/16 numerical correctness and CUDA graphs
  • all 18 measured head requests completed with valid lengths

Stack

WIP blocker

  • restore the runtime gate or recover the lost producer/consumer overlap before landing

Signed-off-by: Sanket Pandit <sanket.pandit@amd.com>
@jhinpan

jhinpan commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The exact #1132 M=8/EP8 point differs from the published M=2/4 EP1 blocker. On current main (9196fa27), #1141 + #1142 + this runtime activation gives:

  • forced 131072/1024 TP8/EP8: 36.25 ms TPOT and 19.9703 tok/s (3 runs), versus 36.80 ms / 19.7647 baseline: -1.49% TPOT, +1.04% throughput
  • normal 4K/1K DSpark: 155.078 tok/s versus 152.949 baseline (+1.39%)
  • exact token identity versus ordinary greedy passes at 4K/1K and 131K/128 under forced rejection

So the activation is beneficial at M=8/EP8, but still regresses the already-reported M=2/4 EP1 points and only moves the #1132 round ratio from 2.157x to ~2.125x. This supports redesigning/gating the orchestration by shape/topology rather than landing the current activation globally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants