Skip to content

feat(attention): add Gluon forward attention backend - #469

Open
bangtianliu wants to merge 2 commits into
AMD-AGI:mainfrom
bangtianliu:bangtian/gluon-forward-attention
Open

feat(attention): add Gluon forward attention backend#469
bangtianliu wants to merge 2 commits into
AMD-AGI:mainfrom
bangtianliu:bangtian/gluon-forward-attention

Conversation

@bangtianliu

@bangtianliu bangtianliu commented Aug 21, 2026

Copy link
Copy Markdown

Description

Adds an opt-in Gluon backend for dense Flash Attention forward on gfx950.

Gluon is not part of the default selection or cross-backend autotuning. It is forward-only, and a pinned unsupported call fails before launch instead of falling back. The port does not load an LLVM pass plugin, pin LLVM/Triton, or select behavior from a compiler version.

Fixes # (N/A)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Vendor the gfx950 FP16/BF16 Gluon forward kernel and record its provenance.
  • Register BackendType.GLUON in the dense attention dispatcher.
  • Add an opaque PyTorch custom-op boundary with fake-tensor and torch.compile(fullgraph=True) support.
  • Add fail-fast eligibility and lifecycle checks for unsupported inputs and backward requests.
  • Add correctness regressions and a correctness-gated forward benchmark.

Supported envelope

Dense rank-4 FP16/BF16 attention on gfx950; BSHD or BHSD storage; MHA, GQA, and MQA; head dimension up to 256; causal and noncausal. Causal requires Sq <= Skv.

Performance

MI350X (gfx950), PyTorch 2.13.0+rocm7.1, Triton 3.8.0+fb, FP16, BSHD, D128, warm cache. The 30-shape set covers causal/noncausal MHA, GQA, and MQA from S128 through S16384. Each row must pass the FP32-reference check at greater than 40 dB output SNR before it is timed. Backends are pinned; no fallback result is included. Gluon uses a fresh autotune cache: 26 general shape keys use the 36-config autotuner; four short-causal rows use the fixed specialization. Tuning and correctness complete before 20 warmups and 100 measured launches. TFLOPS use the attended FLOP count, including the causal fraction.

Backend Correct Arithmetic mean TFLOPS
Gluon 30/30 480.52
AITER 30/30 459.69

The geometric mean of the 30 per-shape Gluon/AITER throughput ratios is 0.830x; the long, high-throughput rows dominate the arithmetic mean. The result is shape-dependent. Selected causal D128 rows:

B S Hq/Hkv Gluon TFLOPS AITER TFLOPS Ratio
32 512 64/64 165.26 294.21 0.562x
16 1024 64/64 446.05 411.21 1.085x
8 2048 64/64 485.22 507.75 0.956x
4 4096 64/64 643.15 573.14 1.122x
2 8192 64/64 700.78 610.26 1.148x
1 8192 32/8 725.04 635.29 1.141x
1 2048 32/1 185.17 465.37 0.398x
1 8192 32/1 724.71 658.79 1.100x

Checklist

  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my feature works
  • New and existing focused tests pass locally with my changes

Vendor and integrate the gfx950 Gluon forward kernel, add explicit dense forward-only dispatch, benchmarks, correctness coverage, and the upstream 05b349b synchronization fixes.
def backward(ctx, dout, *args):
if ctx.backend == BackendType.GLUON:
raise AssertionError(
"internal contract violation: gluon flash-attn is forward-only and must not reach backward"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is forward FA kernel only need for client ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds forward first. Gluon backward is the next focus and will follow separately.



@pytest.mark.skipif(
os.environ.get("PRIMUS_TURBO_TEST_LONG") != "1",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove debug env.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please merge this file into test_attention.py.

_GFX950_DEVICE_CACHE: dict[torch.device, bool] = {}


def _is_gfx950_device(device: torch.device) -> bool:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use this func.

def is_gfx950() -> bool:
return get_device_compute_capability() == (9, 5)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of this unit test file ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a self-added vendoring-policy check rather than existing CI coverage. I removed this test and the corresponding checker to keep the PR focused on the Gluon backend.

@bangtianliu
bangtianliu force-pushed the bangtian/gluon-forward-attention branch from c4689f6 to 6196e77 Compare August 21, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants