[CUDA] Deterministic token-doubling / corrupted output on H200 (Hopper, sm_90); CPU path is clean
Cross-reference: microsoft/foundry-local#873. Filing here because the corruption appears to originate in the ORT-GenAI CUDA generation/attention path rather than in the Foundry Local packaging (details below).
Symptom
On 8× NVIDIA H200 (Hopper, sm_90), CUDA inference produces deterministic token-doubling / gibberish at temperature=0, while the CPU path on the same models is perfectly clean.
Examples (Qwen2.5-1.5B-Instruct, CUDA, temperature=0, deterministic across repeated runs):
| Prompt |
CUDA output |
CPU output |
Count from 1 to 5 |
Sure (lyyes)) (identical across 3 runs) |
Sure! Here is the count from 1 to 5:\n\n1\n2\n3\n4\n5 |
Count: 1 2 3 4 5 |
The:\n count is is 11 112 11131 111144 11 |
(correct) |
Note the token-doubling signature: is is, 11 112 11131 111144. Phi-4-mini-instruct on CUDA produces pure gibberish. Affects multiple dense (non-MoE) models (Qwen2.5-1.5B, Qwen2.5-7B, Phi-4-mini).
This is not sampling (deterministic at temp=0) and not model-specific — it is a CUDA compute correctness problem on Hopper.
Environment
- GPU: 8× NVIDIA H200 (Hopper, compute capability 9.0), driver 580.105.08, CUDA 12.x runtime
- Observed via Microsoft Foundry Local 0.10.0 and 0.10.1, which bundle:
- ONNX Runtime 1.26.0 (
libonnxruntime.so)
- an ORT-GenAI "Foundry" build (
libonnxruntime-genai-cuda.so)
- Models: ORT-GenAI-format Qwen2.5 / Phi-4-mini (
past_present_share_buffer=true, GroupQueryAttention)
Why this looks like an ORT-GenAI-layer issue, not ORT-core
The closest published precedent is onnxruntime #27079 ("[CUDA] Qwen3 Model quality regression" — same CUDA-gibberish-vs-clean-CPU signature), fixed by onnxruntime PR #27108 (GQA QKV "buffer pointer misconfiguration", merged 2026-01-23). But that fix is already present in the build I'm running:
libonnxruntime.so reports version 1.26.0 (released ~3.5 months after #27108).libonnxruntime_providers_cuda.so contains the post-fix fused kernel symbols (UnpackRoPE… / PrepareQKV).
Additionally, upgrading Foundry Local 0.10.0 → 0.10.1 swapped in a different libonnxruntime-genai-cuda.so (the ORT-core libonnxruntime_providers_cuda.so was byte-identical), yet the corruption was unchanged. That points at the generation loop / KV-cache position / attention-mask handling in the ORT-GenAI CUDA path on Hopper, rather than the ORT-core attention kernel math.
Questions
- Is deterministic token-doubling on Hopper (
sm_90) a known issue in the ORT-GenAI CUDA generation path (e.g. past_present_share_buffer handling, position/mask tracking, or a GQA/XQA sm_90 kernel)?
- If so, which ORT-GenAI version / commit fixes it?
- What is the recommended way to reproduce on a public onnxruntime-genai build (model-builder recipe +
og.generator snippet) so this can be confirmed outside the Foundry build? I can run repros on the H200 host.
Happy to provide additional diagnostics.
[CUDA] Deterministic token-doubling / corrupted output on H200 (Hopper, sm_90); CPU path is clean
Cross-reference: microsoft/foundry-local#873. Filing here because the corruption appears to originate in the ORT-GenAI CUDA generation/attention path rather than in the Foundry Local packaging (details below).
Symptom
On 8× NVIDIA H200 (Hopper,
sm_90), CUDA inference produces deterministic token-doubling / gibberish attemperature=0, while the CPU path on the same models is perfectly clean.Examples (Qwen2.5-1.5B-Instruct, CUDA,
temperature=0, deterministic across repeated runs):Count from 1 to 5Sure (lyyes))(identical across 3 runs)Sure! Here is the count from 1 to 5:\n\n1\n2\n3\n4\n5Count: 1 2 3 4 5The:\n count is is 11 112 11131 111144 11Note the token-doubling signature:
is is,11 112 11131 111144.Phi-4-mini-instructon CUDA produces pure gibberish. Affects multiple dense (non-MoE) models (Qwen2.5-1.5B, Qwen2.5-7B, Phi-4-mini).This is not sampling (deterministic at temp=0) and not model-specific — it is a CUDA compute correctness problem on Hopper.
Environment
libonnxruntime.so)libonnxruntime-genai-cuda.so)past_present_share_buffer=true, GroupQueryAttention)Why this looks like an ORT-GenAI-layer issue, not ORT-core
The closest published precedent is onnxruntime #27079 ("[CUDA] Qwen3 Model quality regression" — same CUDA-gibberish-vs-clean-CPU signature), fixed by onnxruntime PR #27108 (GQA QKV "buffer pointer misconfiguration", merged 2026-01-23). But that fix is already present in the build I'm running:
libonnxruntime.soreports version 1.26.0 (released ~3.5 months after #27108).libonnxruntime_providers_cuda.socontains the post-fix fused kernel symbols (UnpackRoPE…/PrepareQKV).Additionally, upgrading Foundry Local 0.10.0 → 0.10.1 swapped in a different
libonnxruntime-genai-cuda.so(the ORT-corelibonnxruntime_providers_cuda.sowas byte-identical), yet the corruption was unchanged. That points at the generation loop / KV-cache position / attention-mask handling in the ORT-GenAI CUDA path on Hopper, rather than the ORT-core attention kernel math.Questions
sm_90) a known issue in the ORT-GenAI CUDA generation path (e.g.past_present_share_bufferhandling, position/mask tracking, or a GQA/XQA sm_90 kernel)?og.generatorsnippet) so this can be confirmed outside the Foundry build? I can run repros on the H200 host.Happy to provide additional diagnostics.