[behroozazarkhalili]

Replaces the pre-Ampere skip on test_train_with_transformers_continuous_batching with a strict xfail on the exact failing condition: pre-Ampere GPU and kernels < 0.16.1.

Continuous batching auto-switched to Flash Attention 3 on pre-Ampere GPUs and crashed (huggingface/transformers#47926). The architecture check in kernels 0.16.1 (huggingface/kernels#774) fixes it, but transformers still allows 0.16.0, so the mark stays until transformers requires kernels>=0.16.1.

Refs #6711.

---

Note

Low Risk
Test-only change to pytest marks; no production training or runtime behavior is modified.

Overview
test_train_with_transformers_continuous_batching no longer skips all pre-Ampere (non-XPU) runs. It now uses a strict xfail only when the GPU is pre-Ampere, the kernels package is installed, and kernels < 0.16.1—the combo that still triggers Flash Attention 3 crashes during continuous batching.

Pre-Ampere environments with kernels>=0.16.1 (architecture check from huggingface/kernels#774) can execute the test as a normal pass/fail. Imports importlib.metadata and importlib.util were added to read the installed kernels version at collection time.

^{Reviewed by Cursor Bugbot for commit 0afee22. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Thanks @behroozazarkhalili. The skip isn't inert for everyone: kernels 0.16.1 fixes it (huggingface/kernels#774, confirmed in huggingface/transformers#47926 (comment)), but transformers still allows 0.16.0. Could you replace the skip with a strict xfail on pre-Ampere + kernels < 0.16.1, with a comment pointing to both and saying it can go once transformers requires kernels>=0.16.1?

[qgallouedec]

Suggested change

	@pytest.mark.skipif(
	not is_ampere_or_newer() and torch_device != "xpu",
	reason="transformers continuous batching switches attention to Flash Attention, which requires an Ampere or "
	"newer GPU, or XPU (see https://github.com/huggingface/transformers/issues/47926)",
	)
	# Continuous batching auto-switches to Flash Attention 3 on pre-Ampere GPUs and crashes
	# (huggingface/transformers#47926). Fixed by the arch check in kernels 0.16.1 (huggingface/kernels#774); remove
	# once transformers requires `kernels>=0.16.1`.
	@pytest.mark.xfail(
	not is_ampere_or_newer()
	and torch_device != "xpu"
	and Version(importlib.metadata.version("kernels")) < Version("0.16.1"),
	reason="Flash Attention 3 on pre-Ampere with kernels<0.16.1 (huggingface/transformers#47926)",
	strict=True,
	)

[qgallouedec]

Suggested change

	is_ampere_or_newer,
	is_ampere_or_newer,

[qgallouedec]

Thanks @behroozazarkhalili. After a closer look, keeping the skip: the only broken setup is pre-Ampere with transformers 5.15-5.17 and kernels 0.16.0 (transformers 5.18 requires kernels>=0.17, and 0.16.1 has the architecture check, huggingface/kernels#774), and our CI never runs it. Narrowing the guard to that exact combination isn't worth the extra conditions. Closing along with #6711.