[AMD][CI] Add a model cache audit that counts per-commit workflows

[michaelzhang-ai]

Motivation

The AMD runners share one HuggingFace cache PVC per cluster, nothing evicts from it, and it fills up — a multi-hundred-GB download then dies partway through with ENOSPC. Deciding what is safe to drop needs a join: register_amd_ci tests → the checkpoints those tests name → whether any AMD workflow actually dispatches them.

Getting that join wrong is expensive. A recent audit of the tw MI35x PVC joined only the two nightly workflows and reported 939 GiB as reclaimable. Every byte of it was live: 289 of 429 AMD registrations are dispatched only by the per-commit pr-test-amd*.yml workflows, whose jobs run on linux-mi35x-gpu-1 and mount the same PVC as the 8-GPU nightly jobs. Deleting by that list would have broken AMD CI on every PR.

Modifications

scripts/ci/utils/amd_cache_audit.py (new) — performs the join, counting per-commit and nightly separately and never reporting one without the other. With --cache-dir or --cache-listing it diffs against a live cache and sorts the result into three buckets, of which only the first is unconditionally safe:

Bucket	Meaning
cached, no AMD test reference	nothing names it — safe to drop
cached, test exists but no job dispatches	becomes needed again the moment someone wires the suite up
needed but missing	a dispatched suite wants it; next run pays the download

Two details it exists to get right, both of which a hand-rolled scan got wrong on the first pass:

• --runner scopes to a cluster by the dispatching job's runs-on label. Without it, tw is compared against every AMD suite and the mi300-only checkpoints all read as missing — 134 rather than 24.
• Model ids resolve through list_stage_models.py's constant table, not string literals. meta-llama/Llama-3.2-1B-Instruct looks unreferenced to a literal search but reaches five per-commit suites through DEFAULT_SMALL_MODEL_NAME_FOR_TEST.

Dispatch detection accepts both --suite <name> and a bare test path, since several MI35x perf steps invoke the file directly rather than going through a suite.

The suite → models extraction is reused from scripts/ci/list_stage_models.py rather than reimplemented; it already handles AST extraction, constant resolution and overrides for any backend, and already supports --backend amd.

scripts/ci/test_amd_cache_audit.py (new) — unit tests for the label/text/cache-path helpers, plus an end-to-end case that runs the real join against the real repo. That last one is the point: no workflow executes this tool, so a refactor in list_stage_models.py would break it silently and nobody would notice until someone deleted the wrong checkpoints. It asserts the contract rather than exact counts, which move whenever a test is added — including specifically that the per-commit bucket is non-empty and that constant-resolved models still appear.

.github/workflows/ci-model-inventory.yml — runs those tests beside the extractor's own, and adds both files to the PR path filter.

Accuracy Tests

Not applicable — CI tooling, no product code.

python3 -m unittest discover -s scripts/ci -p 'test_amd_cache_audit.py' → 15 tests, ~15 s, stdlib only, no GPU and no sglang import.

Behaviour on the current tree: --runner mi35x resolves nightly_only 39 / per_commit_only 3 / both 2 / undispatched 75 over 157 checkpoints; a cache diff against a planted listing correctly sorts an unreferenced entry into "safe to drop" and reports the rest as missing.

Speed Tests and Profiling

Not applicable. The end-to-end test adds ~15 s to ci-model-inventory, which is an ubuntu-latest job.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations. — the module docstring documents usage.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers — AMD CI owners are the right audience here.
3. Trigger CI tests with comments or contact authorized users to do so.
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #32673641979
Latest PR Test (Extra): ❌ Run #32673641830
Latest PR Test (AMD ROCm 7.2): ❌ Run #32673642000