bug: run_eval leaves large MIGraphX .mxr caches in the repository root

[KayMKM]

Summary

Running the E2E evaluation harness with --ep migraphx --device gpu leaves MIGraphX compiled-model cache files (.mxr) in the process working directory. When the harness is launched from the repository root, the files accumulate there as untracked files and are not removed by --clean-cache.

A full evaluation run created 163 files totaling 178,367,026,225 bytes (166.12 GiB). The largest single file was 6.96 GiB.

Reproduction

uv run python scripts/e2e_eval/run_eval.py `
  --eval-type both `
  --ep migraphx `
  --device gpu `
  --clean-cache `
  --continue `
  --timeout 3600 `
  --output-dir eval_results/2026-08-03/migraphx `
  --verbose

After the run, the repository root contains files such as:

21000-52d4355587658066-560872bf9988f016-519c70b1cf0f20ca.mxr
21000-1037e9c44326a6ec-560872bf9988f016-bc289651c53017ef.mxr

Actual behavior

• 163 .mxr files were created between 2026-08-07 06:47 and 2026-08-08 15:48.
• Total size: 178,367,026,225 bytes (166.12 GiB).
• Largest file: 6.96 GiB.
• The files remain after the run and appear as untracked files in git status.
• --clean-cache does not remove them.
• The file header identifies them as MessagePack-serialized MIGraphX programs with migraphx_version set to 2.16.0 and target gpu.

The filename also matches ONNX Runtime's MIGraphX model-cache format:

<MIGraphX version>-<graph hash>-<GPU architecture hash>-<input shape hash>.mxr

21000 is the hexadecimal encoding of MIGraphX 2.16.0.

Expected behavior

E2E evaluation should not leave large execution-provider caches in the repository root. MIGraphX model caches should either:

• be written to a managed WinML cache or per-run temporary/output directory, and be covered by --clean-cache; or
• be disabled for the E2E run if reuse is not intended.

Investigation

• _run_subprocess() calls subprocess.Popen() without setting cwd, so child processes inherit the harness working directory.
• The current stray-CWD cleanup covers UUID-prefixed ONNX/context files, sym_shape_infer_temp*, and timing_log.csv, but not MIGraphX .mxr cache names.
• ONNX Runtime's MIGraphX EP saves the compiled migraphx::program under its model cache path. On this machine that path resolved to the current working directory even though ORT_MIGRAPHX_MODEL_CACHE_PATH was unset and the WinML CLI did not explicitly provide migraphx_model_cache_dir.

Potential fixes include routing migraphx_model_cache_dir to a managed cache directory, or conservatively recognizing the MIGraphX cache filename pattern in the E2E cleanup. Adding .mxr to .gitignore could be a defense-in-depth measure, but would not address disk growth.

Environment

• Repository commit: e27a0fafdbd07e3be610e064c907fa32894f5435
• WinML CLI: 0.2.0
• ONNX Runtime package: onnxruntime-windowsml 1.24.5.202604171637
• MIGraphX EP package: MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8 version 1.8.56.0
• MIGraphX serialized program version: 2.16.0
• GPU: AMD Radeon 890M Graphics
• OS: Windows 11 Enterprise Insider Preview, version 10.0.26310