Repository navigation
Implement cache pinning on Linux, and report that it measures flat there - #35
Conversation
PinProcessToLargestCache() has been reachable only through an environment variable. Everything else the runner does is a command-line flag, so a harness that wants pinning has to reach outside the argument list to get it, and an A/B harness has no way to force it off for the control arm when the environment already asks for it. --cache-affinity enable pinning --no-cache-affinity disable it, overriding MODERNGEKKO_CACHE_AFFINITY The flag wins over the environment in both directions, which is what an A/B needs. argv is scanned in PinProcessToLargestCache() rather than in RunMain(), because pinning has to happen before the run starts. Both spellings are accepted by the normal parser too, on every platform, so a shared script can pass one without knowing the host. The existing comment cited a single measurement (55.41 -> 70.42 fps). Replaced with four titles on a 9950X3D, C backend, three interleaved rounds per scene, same module and savestate throughout: Pokemon Colosseum +20.6% Mario Kart: Double Dash!! +32.9% Luigi's Mansion +11.7% Skyward Sword +35.6% Twelve of twelve scenes improved, +5% to +43%, mean +25.2%. Pinning also cuts run-to-run spread sharply -- one Colosseum scene went from +-0.297 to +-0.015 standard error over three samples. Default stays off, and a second measurement says why more precisely than "one machine, one topology" did. On a 5900X -- two CCDs, 32 MB of L3 each, no asymmetry -- the same A/B over four CPU-saturated scenes came out at -1.4%, two of them showing a small consistent loss. So this is not migration avoidance in general: pinning on a symmetric part halves the cores the worker threads can use and buys nothing back. The win depends on one die carrying more cache than the other. Verified on Windows: --cache-affinity alone gives a 0xFFFF process affinity mask, and --no-cache-affinity with MODERNGEKKO_CACHE_AFFINITY=1 set gives 0xFFFFFFFF.
The Windows path pins to the cores sharing the largest last-level cache.
Linux exposes the same topology through sysfs, so the rule ports directly:
LargestSharedCache() reads /sys/devices/system/cpu/cpu*/cache/index*,
de-duplicates the domain reported once per CPU in
it, and takes the largest by size, ties going to
the first in CPU order
ParseCpuList() "0-5,12-17" -> the CPUs in it
ParseCacheSize() "32768K" -> bytes
ApplyCacheDomain() sched_setaffinity(0, ...)
Two details worth stating. Malformed input yields an empty result rather
than a partial one, because pinning to a wrong domain is worse than not
pinning. And the affinity call takes 0, not a pid: a pid argument moves
only the calling thread on some kernels, and Dolphin's workers are spawned
after this runs.
AffinityEnabled() and PinResult move above the platform split so both
paths share the same opt-in rule. The sysfs root is a parameter so a test
can point it at a fixture tree, matching why the Windows rule takes a
buffer instead of calling the OS itself. The test gains a Linux half
covering equal-CCD and larger-victim-cache layouts, malformed lists, no
L3 at all, and a missing tree; CMake now builds it on Linux too.
It does not help on the hardware available. Most MKDD savestates sit at
1.0x and 59.9 fps, keeping up with the guest, and cannot show a cache
effect at all; screening picked the two that are genuinely CPU-limited.
On an idle 5900X -- two CCDs, 32 MB L3 each -- six interleaved rounds over
four CPU-saturated scenes (0.50x-0.61x):
bench (4-player race) 0.5025 -> 0.5103 +1.6%
play-GM4E01-019 0.6062 -> 0.5884 -2.9%
play-GM4E01-015 0.6127 -> 0.6168 +0.7%
play-GM4E01-003 0.5978 -> 0.5683 -4.9%
overall -1.4%
Mean and median agree in sign on every scene, so this is not an
underpowered null. Two scenes lose consistently, and plausibly: pinning to
one CCD halves the cores available to the worker threads, and on a part
where the other die's cache is exactly as good, nothing is bought back.
So the mechanism is cache asymmetry, not migration avoidance. The +25.2%
on Windows came from a 9950X3D where one die carries 96 MB and the other
32 MB. No Linux host here has that, so the case this exists for is
implemented and tested but unmeasured.
Default stays off, as on Windows.
Verified: unit tests pass under GCC 13.3 and clang 18.1 with -Wall
-Wextra; the pin applies in a real run ("pinned to the 12 CPUs sharing the
largest L3 (32 MB)"); MSVC still builds and the Windows flag still works.
50be274 to
ecd3188
Compare
|
Independent data point that I think sharpens the null here rather than contradicting it: on this same 5900X, cache capacity is decisive even though cache domain choice measures flat. Those are separable, and it is worth saying so explicitly, because "pinning is flat on Linux" reads easily as "this part is not cache-bound" — which it very much is. Same host as your table (5900X, two symmetric 32 MB CCDs, idle, headless), MKDD
Absolute on Linux: C 32.97 → LLVM 39.57 → SIM 52.03 fps. The relevant property of SIM here is that it shrinks the recompiled module. Same game, same savestate, same DolRecomp commit:
Two things fall out. Plain LLVM's win is codegen — branch misses drop 56% — but it costs memory-system pressure: iTLB misses go up 22% against C, which is what a 404 MB code module does to a TLB. SIM keeps the branch improvement (−57%, flat against LLVM) and additionally removes the footprint cost: against plain LLVM it cuts LLC misses 40%, iTLB 50%, L1i 24%, with branch misses unchanged. That residual is a pure working-set effect and it tracks 404 MB → 166 MB. That is consistent with your result, and I think it explains it. Pinning can only choose which 32 MB domain you land on; when both domains are identical there is nothing to choose, so the null is the expected answer. It says nothing about whether 64 MB is enough to hold the workload — and these numbers say it is not, by a wide margin. The 9950X3D's 96 MB absorbs plain LLVM's footprint well enough that the same change is worth 5% instead of 56%, which is the same asymmetry story from the other direction. Practical consequence for the "merge for parity, or hold for a V-Cache Linux part" question: a Linux V-Cache part would likely show a smaller SIM win than this 5900X does, not a larger one, because more L3 means less footprint pressure to recover. So if the goal is to demonstrate what pinning buys, the scene selection problem you hit is real but secondary — the bigger lever is that the arm has to be cache-starved in the first place. It also argues that benchmarking footprint-reducing work exclusively on the X3D box will systematically understate it. Caveats, so these are read at the right precision:
|
2 participants
Stacked on #34. GitHub will not let me base a PR on a branch that lives only on my fork, so this opens against
masterand therefore contains #34's commit as well. Review #34 first; the second commit is what this PR is about. If #34 lands, this rebases to a single commit.What
The Windows path pins the process to the cores sharing the largest last-level cache. Linux exposes the same topology through sysfs, so the rule ports directly:
LargestSharedCache()/sys/devices/system/cpu/cpu*/cache/index*, de-duplicates the domain reported once per CPU in it, takes the largest by size, ties to the first in CPU orderParseCpuList()"0-5,12-17"→ the CPUs in itParseCacheSize()"32768K"→ bytesApplyCacheDomain()sched_setaffinity(0, ...)Two details worth calling out:
0, not a pid. A pid argument moves only the calling thread on some kernels, and Dolphin's workers are spawned after this runs.AffinityEnabled()andPinResultmove above the platform split so both paths share one opt-in rule. The sysfs root is a parameter so a test can point it at a fixture tree — the same reason the Windows rule takes a buffer rather than calling the OS itself.It measures flat, and that turns out to be the interesting part
Choosing the workload mattered more than anything else here. Screening 11 MKDD savestates showed most sit at exactly ~1.0x / 59.9 fps — they are keeping up with the guest, have headroom, and cannot show a cache effect at all. Only two were genuinely CPU-limited. Two earlier attempts at this measurement used scenes that were too light (one set ran at 3.09x realtime) and are not reported here.
Final run: idle 5900X — two CCDs, 32 MB L3 each — four CPU-saturated scenes (0.50x–0.61x), six interleaved rounds each, headless.
Mean and median agree in sign on every scene, so this is not an underpowered null. Two scenes lose consistently, and plausibly: pinning to one CCD halves the cores available to Dolphin's worker threads, and on a part where the other die's cache is exactly as good, nothing is bought back.
So the mechanism is cache asymmetry, not migration avoidance — which corrects how I described it in #34, and is a better reason for the default to stay off. The +25.2% there came from a 9950X3D where one die carries 96 MB and the other 32 MB. No Linux host I have access to has that asymmetry, so the case this code exists for is implemented and tested but unmeasured.
Merge it for parity of mechanism, or hold it until someone can run it on a Linux V-Cache part — both are reasonable, and I would rather the choice be made with the null visible than buried.
Default stays off, as on Windows.
Testing
tests/cache_affinity_test.cppgains a Linux half: equal-CCD and larger-victim-cache fixture layouts, malformed cpu lists, no level-3 entries at all, and a missing tree. CMake builds it on Linux as well as Windows; elsewhere it is a no-opmain.-Wall -Wextra[perf] pinned to the 12 CPUs sharing the largest L3 (32 MB)