Caching is inconsistent in FFI CI

[spacebear21]

We pin the rust version so the cache should always hit as long as the rust dependencies haven't changed.

Here is an example where cache was hit: https://github.com/payjoin/rust-payjoin/actions/runs/18139263858/job/51625878478

And two where it wasn't (despite the cache keys matching exactly, I would expect at least one of these to hit):

• https://github.com/payjoin/rust-payjoin/actions/runs/18170784569/job/51724615262?pr=1060
• https://github.com/payjoin/rust-payjoin/actions/runs/18171737197/job/51727834511?pr=1130

[emmanuelist]

Had a dig at this one. The keys are fine. The cache is just getting evicted
before anything can hit it.

Where the repo cache sits right now:

• 10.46 GiB used against GitHub's 10 GB per-repo limit, so we're over and
  GitHub is evicting least-recently-used entries continuously.
• 1031 entries total, and every one of them belongs to refs/pull/1809/merge.
  A single PR's run is holding the whole quota.
• Nothing on refs/heads/master at all. Querying ?ref=refs/heads/master
  gives back total_count: 0.

What's actually taking up the space:

Entry type	Count	Share
nix store paths (.nar.z)	510	49.5%
nix .narinfo-*	506	49.1%
v0-rust-* (the build caches)	15	1.5%

magic-nix-cache-action writes one entry per nix store path, so roughly 1016
nix entries are crowding out the 15 Swatinem/rust-cache entries that the jobs
actually want back.

I think that accounts for the inconsistency. Cache scope is per-ref: a PR can
restore from its own ref or from the default branch, never from another PR.
Master has nothing on it, so every PR starts cold, and whether a re-run hits
comes down to whether someone else's nix flood evicted your entry in the
meantime. Same key, different outcome, depending who ran last.

It doesn't help that dart.yml, javascript.yml and python.yml only fire on
pull_request, and csharp.yml otherwise only on tag pushes. None of the four
ever runs on a push to master, so none of them can seed a cache for PRs to
inherit. rust.yml and format.yml do run on push, and even those have seeded
nothing, which I think just confirms the eviction.

@benalleng you flagged this in #1798 already ("Perhaps it is worth exploring
whether or not this job is totally worth it if our repo immediately fills the
cache space"). It does fill it, more or less immediately.

Options as I see them:

1. Drop magic-nix-cache-action from the FFI workflows and lean on
   Swatinem/rust-cache. Smallest change, frees the quota straight away, but
   we give up nix caching.
2. Swap to nix-community/cache-nix-action,
   which keeps the store as one bounded entry instead of thousands.
3. Take nix caching off the GitHub quota entirely, either FlakeHub or Cachix.

I don't have a strong preference, and option 1 does trade away something real,
so I'd rather you pick. Happy to write up whichever one you want.

Disclosure: I used Claude Code to pull these numbers and draft this comment.

[emmanuelist]

Correction on my numbers above. I measured entry counts and then wrote as if I had
measured bytes, and it turns out that matters a lot here.

Actual split right now, fully paginated and cross-checked against
/actions/cache/usage:

	entries	size
magic-nix-cache (.nar.z / .narinfo)	1593	1.11 GiB
Swatinem/rust-cache (v0-rust-*)	15	9.52 GiB
total	1599	10.17 GiB

So the nix entries are around 0.7 MB each and the rust-cache ones around 635 MB. All
those thousands of nix entries are noisy but they are not what fills the quota.
rust-cache is doing that pretty much on its own.

The parts I had right still stand: magic-nix-cache does write one entry per store
path, cache scope is per-ref, and master has nothing on it for PRs to inherit. What I
got wrong was saying the nix entries were crowding rust-cache out. They weren't.

Which means #1814 takes the entry count right down and stops the per-store-path
churn, but it won't stop the eviction this issue is actually about. I should have
checked that before opening it.

The lever that does look right is save-if on rust-cache:

save-if: Determines whether the cache should be saved. If false, the cache is
only restored.

None of the 13 rust-cache call sites set it, so every PR run saves its own copy, and
those are running 0.1 to 1.5 GiB each. Something like:

- uses: Swatinem/rust-cache@v2
  with:
    save-if: ${{ github.ref == 'refs/heads/master' }}

would have PRs restore and only master save. That drops the PR-scoped entries, and
as a side effect it finally puts a cache on master for PRs to inherit, which is the
other half of what's broken here.

Downside is a PR that changes dependencies gets a colder build until master catches
up, so it's a real tradeoff rather than a free win. Happy to open it as a separate PR
once #1814 is in, if you want to go that way.

Disclosure: I used Claude Code to pull these numbers and draft this comment.

[benalleng]

I think for any PR that changes the dep tree it is ok to not get the full benefit of caching.

[cderinbogaz]

Disclosure: I work on Runnerhut. The corrected byte totals matter more than the original entry-count diagnosis. For #1824, I would verify a successful default-branch writer actually runs for each FFI cache family; making PRs restore-only cannot create a default-branch cache if that workflow never executes there. Then test an unchanged-dependency PR and a dependency-bump PR, tracking bytes, hit rate and build time. This checks the accepted cold-dependency tradeoff without claiming a new runner will change GitHub's cache scoping.