Skip to content

Update caching docs for canonical-overlay removal #578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
frostebite merged 7 commits into from
Aug 15, 2026
Merged

Update caching docs for canonical-overlay removal #578

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
33a7611
Update caching docs for canonical-overlay removal
frostebite Aug 12, 2026
2834bd3
Reflect move-directory as the new default cache mode
frostebite Aug 12, 2026
837ef94
Fix broken link and formatting flagged by CI
frostebite Aug 12, 2026
a7e00a1
Remove all remaining canonical-overlay documentation, not just the note
frostebite Aug 12, 2026
75b62a5
Merge branch 'main' into remove-canonical-overlay-cache-docs
frostebite Aug 14, 2026
5a8550b
Merge branch 'main' into remove-canonical-overlay-cache-docs
frostebite Aug 14, 2026
ac4ff82
Merge branch 'main' into remove-canonical-overlay-cache-docs
frostebite Aug 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 12 additions & 202 deletions docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,191 +301,6 @@ preserve asset imports (textures, meshes, shader cache), which are profile-indep
See [Self-Hosting and Orchestrator](self-hosting-and-orchestrator#profile-switching) for the
fingerprinting pattern.

## Canonical Cache + Overlay (Advanced)

For studios running multiple self-hosted runners on the same physical host with multi-GB Library
folders, the existing `localCacheMode` values trade off in known ways:

- `tar` — pays archive + extract on every restore.
- `move-directory` — instant atomic same-volume rename, but **consume-once**: only one runner
takes the cache; the next runner cold-starts.
- `copy-directory` — multi-runner-safe, but pays full byte-copy cost on every restore. For a
30 GB Library folder this is minutes per build.

The opt-in `canonical-overlay` mode combines the multi-runner safety of `copy-directory` with the
zero-copy speed of `move-directory`. The pattern is well-known from VFSForGit, Scalar, the Nix
store, and Bazel CAS: write the cache **once** to a content-addressed canonical store, and per-
runner overlays consume it via OS-native hardlinks (NTFS on Windows, ext4/xfs/btrfs/zfs on Linux).

### When to use it

- Multiple self-hosted runners share the same physical host and filesystem.
- Library folder is multi-GB; per-restore copy or extract is unacceptably slow.
- Cancel-in-progress is foundational (CI cancels mid-flight pushes regularly), so the cache must
survive SIGKILL of the publishing job without corruption.

### Architecture

```
<canonicalRoot>/<cacheKey>/
Library/
<sha-A>/ # canonical version A (hardlinked into runner overlays)
<sha-B>/ # canonical version B (current)
latest -> <sha-B> # directory junction / symlink

<runner workspace>/Library/
<files> # hardlinks pointing at canonical bytes
```

The canonical store is written **atomically** by `Publish-Canonical`:

1. Walk the runner's freshly-built Library; hardlink files into `<sha-B>-staging/`.
2. Write `.cache_complete` marker into the staging directory.
3. Atomic rename `<sha-B>-staging/` → `<sha-B>/`. This is the publish moment.
4. Atomic update of the `latest` pointer.

If PostUnityJob is cancelled mid-publish (SIGKILL during the staging walk), the rename never
happens. The orphan staging directory is harmless and gets cleaned up by the next build. The
existing canonical version and `latest` pointer are intact.

When PreUnityJob runs on any runner, `Materialize-Overlay` reads the `latest` pointer and creates
hardlinks from canonical into a per-runner overlay directory. Hardlink creation is one syscall per
file (~0.1 ms on Windows). For a Library with 200,000 files, materialization is ~20 s — orders of
magnitude faster than copying multi-GB content.

### Hardlink safety contract

Not every Library subdirectory is safe to hardlink. The invariant: hardlink is safe only if the
writer uses **write-temp-then-rename**, not in-place modification. In-place modification would
write through the hardlink to canonical bytes, corrupting the canonical store for every consumer.

The default classifier targets Unity's Library structure:

| Subdirectory | Strategy | Why |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------- |
| `PackageCache/<package>@<hash>/` | Directory junction | Unity treats packages as immutable; new version = new `@<hash>` directory |
| `ScriptAssemblies/`, `Artifacts/`, `BurstCache/`, `Bee/artifacts/`, `MetadataGenerator/`, `ShaderCache/`, `StateCache/`, `UIBuilder/`, `HDRPLibrary/` | Hardlink | Roslyn, Bee, AssetDatabase v2 all use write-temp-then-rename |
| `PackageManager/projectResolution.json`, `PackageManager/ProjectCache`, `Bee/*.dag`, `Bee/*.dag.outputdata`, `Bee/*-inputdata.json`, `LastSceneManagerSetup.txt` | Per-runner copy | Contain absolute workspace paths; need cross-runner repair |
| `AnnotationManager`, `EditorOnly`, `EditorUserBuildSettings.asset`, `EditorUserSettings.asset`, `CurrentLayout-*.dwlt` | Skip | Editor session state; not part of the build cache |
| (default) | Hardlink | Conservative default for unknown subtrees |

For non-Unity engines, override the classifier with `canonicalCacheClassifier`:

```yaml
- uses: game-ci/unity-builder@v4
with:
localCacheMode: canonical-overlay
canonicalCacheClassifier: |
{
"default": "hardlink",
"rules": [
{ "pattern": "imported/**", "strategy": "junction" },
{ "pattern": "session/**", "strategy": "skip" }
]
}
```

### Configuration

```yaml
- uses: game-ci/unity-builder@v4
with:
localCacheEnabled: true
localCacheMode: canonical-overlay
canonicalCacheRoot: D:/CI/Canonical # falls back to <localCacheRoot>/canonical
canonicalCacheVersionRetention: 2 # keep last 2 SHA versions per key
cacheMaterialize: prepared # sub-second hydration
cacheSentinelCanary: true # defense-in-depth
```

### Sub-second hydration with `prepared` materialize

Eager materialize (the default for `canonical-overlay`) creates the overlay during PreUnityJob,
on the build's critical path. For a 200k-file Library this is ~20 s.

`cacheMaterialize: prepared` shifts the overlay creation off the critical path entirely. After a
successful build:

1. PostUnityJob publishes the canonical version.
2. PostUnityJob then hardlink-clones the canonical into `<overlay>-prepared/`.
3. The next PreUnityJob detects the prepared overlay, atomic-renames it into place, and starts
the build immediately.

Sub-second hydration in the steady state. If the prepared overlay's SHA doesn't match the current
canonical's `latest` SHA (a different runner published a newer version in between), it falls
through to live materialize.

### Failure modes and self-heal

| Scenario | Behaviour |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Cancel during canonical publish | Orphan `<sha>-staging/` directory; existing canonical version intact. Next build cleans up. |
| Cancel during prepared overlay build | Orphan `<overlay>-prepared/` directory; harmless. Next build falls through to live materialize. |
| Overlay missing or corrupt | `materializeOverlay` rebuilds deterministically from canonical. Idempotent. |
| Canonical missing (host loss, disk failure) | Next successful build re-establishes canonical via `publishCanonical`. One cold path back to clean state. |
| Sentinel canary mismatch (when enabled) | Overlay is discarded; falls through to live materialize. |

### Cross-platform behaviour

The strategy degrades gracefully when filesystems don't support hardlinks. Build never fails
because of strategy unavailability.

| OS / Filesystem | Behaviour |
| --------------------------- | ----------------------------------------------------------------------------------------------- |
| Windows NTFS | Full implementation — hardlinks for files, directory junctions for read-only subtrees |
| Windows ReFS | Hardlinks work; reflinks (per-file COW) documented as future stretch |
| Linux ext4 / xfs | Hardlinks work; junctions emulated via symlinks or per-runner copy |
| Linux btrfs / zfs | Hardlinks work; reflinks (block-level COW) documented as future stretch |
| macOS APFS | Hardlinks work; clonefile reflinks documented as future stretch |
| Containerised runs (Docker) | Falls back to `move-directory` — canonical store doesn't make sense across container boundaries |
| Cross-volume runners | Falls back to `move-directory` — hardlinks can't cross drive letters / mount points |

When falling back, a structured warning is emitted via `OrchestratorLogger`.

### Limitations

1. **Single-host topology only.** All runners must share one filesystem. Multi-host CI farms need
different infrastructure (S3/MinIO via the existing `storageProvider` rclone path, or planned
future strategies — see "Future strategies" below).
2. **NTFS, ext4, xfs needed for full benefit.** ReFS, btrfs, zfs, APFS work via hardlinks but
reflink-aware variants (per-file copy-on-write — strictly better) are documented as future
stretch options.
3. **`Remove-Item -Recurse` on junctions has historically been unsafe in some PowerShell
versions.** The implementation uses `cmd /c rmdir /s /q` for junction-safe directory removal
on Windows.
4. **1023 hardlinks per inode on NTFS.** Not a practical constraint at typical fleet sizes;
`canonicalCacheVersionRetention: 2` (default) limits inode-link count.
5. **Modify-in-place writes propagate to canonical bytes.** Mitigated by the classifier audit
above. Unknown subtrees default to hardlink (conservative for read-mostly use); the
`cacheSentinelCanary` flag provides defense-in-depth.
6. **Tested at scale on Windows NTFS in a 10-runner farm.** The Linux hardlink path lands with
platform-gated tests but has not been validated at multi-GB scale yet.

### Performance characteristics

| Configuration | Materialize time on critical path |
| --------------------------------------------------------- | ------------------------------------------- |
| `tar` | Minutes for multi-GB archive + extract |
| `copy-directory` | Minutes for multi-GB byte copy |
| `move-directory` | Sub-second, but consume-once across runners |
| `canonical-overlay`, eager materialize | 15-30 s for ~200k files (hardlinks only) |
| `canonical-overlay`, eager + PackageCache junctions | 2-8 s |
| `canonical-overlay`, prepared overlay (steady state) | < 1 s (atomic rename) |
| `canonical-overlay`, prepared overlay (canonical changed) | 2-8 s (live materialize fallback) |

### Future strategies in this taxonomy

The `localCacheMode` value space is extensible. Future strategies expected to follow the same
shape (none implemented yet — RFC welcome):

- **`reflink-overlay`** — block-level copy-on-write via Linux btrfs/xfs/zfs reflinks, macOS
APFS clonefile, or Windows ReFS. Equivalent to `canonical-overlay` but per-file COW (no
modify-in-place footgun).
- **`bind-mount`** — read-only bind mount of canonical with overlayfs (Linux) or junction-based
Windows variants for filesystems that don't support fine-grained hardlinks.
- **`nfs-passthrough`** — single-source-of-truth pattern; treat a network mount as canonical and
skip the publish step entirely. Trades network read latency for zero local disk usage.

## Self-Hosted Operational Lessons

The following operational lessons apply when running orchestrator on long-lived self-hosted
Expand Down Expand Up @@ -542,20 +357,15 @@ caching strategies above — fewer wasted builds means less cache churn.

## Inputs Reference

| Input | Description |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `cacheKey` | Override the cache key used for cache isolation |
| `cacheCheckpointInterval` | Minutes between Library checkpoints; `0` disables |
| `cacheSaveOnFailure` | Save a partial cache after non-zero build exit |
| `cacheRetentionDays` | Remove cache entries older than N days |
| `maxRetainedWorkspaces` | Number of retained full workspaces to keep |
| `maxCacheEntries` | Max tar snapshots to retain per cache folder (default: 2) |
| `minCacheEntries` | Minimum cache entries to keep during age-based GC (floor) |
| `skipCache` | Skip cache restore entirely |
| `useCompressionStrategy` | Use LZ4 compression for cache archives |
| `localCacheMode` | One of `tar`, `move-directory`, `copy-directory`, `canonical-overlay` |
| `canonicalCacheRoot` | Path for the canonical store (when `localCacheMode: canonical-overlay`); falls back to `<localCacheRoot>/canonical` |
| `canonicalCacheClassifier` | JSON describing per-subdirectory hardlink/junction/copy/skip strategy; defaults target Unity Library |
| `canonicalCacheVersionRetention` | How many canonical SHA versions to keep per cache key (default: 2) |
| `cacheMaterialize` | `eager` (live materialize) or `prepared` (atomic-rename a pre-built overlay for sub-second hydration) |
| `cacheSentinelCanary` | Defense-in-depth corruption check; writes a known-content file into the overlay and verifies on consume |
| Input | Description |
| ------------------------- | ---------------------------------------------------------- |
| `cacheKey` | Override the cache key used for cache isolation |
| `cacheCheckpointInterval` | Minutes between Library checkpoints; `0` disables |
| `cacheSaveOnFailure` | Save a partial cache after non-zero build exit |
| `cacheRetentionDays` | Remove cache entries older than N days |
| `maxRetainedWorkspaces` | Number of retained full workspaces to keep |
| `maxCacheEntries` | Max tar snapshots to retain per cache folder (default: 2) |
| `minCacheEntries` | Minimum cache entries to keep during age-based GC (floor) |
| `skipCache` | Skip cache restore entirely |
| `useCompressionStrategy` | Use LZ4 compression for cache archives |
| `localCacheMode` | One of `move-directory` (default), `copy-directory`, `tar` |
Loading
Loading