Remove canonical-overlay cache mode - #40
Conversation
The canonical-overlay caching strategy (per-runner hardlink/junction overlays materialized from a single content-addressed store) added significant complexity and per-file syscall overhead for large Library trees, with a fully silent perf cliff on cross-volume hardlink failure (EXDEV fell back to a full copy with no logging at any level). Removes canonical-cache-service.ts and its wiring from local-cache-service.ts and plugin-lifecycle.ts. Setting localCacheMode: canonical-overlay now throws a clear error pointing users at move-directory or tar instead, rather than silently doing something else. tar, move-directory, and copy-directory remain as local cache modes; distributed caching remains available via the existing rclone/S3 built-in container hooks (see docs/03-github-orchestrator/07-advanced-topics/05-hooks/05-built-in-hooks.mdx), unaffected by this change. See game-ci/roadmap#11 for the full caching audit and rationale. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe change removes canonical-overlay cache support from action inputs, environment mappings, local cache contracts, restore/save workflows, and lifecycle configuration. The lifecycle now rejects the removed mode and supports tar, copy-directory, and move-directory modes. ChangesCanonical-overlay removal
Estimated code review effort: 3 (Moderate) | ~20 minutes Mergeability Score: 🔵 Low · up to The cache-mode removal is otherwise localized, but the deprecation error can be bypassed when local caching is disabled, leaving invalid configuration accepted silently; this is a bounded configuration-consistency risk requiring owner awareness or follow-up. Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/model/orchestrator/services/cache/local-cache-service.ts (1)
223-225: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMatch directory modes explicitly instead of "not tar".
Line 223 routes every non-
tarvalue torestoreDirectoryCache. The caller insrc/plugin-lifecycle.ts(lines 612-615) passesconfig.localCacheMode as any, so the compiler does not verify the value. An unknown or removed mode string silently selects directory restore rather than failing.The lifecycle layer rejects
canonical-overlayat config-read time, so this is not reachable today. An explicit allow-list keeps the dispatch safe if that guard changes.saveCacheFolderat line 331 has the same shape.♻️ Proposed explicit mode matching
- if (options.restoreMode && options.restoreMode !== 'tar') { + if (options.restoreMode === 'move-directory' || options.restoreMode === 'copy-directory') { return LocalCacheService.restoreDirectoryCache(projectPath, cachePath, folder, options); }Apply the same change at line 331:
- if (options.saveMode && options.saveMode !== 'tar') { + if (options.saveMode === 'move-directory' || options.saveMode === 'copy-directory') {Then drop the
as anycasts insrc/plugin-lifecycle.tssoLocalCacheModeis enforced at the call sites.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/model/orchestrator/services/cache/local-cache-service.ts` around lines 223 - 225, Update the mode dispatch in LocalCacheService restore and save flows to call directory handling only for the explicitly supported directory mode, rather than any value other than "tar"; preserve tar handling and reject or otherwise avoid silently accepting unknown modes. Remove the as any casts at the plugin-lifecycle call sites so LocalCacheMode is enforced by the type system.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/plugin-lifecycle.ts`:
- Around line 275-284: Move the canonical-overlay rejection from the cache
restore/save path into configuration construction, or explicitly evaluate the
lazy config.localCacheMode getter during lifecycle initialization. Ensure
canonical-overlay always throws during initialization, even when local caching
is disabled, while preserving valid mode handling.
---
Nitpick comments:
In `@src/model/orchestrator/services/cache/local-cache-service.ts`:
- Around line 223-225: Update the mode dispatch in LocalCacheService restore and
save flows to call directory handling only for the explicitly supported
directory mode, rather than any value other than "tar"; preserve tar handling
and reject or otherwise avoid silently accepting unknown modes. Remove the as
any casts at the plugin-lifecycle call sites so LocalCacheMode is enforced by
the type system.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ca6e254e-4246-4f51-bcfc-493396696f13
📒 Files selected for processing (5)
action.ymlsrc/model/orchestrator/services/cache/canonical-cache-service.test.tssrc/model/orchestrator/services/cache/canonical-cache-service.tssrc/model/orchestrator/services/cache/local-cache-service.tssrc/plugin-lifecycle.ts
💤 Files with no reviewable changes (2)
- src/model/orchestrator/services/cache/canonical-cache-service.test.ts
- src/model/orchestrator/services/cache/canonical-cache-service.ts
| const mode = getInput('localCacheMode') || 'tar'; | ||
| if (mode === 'canonical-overlay') { | ||
| throw new Error( | ||
| '[plugin-lifecycle] localCacheMode: canonical-overlay has been removed. ' + | ||
| "Use 'move-directory' (recommended for retained runners/build farms) or 'tar' " + | ||
| '(recommended for remote/ephemeral caches, or combine with rclone/S3 built-in ' + | ||
| 'container hooks — see the caching docs for details).', | ||
| ); | ||
| } | ||
| return mode; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map lifecycle declarations before inspecting configuration construction.
ast-grep outline src/plugin-lifecycle.ts --items all
# Find configuration initialization and every read of the lazy getter.
rg -n -C 5 '\blocalCacheMode\b|createPluginLifecycle|pluginLifecycle|const config|let config' src/plugin-lifecycle.tsRepository: game-ci/orchestrator
Length of output: 3500
Validate canonical-overlay during configuration initialization.
config.localCacheMode is a lazy getter and is read only during cache restore and save. If local caching is disabled, canonical-overlay can bypass validation. Move this validation into configuration construction or force evaluation during lifecycle initialization.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/plugin-lifecycle.ts` around lines 275 - 284, Move the canonical-overlay
rejection from the cache restore/save path into configuration construction, or
explicitly evaluate the lazy config.localCacheMode getter during lifecycle
initialization. Ensure canonical-overlay always throws during initialization,
even when local caching is disabled, while preserving valid mode handling.
* Default localCacheMode to move-directory; add EXDEV copy fallback Switches the default local cache mode from tar (archive+extract, copies bytes twice) to move-directory (O(1) atomic rename), matching child-workspace-service's already-proven pattern. tar remains available as an explicit opt-in, e.g. for distributing cache entries via the existing rclone/S3 built-in container hooks. Adds a renameOrCopy fallback in both local-cache-service.ts and child-workspace-service.ts: a cross-volume rename (EXDEV) or permissions failure (EPERM) now degrades to a copy + source cleanup instead of throwing and losing the cache entirely. Previously this was an unhandled failure mode in both services. Updates two existing tests that implicitly relied on tar being the default to explicitly request tar mode, and adds new coverage for the new default and the EXDEV fallback on both restore and save paths. Part of the game-ci/roadmap#11 caching workstream, following on from #40 (canonical-overlay removal). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Rename localCacheMode input to cacheMode "local" no longer accurately describes this input now that distributed caching via the rclone/S3 built-in container hooks is a supported (if not-yet-unified, see #42) path. localCacheMode keeps working as a deprecated alias — existing workflows aren't broken, but get a one-time deprecation warning pointing at cacheMode. Internal call sites now read config.cacheMode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
It was experimental and removed for real problems (#40) - the RFC skeleton doesn't need to keep citing it as prior art. Rephrased the per-subtree strategy and isCapable comments to describe the concept on its own terms instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Removes the
canonical-overlaylocal cache mode entirely, per the decision in game-ci/roadmap#11 (workstream 1).localCacheMode: canonical-overlaynow throws a clear error at config-read time instead of running, pointing users atmove-directoryortar.canonical-cache-service.ts+ its test, and all wiring inlocal-cache-service.tsandplugin-lifecycle.ts(including the now-unusedcanonicalCacheRoot,canonicalCacheClassifier,canonicalCacheVersionRetention,cacheMaterialize,cacheSentinelCanaryinputs).tar,move-directory, andcopy-directorylocal cache modes are unaffected.docs/03-github-orchestrator/07-advanced-topics/05-hooks/05-built-in-hooks.mdx) are unaffected — this PR only touches the local, single-machine cache modes.Net: -1287 lines, 5 files changed, no other behavior touched.
Why
Full audit and rationale live in game-ci/roadmap#11 (workstream 1) — summarizing the parts that motivated this specific PR:
What
canonical-overlaydid: kept one content-addressed copy of a cache folder (e.g.Library) per<cacheKey>/<folder>/<sha>/, computed via a structural SHA over relative paths+sizes. On publish, it copied/hardlinked the source into a staging directory, atomically renamed it into the canonical store, and flipped alatestjunction/symlink. On restore, it hardlinked (or junction'd/copied, per a per-subtree classifier) from the canonical store into a per-job overlay directory.Why it's being removed, not just deprioritized:
EXDEVacross filesystem volumes, and the fallback to a full copy was not logged at any level, not even debug — a user could be silently getting full-copy performance while believing they were getting zero-copy hardlinks, with no signal anywhere in the logs.Library/after copying it into the canonical store — the job's own copy was simply left behind, duplicated, on every publish.replicateTree) and the SHA computation (computeVersionSha) did a full recursive per-file walk with a syscall per file — for Library trees with 100k+ small files (common on large Unity projects, exactly the case this feature targeted), this overhead dominated even though bytes weren't being copied.pruneOldVersions(sorts/removes canonical versions by mtime) had no locking against a concurrentmaterializeOverlayreadinglatest— two simultaneous publishers could race.execSync('cmd /c ...'), a "prepared overlay" pre-fetch optimization) for a local-filesystem-only mechanism, whenchild-workspace-service.ts's much simplerfs.renameSync-based move approach already exists in this codebase and has none of the above problems.Given all of that, the maintainer's call was to remove it outright rather than harden it in place, and let the simpler
move-directory/tar/copy-directorymodes (plus the existing rclone/S3 hooks for distributed caching) cover the space instead.What's NOT in this PR (tracked separately in game-ci/roadmap#11)
LocalCacheModefromtartomove-directory— a separate, distinct behavior change from this removal, tracked as a follow-up.EXDEV→copy-directoryautomatic fallback formove-directory/child-workspace-service(today a cross-volume rename just throws) — follow-up.CacheBackendinterface design unifying local disk modes with the existing rclone/S3 hooks — a larger design task (RFC), not a mechanical removal.cache warm/ scheduled-hydration command — depends on theCacheBackendwork above landing first.Test plan
yarn typecheck— clean, 0 errorsyarn lint(oxlint) — 0 errors (390 pre-existing warnings, unrelated to this change)yarn test(vitest) — full suite: 60 files, 920 passed, 1 pre-existing skip, 0 failurescanonical-cache-serviceafter deletionRef: game-ci/roadmap#11
Summary by CodeRabbit
canonical-overlaylocal cache mode.