Fix unbounded memory growth in sst dev from retained esbuild contexts - #6960
Open
mph6083 wants to merge 1 commit into
Open
Fix unbounded memory growth in sst dev from retained esbuild contexts#6960mph6083 wants to merge 1 commit into
mph6083 wants to merge 1 commit into
Conversation
In dev mode the Node runtime kept an esbuild incremental BuildContext per function alive for the whole session and never disposed any of them. Each context privately caches the function's entire parsed dependency graph (source text, ASTs, sourcemaps), which retains hundreds of MB per function for non-trivial graphs. Sessions on large apps grew to tens of GB of live, GC-unreclaimable heap; GOMEMLIMIT could not contain it. - Keep dev build contexts in an LRU cache capped by SST_BUILD_CONTEXT_CACHE (default 4, 0 disables caching) and Dispose evicted contexts. Contexts are pinned while a build is in flight so they cannot be evicted mid-rebuild. - Stop retaining the full BuildResult per function. ShouldRebuild now checks a compact set of absolute input paths recorded once per build, instead of re-unmarshaling the full metafile JSON for every function on every file change. - Add SST_PPROF=1 to expose net/http/pprof on the dev server under /debug/pprof/ so this class of issue can be profiled in a running session.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
sst devkeeps one esbuild incremental build context per function, forever. Each context holds the parsed module graph of that function's bundle. For a handler that pulls in@aws-sdkplus a shared core package, that's 220-550 MB of live heap, and nothing ever disposes it.Worse, the retained unit is effectively the build, not the function: rebuilding the same handler after a code change grows memory again each time. On our monorepo (311 handlers), RSS hit 12.4 GiB after building 28 of them, and a developer doing a normal edit-test loop routinely reaches 100+ GiB over a working session.
GOMEMLIMITcan't contain it because the bytes are reachable, so the GC has no authority to drop them. The only reclaim today is restartingsst dev.Measured on 4.17.1, editing one already-built handler 8 times with the distinct-handler count held constant: +1972 MB, about 219 MB per rebuild, monotonic. Repeat invocations without edits cost nothing, so the leak is entirely in the build path.
ShouldRebuildadded to this: it kept the fullBuildResultper function (metafile JSON included) and re-parsed that JSON on every file-change event.Fix
Build contexts now live in an LRU cache capped at 4 entries by default. Evicted contexts get
Dispose()d, which frees the module graph. A context is pinned while a build is running on it, so it can't be evicted mid-build; concurrent builds of the same function that race on context creation dispose the loser.The cap is tunable with
SST_BUILD_CONTEXT_CACHE. Setting it to 0 disables context caching entirely.ShouldRebuildnow works off a plain set of absolute input paths, extracted from the metafile once per build, instead of retaining and re-parsing the whole build result. This also means dependency tracking keeps working for functions whose context was evicted.The trade-off: if you're actively editing more than 4 functions in rotation, the coldest one loses its incremental-build state and pays a full rebuild on the next change. That's the same cost as the function's first dev build, in exchange for a bounded memory ceiling.
Separately,
SST_PPROF=1now mounts the standardnet/http/pprofhandlers on the dev server under/debug/pprof/. Pinning this leak down required sampling/proc/<pid>/memby hand because no heap profile was reachable from a runningsst dev; next time it should be onego tool pprofinvocation.Testing
go test ./pkg/runtime/node/passes. Two new tests:TestDevBuildContextsAreBounded: builds 5 functions with the cap set to 2 and asserts no more than 2 live contexts remain.TestDevBuildShouldRebuildTracksInputs: with the cap set to 1, verifiesShouldRebuildstill matches input files for an evicted function, rejects unrelated files, and that rebuilding an evicted function works.