Skip to content

Fix unbounded memory growth in sst dev from retained esbuild contexts - #6960

Open
mph6083 wants to merge 1 commit into
anomalyco:devfrom
mph6083:fix/dev-esbuild-context-leak
Open

Fix unbounded memory growth in sst dev from retained esbuild contexts#6960
mph6083 wants to merge 1 commit into
anomalyco:devfrom
mph6083:fix/dev-esbuild-context-leak

Conversation

@mph6083

@mph6083 mph6083 commented Aug 5, 2026

Copy link
Copy Markdown

Problem

sst dev keeps one esbuild incremental build context per function, forever. Each context holds the parsed module graph of that function's bundle. For a handler that pulls in @aws-sdk plus a shared core package, that's 220-550 MB of live heap, and nothing ever disposes it.

Worse, the retained unit is effectively the build, not the function: rebuilding the same handler after a code change grows memory again each time. On our monorepo (311 handlers), RSS hit 12.4 GiB after building 28 of them, and a developer doing a normal edit-test loop routinely reaches 100+ GiB over a working session. GOMEMLIMIT can't contain it because the bytes are reachable, so the GC has no authority to drop them. The only reclaim today is restarting sst dev.

Measured on 4.17.1, editing one already-built handler 8 times with the distinct-handler count held constant: +1972 MB, about 219 MB per rebuild, monotonic. Repeat invocations without edits cost nothing, so the leak is entirely in the build path.

ShouldRebuild added to this: it kept the full BuildResult per function (metafile JSON included) and re-parsed that JSON on every file-change event.

Fix

Build contexts now live in an LRU cache capped at 4 entries by default. Evicted contexts get Dispose()d, which frees the module graph. A context is pinned while a build is running on it, so it can't be evicted mid-build; concurrent builds of the same function that race on context creation dispose the loser.

The cap is tunable with SST_BUILD_CONTEXT_CACHE. Setting it to 0 disables context caching entirely.

ShouldRebuild now works off a plain set of absolute input paths, extracted from the metafile once per build, instead of retaining and re-parsing the whole build result. This also means dependency tracking keeps working for functions whose context was evicted.

The trade-off: if you're actively editing more than 4 functions in rotation, the coldest one loses its incremental-build state and pays a full rebuild on the next change. That's the same cost as the function's first dev build, in exchange for a bounded memory ceiling.

Separately, SST_PPROF=1 now mounts the standard net/http/pprof handlers on the dev server under /debug/pprof/. Pinning this leak down required sampling /proc/<pid>/mem by hand because no heap profile was reachable from a running sst dev; next time it should be one go tool pprof invocation.

Testing

go test ./pkg/runtime/node/ passes. Two new tests:

  • TestDevBuildContextsAreBounded: builds 5 functions with the cap set to 2 and asserts no more than 2 live contexts remain.
  • TestDevBuildShouldRebuildTracksInputs: with the cap set to 1, verifies ShouldRebuild still matches input files for an evicted function, rejects unrelated files, and that rebuilding an evicted function works.

In dev mode the Node runtime kept an esbuild incremental BuildContext per
function alive for the whole session and never disposed any of them. Each
context privately caches the function's entire parsed dependency graph
(source text, ASTs, sourcemaps), which retains hundreds of MB per function
for non-trivial graphs. Sessions on large apps grew to tens of GB of live,
GC-unreclaimable heap; GOMEMLIMIT could not contain it.

- Keep dev build contexts in an LRU cache capped by SST_BUILD_CONTEXT_CACHE
  (default 4, 0 disables caching) and Dispose evicted contexts. Contexts are
  pinned while a build is in flight so they cannot be evicted mid-rebuild.
- Stop retaining the full BuildResult per function. ShouldRebuild now checks
  a compact set of absolute input paths recorded once per build, instead of
  re-unmarshaling the full metafile JSON for every function on every file
  change.
- Add SST_PPROF=1 to expose net/http/pprof on the dev server under
  /debug/pprof/ so this class of issue can be profiled in a running session.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant