Skip to content

Fix concurrent map crash in dev server's Lambda bridge - #6970

Open
kieranm wants to merge 2 commits into
anomalyco:devfrom
kieranm:fix-dev-function-map-race
Open

Fix concurrent map crash in dev server's Lambda bridge#6970
kieranm wants to merge 2 commits into
anomalyco:devfrom
kieranm:fix-dev-function-map-race

Conversation

@kieranm

@kieranm kieranm commented Aug 17, 2026

Copy link
Copy Markdown

Fixes #6567

Problem

sst dev crashes with fatal error: concurrent map read and map write under concurrent Lambda invocations. nextChan and workers in cmd/sst/mosaic/aws/function.go are written by the event loop goroutine while the lambda runtime-API HTTP handlers read them from request goroutines, with no synchronization. A cold-start burst (e.g. the first page load of an app with many functions) reliably kills the whole dev process — most recently reproduced on 4.17.1 / macOS ARM64, matching the reports in #6567.

internal/runtime/maps.fatal({...})
github.com/sst/sst/v3/cmd/sst/mosaic/aws.function.func1(...)
        cmd/sst/mosaic/aws/function.go:90

Fix

One mutex guarding both maps, with two small locked accessors (loadNextChan, loadWorker) used by the handlers and the event loop. The handler-side loadNextChan also creates the channel on first access (same buffer size the event loop used), which removes the nil-channel wait for a worker whose init message hasn't arrived yet. The WorkerInfo.CurrentRequestID field — written by the event loop, read by the per-worker log pump goroutine — is guarded by the same mutex.

Channel operations stay outside the lock, so blocking behaviour is unchanged.

Reproduction / verification

The included regression test drives both sides concurrently with no infrastructure (a MessageInit for an unknown function id performs the racing map write and returns before any bridge/project access):

  • Without the fix: go test -race -count=1 -run TestFunctionConcurrentInitAndNext ./cmd/sst/mosaic/aws/ fails in ~0.02s with the race pinned to the handler's map read.
  • With the fix: passes.

Note the race detector isn't enabled in CI's go test, so the test passes harmlessly there on both versions; it exists so the race stays reproducible with -race.

Also verified against a real app (60+ functions) with a race-instrumented build of the CLI: before the fix, page-load bursts produced multiple data-race reports including the fatal site above; after, none.

vimtor and others added 2 commits February 20, 2026 12:26
nextChan and workers in cmd/sst/mosaic/aws/function.go are written
by the event loop while the lambda runtime-API handlers read them
from request goroutines. Under a burst of concurrent invocations a
read races a write and the process dies with a concurrent map
fatal. Guard all access to both maps (and the CurrentRequestID
field read by the per-worker log pump) behind a mutex, and add a
regression test that reproduces the race deterministically under
go test -race.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kieranm
kieranm force-pushed the fix-dev-function-map-race branch from d29ac41 to 9ea3a24 Compare August 17, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent map panic in dev server under load

2 participants