Fix concurrent map crash in dev server's Lambda bridge - #6970
Open
kieranm wants to merge 2 commits into
Open
Conversation
nextChan and workers in cmd/sst/mosaic/aws/function.go are written by the event loop while the lambda runtime-API handlers read them from request goroutines. Under a burst of concurrent invocations a read races a write and the process dies with a concurrent map fatal. Guard all access to both maps (and the CurrentRequestID field read by the per-worker log pump) behind a mutex, and add a regression test that reproduces the race deterministically under go test -race. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kieranm
force-pushed
the
fix-dev-function-map-race
branch
from
August 17, 2026 15:01
d29ac41 to
9ea3a24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6567
Problem
sst devcrashes withfatal error: concurrent map read and map writeunder concurrent Lambda invocations.nextChanandworkersincmd/sst/mosaic/aws/function.goare written by the event loop goroutine while the lambda runtime-API HTTP handlers read them from request goroutines, with no synchronization. A cold-start burst (e.g. the first page load of an app with many functions) reliably kills the whole dev process — most recently reproduced on 4.17.1 / macOS ARM64, matching the reports in #6567.Fix
One mutex guarding both maps, with two small locked accessors (
loadNextChan,loadWorker) used by the handlers and the event loop. The handler-sideloadNextChanalso creates the channel on first access (same buffer size the event loop used), which removes the nil-channel wait for a worker whose init message hasn't arrived yet. TheWorkerInfo.CurrentRequestIDfield — written by the event loop, read by the per-worker log pump goroutine — is guarded by the same mutex.Channel operations stay outside the lock, so blocking behaviour is unchanged.
Reproduction / verification
The included regression test drives both sides concurrently with no infrastructure (a
MessageInitfor an unknown function id performs the racing map write and returns before any bridge/project access):go test -race -count=1 -run TestFunctionConcurrentInitAndNext ./cmd/sst/mosaic/aws/fails in ~0.02s with the race pinned to the handler's map read.Note the race detector isn't enabled in CI's
go test, so the test passes harmlessly there on both versions; it exists so the race stays reproducible with-race.Also verified against a real app (60+ functions) with a race-instrumented build of the CLI: before the fix, page-load bursts produced multiple data-race reports including the fatal site above; after, none.