sst deploy/diff/refresh hangs indefinitely — pulumi engine blocked on LanguageRuntime.Run (Node program never returns)

[MuzzleFork]

Summary

sst deploy, sst diff, and sst refresh all hang indefinitely after the StaticSite dynamic resources are processed. Every process (pulumi engine, language host, all resource providers, the Node program, the dynamic-provider) sits at 0% CPU — a lost-wakeup deadlock, not a slow AWS poll.

A goroutine dump of the pulumi engine shows it is blocked waiting for the Node program to finish (LanguageRuntime.Run never returns): the program registers resources, then hangs before completing, so the engine's step executor and deployment goroutines idle forever waiting for resources that never arrive.

Environment

• SST 4.12.2 (also reproduced on 4.17.1)
• @pulumi/pulumi 3.215.0, pulumi engine binary v3.215.0, @pulumi/aws 7.20.0
• Node 24.15.0 (also reproduced on 22.23.2)
• macOS 26.5.2 (arm64)
• App: ~560–635 resources including 4 sst.aws.StaticSite components (each: command:local:Command builder + BucketFiles (AssetFiles) + DistributionInvalidation + KvKeys + CloudFront distribution).

Symptoms

• Reproduces deterministically on two independent stages.
• Reproduces in deploy, diff, and refresh.
• Reproduces even with the last-known-good commit's graph on an empty diff (no changes to apply) — so it is not the application code / infra delta.
• Onset ~2026-07-29; the same graph deployed cleanly the day before with the identical pinned toolchain.
• Confirmed on a clean CI Linux runner (GitHub Actions), not just macOS. A deploy job applied all four StaticSites' builders + AssetFiles, then went totally silent for ~11.5 minutes right after one site's AssetFiles before being cancelled — with the identical idle process tree (sst deploy / pulumi / pulumi-language-nodejs / esbuild / pulumi-resource-aws). So it is not a single-host/environment issue — it is the graph + toolchain.

Evidence

PULUMI_DEBUG_GRPC trace

The last completed RPCs are always the StaticSite dynamic resources:

Check/Diff  ...StaticSite$pulumi-nodejs:dynamic:Resource::SiteDAssetFiles.sst.aws.BucketFiles  -> response_completed
Check/Diff  ...StaticSite$pulumi-nodejs:dynamic:Resource::SiteDInvalidation.sst.aws.DistributionInvalidation -> response_completed

After that, only /pulumirpc.LanguageRuntime/Run remains in-flight (never returns) and no further RPC is ever emitted.

Engine goroutine dump (GOTRACEBACK=all + SIGQUIT)

goroutine 127 [select, 1 minutes]:
    google.golang.org/grpc ... (*csAttempt).recvMsg -> waitOnHeader          # blocked awaiting the RPC response
    github.com/pulumi/pulumi/sdk/v3/proto/go.(*languageRuntimeClient).Run(...)
    github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin.(*langhost).Run(...)
    github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*evalSourceIterator).forkRun.func1.1()

goroutine 128 [chan receive, 1 minutes]:
    github.com/pulumi/pulumi/pkg/v3/resource/deploy.newStepExecutor.func1()   # step-executor worker, idle (no steps)

goroutine 119/120 [chan receive, 1 minutes]:
    github.com/pulumi/pulumi/pkg/v3/engine.newDeployment.func1/func2()        # deployment, idle

The engine is doing the right thing (awaiting Run); the Node program is the one that never finishes. The Node program host (@pulumi/pulumi/cmd/run/index.js) and the dynamic-provider process are both idle in the libuv event loop (uv__io_poll/kevent); all Go providers are in pthread_cond_wait.

What we ruled out

Hypothesis	Result
Application code / infra delta	Empty diff of the last-good graph still hangs
Corrupt/oversized state	An untouched stage's state (unchanged since a clean deploy) hangs; state is structurally clean (no dependency cycles, no dangling refs, no duplicate providers — a single default_7_20_0 + the intentional us-east-1 and cross-account-DNS providers)
Version regression (ours)	sst 4.12.2 pinned for months; @pulumi/pulumi 3.215.0 + @pulumi/aws 7.20.0 are identical across the entire SST 4.x line. The exact same toolchain deployed cleanly for ~3 months and began hanging 2026-07-29 with zero version movement — so it is not a version bump on our side, and there is no older SST 4.x to pin to.
SST version	4.12.2 and 4.17.1 (both bundle pulumi 3.215.0)
Node version	24.15 and 22.23
Local plugin/platform cache	Original hangs were on a clean setup; also reproduces on a clean CI Linux runner
PULUMI_NODEJS_AUTO_STACK_GLOBAL=0	The AsyncLocalStorage regression guard (per pulumi #20685/#20737) — still hangs
GODEBUG=asyncpreemptoff=1	Go-scheduler async-preemption off — still hangs (consistent with the hang being in the Node program, not the Go engine)
Reduced parallelism	PULUMI_NODEJS_PARALLEL / --parallel variations — no effect
Cross-account DNS auth	The provider's sts:AssumeRole into a role in a separate (shared) account succeeds in ~2s; cross-account Route53 reads/writes work
StaticSite builder churn (Date.now trigger)	SKIP=1 (ignoreChanges on builders) still hangs

Reproduction — isolated to a component/interaction, NOT size or any single feature

We built a fresh throwaway SST app (same sst 4.12.2 / pulumi 3.215.0) and progressively added every structural trait of the failing app, then scaled it past the failing app's resource count. Every variant deploys cleanly:

Fresh-stack repro variant	~Resources	Result
4 domain-less StaticSites + 3 Lambdas	~70	✅ deploys
+ cross-account DNS provider (assume-role) + Route53 records	~80	✅ deploys
+ custom domains → ACM certs (us-east-1 provider) + cross-account cert-validation/alias records — 3 aws providers, same as the failing app	~110	✅ deploys
+ real shell-out builders (command:local:Command) replacing every deploy, changing output → BucketFiles reconcile each deploy	~110	✅ deploys
+ padding to 230 / 350 / 470 / 590 / 710 resources (Lambdas, Dynamo tables, Secrets)	up to 710	✅ deploys
+ 30 command:local:Command created and replaced (> the app's 27)	~740	✅ deploys

The failing app is 578 resources (68 LinkRef, 63 route53 Record, 42 BucketObject, 27 command:local:Command, 22 LogGroup, 21 Secret/Function/Role/Lambda, 16 pulumi-nodejs:dynamic:Resource, 14 Bucket, 13 Dynamo/Certificate, 3 aws providers). A fresh stack of 710 resources (+23% vs the app) with identical structural features deploys cleanly — so it is not resource count / graph size, and not any of: the toolchain, cross-account DNS, ACM certs, the 3-provider setup, StaticSite dynamic resources, builders, or command count. The trigger is a specific component or interaction present only in the real graph. The un-replicated candidates are an APIGW WebSocket stack (custom authorizer Lambda + routes + integrations + deployment) and SQS queues + subscriber Lambdas (event-source mappings); pinpointing needs bisecting the real app (its components are interwired).

Questions

1. Is this a known step-executor / language-host (evalSourceIterator.forkRun ↔ Run) lost-wakeup in this pulumi version? It is not resource-count-driven (a fresh 710-resource stack is fine); it appears tied to a specific resource/component type or an interaction only the real graph has.
2. The hang is in the Node program (LanguageRuntime.Run never returns) while the engine waits — is there a known lost-wakeup in the nodejs language host or the pulumi-nodejs dynamic-provider RPC path at high concurrency? (--parallel reductions had no effect.)
3. Is there a fixed @pulumi/pulumi / engine version to pin to (given 3.215.0 is what SST 4.x bundles, and 3.202.0 introduced at least two regressions — #20737, #21865)?
4. Given it worked for months then began hanging with an unchanged pinned toolchain, is there an out-of-band input (a plugin/registry fetch, an AWS-side behaviour) that could flip a latent race?

Repro capture commands

# reproduce with tracing
PULUMI_DEBUG_GRPC=/tmp/grpc.json GOTRACEBACK=all npx sst diff --stage <stage> --verbose
# once wedged (0% CPU, no new RPCs), dump the engine goroutines:
kill -QUIT "$(pgrep -f 'sst/bin/pulumi' | head -1)"   # dump -> .sst/log/pulumi.err.log

[MuzzleFork]

Minimal reproduction: https://github.com/MuzzleFork/sst-nodejs-plugin-deadlock-repro

I've narrowed this to a single trigger: a Function that loads a custom esbuild plugin via nodejs.plugins. Stripped to the smallest failing case — 3 trivial Lambdas + one no-op plugin that stubs a module to export {}:

• REPRO_PLUGIN=0 npx sst deploy → deploys in ~30s (✓ Complete).
• npx sst deploy (plugin on, the default) → wedges forever at the ~ Deploy line, the pulumi process at 0% CPU, no error, no timeout.

The only difference between the two runs is the nodejs: { plugins: "esbuild-plugin.mjs" } line on the Function.

Likely cause: any plugin (any onResolve/onLoad) routes bundling through the esbuild-service ↔ Node callback channel, and that esbuild context doesn't appear to be disposed after the build. Its open handle keeps the Node event loop alive, so the program never returns from LanguageRuntime.Run and the engine waits on it indefinitely — which matches the "0% CPU, blocked on LanguageRuntime.Run" symptom in the original report.

Confirmed on sst 4.12.2 (pulumi 3.215.0), Node v24.15.0. Full steps + root-cause notes are in the repo README.