`sst dev` never reaps exited function workers — zombies accumulate for the lifetime of the process

[cheruvian]

Summary

sst dev spawns one runtime worker process per function instance, but never calls cmd.Wait() on them. When a worker exits, its process-table entry is never collected, so it stays as a <defunct> zombie for as long as sst dev runs. On a long dev session this accumulates without bound.

This is distinct from #6133 (merged). That fixed orphans — child processes left running after a sst dev timeout, addressed with process-group kills. This is the opposite lifecycle bug: the parent is alive and healthy, the children are already dead, and nobody reaps them. I am on 4.17.1, which includes #6133, and still see it.

Environment

• sst 4.17.1 (sst-darwin-arm64)
• macOS 26.5.2, arm64
• node 22.22.0, pnpm
• sst dev (multiplexer mode, default)

Evidence

Sampling the live sst dev process (PID 47795) every 5s shows an exact 1:1 correlation between workers exiting and zombies appearing:

time	live nodejs-runtime workers	zombies under sst dev
22:27:44	1 → 3 (2 spawned)	36
22:27:54	3 → 2 (1 exited)	36 → 37
22:28:46	2 → 0 (2 exited)	37 → 39

Every worker exit produces exactly one permanent zombie. After ~25 minutes of ordinary development this process had 39; an earlier session accumulated 39 in its first 6 minutes under heavier reload activity (~2.7/min).

$ ps -Ao ppid,state | awk '$1==47795 && $2 ~ /^Z/' | wc -l
39

$ ps -Ao pid,ppid,state,comm | awk '$2==47795 && $3 ~ /^Z/' | head -3
51347 47795 Z+ <defunct>
51687 47795 Z+ <defunct>
51776 47795 Z+ <defunct>

The parent is alive the whole time, so these are never reparented to PID 1 and never reaped by init:

$ ps -o pid,ppid,etime,rss,comm -p 47795
  PID  PPID   ELAPSED     RSS COMM
47795 47789     25:26 2798240 .../sst-darwin-arm64/bin/sst

Cause

pkg/runtime/node/node.go starts the worker and never waits on it:

func (r *Runtime) Run(ctx context.Context, input *runtime.RunInput) (runtime.Worker, error) {
	cmd := process.Command(
		"node", "--enable-source-maps", "--no-warnings",
		filepath.Join(path.ResolvePlatformDir(input.CfgPath), "/dist/nodejs-runtime/index.js"),
		filepath.Join(input.Build.Out, input.Build.Handler),
		input.WorkerID,
	)
	// ...
	cmd.Start()   // no corresponding cmd.Wait()
	return &Worker{stdout, stderr, cmd}, nil
}

func (w *Worker) Stop() {
	process.Kill(w.cmd.Process)   // kills, but never reaps
}

In Go, a process started with Cmd.Start() and never Cmd.Wait()ed stays a zombie until its parent exits. Stop() signals the child but never collects the status, so the entry persists.

grep -c 'cmd\.Wait()' returns 0 for all three runtimes — the only .Wait() in each file is a sync.WaitGroup used for log pumping:

file	cmd.Start()	cmd.Wait()	process.Kill
pkg/runtime/node/node.go	1	0	1
pkg/runtime/python/python.go	1	0	1
pkg/runtime/golang/golang.go	1	0	1

So this affects every runtime, not just Node.

Suggested fix

Reap the child after signalling it, e.g. in Worker.Stop():

func (w *Worker) Stop() {
	process.Kill(w.cmd.Process)
	_ = w.cmd.Wait()   // collect the status so the entry is released
}

Or go func() { _ = cmd.Wait() }() after Start() if the exit status is wanted for logging/restart decisions. The same change applies to the python and golang runtimes.

Impact

Each zombie holds only a process-table entry, so this is not a memory problem — but it is unbounded, and a long-lived sst dev on a busy machine contributes to PID pressure. It also makes ps output noisy and can trip process-count monitoring.