CPU backend: effective parallelism collapses after the second HTTP connection with buffered synthesis (OpenMP team accumulation)

[minamikik]

Summary

On the CPU backend, buffered (wav) synthesis served from a new HTTP connection leaves behind an OpenMP helper team bound to that connection's worker thread. Each new connection adds another team (n_threads - 1 live threads), and from the second connection onward the effective parallelism of synthesis collapses to roughly 2 cores. The process never recovers, even after hours of idle. Streaming (pcm) requests do not accumulate teams by themselves, but once a process has served buffered synthesis, streaming requests in that process inherit the degraded state.

Reproduced at pin a8a7716 on two different machines, so this does not appear to be tied to one CPU topology:

machine	logical CPUs	server threads (auto)	effective cores, 1st request	effective cores, 2nd+	live threads per connection
AMD (64C/128T), Linux/WSL2	128	32	31.9-32.9	2.0-2.2	+31
Intel Core Ultra 5 235 (14C/14T), Linux/WSL2	14	7	6.92	2.73-2.74	+6

The per-connection thread increment is exactly n_threads - 1 in both cases. Effective cores were measured as the delta of utime + stime from /proc/<pid>/stat divided by wall time; live threads as the entry count of /proc/<pid>/task.

Reproduction

1. Start tts-server with the CPU backend (no --max-batch).
2. Send a buffered wav synthesis request from a fresh HTTP connection. First request runs at full parallelism.
3. Close the connection, open a new one, send the same request again. Wall RTF degrades by ~3.4x on 14 cores (~10x on the 128-thread machine), and effective core use drops to ~2.

Fixed input/voice/seed across runs; observed with a 0.6B talker at Q4_K_M. The degraded state persists for the process lifetime (observed no recovery after 3h14m idle).

Suspected mechanism

The server invokes buffered synthesis directly on the per-connection HTTP worker thread. With the GGML CPU backend given a thread count but no persistent threadpool, GGML enters an OpenMP parallel region whose team is mastered by the calling thread. The OpenMP runtime caches one team per distinct calling thread, and connection workers live for the process lifetime, so teams accumulate per connection. Streaming synthesis runs on a short-lived per-request thread, whose helpers exit with it — consistent with streaming not accumulating.

What we have not root-caused is why the presence of residual cached teams pins new parallel regions to ~2 effective cores.

Mitigation (verified)

• --max-batch 2 avoids the issue entirely: synthesis is funneled into the internal batch worker, thread count stays fixed after the first request, and full parallelism is retained (measured equal wall RTF to the healthy first request across 4+ consecutive connections, on both machines). Concurrent requests are processed in parallel slots; requests beyond the slot count queue in waves; memory overhead was ~8 MiB peak RSS. These concurrency measurements may also be relevant to the questions raised in Concurrency support and RTF performance #27.
• Building with -DGGML_OPENMP=OFF also avoids the accumulation, at a moderate throughput cost (~33 → ~28 effective cores on the 128-thread machine).
• The CUDA backend is unaffected.

Since the help text describes --max-batch as GPU-oriented, it may be worth documenting it as the recommended CPU serving configuration, or routing CPU synthesis through a long-lived worker (or a persistent GGML threadpool) by default.

Happy to provide more measurements or test patches.