[MikeBishop]

Summary

index --cleanup --concurrency N doesn't keep N summarizations running. processBatch() in src/indexer.ts runs fixed batches: it takes N items, awaits Promise.all, and only then starts the next N. One slow item idles the other N−1 slots until it finishes.

In a real backfill with --concurrency 8, one long conversation (hierarchical summary, chunks summarized one after another) held the run at a single summarizer subprocess from 17:12 until about 17:58. The first 5 minutes had produced 39 summaries; that stretch produced none. Details in #204.

Fix

Replace the fixed batches with a worker pool: N workers each pull the next item as soon as their current one finishes.

• Sync process hits 14GB RSS due to overlapping SDK subprocesses during summarisation #61's bound still holds. Peak concurrency is still N, which is what keeps live summarizer subprocesses bounded.
• Results stay in input order.
• New module: the pool lives in a new src/concurrency.ts (mapWithConcurrency) so it can be tested on its own. processBatch() in src/indexer.ts keeps its signature and call sites and now delegates to it. That also keeps this PR from conflicting with fix(indexer): don't resume the parent session to summarize a subagent (#205) #208, which edits the code right next to processBatch().

dist/ is regenerated with npm run build. dist/mcp-server.js doesn't bundle the indexer and isn't included.

This covers part 1 of the proposal in #204, which gets most of the benefit: a long conversation now ties up one slot while the others keep working through the queue. Part 2, running its chunks through the same limit, would only help once it's the last item left, so I've left it out for now.

Test plan

• New test/concurrency.test.ts (4 tests):
  • results come back in input order;
  • at most concurrency items run at once (peak measured: 4 of 4);
  • with concurrency 2 and item 0 blocked, items 1–5 all finish through the free slot. This one fails against main, where only item 1 can finish;
  • an empty list, and concurrency larger than the list.
• npm test (full suite): 354/354 pass, 64 test files.
• npm run build: clean.
• Merges cleanly with fix(summarizer): treat transcript as data and reject non-summary replies (#206) #207, fix(indexer): don't resume the parent session to summarize a subagent (#205) #208 and fix(install): install runtime dependencies only (--omit=dev) (#203) #209; with all four merged, npm test passes 366/366.

Refs #204.

---

Drafted with AI assistance (Claude Code). I reviewed the change; the timings are from my own backfill run.

🤖 Generated with Claude Code