feat: bound crash-recovery respawn, expose spool depth, make worker timeouts configurable

[Brian Krabach (bkrabach)]

Summary

Three opt-in changes to fix crash-recovery runaway and add observability to the durable queue.

Motivating incident (real, measured on a host install):

• Durable queue spool grew to 38 GB / 583 files (largest 4.9 GB) with no visible signal
• On restart: lifespan startup logged crash recovery respawned 94/94 drainers
• Took ~4 minutes, peaked at 43.9 GB RSS
• Killed by kernel OOM killer; systemd restarted repeatedly
• 16 worker boots per 30 minutes for two days, serving zero requests
• Backlog could never shrink (cycle repeated every boot)
• Only symptom: graph silently stopped updating; nobody noticed for two days

Changes

1. Bounded crash-recovery respawn

• New config option: crash_recovery_respawn_limit: int | None = None
• Lifespan respawn loop now slices recovered[:limit] when finite limit is set
• Default None preserves today's unbounded behaviour bit-for-bit — this PR is a no-op for any deployment that sets nothing
• Deferred sessions are never touched (no read, no write, no drainer) — they remain exactly as durable and recoverable as before
• Later boot's recover() reports deferred sessions again; a new event spawns their drainer via get_or_create()
• recovered is already sorted, so deferred sessions deterministic across restarts
• When cap trips: logs at WARNING (not INFO) with exact respawned/deferred counts and setting name
• Deferred backlog never silent — silence is what hid the 38 GB spool

2. Spool depth on /status

• New QueueManager.spool_stats() — stat-only directory scan (never reads file content)
• 5s cached (separate from derive_all_stats's 1s cache) — safe under frequent polling even with thousands of files
• Returns exactly: {"pending_sessions": int, "spool_bytes_total": int}
• /status is unauthenticated, honours existing aggregate-only contract: two integers, no session ids, no workspace names
• Impact: silent two-day outage becomes five-minute one

3. Configurable gunicorn timeouts

• New config options: gunicorn_worker_timeout: int = 30, gunicorn_graceful_timeout: int = 10
• Replace hardcoded literals in run()
• Defaults exactly match previous values (no behaviour change)
• Server whose startup is O(backlog) can now accommodate that

Verification

uv sync && uv run pytest tests/ -q
→ 1913 passed, 4 skipped, 1 error in 137s

The single error is pre-existing and environmental:

• tests/integration/test_docker_image.py performs a real docker build
• Hit transient 502 Bad Gateway from pythonhosted.org during container install
• Touches none of the changed files
• Flagged for context rather than omitted

New tests cover:

• Unbounded-by-default behavior
• Cap enforced with warning and exact respawned/deferred counts
• Deferred sessions verified intact and recoverable on subsequent boot
• Edge case: cap=0
• /status spool block present and correctly typed
• Reflects real backlog; leaks no session ids or workspace names
• spool_stats(): byte/session counting, exclusion of fully-committed and dead-letter-only sessions, TTL caching
• Both gunicorn timeouts: defaulting and overriding end-to-end through run()

python_check clean on all 7 files (pre-existing warnings confirmed by diffing against git show HEAD:)

Maintainer Note

Change 1 alters startup semantics only when explicitly opted into. The maintainer may reasonably want to discuss whether the default should eventually become finite — that discussion is deliberately deferred by keeping it unbounded here, making this PR safe to merge without changing any existing deployment's behaviour.

Files Changed

• context_intelligence_server/config.py — config options
• context_intelligence_server/main.py — lifespan respawn, /status stats
• context_intelligence_server/queue_manager.py — spool_stats()
• tests/test_config.py — config tests
• tests/test_main.py — /status and respawn tests
• tests/test_queue_manager.py — spool_stats() tests
• tests/test_run_entrypoint.py — gunicorn timeout tests

Generated with Amplifier

[Diego Colombo (colombod)]

Added hardening commit 3ace10e — in service of this PR's goal

Hi Brian Krabach (@bkrabach) — I pushed one commit on top of this branch to make PR #73's own promise (robust/reliable durable queue on ACA) hold end-to-end. It's a no-op on the default path (crash_recovery_respawn_limit=None); every existing deployment is unaffected. It closes four gaps where the current PR's mechanisms were incomplete:

1. /status must never 500 — it's the ACA health probe. spool_stats() used iterdir() (raises on a missing/unavailable queue dir) where every sibling reader uses glob() (empty, no raise), and its offset read was unguarded. On the unauthenticated /status, an uncaught exception = failed probe = container restart loop. It now degrades to a sentinel {-1,-1,-1} (not cached, self-heals) and skips a per-file corrupt offset.

2. A corrupt .offset also 500'd /status via derive_all_stats() (runs before spool_stats in get_status) — pre-existing on main. It now degrades that offset to committed=0 and surfaces the condition as an aggregate spool.corrupt_offsets count. Deliberately no logging (the probe is polled — a per-scan warning would flood); operators watch the field.

3. A finite crash_recovery_respawn_limit permanently stranded the deferred backlog. recover() runs only at boot, so a finite cap (or cap=0) drains the head and leaves the tail until a restart or a new event — for completed sessions, never. Added a bounded periodic sweep (crash_recovery_sweep_interval_seconds=300, active only under a finite ceiling) that tops the drainer pool back up to the ceiling; idempotent get_or_create keeps live recovered drainers ≤ the ceiling while the tail advances. All three new settings are now documented in server-config.example.yaml.

4. Bounded recovery/stats memory — the actual OOM cause. recovery_seed_counts(), _complete_data_end(), and _count_dead() read whole (multi-GB) spool files into RAM via read_bytes() before the cap could apply — so the cap capped the respawn loop but not the dominant boot cost. Replaced with 1 MiB streaming (backward-scan for last newline; chunked newline counts), numerically identical.

Evidence

• No regression: uv run pytest -m "not neo4j" → 1856 passed, 4 skipped; pyright clean on changed source.
• Memory: 1 GiB log, isolated processes — peak RSS 1559 MB → 37 MB, identical counts (before,pending,dead). Extrapolated to the incident's 4.9 GB file: ~7.5 GB → ~37 MB.
• Health probe: real gunicorn+uvicorn single worker — /status returns 200 under normal / corrupt-offset / missing-queue-dir / recovery.
• Sweep: cap=2, 5 recoverable sessions → all drain across bounded passes (≤ cap live at once), deferred tail not stranded.
• Deployment: launched in an isolated Incus environment — boots single-worker without Neo4j (tolerated) and serves /status 200 with the spool block.

Happy to split any of these into separate commits, reword, or drop #3's default-on sweep to opt-in if you'd prefer — your call on the merge shape.