docs: operational hardening — bound memory, stop silent crash loops

[Brian Krabach (bkrabach)]

Motivating Incident (Real, Measured)

On a production-scale host (121 GB RAM / 3.7 TB disk):

• The durable queue spool grew to 38 GB / 583 files (largest 4.9 GB) with no signal
• Neo4j ran with no heap cap and no container memory limit, auto-sized from host RAM → 22 GB
• On restart: startup logged crash recovery respawned 94/94 drainers, took ~4 minutes, peaked at 43.9 GB RSS
• Kernel OOM killer SIGKILLed the process
• The systemd unit had Restart=on-failure with no StartLimitBurst → 16 worker boots per 30 minutes for two days, serving zero requests, burning 9.5 CPU-hours
• Only symptom: a graph that silently stopped updating. Nobody noticed for two days.
• Remediation, verified: 14/14 probes returned 200 over 10 minutes, memory plateaued at 1.2 GB, 1 worker, 0 restarts.

---

Changes

docs/operational-hardening.md (NEW, 325 lines)

The complete incident narrative, the four operational guards, a detection table ("how to tell this is happening to you"), and an explicit capability/limitation matrix ("what buys you what").

docs/service-setup.md

• §2's Neo4j docker run now carries --memory/--memory-swap plus explicit heap/pagecache env vars
• §5's systemd unit heredoc now carries StartLimitIntervalSec, StartLimitBurst, MemoryMax, MemoryHigh
• §9 gained three troubleshooting rows for OOM/memory pressure scenarios
• (This repo ships no systemd unit file, only a macOS plist template — this heredoc is the unit template, so guards now apply by default to anyone following the primary install path)

README.md

New "Resource Limits and Crash-Loop Guards" section linking docs/operational-hardening.md.

server-config.example.yaml

• Expanded write_concurrency comment carrying real evidence from a sibling deployment: "default 8 thundered neo4j's ~20.6GiB tx-memory ceiling on a restart-with-backlog → 307 OOM/hr"
• Added queues_path operational warning

docs/local-development.md

One bullet naming the deliberate absence of memory limits on dev Neo4j, with a pointer to the operational doc.

---

Evidence (All Empirically Validated on Real Hosts)

✅ Neo4j heap config propagation:
Started throwaway neo4j:5.26.22-community with NEO4J_server_memory_heap_max__size=512m; confirmed it landed as server.memory.heap.max_size=512m in neo4j.conf and -Xmx524288k on the JVM.

✅ Container memory limit interaction:
Ran same image with --memory 2g and no heap env: neo4j.conf got only an auto-computed pagecache size and the JVM carried no -Xmx at all — which is why the doc recommends both flags, and corroborates the incident (no limit ⇒ sizing reads the whole 121 GB host).

✅ systemd configuration validation:
systemd-analyze verify on a deliberately-wrong unit printed Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.` while exiting 0 (silent misconfiguration). It also flagged three unrelated real units on the test box making that exact mistake. The unit as written in service-setup.md was extracted and re-verified clean.

✅ systemd memory limit precision:
systemd-run --user -p MemoryMax=24G -p MemoryHigh=20G produced exactly the byte values quoted in the doc.

✅ Bash and config syntax:
All 44 bash blocks across the four docs pass bash -n; the YAML still parses; all markdown link targets resolve.

---

Note for Maintainer

The docs currently recommend write_concurrency: 4 as a should, not a consider. Evidence: one sibling deployment's config comment (above) plus a second host carrying the same hazard untested. If you'd rather this read as a recommendation than an instruction, §4 of docs/operational-hardening.md and the server-config.example.yaml comment are the two places to soften language from "should" to "consider".

---

Generated with Amplifier