perf(io): sustained 25% IO PSI from swap thrashing on single mmcblk0 device — not fsync-bound

[adrianwedd]

IO PSI Evidence (TRACK C — C3)

Measured live 2026-08-20T21:30 AEST on picar.local.

Current IO Pressure

some avg10=21.63 avg60=22.98 avg300=33.58 total=3948401799
full avg10=12.69 avg60=17.43 avg300=26.94 total=3153136325

Interpretation: IO pressure is significant and sustained:

• some (at least one task stalled on IO): 22-34% of time across all windows
• full (all tasks stalled on IO): 13-27% of time across all windows
• The 5-minute average (avg300) is higher than the 10-second average (avg10), indicating this is a persistent condition, not a transient spike.

For context, the Pi has a single eMMC/SD card (mmcblk0) as its only storage. CPU PSI is low (3-4%) and memory PSI is near zero (0-1.5%), so IO is the dominant pressure axis.

Top IO Writers (5-second sample, /proc/PID/io delta)

Only three processes showed measurable IO activity during a 5-second quiet-period sample:

PID	Comm	Write KB/s	Read KB/s	WChar KB/s	Notes
946	go2rtc	0.0	0.0	2460	Camera stream relay — writes to pipe/socket, not disk
1129	rpicam-vid	0.0	0.0	1220	Camera capture — writes to pipe, not disk
1060	python3 (px-wake-listen)	1.6	0.0	0.1	Log file writes

Key finding: No SPARK daemon is producing significant disk IO during idle periods. The wchar from go2rtc/rpicam-vid is pipe/socket traffic to the camera pipeline, not filesystem writes.

Cumulative IO (since boot, 3h58m)

PID	Comm	write_bytes	read_bytes	wchar	rchar	Notes
2181	claude	140 MB	102 MB	139 MB	886 MB	Resident brain session — highest disk writer
5670	python	45 MB	16 MB	43 MB	304 MB	Likely px-mind or px-post
1060	python3 (wake-listen)	29 MB	435 MB	20 MB	3166 MB	High rchar = model file reads at startup
856	uvicorn	0.5 MB	41 MB	5.9 MB	2305 MB	API server — mostly reads
1028	ffmpeg	0	41 MB	0	193 KB	Frigate stream processing

fsync-heavy Services

atomic_write() in src/pxh/state.py uses mkstemp + fsync + os.replace on every state write. Callers:

Module	Call sites	Frequency
state.py	update_session(), ensure_session(), log rotation	Per voice turn + per session update
health.py	record_success() / record_failure()	Every daemon tick (throttled by min_interval_s)
brain.py	ask_brain() mailbox writes, meter, validation	Per Claude request

The health.py throttle is load-bearing here: px-alive ticks 2x/second and without min_interval_s would produce 120 fsyncs/minute. The throttle was designed for exactly this SD-card wear concern.

Disk Device Stats

mmcblk0: 110493 reads, 99626 writes, write_ticks=199494726 (199s total write time)
mmcblk0p2 (root): 110049 reads, 99623 writes

Assessment

The IO PSI is real and sustained (~25% full pressure), but it is NOT caused by SPARK daemons' fsync patterns. The evidence:

1. During a 5-second quiet sample, no SPARK process wrote more than 1.6 KB/s to disk
2. The highest cumulative disk writer is claude (140 MB in 4 hours = ~10 KB/s average) — the resident brain session's tmux journal
3. The IO pressure is more likely caused by:
   • Swap activity (339 MB of 512 MB swap in use, 217 MB from px-wake-listen alone)
   • Background filesystem journaling (ext4 jbd2)
   • Camera pipeline buffer flushes (go2rtc/ffmpeg reading SD card for model files or writing temporary frames)

The 217 MB of px-wake-listen swap is a contributing factor: swap-in/swap-out generates IO on mmcblk0, which is the same device as rootfs. This creates a feedback loop: memory pressure -> swap -> IO pressure -> all tasks stall.

Conclusion

No actionable top IO offender was identified among SPARK's fsync-heavy services. The IO PSI is driven by:

1. Swap thrashing (217 MB from px-wake-listen, 28 MB from GLaDOS) — addressable by reducing memory footprint (see perf(voice): px-wake-listen holds ~689 MiB anonymous, 446 MiB in one unreclaimable heap (900 MiB peak) #219, perf(tts): px-tts-glados holds 463 MiB steady (228 MiB model weights, 95% unreferenced) — 0 synthesize calls in 7 days #246)
2. Camera pipeline (go2rtc + rpicam-vid + ffmpeg) — high wchar but mostly pipe traffic, not direct disk writes
3. Single-device architecture — rootfs, swap, and camera storage all share mmcblk0

Recommendation: The most effective IO PSI reduction is memory footprint reduction (fewer pages swapped = fewer IO operations). No filesystem redesign needed. The existing atomic_write fsync pattern is well-throttled and not the bottleneck.

Not assumed to be #217 (brcmfmac wedge) — the IO pressure is storage-device-bound, not WiFi-SDIO-bound.

[adrianwedd]

Grooming 2026-08-26 — promoted: this now has a proven user-visible consequence

When filed, this issue's own conclusion was "no actionable top IO offender…
recommendation: reduce memory footprint". #219 has since assessed that reduction
and declined it — every option costs wake accuracy, latency or CPU (closed
with that reasoning recorded). So the lever this issue pointed at is gone.

Meanwhile the cost of the pressure became measurable. Per the new evidence in
#283: 110 instrumented arecord overruns have a median
psi_io_some_avg10 of 91.6 %, with 94 % of them above 50 %. IO pressure is
now a demonstrated causal link from swap-on-mmcblk0 to deleted seconds of a
child's utterance to hallucinated transcripts.

Current state, live 2026-08-26 (much milder than the 25 % sustained originally
measured, but spiky):

/proc/pressure/io: some avg10=8.56 avg60=4.76 avg300=16.08
                   full avg10=6.97 avg60=3.99 avg300=13.44

…while individual overrun moments record psi_io_some_avg10_at of 78–99.98.
The average is not the story; the spikes are, and they land exactly on capture.

What changes about the ask. "Reduce memory footprint" is closed off. The
remaining levers are structural and were listed here as the third cause without
being pursued:

• the single-device architecture — rootfs, swap and camera all on mmcblk0.
  Moving swap off the SD card (USB SSD, or zram so it never touches the device
  at all) is the one change that breaks the memory-pressure → swap → IO → stall
  feedback loop without touching model selection. zram in particular costs
  nothing but CPU and needs no hardware.
• IOWeight=/IOSchedulingClass= on the camera pipeline vs. px-wake-listen,
  now that the io controller is available (ops(kernel): restore usable memory accounting and pressure instrumentation #218 closed — cgroup.controllers
  includes io). Capture should not lose the device to a stream relay.

Both are configuration, not code. Worth measuring the zram option first: it is
reversible in one reboot and directly targets the mechanism.

[adrianwedd]

zram mitigation deployed; pause for natural measurement

The structural lever identified in the grooming comment is now live: zram0 1 GiB / zstd / priority 100, with /var/swap retained at priority -2, swappiness=100, page-cluster=0 (commit 114bdffb). Functional verification passed and IO/cgroup instrumentation remains intact.

The camera/Frigate ffmpeg path still reports write_bytes=0, so the proposed IOWeight= step is not justified by current evidence and was not applied.

This issue should now sit in observation rather than accumulate more tuning. Over the next few days compare natural mmcblk0/PSI/swap behaviour and #283 overrun data against the pre-zram baseline. If zram breaks the swap→SD→stall feedback loop materially, this issue can be narrowed to any residual non-swap IO pressure or closed. If not, the next investigation should identify the remaining mmcblk0 stall source under the cleaner regime.