82 orphaned tmp*.tmp files in state/health/ — atomic_write cannot clean up after a SIGKILL

[adrianwedd]

state/health/ on the live robot holds 82 orphaned tmp*.tmp files dating from 2026-08-06 to 2026-08-23. Found while working #291; filed separately because the cause is not in the consolidation path and the fix is a design decision, not a one-liner.

What they are

pxh.state.atomic_write() writes via tempfile.mkstemp(dir=path.parent, suffix=".tmp") -> write -> flush -> fsync -> chmod 0644 -> os.replace. Its except BaseException: unlinks the temp on any exception.

The orphans are all mode 0600 (i.e. pre-chmod) and split into two sizes:

• 169 bytes / 338 bytes - a complete health record. Write and flush finished; chmod/os.replace never ran.
• 0 bytes - killed before the flush.

That is the signature of the process being killed, not of an exception: SIGKILL gives Python no chance to run the except branch. fsync on this SD card has been measured taking seconds under load, which is a wide enough window to be hit repeatedly.

Why the dates matter

  1 2026-08-06 pi        1 2026-08-19 pi
  3 2026-08-14 pi        7 2026-08-19 root
  9 2026-08-15 pi        1 2026-08-20 pi
  8 2026-08-15 root      4 2026-08-20 root
 16 2026-08-16 root      5 2026-08-21 root
  7 2026-08-17 root      1 2026-08-22 pi
 11 2026-08-18 root      1 2026-08-23 pi
                         7 2026-08-23 root

The root-owned ones (px-alive, px-battery-poll) peak on 2026-08-16 - the px-alive restart storm - and nothing has been produced since 2026-08-23, i.e. since the #286/#219 memory-pressure and restart fixes landed. So the leak is a symptom of processes dying, and the underlying trigger is largely already fixed.

What is left

The residue is still a real (small) problem: 82 root-owned 0600 files in a 1777 directory that nothing will ever clean up, and the count grows on every future SIGKILL.

atomic_write cannot fix this from inside the dying process. Options:

1. A sweep in health._ensure_health_dir() - unlink tmp*.tmp older than, say, 1h. Cheap, local, but adds a directory scan to a hot write path, and health writes must never raise.
2. A sweep at px-mind / px-alive startup - runs once, no hot-path cost, but only covers dirs those daemons know about.
3. A generic state.sweep_stale_temps(dir, older_than_s) called by whoever owns each directory.

Whichever is chosen it must respect the cross-user rule in CLAUDE.md: state/health/ is 1777 precisely because root and pi both write there, so a pi sweeper must tolerate EPERM on root-owned files (sticky bit blocks unlink) rather than raising.

The live files have deliberately not been deleted - they are the evidence.

[adrianwedd]

Grooming 2026-08-26 — the residue is still growing: 82 → 95

$ ls state/health/tmp*.tmp | wc -l
95

Thirteen more since this was filed on 2026-08-24. The body's reading that "the
underlying trigger is largely already fixed" needs qualifying: whatever is being
SIGKILLed is still being SIGKILLed. Worth checking the ownership/date split
again before choosing a fix — if the new ones are root-owned, the px-alive
restart path is still killing mid-fsync (and note #287: px-alive is being
SIGTERM'd by a systemd start-timeout ~4×/day, which is a plausible producer
the original analysis predates).

The three options in the body are unchanged. Option 2 (a sweep at daemon
startup) reads best: it keeps the hot write path clean, and a daemon that just
started is exactly the process that knows a crash may have happened. The
cross-user constraint stands — a pi sweeper must tolerate EPERM on
root-owned files under the sticky bit rather than raising.

Live files still preserved as evidence.