A transient filesystem error in persist_cursor kills the whole overnight run, while its two neighbours survive the same errno

[dmccoystephenson]

Summary

An overnight run on 2026-09-05 ended after tending exactly one repo of eleven candidates. The tend itself succeeded; the batch loop then died persisting the cursor.

gardener: overnight starting — 38 repo(s) in garden, strategy=random, budget=6.0h, 34 repo(s) already attempted this cycle
gardener: overnight dispatching tend for Stephenson-Software/trace (1/11 candidates this run)...
...
gardener: error: [Errno 38] Function not implemented
gardener: state.record_run failed (non-fatal): [Errno 38] Function not implemented: '/home/userland/.local/state/gardener'
gardener: notification failed (non-fatal): [Errno 38] Function not implemented: '/home/userland/.local/state/gardener/notify.env'
gardener: finished tending Stephenson-Software/trace
Traceback (most recent call last):
  File "/usr/local/bin/gardener", line 6, in <module>
  File "gardener/cli.py", line 1956, in main
  File "gardener/cli.py", line 1276, in cmd_overnight
  File "gardener/cli.py", line 1219, in persist_cursor
  File "gardener/overnight.py", line 261, in write_attempted
  File "gardener/overnight.py", line 187, in _load_cursor_file
  File "/usr/lib/python3.10/pathlib.py", line 1290, in exists
OSError: [Errno 38] Function not implemented: '/home/userland/.local/state/gardener/overnight_cursor.json'

The interesting part is the three lines above the traceback. The same ENOSYS, in the same burst, against the same directory, hit three call sites:

• state.record_run — caught, logged (non-fatal), run continues.
• the Discord notification — caught, logged (non-fatal), run continues.
• persist_cursor — uncaught, process exits, ten remaining candidates never dispatched.

Why it matters

ENOSYS on a stat is not a real "unsupported operation" here — it is the Android/proot filesystem under this sandbox misbehaving transiently. It was gone by the time the failure was noticed: the same Path.exists() and Path.stat() on the same file succeeded immediately afterward, returning a valid 1257-byte cursor.

So a momentary blip in a bookkeeping write ended a six-hour budget after one repo. That is the same failure shape #44 addressed for device-global errors — a transient condition being charged far more than it costs — except here it does not even fail a repo, it fails the loop that dispatches them.

The irony worth noting: persist_cursor is on this path because of #42/#43, which moved cursor writes inside the batch loop so a task-swipe kill could not forget what had been attempted. Making the cursor durable also made it a per-batch crash point, and it is the only one of the three post-tend bookkeeping steps without a guard.

Suggested fix

Guard persist_cursor the way its two neighbours in cmd_overnight are already guarded — catch OSError, log it as non-fatal, and continue the batch loop. The cost of a skipped cursor write is a repo possibly re-attempted next cycle, which is enormously cheaper than losing the rest of the night.

_load_cursor_file calling Path.exists() before reading is also a check-then-act that a try: read / except FileNotFoundError: would avoid, removing one of the two stat calls that can raise here. Worth doing regardless, since it is strictly less filesystem interaction on a filesystem that demonstrably lies.

If a transient-error retry is wanted rather than a plain skip, note that the existing AUTH_RETRY_BACKOFF_SECONDS machinery is keyed on dispatch classification, not on local I/O; a short inline retry around the write would be the simpler shape.

Reproduction

Not reproducible on demand — it depends on the sandbox filesystem faulting. The evidence is the log above, and the fact that all three call sites saw the same errno within the same second while only one of them was fatal.

Filed by Claude during an automated triage pass; the traceback above is from a real run and the post-crash filesystem state was verified by hand.