Skip to content

daemon: detect host suspend and recover the transport on resume - #442

Merged
TeoSlayer merged 1 commit into
mainfrom
fix/sleep-wake-wedge
Jul 29, 2026
Merged

daemon: detect host suspend and recover the transport on resume#442
TeoSlayer merged 1 commit into
mainfrom
fix/sleep-wake-wedge

Conversation

@TeoSlayer

Copy link
Copy Markdown
Collaborator

Closing a laptop lid wedges the daemon and it does not reliably self-heal — the operator has to run pilotctl daemon restart by hand.

Why it doesn't self-heal

There is no suspend/resume awareness anywhere in the tree — no IORegisterForSystemPower, no wake notification, nothing. On resume every UDP path is stale: the NAT mapping the beacon punched is gone, peer sessions are dead, and the pooled registry conn is half-open with no liveness ping of its own.

Recovery therefore falls through the ordinary rx-silence path, which takes ~7m40s at best (3 soft attempts + threshold) and frequently never completes.

Field evidence — 344 withheld restarts on one laptop

301  "never progressed"            87%
 39  "process too young"           11%
  4  "registry also unreachable"    1%

The dominant blocker is the never-progressed guard. It withholds the hard exit while PktsRecv has not advanced once this process, reasoning a restart would reproduce the same state and boot-loop.

That is correct for a host whose inbound never worked. It is wrong after a suspend, where the transport worked before the gap and a restart is exactly what recovers it — which is what the operator ends up doing by hand.

(Worth noting: the registry check I initially suspected accounts for 1.2%. The logs corrected the theory.)

Changes

Detect the suspend. A ticker set to fire every rxWatchdogTickInterval that instead fires 4x late means this process was not running. Detection is deliberately clock- and OS-agnostic — measured from observed ticker lateness, not a power API — so it behaves identically on macOS, Linux and in a paused VM, with no platform-specific code.

On detection: re-punch via RegisterWithBeacon, force a fresh registry connection, and re-baseline the wedge counters — whose pre-suspend values describe a different epoch and would otherwise read as a multi-hour rx silence and burn all soft attempts immediately.

Unlock the never-progressed guard once a suspend has been seen, so the hard exit can still escalate if the proactive recovery didn't take. The restart-loop circuit breaker still bounds churn.

Recovery goes from ~7m40s-or-never → one tick (≤30s).

Tests

  • a wedge with no inbound this process must not exit without a suspend (the boot-loop the guard exists to prevent) and must exit with one, at exit code 86
  • resume re-baselines counters and sets the flag
  • the gap threshold sits above ordinary tick jitter but below a realistic lid-close

Full pkg/daemon suite green.

🤖 Generated with Claude Code

Closing a laptop lid wedges the daemon and it does not reliably self-heal;
the operator has to run `pilotctl daemon restart` by hand.

There was no suspend/resume awareness anywhere in the tree — no
IORegisterForSystemPower, no wake notification, nothing. On resume every UDP
path is stale: the NAT mapping the beacon punched is gone, peer sessions are
dead, and the pooled registry conn is half-open with no liveness ping of its
own. Recovery therefore had to fall through the ordinary rx-silence path,
which takes ~7m40s at best (3 soft attempts + threshold) and frequently never
completes at all.

Field evidence from one laptop's daemon.log — 344 withheld restarts:

    301  "never progressed"            (87%)
     39  "process too young"           (11%)
      4  "registry also unreachable"   (1%)

The dominant blocker is the never-progressed guard: it withholds the hard
exit while PktsRecv has not advanced once this process, reasoning that a
restart would reproduce the same state and boot-loop. That is correct for a
host whose inbound never worked. It is wrong after a suspend, where the
transport worked before the gap and a restart is precisely what recovers it —
which is what the operator ends up doing manually.

Two changes:

  - Detect the suspend. A ticker set to fire every rxWatchdogTickInterval
    that instead fires 4x late means this process was not running. Detection
    is deliberately clock- and OS-agnostic — measured from observed ticker
    lateness, not a power API — so it behaves identically on macOS, Linux and
    inside a paused VM, and needs no platform-specific code. On detection:
    re-punch via RegisterWithBeacon, force a fresh registry connection, and
    re-baseline the wedge counters, whose pre-suspend values describe a
    different epoch and would otherwise read as a multi-hour rx silence and
    burn soft attempts immediately.

  - Unlock the never-progressed guard once a suspend has been seen, so the
    hard exit can still escalate if the proactive recovery above did not take.
    The restart-loop circuit breaker still bounds churn.

Recovery goes from ~7m40s-or-never to one tick (<=30s), without waiting on
guards that were reasoning about a different failure.

Tests: a wedge with no inbound this process must NOT exit without a suspend
(the boot-loop the guard exists to prevent) and MUST exit with one; resume
re-baselines counters and sets the flag; and the gap threshold is checked to
sit above ordinary tick jitter but below a realistic lid-close.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer merged commit e887f87 into main Jul 29, 2026
14 checks passed
@TeoSlayer
TeoSlayer deleted the fix/sleep-wake-wedge branch July 29, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants