[aditya-rai-5]

What

Retry engine health probes during startup adoption for up to 30 seconds instead of stopping an engine after one failed probe.

Why

A busy engine, such as an MLX server handling a long generation, can miss a single 5-second health probe. Restarting the agent would then stop the surviving engine and trigger an unnecessary model reload.

Fixes #2406

How

• Retry failed adoption probes after a 5-second delay, within a 30-second deadline.
• Preserve the engine and its record if adoption is cancelled.
• Keep existing process-identity, executable, and bind-address checks.
• Add regression tests for recovery after a timeout and cancellation during retries.
• Verify persistent health failures still stop the engine after the retry window. Tests use virtual time to avoid real delays.

Validation completed:

• Adoption tests with the race detector.
• make test-affected.
• Formatting, generation, vet, and Linux/Darwin lint checks.
• The full make gate pass.

AI assistance: Codex generated the tests.

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[Defilan]

Thanks for picking this up, and the synctest tests are a nice way to cover it. I'd like to change the approach, though, because of where adoption runs.

The stall. startAdoption runs in Start before prepareRegistry, the relay ingress, the heartbeat loop and the watcher (pkg/agent/agent.go:806), and adoptEngines handles records one at a time. So each engine that isn't serving now holds all of that for 30 s instead of about 5 s, and several stuck engines add up. In relay mode the ingress is the only way the cluster reaches an engine, so healthy adopted engines on the same Mac stay unreachable until adoption finishes.

Long generations. An engine that doesn't answer /health for more than 30 s is still stopped, and a long generation can outlast that. That's the case #2406 was filed for.

What I'd suggest. The other option in #2406: when an engine passes the identity, executable and bind host checks but misses its probe, adopt it as unhealthy and leave it to the health monitor. The monitor already has the right behavior: the first failure withdraws the endpoint, and 3 consecutive failures at its 30 s interval restart the engine. That removes the stall and gives a busy engine about 90 s.

One catch: the monitor assumes anything it tracks has finished loading (see the comment on applyResult), so a record still in the provisional state, where the engine never finished loading, should keep today's behavior and be stopped.

Timing. I'm also holding agent changes for a little while after 0.10.17, so this won't merge right away even once it's updated. I've started CI so you get results in the meantime.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!