[Purple10101]

The failure this prevents

A reboot can leave every bridge-network container unable to resolve any hostname, indefinitely. Measured on jonathan-node-1 on 2026-10-05, where the ADS-B feed was dead for 67 minutes and looked exactly like a bad release:

18:43:28  NetworkManager-wait-online finishes, network-online.target is met
18:43:32  tar1090 starts; dockerd writes its resolv.conf from the host's
18:43:40  docker.service reaches active
18:43:45  NetworkManager finally writes /etc/resolv.conf   <- 13s too late
18:43:46  first getaddrinfo EAI_AGAIN

Symptom in the container: getaddrinfo EAI_AGAIN on every upstream, and the proxy serving X-Data-Source: none with 0 aircraft. The host resolved fine throughout.

Why it happens

network-online.target is not a promise that DNS works. It is satisfied when NetworkManager reports a connection is up, which here was 17 s before NetworkManager wrote the resolver configuration.

A bridge container's resolver is fixed at container start. It gets nameserver 127.0.0.11, Docker's embedded resolver, which forwards to whatever upstreams dockerd read from the host's /etc/resolv.conf at that moment. Read too early and the container keeps a dead upstream for its whole lifetime. Nothing re-reads it, and the container cannot see or fix the host file. Only a container restart regenerates it.

Containers on host networking are unaffected, which is why this presents as one service being broken rather than as a network fault.

Why nothing recovered it

Four independent mechanisms all miss it, each verified on the node:

Mechanism	Why it missed
restart: always	The proxy catches the error and keeps serving, so the process never exits. RestartCount stayed 0 across the outage.
Container healthcheck	/healthcheck.sh checks /run/readsb/aircraft.json freshness. readsb rewrites it every second regardless, so the container stayed healthy.
Healthcheck acting on failure	Docker's restart policy acts on process exit, not on healthcheck failure.
RSPduo watchdog	Monitors blah2 API staleness. The radar was unaffected.

Downstream saw no error either: HTTP 200 with an empty aircraft list. retina-telemetry reports nothing about the feed.

The change

One ExecStartPre on docker.service that waits for a non-loopback nameserver in /etc/resolv.conf. Bounded at 60 s, and exits 0 even on timeout, because a failing ExecStartPre would stop the daemon starting at all, which is worse than the race. The real race was 13 s.

Deliberately not done: pinning explicit nameservers in /etc/docker/daemon.json. It would fix the timing, but hardcodes resolvers a given site may not route to, and nodes sit on networks we do not control. Waiting for whatever the site's DHCP provides is correct everywhere.

Noted but not acted on: the rootfs also carries the image build environment's own /etc/resolv.conf until NetworkManager replaces it. On jn1 that was nameserver 127.0.0.53 plus a *.gx.internal.cloudapp.net search domain, neither of which can be generated on a Pi (the node has no systemd-resolved unit at all). That is the file dockerd read. Waiting makes its content irrelevant, so clearing it would be hygiene rather than a fix, and not worth risking an image build without a resolver mid-run.

Testing

The script against 8 resolv.conf fixtures:

fixture	behaviour
stale 127.0.0.53	waits, then proceeds
healthy (NetworkManager)	proceeds immediately
empty / comments only	waits, then proceeds
::1	waits, then proceeds
IPv6 only	proceeds immediately
loopback plus real	proceeds immediately
rewritten mid-wait	unblocks on the rewrite

And on jn1 itself: against its real resolver configuration (exits in 0 s) and against the exact stale content it carried at boot (waits, then proceeds). The drop-in could not be installed there to test the boot path, as the node has no passwordless sudo; that path is only exercised by a reboot.

Docs

owl-os keeps role reasoning in the role header rather than in docs/, as wifi_reconnect does, and the header here carries the full timeline and rationale. Nothing in docs/ describes container DNS or service ordering, so no doc change applies.

🤖 Generated with Claude Code

Update: added self-healing (3 commits now)

The first commit prevents the race on future boots. On its own that was not enough, because it ships in an OS image: every node updated before it arrives stays broken until a human notices. Three of three nodes checked were broken simultaneously today.

Second commit: hardening

docker.service runs with TimeoutStartSec=infinity on these nodes, so systemd will not rescue a hung ExecStartPre. A Docker daemon that never starts is far worse than the race. Three independent guards now:

ExecStartPre=-/usr/bin/timeout -k 10 70 /usr/local/sbin/wait-for-dns-config 60

-k 10 is load bearing: plain timeout sends only SIGTERM, which a wedged process can ignore, after which timeout itself waits forever. Measured at 12 s and still running until an outer guard SIGKILLed it, against 5 s for the same process under -k.

Third commit: the watchdog

A systemd timer every 5 minutes restarts tar1090 when, and only when, the container cannot resolve any host in its own ADSB_UPSTREAMS.

Why the trigger is resolution and not "the feed has no data":

• It catches the masked case. fairforest-b has its own ADS-B receiver, so the proxy served X-Data-Source: local and looked healthy while its remote fallback was dead. It sat that way for nearly two hours and nothing in the stack would ever have reported it.
• It does not fire during a real upstream outage. adsb.retina.fm served HTTP 200 with an empty sky for about two weeks in September. If DNS resolves, a restart cannot help, and restarting every cycle would be churn that hides the actual problem.

A probe from the host would not work at all, incidentally: the host's DNS is fine throughout. The fault is confined to the container's namespace.

Two bugs testing found, both the shape this is meant to end

• exec 9>"$RESTART_LOCK" || exit 0 meant an unwritable lock path disabled the watchdog permanently and silently. The lock guards operations that recreate the whole compose project; this restarts one container, so a lock we cannot open must not veto it. It now proceeds and logs.
• A failed state-file write was swallowed, disabling the cooldown that stops a host-level DNS outage becoming a restart every 5 minutes. It now warns.

Testing

8 decision paths against a stubbed docker:

case	expected	result
healthy, resolves	no action	pass
container not running	no action	pass
cannot resolve, first time	restart	pass
cannot resolve, cooldown active	suppressed	pass
cooldown expired	restart	pass
unwritable lock	restart + warning	pass
unwritable state	restart + WARNING	pass
empty ADSB_UPSTREAMS	falls back, restart	pass

Then end to end on jn1, by breaking resolution inside the live container:

healthy run        -> no restart (correct)
resolution broken  -> "cannot resolve any of [adsb.retina.fm api.adsb.lol]; restarting it"
                      container restarted, resolves ok, X-Data-Source: adsb.retina.fm
immediate repeat   -> "restarted 19s ago; waiting out the 900s cooldown"

Still only exercised by a reboot

The ExecStartPre boot path remains untested: neither node has passwordless sudo, so the drop-in could not be installed. The watchdog is now the backstop for exactly that uncertainty. Suggested gate after the first image carrying this: reboot a node and check docker exec tar1090 getent hosts adsb.retina.fm.

Fourth node checked: owl-debb (2026-10-06)

Different site, different resolver (192.168.0.50, search ht.home), so this is not network specific.

It is currently healthy, and that is not reassuring. All the preconditions are present:

tar1090 network mode:  retina-node_blah2   (bridge, same as the others)
docker.service:        TimeoutStartSec=infinity, no drop-ins
docker active:         2026-10-05 18:44:33
resolv.conf written:   2026-10-05 18:44:35   <- 2s AFTER dockerd

The race happened here too. It just did not land on this container, because the container was not started at boot:

tar1090 Created=2026-10-06T07:10:13   StartedAt=2026-10-06T07:10:14

Created seven minutes before I looked, roughly 12 hours after boot. The reason is a second, unrelated fault: owl-debb has no SDR, so blah2 crash-loops ([RspDuo] MaxDevs=1023 NumDevs=0 / Error: No devices found, RestartCount=16, status=restarting). The RSPduo watchdog therefore sees stale blah2 API data every 5 minutes and runs a compose down/up, recreating the whole stack. Every recreate regenerates resolv.conf from the current, correct host file.

So owl-debb is immune to this bug as a side effect of being permanently broken in a different way. Fix its SDR and it becomes exposed like the rest.

It also confirms the provenance argument: when dockerd reads the host file late it gets the real local values (ExtServers: [host(192.168.0.50)], search ht.home), where jn1, jn2 and ffb all got ExtServers: [host(127.0.0.53)] with a *.gx.internal.cloudapp.net search domain. Same image, so that stale content is the build environment's own file, read before NetworkManager replaced it.

Fleet tally

node	preconditions	state at boot	why
jn1	yes	broken	lost the race by 13s
jn2	yes	broken	lost the race
ffb	yes	broken, masked	lost by 6s; local receiver hid it
owl-debb	yes	not exposed	stack recreated every 5 min by an unrelated fault

Four of four have the preconditions. Three of four were broken simultaneously. The fourth is only spared by a fault of its own.

Fifth node: owl-ded9 (2026-10-06). Broken, and the cleanest proof yet.

owl-ded9 and owl-debb share a site, a resolver (192.168.0.50), an OS image and an identical 2 s race margin. They ended up in opposite states, and the reason is exactly the mechanism this PR addresses.

                       owl-debb            owl-ded9
docker active          18:44:33            18:44:16
resolv.conf written    18:44:35  (+2s)     18:44:18  (+2s)
tar1090 StartedAt      07:10:14 (next day) 18:44:12
                       12h AFTER the write  6s BEFORE the write
container resolver     host(192.168.0.50)   host(127.0.0.53)
                       search ht.home       search ...cloudapp.net
resolves               ok                   FAIL
feed                   adsb.retina.fm       X-Data-Source: none

owl-ded9 had 5653 EAI_AGAIN entries and zero successful fetches since boot: the feed was dead for 12.5 hours and nothing reported it.

The only difference between the two is whether anything recreated the container after boot. owl-debb has no SDR, so blah2 crash-loops and the RSPduo watchdog runs a compose down/up every 5 minutes. owl-ded9 has a working SDR (blah2 running, RestartCount=2), so its stack stayed up and stayed broken.

That is the whole argument for this PR in one comparison: the outcome is decided by a 6-second accident of ordering, and a node that is otherwise working correctly is the one that stays broken.

Fixed by hand with docker restart tar1090: resolver now host(192.168.0.50), X-Data-Source: adsb.retina.fm.

Fleet tally

node	preconditions	gap	at boot	why
jn1	yes	+5s	broken	container started 13s before the write
jonathan-node-2	yes	-	broken	same
fairforest-b	yes	+6s	broken, masked	local receiver served source: local
owl-ded9	yes	+2s	broken	container started 6s before the write
owl-debb	yes	+2s	not exposed	stack recreated every 5 min by an unrelated fault

Five of five have the preconditions. Four of five were broken simultaneously. The one that was not is spared only by a fault of its own.

Unrelated observation, not chased

owl-ded9's RECEIVER_LAT/LON is 42.241528,-72.648361, which is exactly jn1's WGBY-TV transmitter coordinate. Its 0-aircraft reading after the fix is genuine (40 nm radius at 03:30 local; 120 nm gives 10 and 250 nm gives 42 at the same point), so the feed is fine. But a receiver sited on a transmitter is worth a separate look, and may bear on the long-standing "owl-ded9 maps are noise" symptom.

Boot path now tested on hardware (owl-ded9, 2026-10-06)

Installed the drop-in on owl-ded9 by hand and rebooted. Watchdog deliberately NOT installed, so this tests the prevention layer in isolation rather than having the watchdog mask a failure.

The reboot alone was inconclusive, and the timeline says why

07:33:03  NetworkManager starts
07:33:06  /etc/resolv.conf written            <- early on this boot
07:33:09  NetworkManager-wait-online finishes
07:33:09  docker.service begins; wait script runs, finds a good file, exits
07:33:13  tar1090 starts (7s AFTER the write)
07:33:17  docker.service ready

Outcome was healthy (ExtServers: [host(192.168.0.50)], resolves ok, 0 EAI_AGAIN, X-Data-Source: adsb.retina.fm, where the same node had 5653 EAI_AGAIN and a dead feed the previous boot). But NM wrote resolv.conf three seconds before docker.service even began, so the race never arose and the wait loop was a no-op. That boot would likely have succeeded without the drop-in too. The ordering of NM's write relative to wait-online varies between boots: late yesterday, early today.

So the condition was forced directly

With /etc/resolv.conf replaced by nameserver 127.0.0.53 (loopback only), then systemctl restart docker:

t+01s  restart in flight (deactivating)
t+13s  docker=activating
t+16s  docker=activating
t+19s  docker=activating
t+22s  docker=activating
t+25s  docker=activating     <- held for 12s while the file was unusable
       [real resolv.conf restored here]
t+28s  docker=active         <- released within 3s

The ExecStartPre held the Docker daemon in activating for 12+ seconds while the resolver configuration was unusable, and released within 3 seconds of it becoming usable. That is the mechanism that prevents the race, demonstrated on the node that actually lost the race yesterday.

Also confirmed on hardware

• systemd-analyze verify docker.service clean.
• systemd registers the entry with ignore_errors=yes, so the leading - is honoured.
• No boot cost: docker.service took 7.05s, against 12.6s measured on jn1 without the drop-in.
• Docker started normally with the drop-in present, repeatedly, across a reboot and two restarts.

Still untested

The Ansible role has not been applied by an image build (no ansible available here). Validated offline only: 7 tasks, standard copy/file modules, both src files present. The drop-in and script content installed by hand are byte-identical to what the role writes.

Node left in this state

owl-ded9 keeps the hand-installed drop-in and script (not the watchdog), so it is protected against its next reboot until an image supersedes it. Everything else restored: resolv.conf correct, 8 containers up, feed healthy.