[HARDENING] Satellite pipeline release-day resilience — fletcher probe depth, cold-start race, backoff tuning

[accius]

Post-release tracking issue from the v26.6.0 deploy (2026-08-27). Prod is serving 25/40 satellites: all amateur birds resolved (AMSAT/SatNOGS fallbacks), but the 15 CelesTrak-only weather sats (GOES/METEOR/ELEKTRO/HIMAWARI/METOP/NOAA + SO-125) stay unresolved an hour after boot.

Diagnostics so far (all verified live)

1. Health lies green: /api/health reports fletcher ok because health.js probes the service's /health — which is fine — while the actual TLE fetches through it fail. Health should probably probe a relay path, not just liveness.
2. Env-name gap (fixed): health.js accepted FLETCHER_URL || TLE_FETCHER_URL but routing config read only the new name. Fixed in hotfix: honor legacy TLE_FETCHER_URL in fletcher routing #1164 (on main + Staging). Not today's trigger, but real.
3. Co-deploy cold-start race: a push to main redeploys openhamclock-production AND fletcher simultaneously. Prod's satellite state machine starts fetching ~5 min after boot against a cold fletcher whose first relayed requests exceed prod's 20s timeout → httpStatusCode 0 → skipped, and retry spacing is long.
4. Fletcher is slow even warm: 6–9s per relayed CelesTrak request via fletcher-production.up.railway.app, vs <1s hitting celestrak.org directly from a residential connection (HTTP 200, 30,167 bytes, 73 sats — upstream is healthy). Cold requests exceed 20s. Suspects: undici IPv6-first dialing on Railway egress (try pinning autoSelectFamily/family 4 in fletcher's fetch), or CelesTrak slow-walking fletcher's shared cloud egress IP.
5. Warm-cache priming of fletcher's 10-min cache with the exact weather-group payload did NOT lead prod to recover within 30 min — so either the state machine's weather-group retry cadence is longer than that after early failures, or requests are failing for a reason not visible externally.

Next steps

• Prod Railway logs for [Satellites] lines (needs Railway access): which state, what status codes, whether blockCelesTrakUntil engaged
• Fletcher logs for inbound/upstream latency split
• Consider: family-pinned fetch in fletcher, shorter warm-up (self-probe on boot), health probe through a relay path, faster weather-group retry after status-0
• Verify weather sats return after the 120-min CelesTrak backoff window would expire (~2h post-boot) even with no changes

Filed as wrap-up so the diagnostic chain isn't lost; prod is degraded (weather sats only), amateur satellite tracking unaffected.

73, K0CJH

[accius]

Resolved on its own, and the timing nails the mechanism: prod sat at 25/40 until roughly 120 minutes after boot, then jumped to 38/40 in one cycle and is now at 39/40. That is the CelesTrak backoff expiring — an early boot fetch through fletcher drew a 403/429, the 120-minute block engaged, and the first post-expiry retry against a warm fletcher filled the entire weather group at once. No lasting outage; amateur birds were never affected.

The one remaining miss is SO-125 (HADES-ICM), which is almost certainly just sparse TLE coverage for a young satellite rather than anything infrastructure-side — it can have its own look if it persists.

Re-scoping this issue to the hardening items so the next release-day doesn't repeat the two-hour degradation window:

• Health should probe fletcher through a relay path (e.g. a cheap CATNR fetch), not just /health liveness — it reported green through the whole incident
• Co-deploy cold-start: a main push restarts prod and fletcher together, and prod's first fetches race a cold relay; consider a short fletcher self-warm on boot or a gentler first-fetch retry before the 120-minute hammer drops
• Fletcher relay latency: 6–9s warm, >20s cold, vs <1s direct — worth pinning the dial family in fletcher's fetch (undici IPv6-first on Railway is the prime suspect) and checking whether CelesTrak throttles its egress IP
• Consider halving the first backoff after a cold boot (e.g. 15 min doubling to 120) so a single unlucky boot-time 429 doesn't cost two hours of weather sats

None of this is urgent now.

73, K0CJH

[accius]

Hardening pass is on Staging as ef52d769. What landed, mapped to the checklist:

Backoff tuning (the actual 2-hour mechanism): the state machine sets each sat's backoffCelestrakUntil optimistically before fetching, and never rolled it back on failure. Now a failed fetch (status 0, 5xx, or rate-limit) rolls the just-marked sats back to a 5-minute retry, so one cold-start timeout no longer costs the whole weather group the full window. A 404 keeps the long backoff since that sat genuinely isn't there (looking at you, SO-125). Policy is a pure function in server/utils/celestrakFailurePolicy.js with tests, and the three copy-pasted failure chains in the FETCH states now share one handler.

Cold-start race: a 403/429 within 15 minutes of boot now blocks CelesTrak for 10 minutes instead of 120. The co-deploy race looks like a ban but is not one; steady-state rate limits still get the full 120.

Fletcher latency (the 6-9s warm / >20s cold suspect): fletcher now sets dns.setDefaultResultOrder('ipv4first') (overridable via FLETCHER_DNS_ORDER=verbatim) and prewarms upstream DNS on boot. If CelesTrak is slow-walking the shared egress IP instead, this won't fix it, but the retry rollback above caps the blast radius either way.

Health probe depth: fletcher's /stats now reports lastUpstreamOkAt / lastUpstreamErrorAt / lastUpstreamStatus, and health.js probes /stats instead of bare /health, reporting degraded with detail like alive but relays failing (HTTP 403, last error 42s ago) when relays are actually failing. Relayed 404s count as ok so a sparse young sat doesn't flap the watchtower. Falls back to /health liveness for older fletcher builds.

Remaining on this issue: watch the next release-day deploy to confirm the window is gone, and pull Railway logs if it isn't.

73, K0CJH