deploy-test (ipv4-full): Let's Encrypt secondary-validation DNS lookup failure stalled the staging canary cert (not #510/#511)

[defangdevs]

What happened

Deploy test (CFN create + destroy) failed on master run
34545745840
(head 2e485a1, merge of #650 — the ttyd per-user-unix-socket change).
deploy-test (ipv4-full, true, false) failed at "Smoke test — wait for
authenticated terminal URL to serve ttyd HTML": curl got
OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
for the full 10-minute window — same client-side symptom as #510/#511, but a
different, new server-side cause.

This is not a regression from #650. #650 only touches the ttyd backend
(unix socket instead of a loopback port) and the Caddyfile's terminal
reverse_proxy target; it never touches caddyfile-header.caddy, the ACME
issuer acme block, or anything DNS/networking-related. The ipv6-outputs
leg of the same run passed. The 10 master Deploy test runs before this one
were all green.

Root cause (from the SSM journalctl -u caddy dump)

Caddy's certmagic issues a staging "canary" cert first (built-in
certmagic behavior, same mechanism as #510/#511) to confirm domain control
cheaply before spending a production attempt. This time the canary itself
never succeeded — every attempt (5 over the 10-minute window, backoff
120s/120s/300s/300s) failed the same way, entirely on Let's Encrypt's side:

{"level":"error","logger":"http.acme_client","msg":"challenge failed",
 "identifier":"32-185-166-161.sslip.io","challenge_type":"tls-alpn-01",
 "problem":{"type":"urn:ietf:params:acme:error:dns","title":"",
 "detail":"During secondary validation: DNS problem: networking error
 looking up A for 32-185-166-161.sslip.io; DNS problem: networking error
 looking up AAAA for 32-185-166-161.sslip.io"}}

Let's Encrypt's multi-perspective ("secondary validation") checks could not
resolve the box's own sslip.io hostname at all, from LE's own validation
vantage points — a DNS lookup failure on LE's side against a wildcard DNS
service, not anything Caddy, the box, or #650 did. Because the canary never
clears, the flow never reaches the production issuer, so Caddy has no cert
to serve for the whole window and every client TLS handshake gets an
internal-error alert.

This is a different signature from #510/#511 (which got past the canary and
into production issuance, then hit a chain-download 404). Filing this as a
new occurrence in the same deploy-test-flake family (see also #186, #190,
#373) rather than reopening #510/#511, since the failing stage and the
upstream error are both different.

Why file rather than fix

Same trade-off #511 already laid out, and it still applies:

• Do nothing — first failure of this shape, self-resolves on Let's Encrypt's
  or sslip.io's side, re-running (or the next push's own deploy-test) would
  very likely go green.
• Extend the smoke test's patience / add a mid-run retry.
• Configure a second issuer (e.g. ZeroSSL) as a certmagic fallback so a
  transient LE validation hiccup doesn't stall the whole box.

Filing so evidence accumulates here if this DNS-validation shape recurs,
rather than each occurrence re-deriving the diagnosis from scratch.

Repro / evidence

• Failing run: https://github.com/defangdevs/agent-box/actions/runs/34545745840
• Job: deploy-test (ipv4-full, true, false) (103097840168)
• Commit: 2e485a1 (merge of fix(web): put ttyd on a per-user unix socket, not a loopback port (#628) #650)
• ipv6-outputs leg on the same run passed.
• Last 10+ master Deploy test runs before this one: all green.

[defangdevs]

Update: this did not self-resolve — it recurred on the very next master run, and again on a manual rerun of that run's job. Three consecutive failures now, three different sslip.io hostnames, same exact error signature each time:

• Run 34545745840 (commit 2e485a1, the one this issue was filed against) — 32-185-166-161.sslip.io
• Run 34546252536 (commit 88a290b, next master push, unrelated diff — only touches caddy's group access on the file drop) — 52-35-254-225.sslip.io
• Manual gh run rerun 34546252536 --failed of the same job — 52-33-101-175.sslip.io

All three: tls-alpn-01 canary obtain fails every attempt for the full 10-minute window with

"During secondary validation: DNS problem: networking error looking up A for <ip>.sslip.io; DNS problem: networking error looking up AAAA for <ip>.sslip.io"

Ruled out sslip.io itself being down: queried each of the three hostnames above directly against 1.1.1.1, 8.8.8.8 and 9.9.9.9 from this box just now — all resolve the A record correctly and instantly (no AAAA, as expected). So this is specifically Let's Encrypt's own secondary/multi-perspective validation network failing to resolve sslip.io's dynamically-generated records from (some subset of) its own vantage points — a known, historically recurring LE-side issue (see e.g. community.letsencrypt.org threads on "secondary validation DNS problem" / "increase in renewal DNS failures" going back to 2020-2021), not something wrong with the box, Caddy config, or any of #650/#654/#653/#656/#659.

Given it's now 3-for-3 rather than a single blip, "just rerun and it'll go green" is looking less safe as the only mitigation. Worth reconsidering the two options already listed above (extend smoke-test patience / configure a fallback issuer such as ZeroSSL so a stuck LE canary doesn't stall the whole box) rather than assuming the next run self-resolves. Leaving this open rather than escalating further since the actual fix (if any) is a design call, not a quick patch.

[defangdevs]

Update: 4th consecutive occurrence, same signature. Run 34550974611 (commit 895c26d, PR #663 — only escapes commas in WebPasswordHash for aws-cli shorthand parsing, nothing DNS/Caddy/networking-related) — 32-188-192-177.sslip.io.

journalctl -u caddy via SSM shows the identical tls-alpn-01 canary failure on every attempt across the full window:

"During secondary validation: DNS problem: networking error looking up A for 32-188-192-177.sslip.io; DNS problem: networking error looking up AAAA for 32-188-192-177.sslip.io"

ipv6-outputs leg passed again, consistent with this being isolated to the ipv4-full smoke test's dependency on a cleared cert.

Not reopening a separate issue since the two mitigation options are already on the table here (extend smoke-test patience past 10 min, or configure a fallback ACME issuer like ZeroSSL so a stuck LE canary doesn't stall cert issuance) — just adding this as another data point given it's now 4-for-4 rather than a rare blip.

🤖 Generated with Claude Code

[defangdevs]

5th consecutive occurrence, identical signature. Run 34552069486 (commit 6cc67f7, master push) — 35-163-111-223.sslip.io.

journalctl -u caddy via the run's own SSM diagnostic step shows the same tls-alpn-01 failure on every attempt across the full 10-minute window:

"During secondary validation: DNS problem: networking error looking up A for 35-163-111-223.sslip.io; DNS problem: networking error looking up AAAA for 35-163-111-223.sslip.io" (ca=https://acme-staging-v02.api.letsencrypt.org/directory)

5-for-5 now across 5 different commits and 5 different sslip.io hostnames. At this point "rerun and it clears" is not a real mitigation — every push to master is hitting this. Not touching the design call (extend smoke-test patience vs. fallback ACME issuer) since that's still open above; just keeping the count current.