Onboarded EVE never POSTs EdgeNodeCerts; attestation never completes (regression on master, ~2026-05-06)

[eriknordmark]

Describe the bug

After onboarding completes, EVE never POSTs its EdgeNodeCerts (POST /api/v2/edgedevice/id/<uuid>/certs) to the controller. As a consequence, controller-side attestation never succeeds: every /api/v2/edgedevice/id/<uuid>/attest POST is rejected because the controller has no device certificates on file.

EVE keeps retrying both GET /api/v2/edgedevice/certs (pulling controller certs) and the attest POST in a tight loop. After a few hours, info publishing also degrades — triggerPublishDevInfoToDest reports Failed to send on PublishDeviceInfo, watchdog flags the zedagentdevinfo / zedagentlocaldevinfo / zedagenthwinfo / zedagentobjectinfo / zedagent-localappinfo files, and /info POSTs to the controller stop entirely.

Recent release

This is a regression on master. It is not observed against an earlier master snapshot from 2026-04-22 (commit 53ac6c5498c9, baseline of the eve-coverage-merge run series). It is observed against master as of 2026-05-09 (commit a17abf269c66, plus PRs #5901, #5914, #5915, #5922, #5923, #5925 merged for unrelated test-coverage work — none of those PRs touch controllerconn/).

Suspected first-bad master commits, all dated 2026-05-05 → 2026-05-07 and all touching pkg/pillar/controllerconn/:

Commit	Date	Subject
894e9c3f0	2026-05-06	Make sure SenderStatusCertMiss is not overwritten
e2885b201	2026-05-05	zedagent: unit tests for bootstrap and global config loading (refactors VerifyLeavesCertChain to extract VerifyLeavesCertChainWithRootPEM)
b61b11f71	2026-05-07	controllerconn/tls: handle SystemCertPool failure

Of these 894e9c3f0 is the most plausible culprit because it changes the return-status semantics of verifyAuthContainerHeader on the dual-cert (primary + backup) fallback path, which is the path that gates whether EVE treats a controller response as "cert missing" (triggering a cert fetch) vs "cert OK". Its commit message is "This is needed to fetch new controller certificates." — which fits the symptom of EVE looping on GET /edgedevice/certs instead of progressing to the EdgeNodeCerts POST.

b61b11f71 is benign (handles x509.SystemCertPool() returning (nil, err) on minimal Linux). e2885b201 is a clean refactor.

How to reproduce

• EVE-OS version: built from lf-edge/eve master at a17abf269c66 (HEAD of the all-PRs branch in ~/lf-edge/eve-coverage-allprs/eve), COVER=y rootfs, kvm-amd64.
• EVE-OS hardware architecture: amd64.
• Controller: locally-built lfedge/adam:0.0.65-pr152 (server: populate EdgeDevConfig.controllercert_confighash adam#152 merged on top of master).
• Eden: locally-built from eden master + a few test PRs.

Steps:

1. Build EVE rootfs from a master snapshot containing 894e9c3f0.
2. Standard eden setup → start → onboard.
3. Watch docker logs eden_adam and the EVE pillar log.

Expected behavior

After onboarding, EVE POSTs its EdgeNodeCerts to the controller. The controller-side attestation flow then succeeds (/attest returns OK, quoteValidate passes). Info publishing continues at the configured interval indefinitely.

Error message

Adam (controller) access log shows:

2026/05/10 07:48:58  EVE requested /api/v2/edgedevice/certs
2026/05/10 07:48:58  EVE requested /api/v2/edgedevice/register
2026/05/10 07:49:04  EVE requested /api/v2/edgedevice/id/<uuid>/info
2026/05/10 07:49:04  EVE requested /api/v2/edgedevice/id/<uuid>/config
2026/05/10 07:49:05  EVE requested /api/v2/edgedevice/id/<uuid>/attest
2026/05/10 07:49:09  deviceCertsGet: no attestation certificates found for device <uuid>
2026/05/10 07:49:13  quoteValidate failed: Z_ATTEST_RESPONSE_CODE_NO_CERT_FOUND, cannot get device certs: no attestation certificates found for device <uuid>

…and 49 further attestation attempts, all with the same NO_CERT_FOUND result. Across 4+ hours of log there are zero POST /api/v2/edgedevice/id/<uuid>/certs (EdgeNodeCerts upload) entries, confirming EVE never sent the device certs.

GET /api/v2/edgedevice/certs (EVE pulling controller certs) appears 30 times in the same window — repeated retries consistent with a persistent SenderStatusCertMiss status from the new code path.

EVE pillar log shows the downstream effect on info publishing:

2026-05-10T10:09:05Z zedagent.go:2002 triggerPublishDevInfoToDest "Failed to send on PublishDeviceInfo"
2026-05-10T10:09:05Z zedagent.go:2002 triggerPublishDevInfoToDest "Failed to send on PublishDeviceInfo"
2026-05-10T10:09:05Z zedagent.go:2002 triggerPublishDevInfoToDest "Failed to send on PublishDeviceInfo"
2026-05-10T10:09:31Z watchdog: file: /run/zedagent-localappinfo.touch:500
2026-05-10T10:09:31Z watchdog: file: /run/zedagent-localdevinfo.touch:500
2026-05-10T10:09:31Z watchdog: file: /run/zedagenthwinfo.touch:500
2026-05-10T10:09:31Z watchdog: file: /run/zedagentobjectinfo.touch:500
2026-05-10T10:09:31Z watchdog: file: /run/zedagentdevinfo.touch:500

Hourly /info POST count to controller (from adam log):

Hour	/info POSTs
07:00	34
08:00	342
09:00	324
10:00	111
11:00	5 (last at 11:43:57)
12:00+	0

Additional context

This was originally filed as eden issue lf-edge/eden#1166 ("lim.test TestInfo times out matching dinfo.systemAdapter.status.ports.ifname") because the visible failure was a 5-minute eden test timeout. Subsequent investigation traced the root cause to EVE never publishing the field — and from there to the broader attestation / info-publish breakage above. The eden side is innocent: byte-identical eden binaries (including lim.test, einfo, pkg/utils/lookup.go, pkg/controller/loaders) successfully passed the same info_test query in 0.16 s against the April 22 baseline EVE.

Suggested first-cause-to-chase:

1. Read 894e9c3f0 and follow how the preserved SenderStatusCertMiss propagates through controllerconn callers — specifically whether any caller now treats (SenderStatusCertMiss, "") as a permanent terminal state instead of a "retry-after-cert-refresh" hint.
2. Get a goroutine dump from a stuck EVE (pkill -SIGQUIT zedbox) to identify what exactly the EdgeNodeCerts publish goroutine is parked on. Likely candidates: tpmmgr waiting on a TPM handle that depends on attestation, or zedagent deferred-event queue blocked behind the failed verify response.
3. Check whether the same regression reproduces against an unmodified master HEAD (i.e. without the test-coverage PRs merged), to confirm the breakage came in via master and not via this workspace's PR set.

Workspaces and artifacts available locally (paths inside this repo's runs of the smoke + eden e2e suites):

• All-PRs (failing): ~/lf-edge/eve-coverage-allprs/eve (branch coverage-allprs).
• Baseline (working): ~/lf-edge/eve-coverage-merge/eve (branch eve-coverage-merge), tip 53ac6c5498c9.
• Adam access log: docker logs eden_adam while either container is running.
• Stage F unit-coverage report: ~/notes/eve-coverage-allprs-stage-F-unit-tests.md.
• Stage H combined-coverage report: ~/notes/eve-coverage-allprs-stage-H-combined.md.
• Original (incorrect) eden-side hypothesis + revised debug plan: ~/notes/eden-1166-debug-plan.md.

[eriknordmark]

Retracting this report. Subsequent investigation found that the EVE under test was running stock lfedge/eve:12.1.0-kvm-amd64, not the COVER=y master+test-PRs build I claimed in the original report. The 12.1.0 image was installed onto IMGB and activated at 10:09 (a few minutes after onboard) due to a stale baseos.base_os_version=12.1.0-kvm-amd64 config that survived in the persistent Docker volume eden_adam_volume from someone's prior eden session on the same machine. (My setup recipe ran docker rm -f eden_adam but not docker volume rm eden_adam_volume, so the leftover controller config was inherited.)

Consequences for this report:

• The "first-bad master commit" suspicion against controllerconn/{authen.go, tls.go} (commits 894e9c3f0, e2885b201, b61b11f71) is not supported — the running binary was 12.1.0, not master, so the diff between baseline and our merged workspace is irrelevant to the actual symptom.
• adam at server: populate EdgeDevConfig.controllercert_confighash adam#152 is also an unlikely cause as @eriknordmark noted — that PR only changes how the controller-cert config-hash is tracked, not the attestation cert handling.
• EdgeNodeCerts were never POSTed to adam during either the COVER period (07:48–10:09) or the 12.1.0 period (10:09 onwards). So if a real bug exists, it's either older than the May 5–7 master commits, or environmental (TPM/firmware/eden-setup). I don't have a clean repro yet.

Closing as not-an-actual-bug-report. If the upcoming clean rerun (against a wiped adam volume, with a verification step that EVE booted with the COVER tag and stayed on it) still shows EdgeNodeCerts never posted, I'll file a fresh issue with proper repro and a goroutine dump of the relevant pillar process.