Skip to content

Report what a device fails to deliver to the controller - #154

Open
eriknordmark wants to merge 2 commits into
lf-edge:mainfrom
eriknordmark:deferred-queue-telemetry
Open

Report what a device fails to deliver to the controller#154
eriknordmark wants to merge 2 commits into
lf-edge:mainfrom
eriknordmark:deferred-queue-telemetry

Conversation

@eriknordmark

@eriknordmark eriknordmark commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Problem

A device reports application, volume, content tree, blob, network instance,
hardware and cluster state to the controller only when that state changes,
and nothing re-asserts it afterwards. So a report the controller does not accept
leaves its view of that object wrong until the object changes again - for a
steady-state application, indefinitely.

Nothing in the API says this is happening. zedcloudMetric.failures /
success describe whether the device could reach the controller, and EVE
deliberately counts a request the controller answered with an error as a success
there, because for connectivity purposes it is one. The result is that a device
sitting on undelivered reports, or one which has discarded them, looks identical
to a healthy device.

This is the telemetry half of lf-edge/eve#6302, where the device silently
discarded state updates on any 4xx or 5xx. lf-edge/eve#6304 fixes the discarding
and reports both conditions in the device log; these fields are what would let a
controller see them without reading device logs.

What this adds

Both additions are wire-compatible - new field numbers only.

deferredQueueMetric, at device level (deviceMetric.deferred_queue = 32).
The queue is per device, not per port: a message is not tied to the port an
attempt to send it happened to use.

field meaning
undeliveredMsgCount messages waiting to be delivered
oldestUndeliveredMsg when the oldest was produced; the difference from now is how stale the controller's view may be
droppedMsgCount messages given up on since boot
rejectedMsgCount of those, ones the controller rejected outright
supersededMsgCount of those, ones a later periodic publication supersedes
lastDroppedMsg when the most recent one was given up on

The counter/timestamp pairing follows the existing failures/lastFailure
convention in the same message. It matters here because these counters live in
memory and reset when pillar restarts, so a controller differencing samples sees
a reset as a decrease, and because metric messages are themselves best-effort,
so samples get missed. A timestamp answers "was anything lost recently" without
either problem.

Three counters per URL (urlcloudMetric 10, 11, 12): retriableErrCount
for answers which may not repeat - a server error, a rate limit, a request
timeout - and rejectedErrCount for outright rejections, which is the
distinction that decides whether the device keeps a message or discards it.
Plus deliveredMsgCount, the messages the controller accepted: narrower than
sentMsgCount, which counts every message the server answered at all, including
the ones it refused. Both remain useful, since for connectivity purposes an
answer is a success whatever it says - so rather than redefine sentMsgCount,
which would break comparability across versions and which nim and diag read as
a connectivity signal, the accepted count is its own field.

An APICapability value, API_CAPABILITY_DEFERRED_QUEUE_METRICS = 23, so a
controller can tell "nothing was dropped" from "this EVE does not report drops".
Report-side capabilities have precedent: API_CAPABILITY_SMART_REPORT (20)
covers SmartAttr / smart_status in info.proto and
API_CAPABILITY_REPORT_TPM_EVENTLOG (21) covers the event log in attest.proto,
so only the value is added here - the enum's own description is left to #153.

Deliberately not included

  • A per-status-code histogram. The controller already knows what it
    answered, so per-code counts from the device are largely redundant - except
    for codes injected between the two, such as a load balancer's 503 or a proxy's
    407, which the two class counters above still capture. Per port x per URL x
    per code, re-sent every metrics interval, is a lot of repetition for that.
  • The type of the dropped message (which ZInfoTypes it carried). It is the
    natural next question for an operator, but reporting it would mean importing
    proto/info into proto/metrics, and that direction is off the table: it
    invites import loops, if not now then as either file grows. Duplicating the
    enum inside proto/metrics would drift, so the type is simply not reported.
    A controller that needs to know which object went stale can compare what it
    last heard per object against oldestUndeliveredMsg.

This change adds no imports at all.

Downstream

Populating these fields is a follow-up in lf-edge/eve (pillar's
controllerconn queue and AgentMetrics), after this lands and is vendored via
make bump-eve-api. No controller change is required to accept them.

@eriknordmark
eriknordmark force-pushed the deferred-queue-telemetry branch from f66191d to 6f02c45 Compare August 14, 2026 13:21
eriknordmark and others added 2 commits August 14, 2026 17:47
A device reports most of its state only when that state changes, and nothing
re-asserts it later, so a report the controller does not accept leaves its
view of the affected object wrong until the object changes again. Nothing in
the API says this is happening: the existing counters describe whether the
device could reach the controller, and a request the controller answered with
an error counts as a success there, because for connectivity purposes it is
one.

Add what only the device knows. At device level, deferredQueueMetric says how
many messages are waiting, when the oldest of them was produced - the
difference from the current time being how stale the controller's view may be
- and how many were given up on, split into those the controller rejected and
those a later periodic publication supersedes, with the time of the most
recent one. Per URL, two counters separate answers which may not repeat, such
as a server error or a rate limit, from outright rejections, which is the
distinction that decides whether the device keeps a message or discards it.

The addition is wire-compatible: new field numbers only, in deviceMetric and
urlcloudMetric.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
No functional changes; generated code and assets only.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: eriknordmark <erik@zededa.com>
@eriknordmark
eriknordmark force-pushed the deferred-queue-telemetry branch from 6f02c45 to 09af143 Compare August 14, 2026 15:56
@eriknordmark
eriknordmark marked this pull request as ready for review August 14, 2026 16:10

int64 retriableErrCount = 10; // Answered with a status which may not repeat, such as a server error or a rate limit; the message is kept and offered again
int64 rejectedErrCount = 11; // Answered with a rejection of the request itself, so the message is given up on
int64 deliveredMsgCount = 12; // Answered with a success status, i.e. accepted; narrower than sentMsgCount, which counts any answer

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they can be unsigned.

Also I thought we're using snake_case in our proto files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants