Report what a device fails to deliver to the controller - #154
Open
eriknordmark wants to merge 2 commits into
Open
Report what a device fails to deliver to the controller#154eriknordmark wants to merge 2 commits into
eriknordmark wants to merge 2 commits into
Conversation
eriknordmark
force-pushed
the
deferred-queue-telemetry
branch
from
August 14, 2026 13:21
f66191d to
6f02c45
Compare
A device reports most of its state only when that state changes, and nothing re-asserts it later, so a report the controller does not accept leaves its view of the affected object wrong until the object changes again. Nothing in the API says this is happening: the existing counters describe whether the device could reach the controller, and a request the controller answered with an error counts as a success there, because for connectivity purposes it is one. Add what only the device knows. At device level, deferredQueueMetric says how many messages are waiting, when the oldest of them was produced - the difference from the current time being how stale the controller's view may be - and how many were given up on, split into those the controller rejected and those a later periodic publication supersedes, with the time of the most recent one. Per URL, two counters separate answers which may not repeat, such as a server error or a rate limit, from outright rejections, which is the distinction that decides whether the device keeps a message or discards it. The addition is wire-compatible: new field numbers only, in deviceMetric and urlcloudMetric. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
No functional changes; generated code and assets only. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
eriknordmark
force-pushed
the
deferred-queue-telemetry
branch
from
August 14, 2026 15:56
6f02c45 to
09af143
Compare
eriknordmark
marked this pull request as ready for review
August 14, 2026 16:10
|
|
||
| int64 retriableErrCount = 10; // Answered with a status which may not repeat, such as a server error or a rate limit; the message is kept and offered again | ||
| int64 rejectedErrCount = 11; // Answered with a rejection of the request itself, so the message is given up on | ||
| int64 deliveredMsgCount = 12; // Answered with a success status, i.e. accepted; narrower than sentMsgCount, which counts any answer |
Contributor
There was a problem hiding this comment.
I think they can be unsigned.
Also I thought we're using snake_case in our proto files.
christoph-zededa
approved these changes
Aug 14, 2026
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A device reports application, volume, content tree, blob, network instance,
hardware and cluster state to the controller only when that state changes,
and nothing re-asserts it afterwards. So a report the controller does not accept
leaves its view of that object wrong until the object changes again - for a
steady-state application, indefinitely.
Nothing in the API says this is happening.
zedcloudMetric.failures/successdescribe whether the device could reach the controller, and EVEdeliberately counts a request the controller answered with an error as a success
there, because for connectivity purposes it is one. The result is that a device
sitting on undelivered reports, or one which has discarded them, looks identical
to a healthy device.
This is the telemetry half of lf-edge/eve#6302, where the device silently
discarded state updates on any 4xx or 5xx. lf-edge/eve#6304 fixes the discarding
and reports both conditions in the device log; these fields are what would let a
controller see them without reading device logs.
What this adds
Both additions are wire-compatible - new field numbers only.
deferredQueueMetric, at device level (deviceMetric.deferred_queue = 32).The queue is per device, not per port: a message is not tied to the port an
attempt to send it happened to use.
undeliveredMsgCountoldestUndeliveredMsgdroppedMsgCountrejectedMsgCountsupersededMsgCountlastDroppedMsgThe counter/timestamp pairing follows the existing
failures/lastFailureconvention in the same message. It matters here because these counters live in
memory and reset when pillar restarts, so a controller differencing samples sees
a reset as a decrease, and because metric messages are themselves best-effort,
so samples get missed. A timestamp answers "was anything lost recently" without
either problem.
Three counters per URL (
urlcloudMetric10, 11, 12):retriableErrCountfor answers which may not repeat - a server error, a rate limit, a request
timeout - and
rejectedErrCountfor outright rejections, which is thedistinction that decides whether the device keeps a message or discards it.
Plus
deliveredMsgCount, the messages the controller accepted: narrower thansentMsgCount, which counts every message the server answered at all, includingthe ones it refused. Both remain useful, since for connectivity purposes an
answer is a success whatever it says - so rather than redefine
sentMsgCount,which would break comparability across versions and which nim and diag read as
a connectivity signal, the accepted count is its own field.
An
APICapabilityvalue,API_CAPABILITY_DEFERRED_QUEUE_METRICS = 23, so acontroller can tell "nothing was dropped" from "this EVE does not report drops".
Report-side capabilities have precedent:
API_CAPABILITY_SMART_REPORT(20)covers
SmartAttr/smart_statusin info.proto andAPI_CAPABILITY_REPORT_TPM_EVENTLOG(21) covers the event log in attest.proto,so only the value is added here - the enum's own description is left to #153.
Deliberately not included
answered, so per-code counts from the device are largely redundant - except
for codes injected between the two, such as a load balancer's 503 or a proxy's
407, which the two class counters above still capture. Per port x per URL x
per code, re-sent every metrics interval, is a lot of repetition for that.
ZInfoTypesit carried). It is thenatural next question for an operator, but reporting it would mean importing
proto/infointoproto/metrics, and that direction is off the table: itinvites import loops, if not now then as either file grows. Duplicating the
enum inside
proto/metricswould drift, so the type is simply not reported.A controller that needs to know which object went stale can compare what it
last heard per object against
oldestUndeliveredMsg.This change adds no imports at all.
Downstream
Populating these fields is a follow-up in lf-edge/eve (pillar's
controllerconnqueue andAgentMetrics), after this lands and is vendored viamake bump-eve-api. No controller change is required to accept them.