Decide the agentctl telemetry command: socket operation, snapshot publication, failure isolation #225

Activity

1. lbacik

   added

   wayfinder:grillingDecision requiring a live human discussionDecision requiring a live human discussion

   on Oct 14, 2026
2. lbacik

   added a parent issue on Oct 14, 2026

   • Agent fleet dashboard: map #222
3. lbacik

   marked this as blocked on Oct 14, 2026

   • Define the telemetry snapshot contract #224
4. lbacik

   marked this as blocking on Oct 14, 2026

   • Define the dashboard project: repo, layout, stack and delivery slices #228
5. lbacik

   changed the title [-]Decide the agent telemetry endpoint: transport, auth, networking, lifecycle[/-] [+]Decide the agentctl telemetry command: socket operation, snapshot publication, failure isolation[/+] on Oct 14, 2026
6. lbacik commented on Oct 14, 2026

   

   lbacik

   on Oct 14, 2026

   OwnerAuthor

   More actions

   Requirements from the resolved ticket "Decide instance discovery and the dashboard's Docker access" (#226): the dashboard runs docker exec --user agent <container-id> /usr/local/bin/agentctl telemetry --json without a TTY, with a 5 s dashboard-side timeout and at most one in-flight call per instance. agentctl telemetry should therefore bound its own runtime well inside 5 s and report failures machine-readably (exit code plus JSON error), so the dashboard can tell 'agent unreachable' from 'snapshot unavailable'.
7. lbacik commented on Oct 14, 2026

   

   lbacik

   on Oct 14, 2026

   OwnerAuthor

   More actions

   Requirements from the resolved ticket "Define the telemetry snapshot contract" (#224):

   • The reply is the schema-v1 snapshot defined there (schema_version, generated_at, sequence, identity, control, current_attempt, last_finished, recent_events, truncated), serialised at most 256 KiB.
   • control reuses build_status, but building the telemetry reply must perform no network I/O: recovery.publication comes from last known values, not the live GitHub lookups status does today (lifecycle.py:2260-2295).
   • Attempt figures are published by the attempt's event loop as an immutable view (including get_context_usage() calls at session start and after compaction); the socket handler only serialises the last published view plus the lifecycle's in-memory last_finished and the redacted event ring buffer.
   • agentctl telemetry failing on an older agent is how the dashboard detects version skew, so the "unknown operation" / unreachable error shape must be machine-readable.
8. lbacik

   self-assigned this

   on Oct 14, 2026
9. lbacik commented on Oct 14, 2026

   

   lbacik

   on Oct 14, 2026

   OwnerAuthor

   More actions

   Resolution

   Decided in a grilling session; the operator asked to take the recommended answer on every question. Facts checked against the code at 8b69ac3.

   Facts that shaped the answer

   • The control server is one daemon thread with a strictly serial accept loop (control_server.py:147-161); a slow handler blocks stop/handoff.
   • _serve has only try/finally, so an exception a handler does not list kills the listener for the rest of the process (filed separately: Control listener thread dies on an unexpected handler exception).
   • control_status (lifecycle.py:2473) holds _control_lock, which the main thread also holds across the GitHub claim (_gated_claim); under it, _recovery_snapshot_locked runs a git dirty check and _publication_progress_locked makes GitHub calls. So status as it is today can neither be the control section nor be polled.
   • TokenLedger has no lock and is mutated only on the attempt's asyncio loop (asyncio.run per dispatch, lifecycle.py:408); no code reads it from another thread today.
   • The socket listens only after provenance verification (__main__.py:140); agentctl imports are light; no CLI command has --json yet.

   1. Protocol

   • New socket operation {"op": "telemetry"} → {"ok": true, "telemetry": <schema-v1 snapshot>}.
   • New subcommand agentctl telemetry --json. On success it prints the bare snapshot to stdout and exits 0. In v1 there is no human format (operators keep using status), and status does not get --json.
   • On failure it prints {"ok": false, "error": {"code": "...", "message": "..."}} to stdout (human text may also go to stderr):

     code	meaning	exit
     agent_unreachable	socket missing, connection refused, or timeout	2 (existing convention)
     unsupported_operation	agent answered "Unknown operation"	1
     snapshot_failed	agent answered but could not build the snapshot, or the reply was unparsable/oversized	1

   • Version skew in practice: agentctl ships in the same image as the agent, so an older instance shows up as argparse rejecting the telemetry subcommand (exit 2, usage on stderr, empty stdout). Dashboard rule: stdout that is not a JSON object = "telemetry not supported by this agent version"; a JSON error object = use its code.

   2. Timeouts

   • agentctl telemetry uses a 3 s connect+read timeout (other commands keep 10 s), so it always answers inside the dashboard's 5 s exec timeout.
   • The client turns an oversized or unparsable reply into snapshot_failed (today send_request lets ValueError escape as a traceback).

   3. Reaching the attempt data: a telemetry module

   One new deep module owns everything the handler needs. Its interface is small.

   • Writer side, called on the attempt's event loop:
     • begin_attempt(seed): at claim, with ids, issue, issue title and configured ceilings.
     • publish_attempt(view): after each observation, i.e. every _check_limits call, a compaction boundary, a subagent start/finish, and the get_context_usage() readings.
     • finish_attempt(summary): at finalisation; updates last_finished.
   • Reader side, on the listener thread:
     • snapshot(control) -> dict: assembles identity, control, the last published attempt view, last_finished and recent events; applies sequence, generated_at and the 256 KiB trimming.
   • Views are immutable (frozen dataclasses or plain dicts built fresh). Publishing is a reference swap under a lock held only for the swap, so the handler never sees a half-updated ledger and never touches TokenLedger itself.
   • The executor gets the publisher as an injected callback, the same way it gets event_log today. After model execution ends the view is frozen and only finish_attempt changes state.
   • Tests exercise the module through snapshot(), feeding views via the writer methods. No socket is needed except for one protocol test in test_control_socket.py style.

   4. The control section without blocking

   • A network-free variant of control_status builds the same build_status payload without the git dirty check and without GitHub lookups. Publication progress and workspace-dirty state come from the last values the lifecycle computed itself; comment_confirmed/pull_request are null = not known yet, per the contract.
   • It takes _control_lock with a 1 s timeout. If the lock is busy (e.g. main thread mid-claim), the reply reuses the last control section built for telemetry. An additive field control_observed_at says when it was built; the dashboard shows the control part as stale beyond one poll interval.

   5. Failure isolation

   • The telemetry handler catches Exception, answers snapshot_failed, and emits a telemetry_failed event (at most one per minute, so a persistent bug does not flood the log). It never raises into the listener.
   • publish_attempt and its callers on the loop never raise into the attempt. A failure to build a view logs once and leaves the previous view in place.
   • Handler cost is O(snapshot size): serialising a ready snapshot of tens of KiB takes milliseconds.
   • Known limitation, accepted for v1: recovery retry still runs publication on the listener thread. While it runs, telemetry polls time out; the dashboard shows "unreachable (timeout)" with the last good snapshot and its age.
   • The generic listener hardening (catch-all per connection) is a prerequisite tracked as its own bug (Control listener thread dies on an unexpected handler exception #236). It applies to all operations, not just telemetry.

   6. Polling cost

   • The dashboard polls telemetry only; control is embedded, so there is one exec per instance per cycle.
   • Agent-side target: the operation answers in under 50 ms. End-to-end cost is roughly one docker exec plus a light Python start-up, about 200 ms.
   • Requirement passed to "Define the dashboard project": default poll interval 5 s, configurable, floor 2 s, with at most one exec in flight per instance (from Decide instance discovery and the dashboard's Docker access #226).

   7. Lifecycle

   Agent state	Telemetry reply
   Config load, provenance verification, wiring (socket not yet bound)	agent_unreachable
   Provenance verification fails	process exits → container "not running" (no exec)
   Startup reconciliation	snapshot; control.intake = "recovering", current_attempt: null, last_finished seeded from disk
   Idle, or intake stopped	snapshot; current_attempt: null
   Claim → model execution → finalisation	current_attempt present (seeded at claim, live during execution, frozen afterwards)
   Shutdown (server stopped)	agent_unreachable

   8. Opt-in

   Always on, no setting. The operation is read-only and lives on the existing 0600, container-local socket, the same exposure as status. A switch would only add a "telemetry disabled" state for the dashboard to handle.

   9. Records

   • ADR 0006 — Serve fleet telemetry through agentctl over the control socket, not an HTTP endpoint on branch docs/fleet-dashboard-glossary (commit 44235f0). It is hard to reverse, surprising without context (why does a dashboard need docker exec?), and the result of a real trade-off.
   • No new CONTEXT.md terms: the operation, module and error codes are implementation detail; Telemetry snapshot and Figure basis already cover the domain language.

   Agent-side changes surfaced (for the later split)

   1. Socket operation telemetry plus agentctl telemetry --json, with the error codes and exit codes above and a 3 s client timeout.
   2. telemetry module (writer/reader interface above) wired into the lifecycle (claim, finalisation) and the executor (injected publisher).
   3. Network-free, lock-timeout variant of control_status; lifecycle keeps last known publication progress and workspace-dirty state; control_observed_at.
   4. Broad exception isolation in the telemetry handler and in loop-side publishing; rate-limited telemetry_failed event.
   5. Prerequisite bug: Control listener thread dies on an unexpected handler exception.
10. lbacik

    closed this as completedon Oct 14, 2026