Agent fleet dashboard: map #222

Destination

A decided specification for an agent fleet dashboard: (a) the list of changes simple-coding-agent needs so that a running instance can expose its state (tokens, context window, budget thresholds, phase, limits), and (b) the definition of a separate dashboard project (repo, stack, architecture, data contract, delivery order). The map ends when both are decided and ready to be split into implementation issues. Building the dashboard itself is not part of this map.

Notes

• Language: Polish with the user in conversation; everything persisted (tickets, comments, docs) in English.
• Skills: grilling tickets call grilling and domain-modeling; the agentctl telemetry ticket also codebase-design; the prototype ticket calls prototype; research tickets call research.
• Today agentctl status returns only control state (intake, active attempt phase/branch, stop plan, recovery, handoff, consecutive errors). Tokens, context size, budget and thresholds live only inside the agent process (TokenLedger, created per attempt, no lock) and partially in JSON Lines logs / $DATA_DIR/logs/<issue>/<timestamp>/.
• The control socket is container-local by design ("container = instance"); no port is published and no Docker socket is mounted. The control server handles connections serially on one thread. Consult docs/agents/operator-onboarding.md and CONTEXT.md.
• Settled in charting:
  • Topology: no HTTP endpoint. The agent gets a read-only agentctl telemetry --json command served over the existing control socket; the dashboard reads it with docker exec <container> agentctl telemetry --json, and v2 control uses the same docker exec agentctl ... path. "No published control port" stays intact. Consequence: the dashboard needs Docker exec already in v1. (Revised after charting: replaced the opt-in HTTP endpoint with token auth.)
  • v1 scope: control state + current-attempt telemetry (token categories, main/subagent context, peak context, budget, soft threshold, hard ceiling, turns, cache hit rate) + recent log events. Provider limits depend on research and live checks. History and log streaming are fog.
  • Delivery: pull (snapshot per docker exec), schema owned by simple-coding-agent and versioned, dashboard tolerates version skew (unknown fields ignored, missing = unavailable).
  • Scope: one Docker host, many agent containers discovered by label. Remote Docker contexts later.
  • Dashboard: separate repo, pure Python (FastAPI + HTMX/SSE to the browser) for v1, read-only in v1; control UI (stop, stop --after, resume, handoff, next issue) in v2; recovery and errors reset stay CLI-only.
  • New terms (Dashboard, Managed instance, Telemetry snapshot) go to CONTEXT.md as they are resolved; the rejected HTTP alternative is recorded in ADR 0006.

Decisions so far

• Research: which limit and context-window data the model provider and SDK expose: Meta usage, compaction events and model_usage are measured; context window, subagent context and cost are estimates or CLI guesses (treat window as configuration); SDK RateLimitEvent is unusable on Meta; provider rate-limit headroom is unavailable in v1 unless read from LiteLLM externally. Details on branch research/provider-limits-and-context-window.
• Decide instance discovery and the dashboard's Docker access: image label discovery re-listed each poll; instance = container name; argv-only docker exec --user agent <id> /usr/local/bin/agentctl telemetry --json (5 s, one in flight); v1 runs as a host process on 127.0.0.1 with the operator's own Docker access, behind a gateway exposing only list+telemetry.
• Prototype the dashboard UI: fleet overview and instance detail: variant A (dense fleet table + detail drawer) with a status-headline column and attention-first sort; drawer order current attempt → control → recent events; USD and charts cut from v1, provenance markers kept; required snapshot fields listed for the contract ticket. Prototype on branch prototype/fleet-dashboard-ui.
• Run the live checks from the provider-limits research: CLI assumes a 200,000 window for the Meta model unless CLAUDE_CODE_MAX_CONTEXT_TOKENS is set (Meta verified at ~1,048,576; filed separately as needs-triage); compact_metadata confirmed; get_context_usage() costs 5 count_tokens requests per session, 2 per compaction, else 0, so the snapshot may call it; gateway forwards no rate-limit headers, so headroom stays unavailable in v1. Results on branch research/provider-limits-and-context-window.
• Define the telemetry snapshot contract: schema v1 = identity (repo, agent/SDK/CLI versions, process start) + control (the status payload, no network I/O) + current_attempt figures each tagged with a figure basis (measured/estimated/configured/sdk_assumed/unavailable) + last_finished (seeded from disk) + 50 redacted recent events; integer major version, additive changes free, missing = unavailable, null section = not applicable; capped at 256 KiB. Glossary: Telemetry snapshot, Figure basis.
• Decide the agentctl telemetry command: socket operation, snapshot publication, failure isolation: {"op":"telemetry"} + agentctl telemetry --json (bare snapshot on success; JSON error with code agent_unreachable/unsupported_operation/snapshot_failed on stdout; 3 s client timeout); a deep telemetry module receives immutable views published from the attempt loop and assembles the snapshot; control via a network-free variant of status with a 1 s lock timeout, falling back to the last built section; handler and publishing isolated from failures; always on; dashboard polls telemetry only every 5 s. ADR 0006 records the rejected HTTP endpoint; listener-crash bug filed as Control listener thread dies on an unexpected handler exception #236.
• Define the dashboard project: repo, layout, stack and delivery slices: lbacik/agent-dashboard (MIT, uv/hatchling, deep modules gateway/collector/fleet/snapshot/presentation/web); asyncio poll cycle (5 s, one exec per instance, global cap 4, single worker), last good snapshot + last poll outcome in memory, no history; loopback-only bind with Host allow-list, SSH tunnel for remote viewing; uv tool install + systemd --user (ADR 0001 in agent-dashboard); SSE pushes the whole table body, drawer refreshes on it; view models are the swap seam, no JSON API in v1; tests on a fake gateway with contract example snapshots owned by the agent repo; slices S1 skeleton+discovery → S2 telemetry table → S3 drawer → S4 v0.1.0. The resolution holds the consolidated, ordered list of agent-side changes for the split.

Not yet specified

Empty: the v1 way is clear. The remaining patches were ruled out of scope once v1 was fully decided (see below).

Out of scope

• Building the dashboard (separate repo, own effort after this map).
• Splitting the decided agent-side changes into ready-for-agent implementation issues: the map ends when they are ready to be split; the split itself follows the map. (The dashboard's first delivery slices are decided inside the map, in the dashboard-project ticket.)
• Kubernetes or other orchestrators.
• Scaling a single agent beyond one instance per container.
• Control from the UI (v2): command set, confirmations, who may issue commands, audit trail, binding each command to the observed container and target repository (recommended: an agent-side expected-target-repository guard in agentctl), CSRF/Origin protection and authenticated non-loopback access. Out of scope because the destination is the v1 read-only specification; a future map.
• History and trends (persistence, retention, sparklines beyond the in-memory state). Out of scope: v1 keeps no history by decision; a future map.
• Live log streaming from the agent (beyond the recent events in the snapshot). Out of scope: v1 shows only the snapshot's recent events; a future map.
• Remote Docker contexts / multiple hosts. Out of scope: v1 covers one Docker host; the docker CLI gateway already honours contexts, so a future map can build on it.