fix(agents): Delegate token usage was silently zero on every live turn

test_delegate_live_token_usage_populated failed live (crashed=False, real
text/tool output, token_usage=None): _parse_usage was never even reached
with a dict, because it was looking in the wrong place entirely.

Confirmed by reading the installed @uipath/delegate-sdk's bundled
dist/index.mjs directly: sendMessage() always resolves to a plain string,
never an object, and no event forwarded through agent.onEvent() ever
carries a `usage` field. The SDK's per-turn token accounting lives only in
its internal store, reachable through DelegateAgent.getLastTurnUsage()
(and the session id through getSessionId()) -- called nowhere in this
agent before now.

- delegate_host.mjs: call both getters right after sendMessage() resolves
  and attach them to the send_ok message (usage, sessionId).
- delegate_agent.py: _handle_send_ok reads them off send_ok's own top
  level instead of a nonexistent nested result dict. _parse_usage now
  reads the getter's real (confirmed, not guessed) shape --
  promptTokens/completionTokens/promptTokensCached/cacheCreationTokens --
  computing uncached_input_tokens = promptTokens - promptTokensCached
  (promptTokens is the OpenAI-style total, promptTokensCached the
  cache-read subset).
- Updated tests to the real send_ok shape and confirmed bucket names.
- .claude/notes/agents.md § Delegate agent records the finding so it
  isn't re-litigated as a guess next time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: akshaylive

.claude/notes/agents.md

@@ -809,6 +809,26 @@ own env-var-driven bootstrapping, not the `DelegateAgent` class itself, which we
 against a real backend). `tests/test_delegate_agent.py::TestStart::test_org_and_tenant_slug_forwarded_into_auth`
 pins it. Passing `DELEGATE_BACKEND_URL` directly instead of `DELEGATE_ENV` skips this whole path.
 
+**Token usage comes from two getters called after `sendMessage()` resolves, not from any event or the
+resolved value itself — confirmed by reading the installed SDK's bundle, not by guessing.** A first
+pass guessed at a flat `usage` dict carried on the resolved `sendMessage()` value or on a forwarded
+event, tried several plausible snake_case/camelCase bucket-name spellings, and silently returned zero
+tokens every turn against a real backend (`tests/test_delegate_agent_live.py::test_delegate_live_token_usage_populated`
+failed live: `record.crashed is False`, real text/tool output, `token_usage=None`). Reading
+`node_modules/@uipath/delegate-sdk/dist/index.mjs` directly settled it: `sendMessage()` always resolves
+to a plain string (never an object), and no event this host forwards via `agent.onEvent()` ever carries
+a `usage` field — the SDK's own internal event vocabulary has no `"usage"` member (that name IS used
+internally, but only inside the SDK's own Zustand store reducer that updates its "Token usage" UI
+panel, never re-emitted through the public `onEvent` bus). The only way to reach it is
+`DelegateAgent.getLastTurnUsage(sessionId?)` (defaults to the just-used session), so
+`delegate_host.mjs`'s `handleSend` calls it — and `getSessionId()` — right after `sendMessage()`
+resolves, and attaches both to the `send_ok` message itself. That getter's shape, read straight off the
+SDK's own `setUsage` store action, is `{promptTokens, completionTokens, promptTokensCached,
+cacheCreationTokens, turnTokenUnits, contextBreakdown}` — `promptTokens` is the TOTAL input token count
+(OpenAI-style, cached + uncached), `promptTokensCached` the cache-READ subset of it, so
+`_parse_usage` computes `uncached_input_tokens = promptTokens - promptTokensCached`. This is now
+CONFIRMED, not a guess, so `_parse_usage`'s docstring no longer marks it `# UNVERIFIED`.
+
 ### Delegate agent pricing
 
 Delegate-served models are keyed under the SDK's own hyphenated ids (e.g. `gpt-5-6-terra`) in

src/coder_eval/agents/delegate/delegate_host.mjs

@@ -21,7 +21,13 @@
 // stdout (host -> coder_eval):
 //   {"type": "init_ok"}
 //   {"type": "init_error", "message": str}
-//   {"type": "send_ok", "result": <sendMessage()'s resolved value, or null>}
+//   {"type": "send_ok", "result": <sendMessage()'s resolved string, or null>,
+//     "usage": <agent.getLastTurnUsage()'s result, or null>,
+//     "sessionId": <agent.getSessionId()'s result, or null>}
+//     -- usage/sessionId are read from these two getters AFTER sendMessage()
+//     resolves, not from the resolved value itself (a plain string) or from
+//     any forwarded event: CONFIRMED (reading the installed SDK's bundled
+//     source) that no event this host forwards ever carries a `usage` field.
 //   {"type": "send_error", "message": str}
 //   {"type": "destroy_error", "message": str}
 //   {"type": "protocol_error", "message": str} -- malformed/unknown stdin command; host keeps running
@@ -88,7 +94,18 @@ async function handleSend(msg) {
     return;
   }
   const result = await agent.sendMessage(msg.prompt, msg.sessionId || undefined);
-  writeLine({ type: "send_ok", result: result ?? null });
+  // CONFIRMED (reading @uipath/delegate-sdk@0.1.12's bundled dist/index.mjs):
+  // sendMessage() resolves to a plain string (the final response text) --
+  // never an object -- and no event this host forwards via agent.onEvent()
+  // ever carries a `usage` field. The SDK's own per-turn token accounting is
+  // internal state, reachable only through these two getters, called here
+  // once the turn (and the backend's own internal "usage" store update) has
+  // settled. Guarded with typeof, not called unconditionally: an older/newer
+  // SDK build that drops either method must degrade to "no usage this turn",
+  // never crash the host.
+  const usage = typeof agent.getLastTurnUsage === "function" ? agent.getLastTurnUsage() : null;
+  const sessionId = typeof agent.getSessionId === "function" ? agent.getSessionId() : null;
+  writeLine({ type: "send_ok", result: result ?? null, usage: usage ?? null, sessionId: sessionId ?? null });
 }
 
 async function handleDestroy() {

src/coder_eval/agents/delegate_agent.py

@@ -11,9 +11,9 @@
 clear ``AgentConfigError`` at ``start()`` (Node.js, ``npm install
 @uipath/delegate-sdk``, UiPath auth). Deliberate scope reductions versus the
 UiPath-internal sibling agent's more hardened adapter (no multi-generation
-transcript splitting, no WAF/SSE/session-conflict/stall-resend recovery,
-best-effort token-bucket field names marked ``# UNVERIFIED``) and every
-``# UNVERIFIED`` spot's rationale live in one place, not scattered:
+transcript splitting, no WAF/SSE/session-conflict/stall-resend recovery) and
+every remaining ``# UNVERIFIED`` spot's rationale live in one place, not
+scattered:
 
 Rationale: .claude/notes/agents.md § Delegate agent
 """
@@ -206,39 +206,46 @@ def _resolve_bundled_skills_path(plugins: list[dict[str, Any]] | None) -> str |
 
 
 def _parse_usage(raw: Any) -> TokenUsage | None:
-    """Best-effort parse of an event/result's ``usage`` payload into ``TokenUsage``.
-
-    UNVERIFIED: the SDK confirms an ``usage`` field exists on at least some
-    events, but not its internal bucket key names. Tries several plausible
-    spellings (snake_case, as the internal sibling agent's protocol used; and
-    camelCase, in case this layer differs) and falls back to 0 for anything it
-    cannot read, mirroring the project's "warn on drift, never raise" contract.
+    """Parse the SDK's per-turn usage payload (from ``getLastTurnUsage()``) into ``TokenUsage``.
+
+    CONFIRMED (reading the installed ``@uipath/delegate-sdk@0.1.12``'s bundled
+    ``dist/index.mjs``): no event this host forwards ever carries a ``usage``
+    field -- the SDK's per-turn token accounting lives only in its internal
+    store, reachable through ``DelegateAgent.getLastTurnUsage()``, which
+    ``delegate_host.mjs`` calls after ``sendMessage()`` resolves and attaches
+    to the ``send_ok`` message as ``usage``. That getter's shape, from the
+    SDK's own ``setUsage`` store action: ``{promptTokens, completionTokens,
+    promptTokensCached, cacheCreationTokens, turnTokenUnits,
+    contextBreakdown}``. ``promptTokens`` is the TOTAL input token count
+    (cached + uncached, OpenAI-style); ``promptTokensCached`` is the
+    cache-READ subset of it, so ``uncached = promptTokens - promptTokensCached``.
+    Falls back to 0 for anything absent (e.g. before the backend's first
+    internal usage report), mirroring the project's "warn on drift, never
+    raise" contract -- a future SDK release renaming one of these fields
+    degrades to zero tokens for that bucket, not a crash.
     """
     if not isinstance(raw, dict):
         return None
 
-    def _int(*keys: str) -> int:
-        for key in keys:
-            value = raw.get(key)
-            if isinstance(value, bool):
-                continue
-            if isinstance(value, int) and value >= 0:
-                return value
-        return 0
-
-    input_tokens = _int("input_tokens", "inputTokens", "uncached_input_tokens")
-    output_tokens = _int("output_tokens", "outputTokens")
-    cache_creation = _int("cache_creation_input_tokens", "cacheCreationInputTokens", "cache_write", "cacheWrite")
-    cache_read = _int("cache_read_input_tokens", "cacheReadInputTokens", "cache_read", "cacheRead")
-    if input_tokens == 0 and output_tokens == 0 and cache_creation == 0 and cache_read == 0:
+    def _int(key: str) -> int:
+        value = raw.get(key)
+        if isinstance(value, bool):
+            return 0
+        return value if isinstance(value, int) and value >= 0 else 0
+
+    prompt_total = _int("promptTokens")
+    prompt_cached = _int("promptTokensCached")
+    output_tokens = _int("completionTokens")
+    cache_creation = _int("cacheCreationTokens")
+    if prompt_total == 0 and output_tokens == 0 and prompt_cached == 0 and cache_creation == 0:
         if raw:
             logger.warning("delegate: usage payload matched none of the known bucket spellings: %r", sorted(raw))
         return None
     return TokenUsage(
-        uncached_input_tokens=input_tokens,
+        uncached_input_tokens=max(prompt_total - prompt_cached, 0),
         output_tokens=output_tokens,
         cache_creation_input_tokens=cache_creation,
-        cache_read_input_tokens=cache_read,
+        cache_read_input_tokens=prompt_cached,
     )
 
 
@@ -795,22 +802,21 @@ def _handle_tool_result(self, msg: dict[str, Any], state: _TurnState, emit: Call
         emit(ToolEndEvent(task_id=self.task_id, turn_id=state.turn_id, tool=telemetry, status=status))
 
     def _handle_send_ok(self, msg: dict[str, Any], state: _TurnState) -> None:
+        # CONFIRMED (reading the installed SDK's bundled source):
+        # sendMessage()'s resolved value is always a plain string, never an
+        # object -- `usage`/`sessionId` are NOT nested under it. delegate_host.mjs
+        # instead reads them off `getLastTurnUsage()`/`getSessionId()` after
+        # sendMessage() resolves and attaches them to this message's own
+        # top level (see delegate_host.mjs's wire-protocol header comment).
         result = msg.get("result")
         if isinstance(result, str):
             state.final_response = result
-        elif isinstance(result, dict):
-            response = result.get("response") or result.get("content")
-            if isinstance(response, str):
-                state.final_response = response
-            session_id = result.get("sessionId")
-            if isinstance(session_id, str) and session_id:
-                self._session_id = session_id
-            usage = _parse_usage(result.get("usage"))
-            if usage is not None:
-                state.usage = usage
-            model = result.get("model")
-            if isinstance(model, str) and model:
-                state.model_used = model
+        session_id = msg.get("sessionId")
+        if isinstance(session_id, str) and session_id:
+            self._session_id = session_id
+        usage = _parse_usage(msg.get("usage"))
+        if usage is not None:
+            state.usage = usage
 
     def _close_open_tools(self, state: _TurnState, emit: Callable[[StreamEvent], None]) -> None:
         for tool_id, telemetry in list(state.open_tools.items()):

tests/test_delegate_agent.py

@@ -251,7 +251,7 @@ async def test_happy_path_text_and_tool(self, patch_exec, tmp_path):
             _line({"type": "message", "content": "here is my answer"}),
             _line({"type": "tool_call", "toolName": "Bash", "toolId": "tool-1", "input": {"command": "ls"}}),
             _line({"type": "tool_result", "toolId": "tool-1", "output": "file.txt"}),
-            _line({"type": "send_ok", "result": {"response": "here is my answer", "sessionId": "sess-1"}}),
+            _line({"type": "send_ok", "result": "here is my answer", "sessionId": "sess-1"}),
         ]
         agent, proc = await _started_agent(patch_exec, events, tmp_path)
         record = await agent.communicate("do something")
@@ -422,45 +422,57 @@ def on_event(self, event: Any) -> None:
         assert record.crashed is False
 
     @pytest.mark.parametrize(
-        ("usage_payload", "expected_input", "expected_output"),
+        ("usage_payload", "expected_uncached_input", "expected_output", "expected_cache_read", "expected_cache_write"),
         [
-            ({"input_tokens": 10, "output_tokens": 5}, 10, 5),
-            ({"inputTokens": 10, "outputTokens": 5}, 10, 5),
-            ({"uncached_input_tokens": 10, "output_tokens": 5}, 10, 5),
+            ({"promptTokens": 10, "completionTokens": 5}, 10, 5, 0, 0),
+            ({"promptTokens": 10, "completionTokens": 5, "promptTokensCached": 4}, 6, 5, 4, 0),
+            (
+                {"promptTokens": 10, "completionTokens": 5, "promptTokensCached": 4, "cacheCreationTokens": 3},
+                6,
+                5,
+                4,
+                3,
+            ),
         ],
     )
     async def test_usage_bucket_spellings_populate_token_usage(
-        self, patch_exec, tmp_path, usage_payload, expected_input, expected_output
+        self,
+        patch_exec,
+        tmp_path,
+        usage_payload,
+        expected_uncached_input,
+        expected_output,
+        expected_cache_read,
+        expected_cache_write,
     ):
-        events = [_line({"type": "send_ok", "result": {"response": "done", "usage": usage_payload}})]
+        events = [_line({"type": "send_ok", "result": "done", "usage": usage_payload})]
         agent, _ = await _started_agent(patch_exec, events, tmp_path)
         record = await agent.communicate("hi")
         assert record.token_usage is not None
-        assert record.token_usage.uncached_input_tokens == expected_input
+        assert record.token_usage.uncached_input_tokens == expected_uncached_input
         assert record.token_usage.output_tokens == expected_output
+        assert record.token_usage.cache_read_input_tokens == expected_cache_read
+        assert record.token_usage.cache_creation_input_tokens == expected_cache_write
 
     async def test_usage_all_zero_is_none_and_warns(self, patch_exec, tmp_path, caplog):
-        events = [_line({"type": "send_ok", "result": {"response": "done", "usage": {"weird_bucket": 3}}})]
+        events = [_line({"type": "send_ok", "result": "done", "usage": {"weird_bucket": 3}})]
         agent, _ = await _started_agent(patch_exec, events, tmp_path)
         with caplog.at_level("WARNING"):
             record = await agent.communicate("hi")
         assert record.token_usage is None
         assert any("usage payload matched none" in r.message for r in caplog.records)
 
-    async def test_model_and_cost_wired_into_record(self, patch_exec, tmp_path):
+    async def test_cost_wired_into_record_for_configured_model(self, patch_exec, tmp_path):
         events = [
             _line(
                 {
                     "type": "send_ok",
-                    "result": {
-                        "response": "done",
-                        "model": "virtuoso-1-5",
-                        "usage": {"input_tokens": 1_000_000, "output_tokens": 1_000_000},
-                    },
+                    "result": "done",
+                    "usage": {"promptTokens": 1_000_000, "completionTokens": 1_000_000},
                 }
             )
         ]
-        agent, _ = await _started_agent(patch_exec, events, tmp_path)
+        agent, _ = await _started_agent(patch_exec, events, tmp_path, model="virtuoso-1-5")
         record = await agent.communicate("hi")
         assert record.model_used == "virtuoso-1-5"
         assert record.token_usage is not None