Repository navigation
Description
Summary
The keep-alive saving credits a returning request when the keeper "decided" to ping its thread. It does not check that a ping row exists on that thread. The read-time gate that was added to catch this (kaSavedGate) checks the whole session, not the thread, so a ping on a sibling subagent satisfies it. On one hosted deployment, 91 of 279 credited rows have no ping on their own thread in the idle span. They carry $47.97 of the $249.51 credit.
What we found
Data (class 2, operational): union of two read-only snapshots of one hosted internal build (core ~v0.6.x), metadata columns only, 2026-10-28 to 2026-11-02. A "credited row" is a real request with keepalive_saved_usd > 0. "Own ping" is a ping row (keepalive = 1) with the same session and thread, between the thread's previous real request and the credited row.
| Credited rows (279, $249.51) | Rows | Credit |
|---|---|---|
| a ping on the row's own thread | 188 | $201.54 |
| no own ping, but a ping on another thread of the session | 75 | $37.03 |
| no ping anywhere in the session in the span | 16 | $10.95 |
- All 91 rows without an own ping have
keepalive_pings = 1. So the write path took them as "a ping happened". - 89 of the 91 are in one session (about 13 concurrent subagent threads), all after 2026-11-01. 85 of the 91 are subagent threads; 90 follow a
tool_usestop. - The gap from the previous request on the thread is 266 to 749 s (median about 380 s). 77 of 91 are over 330 s.
- Typically the returned request read all of its previous prefix (median read / previous prefix = 1.0), so the entry was alive past 5 minutes with no ping of ours on it. The credit takes the whole
cache_readbecauseKeepAliveRefreshedis 0 (no ping read anything), sokeepaliveSavedUSDfalls back ton = e.CacheRead. - Of the 91, the read-time gate (
kaSavedGate) still passes 62 ($30.78) because some other thread's ping row read the cache inside the window, and rejects 29 ($17.20). - 37 of the 91 sit on threads that had never been pinged before.
This is not the same count as the thread-key question: the thread id stored on the row is the one passed to arriveAt (proxy/proxy.go: threadID = th.ID, then e.ThreadID = c.thread, proxy/dashcapture.go), so the stored label and the lookup key agree at request time. I did not test relabelling directly; the 91 are the rows where the stored thread id has no ping, so a relabel after the fact is not excluded.
Why it happens (source at b4dfbfe; the cause of the missing ping rows is not proven)
Keeper.Sweepincrementse.pingsunder its lock before the ping is sent (keepalive/keeper.go,e.pings++before building thepingJob).Arrivereturns that count and it becomesKeepAlivePings(dash/event.go, field doc: "dispatch intent ... not the same as how many were sent").firecan return with no request and no row:k.sparerefusing the tenant's headroom (k.skipped.Add(1); return), a body that cannot be built, or a transport error.giveBackFailedPingundoes the count only for a transport error. The tenant limiter keeps a quarter of its budget back (reserveFrac,keepalive/ping.go), so a session with a dozen threads due at the same second is the likely case. I cannot see the skipped counter in the snapshot, so this is a hypothesis.keepaliveSavedUSD(dash/event.go) gates onKeepAlivePings > 0, which step 2 leaves true. The comment there records the same leak at a smaller size on an older corpus (7 of 105).- The read-time repair
kaSavedGate(dash/keepalive.go) joins ping rows ontenant_idandsession_idonly (kp.session_id = <alias>session_id), not onthread_id. Dashboard queries: use the thread id where they ask about one cache entry #427 lists the other dashboard queries that needthread_id; this one is not in its table.
Related, not the same: #853 (the credit counts a shared 20k head that siblings keep warm) is about how much of a real rescue is ours. This is about rows with no rescue by us at all. #427 is the general "key by thread" change; this is one more query for it.
Suggested fix
- In
kaSavedGate, addAND kp.thread_id = <alias>thread_id(rows from before feat(keepalive,dash): key cache attribution and keep-alive per thread #426 havethread_id = ''and keep today's result). - Count the intent only when it is sent: move
KeepAlivePingsto the number of ping rows actually booked, or havefirehand back the count (asgiveBackFailedPingdoes) on a limiter refusal. - Do not fall back to the whole
cache_readwhenKeepAliveRefreshed == 0; credit 0 or the head only. - Test: a session with two threads where only one is pinged; the other thread's return must not be credited.
Who it affects
Any deployment with parallel subagents and the keep-alive credit shown on the dashboard. It overstates the credit and the net keep-alive figure. It does not change what is billed. On this deployment about 19% of the credit by value ($47.97 of $249.51), concentrated in one session. One developer's session holds most of it, so do not read the percentage as a rate.
Limits
- One deployment, 5 days, one session carries 89 of the 91 rows.
- Whether the missing pings were refused by the limiter is not shown by the data.
- The entry survived without our ping in these rows; why (sibling prefix, longer provider lifetime, a ping I cannot see) is unknown.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsNew/ToDo
Activity