Skip to content

P0: measured token usage for the fleet -- Claude Code + agy, by project / model / lane / hour, with the top levers #4162

Open
Open
P0: measured token usage for the fleet -- Claude Code + agy, by project / model / lane / hour, with the top levers#4162
Labels
P1High priorityowner:la-0.71Owned by the 0.71 look-ahead slotsize:MK̂ 3-4 h
Milestone
0.73.0

Description

@noahgift
noahgift
opened on Sep 24, 2026

Operator P0 (2026-09-24, relayed verbatim by the release cop): "we need to get a handle on our token usage both for claude and agy...lets solve this issue ASAP along side the triage issue before .7".

A MEASURED token-usage accounting for the fleet: Claude Code (every session transcript's per-message usage) and agy (quorum lane envelopes, agy's conversation files and logs), by project, model, lane type and hour, plus the top levers with numbers. No estimates.

Deliverable: a re-runnable read-only script under scripts/ and the tables on this issue (measured results follow as a comment).

Activity

  1. noahgift
    added this to the 0.70.0 milestone on Sep 24, 2026
  2. noahgift
    added
    P0Critical priority
    on Sep 24, 2026

  3. noahgift commented on Sep 24, 2026

    @noahgift
    noahgift
    on Sep 24, 2026
    MemberAuthor

    Measured with scripts/token_usage_report.py on branch feat/token-usage-report @ 4ffb330. Host: lambda. gx10 is not covered.

    Token usage, MEASURED -- 09-23 06:58 to 09-24 06:58 UTC (24 h), host noah-Lambda-Vector

    python3 scripts/token_usage_report.py --hours 24 (read-only). Raw token counts as each tool recorded them; no price weighting, no estimates. Claude: 38,065 assistant messages (de-duplicated by message id). agy: 171 lane/probe envelopes in 42 quorum rounds found on disk, 538 agy conversations, 306 429 lines in agy's logs.

    A1. Claude totals

    input cache write cache read output all
    93,882 171,490,970 12,707,684,141 21,553,465 12,900,822,458

    A2. Claude by project

    project sessions messages cache read cache write output all tokens share
    aprender 60 22,486 7,946,589,543 94,081,386 14,165,208 8,054,889,543 62.4%
    infra 4 5,297 1,763,384,126 28,135,555 2,440,030 1,793,976,695 13.9%
    rmedia 4 3,241 1,444,365,761 17,055,430 2,251,833 1,463,679,938 11.3%
    ruchy 1 3,232 522,078,449 13,124,243 708,759 535,917,915 4.2%
    paiml-implement 3 978 422,899,517 4,977,236 730,376 428,610,377 3.3%
    apex 2 1,487 381,950,636 9,729,411 887,313 392,571,188 3.0%
    xpile 1 1,342 226,416,109 4,328,964 369,561 231,117,668 1.8%
    tmp 2 2 0 58,745 385 59,134 0.0%

    A3. Claude by model and lane (main-session turn vs subagent)

    model lane messages output all tokens share
    claude-opus-5-5 main 23,364 19,026,048 11,316,899,750 87.7%
    claude-sonnet-5 subagent 8,056 394,610 919,333,078 7.1%
    claude-opus-5-5 subagent 3,684 457,341 330,732,272 2.6%
    claude-sonnet-5 main 1,551 1,596,708 218,143,552 1.7%
    claude-haiku-4-5-20251001 subagent 946 1,760 61,557,451 0.5%
    claude-fable-5-1 subagent 290 29,819 27,931,856 0.2%
    claude-fable-5-1 main 73 38,700 23,669,073 0.2%
    claude-haiku-4-5-20251001 main 25 8,479 2,555,426 0.0%
    main 63 0 0 0.0%
    subagent 13 0 0 0.0%

    A4. Claude subagent lanes by agent type

    agent type lanes messages all tokens share of all Claude
    general-purpose 326 9,833 1,079,854,234 8.4%
    paiml-impl-worker 42 2,115 182,971,299 1.4%
    paiml-agy-delegate 43 882 65,754,785 0.5%
    Plan 6 104 7,580,355 0.1%
    Explore 2 55 3,393,984 0.0%

    A5. Top 10 Claude sessions

    session project models main turns subagent msgs span h all tokens cache read / main turn share
    78401db7 aprender ,claude-opus-5-5 2,196 27 22.8 1,069,305,470 482,745 8.3%
    afdfb0bd rmedia ,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 1,280 128 24.0 661,535,286 501,544 5.1%
    f43c652b aprender ,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 1,223 304 24.0 652,175,907 505,911 5.1%
    3494c400 aprender ,claude-fable-5-1,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 1,201 511 23.5 638,718,403 477,880 5.0%
    ae16b7a2 aprender ,claude-fable-5-1,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 882 124 23.9 538,126,130 592,677 4.2%
    3fbea830 ruchy ,claude-opus-5-5,claude-sonnet-5 593 2,639 21.4 535,917,915 501,492 4.2%
    e8ae1d73 infra ,claude-fable-5-1,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 1,000 644 16.8 529,814,724 454,615 4.1%
    37f7875c infra ,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 686 886 24.0 504,188,476 584,707 3.9%
    e0f49495 aprender ,claude-haiku-4-5-20251001,claude-opus-5-5,claude-sonnet-5 804 1,111 22.2 493,174,490 454,294 3.8%
    2c9facfd aprender ,claude-haiku-4-5-20251001,claude-opus-5-5 853 40 24.0 486,685,368 563,425 3.8%

    A6. Claude per hour (UTC)

    hour sessions messages all tokens
    09-23 06:00 10 33 17,259,005
    09-23 07:00 16 1,117 478,143,666
    09-23 08:00 20 1,397 518,294,783
    09-23 09:00 21 1,346 575,135,980
    09-23 10:00 19 1,279 479,716,809
    09-23 11:00 20 1,179 460,782,031
    09-23 12:00 22 1,167 426,014,089
    09-23 13:00 23 857 399,269,375
    09-23 14:00 23 1,594 650,141,983
    09-23 15:00 30 2,299 761,193,105
    09-23 16:00 27 2,054 623,048,835
    09-23 17:00 38 2,732 839,461,326
    09-23 18:00 39 2,503 966,622,228
    09-23 19:00 34 3,318 967,849,230
    09-23 20:00 35 2,674 700,709,223
    09-23 21:00 17 386 165,497,457
    09-23 22:00 15 224 61,930,675
    09-23 23:00 16 379 122,071,327
    09-24 00:00 32 2,819 981,609,947
    09-24 01:00 29 2,611 761,068,953
    09-24 02:00 28 479 162,161,245
    09-24 03:00 1 2 0
    09-24 04:00 26 592 269,424,287
    09-24 05:00 28 2,153 614,241,022
    09-24 06:00 31 2,871 899,175,877

    B1. agy totals (quorum lane + probe envelopes on disk)

    envelopes lanes probes rounds input output thinking cache read total 429 in envelopes
    171 110 61 42 56,587,720 2,014,084 1,340,568 572,883,094 58,601,804 29

    B2. agy quorum lanes by project

    project rounds lanes total tokens avg input / lane avg lane minutes
    aprender 29 87 52,086,067 578,653 8.3
    xpile 3 9 3,506,387 373,857 6.7
    infra 5 14 2,246,102 151,726 2.4

    B3. agy per hour (UTC)

    hour agy conversations (every use) quorum lanes lane tokens 429 envelopes
    09-23 06:00 0 3 1,885,553 0
    09-23 07:00 0 6 642,931 0
    09-23 08:00 0 9 3,455,610 1
    09-23 09:00 0 3 2,486,008 0
    09-23 11:00 0 3 2,956,714 0
    09-23 18:00 0 6 1,700,886 2
    09-23 19:00 0 6 654,970 2
    09-24 00:00 138 27 16,496,860 7
    09-24 01:00 99 15 9,779,759 6
    09-24 02:00 38 9 6,612,457 0
    09-24 03:00 9 0 0 0
    09-24 04:00 25 0 0 0
    09-24 05:00 92 13 6,121,218 8
    09-24 06:00 137 10 5,808,838 3

    C. Levers, measured from the records above

    • Claude cache re-reads: 98.5% of all Claude tokens are cache READS (12,707,684,141). Every main-session turn re-reads the whole context, so a long-lived session pays its context size on every turn. The top 10 sessions hold 47.4% of all Claude tokens. (Raw COUNTS: a cache-read token is billed at a fraction of an input token, so this share of tokens is not the share of cost or of plan quota.)
    • Repeat quorum rounds: 3 of 34 tickets with lane envelopes on disk took more than one round (PMAT-080 x2, PMAT-534 x2, PMAT-775 x2); those tickets' lanes used 3,309,232 agy tokens.
    • Repeat rounds, from committed receipts (aprender, every branch): 49 quorum receipts committed for 41 tickets; 6 tickets committed more than one (PMAT-4134 x3, PMAT-4071 x3, PMAT-4080 x2, PMAT-4081 x2, PMAT-4029 x2, PMAT-3952-greedy2 x2). A committed receipt is one round that was kept; rounds rerun without a commit are not in this count.
    • Lane brief size: an agy quorum lane reads 514,433 input tokens on average (max 2,618,052) plus 5,206,251 cache-read tokens on average; three lanes read it per round.
    • Concurrency: at peak 13 quorum lanes ran at once on this host (from each lane's end time and duration). 538 agy conversations started in the window; 171 were quorum lanes or probes found on disk.

    D1. Top 10 sessions by cache read (main-session turns): the drivers

    session repo cache read turns median context peak context turns > 300k images read image bytes (base64)
    78401db7 aprender 1,060,110,083 2,184 488,170 966,466 1,596 74 4,509,884
    afdfb0bd rmedia 641,976,940 1,279 487,928 964,532 934 36 5,259,600
    f43c652b aprender 618,729,196 1,221 496,855 966,872 897 0 0
    3494c400 aprender 573,934,434 1,198 471,359 966,314 843 0 0
    ae16b7a2 aprender 522,741,219 879 642,019 966,260 744 0 0
    2c9facfd aprender 480,602,368 850 585,506 966,310 729 0 0
    6b740d81 aprender 476,397,412 867 583,272 966,760 654 0 0
    e8ae1d73 infra 454,615,441 999 457,664 960,073 678 0 0
    c48f7854 aprender 436,976,587 941 446,601 966,451 685 0 0
    2e05d836 rmedia 422,651,325 928 433,796 969,429 664 54 11,702,676

    Main-session turns in the window: 25,013; above 300k context: 16,693 (66.7%); their cache reads: 87.0% of all main-turn cache reads.

    D2. Sessions that read images

    session images image bytes (base64) main turns after the first image
    2e05d836 54 11,702,676 852
    afdfb0bd 36 5,259,600 1,194
    78401db7 74 4,509,884 2,048
    7f01a46c 9 535,348 346
    fb464f10 2 202,660 260

    E. Levers, with ESTIMATED savings (arithmetic on the measured per-turn records; labelled estimates)

    • E1. Cap the context at 200,000 (compact / restart the session there): the context above 200,000 re-read on every main turn sums to 6,856,632,015 tokens, 60.0% of all main-turn cache reads -- the ceiling of what a cap saves (an estimate: compaction itself costs a summary turn).
    • E2. Poll turns: 201 main turns wrote < 100 output tokens over a > 200k context (sleep / status polls, one-line acks); they re-read 111,344,131 tokens (1.0% of main-turn cache reads). Waiting in ONE long tool call (or a background monitor that wakes the session) instead of repeated short turns saves up to that (estimate).
    • E3. Images stay in context: 175 images read in 5 sessions stay in the context for every later turn. At the documented upper bound of ~1,600 tokens per image, that is up to 390,684,800 re-read tokens (3.4% of main-turn cache reads; an upper-bound estimate -- an image's real cost depends on its pixel size, which the transcript does not record).

  4. noahgift commented on Sep 24, 2026

    @noahgift
    noahgift
    on Sep 24, 2026
    MemberAuthor

    Update: what a context cap COSTS (scripts/token_usage_report.py @ 296562b, lambda, last 24 h).

    F1. Real compactions in the window (measured)

    compactions auto / manual median context before median summary written median context of the next turn
    27 25 / 2 968,681 14,004 80,241

    F2. A context cap, replayed on the measured turns (ESTIMATE): compactions it forces, what they cost, and the NET saving

    cap compactions forced compaction cost (read + summary + restart) re-reads avoided NET saving net / all main-turn context
    200,000 300 88,767,420 8,118,922,182 8,030,154,762 69.5%
    300,000 153 60,636,238 6,981,402,838 6,920,766,600 59.9%
    400,000 101 50,077,317 5,888,243,683 5,838,166,366 50.5%
    600,000 58 40,331,121 3,839,642,421 3,799,311,300 32.9%

    F2 replays each session's real per-turn context growth; when the simulated context passes the cap it compacts at the MEASURED cost (the summarizer reads the whole context once, writes the median summary 14,004, the next turn starts from the median post-compaction context 80,241). What it cannot see: work redone after a compaction (files re-read, lost state) -- so the NET is an upper bound on the saving, and the compaction count is exact for this replay.

    E1's line in the earlier report called the above-cap sum the ceiling of the saving. That was wrong: a compacted session restarts near its ~80k baseline, far below the cap. F2 is the replay.

  5. noahgift
    added 4 commits that reference this issue on Sep 24, 2026

    feat(scripts): token report -- what a context cap COSTS: real compact…

    296562b

    fix(tokens): agy usage by model (B2b); F2 no longer charges the resta…

    4e6625f

    chore(tokens): keep only the r1 quorum artifact, not its lane logs — G…

    f38703f

    audit(tokens): GH-4162 quorum r2 AGREED 2/2 agy; haiku seat pending t…

    465e7f8

  6. noahgift commented on Sep 24, 2026

    @noahgift
    noahgift
    on Sep 24, 2026
    MemberAuthor

    Re-measured on lambda (24h to 09-24 20:15Z) at branch GH-4162-token-usage-report @f41f00274 (pushed, no PR: cap 10/10, for fold).

    Fix: project attribution now uses git metadata (repo dir, or the linked-worktree .git file). The old fallback cwd.split('/')[1] put 122 sessions / 215M tokens (2.3%) under "mnt". The mnt row is gone and worktrees fold into their repos. --self-test passes 9/9 rows; the mutant that drops the worktree parse gives 7/9, rc=1.

    A2. Claude by project

    project sessions messages cache read cache write output all tokens share
    aprender 89 32,604 5,236,737,549 127,862,756 13,519,631 5,378,214,890 58.3%
    infra 66 11,030 1,683,267,636 45,234,428 4,731,814 1,733,267,641 18.8%
    rmedia 4 4,040 688,028,057 20,181,418 1,385,892 709,607,965 7.7%
    paiml-implement 2 2,421 388,502,541 8,034,770 1,171,982 397,715,363 4.3%
    apex 2 2,761 381,140,655 10,238,946 908,728 392,298,001 4.3%
    ruchy 19 2,470 303,574,195 9,972,914 729,635 314,283,394 3.4%
    xpile 1 1,985 273,014,564 6,008,317 582,342 279,611,677 3.0%
    forjar 23 244 16,138,943 1,096,589 134,960 17,372,066 0.2%
    unattributed:/home/noah 1 42 3,853,313 166,206 19,520 4,039,123 0.0%
    unattributed:budget 4 13 310,687 61,520 3,902 376,229 0.0%
    unattributed:tmp-e9 6 13 282,258 72,959 3,205 358,538 0.0%
    unattributed:src 1 5 305,800 5,154 1,285 312,249 0.0%
    unattributed:.claude 1 1 104,223 850 4,815 109,890 0.0%
    unattributed:tmp/aprender-30-budget 1 3 64,939 14,255 477 79,699 0.0%
    unattributed:tmp/probe-406-2318918 1 4 35,331 17,440 841 53,620 0.0%
    unattributed:/tmp 1 1 0 0 0 0 0.0%

    Scope: this host only. gx10 is not covered (no builds run there; its disk is full).

  7. noahgift
    added 5 commits that reference this issue on Sep 24, 2026

    chore(roadmap): GH-4162 kind: code (kind-gate)

    fd55054

    fix(token-report): F2 cap replay follows real compactions; gitdir and…

    8a8cc42

    audit(GH-4162): Claude-seat quorum r3 FAIL -> r4 PASS 2/2 (degraded: …

    4d98c1d

    Merge remote-tracking branch 'origin/GH-4162-token-usage-report' into…

    e461983

    regen: roadmap aggregate after folding #4162

    0c499cc
  8. noahgift
    removed this from the 0.70.0 milestone on Sep 26, 2026

  9. 3 remaining items

  10. noahgift
    added
    owner:la-0.71Owned by the 0.71 look-ahead slot
    on Oct 4, 2026
  11. noahgift
    modified the milestones: 0.70.1, 0.71.0 on Oct 4, 2026

  12. noahgift commented on Oct 12, 2026

    @noahgift
    noahgift
    on Oct 12, 2026
    MemberAuthor

    Status (la-0.71, owner per label), 2026-10-12.

    Agent: la-0.71

  13. noahgift
    added
    P1High priority
    and removed
    P0Critical priority
    on Oct 14, 2026

  14. noahgift commented on Oct 14, 2026

    @noahgift
    noahgift
    on Oct 14, 2026
    MemberAuthor

    P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.

  15. noahgift commented on Oct 14, 2026

    @noahgift
    noahgift
    on Oct 14, 2026
    MemberAuthor

    merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  16. noahgift
    removed a parent issue on Oct 14, 2026
  17. noahgift
    closed this as not plannedon Oct 14, 2026

  18. noahgift commented on Oct 14, 2026

    @noahgift
    noahgift
    on Oct 14, 2026
    MemberAuthor

    Moved: this sub-ticket is now a row in #5322 (Speed measurement harness: resource receipts, observability and the ollama/llama.cpp ratio), split out of #5305 by theme.

  19. noahgift
    reopened this on Oct 23, 2026
  20. noahgift
    added a parent issue on Oct 23, 2026
  21. noahgift
    modified the milestones: 0.71.0, 0.73.0 on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityowner:la-0.71Owned by the 0.71 look-ahead slotsize:MK̂ 3-4 h

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions