Skip to content

usage: count Codex reasoning tokens as thinking - #1794

Merged
askalf merged 4 commits into
masterfrom
codex-reasoning-thinking
Nov 2, 2026
Merged

askalf merged 4 commits into
masterfrom
codex-reasoning-thinking

Conversation

@askalf

@askalf askalf commented Nov 2, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

The thinking share from #1783 now covers GPT traffic. Every Codex row was recorded with thinkingTokens: 0 (hardcoded in reportCodexOutcome), so dario usage, --by-key and /analytics showed no reasoning on any ChatGPT-subscription lane, however much the model reasoned.

  • reasoningTokensFromResponses() in codex-backend.ts reads output_tokens_details.reasoning_tokens from the Responses usage block. OpenAI counts it inside output_tokens, the same way Anthropic counts thinking inside output, so it fits the existing field as is. Rounded, capped at the output it is part of, 0 when absent or malformed.
  • splitResponsesUsage() carries it as reasoning; CodexForwardOutcome gains reasoningTokens, set on both report sites (the translated Messages route and the Responses passthrough); reportCodexOutcome records it as thinkingTokens.
  • Nothing on the wire changes; this is accounting only. Ledger rows written before read as before.

Why: on one deployment's last 7 days, 22% of all output tokens were GPT through the Codex engine, and the thinking line could say nothing about any of it. With the per-key default effort from #1786 able to set reasoning.effort on those lanes, the share is the number that says which of them need it.

How to test

  • npm run build
  • node test/ledger-thinking-codex.mjs (new, 12 checks: the reader's tolerance and cap, splitResponsesUsage, then a real startProxy with a loopback Codex stub on both GPT routes, checked on the window record, the ledger cell, the consumer summary with its price, and the Thinking: line; 7 of them fail against the proxy without the record change). It is its own file because the proxy reads the Codex base URL and the accounts folder once per process.
  • Unchanged and green: codex-backend, ledger, ledger-wiring, ledger-thinking, ledger-per-key-tokens, analytics-recording, analytics-stream, responses-inbound-wiring, anthropic-responses-translate.test, anthropic-responses-stream.test, key-effort, keys-budget-proxy, metrics, tui-*.

Checklist

  • npm run build passes
  • npm test passes (CI; the suites above were run locally)
  • Touches src/? Then a bullet under ## [Unreleased] in CHANGELOG.md is in this PR
  • Version left alone; no ## [x.y.z] heading added
  • No issue auto-close wording
  • Touches proxy.ts: one field on the Codex analytics record; test/compat.mjs in CI
  • No new runtime dependencies added
  • No tokens/secrets in code or logs
  • README and site test count 287 -> 288 (one new test file)

Every Codex row was recorded with thinkingTokens 0, so the thinking share from #1783 was blind on GPT lanes. The Responses usage block carries output_tokens_details.reasoning_tokens inside output_tokens, the same way Anthropic counts thinking; reasoningTokensFromResponses() reads it (rounded, capped at output, 0 when absent), splitResponsesUsage() and CodexForwardOutcome carry it from both GPT routes, and reportCodexOutcome records it. Accounting only; nothing on the wire changes. New test/ledger-thinking-codex.mjs (12 checks, 7 fail without the record change).
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests Test suite changes labels Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

📦 This PR will not cut a release

version stays at 6.20.0 but 5 shipping file(s) changed: README.md, docs/pool-keys-and-analytics.md, site/index.html (+2 more)

cc-drift-auto-release tags and publishes only when package.json's
version moves on master. As it stands this PR merges green and ships
nothing — the change sits on master until some later PR's bump carries it
out.

If that is intended (revert, docs-only refactor, or a stacked PR whose
partner carries the bump), ignore this. Otherwise bump all three slots
that scripts/preflight.mjs checks, in one commit:

  • package.json → 6.20.1
  • package-lock.json → .version and .packages[""].version
  • CHANGELOG.md → a ## [6.20.1] - 2026-11-02 section under ## [Unreleased]

This comment is advisory and updates itself on every push; it never
blocks the merge.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Live test: ✅ PASS

Head b04ef19cad0a318967ade3c856395c8f646749be on the self-hosted runner: All stages PASS: install/build/cc-detect/cli/doctor/live-probe(pong)

Stages
install: PASS
build: PASS
cc-detect: PASS
cli: PASS
doctor: PASS (exit 1, 19 rows; no credentials, reported cleanly)
live: PASS (pong)

Full workflow run

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. Codex Responses usage now carries bounded reasoning-token counts through both forwarding routes into analytics and the ledger, with coverage for the reader, routes, summaries, and displayed share.

Minor:

  • test/ledger-thinking-codex.mjs:2: This introductory block narrates the motivation and structure of the patch rather than documenting a test-specific constraint. It adds a large maintenance burden without helping the assertions establish their behavior.

check-readme-line-count failed on this branch after the master merges: src/ measures 44,018 lines across 88 files against a stated ~43k (the site still said ~42k across 87).

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. This carries Responses reasoning token usage through both Codex forwarding paths into analytics and ledger thinking totals. The reader validates and caps the value against output, and the new end-to-end coverage exercises translated and passthrough routes, absent details, and the ledger summaries.

@askalf
askalf merged commit cd96e74 into master Nov 2, 2026
27 checks passed
@askalf
askalf deleted the codex-reasoning-thinking branch November 2, 2026 20:15
@askalf askalf mentioned this pull request Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation tests Test suite changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants