Repository navigation
Feat/backend boundary - #7
Merged
Merged
Conversation
Collectors now report input, cache read, cache write, and output as disjoint counters with reasoning as a breakdown of output. Derive a missing total from those four counters instead of also adding reasoning, and return output and cache counters from the token_usage summary projection so charts can split cached input from uncached input. Rewrite stored rows for both SQLite and Postgres: remove OpenAI-style cached input from input_tokens and recompute total_tokens, which older collectors set to input_tokens alone and so dropped Anthropic cache reads and writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Viewing a large session loaded every accumulated request snapshot at once
(66 MB for a 1,590-event session). Add three read paths so clients can load
only what they show:
- GET /v1/agent-usage/events/{event_pk} returns one event verbatim.
- GET /v1/agent-usage/sessions/{session_pk}/outline returns the timeline
without text or metadata bodies, plus text length, a short preview, and
attachment counts (about 1 MB for the same session).
- GET /v1/agent-usage/sessions/{session_pk}/trajectory merges the timeline
so each request and response contributes only its new content: new
request paragraphs, response text, and tool calls paired with their
first-seen results. Side requests such as classifiers form their own
streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Store disjoint input, cache read, cache write, and output token counters alongside the session outline and trajectory endpoints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add GET /v1/agent-usage/sessions, returning one page of owner-scoped sessions ordered by their latest event, with turn, event, and token counts plus the device host and platform. Sessions can be filtered by an activity window and Agent name, with Claude aliases matched together. Implemented for both the SQLite and PostgreSQL backends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep sessions with at least one event from the given LLM provider or model, so the session index can answer which sessions used a provider or model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude Code and Codex ask a model to name each session and return
{"title": ...}. Recognize those calls at ingestion by their official naming
instruction plus a response made only of agreeing title objects, store the
result in session_titles, and expose title/purpose on session info and the
session list. Codex naming threads are marked purpose=title_generation.
A one-time backfill covers events stored before capture existed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex compacts its context by sending a normal request that ends with a fixed "CONTEXT CHECKPOINT COMPACTION" prompt, and replays the answer at the start of the compacted history behind a fixed "Another language model started to solve this problem" sentence. The trajectory showed the prompt as user input, the summary as an assistant reply, and the summary again as user input. Recognize both fixed texts in the legacy and the snapshot projection. The prompt and the summary from its response form one compaction item, and the replayed handoff is hidden. When the summary request was not observed, the handoff alone records the compaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…entity and order" This reverts commit 9904c12.
Codex names a task from a separate thread whose requests carry no link to the task (verified on 169 captured naming requests across Codex 0.154-0.159). The naming request quotes the task's user message, and Codex starts the naming turn right after the task turn. A task session is matched when its turn ID (UUIDv7) starts within 60 s of the naming turn on the same device and its latest user message opens with the quoted prompt. Only a single candidate is matched; a later second candidate withdraws the match. The title is then recorded for the task session, and the naming thread keeps a link to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…models Query endpoints accept repeated agent_name, session_id, llm_provider, and llm_model keys; values within one filter match any, filters still combine. Single-value calls are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SQLite summary loaded every matching event row, including text and metadata that hold whole request bodies, parsed the metadata JSON, and grouped in memory. A week of usage meant reading over 2 GB to add up token counts. It now groups in one query that mirrors the PostgreSQL aggregate, reading only the token, grouping, session, and turn columns, with the time bounds always bound so the owner-time index limits the scan. Summaries now cover every event in range, as on PostgreSQL, instead of only the newest 100,000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…exes Token columns sit after the event text in each row, so reading them walks the text's overflow pages. Two indexes carry every column the usage summary and the session list read, and the session list counts an indexed column, so both queries no longer touch event rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # crates/abyss-backend/src/storage/sqlite/repository.rs
Event text and metadata hold whole request bodies, so ten events came to about 8 MB while an event list shows a few hundred bytes of each. With fields=list the events endpoint returns what a list shows, and neither storage backend reads text, its digest, or metadata, nor loads attachments. Without the parameter the response is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stream was named by every system message in a request. Claude Code puts a billing header with a per-request hash and previous-request id at the top of its system prompt, a context budget at its end, and system messages inside the conversation that grow with it, so nearly every request opened a new stream and replayed the whole conversation. A 480-turn session came to 1,255 streams and 349,000 items, 227 MB, with one user message 2,167 times. Streams are now named by the system prompt the conversation opens with, without the billing header and context budget lines, and those lines are also left out when telling system prompts apart. The same session has 6 streams and 6,837 items, 10 MB, and keeps every distinct item it had. The archived bodies keep both lines, so billing can still be counted from them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atches A search for a common word took 10 to 50 seconds on a real database while the full-text match itself took milliseconds. Every step after the match paid for every hit: fragments were built for all hits of the page's sessions before keeping three per session, every hit was loaded to dedupe sessions in Rust, three queries re-ran the match, reading session keys through the FTS table loaded each matching document, and ordering by observation time read through event text that sits before it in the row. One set of expressions now ranks hits by rowid and bm25 alone, reads keys from the FTS content table, and takes observation times from the owner-time covering index when hits are many or from event rows when few, applying the time window after that join. Window functions give each session's match count and strongest event, so sessions are ordered and paged in SQL with the total in the same query, and fragments are built only for the matches returned. On the user's 41,000 events, "entry" goes from 13.8 to 0.24 seconds and "the" from 36.9 to 0.79, with identical responses across a 21-query benchmark, which is added under scripts/bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… bodies Titles and trajectories read the collector's role-aware `metadata.conversation`; the provider-body decoders are removed, so the backend never parses raw request or response bodies. Turns are numbered by the backend from the traffic's turn identity (`codex_turn_id`, `claude_turn_id`, `turn_id`, then provider ids); the sender's `turn_index` is ignored. A Claude Code event without a turn id joins the turn of a same-session `cc_prompt_id`, else the latest turn started at or before it. Claude `thread` continue requests join the stream of the response they continue. Session titles come from collector fields only: a `session_title` names its own session and outranks the first-user-input default, which never replaces it; `purpose` requests are not user input; Codex `thread_source == "thread_title"` sessions are marked `title_generation`. The Codex link heuristic, text-based naming recognition and their backfills are removed. Stored titles and links are kept; no migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI installs the latest stable toolchain, now 1.99.0. Its `redundant_field_names` fires on code generated by diesel_derives 2.3.9 for `QueryableByName`; diesel 2.3.14 / diesel_derives 2.3.10 no longer trigger it (lockfile only). Empty-collection test asserts now use `assert_eq!` for the new `assert_is_empty` lint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The project is not released, so local databases are rebuilt instead of migrated. SQLite and PostgreSQL each keep a single `20261014150000_initial_schema`; the new version makes a database created from the old migrations fail at migration time instead of running against a different schema. - Fold in `session_titles` (with `parent_session_pk`) and the SQLite covering indexes behind the usage summary and the session list. - Add `agent_turns.turn_key` with `UNIQUE (session_pk, turn_key)`, and the exchange identities `stream_id`, `response_id`, and `previous_response_id` on `llm_usage_events`. - Drop `disjoint_token_counters` (it only rewrote old rows), the `schema_backfills` table with the startup title backfill, the unused `session_titles.naming_turn_id` / `naming_prompt` columns, and the SQLite FTS `commands` / `file_paths` columns. The Diesel schemas follow. Session title capture, sharing these files with the backfill, now takes the collector's `naming_thread` flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PostgreSQL readiness check selects a bound `1` through Diesel. The search outbox backfill becomes `insert_into(..).values(select)` with `on_conflict(..).do_nothing()`, and its completion check `select(not(exists(..)))`; `RawBackfillState` is gone. The same rewrite of the PostgreSQL upserts (`excluded()` and typed `coalesce` / `least` / `greatest`) and of the content-free list placeholder columns lives in the repository files and lands with the ingest change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… turn_id The backend uses the values abyss-rs sends instead of reinterpreting them. - Agent names, providers, devices, session ids, models, and text are stored exactly as received: no trimming, slugs, case folding, Claude alias folding, or `hostname` alias, and empty text stays empty. Claude Desktop is no longer merged into Claude Code. Agent and provider filters match exactly, only trimming and de-duplicating the query values. - Token counters are stored as sent and absent ones as zero; totals are no longer derived and negative counts no longer rejected. - Events are placed by the top-level `turn_id`: events naming a turn share it, the first named turn adopts the turn opened without one, events naming none join the turn in progress when observed, and the session's turns and events are renumbered by start time when a late event reorders them. The metadata turn-identity chain and the Claude Code `cc_prompt_id` attachment are removed. `naming_thread` is passed on to session titles. - `stream_id`, `response_id`, and `previous_response_id` are stored and returned with each event. - Dead reads are removed: the response-chain timeline ordering, the outline `token_usage_estimated`, and the session `working_directory`. - The PostgreSQL upserts use `excluded()` and typed `coalesce` / `least` / `greatest`, and the content-free list binds its placeholder columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tool inputs were parsed as JSON to pull `command` / `cmd` / `file` / `file_path` / `path` values, plus the session working directory, into separate commands and file paths fields. The backend no longer guesses tool argument names: the search projection, the SQLite FTS snippets, and the Elasticsearch document, mapping, query fields, and highlights drop both fields. Tool input stays searchable through `tool_content`. Search results report the Agent name as stored, and test fixtures carry the new exchange identity fields and explicit token counters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eported Trajectories no longer interpret prompt text or provider-specific fields. - Streams: a request whose `previous_response_id` names a known response joins that response's stream, else a request with a `stream_id` joins that stream, else it opens a stream of its own; a response joins the stream of its exchange by `response_id`. The system-prompt hash, `claude_thread`, latest-request inference, and the billing header / `<total_tokens>` stripping are removed. - Continuations and responses show every message they carry; only full requests are compared with the stream history. - Roles are kept as reported. Items are `message` with the original `role`, or `text`, `tool_call`, `tool_result`, and `compaction`. The opening instructions are the first message when `instructions_present` is set and show only when they change; messages without text are left out. - Compaction is recognized from `purpose == "compaction"` on the exchange instead of Codex prompt text, and its response is matched by `response_id`. - Conversations require `completeness`; the legacy-collector branches that fell back to flattened text are removed. - `token_usage_available` means any counter is non-zero, and `token_usage_estimated` is gone. - Session titles mark naming threads from `naming_thread` instead of Codex `thread_source`, and take the first `user` message as default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Fixtures send `codex` / `openai` as abyss-rs does, explicit token counters through a `token_usage()` helper (totals are the disjoint sum, so the summary total is 36), and exchange identities at the top level instead of in metadata. - New checks: `agent_name=Codex`, `agent_name=claude-desktop`, and `llm_provider=OpenAI` match nothing, and events return the exchange identities they were ingested with. - `claude_turns_api.py` becomes `turns_api.py`, covering `turn_id` placement, adoption of the opening turn, and numbering by start time. - Title fixtures mark naming threads with `naming_thread`; the mock Elasticsearch drops the commands and file paths fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The backend no longer interprets Agent or provider internals. It uses only what abyss-rs reports on each
AgentEvent:turn_id,stream_id,response_id,previous_response_id;All provider-specific reading now lives in abyss-rs.
Changes
The feature consists of the last 6 commits,
59a4daa..366ac5e:59a4daarefactor(schema)!: SQLite and PostgreSQL each keep a single initial schema,20261014150000_initial_schema.agent_turns.turn_key.stream_id,response_id, andprevious_response_idon events.28a9b14refactor(storage): replaces the hand-written SQL with the Diesel DSL.1587b9cfeat(ingest)!:turn_idand renumbered by start time.6b9b6f6feat(search)!: indexes tool input as written. The guessedcommandsandfile_pathsfields are removed from FTS and Elasticsearch.5f82fd4feat(trajectory)!: places streams by the declared keys.stream_idjoins that stream; otherwise it opens a new stream.message(with the reportedrole),text,tool_call,tool_result, orcompaction.purpose.claude_thread, and the billing-header handling.366ac5etest(blackbox): the blackbox scripts follow the new ingest contract.Breaking changes
turn_id,naming_thread,stream_id,response_id, andprevious_response_idare read from the top level of the event.claude_thread, andclaude_code_billingare no longer read.rolefield;token_usage_estimatedis removed.Dependencies and merge order
feat/backend-boundaryproduces the new top-level fields. Not pushed yet.feat/backend-boundaryrenders the new trajectory shape. Not pushed yet.Deploy the backend and the dashboard together: the current dashboard cannot read the new trajectory items.
Commit range note
This branch is based on local
main(c45a9d1). Localmainis 29 commits ahead oforigin/main, because other features were integrated locally and not pushed yet, so those commits also appear here.Intermediate commits of this feature do not compile on their own. Please merge the feature as a whole (or squash).