Repository navigation
Feat/structured sessions - #13
Open
AuceptinFang wants to merge 27 commits into
Open
AuceptinFang wants to merge 27 commits into
AuceptinFang wants to merge 27 commits into
Conversation
- Conversation reads `entries` (role, text, call_id, name, item_id) in place of the retired `messages`; default titles take the first user entry. - The trajectory shows every entry each event carries, in order: messages with their reported role, tool calls and results from their entries with their call id. Requests no longer hide history their stream has seen, and tool results are neither de-duplicated nor attached to their call. - Trajectory tests become two case tables plus a few focused tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New `llm_usage_event_stream_facts` table, one row per new event, written in the ingest transaction; the event table keeps its columns. - A request keeps where its new part starts, whether it opens a turn, the last identity of its last run and the roles of the model-side entries after it, and, without model output, the call ids of its tool results. A response keeps the identities it emitted and its output skeleton. - Facts come only from a completely decoded conversation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add GET /v1/agent-usage/sessions/{session_pk}/streams. A query groups the
session's events into exchanges, finds each request's base (pointer, last
output, results only) or why it has none, and cuts segments at roots,
turns, and branches. It returns structure only: ids, bases, rules, new
part offsets, and segments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The session page reads one layer per request, by session_pk:
- GET /sessions/{session_pk}/header: agents, time span, event, exchange,
turn and segment counts, token totals, providers and models.
- GET /sessions/{session_pk}/streams: segments only, each identified by its
first exchange (head), with its kind, parent head, the exchange it forks
from, time span and exchange count.
- GET /sessions/{session_pk}/segments/{head}: one page of a segment's
exchanges from head on (limit, after), each with its base, rule or reason,
new part, turn start, times, model, tokens, and a preview of its request
and response.
- GET /events/{event_pk}/conversation: an event's entries through the same
window as a preview (entries runs, text_preview).
Previews give the asked runs of entries as reported, cut texts carrying
their full UTF-16 length, and count what was left out.
streams.rs splits into facts, base, segmentation and test fixtures; the
code moves without change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ents Events carry their content only in conversation entries: - Ingest takes no text; responses drop text and text_sha256; /outline drops text_chars and preview. The text columns stay in the schema and are no longer written or read. - Search indexes every entry's text in order and the names of called tools; tool_content stays empty. - Trajectory drops its text fallback; a compaction summary is the response's assistant entries. Each request's stream facts keep a label: the last user entry with text in its new part, as its index, its first 50 UTF-16 code units, and its whole length, in three columns of the stream facts migration. /streams gives each segment the label of its first request, and the session header's title is the label of the session's first request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /sessions/{session_pk}/segments/{head} returns the exchanges from head
to the segment's end at once, as the session page design asks; limit,
after, and next go away.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SQLite
Search follows the confirmed search semantics: every event carrying the
query matches, requests and responses alike, inherited context included,
and each hit says whether it falls in what the event added.
- Pieces are the text and the name of every conversation entry. Each
distinct piece is stored once per owner (search_texts) and every piece
of an event refers to it (search_refs); a contentless trigram index over
the folded texts narrows candidates that the backend then verifies.
usage_events_fts goes away. The tables join the stream facts migration;
ingest writes the index in the event's transaction and startup
backfills events stored before it.
- Matching is a case-insensitive literal substring under a simple
per-character fold, shared through search::matching.
- Session search adds new_match_count and each match's new; matches and
fragments prefer hits in what an event added.
- GET /sessions/{session_pk}/search gives the hits within one session with
their UTF-16 ranges and the head of each hit's segment.
- A request's new part starts at its stored new_from; events without
stream facts count from 0. PostgreSQL keeps its Elasticsearch search
and answers in-session search with 503.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turns no longer come from a collector-declared turn_id:
- agent_turns, the events' turn_pk and turn_index, and ingest's turn
placement go away, as do the turn_id and naming_thread ingest fields
and the timeline, outline, and trajectory session endpoints built on
them. The schema changes are folded into the initial schema.
- Turn counts in the session list, the summary, and the session header
count the requests whose stream facts open a turn.
- A session title is the label of its earliest request, read from that
request's stream facts; the list, the header, and search return the
same EntryLabel. The first-user-input title and agent-reported titles
go away, and the facts migration rebuilds the title anchors.
- Session structures are cached by a per-session structure revision that
ingest advances. GET /events/{event_pk}/location, a single event, and
/events?locate=true give an event's current segment, its one-based
position, its kind, and its exchange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /summary takes an optional tz, an IANA zone name, that decides which local day the day dimension groups by; without it days stay UTC, and an unknown zone is rejected. from and to remain instants. PostgreSQL groups AT TIME ZONE; SQLite, without a zone database, groups by the zone's UTC offsets over the queried range, computed with chrono-tz. Ported from feat/local-day-usage d1f2d43 onto the current summary query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ingest stores the subagent_id, parent_subagent_id, and root_turn_id an event declares in its stream facts. A sub-agent's root segment attaches to a segment of its parent stream: the first one holding its declared root turn, else the earliest one whose time range holds its start. Attachment only decides where a segment shows; bases and segments are unchanged. /streams gives each segment subagent_id, attached_to, and attached_by, and /segments gives each exchange its declared ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stream facts keep the turn_id ingest field (the facts migration's
root_turn_id column becomes turn_id). Each exchange's turn is derived with
the session structure:
- In a session where any exchange declares a turn, a declared exchange
belongs to that turn. An undeclared one inherits its base's turn; with
no base it joins the turn that last began, and before any turn it
belongs to none.
- In a session without declarations, an exchange that starts a turn opens
one named by its exchange id; the rest follow as above.
- Turns are numbered from one by first appearance.
A Turn segment is cut where an exchange's turn differs from its base's, so
a segment holds one turn. Segments and exchanges carry turn_id and turn,
and an event's location gives its turn in place of segment_index. The
session list, the summary, and the header count distinct declared turns,
or turn-starting requests where nothing is declared; sub-agent attachment
compares turn_id.
GET /v1/agent-usage/sessions/{session_pk}/turns gives one node per turn:
its executors grouped by sub-agent, the main line first, each split into
the root-to-leaf path of its segments with the most exchanges and the
other segments; and whether the main line begins without a base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the token comparison queries (#10) and the 1.0.1 and 1.1.0 releases. The response-id fallback turns of #8 are not taken: turns are derived from the stream facts and the declared turn_id, so agent_turns, ingest's turn placement, and scripts/tests/turns_api.py stay retired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clippy 1.99 adds assert_is_empty, and CI installs the latest stable, so the nine assert!(x.is_empty()) checks failed both Clippy runs under -D warnings. They now compare against an empty value, which also shows the value when a check fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trailing is a reserved word in PostgreSQL, so the event stream facts migration failed with a syntax error and the backend could not start on Postgres. Diesel already quotes the column in its queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each executor of a turn now carries `parent_subagent_id`, the parent the first exchange of its earliest segment declares, and the executors come depth first by who started whom: the main line first, each executor followed by the sub-agents it started, siblings by start. A sub-agent whose declared parent takes no part in the turn sits at the top level. The declared parent was used only to attach sub-agent root segments, so a turn listed its sub-agents flat and forked workers lost their parent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The summary joins stream facts by event id since turns are counted from them, but the summary cover index lost turn_pk without gaining id, so the summary read every event row in its range. Event rows hold the conversation in metadata, a page per event: a week read 183 MB instead of 10 MB, and took seconds whenever those pages were out of the file cache. The index now ends with id, and a test pins the summary plan to covering indexes. The comments no longer describe event text, which ingest has stopped writing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
matched_events filtered by owner and id. SQLite without statistics takes the owner for a narrow filter, so it scanned an owner-first index over every event: since the summary cover index also holds id, the whole 15 MB index rather than the 8 MB owner-time index. The ids now find the events and the unary `+` keeps the owner a filter on them, as hits_in_session does; the page's events read 43 pages instead of 4489. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An `assistant` or `reasoning` entry with neither a call id nor an item id and with text that is not blank now has an identity: `text:<role>:<sha256 hex of its exact text>`. A response lists it among its emitted ids and keys its skeleton entry by it; a request whose last identity-bearing model entry is such text takes it as its last identity, and only entries after it trail. The base rule is unchanged: exactly one other exchange emitted it, and it said everything that trails it. Chat Completions answers and `reasoning_content`, and Anthropic text without thinking, carry no id, so each new user turn used to start a new stream. A replayed answer now continues the exchange that gave it, also behind the blank thinking OpenCode inserts before an answer that had none. Text the client rewrote has no emitter (`unknown_ids`), and text more than one reply said is `multiple_responses`; blank text still has no identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Facts are read only at ingest, so events stored before text identities keep the facts an older backend read. A data-only migration, 20261029120000, has the backend read the facts of every stored event again from its metadata in the transaction that records it, in SQLite and PostgreSQL alike. Events are paged by id, 64 at a time. An upsert writes only facts that differ, keeps the ids the client declared, and gives an event stored before facts were read its first row; each session with a written row gets one new structure revision. Ingest and the rebuild share one reader of an event's facts. PostgreSQL refuses an empty query, so its migration runs `SELECT 1`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inued A turn's exchanges are grouped by the sub-agent they declare; in a session with no turn_id, subagent_id or parent_subagent_id at all, by the line of context they continue, cut only where more than one successor of an exchange is continued. Within a group an exchange is a side exchange when nobody continued its output while another successor of the same base was continued, or when it has no base and the group continues something else in the turn; a group with nothing else in a turn keeps them all. The main line is everything else. Segments stay as the unit the session page reads: a group's main line is one segment and every side exchange a segment of its own. The longest-path main line and sub-agent attachment are gone; attached_to and attached_by are always null. Without declared turns, a turn-opening request opens a turn only when it continues a context (it holds model output or names the response it continues) or when no turn began before it. Stream facts keep that as continues_context, so the session list and the summary count turns the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ingest payload carries the conversation at the top level, as the AgentEvent does, instead of under metadata.conversation. Events store it in a column of their own, conversation (nullable JSON), and the event object the API returns has it at the top level too; metadata keeps purpose, omitted, headers and path. Migration 20261029140000 moves every stored metadata.conversation into the new column and drops it from metadata. Stream facts, search and the previews read the column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne before it A request whose last identity more than one other exchange emitted, such as an answer a /loop gives word for word every round or a short text said again after signed thinking, used to have no base (`multiple_responses`). The context is resent as it grew, so the exchange it continues saw, just before it spoke, the identity the request holds before the last one, or said that identity itself. Requests now store that identity as the fact `previous_identity` (repeats of the last identity skipped), and a second pass over the first bases keeps the emitter that is its one emitter or whose first base is that emitter; with nothing before the last identity, the emitter that had no context and said it first. Exactly one kept is the base; none means the client rewrote the context (`unknown_ids`); more than one, or no single emitter of the identity before, stays `multiple_responses`. Bases the first pass found do not change. `20261030120000_stream_facts_previous_identity` adds the column and, since only an event's conversation holds it, reads the stream facts of every stored event again in the same transaction, once, together with the text identity rebuild when both are pending. On a copy of the shared local database, 235 of 236 `multiple_responses` exchanges gain a base and one compacted request becomes `unknown_ids`; no other exchange changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the Postgres migration fix that quotes the trailing column and the Clippy 1.99 assert_is_empty fixes. Tests changed since on main keep their content in the assert_eq! form, and the two empty checks main added since follow it too. The Postgres summary counted first openers with e.id IN first_openers, which only SQLite accepts; it now selects their ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s checked The session header filtered events by owner and session together. SQLite picked the owner's time index for that and read the row of every event the owner has, so each header took about 60 ms on a database of 77k events, however small the session, and grew with the owner's events. The header now checks the owner on the session row and reads the events by session alone, with the session index: 0.6 ms for a session of 76 events, 21 ms for one of 16.6k, on the same copy. The events a session structure is derived from were filtered the same way. SQLite picks the session index there today only because the order matches it; their one caller checks the owner on the session row before loading them, so they are read by session alone too. PostgreSQL does the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… migration The five migrations since the initial schema become one, 20261030130000_event_conversation_and_stream_facts, which only creates the columns and tables. Databases from earlier releases are not upgraded, so the data moves and the one-time rebuilds of stream facts and title anchors go, and migrations run through Diesel alone again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Event rows no longer declare the text and text_sha256 columns, and the SQLite initial schema creates the search text tables directly instead of an FTS5 projection the next migration dropped. - SQLite no longer indexes stored events at startup; ingest indexes each event in its own transaction. - Session title tests and the title contract script cover only the current first-request label rule. - README describes the current title object and lists every route; search fixtures and the bench script use neutral text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sessions are now built from the collector's structured
conversationentries instead of re-parsed bodies or a declared turn hierarchy.conversation.entries(top-levelconversation, stored in its own column) and stores per-event stream facts (new part, last/previous identity, trailing roles, result/emitted ids, label, declaredsubagent_id/parent_subagent_id/turn_id) beside each event.turn_id, a turn's exchanges grouped, executors ordered by who started whom, model text without ids identified by its digest, repeated identities told apart by the identity before them.sessions(list),sessions/{pk}/header,/streams,/turns,/segments/{head},/search(in-session),events/{pk},events/{pk}/conversation,events/{pk}/location.agent_turns,turn_pk,turn_index) and the session timeline/outline/trajectory endpoints.Breaking changes
feat/structured-sessions) together.20261014150000_initial_schemais edited and one migration is added,20261030130000_event_conversation_and_stream_facts. Databases created by earlier releases are not upgraded — recreate them.Fixes included
trailingcolumn; the summary query selects first openers withIN (SELECT id …).