Summary
For JSONL providers, assistant turns are flattened into a single Content string
with thinking and tool-use markers inlined (e.g. [Thinking] ... [/Thinking],
[Read: path], [Bash] $ cmd, [Tool: name]). This was chosen in #75 to keep
export, FTS, and block ordering working ("matching Claude/Amp pattern"). The
tradeoff: real conversation, thinking, and tool calls all land in the message
location, so messages_only search cannot separate dialogue from tool/thinking
noise — and downstream consumers can't clean it either, because the markers get
truncated away inside FTS windows.
Root cause
Pi stores each assistant turn as a structured content list of blocks
({type:"thinking"}, {type:"toolCall"}, {type:"text"}). The Pi provider
renders these into one Content string with inline markers. Since #75 says this
matches the Claude/Amp pattern, the same tradeoff presumably applies to those
providers too, not just Pi.
Symptoms (reproduced on Pi)
messages_only filtering breaks. session search filters
location == "message" to search only conversation, but Pi's tool commands and
file paths are rendered into message content and can't be excluded. A keyword
that only occurs in a bash command or a read path still matches a
messages-only search.
- Snippets are polluted with
[Thinking], [Read: ...], [Bash] $ ...,
[Tool: ...] instead of actual dialogue.
- Truncated FTS windows create unfixable noise. When the snippet window lands
mid-thinking-block, the opening/closing delimiters are cut off and only raw
thinking text remains, so no downstream regex cleanup can detect or remove it.
Example indexed Content (Pi):
[Thinking] ... [/Thinking]
Let me explore...
[Read: /path/to/README.md]
[Bash]
$ find /path -name "*.ts"
Observed on a real Pi session: of 40 messages with has_thinking=true,
0 had a non-empty thinking_text — thinking exists only inside the inline
[Thinking] marker in Content. Tool calls, by contrast, are extracted into the
structured tool_calls array.
Proposed direction
Keep export/FTS/block-ordering working, but stop inlining markers into Content:
Content = user messages + text blocks only (real dialogue).
- Tool calls: already available as structured
tool_calls (surfaced by
session messages); export/FTS can draw from them.
- Thinking: extract it into
thinking_text (currently empty for Pi) so Content
no longer needs the inline [Thinking] block.
- A separate search mode (or the existing
--include-tools) can still search
thinking/tool content through those fields.
This makes messages_only mean "conversation only" uniformly across providers, and
eliminates the truncated-window noise entirely.
Happy to open a PR — would like to align on direction first, in particular whether
to handle this uniformly across the Claude/Amp/JSONL providers or
provider-by-provider.
Summary
For JSONL providers, assistant turns are flattened into a single
Contentstringwith thinking and tool-use markers inlined (e.g.
[Thinking] ... [/Thinking],[Read: path],[Bash] $ cmd,[Tool: name]). This was chosen in #75 to keepexport, FTS, and block ordering working ("matching Claude/Amp pattern"). The
tradeoff: real conversation, thinking, and tool calls all land in the
messagelocation, so
messages_onlysearch cannot separate dialogue from tool/thinkingnoise — and downstream consumers can't clean it either, because the markers get
truncated away inside FTS windows.
Root cause
Pi stores each assistant turn as a structured
contentlist of blocks(
{type:"thinking"},{type:"toolCall"},{type:"text"}). The Pi providerrenders these into one
Contentstring with inline markers. Since #75 says thismatches the Claude/Amp pattern, the same tradeoff presumably applies to those
providers too, not just Pi.
Symptoms (reproduced on Pi)
messages_onlyfiltering breaks.session searchfilterslocation == "message"to search only conversation, but Pi's tool commands andfile paths are rendered into message content and can't be excluded. A keyword
that only occurs in a
bashcommand or areadpath still matches amessages-only search.
[Thinking],[Read: ...],[Bash] $ ...,[Tool: ...]instead of actual dialogue.mid-thinking-block, the opening/closing delimiters are cut off and only raw
thinking text remains, so no downstream regex cleanup can detect or remove it.
Example indexed Content (Pi):
Observed on a real Pi session: of 40 messages with
has_thinking=true,0 had a non-empty
thinking_text— thinking exists only inside the inline[Thinking]marker inContent. Tool calls, by contrast, are extracted into thestructured
tool_callsarray.Proposed direction
Keep export/FTS/block-ordering working, but stop inlining markers into
Content:Content= user messages +textblocks only (real dialogue).tool_calls(surfaced bysession messages); export/FTS can draw from them.thinking_text(currently empty for Pi) soContentno longer needs the inline
[Thinking]block.--include-tools) can still searchthinking/tool content through those fields.
This makes
messages_onlymean "conversation only" uniformly across providers, andeliminates the truncated-window noise entirely.
Happy to open a PR — would like to align on direction first, in particular whether
to handle this uniformly across the Claude/Amp/JSONL providers or
provider-by-provider.