Skip to content

messages-only search polluted by inline thinking/tool markers (Pi, and likely Claude/Amp) #1446

Description

@zh-xl-kang

Summary

For JSONL providers, assistant turns are flattened into a single Content string
with thinking and tool-use markers inlined (e.g. [Thinking] ... [/Thinking],
[Read: path], [Bash] $ cmd, [Tool: name]). This was chosen in #75 to keep
export, FTS, and block ordering working ("matching Claude/Amp pattern"). The
tradeoff: real conversation, thinking, and tool calls all land in the message
location, so messages_only search cannot separate dialogue from tool/thinking
noise — and downstream consumers can't clean it either, because the markers get
truncated away inside FTS windows.

Root cause

Pi stores each assistant turn as a structured content list of blocks
({type:"thinking"}, {type:"toolCall"}, {type:"text"}). The Pi provider
renders these into one Content string with inline markers. Since #75 says this
matches the Claude/Amp pattern, the same tradeoff presumably applies to those
providers too, not just Pi.

Symptoms (reproduced on Pi)

  1. messages_only filtering breaks. session search filters
    location == "message" to search only conversation, but Pi's tool commands and
    file paths are rendered into message content and can't be excluded. A keyword
    that only occurs in a bash command or a read path still matches a
    messages-only search.
  2. Snippets are polluted with [Thinking], [Read: ...], [Bash] $ ...,
    [Tool: ...] instead of actual dialogue.
  3. Truncated FTS windows create unfixable noise. When the snippet window lands
    mid-thinking-block, the opening/closing delimiters are cut off and only raw
    thinking text remains, so no downstream regex cleanup can detect or remove it.

Example indexed Content (Pi):

[Thinking] ... [/Thinking]
Let me explore...

[Read: /path/to/README.md]
[Bash]
$ find /path -name "*.ts"

Observed on a real Pi session: of 40 messages with has_thinking=true,
0 had a non-empty thinking_text — thinking exists only inside the inline
[Thinking] marker in Content. Tool calls, by contrast, are extracted into the
structured tool_calls array.

Proposed direction

Keep export/FTS/block-ordering working, but stop inlining markers into Content:

  • Content = user messages + text blocks only (real dialogue).
  • Tool calls: already available as structured tool_calls (surfaced by
    session messages); export/FTS can draw from them.
  • Thinking: extract it into thinking_text (currently empty for Pi) so Content
    no longer needs the inline [Thinking] block.
  • A separate search mode (or the existing --include-tools) can still search
    thinking/tool content through those fields.

This makes messages_only mean "conversation only" uniformly across providers, and
eliminates the truncated-window noise entirely.

Happy to open a PR — would like to align on direction first, in particular whether
to handle this uniformly across the Claude/Amp/JSONL providers or
provider-by-provider.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions