Skip to content

fix(extension): tb thread always returns empty — getRaw() headers, angle brackets, downstream search - #4

Closed
dboeckenhoff wants to merge 5 commits into
vitalio-sh:mainfrom
dboeckenhoff:main
Closed

dboeckenhoff wants to merge 5 commits into
vitalio-sh:mainfrom
dboeckenhoff:main

Conversation

@dboeckenhoff

Copy link
Copy Markdown

#3

Daniel Böckenhoff added 2 commits May 9, 2026 06:20
Three bugs in the thread endpoint and one missing search parameter:

1. getFull() doesn't expose RFC 2822 headers (References, In-Reply-To,
   Message-ID) in the headers map in practice. Switch to getRaw() and
   parse headers directly from the raw message bytes.

2. Message-IDs extracted from the References header include angle
   brackets (<msgid@domain>) but messages.query({ headerMessageId })
   expects them stripped. Add stripBrackets() before querying.

3. The algorithm only found upstream messages (those referenced by the
   queried message). Replies that arrived after it don't appear in its
   own References chain. Add a subject-based pass to catch downstream
   messages too. Strips Re:/WG:/AW:/Fwd: prefixes and [list-name]
   brackets before searching.

Also expose headerMessageId as a filter in the /messages/search POST
endpoint — messenger.messages.query() supports it but the HTTP layer
didn't pass it through.
- extension/src/thread-utils.js: extract pure functions (parseHeader,
  stripAngleBrackets, parseReferences, normalizeSubject, buildThreadIds)
  so they can be unit-tested outside Thunderbird's extension context.
  Also found and fixed two edge-case bugs: stripAngleBrackets and
  normalizeSubject now trim whitespace before applying their regexes.

- test/thread-utils.test.mjs: 29 unit tests covering RFC 2822 header
  parsing, bracket stripping, References tokenization, subject
  normalization (all German/English reply prefixes, list-name brackets,
  stacking), and full buildThreadIds including forwarded-message bleed.

- test/quick-test.mjs: 3 new integration assertions:
  · headerMessageId is forwarded through /messages/search
  · /messages/:id/thread returns an array
  · thread results are sorted by date and count matches length
@dboeckenhoff

Copy link
Copy Markdown
Author

Problem

tb thread <id> consistently returns {"thread":[],"count":0} regardless of message.

Root causes

1. getFull() doesn't expose RFC 2822 headers

The thread handler extracted References/In-Reply-To/Message-ID from
messenger.messages.getFull(msgId).headers, but these keys are absent in
practice — getFull() doesn't populate the top-level message header map
reliably. The existing /messages/:id/raw endpoint proves getRaw() has
them.

Fix: parse headers from getRaw() instead, with folded-header unfolding.

2. Angle brackets not stripped before messages.query({ headerMessageId })

The References header contains IDs in <msgid@domain> form. The stored
msg.headerMessageId field omits the brackets. Querying with brackets
included returns nothing.

Fix: strip < > from each extracted token before querying.

3. Downstream messages not found

The current algorithm only finds messages referenced by the queried message
(ancestors). Replies that arrived after it don't appear in its own References
chain.

Fix: after the References pass, run a subject-based
messages.query({ subject: normalizedSubject }) to catch downstream replies.
Strips Re:/WG:/AW:/Fwd:/FW: prefixes and [list-name] brackets before
searching.

Bonus: headerMessageId not exposed in /messages/search

messenger.messages.query() supports headerMessageId but the HTTP search
endpoint didn't pass it through. One-line fix.

Tests

  • extension/src/thread-utils.js: pure functions extracted from the fix
    (header parsing, bracket stripping, subject normalization) so they can be
    unit-tested outside Thunderbird's extension context.
  • test/thread-utils.test.mjs: 29 unit tests. Caught two real edge-case bugs
    during test writing (missing trim before regex), fixed in both files.
  • test/quick-test.mjs: 3 new assertions — headerMessageId forwarding,
    thread returns sorted array, count matches length.

Daniel Böckenhoff added 3 commits May 26, 2026 21:14
Destructured only { hours, limit } from body, dropping accountId entirely.
Now extracts accountId; collects limit*5 messages up-front when filtering
(accounts are interleaved in the raw query stream), then filters by
msg.folder.accountId and slices to limit.
Previous approach post-filtered after collecting a capped batch, so
accounts with low message volume (e.g. Gmail) got zero results if the
batch filled up with messages from a busier account (IPP).

Fix: pass accountId into collectMessages so filtering happens inside
the pagination loop — keeps fetching pages until limit matching messages
are found or the result set is exhausted.
@vitalio-sh

Copy link
Copy Markdown
Owner

Thank you @dboeckenhoff — your root-cause analysis (getFull() missing the headers, angle brackets in headerMessageId queries) was exactly right, and the fix ships in 1.1.0 via #25 with you as co-author. Verified against a live Thunderbird with Gmail threads.

A few adjustments made during integration:

  • thread-utils.js is now loaded by the manifest and used by background.js, so the tests exercise the real code instead of a copy.
  • Subject-based downstream matches must equal the normalized subject exactly (Thunderbird's subject query is substring), exclude junk, and are labelled threadMatch: "subject" vs "references".
  • In-Reply-To is parsed like References, and the getFull() fallback strips brackets too.
  • /recent also honours unreadOnly, which the CLI sends but the extension ignored.

Closing in favour of #25.

@vitalio-sh vitalio-sh closed this Sep 14, 2026
jctots pushed a commit to jctots/thunderbird-cli that referenced this pull request Sep 14, 2026
… bounded parallelism

Integrates the extension parts of PR vitalio-sh#4 and PR vitalio-sh#20, with fixes:

Thread (from vitalio-sh#4)
- Parse References / In-Reply-To / Message-ID from getRaw() (getFull()
  doesn't expose them) and strip angle brackets before
  messages.query({ headerMessageId }) — previously `tb thread` always
  returned an empty list. getFull() fallback now strips brackets too.
- Downstream replies found by subject are limited to exact normalized
  subject matches (Thunderbird's query is substring), exclude junk, and
  are labelled threadMatch: "subject" vs "references".
- thread-utils.js is loaded by the manifest and is the single source for
  the parsing helpers instead of a test-only copy.

Recent / search (from vitalio-sh#4)
- /recent honours accountId, and now unreadOnly as well; the CLI sent
  both but the extension ignored them.
- /messages/search accepts headerMessageId.

Connection (from vitalio-sh#20, reworked)
- Reconnect backoff 3s → 6s → 12s → 15s cap (PR used 60s, which would
  leave a restarted bridge waiting up to a minute), reset on connect,
  immediate retry on idle → active ("idle" permission).
- Socket handlers are bound to their own socket, so a late close from a
  replaced socket can't null the live one (main crashed on ws.send).

Throughput (from vitalio-sh#20)
- read-batch, fetch, bulk tag/fetch and thread lookups run through a
  shared queue capped at 8 concurrent operations, preserving order.
- Attachment base64 encoded in 32 KB chunks instead of byte by byte.

Not taken from vitalio-sh#20: the 30s folder-info cache (stale unread counts
right after mark-read; Thunderbird is the source of truth), the
/debug/compose-capabilities endpoint, and the 15s resume-poll interval.

test/extension.test.mjs loads the real background scripts in a vm with a
mocked messenger API (39 checks); against the previous extension the
thread, recent, search and reconnect checks fail.

Co-authored-by: Daniel Böckenhoff <dboe@ipp.mpg.de>
Co-authored-by: le-dawg <3172733+le-dawg@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019uoUWTHVsDJpMJCVNFZz5L
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants