[AlexErrant]

Problem

With Search Full Text on, clips whose CF_UNICODETEXT data is over 4000 bytes (~2000 characters) only match on their first ~500 characters (the description). Shorter clips search normally. Wildcard, simple, and regex search are all affected.

Regression

This broke in 1522219 (#666), which replaced the bundled SQLite 3.33.0 with SQLite3MultipleCiphers (SQLite 3.45.2).

Cause

Newer SQLite caches large values read from overflow pages (vdbeColumnFromOverflow, for values over 4000 bytes). For a BLOB, that path calls sqlite3VdbeMemSetStr(..., 0, ...), which tags the value as UTF-8 whatever the database encoding is. Ditto's database is UTF-16LE, so when LIKE (or Ditto's regexp) converts Data.ooData to text, it reads the UTF-16 bytes as UTF-8 and stops at the first zero byte, which comes right after the first character.

Fix

Search substr(Data.ooData, 1) instead of Data.ooData. SQLite re-tags a function's BLOB result with the database encoding (OP_Function calls sqlite3VdbeChangeEncoding), so the full text converts correctly.

Testing

• On a real database with about 49k clips, I ran ooData LIKE '%e%' over CF_UNICODETEXT rows that contain an "e" after the first character. It matched 38,118 / 38,118 rows of 4000 bytes or less, but only 31 / 2,316 rows over 4000 bytes. With substr(ooData, 1), it matched 2,316 / 2,316.
• I ran Ditto's generated search SQL against the same database, compiled with master's bundled amalgamation (3.53.2). The old query returns no results; the new one returns the expected clip.
• I built Release x64 and confirmed that the app now finds a phrase deep inside a long clip.