[YevheniiKotyrlo]

Problem

zero-cache gathers its replica's planner statistics from a sample, under the analysis_limit = 1000 that getPragmaConfig fixes, and SQLite takes each sqlite_stat1 average from the rows it sampled. So an index led by a column with few values, such as a soft-delete flag, reads as selective, and SQLite walks it instead of seeking a more selective one. On a million-row table with deleted = 0 on 99% of rows, WHERE deleted = 0 AND group_id IN (…) ORDER BY created_at DESC, id DESC LIMIT 50 plans:

analysis_limit	sqlite_stat1 of (deleted, created_at, id)	Plan	Median, groups whose rows are old
1000	1000000 1001 1 1	walks (deleted, created_at, id), filtering every row	458 ms
0	1000000 500000 1 1	seeks (group_id) and sorts	3 ms

The probe and its measurements: #6732 (comment)

Solution

ZERO_REPLICA_ANALYSIS_LIMIT (--replica-analysis-limit; default 1000, 0 reads every row) sets the analysis_limit under which the replicator's write worker, the replica's one writer, alone gathers its statistics, in its own thread (services/replicator/schema/statistics.ts):

• When it opens the replica, it runs ANALYZE main if _zero.statistics (replica schema v19) records no limit or another one, then records the limit; otherwise it runs PRAGMA optimize over every table, as before. Statistics an earlier zero-cache left are recorded as NULL and kept under 1000, the limit it set, so an upgrade at the default keeps them.
• After a replicated schema change, it runs PRAGMA optimize.
• Both PRAGMA optimize passes run SQLite's default mask without 0x10, and the limit is recorded, because PRAGMA optimize swaps a limit of 0, or of 2000 and above, for a temporary one of at most 2000 under that bit, and re-analyzes an analyzed table only once its size changes tenfold.
• Every other connection to the replica opens it read-only or closes with {optimize: false}, and an initial sync's ChangeProcessor runs no pass. So in a deployment without a backup, the change-streamer's poller no longer analyzes a table replicated empty and filled since; the replicator does at its next start or schema change.
• When the limit changes, the next start analyzes every table in one transaction before the replicator is ready. Under 0 that pass, and each schema change's pass inside its commit, read every row of the tables they analyze, about 750 ms per million rows with two indexes: the cost chore(zero-cache): Run PRAGMA optimize instead of analyze #6426 took off startup, now opt-in. A view-syncer under another limit than the replication-manager's analyzes every table at each restore.

Tests

packages/zero-cache/src/services/replicator/schema/statistics.test.ts starts the replicator as server/replicator.ts does, setupReplica and then the write worker, on the published harnesses. It covers:

• the option's default, environment variable, flag and bounds;
• a synced replica under limits 0, 1000 and 5000, a changed limit, a table grown tenfold, a table left without statistics that the poller polls past, an earlier zero-cache's statistics, and a rollback past v19 and forward;
• the poller, analyzeQuery, the preparation, the serving copy's source and the write worker's stop, none of which gathers statistics;
• a replicated index under limits 0 and 5000, and an initial sync, which gathers none;
• /plannerz reporting the record, or none for a replica that predates it.

With only the new module added to main, every case fails but the tenfold one, which main's preparation already passes.

Verification

• statistics.test.ts 25 passed · pnpm run lint clean · oxfmt --check clean · syncpack lint clean · check-types passes in every package; turbo's zbugs#build, which it runs first, fails on Windows on main too.
• Not run here: the change sources' and the restore's closes and the inspector's read-only open, which need Postgres, a custom change source, a litestream backup or a view-syncer; and zero-cache's unpublished suites, which meet v19, _zero.statistics, getPragmaConfig's limit and the test harness's default limit.

[YevheniiKotyrlo]

Measurements behind the table in the description.

A standalone probe against @rocicorp/zero-sqlite3 1.1.7 (SQLite 3.54.0), the build main installs: an in-memory table of 1,000,000 rows, deleted 0 on 99% of them, 2,000 groups, with groups 1–10 holding only the oldest 10,000 rows, as a small or closed group's would be. The same table is analysed under analysis_limit = 1000 and under 0, then each scope runs 15 times after one warm-up. AMD Ryzen 9 9950X3D, Windows 11 Pro, Node 26.8.1.

analysis_limit	ANALYZE	item_deleted_created stat1	item_group stat1	Plan	Five spread groups, median (min–max)	Five oldest groups, median (min–max)
1000	1.9 ms	1000000 1001 1 1	1000000 501	item_deleted_created (deleted=?), bloom filter	8.23 ms (7.99–8.33)	457.78 ms (417.28–501.95)
0	749.3 ms	1000000 500000 1 1	1000000 500	item_group (group_id=?), temp B-tree for ORDER BY	1.93 ms (1.86–2.51)	3.14 ms (2.60–3.28)

The sample reads deleted as about 1,000 rows per value, so walking the index that serves the ORDER BY looks nearly free; it is cheap only while the scope's newest rows come early in that order. Full statistics give the true 500,000, and SQLite seeks the groups instead.

probe.mjs

// What SQLite's statistics and plan are for a large table led by a low-cardinality column, gathered
// under an analysis limit and without one, and what each plan costs.
//
//   node probe.mjs <zero-sqlite3 package folder> [rows] [runs]
//
// The table models a soft-deleted list scoped to a few groups: `deleted` is 0 on almost every row, so
// an index led by it serves the ORDER BY while matching nearly the whole table, and the `group_id`
// index is the selective one. Groups 1-10 hold only the oldest rows, as a small or closed group would,
// so an ordered walk reaches their rows last.
import {createRequire} from 'node:module';
import path from 'node:path';

const [packageFolder, rowsArgument = '1000000', runsArgument = '15'] = process.argv.slice(2);
if (packageFolder === undefined) {
  throw new Error('usage: node probe.mjs <zero-sqlite3 package folder> [rows] [runs]');
}
const Database = createRequire(import.meta.url)(path.resolve(packageFolder));
const ROWS = Number(rowsArgument);
const RUNS = Number(runsArgument);
const GROUPS = 2000;
const OLD_GROUPS = 10;
const OLD_ROWS = 10000;
const SCOPES = {spread: JSON.stringify([107, 501, 902, 1303, 1804]), oldest: JSON.stringify([1, 2, 3, 4, 5])};
const QUERY = `SELECT id FROM item
  WHERE deleted = 0 AND group_id IN (SELECT value FROM json_each(?))
  ORDER BY created_at DESC, id DESC LIMIT 50`;

const buildDatabase = () => {
  const database = new Database(':memory:');
  database.exec(`
    CREATE TABLE item (id INTEGER PRIMARY KEY, deleted INTEGER NOT NULL, group_id INTEGER NOT NULL, created_at INTEGER NOT NULL);
    CREATE INDEX item_deleted_created ON item (deleted, created_at, id);
    CREATE INDEX item_group ON item (group_id);
  `);
  const insert = database.prepare('INSERT INTO item VALUES (?, ?, ?, ?)');
  database.transaction(() => {
    for (let id = 1; id <= ROWS; id++) {
      const groupId = id <= OLD_ROWS ? (id % OLD_GROUPS) + 1 : ((id * 7919) % (GROUPS - OLD_GROUPS)) + OLD_GROUPS + 1;
      insert.run(id, id % 100 === 0 ? 1 : 0, groupId, id);
    }
  })();
  return database;
};

const median = values => values.toSorted((left, right) => left - right)[Math.floor(values.length / 2)];

const measure = (database, scope) => {
  const statement = database.prepare(QUERY);
  statement.all(scope);
  const samples = Array.from({length: RUNS}, () => {
    const start = performance.now();
    statement.all(scope);
    return performance.now() - start;
  });
  return {
    medianMs: Number(median(samples).toFixed(2)),
    minMs: Number(Math.min(...samples).toFixed(2)),
    maxMs: Number(Math.max(...samples).toFixed(2)),
  };
};

const report = analysisLimit => {
  const database = buildDatabase();
  database.pragma(`analysis_limit = ${analysisLimit}`);
  const start = performance.now();
  database.exec('ANALYZE');
  const analyzeMs = Number((performance.now() - start).toFixed(1));
  const stat1 = database.prepare("SELECT idx, stat FROM sqlite_stat1 WHERE tbl = 'item' ORDER BY idx").all();
  const plan = database
    .prepare(`EXPLAIN QUERY PLAN ${QUERY}`)
    .all(SCOPES.spread)
    .map(row => row.detail);
  const timing = Object.fromEntries(Object.entries(SCOPES).map(([name, scope]) => [name, measure(database, scope)]));
  database.close();
  return {analysisLimit, analyzeMs, stat1, plan, timing};
};

console.log(JSON.stringify({sqlite: new Database(':memory:').prepare('SELECT sqlite_version() AS v').get().v, rows: ROWS, runs: RUNS}));
for (const analysisLimit of [1000, 0]) {
  console.log(JSON.stringify(report(analysisLimit), null, 2));
}

Run with node probe.mjs node_modules/@rocicorp/zero-sqlite3 1000000 15.