build: add a wa-sqlite rebuild recipe with a 125-database attach limit

README.md

@@ -160,6 +160,26 @@ registerMemoryWriteAheadSharedFiles({
 
 Structured cloning shares the underlying `SharedArrayBuffer`s; it does not copy the database bytes.
 
+## Raising the attached-database limit
+
+The `wa-sqlite.wasm` published in `@journeyapps/wa-sqlite` 1.7.0 keeps SQLite's default `SQLITE_MAX_ATTACHED=10`, so one connection can attach at most 10 databases. [`wasm/Dockerfile`](https://github.com/whygee-dev/memory-write-ahead-vfs/blob/main/wasm/Dockerfile) rebuilds the same wa-sqlite source with a higher limit, 125 by default, which is SQLite's ceiling:
+
+```bash
+pnpm build:wasm
+```
+
+This writes `wasm/dist/wa-sqlite.mjs` and `wasm/dist/wa-sqlite.wasm`, then checks both against [`wasm/SHA256SUMS`](https://github.com/whygee-dev/memory-write-ahead-vfs/blob/main/wasm/SHA256SUMS). For another limit, run Docker directly; the hash check only applies to the default:
+
+```bash
+docker build --build-arg SQLITE_MAX_ATTACHED=32 --output wasm/dist wasm
+```
+
+Copy both files into your app and import the rebuilt `wa-sqlite.mjs` in place of `@journeyapps/wa-sqlite/dist/wa-sqlite.mjs`. Keep `@journeyapps/wa-sqlite` installed for `SQLite.Factory` and the VFS base classes. The glue loads `wa-sqlite.wasm` from its own directory unless you pass `locateFile` or `wasmBinary` to the factory.
+
+The recipe pins the inputs that decide the output bytes: the `emscripten/emsdk:4.0.9` image by digest, [powersync-ja/wa-sqlite](https://github.com/powersync-ja/wa-sqlite) at `e4bb5532c8`, the commit published as `@journeyapps/wa-sqlite` 1.7.0 (SQLite 3.53.0, PowerSync core v0.4.13), and the SHA-256 of the downloaded PowerSync core library and of the generated SQLite amalgamation. It builds only the synchronous `wa-sqlite` variant this VFS uses, with the upstream flags (`-Oz -flto`, `WASM_BIGINT=0`); the attach limit is the only configuration change. Two builds with `--no-cache` on the same host produced byte-identical files; other hosts were not tried.
+
+The npm artifact is not the same build with one flag changed: its compile options report clang 23, while the recipe uses Emscripten 4.0.9 (clang 21), the version the fork's CI pins. Rebuild for the attach limit, not for speed: in [the comparison benchmark](https://github.com/whygee-dev/memory-write-ahead-vfs/blob/main/benchmarks/wasm-build-comparison/README.md), the rebuilt build ran within 5% of the npm build on every SQL workload, slightly slower on each.
+
 ## Capacity options
 
 ```ts

benchmarks/wasm-build-comparison/README.md

@@ -0,0 +1,77 @@
+# wa-sqlite build comparison
+
+This benchmark compares the `wa-sqlite.wasm` published in `@journeyapps/wa-sqlite` 1.7.0 ("stock") with the one built by [`wasm/Dockerfile`](../../wasm/Dockerfile) ("rebuilt"). Both arms run the same SQL through Memory Write-Ahead VFS in a dedicated worker, with the same wa-sqlite JavaScript API; only the Emscripten glue and the wasm binary change.
+
+## What differs between the two builds
+
+|                                            | stock                              | rebuilt                                    |
+| ------------------------------------------ | ---------------------------------- | ------------------------------------------ |
+| Source                                     | npm `@journeyapps/wa-sqlite` 1.7.0 | [`wasm/Dockerfile`](../../wasm/Dockerfile) |
+| SQLite                                     | 3.53.0                             | 3.53.0                                     |
+| Compiler in `PRAGMA compile_options`       | clang-23.0.0                       | clang-21.0.0                               |
+| `MAX_ATTACHED` in `PRAGMA compile_options` | 10                                 | 125                                        |
+| `wa-sqlite.wasm` size                      | 1,128,032 bytes                    | 1,129,991 bytes                            |
+| `wa-sqlite.wasm` SHA-256                   | `d167becf5a69…`                    | `91defe9a35e0…`                            |
+
+A difference between the arms can come from the compiler or from the attach limit; this benchmark does not separate the two.
+
+## Result
+
+Run on 2026-10-07 with Chromium 148.0.7778.96 on an Apple M5 Pro (18 logical CPUs, darwin arm64), 16 runs per arm. The machine was not idle: the 1-minute load average was 8.43 when the benchmark started and 7.56 when it ended.
+
+| Workload          | stock, ms: median (min–max) | rebuilt, ms: median (min–max) | rebuilt against stock |
+| ----------------- | --------------------------- | ----------------------------- | --------------------- |
+| `filterScan`      | 46.72 (43.83–52.36)         | 46.89 (45.01–50.15)           | +0.4%                 |
+| `join`            | 11.86 (11.50–15.02)         | 12.10 (11.46–15.28)           | +2.0%                 |
+| `groupedLeftJoin` | 12.80 (11.05–14.01)         | 13.44 (12.57–14.94)           | +5.0%                 |
+| `orderByText`     | 45.63 (41.88–54.92)         | 47.01 (44.01–50.61)           | +3.0%                 |
+| `insert`          | 23.89 (22.78–25.95)         | 24.48 (23.31–26.06)           | +2.5%                 |
+| `seed`            | 164.15 (155.18–184.40)      | 167.60 (158.62–172.60)        | +2.1%                 |
+| `instantiate`     | 3.93 (3.62–4.43)            | 3.26 (3.00–3.51)              | −17.0%                |
+
+The rebuilt build is not faster. On the five SQL workloads and the seed, its median is 0.4% to 5.0% above the stock build's, and the two ranges overlap on every one of them. Instantiation is 0.67 ms shorter. Every run of both arms returned the same row counts and the same result hashes.
+
+## Workload
+
+A seeded generator in [`dataset.ts`](./dataset.ts) builds the same database in every run: 1,000 categories and 30,000 items with 20 columns mixing text, integers, reals, and nullable values, plus an index on `item.category_id`.
+
+| Workload          | What runs                                                                                               |
+| ----------------- | ------------------------------------------------------------------------------------------------------- |
+| `filterScan`      | `SELECT *` with a filter on two unindexed columns, returning 17,900 of the 30,000 rows                  |
+| `join`            | items joined to their category, filtered on three of the 20 category families, ordered by item id       |
+| `groupedLeftJoin` | every category left-joined to its active items, grouped with `COUNT`, `SUM`, `AVG`, `MAX`               |
+| `orderByText`     | 4 columns of all 30,000 items ordered by a text column, without an index                                |
+| `insert`          | 5,000 rows with all 20 columns in one transaction through one prepared statement, into an indexed table |
+| `instantiate`     | the glue's module factory compiling and instantiating the wasm bytes, once per run                      |
+| `seed`            | creating the schema and inserting the 31,000 rows and the index in one transaction, once per run        |
+
+Each read is timed from `prepare` until every row has been read into JavaScript values with `sqlite3.row`, so the time includes wa-sqlite's per-row JavaScript overhead, which is the same in both arms.
+
+## Method
+
+- One run opens a new browser context and page, starts a dedicated worker, loads one arm's glue, fetches its wasm bytes, and passes them as `wasmBinary`. The worker opens a Memory Write-Ahead VFS database in WAL mode with a 64 MiB page cache, which holds the whole database, so warm reads do not go through the VFS read path.
+- Each run executes 2 warmup rounds, which are discarded, then 5 measured rounds. A round runs the four reads, then the insert. A run's value for a workload is the median of its 5 measured rounds; the table reports the median, minimum, and maximum of those values across runs.
+- Arms alternate in ABBA and BAAB blocks so that each arm runs equally often early and late. One priming run per arm, before the first block, is discarded.
+- Every run hashes each read result and the inserted rows; all runs of both arms must produce the same hashes. The worker also hashes the wasm bytes it loaded and the test checks them against the files on disk.
+- [Raw runs](./results/npm-1.7.0-vs-rebuilt.json) include every measured round and the 1-minute load average when each run started.
+
+## What this does not measure
+
+- Attached databases: the workload uses one database, so it does not exercise the code paths that the higher attach limit changes.
+- Other wa-sqlite variants (asynchronous, JSPI, multiple ciphers), other VFS implementations, other browsers, or other machines.
+- Memory use, download size, page-load time, or the cost of fetching the wasm over a network.
+- Concurrency between workers or connections; every run uses one connection.
+- Prepared-statement reuse for reads; each read prepares its statement again.
+
+## Reproduce
+
+Requirements: Docker, Node.js 20+, pnpm 10.27.0, and Playwright Chromium.
+
+```bash
+pnpm install --frozen-lockfile
+pnpm exec playwright install chromium
+pnpm build:wasm
+pnpm benchmark:wasm
+```
+
+The raw result is written to `benchmark-results/wasm-build-comparison.json`. `BENCHMARK_RUNS_PER_ARM` (even, default 16) changes the number of measured runs, and `WASM_BENCHMARK_CANDIDATE_DIR` points the rebuilt arm at another directory inside the repository, for example a build with a different `SQLITE_MAX_ATTACHED`.

benchmarks/wasm-build-comparison/benchmark.spec.ts

@@ -0,0 +1,246 @@
+import { createHash } from 'node:crypto';
+import { existsSync, readFileSync } from 'node:fs';
+import { mkdir, writeFile } from 'node:fs/promises';
+import { cpus, loadavg, platform, arch } from 'node:os';
+import { dirname, isAbsolute, join, relative, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { expect, test } from '@playwright/test';
+import memoryWriteAheadPackage from '../../package.json' with { type: 'json' };
+
+import { DATASET, WORKLOAD } from './dataset.js';
+import type { WasmBuildRunRequest, WasmBuildRunResult, WorkloadName } from './worker.js';
+
+type ArmName = 'stock' | 'rebuilt';
+
+type Arm = {
+    directory: string;
+    name: ArmName;
+    source: string;
+};
+
+type Summary = {
+    max: number;
+    median: number;
+    min: number;
+    runs: number;
+};
+
+const REPOSITORY_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
+
+const readPositiveEvenInteger = (name: string, fallback: number) => {
+    const value = Number(process.env[name] ?? fallback);
+    if (!Number.isInteger(value) || value < 2 || value % 2 !== 0) {
+        throw new Error(`${name} must be a positive even integer; received ${process.env[name]}`);
+    }
+    return value;
+};
+
+const readPackageVersion = (packageName: string) => {
+    const packageJson = JSON.parse(
+        readFileSync(join(REPOSITORY_ROOT, 'node_modules', packageName, 'package.json'), 'utf8')
+    ) as { version: string };
+    return packageJson.version;
+};
+
+const sha256File = (path: string) => createHash('sha256').update(readFileSync(path)).digest('hex');
+
+const round = (value: number) => Math.round(value * 100) / 100;
+
+const median = (values: readonly number[]) => {
+    const sorted = [...values].sort((left, right) => left - right);
+    const middle = Math.floor(sorted.length / 2);
+    const upper = sorted[middle] ?? Number.NaN;
+    return sorted.length % 2 === 0 ? ((sorted[middle - 1] ?? Number.NaN) + upper) / 2 : upper;
+};
+
+const summarize = (values: readonly number[]): Summary => ({
+    max: round(Math.max(...values)),
+    median: round(median(values)),
+    min: round(Math.min(...values)),
+    runs: values.length,
+});
+
+const resolveCandidateDirectory = () => {
+    const directory = resolve(REPOSITORY_ROOT, process.env.WASM_BENCHMARK_CANDIDATE_DIR ?? 'wasm/dist');
+    const relativeDirectory = relative(REPOSITORY_ROOT, directory);
+    if (relativeDirectory.startsWith('..') || isAbsolute(relativeDirectory)) {
+        throw new Error('WASM_BENCHMARK_CANDIDATE_DIR must be inside the repository so Vite can serve it');
+    }
+    if (!existsSync(join(directory, 'wa-sqlite.mjs')) || !existsSync(join(directory, 'wa-sqlite.wasm'))) {
+        throw new Error(`${relativeDirectory} has no wa-sqlite.mjs and wa-sqlite.wasm; run pnpm build:wasm first`);
+    }
+    return relativeDirectory.split('\\').join('/');
+};
+
+const createSchedule = (runsPerArm: number): ArmName[] =>
+    Array.from({ length: runsPerArm / 2 }, (_, block): ArmName[] =>
+        block % 2 === 0 ? ['stock', 'rebuilt', 'rebuilt', 'stock'] : ['rebuilt', 'stock', 'stock', 'rebuilt']
+    ).flat();
+
+test('compares the stock and rebuilt wa-sqlite builds through Memory Write-Ahead VFS', async ({ browser, baseURL }) => {
+    test.setTimeout(60 * 60_000);
+
+    const runsPerArm = readPositiveEvenInteger('BENCHMARK_RUNS_PER_ARM', 16);
+    const arms: readonly Arm[] = [
+        {
+            directory: 'node_modules/@journeyapps/wa-sqlite/dist',
+            name: 'stock',
+            source: `npm @journeyapps/wa-sqlite ${readPackageVersion('@journeyapps/wa-sqlite')}`,
+        },
+        { directory: resolveCandidateDirectory(), name: 'rebuilt', source: 'wasm/Dockerfile' },
+    ];
+    const armByName = new Map(arms.map((arm) => [arm.name, arm]));
+    const artifacts = arms.map((arm) => ({
+        directory: arm.directory,
+        glueSha256: sha256File(join(REPOSITORY_ROOT, arm.directory, 'wa-sqlite.mjs')),
+        name: arm.name,
+        source: arm.source,
+        wasmSha256: sha256File(join(REPOSITORY_ROOT, arm.directory, 'wa-sqlite.wasm')),
+    }));
+
+    const runArm = async (armName: ArmName) => {
+        const arm = armByName.get(armName);
+        if (!arm) {
+            throw new Error(`Unknown arm ${armName}`);
+        }
+        const context = await browser.newContext({ baseURL: baseURL ?? 'http://127.0.0.1:5175' });
+        try {
+            const page = await context.newPage();
+            await page.goto('/test-host.html');
+            const request: WasmBuildRunRequest = {
+                glueUrl: `/${arm.directory}/wa-sqlite.mjs`,
+                wasmUrl: `/${arm.directory}/wa-sqlite.wasm`,
+            };
+            return await page.evaluate(
+                (runRequest) =>
+                    new Promise<WasmBuildRunResult>((resolveRun, rejectRun) => {
+                        const worker = new Worker('/benchmarks/wasm-build-comparison/worker.ts', { type: 'module' });
+                        worker.onmessage = (
+                            event: MessageEvent<
+                                { ok: true; result: WasmBuildRunResult } | { ok: false; message: string }
+                            >
+                        ) => {
+                            worker.terminate();
+                            if (event.data.ok) {
+                                resolveRun(event.data.result);
+                            } else {
+                                rejectRun(new Error(event.data.message));
+                            }
+                        };
+                        worker.onerror = (event) => {
+                            worker.terminate();
+                            rejectRun(new Error(event.message || 'Benchmark worker failed to load'));
+                        };
+                        worker.postMessage(runRequest);
+                    }),
+                request
+            );
+        } finally {
+            await context.close();
+        }
+    };
+
+    const loadAverageAtStart = loadavg().map(round);
+    for (const arm of arms) {
+        console.info(`[wasm-build-benchmark] priming ${arm.name}`);
+        await runArm(arm.name);
+    }
+
+    const schedule = createSchedule(runsPerArm);
+    const runs: Array<{ arm: ArmName; loadAverage1m: number; result: WasmBuildRunResult; run: number }> = [];
+    for (const [index, armName] of schedule.entries()) {
+        const loadAverage1m = round(loadavg()[0] ?? 0);
+        console.info(`[wasm-build-benchmark] run ${index + 1}/${schedule.length} ${armName}`);
+        runs.push({ arm: armName, loadAverage1m, result: await runArm(armName), run: index + 1 });
+    }
+    const loadAverageAtEnd = loadavg().map(round);
+
+    const firstRun = runs[0];
+    if (!firstRun) {
+        throw new Error('The benchmark recorded no runs');
+    }
+    for (const run of runs) {
+        expect(run.result.crossOriginIsolated).toBe(true);
+        expect(run.result.fingerprints).toEqual(firstRun.result.fingerprints);
+        expect(run.result.rowCounts).toEqual(firstRun.result.rowCounts);
+        expect(run.result.build.wasmSha256).toBe(artifacts.find((artifact) => artifact.name === run.arm)?.wasmSha256);
+    }
+
+    const workloadNames = Object.keys(firstRun.result.measuredMs) as WorkloadName[];
+    const summaries = Object.fromEntries(
+        arms.map((arm) => {
+            const armRuns = runs.filter((run) => run.arm === arm.name).map((run) => run.result);
+            return [
+                arm.name,
+                {
+                    ...Object.fromEntries(
+                        workloadNames.map((name) => [
+                            name,
+                            summarize(armRuns.map((run) => median(run.measuredMs[name]))),
+                        ])
+                    ),
+                    instantiate: summarize(armRuns.map((run) => run.instantiateMs)),
+                    seed: summarize(armRuns.map((run) => run.seedMs)),
+                },
+            ];
+        })
+    );
+
+    const result = {
+        resultFormat: 'wasm-build-comparison-v1',
+        runDate: new Date().toISOString().slice(0, 10),
+        environment: {
+            browser: `Chromium ${browser.version()}`,
+            crossOriginIsolated: firstRun.result.crossOriginIsolated,
+            hardwareConcurrency: firstRun.result.hardwareConcurrency,
+            host: {
+                arch: arch(),
+                cpuModel: cpus()[0]?.model ?? 'unknown',
+                logicalCpus: cpus().length,
+                platform: platform(),
+            },
+            loadAverageAtEnd,
+            loadAverageAtStart,
+        },
+        runtimeVersions: {
+            memoryWriteAheadVfs: memoryWriteAheadPackage.version,
+            playwright: readPackageVersion('@playwright/test'),
+            vite: readPackageVersion('vite'),
+            waSqlite: readPackageVersion('@journeyapps/wa-sqlite'),
+        },
+        configuration: {
+            dataset: DATASET,
+            primingRunsPerArm: 1,
+            runsPerArm,
+            schedule: 'ABBA/BAAB blocks, fresh browser context, page and worker per run',
+            summaryUnit: 'per-run median of measured rounds; instantiate and seed are one sample per run',
+            vfs: 'MemoryWriteAheadVFS in a dedicated worker',
+            workload: WORKLOAD,
+        },
+        arms: artifacts.map((artifact) => {
+            const build = runs.find((run) => run.arm === artifact.name)?.result.build;
+            return { ...artifact, ...build };
+        }),
+        fingerprints: firstRun.result.fingerprints,
+        rowCounts: firstRun.result.rowCounts,
+        summaries,
+        runs: runs.map(({ arm, loadAverage1m, result: { instantiateMs, measuredMs, seedMs }, run }) => ({
+            arm,
+            instantiateMs: round(instantiateMs),
+            loadAverage1m,
+            measuredMs: Object.fromEntries(
+                Object.entries(measuredMs).map(([name, values]) => [name, values.map(round)])
+            ),
+            run,
+            seedMs: round(seedMs),
+        })),
+    };
+
+    const outputPath = resolve(
+        REPOSITORY_ROOT,
+        process.env.BENCHMARK_OUTPUT ?? 'benchmark-results/wasm-build-comparison.json'
+    );
+    await mkdir(dirname(outputPath), { recursive: true });
+    await writeFile(outputPath, `${JSON.stringify(result, null, 2)}\n`, 'utf8');
+    console.info(`[wasm-build-benchmark] ${JSON.stringify(summaries)}`);
+});