Spill large sorts to disk; cap the page cache for fixed-heap builds#40
Open
UKTailwind wants to merge 9 commits into
Open
Spill large sorts to disk; cap the page cache for fixed-heap builds#40UKTailwind wants to merge 9 commits into
UKTailwind wants to merge 9 commits into
Conversation
The default allocator hands SQLite one gc_alloc() block per allocation,
so SQLite's structures live on MicroPython's GC heap. The conservative
collector cannot follow SQLite's interior/tagged pointers, so a
gc.collect() -- explicit, or automatic under memory pressure -- while a
connection is open frees live SQLite memory: SQLITE_CORRUPT ("malformed
database schema") or a hard hang. Bulk inserts die once auto-GC fires
under the accumulated garbage.
Switch to SQLite's MEMSYS5 pool (SQLITE_CONFIG_HEAP): one block the GC
never sub-collects, so gc.collect() is safe with a live connection. The
pool is reserved lazily on the first connect() -- a program that never
opens a database reserves nothing -- rooted via MP_REGISTER_ROOT_POINTER
so the GC keeps it. MEMSYS5_HEAP_SIZE defaults to a small 128 KB so it
fits constrained targets (a plain RP2040/ESP32 has only a few hundred KB
of RAM); a board with more RAM raises it, e.g. -DMEMSYS5_HEAP_SIZE=0x400000.
The page cache defaults to half the pool so it always fits inside it.
Also fix two bugs that stopped the MEMSYS5 branch working at all: the
undefined HEAP_SIZE (-> MEMSYS5_HEAP_SIZE), and the SQLITE_CONFIG_HEAP
minimum-allocation argument, which was 0 -- MEMSYS5 turns that into a
1-byte atom and cripples the buddy allocator; pass 64.
Verified on rp2 (RP2350, 8 MB PSRAM, 4 MB pool) and the unix port: an
8000-row insert with gc.collect() forced mid-transaction runs at flat
memory with no corruption.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
execute() registered every cursor in a strong-referenced list on the connection and removed it only in __del__; but cursors have no GC finaliser and are pinned by that list, so a loop of `con.execute(insert, row)` accumulated one prepared statement (~2 KB VDBE) per row until the connection closed, and the GC could never reclaim any of it. (With SQLite on the GC heap this also meant auto-GC fired ever sooner and corrupted the database.) Track a cursor in the connection list only while it holds a live statement -- idempotent register/deregister guarded by a `registered` flag -- and autoclose statements that return no rows: after the first step, if sqlite3_column_count()==0 (INSERT/UPDATE/DELETE/DDL) finalize the statement inline and drop the cursor. rowcount is captured first and lastrowid reads from the connection, so both survive on the returned cursor; SELECT cursors keep their statement for fetching. A finalized or result-less cursor now iterates empty and reports description=None instead of raising. Also fix connection.close(): it cleared cursors.items[0] on every iteration instead of items[i] and never reset len, leaving the list full of dangling pointers. With this (and the MEMSYS5 heap), a bulk insert in one transaction runs at flat memory with no chunk/close/reopen workaround -- ~3x faster on rp2 in testing (610 vs ~212 rows/s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Row is meant to be a tuple of the column values with one extra hidden slot holding the cursor, which usqlite_row_attr reads at items[len] to build .keys. Three bugs left it unusable (row_type defaults to tuple, so they went unnoticed): - the row factory never stamped usqlite_row_type on the object, so it was a plain tuple and .keys raised AttributeError; - it left len = columns+1, so the trailing cursor slot leaked into indexing / len / iteration, and .keys read items[columns+1] out of bounds; - keys() used sqlite3_data_count() (0 once the fetch has stepped past the row) instead of sqlite3_column_count(). Stamp the type, set len=columns (the block stays columns+1 wide so the GC still keeps the cursor referenced), and count columns with sqlite3_column_count(). Now `row.keys` returns the column-name tuple and `dict(zip(row.keys, row))` works, with the cursor slot hidden. Note .keys is a property, not a method. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bindParameter() passed a NULL destructor to sqlite3_bind_text/blob, which is SQLITE_STATIC: SQLite keeps the caller-supplied pointer and does not copy. That pointer is into a MicroPython str/bytes object on the GC heap, so if the argument is transient -- built inline for the call and dropped afterwards -- the collector can free it while the prepared statement is still bound to it, and a later step() reads freed memory. Pass SQLITE_TRANSIENT so SQLite copies the bytes at bind time, matching what CPython's sqlite3 does and what callers expect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An open database file is a MicroPython stream object (io.open()), but the only reference to it lives in MPFILE.stream inside SQLite's sqlite3_file, which SQLite allocates from its own dedicated heap (MEMSYS5). The MicroPython GC does not walk SQLite's heap as object memory, so nothing strongly reachable keeps the stream alive: after a gc.collect() while a connection is open the stream could be reclaimed, and the next read/write would touch a freed object. It survived only by luck -- the collector conservatively scanning SQLite's heap and happening to spot the pointer -- which is memory-layout dependent and unreliable. Keep an explicit strong reference instead: pin every stream in a GC-visible list held from a MP_REGISTER_ROOT_POINTER on open, and swap-remove it on close. The unpin is pure C (no allocation, cannot raise) so it is safe on the close/finalize path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every SELECT kept its prepared statement open, pinned by the connection's cursor list, until the cursor or the connection was closed. Because the list holds a strong reference the cursor is never garbage, so its __del__ never runs and gc.collect() cannot reclaim it -- a long-lived connection running many SELECTs without closing each cursor accumulated open statements in the fixed SQLite heap until it hit "out of memory". The idiomatic `for row in con.execute(sql): ...` leaked one statement per call (observed: OOM after ~2200 iterations on an RP2350B with a 4 MB heap). Free the statement the moment stepping reaches SQLITE_DONE (cursor_finish in stepExecute), and drop the cursor from the connection's list. rowcount is left untouched and the result-column names are cached first, so the cursor stays usable for rowcount/lastrowid/.keys afterwards. This also subsumes the previous "finalize result-less statements after execute" special case, which is removed. Fully-consumed cursors (for/list/fetchall/fetchmany, and empty or result-less statements) now run at flat memory; a partially fetched cursor that is dropped without close() still holds its statement until the connection closes, as before. Verified on an RP2350B (flash and SD): 6000 idiomatic-loop iterations at constant sqlite3_memory_used(), .keys intact on rows from an exhausted cursor, rowcount preserved, and PRAGMA integrity_check ok throughout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The page cache defaulted to half the dedicated pool (2 MB on a 4 MB board). SQLite ties the sorter's spill threshold to the cache, so a big cache let an unindexed ORDER BY / GROUP BY / index build accumulate ~3 MB of temp b-tree in RAM before spilling -- climbing to the edge of the pool and thrashing (a 20k-row report could hang the board until reset). The sorter already spills to temp files through the VFS; it just spilled far too late. Cap the cache low (~256 KB, and never more than 1/8 of a small pool) so the temp b-tree spills to disk early and stays bounded. Measured on an RP2350B (4 MB pool): the 20k-row join+group+order report that previously thrashed for 150s+ now completes in ~19s at ~530 KB peak -- a >6x drop in peak memory with no speed cost (spilling to flash is cheap; even on SD the same query went from thrash-to-reset to a clean 20s). Because large sorts are now bounded sub-MB rather than pool-sized, the pool no longer has to be large to run them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SQLite asks the VFS for anonymous temporary files -- the sorter's
merge-spill and temp b-trees for large ORDER BY / GROUP BY / DISTINCT /
index builds -- by calling xOpen with a NULL name and expecting the VFS to
invent one. usqlite_file_open dereferenced that NULL (strlen/strcpy),
hard-faulting the board: a big enough sort didn't fail gracefully, it
crashed the machine (leaving orphaned journals behind).
Give anonymous temp opens a unique name in a temp directory instead, so
the sorter actually spills to disk and the sort completes at bounded
memory. The name defaults to USQLITE_TEMP_DIR (the flash root) and honours
sqlite3_temp_directory, so PRAGMA temp_store_directory can redirect temp
files (e.g. to '/sd' to spare flash). SQLite sets SQLITE_OPEN_DELETEONCLOSE
on these, so the existing close path removes them -- no temp files leak.
Supporting changes:
- xAccess now answers the SQLITE_ACCESS_READWRITE probe (via os.stat)
so PRAGMA temp_store_directory validates instead of being rejected as
"not a writable directory". Existence probes still return 0, so
file-creation and journal behaviour are unchanged.
- xFullPathname guards against a NULL name and bounds the copy.
- temp files (fresh unique names, and root-level paths that confuse the
existence probe) skip the exists() check and are created write-new.
Verified in the emulator (same 4 MB dedicated heap as the board): a
cross-join producing a 7.7 MB sort that previously segfaulted now
completes at ~1.2 MB peak, repeatably, with no leaked temp files and
PRAGMA integrity_check ok; an unwritable temp dir now raises a catchable
OSError instead of crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Large sorts can spill several megabytes to a temp file, and the churn is
better aimed at the removable SD card than at the soldered flash. When
neither the caller (PRAGMA temp_store_directory) nor sqlite3_temp_directory
has chosen a location, prefer USQLITE_TEMP_SD ('/sd') if it is mounted,
falling back to the flash root (USQLITE_TEMP_DIR) otherwise -- so a board
with a card spares its flash automatically, and one without still works.
Verified in the emulator (same dedicated heap as the board): with /sd
mounted a 7.7 MB sort spills to /sd (nothing on flash) and completes,
cleaned up after; with no /sd it falls back to the flash root without
crashing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author
|
Superseded by #41 (this branch is included there, plus the wider hardening and a regression suite). Happy to keep this open if piecemeal merging is preferred. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two improvements for large queries when SQLite runs in a fixed, board-sized dedicated heap. Both build on the MEMSYS5 heap from #38, so this PR is stacked on #38 — until that merges, the diff here also shows #38's six commits; the two commits unique to this PR are the last two (
config: cap the page cache…andvfs: support temp files…).config: cap the page cache. The default page cache was half the pool. SQLite ties the sorter's spill threshold to the cache, so a big cache let an unindexed
ORDER BY/GROUP BY/ index build accumulate megabytes of temp b-tree in RAM before spilling — climbing to the edge of the pool and thrashing (a 20k-row grouped report could hang the board until reset). Capping the cache low (~256 KB, never more than 1/8 of a small pool) makes the temp b-tree spill early, so big sorts stay bounded (sub-MB) with no measured speed cost. On a 4 MB pool the same report went from a 150 s+ thrash to a clean ~19 s at ~0.5 MB.vfs: support temp files (and stop crashing). SQLite asks the VFS for anonymous temporary files — the sorter's merge spill and temp b-trees for large sorts/index builds — by calling
xOpenwith a NULL name. The VFS dereferenced that NULL (strlen/strcpy), hard-faulting the board: a big enough sort didn't fail gracefully, it crashed the machine. Temp opens now get a unique name in a temp directory, so the sorter spills to disk and large sorts complete at bounded memory instead of crashing. Defaults to the flash root and honoursPRAGMA temp_store_directory(e.g.'/sd'to spare flash);SQLITE_OPEN_DELETEONCLOSEcleans the files up. Also makesxAccessanswer the writable-directory probe, soPRAGMA temp_store_directoryvalidates instead of being rejected.Verified on an RP2350B (4 MB dedicated heap): a cross-join producing a ~5 MB sort that previously crash-reset the board now completes at ~1.2 MB, spilling to a temp file that is created during the sort and removed after — on both LittleFS (flash) and FatFS (SD).
Note: the page-cache cap is an opinionated default tuned for a small fixed heap; if you'd rather not change the default, the temp-file commit stands on its own — happy to drop the cache commit.