Found while reviewing #1 for efficiency; measured, not reasoned.
load.load_files runs load_file strictly sequentially, and each load_file is ~2 s of pure-Python parsing followed by ~4 s of DuckDB insertion (slurm/README.md). The two never overlap, so a third of every file's time has one side idle.
DuckDB's Python client releases the GIL during execute — verified: a CPU-bound Python thread ran at 84% of its solo speed alongside a 0.34 s DuckDB query. So parsing file N+1 in a thread genuinely overlaps the insert of file N; no subprocess and no pickling.
Sketch: a one-slot prefetch with ThreadPoolExecutor(max_workers=1) — submit parse_file(next_path) before calling load_parsed for the current file, then future.result() at the top of the next iteration. load_parsed/load_file need no change and the DuckDB connection stays on the main thread.
Payoff: hides ~2 s of every ~6 s file, so roughly 40 minutes off a ~1,300-file, 2–3 hour load.
Cost: one extra parsed file resident (~0.9 GiB today; less if #25 lands first, which would also make this the cheaper of the two to hold). Worth confirming the peak-RSS effect against slurm/README.md's sizing before changing what that document recommends.
Do #25 first — it shrinks the thing this holds a second copy of.
Found while reviewing #1 for efficiency; measured, not reasoned.
load.load_filesrunsload_filestrictly sequentially, and eachload_fileis ~2 s of pure-Python parsing followed by ~4 s of DuckDB insertion (slurm/README.md). The two never overlap, so a third of every file's time has one side idle.DuckDB's Python client releases the GIL during
execute— verified: a CPU-bound Python thread ran at 84% of its solo speed alongside a 0.34 s DuckDB query. So parsing file N+1 in a thread genuinely overlaps the insert of file N; no subprocess and no pickling.Sketch: a one-slot prefetch with
ThreadPoolExecutor(max_workers=1)— submitparse_file(next_path)before callingload_parsedfor the current file, thenfuture.result()at the top of the next iteration.load_parsed/load_fileneed no change and the DuckDB connection stays on the main thread.Payoff: hides ~2 s of every ~6 s file, so roughly 40 minutes off a ~1,300-file, 2–3 hour load.
Cost: one extra parsed file resident (~0.9 GiB today; less if #25 lands first, which would also make this the cheaper of the two to hold). Worth confirming the peak-RSS effect against
slurm/README.md's sizing before changing what that document recommends.Do #25 first — it shrinks the thing this holds a second copy of.