Skip to content

Write document files synchronously by default - #30

Open
danielebarbaro wants to merge 1 commit into
mainfrom
fix/document-store-fork
Open

danielebarbaro wants to merge 1 commit into
mainfrom
fix/document-store-fork

Conversation

@danielebarbaro

Copy link
Copy Markdown
Collaborator

Closes #26.

addDocument() forked a child for every document whenever ext-pcntl was loaded. The child ended with exit(0), running the parent's shutdown functions and destructors, and children were reaped only in save().

Changes

  • Document files are written synchronously by default. New asyncWrites parameter (default false) on the VectorDatabase constructor and open() opts back in.
  • Async children end with posix_kill(getmypid(), SIGKILL), so inherited shutdown functions and destructors never run. Async therefore needs ext-pcntl and ext-posix, and falls back to sync writes otherwise.
  • Finished children are reaped with WNOHANG on every async write.
  • A child can no longer report failure through its exit status, so the parent checks that the file exists when reaping and throws a RuntimeException if not. Before, a failed write went unnoticed and an exception in the child unwound into the caller's code.

Tests

tests/Persistence/DocumentStoreTest.php, including a subprocess test that registers a shutdown function and a destructor and asserts only the parent PID runs them. The same script against the previous DocumentStore logs both from the child PID too.

Notes

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Comparison

🔴 Significant regressions detected (>5%) — 34 metrics compared

xs

Metric Baseline Current Delta Status
insert (ops/s) 19.20 ops/s 22.79 ops/s +18.7% 🟢
insert (memory delta) 48.00 MB 48.00 MB +0.0% 🟢
vector_search (QPS) 224.27 queries/s 240.44 queries/s +7.2% 🟢
vector_search (memory delta) 0.00 MB 0.00 MB N/A 🟢
text_search (QPS) 429.49 queries/s 506.05 queries/s +17.8% 🟢
text_search (memory delta) 2.00 MB 2.00 MB +0.0% 🟢
hybrid_search (QPS) 128.25 queries/s 144.74 queries/s +12.9% 🟢
hybrid_search (memory delta) 0.00 MB 0.00 MB N/A 🟢
update (ops/s) 16.62 ops/s 19.34 ops/s +16.4% 🟢
update (memory delta) 8.00 MB 8.00 MB +0.0% 🟢
delete (ops/s) 728,397.90 ops/s 771,360.32 ops/s +5.9% 🟢
delete (memory delta) 0.00 MB 0.00 MB N/A 🟢
save (MB/s) 0.02 MB/s 0.03 MB/s +50.0% 🟢
save (disk size) 13.12 MB 13.12 MB +0.0% 🟢
save (memory delta) 56.00 MB 56.00 MB +0.0% 🟢
open (MB/s) 79.34 MB/s 100.43 MB/s +26.6% 🟢
open (memory delta) 26.00 MB 26.00 MB +0.0% 🟢

small

Metric Baseline Current Delta Status
insert (ops/s) 18.84 ops/s 22.84 ops/s +21.2% 🟢
insert (memory delta) 0.00 MB 0.00 MB N/A 🟢
vector_search (QPS) 220.38 queries/s 250.04 queries/s +13.5% 🟢
vector_search (memory delta) 0.00 MB 0.00 MB N/A 🟢
text_search (QPS) 426.40 queries/s 513.30 queries/s +20.4% 🟢
text_search (memory delta) 0.00 MB 0.00 MB N/A 🟢
hybrid_search (QPS) 127.96 queries/s 148.27 queries/s +15.9% 🟢
hybrid_search (memory delta) 0.00 MB 0.00 MB N/A 🟢
update (ops/s) 16.31 ops/s 19.20 ops/s +17.7% 🟢
update (memory delta) 0.00 MB 0.00 MB N/A 🟢
delete (ops/s) 657,145.50 ops/s 741,301.38 ops/s +12.8% 🟢
delete (memory delta) 0.00 MB 0.00 MB N/A 🟢
save (MB/s) 0.02 MB/s 0.03 MB/s +50.0% 🟢
save (disk size) 13.12 MB 13.12 MB +0.0% 🟢
save (memory delta) 2.00 MB 0.00 MB -100.0% 🟢
open (MB/s) 65.14 MB/s 82.35 MB/s +26.4% 🟢
open (memory delta) 2.00 MB 4.00 MB +100.0% 🔴

@danielebarbaro

Copy link
Copy Markdown
Collaborator Author

The flagged row is a 2 MB shift between phases, not extra memory. In the small scenario save (memory delta) dropped from 2 MB to 0 MB while open (memory delta) rose from 2 MB to 4 MB, so the total is unchanged. PHP's allocator grows in 2 MB chunks, which is why a single chunk moving from one phase to the next reads as +100%.

#29 and #30 show exactly the same pattern while touching unrelated code, and neither changes the open() path, so the baseline most likely predates a change already on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Drop per-document fork in DocumentStore::write()

1 participant