Native INSERT: ~+40% via bulk column encoders + streamed Native blocks - #143
Merged
Conversation
Two changes to the Native write path, targeting a 30% INSERT throughput gain: - Bulk column encoders for the non-numeric scalars that dominated encoding: String via a compiled write_string_column (varint + UTF-8 into one bytearray, pure fallback included), Date and naive DateTime straight from a stdlib array. Array(String) benefits too (its flat child encodes as String). The encode step goes ~645k -> ~1.46M rows/sec on the mixed-type benchmark. - Stream the INSERT body as a sequence of Native blocks (rows_to_native_stream, an async generator passed straight to the HTTP backend as a chunked body) instead of one pre-encoded blob. ClickHouse then inserts one block while the client encodes the next, overlapping the client-side encode with the server-side insert — the bulk of the win. Result: Native INSERT ~330k -> ~450k rows/sec end-to-end (load-invariant +39% in a same-process A/B vs a single-blob insert), now ahead of clickhouse-connect and clickhouse-driver on this workload. The split body is not atomic on a client-side encoding error (documented); valid uniformly-typed rows never hit that. Full suite 921 passed (both HTTP backends), pure-Python fallback green. perf_insert/ holds the iteration log; docs and benchmarks refreshed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…omicity The streamed multi-block INSERT trades cross-block atomicity for throughput: if encoding raises partway through a large insert, the blocks already sent are committed. Make that an explicit, controllable choice instead of a silent one: - New ChClient(insert_block_size=...) (default 8192). Inserts that fit in one block stay all-or-nothing; insert_block_size=0 always sends a single atomic block (encode fully, then POST) at the cost of the encode/insert overlap. - Corrected the rows_to_native_stream docstring (an error can surface mid-stream on a single bad value, not only on the first block) and documented the trade-off in the README and CHANGELOG. Tests cover both the multi-block streaming path and the atomic single-block path. Full suite 929 passed, pure fallback green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A performance pass on the Native write path: INSERT throughput goes from ~330k to ~450k rows/sec on the mixed-type benchmark (50k rows, M1 Pro, CH 26.5).
Changes
write_string_column(varint + UTF-8 into one bytearray, pure fallback included), Date and naive DateTime straight from a stdlibarray.Array(String)benefits too. Encode step: ~645k → ~1.46M rows/sec.rows_to_native_stream, an async generator handed to the HTTP backend as a chunked body). ClickHouse inserts one block while the client encodes the next, overlapping the client-side encode with the server-side insert — this is the bulk of the win (load-invariant +39% vs a single-blob insert in a same-process A/B).