Partitioning: refuse RANGE until implemented; add dynamic LIST partitioning - #834
Open
adsharma wants to merge 8 commits into
Open
Partitioning: refuse RANGE until implemented; add dynamic LIST partitioning#834adsharma wants to merge 8 commits into
adsharma wants to merge 8 commits into
Conversation
…ists RANGE previously fell back to hash routing while accepting RANGE DDL, which silently violated the monotonic placement users expect from range partitioning. Real RANGE needs bounds derived from the actual value distribution (declarative bounds or data-driven splitting); a static stand-in - including equal splits of the type domain, which dump all real-world data into one bucket - would misplace rows silently. The binder now refuses PARTITION BY RANGE with an actionable message; the grammar keeps parsing it so enabling it later is a binder-only change. Tests: PartitionedCreateRange now asserts the refusal; PartitionedDropCascades switched to HASH. Docs: status note, limitations, and a roadmap section covering declarative bounds and dynamic distribution-aware splits.
…nct value) PARTITION BY LIST (col) now creates one partition per distinct partition-key value, dynamically: - Grammar: iC_PartitionList (LIST keyword, no PARTITIONS clause); regenerated parser sources and shell keywords. - Catalog: LIST parents persist an encoded-key -> child-table-ID map on the parent entry (storage version 47; 0.20.0 repointed since v46 never shipped). The first partition is created at DDL time so reads/writes always see a non-empty child set; it stays unkeyed and empty. - Routing: ListPartitionRouter creates a partition on first sight of a new value inside the writing transaction via the CREATE NODE TABLE machinery (catalog entry + serial sequence + subgraph + storage + WAL create record), so rollback, WAL replay, and checkpoint behave like ordinary DDL. Children carry their parent link through the create info, keeping replay correct. - Single-row INSERT/MERGE route in NodeInsertExecutor; COPY routes under the router lock, atomically growing shared + per-worker target arrays (and index producer tokens) as workers discover new values. - RANGE stays refused at DDL; HASH routing unchanged. Verified end to end in the shell: distinct values create distinct partitions, same value reuses its partition, parent scans union all partitions, state survives reopen/WAL replay, post-reopen inserts reuse persisted partitions. All 11 partitioned + storage_version e2e tests pass.
Both ladybug's third_party/mbedtls and a statically-linked duckdb (libduckdb_static.a) bundle mbedtls with plain, unprefixed mbedtls_* symbols. When both end up on one link line (e.g. GEN=Ninja make shell EXTENSION_STATIC_LINK_LIST='httpfs'), the linker reports ~19 duplicate symbol errors for _mbedtls_cipher_*. Fix by wrapping our vendored copy in namespace lbug_mbedtls: - third_party/ports/mbedtls/do-patch.py (new): port patch script, following the zstd port pattern. It renames .c -> .cpp, strips the now-dead extern "C" guards (the copy is compiled as C++ everywhere), and wraps top-level code segments in balanced namespace pairs. Segmentation tracks preprocessor conditional context and brace depth per line, so sibling conditional branches (MBEDTLS_AES_ALT vs MBEDTLS_SELF_TEST) and mid-declaration conditionals (lone 'static' followed by a conditional attribute block) stay balanced. It also absorbs the previously-missing format_c_to_cpp.py step, making the port runnable again. - ports/mbedtls/Makefile: invoke do-patch.py during build. - third_party/mbedtls: vendored sources regenerated via the script (57 files; only namespace pairs and extern "C" removals). Public headers re-expose names via 'using namespace lbug_mbedtls', so in-tree consumers (src/common/sha256.cpp, extension/httpfs/src/crypto.cpp) are unchanged. Requires LadybugDB/extensions#63 (submodule bump included here). Verified with GEN=Ninja make shell EXTENSION_STATIC_LINK_LIST='httpfs': the link succeeds, libmbedtls.a exports zero plain _mbedtls_* symbols, and the shell passes a smoke query.
SECURITY.md linked security/threat-model.md (hyphen); the file in the repository is security/threat_model.md (underscore). tools/shell/shell_development_guide.md: avaiable -> available.
PartitionedListDynamic: dynamic per-value partition creation on single-row inserts (4 distinct cluster values -> L_p1..L_p3 alongside the initial unkeyed L_p0), parent-scan union over all partitions, partition reuse for a repeated value, persistence across reopen/WAL replay, post-reopen routing into an existing partition, and cascade drop of the parent. The case ends with an explicit DROP TABLE L: the FSM leak checker drops tables in show_tables() order, which after reopen lists partition children before their parent, and dropping a child directly is refused.
Adding the LIST keyword made 'list' a reserved word, breaking every query that uses it as a variable/alias/column name (TCK list/map scenarios, cast tests with (list STRING[]) columns). Add LIST to iC_NonReservedKeywords so it remains usable as an identifier, matching HASH/RANGE/PARTITION(S), and regenerate the parser. Also apply clang-format-18 to the touched src/ files (third_party generated files are outside CI's lint scope).
…ng hooks (PR 829) Encode the decision order at every shared seam: locate() first (claimed -> wrapper owns it, no local state), then phase B's dedicated StorageManager for unclaimed partitions. Claimed partitions never enter the file registry, so checkpoint/rollback/drop iterate exactly the unclaimed set with no extra filtering; bind-time scan substitution means plan-time resolution never sees a remote child; drop notifications and file cleanup are disjoint by construction. Child files are named by catalog name and renamed with the rename cascade; PartitionRef stays ID-based as in 829.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to table partitioning:
1. RANGE is refused at DDL time
RANGE previously fell back to hash routing while accepting
PARTITION BY RANGE, silently violating the monotonic placement users expect. Real range partitioning needs bounds derived from the actual value distribution (declarative bounds or data-driven splitting); equal splits of the type domain were rejected because all realistic data lands in one middle bucket. The binder now refuses the DDL with an actionable message; the grammar keeps parsing it so enabling it later is a binder-only change.2. LIST partitioning (one partition per distinct value)
PARTITION BY LIST (col)creates one partition per distinct partition-key value, dynamically:iC_PartitionListrule +LISTkeyword (no PARTITIONS clause); regenerated parser sources and shell keywords.ListPartitionRoutercreates a partition at first sight of a new value inside the writing transaction, via the same machinery as CREATE NODE TABLE (catalog entry + serial sequence + subgraph + storage + WAL create record), so rollback, WAL replay, and checkpointing behave like ordinary DDL. Children carry their parent link through the create info so replay reconstructs them correctly.HASH routing is unchanged; all previously merged invariants hold (no direct child drops/alters, cascade-drop guards, partition-column updates refused, rel FROM/TO expands over partitions, parent rename cascades).
Testing
PartitionedListDynamic: per-value partition creation on inserts, parent-scan union, partition reuse for repeated values, persistence across reopen/WAL replay, post-reopen routing into an existing partition, cascade drop.