Partitioning: refuse RANGE until implemented; add dynamic LIST partitioning

SECURITY.md

@@ -14,4 +14,4 @@ To report a security issue, please email [security@ladybugdb.com](mailto:securit
 
 ## Threat Model
 
-Ladybug's current threat model is documented in [security/threat-model.md](security/threat-model.md).
+Ladybug's current threat model is documented in [security/threat_model.md](security/threat_model.md).

docs/partitioning.md

@@ -9,8 +9,10 @@ protocol such as ADBC).
 
 > Status: **v1 (foundational).** DDL, catalog, storage, persistence, drop-cascade and query
 > (read over all partitions) are implemented. Write-routing (COPY / CREATE / MERGE into the
-> parent) is implemented for HASH partitioning; RANGE reuses the same deterministic hash routing
-> until declarative range bounds land. Predicate-based partition pruning is still future work.
+> parent) is implemented for HASH partitioning. **`PARTITION BY RANGE` is refused at DDL time**:
+> real range partitioning must derive partition bounds from the actual value distribution, which
+> is not implemented yet — refusing beats silently falling back to hash routing. Predicate-based
+> partition pruning is still future work.
 >
 > Invariant-protection implemented in response to design review: partitions cannot be dropped or
 > altered individually, `DROP GRAPH` refuses node-table subgraphs, dropping a partitioned table is
@@ -130,15 +132,11 @@ CREATE NODE TABLE Orders (
     region STRING,
     amount INT64
 ) PARTITION BY HASH (region) PARTITIONS 4;
-
--- Range partitioning (bounds are derived; see design below).
-CREATE NODE TABLE Events (
-    id    INT64 PRIMARY KEY,
-    ts    TIMESTAMP,
-    value DOUBLE
-) PARTITION BY RANGE (ts) PARTITIONS 5;
 ```
 
+The grammar still accepts `PARTITION BY RANGE (<col>) PARTITIONS n`, but the binder refuses it:
+RANGE partitioning is not implemented yet (see the status note above).
+
 The grammar extension lives in `src/antlr4/Cypher.g4`:
 
 ```
@@ -233,10 +231,8 @@ Because each partition is a real node table, you can also address a specific par
 
 * **Write-routing.** `COPY INTO <parent>`, `CREATE (n:<parent>)` and `MERGE` against a
   partitioned parent are routed into the partition matching each row's partition-key value. HASH
-  partitions use the same value hashing as the built-in `HASH()` function; RANGE partitions
-  currently reuse that hash (there are no declarative bounds yet), which keeps writes
-  deterministic and findable because parent reads union over all partitions. Primary-key
-  uniqueness is enforced per partition, not across the parent.
+  partitions use the same value hashing as the built-in `HASH()` function. Primary-key uniqueness
+  is enforced per partition, not across the parent.
 * **No partition pruning on predicates.** A `WHERE` on the partition key is not yet used to skip
   partitions; all partitions are scanned and unioned.
 * **`ALTER` is limited to `RENAME`.** Renaming the parent renames its `<parent>_p<i>` partitions
@@ -246,8 +242,9 @@ Because each partition is a real node table, you can also address a specific par
   partitions; delete and re-insert instead.
 * **Rels attach per partition.** `FROM <parent>` expands to one pair per partition; rel pattern
   writes against the parent are refused (use a specific partition) until runtime rel routing lands.
-* **Range bounds are not declarative.** `PARTITIONS n` builds the range split; per-partition bound
-  lists (`PARTITION p0 VALUES < (...), p1 VALUES FROM ... `) are future work.
+* **RANGE is refused at DDL time.** Meaningful range partitioning needs bounds derived from the
+  actual value distribution (declarative bounds or data-driven splitting); a static stand-in would
+  misplace rows silently. See the roadmap.
 * **No remote partitions yet.** Only local (in-process) partition subgraphs exist today.
 
 ## Roadmap
@@ -258,17 +255,23 @@ The canonical path is `COPY INTO Orders FROM file`. Implemented:
 
 * `BoundCopyFromInfo`/`NodeBatchInsertInfo` carry the partition method, partition-key column id,
   and the child table list (`common::NodePartitionWriteInfo`).
-* `NodeBatchInsert` evaluates each row's partition-key value, computes `hash(value) % n` (for both
-  HASH and, for now, RANGE), and routes consecutive same-partition runs into the correct child
-  `NodeTable`'s node group. Each child is an ordinary `NodeTable`, so its own WAL/MVCC machinery
-  (`appendToLastNodeGroup` + commit/undo records) applies the routed write.
+* `NodeBatchInsert` evaluates each row's partition-key value, computes `hash(value) % n`, and
+  routes consecutive same-partition runs into the correct child `NodeTable`'s node group. Each
+  child is an ordinary `NodeTable`, so its own WAL/MVCC machinery (`appendToLastNodeGroup` +
+  commit/undo records) applies the routed write.
 * Primary-key duplicate detection is per partition (each child has its own PK index), matching how
   a direct `COPY` into a partition subgraph behaves.
 * Single-row `CREATE`/`MERGE` routes at runtime in `NodeInsertExecutor`: the partition key is
   evaluated, the matching child table is selected, and the row is inserted there.
 
-Remaining for RANGE: replace the hash stand-in with real bound comparison once declarative range
-bounds (`PARTITION p0 VALUES < (...) ...`) are implemented.
+### 1b. RANGE partitioning (not implemented; DDL refuses it)
+
+Real RANGE needs bounds that reflect the data: either user-declared bounds
+(`PARTITION p0 VALUES < (...)`) or dynamic, distribution-aware splitting of the input (min/max or
+equi-depth histograms computed during COPY, persisted with the parent). Equal splits of the *type
+domain* were considered and rejected: every realistic timestamp lands in one middle bucket.
+Until one of those exists, the binder refuses `PARTITION BY RANGE` instead of silently routing by
+hash.
 
 ### 2. Rel writes against the parent
 
@@ -284,18 +287,36 @@ Push a predicate on the partition column into the scan: for HASH only equality
 filter-push-down / scan selection (`scan->setNumPartitionsToScan`, etc.) once the partition bounds
 are materialized.
 
-### 4. Declarative range bounds
-
-Extend the grammar to accept per-partition bounds:
-
-```cypher
-CREATE NODE TABLE Events (...) PARTITION BY RANGE (ts) (
-    PARTITION p2023 VALUES < DATE '2024-01-01',
-    PARTITION p2024 VALUES >= DATE '2024-01-01' AND < DATE '2025-01-01'
-);
-```
-
-### 5. Row movement, ALTER propagation, and DETACH PARTITION
+### 4. RANGE partitioning: dynamic, distribution-aware splits
+
+Unblocks `PARTITION BY RANGE` (currently refused at DDL). Two shapes, in increasing ambition:
+
+* **Declarative bounds** — extend the grammar to accept per-partition bounds:
+  ```cypher
+  CREATE NODE TABLE Events (...) PARTITION BY RANGE (ts) (
+      PARTITION p2023 VALUES < DATE '2024-01-01',
+      PARTITION p2024 VALUES >= DATE '2024-01-01' AND < DATE '2025-01-01'
+  );
+  ```
+* **Dynamic splitting** — derive bounds from the actual value distribution (min/max or equi-depth
+  histograms computed over the COPY input, persisted with the parent; late rows outside the
+  learned range go to an overflow partition or trigger resplitting). This is what makes
+  `RANGE(ts)` place rows monotonically without asking the user for bounds.
+
+### 5. LIST partitioning (per-distinct-value partitions) — IMPLEMENTED
+
+`PARTITION BY LIST (col)` creates one partition per distinct key value on demand: 100 distinct
+cluster IDs → ~100 partitions. The first partition is created at DDL time (unkeyed, stays empty);
+every other partition is created at first sight of a new value, inside the writing transaction,
+via the same machinery as CREATE NODE TABLE (catalog entry + serial sequence + subgraph +
+storage + WAL create record), so rollback, WAL replay, and checkpointing behave like ordinary DDL.
+The encoded-key → child-table-ID map persists on the parent entry (storage version 47). Writes
+route through `ListPartitionRouter` (single-row inserts and COPY alike; COPY grows its target
+arrays under the router lock as workers discover values). Rel patterns bound against a LIST
+parent attach to the partitions that exist when they are bound; partitions created later need new
+rel tables (see roadmap item 2).
+
+### 6. Row movement, ALTER propagation, and DETACH PARTITION
 
 Lift the v1 restrictions in dependency order:
 
@@ -308,7 +329,53 @@ Lift the v1 restrictions in dependency order:
   table and back, the PostgreSQL-style escape hatch that today is approximated by "drop the
   parent"; also revisit cascade-dropping dependent rels at that point.
 
-### 6. Remote partitions over a columnar protocol (ADBC / Arrow Flight)
+### 6b. Per-partition storage files (`test.<parent>_p<i>.db`) — DESIGNED, NOT IMPLEMENTED
+
+Partition children currently share the parent's StorageManager, so their bytes live inside
+`test.db`. Goal: each partition gets its own file, like graphs created via CREATE GRAPH
+(`DatabaseManager::createGraph` builds a per-graph Catalog + StorageManager at path
+`StorageUtils::getGraphPath(dbPath, name)`).
+
+Phase B1 (shared catalog, separate files):
+* Registry: `table_id_t -> std::unique_ptr<StorageManager>` owned by DatabaseManager.
+* Creation: DDL seed partitions and ListPartitionRouter creations build a dedicated
+  StorageManager for the child (path = getGraphPath(dbPath, childName)) instead of calling
+  main-SM createTable; register it.
+* Resolution: all `StorageManager::Get(...)->getTable(id)` sites that can see a partition child
+  go through one helper that checks the registry, then lazily opens the child's SM from catalog
+  metadata on first touch after reopen. Known sites: plan_mapper.cpp:270 (scans), map_insert.cpp,
+  node_batch_insert.cpp (init + growth), insert_executor.cpp resolveTableForNodeID,
+  partition_routing.cpp pre-check.
+* Lifecycle: checkpoint iterates registered SMs; DROP-parent cascade closes+deletes files;
+  rolled-back dynamic partitions delete their file; WAL replay recreates via the same creation
+  helper.
+
+Compatibility with PR #829 (`common::PartitionRoutingHooks`, remote partition subgraphs):
+
+* The hooks and B split the same seams along orthogonal axes: 829 handles *claimed* (remote)
+  partitions, B gives *unclaimed* (local) partitions their own file. Decision order at every
+  shared seam is: consult `locate` first; claimed -> wrapper owns it, no local state (existing
+  829 behavior); unclaimed -> B's dedicated StorageManager.
+* Creation (`storage_manager.cpp:271`) becomes one decision tree: claimed -> onPartitionCreate
+  only; unclaimed -> build + register the per-partition SM. Because claimed partitions never
+  reach B's registry, checkpoint/rollback iterate exactly the unclaimed set with no extra
+  filtering -- the two mechanisms cannot double-handle a partition.
+* Reads: 829 swaps claimed partitions for scan-function-backed substitutes at bind time
+  (`expandPartitionedNodeTables`), so plan-time resolution (B's helper) never sees a remote
+  child; no ordering hazard.
+* Drops: onPartitionDrop fires for claimed partitions; B closes+deletes files and registry
+  entries for unclaimed ones. Disjoint by construction.
+* File naming follows the child's catalog name (getGraphPath scheme); parent-rename cascades
+  must rename child files alongside entry renames. PartitionRef stays ID-based as in 829;
+  names are only consulted when opening/reopening a file.
+* Phase A remains compatible: promoting an unclaimed partition to a standalone graph-database
+  changes what its registry entry holds (Catalog+SM instead of bare SM), not the seams.
+
+Phase A (later): promote each partition to a full standalone graph-database (own Catalog like
+CREATE GRAPH), making cross-partition queries identical to cross-graph ones; requires
+catalog-aware table-ID resolution everywhere.
+
+### 7. Remote partitions over a columnar protocol (ADBC / Arrow Flight)
 
 Each partition subgraph already *is* a `NodeTableCatalogEntry`. A remote partition would be a
 flavor whose storage lives on a server:

scripts/antlr4/Cypher.g4

@@ -140,6 +140,8 @@ KEY : ( 'K' | 'k' ) ( 'E' | 'e' ) ( 'Y' | 'y' ) ;
 
 LIMIT : ( 'L' | 'l' ) ( 'I' | 'i' ) ( 'M' | 'm' ) ( 'I' | 'i' ) ( 'T' | 't' ) ;
 
+LIST : ( 'L' | 'l' ) ( 'I' | 'i' ) ( 'S' | 's' ) ( 'T' | 't' ) ;
+
 LOAD : ( 'L' | 'l' ) ( 'O' | 'o' ) ( 'A' | 'a' ) ( 'D' | 'd' ) ;
 
 LOGICAL : ( 'L' | 'l' ) ( 'O' | 'o' ) ( 'G' | 'g' ) ( 'I' | 'i' ) ( 'C' | 'c' ) ( 'A' | 'a' ) ( 'L' | 'l' ) ;
@@ -386,14 +388,18 @@ iC_CreateNodeTable
 // PostgreSQL-style partitioning. A node table can be declared as a partitioned parent,
 // where each partition is backed by its own subgraph (a hidden node table).
 iC_PartitionBy
-    : PARTITION SP BY SP ( iC_PartitionRange | iC_PartitionHash ) ;
+    : PARTITION SP BY SP ( iC_PartitionRange | iC_PartitionHash | iC_PartitionList ) ;
 
 iC_PartitionHash
     : HASH SP? '(' SP? oC_PropertyKeyName SP? ')' SP PARTITIONS SP oC_IntegerLiteral ;
 
 iC_PartitionRange
     : RANGE SP? '(' SP? oC_PropertyKeyName SP? ')' SP PARTITIONS SP oC_IntegerLiteral ;
 
+// List partitioning: one partition per distinct partition-key value, created on demand.
+iC_PartitionList
+    : LIST SP? '(' SP? oC_PropertyKeyName SP? ')' ;
+
 iC_CreateRelTable
     : CREATE SP REL SP TABLE ( SP GROUP )? ( SP iC_IfNotExists )? SP oC_SchemaName
         SP? '(' SP?
@@ -1105,6 +1111,7 @@ iC_NonReservedKeywords
         | STRUCT
         | L_SKIP
         | LIMIT
+        | LIST
         | TRANSACTION
         | TYPE
         | USE

scripts/antlr4/hash.md5

@@ -1 +1 @@
-27eca67648a4fc8b77d5182a315a9eb4
+c6ef6ea298ca84a5c080bae09a29c9db

src/antlr4/Cypher.g4

@@ -119,14 +119,18 @@ iC_CreateNodeTable
 // PostgreSQL-style partitioning. A node table can be declared as a partitioned parent,
 // where each partition is backed by its own subgraph (a hidden node table).
 iC_PartitionBy
-    : PARTITION SP BY SP ( iC_PartitionRange | iC_PartitionHash ) ;
+    : PARTITION SP BY SP ( iC_PartitionRange | iC_PartitionHash | iC_PartitionList ) ;
 
 iC_PartitionHash
     : HASH SP? '(' SP? oC_PropertyKeyName SP? ')' SP PARTITIONS SP oC_IntegerLiteral ;
 
 iC_PartitionRange
     : RANGE SP? '(' SP? oC_PropertyKeyName SP? ')' SP PARTITIONS SP oC_IntegerLiteral ;
 
+// List partitioning: one partition per distinct partition-key value, created on demand.
+iC_PartitionList
+    : LIST SP? '(' SP? oC_PropertyKeyName SP? ')' ;
+
 iC_CreateRelTable
     : CREATE SP REL SP TABLE ( SP GROUP )? ( SP iC_IfNotExists )? SP oC_SchemaName
         SP? '(' SP?
@@ -838,6 +842,7 @@ iC_NonReservedKeywords
         | STRUCT
         | L_SKIP
         | LIMIT
+        | LIST
         | TRANSACTION
         | TYPE
         | USE

src/antlr4/keywords.txt

@@ -61,6 +61,7 @@ IS
 JOIN
 KEY
 LIMIT
+LIST
 LOAD
 LOGICAL
 MACRO

src/binder/bind/bind_ddl.cpp

@@ -261,13 +261,36 @@ BoundCreateTableInfo Binder::bindCreateNodeTableInfo(const CreateTableInfo* info
     std::optional<BoundPartitionInfo> partitionInfo;
     if (extraInfo.partitionInfo.has_value()) {
         const auto& parsed = *extraInfo.partitionInfo;
-        auto method = parsed.method == ParsedPartitionMethod::HASH ? BoundPartitionMethod::HASH :
-                                                                     BoundPartitionMethod::RANGE;
-        if (parsed.numPartitions == 0) {
-            throw BinderException("Number of partitions must be greater than 0.");
+        BoundPartitionMethod method;
+        switch (parsed.method) {
+        case ParsedPartitionMethod::HASH:
+            method = BoundPartitionMethod::HASH;
+            break;
+        case ParsedPartitionMethod::LIST:
+            // LIST partitions dynamically: one partition per distinct key value, created on
+            // demand. It takes no PARTITIONS clause.
+            method = BoundPartitionMethod::LIST;
+            validatePartitionColumn(propertyDefinitions, parsed.columnName);
+            partitionInfo = BoundPartitionInfo(method, parsed.columnName, 0 /* numPartitions */);
+            break;
+            // Real range partitioning must split on the actual distribution of values (equal
+            // domain splits would dump all real-world data into one bucket). Until that dynamic
+            // splitting exists, refuse the DDL rather than silently falling back to hash routing.
+        case ParsedPartitionMethod::RANGE:
+            throw BinderException(
+                "RANGE partitioning is not implemented yet. RANGE requires partition bounds "
+                "derived from the actual value distribution, which is future work. Use PARTITION "
+                "BY HASH instead.");
+        default:
+            UNREACHABLE_CODE;
+        }
+        if (method == BoundPartitionMethod::HASH) {
+            if (parsed.numPartitions == 0) {
+                throw BinderException("Number of partitions must be greater than 0.");
+            }
+            validatePartitionColumn(propertyDefinitions, parsed.columnName);
+            partitionInfo = BoundPartitionInfo(method, parsed.columnName, parsed.numPartitions);
         }
-        validatePartitionColumn(propertyDefinitions, parsed.columnName);
-        partitionInfo = BoundPartitionInfo(method, parsed.columnName, parsed.numPartitions);
     }
     auto boundExtraInfo = std::make_unique<BoundExtraCreateNodeTableInfo>(extraInfo.pKName,
         std::move(propertyDefinitions), std::move(storage), std::move(storageFormat),

src/catalog/catalog.cpp

@@ -635,6 +635,11 @@ CatalogEntry* Catalog::createNodeTableEntry(Transaction* transaction,
     for (auto& definition : extraInfo->propertyDefinitions) {
         entry->addProperty(definition);
     }
+    if (extraInfo->partitionParentTableID != common::INVALID_TABLE_ID) {
+        // Dynamically created LIST partition child: register the parent link so reads expand
+        // to it and writes can resolve it.
+        entry->setParentInfo(extraInfo->partitionParentTableID, extraInfo->partitionChildIndex);
+    }
     entry->setHasParent(info.hasParent);
     createSerialSequence(transaction, entry.get(), info.isInternal);
     auto catalogSet = info.isInternal ? internalTables.get() : tables.get();
@@ -654,7 +659,15 @@ CatalogEntry* Catalog::createNodeTableEntry(Transaction* transaction,
         auto partitionColumnID = parent->getPropertyID(partitionInfo.columnName);
         parent->setPartitionInfo(partitionInfo.method, partitionInfo.columnName, partitionColumnID,
             partitionInfo.numPartitions);
-        for (auto i = 0u; i < partitionInfo.numPartitions; i++) {
+        // LIST starts with one partition and grows on demand; HASH creates its full set here.
+        // LIST's initial partition keeps the >=1-partition invariant that reads and writes rely
+        // on (partition expansion never yields an empty child set). It stays unkeyed and empty:
+        // rows always route to the partition created for their own key value.
+        const auto numInitialPartitions =
+            partitionInfo.method == binder::BoundPartitionMethod::LIST ?
+                1 :
+                partitionInfo.numPartitions;
+        for (auto i = 0u; i < numInitialPartitions; i++) {
             auto childName = std::format("{}_p{}", info.tableName, i);
             auto child = std::make_unique<NodeTableCatalogEntry>(childName,
                 extraInfo->primaryKeyName, extraInfo->storage, extraInfo->storageFormat);