Repository navigation
Fix engine bugs and CPU fallbacks found on science datasets - #7
Merged
Merged
Conversation
The grouped family key of shared_subplan_materialization compared group keys, aggregate expressions and predicate-wildcarded sources only, so two aggregates whose inputs differed by a Filter or Join between the aggregate and the scan were merged and one branch read the other's aggregate. A CTE read twice, once under `WHERE col IS NOT NULL`, returned the unfiltered groups. The key now includes the input shape, computed against the columns the operators above read: sorts and column-only projections are skipped, a join keeps its type, keys, residual, null equality and the read output columns with their side, and filters and computed projections add the columns they read. Below a limit, window, union or any other operator, and for floating statistical aggregates, the shape is exact. TPC-DS q27's rollup branches, which differ only in column pruning and join output order, still merge; applied merges match the previous build on TPC-H and TPC-DS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… limit A FixedSizeList<Float32, 2048> column with 2.9M rows (5.9e9 elements) failed every scan, filtered or limited, because one planned source chunk spanned enough row groups that the cuDF shim's concatenation of reader passes exceeded cudf::size_type. The client saw "result size computation overflowed the platform range" with no column or remedy. BackendCapabilities gains max_column_elements (cuDF: i32::MAX). Chunk, batch, retry and scan-limit planning keep every read's largest column under it, counted from Parquet footer leaf value counts, or rows times fixed list widths without a footer. A row group that alone exceeds the limit fails planning with a typed size_overflow naming the column and the row-group size that fits. libcudf concatenation is pre-checked per column, and the client message names the column, element count, limit and the row count that fits at its width. One relation input above the limit is still unsupported and now fails with that message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ecution DataFusion inlines a CTE or view at every reference, so a KMeans, Louvain or UMAP call in a CTE read twice ran twice and the two references disagreed (170 of 200,000 rows in a KMeans self-join). Island reuse was limited to cuGraph similarity with repeatable inputs, and cuVS and cuML named each physical copy separately, so no copy could be matched. Every cuGraph, cuVS and cuML call site now gets one invocation identity at logical planning (GpuFunctionInvocation), kept through plan rebuilds and part of node equality, so separately written calls stay independent. Inside a composed island, every GPU function result is reused by invocation and the function's own output-plan fingerprint; the repeatability gate and the input-scope walk are removed. Host occurrences of one invocation share one manifest entry: the first reader polled runs the function and the others replay its batches, charged to the DataFusion memory pool. References split between an island and a host boundary, and a cuGraph copy that absorbed a LIMIT, still evaluate separately; the design page documents both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A read known to return no rows (scan LIMIT 0, every file or row group pruned) built its zero-row frame by uploading an empty Arrow batch, which counts as a host Arrow import and breaks device-resident contracts once a dynamic filter prunes a whole source. Runtime::empty_dataframe_from_arrow_schema maps each field through the host-import type mapping and builds the column from a cudf type tree in the new shim call cudf_column_create_empty_from_type_tree_ex: a LIST gets an empty INT32 offsets child and its element child, a STRUCT its field children, as cudf::empty_like does. No device memory is allocated and the columns carry the runtime's provenance. The engine's empty local Parquet frame uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… unmatched rows Route discovery accepted only inner joins, so `x IN (SELECT ...)`, which plans as a left semi join, read every row group of the probe source: on 2.61 billion rows a one-column count took 0.59 s and 8.5 GiB against 0.06 s and 1.2 GiB for the literal list. join_type_discards_unmatched_rows now decides which input a key payload may reduce: both inputs of inner and semi joins, the right input of left outer, non-null-aware anti and mark joins, neither input of null-aware anti, full and cross joins. Key routes, keyset-constraint proofs and residual-equality candidates all use it. The TPC-H rule golden gains routes on q04, q17, q18 and q20. The same count now takes 0.05 s and 1.2 GiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A merge of sorted partitions was rejected unless every sort key was a plain column, although a native sort lowers expression keys, so `ORDER BY value / global_pr DESC LIMIT 20` at the default partition count ran mixed with 23 host boundaries. Orderings are now compared by full PhysicalSortExpr equality and the merge accepts any key the native sort lowers. InterleaveExec, DataFusion's UNION ALL over identically hash-partitioned inputs, had no native form; a ChEMBL activity-cliff query fell back with 25 host boundaries and took 39.9 s. It now lowers through the native union path and runs fully native in 1.93 s with the same counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An edge guard rejected a left anti join whose build is a group-only aggregate unless its probe input was proven distinct, and rejected any grouped build with measures. It began as a recognizer rule for one set-operation envelope and was promoted to generic edge capability, but every bounded anti path keeps each unmatched probe row and DataFusion plans EXCEPT DISTINCT with an explicit dedup. `LEFT JOIN (SELECT DISTINCT ...) d ... WHERE d.k IS NULL` over a join output fell back to the CPU for no reason. Both arms and their reasons (left_anti_build_requires_dedup_aggregate, left_anti_probe_requires_dedup_proven_input) are removed. A GPU case keeps duplicate probe rows equal to DataFusion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default scan read the gitignored docs/ and note/ trees, which git and CI never see, so local private files failed the full gate. Its self-test for a checkout without those trees ran the real checkout's script, which scans the checkout containing it, and so read the same private files. The default target is website/docs; explicitly named paths still work. The self-test runs a copy of the script inside a scratch checkout with violations under docs/ and note/ and one under website/docs/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
morristai
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Engine fixes from running Algeon on public science datasets (WM-811K, ChEMBL 37, Open Targets, Google Willow QEC). One commit per fix.
Correctness
ff203c9bf):shared_subplan_materializationmerged grouped aggregates whose inputs differed below the aggregate, so a CTE read twice returned unfiltered groups. The family key now includes the input shape.20f90fd3d): a KMeans CTE joined to itself disagreed on 170 of 200,000 rows. Each cuGraph/cuVS/cuML call now evaluates once per query execution.Blocking
1ab47a23b): chunk planning keeps every read under the backend's element limit; one oversized relation fails with a typed message naming the column and row cap.Performance and coverage
IN (SELECT ...)read the whole source (58dbe35c7,b581e0b5a): dynamic-filter routes now cover every join input that drops unmatched rows (semi joins, the right input of outer/anti/mark joins). On 2.61 B rows: 0.59 s / 8.5 GiB to 0.05 s / 1.2 GiB. A fully pruned source now builds its empty frame on the device instead of importing a host batch.8ec4056d7,21339a271): expression-keyed sort-preserving merges,InterleaveExec, and left anti joins over aSELECT DISTINCTbuild now run native. A ChEMBL activity-cliff query went from 39.9 s mixed to 1.93 s native with identical counts.CI
82c56d23e: the default prose check scanswebsite/docsonly; gitignoreddocs/andnote/are out of scope.Validation
Full pre-merge gate passed on
82c56d23e. Release-build reruns on the real datasets match DuckDB or numpy references, or earlier results.Not in this PR:
x = c OR x IN (SELECT ...)(mark join) still reads the whole source; nativeAPPROX_PERCENTILE_CONTandCAST(string AS integer).🤖 Generated with Claude Code