chore: Update to datafusion@7e015b74af0928187a77cdf9df23b587e349ebf9. - #77
Draft
stuhood wants to merge 15 commits into
Draft
chore: Update to datafusion@7e015b74af0928187a77cdf9df23b587e349ebf9.#77stuhood wants to merge 15 commits into
stuhood wants to merge 15 commits into
Conversation
)": see datafusion-contrib#618 (comment) This reverts commit 2dc7a36.
As part of [this](paradedb/paradedb#5553) switch to pre-release DataFusion 55. This PR updates the DF rev used, fixes the compile errors, updates a bunch of test outputs, and removes the unnecessary statistics wrapper. --------- Co-authored-by: Mohammad Dashti <mdashti@gmail.com>
The prost message types carry no tonic dependency, so a transport that is not gRPC can speak the same wire shape without pulling in the gRPC stack. Only the tonic client and server stay gated; the generator emits those gates so a regeneration cannot drop them. tonic-prost feeds only the generated client and server, so it moves behind the feature too. Co-authored-by: Stu Hood <stuhood@gmail.com>
The benchmarks crate's dev-dependency on the lib re-unified grpc into every test build, so a genuine no-gRPC test run was impossible; the dataset suites move into the benchmarks crate and the gRPC-coupled test utilities gate behind grpc. A unit-test-no-grpc job then runs the whole lib suite with the feature off. Co-authored-by: Stu Hood <stuhood@gmail.com>
InProcessChannelResolver routes the three protocol methods straight to a co-located Worker, with no gRPC, no IPC, and no serialization round-trip: the reference implementation of the protocol for a co-located worker, and the first transport that exercises the abstraction with grpc off. Its end-to-end test (a distributed GROUP BY across tasks) runs under the no-gRPC CI job. Co-authored-by: Stu Hood <stuhood@gmail.com>
A transport that returns worker metrics out-of-band, rather than over the coordinator_channel return stream, needs its driver to decode the frames and file them into the executed plan's store before the per-task EXPLAIN rewrite reads it: metrics_store() with a public insert, a no-gRPC decode_task_metrics, and the frame builders collect_plan_metrics_protos and set_received_time. The metrics codec moves out from behind grpc with them, and the gRPC client drops its private copy of the decode. The test pins the frame-to-store path. Co-authored-by: Stu Hood <stuhood@gmail.com>
In support of that: * Added a DispatchPlanSource hook for coordinator-sourced plans. * Exposed produce-side partition routing off the network boundaries. * Routed dispatched plans to workers over the shm mesh. * fix(shm): stream oversized frames through the ring (#51)
…y workflow. Squashed from the following commits: * Incorporate the `paradedb/actions` rebase sync job. (#32, #33) * Run CI on `target-patch-*` branches. (#35) * Removed the docs-deploy GitHub Pages workflow. * Add snapshot tagging job (#38) * Fix Slack mentions by always mentioning at-most the approvers (#41) * ci: bump paradedb/actions to v8 for the upstream sync (#42) * ci: Add explicit permissions to the sync workflows (#43) * Use actions v10 for upstream sync alerts (#54)
I had forgotten this one when moving to GitHub Teams
As described in the new `README.md`, this change moves the `shm` module from push based to pull based. In practice, this means that when a worker starts up, it idles until it receives the new `ExecuteTaskFrame` message over the wire (corresponding to the same message on the gRPC side), and only then begins executing a particular task. ---- This does _not_ actually result in a reduction in lines of code. But I'm confident that better aligning with gRPC will allow us to more easily fix paradedb/paradedb#5809 ... and #57 (in particular) becomes much smaller.
…ies (#67) # Ticket(s) Closed - N/A ## What This PR scales a `Partitioning::Range` layout to `UnknownPartitioning` when a network boundary scales a stage's properties. ## Why The old arm was a `debug_assert!` plus an unscaled clone. Release builds kept the clone, so `NetworkCoalesceExec` scaled every other layout but not ranges, and the consumer requested a partition set the producers never served. Both sides then waited on each other for the life of the query. That is the six-hour silent hang in the paradedb/paradedb#5892 benchmark runs: with `partition_by` on join keys, a co-partitioned join under a bare `LIMIT` reports its range layout up to the gather, and nothing reshuffles in between. ## How A task-scaled range layout has no representation: the consumer side of a coalesce boundary sees every input task's ranges repeated, and `RangePartitioning` cannot express repeated split points. The boundary math and the coalesce consumer run on counts alone, so an unknown layout with the scaled count behaves exactly like the flat layouts, and the rows never move. The range property is lost above the boundary; #68 tracks carrying it across so consumer-side joins can stay co-partitioned. Earlier revisions of this PR failed loudly instead, then broke the layout with a repartition under the gather. Both are unnecessary once the boundary scales the count correctly. ## Tests - Unit test: a range layout scales to `UnknownPartitioning` with the multiplied count. - The paradedb/paradedb#5894 regress test (`partition_by` indexes, a 3-way co-partitioned join, a bare `LIMIT 5`) passes against this rev. - `cargo test --lib` green in both feature configs; CI clippy clean.
## What Add task identity to shared-memory MPP data streams, allowing several tasks from the same stage to run on one producer process. ## Why DataFusion can plan more logical tasks than there are worker processes. When several tasks from one stage run on the same producer process and write to the same output partition, the prior transport identity `(sender_proc, stage_id, partition)` is ambiguous: one task EOF could close another task stream. Handling PostgreSQL launches with fewer attached workers additionally depends on [ParadeDB #5809](paradedb/paradedb#5809), which creates the shared-memory mesh from the workers that actually attached. That gives the sender and receiver the same modulo base; this PR then keeps the logical task streams that share a producer process distinct. ## How - Extend data-frame identity to `(sender_proc, stage_id, task_id, partition)`. - Carry `task_id` through batch, EOF, cancellation, and oversized-frame chunk reassembly. - Register consumer streams with the same task-aware identity. - Permit `proc_for_task` to multiplex several tasks onto one worker safely. - Add a deterministic five-task/three-worker in-process query regression and a focused EOF-isolation test. ## Tests - `cargo +1.97.1 test -p datafusion-distributed --lib shm::transport::tests -- --nocapture` - `cargo +1.97.1 test -p datafusion-distributed --lib --no-run` - `cargo +1.97.1 check -p datafusion-distributed`
…73) ## Ticket(s) Closed - Closes #70 ## What This PR keeps the control-plane registries and the drain slot alive when a ring's data senders drop. ## Why A peer that completes normally drops the ring's last data sender, and the drain then failed the whole inbox scope: unfinished channels plus the feed, plan, and request registries. The surviving worker may still be waiting on leader-origin frames (`ExecuteTask` requests, dispatched plans), so a clean completion killed independent work. That is the `joinscan_concurrent` flake in CI ([community](https://github.com/paradedb/paradedb/actions/runs/31221693830/job/93007528649), [enterprise](https://github.com/paradedb/paradedb-enterprise/actions/runs/31215016161/job/92986410089)). A detach only says the data plane is gone; control senders never join `sender_count` and can still publish. ## How The `Detached` arm now calls `detach_data_scope`: fail only the channels still awaiting their `Eof`, keep the registries and the receiver slot. The full `fail_scope` stays for the `Error` arm and teardown, so unexpected producer loss still fails dependent work. ## Tests - `peer_completion_detach_keeps_leader_control_plane_alive`: fails without the fix. - `producer_loss_fails_channels_still_awaiting_eof`: checks the loss path.
…entation. (#66) Adds an explicit `TaskError` message type, and ensures that task errors and panics are sent across the wire. In order to do that, moves to better encapsulate the `run_execute_task_loop` method.
6 tasks
paradedb-github-bot
Bot
force-pushed
the
main
branch
5 times, most recently
from
August 14, 2026 18:34
7b897e1 to
bbaabda
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.