Skip to content

refactor: simplify null-aware and join-type branching in hash join - #24957

Merged
jayzhan211 merged 1 commit into
apache:mainfrom
jayzhan211:join-refactor
Sep 6, 2026
Merged

refactor: simplify null-aware and join-type branching in hash join#24957
jayzhan211 merged 1 commit into
apache:mainfrom
jayzhan211:join-refactor

Conversation

@jayzhan211

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • N/A

Rationale for this change

HashJoinExec carries a null_aware: bool that is only legal for three
(join_type, partition_mode) combinations. Because the flag and the join type
were tracked separately, every use site had to re-derive which combination it
was in, with conditions like self.null_aware && self.join_type == JoinType::LeftAnti
scattered across the builder and the stream. process_probe_batch and
process_unmatched_build_batch were interrupted five times by such blocks, so
the main probe path was hard to follow, and the builder validated the legal
combinations with a five-clause hand-written truth table.

Separately, several predicates over JoinType were spelled as hand-maintained
lists of variants (maintains_input_order, the EmissionType match, the
swap-inputs projection check). Each list has to be revisited whenever a variant
is added, as happened when RightMark landed, and none of them says why a
variant belongs in the list.

What changes are included in this PR?

Two related cleanups, no behavior change:

  1. NullAwareMode enum (hash_join/exec.rs, hash_join/stream.rs)

    • New pub(super) enum NullAwareMode { LeftAnti, RightAnti, LeftMark { correlated: bool } },
      with a single try_new holding the whole legality table. The builder's
      validation becomes one call; the error messages are byte-identical.
    • HashJoinExec::null_aware stays a public bool, so there is no API change.
      The mode is derived once in execute and passed down.
    • collect_left_input takes one Option<NullAwareMode> instead of two
      separately derived booleans, which also lets a now-unfulfilled
      clippy::fn_params_excessive_bools expectation be dropped.
    • The null-aware logic in the stream moves into four named helpers
      (null_aware_skip_probe_batch, drop_null_probe_keys,
      null_aware_left_anti_final_indices, null_aware_left_mark_column), each
      documenting the three-valued-logic rule it implements. The probe path now
      reads top to bottom, and the final stage is a single match on the mode.
  2. Named predicates instead of variant lists (joins/utils.rs)

    • emits_unmatched_left_rows drives the EmissionType decision.
    • is_existence_join drives the swap-inputs projection check.
    • maintains_input_order becomes !need_produce_result_in_final(join_type),
      whose complement was exactly the old hand-written list, with a comment
      explaining that emitting rows from the build-side bitmap breaks probe order.
    • lr_is_preserved is left alone; it already reads as a clear truth table.

Note for reviewers: emits_unmatched_left_rows deliberately excludes LeftSemi
so the EmissionType result is unchanged, even though the hash join does emit
LeftSemi rows from the bitmap in the final stage. That inconsistency is
pre-existing and out of scope here.

What is the testing strategy for this PR?

No new tests: the change is behavior-preserving, so the value is in the existing
coverage continuing to pass.

  • cargo test -p datafusion-physical-plan --lib hash_join
  • sqllogictest files null_aware_anti_join.slt, null_aware_mark_join.slt,
    joins.slt, subquery.slt
  • cargo clippy --all-targets --all-features -- -D warnings and cargo fmt --all

The builder validation tests in hash_join/exec.rs already assert on the exact
null-aware error strings, which are unchanged, so they cover NullAwareMode::try_new.

Are there any user-facing changes?

No. No public API changes, and EXPLAIN output is unchanged. The three
JoinType predicates were each verified to cover exactly the same set of
variants as the lists they replace.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Sep 6, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.68786% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.62%. Comparing base (406c0c6) to head (da55917).
⚠️ Report is 34 commits behind head on main.

Files with missing lines Patch % Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs 97.47% 2 Missing and 1 partial ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs 97.82% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24957      +/-   ##
==========================================
- Coverage   81.62%   81.62%   -0.01%     
==========================================
  Files        1123     1124       +1     
  Lines      409524   412193    +2669     
  Branches   409524   412193    +2669     
==========================================
+ Hits       334294   336437    +2143     
- Misses      55594    55939     +345     
- Partials    19636    19817     +181     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JoinType::Inner
| JoinType::LeftSemi
| JoinType::RightSemi
| JoinType::Right

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here says everything not emitting unmatched build rows is emitted incrementally. LeftSemi emits matched build rows from the final bitmap, not incrementally.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'll open an issue or a PR for this.

@jayzhan211
jayzhan211 added this pull request to the merge queue Sep 6, 2026
Merged via the queue into apache:main with commit 3266eaa Sep 6, 2026
41 checks passed
@jayzhan211
jayzhan211 deleted the join-refactor branch September 6, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants