Skip to content

[PERF] simplify conditional_join result materialization (blocked by #1633) - #1638

Draft
samukweku wants to merge 4 commits into
issue-1627-anti-joinfrom
issue-1632-result-materialization
Draft

[PERF] simplify conditional_join result materialization (blocked by #1633)#1638
samukweku wants to merge 4 commits into
issue-1627-anti-joinfrom
issue-1632-result-materialization

Conversation

@samukweku

@samukweku samukweku commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Dependency

Blocked by #1633. Do not merge this PR until #1633 has merged. This PR is intentionally based on issue-1627-anti-join so the benchmark and refactor cover left_anti and right_anti. After #1633 merges, retarget/rebase this branch onto dev and mark this PR ready for review.

Closes #1632.

Summary

  • replace branch-specific result assembly with synchronized left/right row indexers
  • use pandas dtype-aware take only when a side contains missing positions, retaining a direct-indexing fast path otherwise
  • preserve row order, indicator categories, NumPy promotion, and extension-array dtypes
  • remove the copied pandas private fill-array helper
  • add mixed-dtype compatibility tests

ELI5

A join first finds matching row numbers, then builds a new table from them. The old code rebuilt that table differently for every join type. This change creates one row-number plan per side; -1 means that side has no row and pandas supplies the correct missing value.

Performance

Apple Silicon, Python 3.14.5, pandas 3.0.3, 50,000 rows, sparse duplicate-key matches, five repetitions. Timing and tracemalloc memory passes ran separately using a one-off local harness that is not included in this PR.

Join 9 cols 32 cols
left 1.04x 1.22x
right 1.21x 1.23x
outer 1.00x 1.10x
left_anti 1.18x 1.05x
right_anti 1.16x 1.16x

Nine of ten sampled cases improved; narrow outer was effectively flat (4.72 ms to 4.74 ms). Peak traced allocation fell in eight cases, was effectively flat for wide outer/right_anti, and rose 3.7% for narrow outer because one combined integer row indexer remains live. The broader 48-case join-type/density/width run found no material runtime regression. End-to-end timing is matching-dominated and stayed within the unchanged inner-path noise band.

Verification

  • project suite: 1,346 passed, 5 skipped, 48 xfailed, 17 xpassed
  • conditional_join focused suite: 253 passed, 1 skipped
  • mixed-dtype materialization regression tests: 12 passed
  • configured pre-commit hooks: passed for all changed files
  • standalone markdownlint: passed for changed Markdown files
  • adversarial comparison against [ENH] Add anti joins to conditional_join #1633: exact parity for numeric/nullable/category/string/datetime/tz/timedelta plus Period, Interval, SparseArray, and empty-input probes

@samukweku
samukweku force-pushed the issue-1632-result-materialization branch from 503706e to 8aecf4e Compare August 20, 2026 03:11
@samukweku
samukweku force-pushed the issue-1627-anti-join branch from 50fc7da to 1416b3e Compare August 20, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant