Skip to content

[PERF] Speed up balanced NumPy pivot_longer reshapes - #1668

Open
samukweku wants to merge 2 commits into
devfrom
issue-1656-stack-fast-paths
Open

[PERF] Speed up balanced NumPy pivot_longer reshapes#1668
samukweku wants to merge 2 commits into
devfrom
issue-1656-stack-fast-paths

Conversation

@samukweku

Copy link
Copy Markdown
Collaborator

PR Description

  • Add a narrow fast path for balanced multi-value pivot_longer specifications backed by one homogeneous NumPy block.
  • Slice already-contiguous value groups positionally instead of repeatedly selecting DataFrames and rescanning their dtypes.
  • Preserve the existing path for extension arrays, mixed or fragmented blocks, and imbalanced specifications.
  • Add exact ordering and dtype coverage for integers, floats, booleans, datetimes, and timedeltas in both appearance-order modes.

Representative median results on pandas 3.0.5:

Input Before After Change
2,500,000 rows x 20 columns 79.7 ms 68.3 ms 14% faster
10,000 rows x 1,000 columns 20.5 ms 18.1 ms 12% faster
1 row x 100,000 columns 69.4 ms 31.4 ms 2.2x faster
1 row x 100,000 columns, 1,000 value groups 252.8 ms 33.4 ms 7.6x faster

Peak Python allocation was unchanged. Nullable, mixed, and imbalanced fallback cases remained within approximately 1% of the baseline.

Correctness was compared against dev over 720 generated combinations spanning empty, tiny, tall, and wide frames; balanced, interleaved, and imbalanced specifications; both ordering modes; dropna and ignore_index; and NumPy, object, categorical, nullable, string, and timezone-aware dtypes. All results matched element-for-element, including indexes and dtypes.

This PR contributes to #1656 without closing the profiling umbrella issue.

PR Checklist

  1. Changes are on the focused issue-1656-stack-fast-paths branch.
  2. Added an entry under CHANGELOG.md's Unreleased section.

Verification

  • pixi run pytest tests/functions/test_pivot_longer.py: 103 passed, 6 xfailed, 1 xpassed
  • Full suite: 1,351 passed, 5 skipped, 48 xfailed, 17 xpassed
  • Focused Ruff check and format check passed
  • git diff --check passed

Relevant Reviewers

@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://pyjanitor-devs.github.io/pyjanitor/pr-preview/pr-1668/

Built to branch gh-pages at 2026-08-21 10:39 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant