Skip to content

Support pandas 3 - #711

Open
ecomodeller wants to merge 1 commit into
mainfrom
support-pandas-3
Open

ecomodeller wants to merge 1 commit into
mainfrom
support-pandas-3

Conversation

@ecomodeller

Copy link
Copy Markdown
Member

Follow-up to #561, which added _normalize_time_to_ns and the pandas2/pandas3 test matrix. The pandas3 leg of that matrix has never run pandas 3, so the compatibility it promised was never verified.

The pandas3 CI leg was testing pandas 2

uv run pip install "pandas>=3.0,<4.0" is followed by uv run, which re-syncs the environment from the lockfile and puts pandas 2 back. From the last full_test run on main, job build (3.12, pandas3):

Successfully installed numpy-2.5.2 pandas-3.0.5
pandas 2.3.3          <- the "Show pandas version" step

Both legs have been running pandas 2.3.3 since the matrix was added. Fixed by installing with uv pip install, setting UV_NO_SYNC: "1" for the job, and asserting the major version against the matrix leg so a silent revert fails the build instead of passing it.

One real pandas 3 break

With the leg fixed, test_matched_aux_variables and test_multiple_forecasts_matched_data fail with KeyError: 'kind'. _parse_dataset tags untagged data vars with kind="auxiliary" (_comparison.py:97) and Comparer.__init__ read those attrs back off the caller's matched_data, relying on _normalize_time_to_ns having returned the same object. pandas 3 defaults datetime64 to microseconds, so that call now genuinely converts and returns a copy, and the mutation stops leaking. Reading from self.data — the parsed dataset, where kind is always set — is correct on both majors. Latent aliasing bug, not a pandas incompatibility.

Honest pandas floor

pandas >= 1.4, < 3.0 is not what is supported: with requires-python >= 3.12, pandas 2.1.1 is the first release with wheels, and xarray 2026.7.0 already requires pandas >= 2.2, as does mikeio. Now pandas >= 2.2, no upper bound.

The floor stays at 2.x rather than moving to 3.0 because supporting both costs one line — nothing in the codebase needs pandas 3, and _normalize_time_to_ns is needed either way (pandas 2 also yields non-ns resolutions from NetCDF/dfs0). A 3.0 floor would push the pandas 3 migration onto downstream analysis code — potpy, blue_ml, fmdap, the notebooks — and remove their ability to pin back, for no gain here. Worth revisiting at 2.0.

Verification

Replayed the CI leg in a throwaway worktree: uv sync --group test --group networks --no-devuv pip install "pandas>=3.0,<4.0"UV_NO_SYNC=1 uv run pytest.

tree pandas result
main 3.0.5 2 failed, 790 passed, 5 skipped
this branch 3.0.5 792 passed, 5 skipped; mypy clean
this branch 2.3.3 737 passed, 60 skipped; ruff + mypy clean

The first row is the point of the CI change: the old workflow ran that code and reported success.

Conflicts

#551 (pandas[output_formatting]) touches the same line and is based on main from before #561 added the < 3.0 cap; when rebasing it, bring the floor to >= 2.2 and note that pandas 3 spells the extra output-formatting. #548 is the same theme but doesn't touch pandas — no conflict. #564 reworks the kind attribute this fix is about; if it lands first the change wants re-checking rather than a blind rebase.

🤖 Generated with Claude Code

The pandas3 leg of the test matrix has never run pandas 3: `uv run pip
install` is followed by `uv run`, which re-syncs the environment from the
lockfile and puts pandas 2 back. The last main run installed pandas 3.0.5 and
then reported `pandas 2.3.3`. Install with `uv pip install`, set UV_NO_SYNC
for the job, and assert the major version so the leg fails loudly instead of
silently testing the wrong thing.

With the leg fixed, two tests fail under pandas 3. `_parse_dataset` tags
untagged data vars with kind="auxiliary" and `Comparer.__init__` read those
attrs back off the caller's dataset, relying on `_normalize_time_to_ns` having
returned the same object. pandas 3 defaults datetime64 to microseconds, so
that call now genuinely converts and returns a copy, and the mutation stops
leaking. Read from the parsed dataset instead.

The pandas floor was `>= 1.4, < 3.0`, which is not what is supported: with
requires-python >= 3.12, pandas 2.1.1 is the first release with wheels, and
xarray already requires pandas >= 2.2. Drop the upper bound, state the real
floor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant