Support pandas 3 - #711
Open
ecomodeller wants to merge 1 commit into
Open
Support pandas 3#711ecomodeller wants to merge 1 commit into
ecomodeller wants to merge 1 commit into
Conversation
The pandas3 leg of the test matrix has never run pandas 3: `uv run pip install` is followed by `uv run`, which re-syncs the environment from the lockfile and puts pandas 2 back. The last main run installed pandas 3.0.5 and then reported `pandas 2.3.3`. Install with `uv pip install`, set UV_NO_SYNC for the job, and assert the major version so the leg fails loudly instead of silently testing the wrong thing. With the leg fixed, two tests fail under pandas 3. `_parse_dataset` tags untagged data vars with kind="auxiliary" and `Comparer.__init__` read those attrs back off the caller's dataset, relying on `_normalize_time_to_ns` having returned the same object. pandas 3 defaults datetime64 to microseconds, so that call now genuinely converts and returns a copy, and the mutation stops leaking. Read from the parsed dataset instead. The pandas floor was `>= 1.4, < 3.0`, which is not what is supported: with requires-python >= 3.12, pandas 2.1.1 is the first release with wheels, and xarray already requires pandas >= 2.2. Drop the upper bound, state the real floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #561, which added
_normalize_time_to_nsand the pandas2/pandas3 test matrix. The pandas3 leg of that matrix has never run pandas 3, so the compatibility it promised was never verified.The pandas3 CI leg was testing pandas 2
uv run pip install "pandas>=3.0,<4.0"is followed byuv run, which re-syncs the environment from the lockfile and puts pandas 2 back. From the lastfull_testrun on main, jobbuild (3.12, pandas3):Both legs have been running pandas 2.3.3 since the matrix was added. Fixed by installing with
uv pip install, settingUV_NO_SYNC: "1"for the job, and asserting the major version against the matrix leg so a silent revert fails the build instead of passing it.One real pandas 3 break
With the leg fixed,
test_matched_aux_variablesandtest_multiple_forecasts_matched_datafail withKeyError: 'kind'._parse_datasettags untagged data vars withkind="auxiliary"(_comparison.py:97) andComparer.__init__read those attrs back off the caller'smatched_data, relying on_normalize_time_to_nshaving returned the same object. pandas 3 defaults datetime64 to microseconds, so that call now genuinely converts and returns a copy, and the mutation stops leaking. Reading fromself.data— the parsed dataset, wherekindis always set — is correct on both majors. Latent aliasing bug, not a pandas incompatibility.Honest pandas floor
pandas >= 1.4, < 3.0is not what is supported: withrequires-python >= 3.12, pandas 2.1.1 is the first release with wheels, and xarray 2026.7.0 already requirespandas >= 2.2, as does mikeio. Nowpandas >= 2.2, no upper bound.The floor stays at 2.x rather than moving to 3.0 because supporting both costs one line — nothing in the codebase needs pandas 3, and
_normalize_time_to_nsis needed either way (pandas 2 also yields non-ns resolutions from NetCDF/dfs0). A 3.0 floor would push the pandas 3 migration onto downstream analysis code — potpy, blue_ml, fmdap, the notebooks — and remove their ability to pin back, for no gain here. Worth revisiting at 2.0.Verification
Replayed the CI leg in a throwaway worktree:
uv sync --group test --group networks --no-dev→uv pip install "pandas>=3.0,<4.0"→UV_NO_SYNC=1 uv run pytest.The first row is the point of the CI change: the old workflow ran that code and reported success.
Conflicts
#551 (
pandas[output_formatting]) touches the same line and is based on main from before #561 added the< 3.0cap; when rebasing it, bring the floor to>= 2.2and note that pandas 3 spells the extraoutput-formatting. #548 is the same theme but doesn't touch pandas — no conflict. #564 reworks thekindattribute this fix is about; if it lands first the change wants re-checking rather than a blind rebase.🤖 Generated with Claude Code