fix: nullable integer streaming stats on pandas < 2 - #192
Merged
Merged
Conversation
test-matrix (3.9) has failed on main since #187: with pandas 1.5, the streaming state converted numeric columns via pd.to_numeric(s).to_numpy(dtype="float64"), which raises "cannot convert to 'float64'-dtype NumPy array with missing values" for nullable (masked) integer columns with NA. fd.clean_timeseries and StreamingCleaner hit it on any Int*/UInt* column with gaps. Pass na_value=np.nan there and in the time-series interpolation helper, which has the same conversion. Also pin the alert step #188 added to fetch-fixtures.yml, which #190's SHA sweep predated.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This was referenced Sep 14, 2026
kevincostner17
added a commit
that referenced
this pull request
Sep 14, 2026
- perf-regression.yml installs with -c constraints/ci.txt like the other gating jobs (left out of #194 to avoid conflicting with #193). - MissForest convergence converts with to_numpy(dtype="float64", na_value=np.nan): the same pandas < 2 masked-array conversion that #192 fixed in the streaming state. Adds a nullable Int16 MissForest test. - contributor-roadmap.md no longer advertises #33 and #31, which are resolved; it points at the good-first-issue label instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test-matrix (3.9), which runs pandas 1.5.3 and numpy 2.0.2, has been failing onmainsince #187 merged:The new test surfaced an existing pandas < 2 bug, not a problem in the median fix.
StreamingStatefolds each numeric column into its running stats withpd.to_numeric(s).to_numpy(dtype="float64"), which pandas 1.5 refuses for maskedInt*/UInt*columns containingNA:So
fd.clean_timeseries/StreamingCleanercrashed on pandas 1.5 for any nullable integer column with gaps.Changes
streaming/_state.py:to_numpy(dtype="float64", na_value=np.nan). TheNaNs are already filtered on the next line.streaming/_timeseries.py:_interp_seriesdoes the same conversion for interpolation, so it gets the samena_valueand computes the float view once..github/workflows/fetch-fixtures.yml: pins the alert step fix: retry transient failures in the weekly online fixture refresh #188 added (actions/github-script@v7) to v9.0.0 by SHA. ci: pin GitHub Actions to Node 24 releases by SHA #190's pin sweep was branched before fix: retry transient failures in the weekly online fixture refresh #188 landed, so it was the only unpinned action left onmain.Verification
pytest tests/test_nullable_int_median.py tests/test_streaming_timeseries.py tests/test_missing.py tests/test_simple.pygives 116 passed. Before this change, the two seasonal cases failed with the error above.pytest tests/test_nullable_int_median.py tests/test_streaming_timeseries.py tests/test_streaming_reports.pygives 80 passed.ruff check src/freshdata/streaming: clean. The workflow parses as YAML, and no unpinneduses:remain.