feat: add fd.clean_excel() companion to fd.clean_csv() - #198
Merged
Merged
Conversation
Read one sheet with pandas.read_excel, run fd.clean, and optionally write the cleaned workbook. The signature mirrors clean_csv, with read_excel_kwargs/to_excel_kwargs in place of the csv kwargs. Formula sanitization is on by default for the written workbook: a raw to_excel stores "=1+1" cells and "=..." headers as live formula cells. A sheet_name that selects several sheets raises TypeError. Adds an `excel` extra (openpyxl), also pulled into `all` and `dev`, and refreshes uv.lock and constraints/ci.txt. Closes #166
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
tests/truthbench/test_inventory.py requires every public name to be classified, so collection failed with 'unclassified FreshData public surface: clean_excel'. Register it as a decision surface on the cleaning adapter, with a clean_excel operation mirroring clean_csv.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #166
Summary
Adds
fd.clean_excel(path, ...), the Excel companion tofd.clean_csv(#19) that #166 asks for.clean_csv. The parameters match one-to-one, exceptread_csv_kwargs/to_csv_kwargsbecomeread_excel_kwargs/to_excel_kwargs. A test checks that the two signatures stay in sync.read_excel_kwargs={"sheet_name": ...}picks another. Asheet_namethat selects several sheets (Noneor a list) raisesTypeErrorinstead of passing a dict tofd.clean.DataFrame.to_excelstores"=1+1"cells and"=..."headers as live formula cells (data_type == "f"). The written workbook therefore goes through the samesanitize_csv_formulasguard asclean_csv.sanitize_formulas=Falseopts out, and the returned DataFrame is never altered.excelextra. It addsopenpyxl>=3.0.7, the minimum pandas 1.5 accepts. It's also inall, and indevso CI runs the tests. freshdata never imports openpyxl itself; if it's missing, pandas raises its own ImportError.uv.lockandconstraints/ci.txtwere refreshed with uv 0.12.10. The only additions areopenpyxlandet-xmlfile.docs/api-reference.mdgets the new function,docs/threat-model.mdgets a row in the sanitization table, andCHANGELOG.mdhas an entry.Tests
New
tests/test_clean_excel.py, 7 tests. The fixtures are built with openpyxl so=strings stay text on input.fd.clean(pd.read_excel(src)), and the written workbook reads back the same.return_report=Trueand thereport=Truealias both return aCleanReport.sheet_nameraisesTypeError.'prefix and there are no formula cells; the returned frame is unchanged. Withsanitize_formulas=False, a formula cell is written.clean_csv, andclean_excelis exported.Verification
pytest tests/test_clean_excel.py tests/test_csv_formula_sanitize.py tests/test_api.py tests/test_phase5_backcompat.py: all pass on Python 3.9 / pandas 1.5.3 and on Python 3.12 / pandas 2.3.3.ruff check: clean.mypy src/freshdata: no issues in 201 files.uv lock --check: passes.mkdocs build --strict: passes.