fix: prevent currency code false-positives inside longer words - #78
binggao1230 wants to merge 6 commits into
Conversation
Letter-only currency codes like ALL (Albanian Lek) and DA were being matched as substrings inside words like ANNUALLY and DAILY. Add negative lookbehind/lookahead for ASCII letters around purely alphabetic symbols in the unsafe currency regex. Fixes scrapinghub#71
2128264 to
a808798
Compare
|
Sonnet reports: price_parser/parser.py:261 — The price_parser/parser.py:268 —
tests/test_price_parsing.py — No The PR description specifically calls out price_parser/parser.py:261 —
Any thoughts? |
|
Addressed the boundary review in What changed:
I did not try to support arbitrary inflections such as Verification:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #78 +/- ##
==========================================
+ Coverage 98.23% 98.44% +0.21%
==========================================
Files 3 3
Lines 113 129 +16
Branches 16 19 +3
==========================================
+ Hits 111 127 +16
Misses 1 1
Partials 1 1
🚀 New features to boost your workflow:
|
|
Please, run |
|
Ran Local verification now passes: |
| _search_safe_currency = or_regex(SAFE_CURRENCY_SYMBOLS).search | ||
| _search_unsafe_currency = or_regex(OTHER_CURRENCY_SYMBOLS).search | ||
| _search_safe_currency = _make_currency_regex(SAFE_CURRENCY_SYMBOLS).search | ||
| _search_unsafe_currency = _make_currency_regex(OTHER_CURRENCY_SYMBOLS).search |
There was a problem hiding this comment.
Is or_regex dead code now?
There was a problem hiding this comment.
Yes — after this change nothing calls it; _make_currency_regex replaced both call sites. It isn't in __all__ and isn't re-exported from price_parser/__init__.py, but it is a public-looking module-level name, so someone could be importing it. Happy to drop it here or leave it and deprecate separately — tell me which you prefer and I'll push it.
There was a problem hiding this comment.
Done in a753e9a — or_regex now emits a DeprecationWarning and its docstring points at _make_currency_regex. Suite still green.
|
Could you complete coverage? |
|
Done in de876d1 — |
|
Please, run pre-commit |
|
Done — |
Fixes #71 — currency codes like ALL (Albanian Lek) and DA are incorrectly matched as substrings within words like ANNUALLY and DAILY.
Problem
The
_search_unsafe_currencyregex built byor_regex()performs a plain alternation without any boundary checks. This meansALLmatches insideANNUALLYandDAmatches insideDAILY, causingPrice.fromstring('1000 ANNUALLY')to returncurrency='ALL'instead ofcurrency=None.Fix
Replace the direct
or_regex()call for unsafe currency symbols with_make_unsafe_currency_regex(), which wraps letter-only currency codes in negative lookbehind/lookahead assertions:(?<![a-zA-Z])ALL(?![a-zA-Z]). Non-alphabetic symbols ($, \€, \¥, etc.) are left unchanged to preserve existing behavior for attached codes like1000USD.This is a safe target-specific fix — only the unsafe-currency regex used for final currency extraction is affected.
Testing
Price.fromstring('1000 ANNUALLY')→currency=None(wasALL)Price.fromstring('1000 DAILY')→currency=None(wasDA)Price.fromstring('1000 ALL')→currency='ALL'(unchanged)Price.fromstring('1000USD')→currency='USD'(unchanged)Price.fromstring('1000 USD')→currency='USD'(unchanged)This pull request was prepared with the assistance of AI, under my direction and review.