Skip to content

fix: prevent currency code false-positives inside longer words - #78

Open
binggao1230 wants to merge 6 commits into
scrapinghub:masterfrom
binggao1230:fix-71-currency-false-positive
Open

binggao1230 wants to merge 6 commits into
scrapinghub:masterfrom
binggao1230:fix-71-currency-false-positive

Conversation

@binggao1230

Copy link
Copy Markdown

Fixes #71 — currency codes like ALL (Albanian Lek) and DA are incorrectly matched as substrings within words like ANNUALLY and DAILY.

Problem

The _search_unsafe_currency regex built by or_regex() performs a plain alternation without any boundary checks. This means ALL matches inside ANNUALLY and DA matches inside DAILY, causing Price.fromstring('1000 ANNUALLY') to return currency='ALL' instead of currency=None.

Fix

Replace the direct or_regex() call for unsafe currency symbols with _make_unsafe_currency_regex(), which wraps letter-only currency codes in negative lookbehind/lookahead assertions: (?<![a-zA-Z])ALL(?![a-zA-Z]). Non-alphabetic symbols ($, \€, \¥, etc.) are left unchanged to preserve existing behavior for attached codes like 1000USD.

This is a safe target-specific fix — only the unsafe-currency regex used for final currency extraction is affected.

Testing

  • Price.fromstring('1000 ANNUALLY')currency=None (was ALL)
  • Price.fromstring('1000 DAILY')currency=None (was DA)
  • Price.fromstring('1000 ALL')currency='ALL' (unchanged)
  • Price.fromstring('1000USD')currency='USD' (unchanged)
  • Price.fromstring('1000 USD')currency='USD' (unchanged)
  • Full test suite: 1059 passed, 0 failures

This pull request was prepared with the assistance of AI, under my direction and review.

Letter-only currency codes like ALL (Albanian Lek) and DA were being
matched as substrings inside words like ANNUALLY and DAILY. Add
negative lookbehind/lookahead for ASCII letters around purely
alphabetic symbols in the unsafe currency regex.

Fixes scrapinghub#71
@binggao1230
binggao1230 force-pushed the fix-71-currency-false-positive branch from 2128264 to a808798 Compare June 24, 2026 06:46
@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Sonnet reports:


price_parser/parser.py:261 —
Regression: alphabetic currency names no longer match in
inflected/plural forms

The (?![a-zA-Z]) lookahead blocks any match where the next character
is a letter. Several symbols in OTHER_CURRENCY_SYMBOLS appear in real
price strings with a trailing inflectional suffix — "100 kroons" fails
to match kroon (EEK), "100 Sucres" fails to match Sucre (XSU), and
any further-inflected form of Slovenian tolarjev (e.g. tolarjevih)
would also fail. The old or_regex matched all of these. These
currencies are obscure/obsolete, but the boundary is applied uniformly
to all alphabetic symbols, so the regression is structural, not limited
to these examples. Worth verifying against the full
CURRENCY_NATIONAL_SYMBOLS list for any in-production currencies with
similar plural forms.


price_parser/parser.py:268 —
Pre-existing but unfixed: SAFE_CURRENCY_SYMBOLS has the same
false-positive problem

_search_safe_currency is compiled with or_regex() which has no
word-boundary logic. SAFE_CURRENCY_SYMBOLS contains the alphabetic
entries "EUR", "euro", "eur", "CHF", "DKK", "lei". Confirmed
false positives: Price.fromstring("10.00", "leiden") returns
currency="lei", Price.fromstring("15.99", "european") returns
currency="euro". The PR halves the false-positive surface but leaves
the higher-priority _search_safe_currency path (which runs first in
extract_currency_symbol) unpatched.


tests/test_price_parsing.py — No
regression test added for the reported bug

The PR description specifically calls out
Price.fromstring('1000 ANNUALLY') → currency='ALL' and
Price.fromstring('1000 DAILY') → currency='DA' as the fixed cases.
Neither is in the test suite. A future refactor of
_make_unsafe_currency_regex or the OTHER_CURRENCY_SYMBOLS list will
re-introduce the bug with no CI signal.


price_parser/parser.py:261 —
Incomplete guard for non-ASCII alphabetic symbols

s.isalpha() returns True for Cyrillic (р, Р), Arabic (ریال),
and other non-ASCII alphabetic symbols, so they get wrapped in
(?<![a-zA-Z])…(?![a-zA-Z]). But [a-zA-Z] only covers ASCII letters —
adjacent Cyrillic or Arabic letters aren't blocked. This is not a
regression (old code had no guard), but it makes the fix inconsistent:
р in a Cyrillic word still matches, defeating the intent for that
script. Replacing [a-zA-Z] with [^\W\d_] (Unicode letters without
digits/underscore) would be uniform across scripts, though care is
needed to preserve digit-adjacency (1000USD must still match, so \b
alone is too strict).


Any thoughts?

@binggao1230

Copy link
Copy Markdown
Author

Addressed the boundary review in dccbcb5.

What changed:

  • Safe and unsafe currency lookup now share the same Unicode-letter boundary logic instead of ASCII-only (?![a-zA-Z]) handling.
  • Added regressions for the original false positives (1000 ANNUALLY, 1000 DAILY) and for the safe-currency fallback cases (leiden -> lei, european -> euro).
  • Added a Unicode-adjacent regression for 1000рруб.
  • Preserved simple plural s matches for longer alphabetic currency names such as euros, kroons, and Sucres.
  • Kept the existing Russian рублей/рубли behavior through a narrow руб suffix rule while preventing the previous труба false positive; the now-fixed Russian examples were moved out of strict xfail.

I did not try to support arbitrary inflections such as tolarjevih; doing that generically would need language-specific morphology or explicit symbol aliases, and a broad boundary relaxation would bring back the false positives this PR is fixing.

Verification:

  • uv run --no-project --with pytest --with attrs python -m pytest tests/test_price_parsing.py::test_currency_not_matched_inside_words tests/test_price_parsing.py::test_currency_boundaries_keep_valid_matches -q
  • uv run --no-project --with pytest --with attrs python -m pytest tests/test_price_parsing.py -q
  • uv run --no-project --with ruff ruff check price_parser/parser.py tests/test_price_parsing.py
  • uv run --no-project --with mypy==1.18.2 --with pytest==8.4.2 --with attrs mypy price_parser tests
  • git diff --check

@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.44%. Comparing base (64e213a) to head (4b1988c).

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #78      +/-   ##
==========================================
+ Coverage   98.23%   98.44%   +0.21%     
==========================================
  Files           3        3              
  Lines         113      129      +16     
  Branches       16       19       +3     
==========================================
+ Hits          111      127      +16     
  Misses          1        1              
  Partials        1        1              
Files with missing lines Coverage Δ
price_parser/parser.py 98.03% <100.00%> (+0.36%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Please, run pre-commit run --all-files.

@binggao1230

Copy link
Copy Markdown
Author

Ran pre-commit run --all-files --show-diff-on-failure and pushed the resulting ruff format change in c653efa.

Local verification now passes:

uv run --with pre-commit pre-commit run --all-files --show-diff-on-failure

Comment thread price_parser/parser.py
Comment on lines -252 to +279
_search_safe_currency = or_regex(SAFE_CURRENCY_SYMBOLS).search
_search_unsafe_currency = or_regex(OTHER_CURRENCY_SYMBOLS).search
_search_safe_currency = _make_currency_regex(SAFE_CURRENCY_SYMBOLS).search
_search_unsafe_currency = _make_currency_regex(OTHER_CURRENCY_SYMBOLS).search

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is or_regex dead code now?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — after this change nothing calls it; _make_currency_regex replaced both call sites. It isn't in __all__ and isn't re-exported from price_parser/__init__.py, but it is a public-looking module-level name, so someone could be importing it. Happy to drop it here or leave it and deprecate separately — tell me which you prefer and I'll push it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s deprecate.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a753e9aor_regex now emits a DeprecationWarning and its docstring points at _make_currency_regex. Suite still green.

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Could you complete coverage?

@binggao1230

Copy link
Copy Markdown
Author

Done in de876d1or_regex now has a direct test (test_or_regex_deprecated) asserting the DeprecationWarning and that the compiled pattern still matches. Patch coverage on the diff should be 100% now; 1074 pass.

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Please, run pre-commit

@binggao1230

Copy link
Copy Markdown
Author

Done — pre-commit run --all-files is clean after narrowing the new deprecation-warning assertion (PT030); pushed in 4b1988c. Full suite: 1074 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problem with annually/daily

2 participants