Context
As identified in the Contributor Roadmap (Level 1: Quick Wins), web-scraped and user-entered text often contains invisible unicode whitespace that can interfere with sentinel detection if not properly tested.
Task
Add unit tests in tests/test_strings.py covering unicode whitespace characters (such as non-breaking space \u00A0 and zero-width space \u200B) to verify they are handled as expected during sentinel normalization and whitespace stripping.
Acceptance Criteria
Pointers
New to the codebase? Comment here and we'll help you get set up! You can also check our First Contribution Guide.
Context
As identified in the Contributor Roadmap (Level 1: Quick Wins), web-scraped and user-entered text often contains invisible unicode whitespace that can interfere with sentinel detection if not properly tested.
Task
Add unit tests in
tests/test_strings.pycovering unicode whitespace characters (such as non-breaking space\u00A0and zero-width space\u200B) to verify they are handled as expected during sentinel normalization and whitespace stripping.Acceptance Criteria
tests/test_strings.pytesting strings with\u00A0(non-breaking space) and\u200B(zero-width space)."\u00A0N/A\u00A0"and"\u200BNA\u200B").strip_whitespace=Trueand defaultfd.clean()behavior.pytest tests/test_strings.py --no-cov.ruff check tests/test_strings.py).Pointers
tests/test_strings.pysrc/freshdata/steps/sentinels.pyNew to the codebase? Comment here and we'll help you get set up! You can also check our First Contribution Guide.