Skip to content

Add unit tests for unicode whitespace and zero-width spaces in sentinel normalization #498

Description

@JohnnyWilson16

Context

As identified in the Contributor Roadmap (Level 1: Quick Wins), web-scraped and user-entered text often contains invisible unicode whitespace that can interfere with sentinel detection if not properly tested.

Task

Add unit tests in tests/test_strings.py covering unicode whitespace characters (such as non-breaking space \u00A0 and zero-width space \u200B) to verify they are handled as expected during sentinel normalization and whitespace stripping.

Acceptance Criteria

  • Add test case(s) in tests/test_strings.py testing strings with \u00A0 (non-breaking space) and \u200B (zero-width space).
  • Test interactions with standard sentinels (e.g., "\u00A0N/A\u00A0" and "\u200BNA\u200B").
  • Ensure tests verify both strip_whitespace=True and default fd.clean() behavior.
  • Tests execute fast and pass cleanly with pytest tests/test_strings.py --no-cov.
  • Code adheres to style guidelines (ruff check tests/test_strings.py).

Pointers


New to the codebase? Comment here and we'll help you get set up! You can also check our First Contribution Guide.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions