Add Haitian Creole (ht) language support locale

[speedyk-005]

Summary

Adds Haitian Creole ("ht") support to dateparser, including absolute dates (months, weekdays, numeric formats) and relative/freshness expressions.

Haitian Creole is spoken by more than 11 million people, and this adds it to the list of supported languages.

I am a native Haitian Creole speaker and validated the translations against native usage.

Sources checked for correctness:

• https://rise2haitiancreole.com/learningcenter/basics-to-haitian-creole/time/
• https://preply.com/en/blog/months-in-creole/

Changes

• Added Haitian Creole translation data in "dateparser_data/supplementary_language_data/date_translation_data/ht.yaml"
• Regenerated "dateparser/data/date_translation_data/ht.py"
• Registered "ht" in "dateparser/data/languages_info.py"
• Added 18 test cases in "tests/test_languages.py"
• Updated "docs/supported_locales.rst" to include "ht"/"ht-HT'"

Testing

• Manual validation using "Locale('ht').translate()" and "parse(languages=['ht'])"
• "pytest tests/test_languages.py -k ht -v" passed

[AdrianAtZyte]

Does CLDR track Haitian Creole? If so, maybe we can get most of this data from there, like we do for most languages, instead of through custom data.

[speedyk-005]

@AdrianAtZyte the CLDR repo contains a Haitian Creole locale (language code "ht"), but the current ht files in common/main do not include the usual date/time data (month/day names and date format patterns), so full date localization is not present in those files.

[AdrianAtZyte]

A few things to fix before this can be merged:

• Tests are not passing: tests/test_data.py::TestLocaleInfo::test_name fails for ht-HT, because the YAML has no locale_specific section and so the locale name stays ht. Since there is no CLDR data for ht, I would drop ht-HT altogether and register "ht": [] in languages_info.py.
• Minutes never parse: nan 5 minit, sa gen 5 min and nan minit sa all return None. The relative keys must use English units, so 0 minit ago, \1 minit ago and in \1 minit need to become minute. Please, add a test for minutes too, none of the new tests cover them.
• Some single tokens look too short to be safe as dictionary words: a, e, m, s, de, en. de: '2' in particular changes what a bare de parses to with language detection. Are these real usage, or completeness? I would drop the ones that are not real usage.
• a is both a year unit and a skip word; skip wins, so the year entry is dead. Probably worth removing one of them.
• The ht entry in language_locale_dict sits between ug and uk, so it looks hand-placed. Please, run dateparser_scripts/order_languages.py so that language_order reflects the actual population data.

[speedyk-005]

Thanks for the review, @AdrianAtZyte! I'll update them accordingly.

Quick note: en and de are actual Haitian Creole words used for counting (en [1], de [2], twa [3]... dis [10]), and e is also used to mean 'and'.

[speedyk-005]

All points are handled, plus one thing that fell out of re-running the data script.

1. Since CLDR has nothing for ht, I registered "ht": [] in language_locale_dict and removed it from language_map. That also fixed the test_name failure on ht-HT.
2. Re-ran order_languages.py, so ht is now ordered by population instead of sitting between ug and uk. But that fresh pull moved a bunch of other languages around too (w3techs data has shifted) and tweaked a few locale lists from CLDR 44.1, e.g. en gained en-AE/en-ID/en-MV and ff lost ff-CM/ff-GN/ff-MR. All 20360 data tests and the language tests still pass.
3. Dropped a (year) and j (day), plus their regex entries. Kept m and s, since ht is French-based and those are real abbreviations.
4. Minute keys fixed. The keys had been changed to minit by an accidental search-and-replace.

[codspeed]

Merging this PR will not alter performance

✅ 7 untouched benchmarks

---

_{Comparing speedyk-005:feat/haitian-creole-locale (5920337) with master (a131f78)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.18%. Comparing base (6b5f17d) to head (5920337).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1375      +/-   ##
==========================================
- Coverage   97.18%   97.18%   -0.01%     
==========================================
  Files         236      237       +1     
  Lines        3059     3058       -1     
==========================================
- Hits         2973     2972       -1     
  Misses         86       86              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[speedyk-005]

Hey, any idea why the CI failed? Was it because of my changes? Let me know!

[AdrianAtZyte]

Was it because of my changes?

Seems so. Can you not reproduce this issues locally?

[speedyk-005]

Thanks! Fixed now!! The CLDR 44.1 refresh regenerated language_locale_dict, silently dropping en-US and the ff-CM/ff-GN/ff-MR locale codes, so the loader's _isvalidlocale rejected them and the affected tests (test_loading, ff tests in test_clean_api) failed.

[AdrianAtZyte]

Please, run pre-commit run --all-files.

[speedyk-005]

Done!

[AdrianAtZyte]

Closer, thanks. A few things left:

• uv.lock is not tracked in this repo, please, drop it from the pull request.
• dateparser/data/languages_info.py does not match what order_languages.py generates: ht is between bm and bn in language_map (the script sorts it alphabetically) and between ug and zgh in language_locale_dict (the script follows language_order). Since the fresh w3techs data reshuffled dozens of other languages and you then had to hand-revert the CLDR fallout, I would rather not mix that refresh into this pull request. Please, keep language_order as it is on master, insert ht right after bm (where the new data put it), and place the ht entries in the other two dicts where the script would.
• m is a month unit in the dictionary, but the simplifications in the same file treat it as minutes (mwens pase 5 m → 5 minit, 5h30m → 5:30). nan 5 m currently parses as 5 months. I would drop m from month and the matching in \1 month regex.
• Minor: semèn sa and ane sa are there, but mwa sa only works as nan mwa sa.

[speedyk-005]

Done!! As for "ane sa", it is more natural than "nan ane sa"

[AdrianAtZyte]

Thanks, but the last three commits do not address the review:

• a64d7ec and cecec6b edit the generated dateparser/data/date_translation_data/ht.py only. The source of truth is ht.yaml, and ht.py must be regenerated from it. tests/test_dateparser_data_integrity.py now fails because they differ. Please, make the changes in the YAML and regenerate.
• You changed semèn sa into nan semèn sa, which is the opposite of what I asked. mwa sa still does not parse without nan; I would add mwa sa to 0 month ago and leave semèn sa as it was.
• The language_order refresh is still there, and ht is still misplaced in language_map and language_locale_dict. Please, revert languages_info.py to master and add only the three ht entries: after bm in language_order, after hsb in language_map, and after bm in language_locale_dict.

[speedyk-005]

My bad!! I'll fix them.

[speedyk-005]

The language_info.py is already reverted, maybe you were seeing an old commit. See: https://github.com/scrapinghub/dateparser/pull/1375/changes

[speedyk-005]

Done! I mirror m shorthand and add un-prefixed mwa sa in ht in 6882682

[AdrianAtZyte]

Thanks, this is close. Two things left:

• ht is still misplaced in language_map: it is after bm, but order_languages.py builds that dict with sorted(language_order), so it goes after hsb.
• e in skip breaks the bare è hour unit. Normalization strips accents, so è becomes e and gets skipped, and anything that ends up as N è without nan or sa gen in front loses its unit and parses as a day of month: 5 è, mwens pase 5 è and mwens pase 5 h (your own simplification turns it into 5 è) all return September 5 instead of 5 hours ago. Dropping e from skip fixes all of them; I checked after regenerating. mè for Wednesday has the same problem, it normalizes to me, which is May, so mè 3 jen translates to may 3 june and returns None. I would drop that abbreviation too.

Please, make the changes in the YAML and regenerate ht.py.

[speedyk-005]

Done!!