Skip to content

feat: Domain config through file, Sandhi improvements, and Kannada numeral supportFeature domain file - #15

Merged
kavyamanohar merged 5 commits into
mainfrom
feature-domain-file
Mar 11, 2026
Merged

feat: Domain config through file, Sandhi improvements, and Kannada numeral supportFeature domain file#15
kavyamanohar merged 5 commits into
mainfrom
feature-domain-file

Conversation

@kavyamanohar

Copy link
Copy Markdown
Member

Summary

  • Domain config standardization (f42de57): Introduced DomainConfig class with factory methods (legal(), medical(), financial(), technical()) and file-based configuration via DomainConfig.from_file(). Bundled
    domain config files added under src/dicterrors/config/.
  • Sandhi alignment scope (eb9a904): Sandhi split/merge rules now apply across all token categories except PUNCT and NUMERAL, improving alignment accuracy for agglutinative Indic text.
  • Hyphen normalization & script-agnostic word boundaries (23ad7d5): Punctuation splitting is now smart — word-medial hyphens (e.g., "ice-cream") are preserved as single tokens during tokenization. Danda (।) added to punctuation list. Word boundary regex made Unicode/script-agnostic so domain terms match correctly adjacent to Indic characters.
  • Kannada numeral normalization (24c69d2): Extended numeral pattern support to recognize Kannada digit sequences alongside existing Malayalam/Hindi patterns. Expanded legal_terms.txt with additional
    terminology.
  • use_sandhi toggle + docs update (502b967): Added use_sandhi: bool = True parameter to align_arrays, text_error_rates, token_error_rates, and compute_sample_errors — allowing Sandhi detection to be disabled
    for non-agglutinative languages. Sandhi Detection toggle added to visualizer sidebar. Mismatch penalty multiplier tuned from 0.5 → 0.2. README and CLAUDE.md updated to reflect new API params.

Test plan

  • uv run python -c "from dicterrors import text_error_rates; print('OK')" — core import
  • uv run python examples/error_report.py — single-sample report
  • uv run python examples/batch_evaluate.py — batch evaluation (needs a predictions JSONL)
  • streamlit run visualizer.py — verify Sandhi Detection toggle appears and works
  • Test use_sandhi=False suppresses Sandhi hits in output

kavyamanohar and others added 5 commits February 14, 2026 15:26
- Add factory methods for domain configs (DomainConfig.legal(), medical(), financial(), technical())
- Bundle config files in package (src/dicterrors/config/)
- Add 200+ legal terms (English abbreviations, Malayalam terms, Latin phrases)
- Fix word boundary bug preventing substring matches
- Support multi-word phrases as single tokens (Police Station, Prima facie)
- Update all examples and docs to use new factory methods

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add normalize_kannada_numerals() function to convert Kannada digits (೦-೯)
  to Arabic numerals for semantic matching
- Fix correct token aggregation in measure_batch.py (properly accumulate
  correct counts instead of calculating from total - errors)
- Expand legal domain vocabulary with 148 new Malayalam legal terms
- Add `use_sandhi: bool = True` param to align_arrays, text_error_rates,
  token_error_rates, and compute_sample_errors for optional Sandhi detection
- Add Sandhi Detection toggle to visualizer sidebar
- Tune mismatch penalty multiplier from 0.5 to 0.2 for better alignment scoring
- Update README.md and CLAUDE.md API docs with use_sandhi param and visualizer toggles

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@kavyamanohar
kavyamanohar merged commit 8468b21 into main Mar 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant