feat: Feature date tokenization - #13
Merged
Merged
Conversation
Use combined denominator (sum of all token categories) for calculating error rates instead of per-category denominators. This provides contextually meaningful metrics that account for class imbalance, preventing misleading rates for sparse categories like legal entities. Changes: - Update token_error_rates() to use combined denominator across WORD, LEGAL, NUMERAL, and PUNCT categories - Update calculate_rates() in batch aggregation to apply same normalization - Add combined_total field to reports for transparency - Add type hints to function signatures Alignment improvements: - Implement category-aware gap penalties (lighter penalty for punctuation) - Rename scoring parameters for clarity (match_reward, mismatch_default_penalty, etc.) - Add type hints to alignment functions Tokenizer improvements: - Add PW and CW to legal entity patterns - Add type hints Visualizer improvements: - Update metric labels for clarity (General WER, Legal WER, etc.) - Reorganize sidebar controls - Update parameter names to match align.py changes Benefits: - Legal entities with sparse occurrence no longer show inflated error rates (e.g., 1 error out of 1 legal word now shows as 0.79% instead of 100% when total corpus has 126 tokens) - All error rates now represent proportion within entire corpus - Cross-category error rates are directly comparable Documentation: - Update README with detailed testing and visualization sections - Document token categories and normalized error rate calculation - Add current status section (implemented vs TODO features) - Clarify that only English legal abbreviations are implemented
- Fix error_report.py: update imports to use correct API functions (legal_aware_tokenizer, align_arrays, token_error_rates) - Fix text_alignment.py: handle token tuples correctly and add visual match indicators - Add reporting.py module with shared formatting functions: format_metrics_dict, format_dataset_table, format_error_counts_table, format_alignment_table - Refactor measure_batch.py and visualizer.py to use shared functions - Add test_reporting.py with comprehensive tests for reporting module - Eliminate code duplication across CLI and web UI tools All tests passing. Backward compatibility maintained. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Remove test_*.py files from git tracking while keeping them on disk for local development. These files are now ignored via .gitignore. Files untracked: - test_batch_aggregation.py - test_combined_denominator.py - test_edge_cases.py - test_reporting.py
Implement optional output_file parameter in compute_sample_errors() to save detailed per-sample error reports as JSONL. Each sample includes category-wise breakdown (WORD/LEGAL/NUMERAL/PUNCT) with error rates, substitutions, insertions, deletions, correct counts, and Sandhi hits. Changes: - measure_batch.py: Add output_file parameter with JSONL export - batch_evaluate.py: Save detailed results to evaluation-detailed.jsonl - .gitignore: Add /test*.py pattern to ignore root-level test files
- Enhance batch evaluation and reporting modules - Update tokenization functionality - Add CLAUDE.md for development documentation - Fix visualizer session state bug (NameError on file uploader interaction) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Create constants.py with shared constants and utility functions - Use full field names throughout (substitutions, insertions, deletions, correct, sandhi_hits) - Remove duplicate category definitions and hardcoded lists - Replace magic numbers with named constants Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace hardcoded legal_aware_tokenizer with domain_aware_tokenizer
supporting any domain (legal, medical, financial, etc.) via DomainConfig.
Breaking changes:
- legal_aware_tokenizer → domain_aware_tokenizer(text, domain_config)
- CAT_LEGAL removed, use domain_config.category
- All functions require explicit domain_config parameter
New features:
- DomainConfig class with list/regex pattern support
- Pre-defined LEGAL_DOMAIN and MEDICAL_DOMAIN
- Custom domains: DomainConfig("medical", ["mg", "ml"], label="MER")
- Dynamic categories and error rate labels
Updated: All example scripts, visualizer, README with API reference
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
kavyamanohar
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Major refactor adding domain-specific error tracking, token normalization, and improved error rate calculations for ASR evaluation in legal/medical/custom domains.
Key Features
Files Changed