diff --git a/.gitignore b/.gitignore index 139acbe..1d55636 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,10 @@ ENV/ .coverage htmlcov/ +# Lint / type-check caches +.ruff_cache/ +.mypy_cache/ + # Jupyter Notebook .ipynb_checkpoints @@ -32,5 +36,21 @@ htmlcov/ *.swp .DS_Store -# Root-level test files (kept locally for development) -/test*.py +# AI/dev harness configuration (kept local, not in public repo) +CLAUDE.md +AGENTS.md +.claude/ + +# Internal-only documentation +docs/internal/ + +# Benchmark / evaluation outputs +# Only examples/ SUBDIRECTORIES are ignored; top-level example scripts stay tracked. +examples/*/ +# User-specific domain configs sitting flat in examples/ or src/scribe/config/ +examples/custom_*.txt +src/scribe/config/custom_*.txt +# Ad-hoc test scripts dropped in examples/ (separate from the tests/ tree) +examples/test_*.py +# Root-level eval result directory +malayalam-model-results/ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..670d5b8 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for describing the origin of the Work and + reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Support. While redistributing the Work or + Derivative Works thereof, You may choose to offer, and charge a + fee for, acceptance of support, warranty, indemnity, or other + liability obligations and/or rights consistent with this License. + However, in accepting such obligations, You may act only on Your + own behalf and on Your sole responsibility, not on behalf of any + other Contributor, and only if You agree to indemnify, defend, and + hold each Contributor harmless for any liability incurred by, or + claims asserted against, such Contributor by reason of your accepting + any such warranty or support. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2026 Adalat AI + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing + permissions and limitations under the License. diff --git a/README.md b/README.md index aa8dd86..39da950 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,32 @@ -# DictErrors — Dictation Error Analysis for Indic Languages +# SCRIBE — Diagnostic Evaluation for Indic & Domain-Specific ASR -[![Python Version](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) +[![PyPI](https://img.shields.io/pypi/v/scribe-eval.svg)](https://pypi.org/project/scribe-eval/) +[![Python](https://img.shields.io/pypi/pyversions/scribe-eval.svg)](https://pypi.org/project/scribe-eval/) +[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) -DictErrors is a specialized tool for analyzing ASR (Automatic Speech Recognition) transcription errors in Indic languages (Malayalam, Kannada). It provides fine-grained error metrics by categorizing tokens into base categories (WORD, NUMERAL, PUNCT) and optional domain-specific categories (LEGAL, MEDICAL, TECH, or custom). +`scribe-eval` is the open-source evaluation framework introduced in the SCRIBE +paper (*Diagnostic Evaluation and Rich Transcription Models for Indic ASR*, +under review at Interspeech 2026). It provides fine-grained error metrics for +ASR systems on Indic languages (Malayalam, Kannada, Hindi, ...) and on +domain-specific transcription (legal, medical, technical). -Domain-critical terminology is shielded from incorrect splitting and tracked separately — so a single misrecognized legal term doesn't inflate your general WER. +Token categories are decomposed into base classes (WORD, NUMERAL, PUNCT) and +optional domain classes (LEGAL, MEDICAL, TECH, or custom). Domain-critical +terminology is shielded from incorrect splitting and tracked separately — so a +single misrecognized legal term doesn't inflate your general WER. ## Installation ```bash -git clone https://github.com/adalat-ai-tech/dict-errors.git -cd dict-errors -uv sync && uv pip install -e . +pip install scribe-eval # core library +pip install 'scribe-eval[visualizer]' # adds Streamlit UI +pip install 'scribe-eval[charts]' # adds matplotlib charts ``` ## Quick Start ```python -from dicterrors import text_error_rates, DomainConfig +from scribe import text_error_rates, DomainConfig ref = "charged u/s 302 IPC on 22.05.2023" hyp = "charged u/s 303 IPC on 22.05.2023" @@ -47,7 +55,7 @@ print(f"NER: {report['NUMERAL']['error_rate']:.2%}") | PUNCT | base | PER | Punctuation marks | | LEGAL | domain | LER | Indian legal terminology (u/s, r/w, PW1, Ext.A) | | MEDICAL | domain | MER | Medical units and dosages (mg, ml, 500mg) | -| TECH | domain | TER | Technical abbreviations (API, SDK, v1.0) | +| TECH | domain | TchER | Technical abbreviations (API, SDK, v1.0) | | Custom | domain | configurable | Define your own with lists or regex patterns | ## Domain Configuration @@ -56,11 +64,16 @@ Factory methods for bundled domains: `DomainConfig.legal()`, `DomainConfig.medic File-based and custom inline configs are also supported. See [docs/domain-configuration.md](docs/domain-configuration.md). +## Examples + +Runnable scripts under [`examples/`](examples/) demonstrate alignment, +single-sample reports, domain-config patterns, and full batch evaluation. +See [`examples/README.md`](examples/README.md) for the full index. + ## Batch Processing ```bash -cd examples/ -uv run batch_evaluate.py --input predictions.jsonl --domain-config config/legal_terms.txt --analysis --chart +uv run examples/batch_evaluate.py --analysis --chart ``` See [docs/batch-processing.md](docs/batch-processing.md) for the Python API, CLI arguments, and output schema. @@ -68,14 +81,62 @@ See [docs/batch-processing.md](docs/batch-processing.md) for the Python API, CLI ## Interactive Visualizer ```bash -streamlit run visualizer.py +pip install 'scribe-eval[visualizer]' +scribe-visualizer ``` See [docs/visualizer.md](docs/visualizer.md). ## Dependencies -- `levenshtein>=0.27.1`, `jiwer>=4.0.0`, `streamlit>=1.53.0`, `tabulate>=0.9.0` +Core: `jiwer>=4.0.0`, `levenshtein>=0.27.1`, `tabulate>=0.9.0` + +Optional extras: `matplotlib` (for `[charts]`), `streamlit` and `pandas` (for `[visualizer]`). + +## Development + +```bash +git clone https://github.com/adalat-ai-tech/scribe-eval.git +cd scribe-eval +uv sync --all-extras --dev # core + [charts] + [visualizer] + [dev] +``` + +### Running tests + +```bash +uv run pytest # full suite +uv run pytest tests/test_analysis.py # one file +uv run pytest -k sandhi # name pattern (-k matches by substring) +uv run pytest -v # verbose, with each test name +uv run pytest --cov=scribe # with coverage +``` + +Tests are organised one file per library module under [`tests/`](tests/), plus +[`tests/test_paper_cases.py`](tests/test_paper_cases.py) for end-to-end golden +cases from the SCRIBE paper. `pytest` itself ships with the `[dev]` extra, so +`uv sync --all-extras --dev` (above) is required first. + +### Lint and format + +```bash +uv run ruff check src tests examples # lint +uv run ruff format src tests examples # auto-format +``` + +See [`docs/architecture.md`](docs/architecture.md) for the module map and a +glossary of project-specific terminology (sandhi, combined denominator, TER, +Accuracy, ...). + +## Citation + +The SCRIBE paper is currently under anonymous review at Interspeech 2026. A +BibTeX entry will be added here once the proceedings or arXiv preprint are +publicly available. Until then, please cite this repository directly via the +"Cite this repository" button on GitHub. + +## License + +Licensed under the [Apache License 2.0](LICENSE). ## Acknowledgements diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..e14e60d --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,228 @@ +# Architecture + +scribe-eval is a five-stage pipeline. Each stage is a module under +`src/scribe/`. Use the high-level API (`text_error_rates`) to run the +whole pipeline, or import individual stages for custom flows. + +## Pipeline + +``` +raw text ──▶ tokenize ──▶ align ──▶ measure ──▶ aggregate ──▶ report + (per side) (paired) (per sample) (across (CLI / UI) + samples) +``` + +| Arrow | Payload | +|---|---| +| tokenize → align | `(tokens, tags, normalized_tokens)` per side | +| align → measure | aligned `[(text, tag), ...]` pairs (gaps as `("**", "GAP")`) | +| measure → aggregate | per-sample report `{WORD: {...}, NUMERAL: {...}, ...}` | +| aggregate → report | `{"overall": ..., "by_dataset": {...}}` | + +## Quick example + +End-to-end, with the high-level API: + +```python +from scribe import text_error_rates, DomainConfig + +ref = "charged u/s 302 IPC on 22.05.2023" +hyp = "charged u/s 303 IPC on 22/05/2023" + +report = text_error_rates(ref, hyp, DomainConfig.legal()) +print(f"WER: {report['WORD']['error_rate']:.2%}") # 0.00% — words match +print(f"LER: {report['LEGAL']['error_rate']:.2%}") # 0.00% — u/s, IPC shielded +print(f"NER: {report['NUMERAL']['error_rate']:.2%}") # 16.67% — 302 → 303 + # (date is normalized away) +``` + +The same flow, stage by stage, when you need finer control: + +```python +from scribe import ( + DomainConfig, domain_aware_tokenizer, align_arrays, + token_error_rates, token_error_details, +) + +domain = DomainConfig.legal() + +# 1. tokenize each side +t1, g1 = domain_aware_tokenizer(ref, domain) # tokens, tags +t2, g2 = domain_aware_tokenizer(hyp, domain) + +# 2. align (Needleman-Wunsch with sandhi/category-aware scoring) +aligned_ref, aligned_hyp, _ = align_arrays(t1, g1, t2, g2) + +# 3. measure (rates and per-token error records) +rates = token_error_rates(aligned_ref, aligned_hyp, domain) +details = token_error_details(aligned_ref, aligned_hyp, domain) +# details: [{"error_type": "substitution", "category": "NUMERAL", +# "ref_token": "302", "hyp_token": "303"}, ...] +``` + +For batch evaluation across a JSONL dataset, see +[batch-processing.md](batch-processing.md). + +## Module map + +| Module | Owns | Key callables | +|---|---|---| +| `tokenize.py` | Splitting text into tagged tokens; date / time / currency detection | `domain_aware_tokenizer` | +| `domain_config.py` | Loading and applying domain-shielding patterns | `DomainConfig`, `.legal()` / `.medical()` / `.technical()`, `.from_file()` | +| `normalize.py` | Canonicalising date / currency / numeral surface forms before comparison | `normalize_token`, `normalize_date`, `normalize_currency` | +| `align.py` | Modified Needleman–Wunsch with token-type-aware scoring; sandhi merge / split detection | `align_arrays`, `DEFAULT_WEIGHTS` | +| `measure.py` | Per-sample error rates and per-token error records | `text_error_rates`, `token_error_rates`, `text_error_details` | +| `measure_batch.py` | JSONL ingestion, per-sample running, per-dataset & overall aggregation | `compute_sample_errors`, `compute_aggregate_metrics`, `aggregate_error_details` | +| `analysis.py` | Category contributions, frequent substitutions / deletions / insertions / sandhi merges / sandhi splits, Token Error Rate (TER) | `compute_error_summary`, `compute_category_contributions`, `compute_frequent_sandhi_merges`, `compute_frequent_sandhi_splits` | +| `reporting.py` | Formatters shared by the CLI and Streamlit UI | `format_metrics_dict`, `format_contribution_table`, `format_alignment_table` | +| `charts.py` | matplotlib chart generation (optional `[charts]` extra) | `category_breakdown_chart` | +| `visualizer/` | Streamlit app and `scribe-visualizer` console script (optional `[visualizer]` extra) | `app.py`, `__main__.py` | +| `constants.py` | Category names and helpers | `CAT_WORD`, `CAT_NUMERAL`, `get_categories(domain_config)` | + +## Stage-by-stage examples + +Small, runnable snippets for the parts of the pipeline you most often +reach into directly. + +### Tokenize + +```python +from scribe import domain_aware_tokenizer, DomainConfig + +tokens, tags = domain_aware_tokenizer("filed u/s 302 IPC", DomainConfig.legal()) +# tokens: ['filed', 'u/s', '302', 'IPC'] +# tags: ['WORD', 'LEGAL', 'NUMERAL', 'LEGAL'] +``` + +Both `u/s` and `IPC` are LEGAL — they're tracked under LER, not WER, so +a misrecognised legal abbreviation doesn't inflate your general word +error rate. `u/s` also stays atomic instead of being split on `/`. + +### Normalize + +```python +from scribe.normalize import normalize_token + +normalize_token("22.05.2023", "NUMERAL") # '22-05-2023' (canonical date) +normalize_token("10,500", "NUMERAL") # '10500' (commas stripped) +normalize_token("೧೫.೦೫.೨೦೨೩", "NUMERAL") # '15-05-2023' (Kannada → Arabic) +``` + +Normalization runs *post-alignment* on each surviving substitution pair: +if both sides normalize to the same string, the pair is reclassified +from "sub" to "correct". Enabled by default; opt out with +`text_error_rates(..., normalize=False)`. + +### Align (sandhi-aware) + +```python +from scribe import align_arrays, domain_aware_tokenizer + +t1, g1 = domain_aware_tokenizer("ഇന്ന് അല്ലെങ്കിൽ", None) +t2, g2 = domain_aware_tokenizer("ഇന്നല്ലെങ്കിൽ", None) +ref, hyp, _ = align_arrays(t1, g1, t2, g2) +# ref: [('MERGE:ഇന്ന് അല്ലെങ്കിൽ', 'WORD')] +# hyp: [('ഇന്നല്ലെങ്കിൽ', 'WORD')] +``` + +The aligner tags merge / split events with `MERGE:` / `SPLIT:` prefixes +on the affected side. Downstream, `measure.py` reads those prefixes and +records the event as a *sandhi correction* — not an error. + +### Measure — rates and per-token records + +```python +from scribe import text_error_rates, text_error_details + +rates = text_error_rates("alpha beta gamma", "alpha delta epsilon", None) +# rates['WORD']: {'error_rate': 0.667, 'substitutions': 2, 'correct': 1, +# 'total_ref': 3, 'sandhi_hits': 0, ...} + +details = text_error_details("alpha beta gamma", "alpha delta epsilon", None) +# [{'error_type': 'substitution', 'category': 'WORD', +# 'ref_token': 'beta', 'hyp_token': 'delta'}, +# {'error_type': 'substitution', 'category': 'WORD', +# 'ref_token': 'gamma', 'hyp_token': 'epsilon'}] +``` + +`text_error_details` is the input to the frequent-error analysis below. +For a sandhi event it emits `{"error_type": "sandhi_merge"|"sandhi_split", ...}` +records (no contribution to sub / ins / del counters). + +### Analyse — frequent errors and sandhi events + +```python +from scribe import ( + text_error_details, text_error_rates, + compute_error_summary, compute_aggregate_metrics, + format_frequent_errors_table, +) + +pairs = [ + ("ഇന്ന് അല്ലെങ്കിൽ നാളെ", "ഇന്നല്ലെങ്കിൽ നാളെ"), # merge + ("ഇന്ന് അല്ലെങ്കിൽ പിന്നെ", "ഇന്നല്ലെങ്കിൽ പിന്നെ"), # merge (repeat) + ("നാളെ വരാം", "നാളെ പോകാം"), # plain sub +] +details = [] +samples = [] +for r, h in pairs: + details.extend(text_error_details(r, h, None)) + samples.append({"detailed_report": text_error_rates(r, h, None), + "source_dataset": "demo"}) + +agg = compute_aggregate_metrics(samples) +summary = compute_error_summary(agg["overall"], details, top_n=5) + +merge_rows = format_frequent_errors_table( + summary["frequent_sandhi_merges"], "sandhi_merge", 5 +) +# [{'Rank': 1, 'Category': 'WORD', 'Reference': 'ഇന്ന് അല്ലെങ്കിൽ', +# 'Hypothesis': 'ഇന്നല്ലെങ്കിൽ', 'Count': 2}] +``` + +`compute_error_summary` returns a single dict with all per-category +contributions, top-N substitutions / deletions / insertions, and the +two new top-N sandhi tables (`frequent_sandhi_merges`, +`frequent_sandhi_splits`). The CLI (`examples/batch_evaluate.py +--analysis`) and the Streamlit visualizer both render straight from +this dict. + +## Where to make a change + +| You want to... | Touch | +|---|---| +| Add a new bundled domain | `src/scribe/config/_terms.txt` + a factory method in `domain_config.py` | +| Recognise a new numeral form (e.g. ISO-8601 dates) | The numeral regex in `tokenize.py`, plus a matching `normalize_*` in `normalize.py` | +| Tweak alignment scoring | `DEFAULT_WEIGHTS` in `align.py` | +| Add a new aggregate metric | The math in `analysis.py`, the formatter in `reporting.py` | +| Add a CLI flag | `examples/batch_evaluate.py` — the CLI lives in `examples/`, not in the library | +| Add a UI element | `src/scribe/visualizer/app.py` | + +## Tests mirror the pipeline + +`tests/` has one file per module (`test_tokenize.py`, `test_align.py`, +`test_measure.py`, `test_normalize.py`, `test_measure_batch.py`, +`test_reporting.py`, `test_domain_config.py`), plus +`test_paper_cases.py` for end-to-end golden cases from the SCRIBE +paper. When you change a module, run its corresponding test file first. + +## Key design decisions + +- **Combined denominator** — error rates are `(category errors) / (total tokens across all categories)`, not `(category errors) / (category tokens)`. Stops sparse categories (e.g. 1 LEGAL error in 1 LEGAL token) reading as 100%. Implemented in `measure.py::token_error_rates`. +- **Domain shielding** — domain entities (`u/s`, `r/w`, `PW1`) are extracted *before* general tokenization so they stay atomic and are tracked under their own category. Implemented across `tokenize.py` + `domain_config.py`. +- **Sandhi awareness** — the alignment step detects when ASR has merged or split adjacent words (common in agglutinative Indic languages) and counts those separately from substitutions. The detected pairs are also surfaced as their own frequent-event tables (`frequent_sandhi_merges`, `frequent_sandhi_splits`) alongside the substitution / deletion / insertion tables, so recurring sandhi patterns are diagnosable at a dataset level. Implemented in `align.py` (detection) + `analysis.py` (aggregation). Disable with `use_sandhi=False` for non-agglutinative languages. +- **Two error-rate views per category** — `error_rate` (errors / category_ref) for in-isolation accuracy, `combined_total` (errors / total_ref) for contribution to overall TER. The Streamlit UI shows both side-by-side. + +## Glossary + +Quick reference for terms used throughout the codebase, docs, and the +SCRIBE paper. Each entry points at the module that owns the concept. + +- **Sandhi** — in agglutinative Indic languages, the phonological merging of adjacent words at morpheme boundaries (e.g. `ഇന്ന്` + `അല്ലെങ്കിൽ` → `ഇന്നല്ലെങ്കിൽ`). ASR systems often produce one form when the reference uses the other. +- **Sandhi correction** — an alignment hit where one reference token spans two hypothesis tokens (split) or two reference tokens collapse into one hypothesis token (merge). Tracked separately from substitutions because the underlying word identity is preserved. See `align.py`. +- **Combined denominator** — the total reference-token count across all categories, used as the divisor for every category's error rate. Prevents 1-error-in-1-token categories from reading as 100%. See `measure.py::token_error_rates`. +- **Domain shielding** — extracting domain-critical multi-character tokens (e.g. `u/s`, `r/w`, `PW1`) before general tokenization so they stay atomic and aren't split on punctuation. See `tokenize.py` + `domain_config.py`. +- **TER (Token Error Rate)** — the headline overall error rate: `(sub + ins + del) / total_ref`, where `total_ref` is the combined-denominator count of reference tokens across all categories. Equivalently, TER is the sum of every category's `error_rate` (since they share the same denominator). +- **Accuracy** — `total_correct / total_ref`, the fraction of reference tokens recovered exactly. **Accuracy and TER are independent quantities** — they do not sum to 100% in general because (a) insertions appear in the TER numerator but not in the reference token count, and (b) sandhi hits count as correct but consume two reference tokens per single hypothesis token. Both numbers are reported side-by-side in the CLI and visualizer. +- **Error rate vs Impact on Total** — every category exposes two numbers. `error_rate = (sub + ins + del) / category_ref` answers "how accurate is the model on this category in isolation". `Impact on Total = (sub + ins + del) / total_ref` answers "how much does this category contribute to TER". Across categories the *Impact on Total* values sum to TER. +- **Gap penalty / DP weight** — in the modified Needleman–Wunsch alignment, the cost of inserting a gap on either side. Tuned per-category in `DEFAULT_WEIGHTS` (align.py); punctuation gaps are cheaper than word or domain gaps because punctuation errors carry less semantic weight. diff --git a/docs/batch-processing.md b/docs/batch-processing.md index a472af0..0f25000 100644 --- a/docs/batch-processing.md +++ b/docs/batch-processing.md @@ -1,6 +1,6 @@ # Batch Processing -DictErrors supports processing multiple samples from a JSONL file and aggregating metrics across datasets. +scribe-eval supports processing multiple samples from a JSONL file and aggregating metrics across datasets. ## Input Format @@ -17,7 +17,7 @@ Each line in the JSONL file must have: ### Basic batch evaluation ```python -from dicterrors import compute_sample_errors, compute_aggregate_metrics, DomainConfig +from scribe import compute_sample_errors, compute_aggregate_metrics, DomainConfig domain = DomainConfig.legal() @@ -43,7 +43,7 @@ for dataset, data in metrics['by_dataset'].items(): ### Error analysis (contributions + frequent errors) ```python -from dicterrors import ( +from scribe import ( aggregate_error_details, compute_error_summary, format_contribution_table, @@ -69,33 +69,37 @@ print(f"Accuracy: {summary['total_correct_pct']:.1f}%") # Formatted tables for display contrib_rows = format_contribution_table(summary["contributions"], domain) -sub_rows = format_frequent_errors_table(summary["frequent_substitutions"], "substitution", top_n=10) -del_rows = format_frequent_errors_table(summary["frequent_deletions"], "deletion", top_n=10) -ins_rows = format_frequent_errors_table(summary["frequent_insertions"], "insertion", top_n=10) +sub_rows = format_frequent_errors_table(summary["frequent_substitutions"], "substitution", top_n=10) +del_rows = format_frequent_errors_table(summary["frequent_deletions"], "deletion", top_n=10) +ins_rows = format_frequent_errors_table(summary["frequent_insertions"], "insertion", top_n=10) +merge_rows = format_frequent_errors_table(summary["frequent_sandhi_merges"], "sandhi_merge", top_n=10) +split_rows = format_frequent_errors_table(summary["frequent_sandhi_splits"], "sandhi_split", top_n=10) ``` +Sandhi merges / splits surface only when `use_sandhi=True` (the default) +and the language has agglutinative compounds. For non-agglutinative +languages (English, Hindi, etc.) the two tables will be empty. + ## CLI (`batch_evaluate.py`) ```bash -cd examples/ - -# Default run -uv run batch_evaluate.py +# Default run (uses the bundled examples/predictions.jsonl sample) +uv run examples/batch_evaluate.py # Custom input/output -uv run batch_evaluate.py \ +uv run examples/batch_evaluate.py \ --input ./my-data/predictions.jsonl \ --output-dir ./results \ --ref-field reference \ --hyp-field hypothesis # With domain config file -uv run batch_evaluate.py \ +uv run examples/batch_evaluate.py \ --input data/predictions.jsonl \ --domain-config config/legal_terms.txt # With detailed error analysis and category breakdown chart -uv run batch_evaluate.py \ +uv run examples/batch_evaluate.py \ --input data/predictions.jsonl \ --analysis \ --top-n 15 \ @@ -106,8 +110,8 @@ uv run batch_evaluate.py \ | Argument | Default | Description | |---|---|---| -| `-i`, `--input` | `./dictation-eval/predictions.jsonl` | Input JSONL file | -| `-o`, `--output-dir` | `./dictation-eval` | Output directory | +| `-i`, `--input` | bundled `examples/predictions.jsonl` | Input JSONL file | +| `-o`, `--output-dir` | `examples/output/` | Output directory (defaults alongside the script) | | `--ref-field` | `transcript_cleaned` | Reference field name | | `--hyp-field` | `prediction` | Hypothesis field name | | `--dataset-field` | `source_dataset` | Dataset identifier field | @@ -124,7 +128,7 @@ Always produced: - `evaluation-detailed.jsonl` — per-sample breakdown (see below) With `--analysis`: -- `analysis_report.txt` — TER, accuracy, category breakdown table, top-N frequent substitutions/deletions/insertions +- `analysis_report.txt` — TER, accuracy, category breakdown table, top-N frequent substitutions / deletions / insertions / sandhi merges / sandhi splits (the last two are only populated for agglutinative languages with sandhi events detected) With `--analysis --chart`: - `category_breakdown.png` — 2-panel stacked bar chart: token matches per category (left panel) and each category's contribution to the overall TER (right panel) @@ -174,4 +178,4 @@ Each line in the detailed output contains: } ``` -All error rates use the combined denominator (total tokens across all categories). See [Normalized Error Rates](../README.md#normalized-error-rates) for details. +All error rates use the combined denominator (total tokens across all categories). See [Combined denominator](architecture.md#glossary) in the architecture glossary for details. diff --git a/docs/domain-configuration.md b/docs/domain-configuration.md index f117347..ee2bc4e 100644 --- a/docs/domain-configuration.md +++ b/docs/domain-configuration.md @@ -1,13 +1,13 @@ # Domain Configuration -DictErrors supports flexible domain-aware tokenization via the `DomainConfig` class. Domain entities are extracted before general tokenization to prevent incorrect splitting (e.g., `u/s` stays as one token) and are tracked separately in error metrics. +scribe-eval supports flexible domain-aware tokenization via the `DomainConfig` class. Domain entities are extracted before general tokenization to prevent incorrect splitting (e.g., `u/s` stays as one token) and are tracked separately in error metrics. ## Factory Methods (Bundled Domains) Three pre-configured domains are bundled with the package: ```python -from dicterrors import DomainConfig, text_error_rates +from scribe import DomainConfig, text_error_rates domain = DomainConfig.legal() # Indian legal terminology domain = DomainConfig.medical() # Medical units and dosages @@ -20,7 +20,7 @@ report = text_error_rates(ref, hyp, domain) |---|---|---|---| | `DomainConfig.legal()` | LEGAL | LER | u/s, r/w, sec., art., v., vs., PW1/PW-1, CW1, Ext.A | | `DomainConfig.medical()` | MEDICAL | MER | mg, ml, cc, mcg, IU, 500mg, 10ml | -| `DomainConfig.technical()` | TECH | TER | API, SDK, CLI, JSON, HTTP, v1.0 | +| `DomainConfig.technical()` | TECH | TchER | API, SDK, CLI, JSON, HTTP, v1.0 | ## File-Based Configuration @@ -67,7 +67,7 @@ REGEX: Ext\.[-\s]*[A-Z]\d* # Matches Ext.A, Ext. B2 ### Bundled Config Files -Sample config files are included in `src/dicterrors/config/`: +Sample config files are included in `src/scribe/config/`: - `legal_terms.txt` — Indian legal terminology - `medical_terms.txt` — Medical units and dosages - `technical_terms.txt` — Technical abbreviations (case-sensitive) @@ -88,13 +88,13 @@ custom = DomainConfig.from_file( ## Inline Custom Domains ```python -from dicterrors import DomainConfig +from scribe import DomainConfig # List-based patterns (automatically escaped) custom = DomainConfig("custom", ["u/s", "r/w"], category="CUSTOM", label="CuER") # Regex pattern (used directly) -technical = DomainConfig("tech", r'API|SDK|CLI|v\d+\.\d+', category="TECH", label="TER") +technical = DomainConfig("tech", r'API|SDK|CLI|v\d+\.\d+', category="TECH", label="TchER") # Use in evaluation report = text_error_rates(ref, hyp, custom) @@ -111,7 +111,7 @@ report = text_error_rates(ref, hyp, None) ## File Location Conventions - **Project configs**: `config/` directory at the repository root -- **Personal configs**: `~/.config/dicterrors/` +- **Personal configs**: `~/.config/scribe-eval/` - **Dataset-specific configs**: Alongside the dataset in the data directory ``` diff --git a/docs/visualizer.md b/docs/visualizer.md index ca79e28..9b50d1b 100644 --- a/docs/visualizer.md +++ b/docs/visualizer.md @@ -1,13 +1,18 @@ # Interactive Visualizer -DictErrors includes a Streamlit-based web UI for exploring alignment and error metrics interactively. +scribe-eval ships a Streamlit-based web UI for exploring alignment and error metrics interactively. ## Launching +Install the `visualizer` extra and run the bundled command: + ```bash -streamlit run visualizer.py +pip install 'scribe-eval[visualizer]' +scribe-visualizer ``` +Any extra arguments are forwarded to `streamlit run` (e.g. `scribe-visualizer --server.port 8502`). + ## Tabs ### Single Sample Analysis @@ -24,7 +29,7 @@ Enter reference and hypothesis text directly. The view updates automatically on - **Error Analysis** expander: - Category contributions table (Ref Tokens, Exact Match, Accuracy, Sub, Del, Ins, Error Rate, Impact on Total) - Category breakdown chart (stacked bar: token matches left, TER contribution right) -- **Frequent Errors** expander: top-N substitutions, deletions, insertions in sub-tabs +- **Frequent Errors** expander: top-N substitutions, deletions, insertions, sandhi merges, sandhi splits across five sub-tabs (the two sandhi tabs are populated only when sandhi detection is enabled and the language has agglutinative compounds) ### Batch Dataset Analysis diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..e9e873d --- /dev/null +++ b/examples/README.md @@ -0,0 +1,43 @@ +# scribe-eval examples + +Each script is runnable with `uv run examples/