refactor: Preparing repo for the release as scribe-eval - #22
Closed
kavyamanohar wants to merge 28 commits into
Closed
refactor: Preparing repo for the release as scribe-eval#22kavyamanohar wants to merge 28 commits into
kavyamanohar wants to merge 28 commits into
Conversation
Establishes the library source layout at src/scribe/, with public API exports in __init__.py and bundled domain configs under src/scribe/config/. Examples, documentation, and the Streamlit visualizer are included. The library is distributed on PyPI as scribe-eval and imported as scribe. Subsequent commits add public-release metadata, license, tests, and CI.
- Add description, Apache-2.0 license, author, keywords, classifiers, project URLs (homepage/repository/issues). - Require Python >= 3.10 (previously 3.11). - Split dependencies: - core: jiwer, levenshtein, tabulate - [charts]: matplotlib (for headless chart generation) - [visualizer]: streamlit, pandas, matplotlib (for the Streamlit UI) - [dev]: pytest, pytest-cov, ruff, mypy, build, twine - Set ruff target-version to py310 to match requires-python. Core install no longer pulls in streamlit, pandas, or matplotlib (~100 MB saved for users who only need the error-analysis API).
- LICENSE: full Apache-2.0 text with copyright "2026 Adalat AI". - src/scribe/__init__.py: module docstring updated to reflect the scribe package identity (no longer "DictErrors") and exposes __version__ read dynamically from installed package metadata via importlib.metadata, so the version lives in a single place (pyproject.toml).
When streamlit is not installed (core-only install of scribe-eval),
running visualizer.py would raise a cryptic ModuleNotFoundError.
The guard intercepts the ImportError and exits with a clear message
pointing the user at:
pip install 'scribe-eval[visualizer]'
Move the Streamlit UI into the scribe package as a subpackage so it
ships inside the installed wheel. Users now run the UI with a single
console command after installing the visualizer extra:
pip install 'scribe-eval[visualizer]'
scribe-visualizer
The command forwards additional arguments to streamlit, e.g.
`scribe-visualizer --server.port 8502`.
- visualizer.py -> src/scribe/visualizer/app.py (git rename)
- Add src/scribe/visualizer/__init__.py (subpackage marker)
- Add src/scribe/visualizer/__main__.py with main() entry point that
invokes streamlit.web.cli on the bundled app.py and preserves the
ImportError guard (hints at the [visualizer] extra).
- pyproject.toml: register `scribe-visualizer` console_script.
- README.md, docs/visualizer.md: update launch instructions.
- Delete main.py (empty uv-init scaffold, nothing imports it).
- Move the bundled sample from examples/dictation-eval/predictions.jsonl
to examples/predictions.jsonl (flat). This sits alongside the example
scripts and is the one file users need to run batch_evaluate.py out of
the box after cloning.
- Expand .gitignore:
- tool caches: .ruff_cache/, .mypy_cache/
- internal-only docs: docs/internal/
- benchmark / evaluation outputs dropped into examples/ subdirectories
(example scripts at examples/*.py remain tracked), user-specific
domain overrides under src/scribe/config/, and any ad-hoc
examples/test_*.py scripts.
The default --input and --output-dir paths now point to ./predictions.jsonl
and ./output/, matching the file shipped at examples/predictions.jsonl.
Anyone who clones the repo can now run, from examples/:
uv run batch_evaluate.py
with no flags and get a working evaluation against the bundled sample.
The output directory ./output/ is automatically excluded from version
control by the examples/*/ ignore rule.
Set up the foundations of a real pytest suite: - tests/conftest.py: shared fixtures for the bundled domain configs. - tests/test_imports.py: pins the public API surface. Every symbol re-exported from scribe.__init__ is enumerated with its expected kind, so downstream-breaking renames or drops fail loudly. Also asserts that __version__ is populated. - tests/test_paper_cases.py: golden cases for the headline empirical claims of the SCRIBE Interspeech submission — the Fig. 2 Malayalam sandhi merge-and-split (ERlex = 0% with sandhi, 100% without), legal domain shielding for u/s / PW1 / Ext.A, and the combined-denominator formula from §3.3. - pyproject.toml: pytest config (testpaths = tests, --strict-markers). Removes the legacy `/test*.py` ignore rule. The legacy root-level test_*.py scripts are kept out of the new tree; their useful logic will be mined into the per-module suite in the next commit.
Adds unit tests across tokenize, align, measure, normalize, measure_batch, and reporting. Core modules (align, normalize, tokenize, domain_config, measure_batch) are at 77-98% line coverage; analysis and reporting layers remain candidates for later expansion. pyproject.toml: omit visualizer/ and charts.py from coverage measurement (UI and optional extras are not unit-testable without browser automation).
Adds tests/test_domain_config.py with 25 tests across 7 classes covering file parsing, metadata, regex / literal patterns, parameter overrides, error handling, integration with the tokenizer, and the bundled-config factory methods (DomainConfig.legal / .medical / .technical).
Anchor the README to the Interspeech 2026 SCRIBE paper, document pip install scribe-eval with the [visualizer] and [charts] extras, add PyPI / Python / Apache-2.0 badges, separate core dependencies from optional extras, and add Citation and License sections. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…category edge Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- custom_domain_file.py: actually load DomainConfig.from_file() against a real config file (examples/sample_legal.txt) instead of falling back to the factory. - sample_legal.txt: small but realistic legal config with literals and regex patterns; demonstrates the file format end-to-end. - text_alignment.py: drop ~40 lines of commented-out alternate examples; keep the Malayalam Sandhi-aware default and pass-through CLI args. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ish) examples/text_alignment.py now runs all four demo pairs by default (sandhi merge/split, token insertion + numeral truncation, spelled-out Kannada numeral, English reorder + punctuation). The same pairs are parametrized in tests/test_align.py to guard against regressions on real-world Indic and English inputs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…with the analysis CLI - examples/README.md: index of the four runnable scripts and bundled data, with one-line descriptions and the public API each exercises. - examples/error_report.py: rewrite as a thin single-sample analog of batch_evaluate.py --analysis. Uses compute_error_summary + format_contribution_table so the per-category breakdown matches the batch CLI exactly. Reuses print_alignment from text_alignment.py for the token-by-token alignment view. - README.md: add a brief Examples section pointing at examples/README.md and simplify the batch_evaluate.py invocation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both --input and --output-dir now resolve relative to the script's
directory when not overridden, so
uv run examples/batch_evaluate.py
from the repo root works the same as `cd examples && uv run ...`.
Outputs land in examples/output/ (already gitignored) instead of
spilling a ./output directory wherever the user happened to be.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A new docs/architecture.md gives a single-page tour for anyone modifying the library: - Pipeline diagram (tokenize → align → measure → aggregate → report) with the payload at each arrow. - Module map covering every src/scribe/ module — what it owns and the key callables it exposes. - "Where to make a change" task → file pointers for common edits. - Tests-mirror-the-pipeline pointer. - Key design decisions: combined denominator, domain shielding, sandhi awareness, two error-rate views. - Glossary defining sandhi, sandhi correction, combined denominator, domain shielding, TER, error rate vs Impact on Total, and the alignment gap penalty. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rmula - TER expansion is now consistently "Token Error Rate" across the glossary, analysis.py, and example docstrings (matches the user-facing label already shown by the visualizer and charts). - The architecture glossary's Accuracy entry replaces the wrong "Accuracy = 1 - TER minus a sandhi correction" claim with the actual formula: total_correct / total_ref. Adds a note explaining why Accuracy and TER do not sum to 100% in general (insertions and sandhi hits). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Token Error Rate (overall) and the technical-domain short label both used "TER", colliding in mixed reports. The technical label is now TchER. LER (Legal) and MER (Medical) unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
….jsonl Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…port token_error_details now emits sandhi_merge / sandhi_split records (parsed from the MERGE: / SPLIT: alignment markers) instead of skipping them. analysis.py adds compute_frequent_sandhi_merges / _splits, included in compute_error_summary; reporting.format_frequent_errors_table accepts the two new pair-shaped error types; batch_evaluate.py prints and saves the two new tables alongside the existing substitution / deletion / insertion sections (no new CLI flag — piggybacks on --analysis). Counting is unchanged: token_error_rates still treats sandhi events as corrections (sandhi_hits++, no contribution to error rate). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s, TER New tests/test_analysis.py with 28 cases covering analysis.py end-to-end: sandhi merge/split record emission and ranking, frequent-error counting and top-N truncation, total error rate (zero / fractional / >1 with insertions), category contributions (counts, empty category, contribution sums to 100), error type distribution, and compute_error_summary key set. Examples use Hindi (आम/केला/सेब, मैंने...खाया) and Malayalam (ഇന്ന് അല്ലെങ്കിൽ ↔ ഇന്നല്ലെങ്കിൽ for sandhi). test_measure.py adds a regression pinning that token_error_rates output is unaffected by the new sandhi-record emission. test_imports.py tracks the two new public symbols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…amples Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repo-wide rename from dicterrors to scribe-eval (PyPI) / scribe (import)
ahead. 28 commits across rename, packaging, tests, docs, and one new feature.
Highlights:
visualizer alongside subs/dels/ins; token_error_rates output unchanged
domain-configuration, visualizer) with runnable per-stage examples
.gitignore hardening