fix(models): verify checksums for every model file, including already-installed files

[kevincostner17]

Summary

Model checksum checks now cover every file of a model, and fd.models.pull checks files that are already installed before it reports success.

Before this change, a pin covered only the primary file. pull hashed only files[0], registry.verify checked only the primary file, and status() reported verified=True while a secondary file such as tokenizer.json went unchecked. When every file already existed and force=False, pull returned early without verifying anything.

• ModelConfig gains file_sha256, a tuple of (name, sha256) pairs. sha256 remains the pin for the primary file, so dataclasses.replace(cfg, sha256=...) keeps working.
• registry.pinned_checksums(cfg) merges both fields into one mapping per file. It raises ModelChecksumError when a pin names an unknown file or two pins for the same file conflict.
• Once a model has any pin, every file must be pinned. A file without a pin raises ModelChecksumError naming it, before anything is downloaded.
• verify() checks every file, so status() reports verified: True only when all files match.
• pull() checks each downloaded file against its own pin before it replaces the installed file. When the files are already installed, it calls verify(): a mismatch raises with a force=True hint and leaves the files in place, and the CLI exits 2.
• docs/semantic-models.md describes the per-file checks and the new pull behaviour.

Models without pins, which is every entry in today's registry, behave exactly as before. Requiring https:// for the download base URL is out of scope.

Follow-up (not in this PR): OnnxEncoder.model_sha256 still reads only cfg.sha256, so a model pinned only through file_sha256 reports "unverified" there.

Tests

• tests/test_models_download.py:
  • a mismatched secondary file is refused and not installed, with no .part files left
  • a model whose files all match its pins installs
  • a partially pinned model raises before any download (issue repro)
  • already-installed files with a corrupt primary raise and are left in place (issue repro)
  • already-installed files with a corrupt secondary raise
  • already-installed files that match their pins return without downloading
  • already-installed unpinned files behave as before
• tests/test_models_registry.py:
  • status() is false for a mismatched secondary file and true once it matches
  • verify() raises for a partially pinned model
  • pinned_checksums merging, unknown-file and conflicting-pin cases
  • every registry entry's pins are either empty or cover every file
• tests/test_cli_models.py: freshdata models pull exits 2 when installed files don't match their pins.

Verification

• ruff check .: passed; changed files are ruff format-clean
• mypy src/freshdata: no issues (202 files)
• pytest -m "not online and not large": Python 3.12: 5150 passed, 13 skipped; Python 3.9 / pandas 1.5: 5146 passed, 17 skipped
• Model checksum checks cover only the primary file, and pull skips them for existing files #346 repro against a local HTTP server: a pull with a pin only on the primary file is refused and tokenizer.json is not installed; a pull over a corrupt installed model.onnx raises ModelChecksumError instead of returning the path.

Closes #346

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 17f9865b-321e-4b79-b021-a7d32e26e908

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)