Prefer tract over CoreML on macOS by default (+ engine kill-switch and per-model override)

[Abhishek5517]

What

Makes tract the default embedding engine on macOS and adds two overrides, so CoreML stops being picked for models where it loses. Fixes the selection half of #19.

Why

On macOS the accelerated ORT path is CoreML. Measured on Apple Silicon with the shipped bge-small model over an identical 30k-document corpus (all release builds):

	tract	CoreML
Throughput (same doc index)	~2x faster at every point	baseline
Peak RSS	743 MB	~4.0 GB
Finished the corpus	yes, 8m35s	no

CoreML loses because ORT fragments the BERT graph into ~97 CoreML/CPU partitions (a copy at every seam) and pays a dynamic-shape recompile tax. The resolver ranked CoreML (pref 100) above tract (pref 50), so it won whenever it warmed up.

Note: an earlier autorelease-leak hypothesis turned out to be a red herring. That ORT bug (onnxruntime#14455) is already fixed upstream in model.mm, and an A/B soak confirmed a Rust-side autorelease wrap changed nothing, so it is not included here.

Change

Layered engine resolution, most explicit first:

1. TRAVSR_EMBED_ENGINE env kill-switch: tract drops every ORT factory; auto/unset keeps the cascade.
2. model.toml macos_engine: auto (default) | tract (per-model kill-switch) | ort (opt back into accelerated-first, for a model benchmarked faster on CoreML).
3. Capability filter (unchanged): tract only if it can run the family.
4. Default flip: on macOS tract outranks the accelerated engine for families tract can run.

The flip is gated to is_macos, so Linux/CUDA keeps accelerated-first. resolve() is split into a testable resolve_with(is_macos) so the macOS-only ordering is deterministic on any CI host. macos_engine defaults to auto, so existing model.toml files need no migration.

Verification

• Unit tests: default and --features ort-coreml suites green (flip, both catalog values, incapable-family error, precedence truth table). fmt + clippy -D warnings clean on both feature sets.

• E2E on a real ort-coreml release build (bge-small):

  scenario	selected backend
  default (auto)	tract
  macos_engine = "ort"	ort/CoreML
  TRAVSR_EMBED_ENGINE=tract over the ort catalog	tract (env wins)

Out of scope (tracked in #19)

• A CoreML partition-count gate is not implementable: the ort crate exposes no partition/node-assignment API. The better lead is CoreML EP tuning (with_static_input_shapes, with_model_format(MLProgram), with_model_cache_dir), which needs its own benchmark.
• The travsr CLI should populate macos_engine per model from the catalog as models are benchmarked. Until then everything defaults to auto (the flip), which is correct for the current catalog. tract cannot run ModernBERT/nomic-bert, where ORT is still used.

[anketpratapsingh]

Review: read in full, built and exercised on Windows

The resolve_with(is_macos) seam works exactly as advertised: all 10 new policy tests run deterministically on this non-macOS host. Full suite 68 passed / 0 failed in release mode, rustfmt and clippy -D warnings clean (toolchain 1.97 here, so the 1.98 lint motivation is taken from CI, same as your review of #22 noted in the other direction).

Design reads right to me: the layering (env kill-switch > catalog override > capability filter > macOS flip) puts the most explicit signal on top; filtering at registry() so capabilities() reports the restricted set keeps the CLI handshake honest; macos_engine = "tract" on an incapable family erroring loudly is the correct fail-closed choice, and the truth-table test pins the whole policy. The kill-switch only ever removing ORT (typos warn and change nothing) is a good property.

Needs action before merge

1. Rebase onto main — trial merge conflicts in src/model.rs. Main (via #22, merged 08:58Z today) already carries the blob_to_f32 as_chunks migration, with an explanatory comment; this branch carries a comment-less copy of the same change (83694cb), so both sides edited the same region and git cannot auto-merge. This is the third branch to carry that hunk — on rebase your copy should drop out, leaving the two real commits. (Verified by trial merge against origin/main; the branch is currently 7 commits behind.)

2. Missing CHANGELOG entry and upgrader note. This flips the macOS default that 1.4.0's own "Notes for upgraders" announced in the opposite direction. Every Apple Silicon install that reindexed under 1.4.0 CoreML will now resolve to tract, trip the backend-provenance warning, and should be told to travsr embed reindex --rebuild — the exact mirror of the note 1.4.0 shipped for the tract-to-CoreML flip. As it stands the diff touches no CHANGELOG at all.

Cross-repo sharp edge worth documenting (non-blocking here, fix belongs in travsr)

A hand-set macos_engine is silently wiped by the next travsr embed init. The CLI's write_model_descriptor (travsr repo, embed_catalog.rs) serializes a closed field set — dim/pooling/query_prefix/n_inputs/truncate_dim/family — and install_models calls it unconditionally at the end, even when every model file was "already present". So the documented operator flow ("set macos_engine = tract for a misbehaving model") survives only until anyone re-runs embed init, which reverts the model to auto with no warning. The env kill-switch is unaffected, and auto is the right default for today's catalog, so nothing breaks now — but the field's doc comment should say the CLI rewrites the file, and the travsr-side fix (carry macos_engine through the writer, or promote it to catalog data as your out-of-scope note suggests) deserves an issue so it does not get lost.

Observation, no action

Interplay with #22 (now on main): flipping macOS from CoreML to tract also moves those installs from the single-submitter regime (one inference loop, whole token budget) to multi-worker --parallel with the divided budget. That is strictly favourable — your measured 743 MB tract RSS predates #22's flattening — but it means macOS perf after rebase will look different (better) than this PR's own benchmark table, which is worth a one-line note when you re-verify post-rebase.

Verdict

The measurement is convincing, the mechanism explains it, the escape hatches are all present, and the tests pin the policy tightly. No objection to the substance — happy to see this merge once it is rebased and carries the upgrader note.

[anketpratapsingh]

Re-review after the rebase, focused on memory and performance impact

Verified the update first: rebased onto main (now MERGEABLE, redundant as_chunks commit gone), changelog added with an explicit Notes-for-upgraders section, and the macos_engine-wipe sharp edge from my first review is now documented both in the changelog and the field's doc comment, with the env switch named as the durable alternative. The only src delta since my first pass is that expanded doc comment — logic is byte-identical, so my earlier functional review stands. Re-ran locally on Windows against the rebased base: 78 passed / 0 failed in release mode, clippy -D warnings clean; all 8 CI checks green.

Memory: no new risk, and a large improvement where it applies

• The change itself cannot leak or accumulate. MacosEngine is a Copy enum on the existing descriptor; the kill-switch is a retain() over a two-to-three element factory Vec at registry construction. No new threads, caches, statics, or retained buffers anywhere in the diff — nothing in the class of problem #735 was about.
• macOS steady state improves dramatically by this PR's own A/B: 743 MB peak vs ~4.0 GB on CoreML for the same corpus. And post-rebase it compounds with the token-budget change now in the base: tract is the multi-worker path, so the divided budget bounds activations as -j scales, where CoreML was single-submitter holding the whole budget. Real-world macOS peaks should land below the PR's own pre-rebase 743 MB figure.
• Off macOS: zero change by default. The flip is is_macos-gated and the env switch is opt-in; bert_with_confirmed_accelerator_chooses_accelerated_ort (run with is_macos=false) pins accelerated-first ordering structurally, and it passes.

Performance: improvement on macOS, unmeasurable cost elsewhere, one bounded regression window

• macOS BERT-family: ~2x throughput per the 30k-doc measurement, and CoreML never finished that corpus at all. Non-BERT families keep ORT (tested), so nothing loses acceleration it needs.
• Runtime cost of the new code: one env read per registry() call and one enum compare in resolve — both run once per model load, never per query or per batch. Unmeasurable.
• One-time upgrade cost, correctly documented: the backend flip trips the provenance warning and recommends a full --rebuild on macOS installs that reindexed under CoreML. Worth knowing that the rebuild itself runs ~2x faster on tract than it would have on CoreML, so the toll is smaller than it sounds.
• The one residual perf gap is the case where CoreML genuinely wins for some future BERT-family model on some Mac: the default costs it acceleration, and the durable escape (macos_engine = "ort") is undermined by the CLI rewrite until the travsr-side fix lands. Note the asymmetry: the env switch can force tract but there is deliberately no env value to force ort. That is defensible (the switch is a kill-switch, not a selector), but if the CLI fix is slow to land, an ort env value would be the cheap interim escape. Per the current catalog no shipped model is in that set, so this is a forward-looking note, not a present regression.

Verdict

Approving. Memory: strictly improved on macOS, untouched elsewhere, nothing that can grow or leak. Performance: improved where it changes anything, with the single documented one-time rebuild cost at upgrade.