From 58ffdd8e5a6605e2ff669372bbcdb8e0bbc03c60 Mon Sep 17 00:00:00 2001 From: Raj Kumar Vishwakarma Date: Sat, 22 Aug 2026 19:18:52 +0530 Subject: [PATCH] Release 1.5.0 Cut the accumulated Unreleased work as 1.5.0 rather than a patch: the release adds two environment variables (TRAVSR_EMBED_ENGINE, TRAVSR_EMBED_TOKEN_BUDGET) and the macos_engine model.toml field, and changes the macOS default engine, which is minor-bump surface under the semver policy this changelog states. Changelog: document the two PRs that landed without entries, #18 (free the old HNSW before reload, cap ORT thread pools, mmap the serving index, chunk the pending query) and #22 (budget inference tokens across workers instead of per worker). README: the Apple Silicon guidance still claimed CoreML runs by default, which the 1.5.0 engine flip reverses. Rewrite that row and the asset intro, and document engine selection on macOS plus the environment variables, none of which were described anywhere. --- CHANGELOG.md | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++- Cargo.lock | 2 +- Cargo.toml | 2 +- README.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++---- 4 files changed, 100 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0978e5..ad2d3ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,11 @@ All notable changes to `travsr-embed` are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [1.5.0] - 2026-08-22 + +Hardening and memory release. Reverses 1.4.0's macOS engine choice on measured +evidence, makes parallel reindex cheap in memory rather than expensive, and +clears the crash residue that could make a killed reindex poison every later run. ### Added - **`TRAVSR_EMBED_ENGINE` kill-switch.** Setting it to `tract` drops every ORT @@ -24,6 +28,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 were already present, so a hand-set value reverts to `auto` on the next `embed init`. Use the env kill-switch for an override that has to survive a reinstall. (Tracked for a CLI-side fix; the env switch is unaffected.) +- **`TRAVSR_EMBED_TOKEN_BUDGET`** overrides the derived per-worker padded-token + budget for tuning. The value is clamped up to one full sequence so a batch can + always hold a single item. ### Changed - **macOS now prefers tract over CoreML by default** for families tract can run @@ -36,8 +43,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 macOS, so Linux/CUDA keeps accelerated-first ordering, and `resolve()` is split into a testable `resolve_with(is_macos)` so the ordering is deterministic on any CI host. +- **Inference tokens are budgeted across all workers, not per worker.** The + padded-token budget bounds the largest activation tensor **one** worker + allocates, and every worker runs one batch at a time, so peak sidecar memory + scaled with `workers x budget`. A fixed per-worker 4,096 therefore made memory + grow linearly with `-j`, which is what made parallelism expensive: on a + 1,453-node repo, 8 workers peaked at 634 MB against 345 MB for the 2-worker + default, for only a 1.7x speedup. A total budget of 8,192 is now divided by + the requested workers, floored at 1,024 and capped at the historical 4,096. + Memory stays roughly flat as workers scale and throughput improves rather than + degrades, because large batches mostly buy `BatchLongest` padding waste: + 8 workers measured 45.4s / 316 MB against 48.1s / 634 MB at the old fixed + budget. One and two workers keep exactly 4,096, so the configurations where + the old default was already right do not regress. Single-submitter backends + (GPU/ORT) run one inference loop whatever `-j` says, so they keep the whole + budget. Batch composition never changes the vectors (attention masking makes + padding inert, verified bit-identical across budgets on texts spanning 12 to + 480 characters), so this is purely a compute/memory trade-off, not a quality + one. +- **The serving path memory-maps the HNSW index** (usearch `view`) on Linux and + macOS instead of copying it into RAM, so the OS pages it in on demand and can + evict under pressure. A viewed index is immutable, so the lazy-embed add + becomes a no-op (the vector still persists to `embed.db` and enters the index + on the next reindex) and reloads re-view. Windows keeps the load fallback, + where a live file mapping takes a sharing lock that would break the reindex + sidecar's save. +- **Both reindex paths stop materialising the whole pending corpus.** A `COUNT` + query provides totals, then 50k rows are fetched, embedded and committed per + pass. Committed chunks drop out of the `NOT EXISTS` filter, so the loop needs + no `OFFSET` and always terminates. ### Fixed +- **A KNN index reload transiently doubled index memory** (travsr#736 RCA). The + reload built the replacement index while the previous one was still alive, a + transient 2x of the full index size at exactly the moment a just-finished + reindex had already elevated memory. The old graph is now freed before the + updated file is loaded, for RAM-copy handles where that 2x is the real + concern. +- **ONNX Runtime sized its thread pools from the host, ignoring cgroup limits** + (travsr#736 RCA). Left unset, ORT probes the host's physical core count, so a + container limited to 2 CPUs on a 64-core host got roughly 64 spinning threads + and permanent throttling. Intra-op threads are now capped at + `available_parallelism()` (cgroup quota-aware on Linux) and inter-op at 1. - **Crash residue from a killed reindex could crash or stall the serving sidecar (travsr#735 follow-up).** Four hardening changes: @@ -240,6 +287,8 @@ Initial release of the `travsr-embed` sidecar (RFC-018). ### Changed - Relicensed from MIT to Apache-2.0. +[1.5.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.4.0...v1.5.0 +[1.4.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.3.0...v1.4.0 [1.3.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.2.0...v1.3.0 [1.2.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.1.0...v1.2.0 [1.1.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.0.0...v1.1.0 diff --git a/Cargo.lock b/Cargo.lock index d7bdc59..8baca88 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2256,7 +2256,7 @@ dependencies = [ [[package]] name = "travsr-embed" -version = "1.4.0" +version = "1.5.0" dependencies = [ "anyhow", "dirs", diff --git a/Cargo.toml b/Cargo.toml index 569c9e5..0ca0548 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "travsr-embed" -version = "1.4.0" +version = "1.5.0" edition = "2021" description = "Travsr embed sidecar: BGE family ONNX fp32, runtime model selection via --model-id" license = "Apache-2.0" diff --git a/README.md b/README.md index dd57ed7..489c63f 100644 --- a/README.md +++ b/README.md @@ -11,14 +11,19 @@ users never build it directly. ## Which release asset do I want? -The default asset for your platform always works. GPU assets are opt-in **except -on Apple Silicon**, where acceleration is free (statically linked, nothing to -install). Intel Macs are CPU-only: ONNX Runtime publishes no prebuilt for +The default asset for your platform always works, and GPU assets are opt-in. +Intel Macs are CPU-only: ONNX Runtime publishes no prebuilt for `x86_64-apple-darwin`, so no ORT-based engine can be built for them at all. +Apple Silicon is the exception in both directions. CoreML is compiled into the +default asset (statically linked, nothing to install), but as of 1.5.0 it is no +longer what runs by default: for model families `tract` can run, tract is both +faster and far cheaper in memory than CoreML on this hardware, so it is +preferred. See [Choosing an engine on macOS](#choosing-an-engine-on-macos). + | Your machine | Asset | Accelerator | Host prerequisite | | --- | --- | --- | --- | -| macOS (Apple Silicon) | default | CoreML (ANE + GPU) | none | +| macOS (Apple Silicon) | default | none by default (tract, CPU); CoreML for families tract cannot run, or opt in per model | none | | macOS (Intel) | default | none (tract, CPU) | none | | Linux x86_64 + NVIDIA GPU | `…-x86_64-unknown-linux-gnu-cuda` | CUDA | CUDA runtime + cuDNN, glibc 2.38+, and a Haswell-or-newer CPU (`x86-64-v3`) | | Windows x86_64 + any GPU | `…-x86_64-pc-windows-msvc-directml.exe` | DirectML (Intel, AMD **or** NVIDIA) | a DirectX 12 GPU | @@ -121,6 +126,45 @@ host (notably: no ORT feature builds on a Windows *GNU* toolchain). Backend selection at runtime is capability-based: each registered backend declares which model families it can run, and the sidecar picks the first backend, in preference order, that can serve the model actually being loaded. +Accelerated engines come first, except on macOS (see below). + +### Choosing an engine on macOS + +Since 1.5.0, macOS prefers `tract` over CoreML for families tract can run. +Measured on Apple Silicon with the shipped bge-small model over an identical +30k-document corpus, release builds: tract ran about 2x the throughput at every +point and peaked at 743 MB RSS, finishing in 8m35s, where CoreML peaked near +4.0 GB and never finished the corpus. CoreML loses because ORT fragments the +BERT graph into roughly 97 CoreML/CPU partitions, paying a copy at every seam, +plus a dynamic-shape recompile tax. The flip is gated to macOS, so Linux/CUDA +and Windows/DirectML keep accelerated-first ordering. + +CoreML is still reached on macOS for families `tract` cannot run (ModernBERT, +nomic-bert), and can be opted back into per model with `macos_engine` in that +model's `model.toml`: + +| `macos_engine` | Effect on macOS | +| --- | --- | +| `auto` (default) | Prefer `tract` for families it can run; fall through to ORT/CoreML otherwise. | +| `tract` | Force `tract`, dropping every ORT engine. A loud error for a family tract cannot run, rather than a silent fallback. | +| `ort` | Restore accelerated-first ordering. Set this only for a model benchmarked faster on CoreML. | + +It defaults to `auto`, so existing `model.toml` files need no migration, and it +has no effect off macOS. + +**`macos_engine` does not survive a reinstall.** `travsr embed init` rewrites +`model.toml` from a closed field set that does not yet include it, and does so +even when the model files were already present, so a hand-set value reverts to +`auto` on the next `embed init`. For an override that has to persist, use the +`TRAVSR_EMBED_ENGINE` environment variable instead. + +### Environment variables + +| Variable | Effect | +| --- | --- | +| `TRAVSR_EMBED_ENGINE` | `tract` drops every ORT engine (both accelerated and ORT CPU) from the cascade, forcing the pure-Rust CPU engine regardless of what is compiled in or what the catalog says. `auto`, or unset, keeps the normal cascade. The switch can only ever remove ORT, so an unrecognised value warns and changes nothing. Also reflected in `--capabilities`, so the handshake never advertises acceleration that has been switched off. | +| `TRAVSR_EMBED_TOKEN_BUDGET` | Overrides the derived per-worker padded-token budget, for tuning. By default a total budget of 8,192 is divided across the requested `--parallel` workers, floored at 1,024 and capped at 4,096, which keeps peak memory roughly flat as workers scale. Single-submitter backends (GPU/ORT) keep the whole budget. | +| `ORT_DYLIB_PATH` | Where the `-directml` (and any `ort-dynamic`) build loads `onnxruntime.dll` from, when not beside the executable. | ### Reporting what a build can do