diff --git a/CHANGELOG.md b/CHANGELOG.md index f0978e5..ad2d3ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,11 @@ All notable changes to `travsr-embed` are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [1.5.0] - 2026-08-22 + +Hardening and memory release. Reverses 1.4.0's macOS engine choice on measured +evidence, makes parallel reindex cheap in memory rather than expensive, and +clears the crash residue that could make a killed reindex poison every later run. ### Added - **`TRAVSR_EMBED_ENGINE` kill-switch.** Setting it to `tract` drops every ORT @@ -24,6 +28,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 were already present, so a hand-set value reverts to `auto` on the next `embed init`. Use the env kill-switch for an override that has to survive a reinstall. (Tracked for a CLI-side fix; the env switch is unaffected.) +- **`TRAVSR_EMBED_TOKEN_BUDGET`** overrides the derived per-worker padded-token + budget for tuning. The value is clamped up to one full sequence so a batch can + always hold a single item. ### Changed - **macOS now prefers tract over CoreML by default** for families tract can run @@ -36,8 +43,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 macOS, so Linux/CUDA keeps accelerated-first ordering, and `resolve()` is split into a testable `resolve_with(is_macos)` so the ordering is deterministic on any CI host. +- **Inference tokens are budgeted across all workers, not per worker.** The + padded-token budget bounds the largest activation tensor **one** worker + allocates, and every worker runs one batch at a time, so peak sidecar memory + scaled with `workers x budget`. A fixed per-worker 4,096 therefore made memory + grow linearly with `-j`, which is what made parallelism expensive: on a + 1,453-node repo, 8 workers peaked at 634 MB against 345 MB for the 2-worker + default, for only a 1.7x speedup. A total budget of 8,192 is now divided by + the requested workers, floored at 1,024 and capped at the historical 4,096. + Memory stays roughly flat as workers scale and throughput improves rather than + degrades, because large batches mostly buy `BatchLongest` padding waste: + 8 workers measured 45.4s / 316 MB against 48.1s / 634 MB at the old fixed + budget. One and two workers keep exactly 4,096, so the configurations where + the old default was already right do not regress. Single-submitter backends + (GPU/ORT) run one inference loop whatever `-j` says, so they keep the whole + budget. Batch composition never changes the vectors (attention masking makes + padding inert, verified bit-identical across budgets on texts spanning 12 to + 480 characters), so this is purely a compute/memory trade-off, not a quality + one. +- **The serving path memory-maps the HNSW index** (usearch `view`) on Linux and + macOS instead of copying it into RAM, so the OS pages it in on demand and can + evict under pressure. A viewed index is immutable, so the lazy-embed add + becomes a no-op (the vector still persists to `embed.db` and enters the index + on the next reindex) and reloads re-view. Windows keeps the load fallback, + where a live file mapping takes a sharing lock that would break the reindex + sidecar's save. +- **Both reindex paths stop materialising the whole pending corpus.** A `COUNT` + query provides totals, then 50k rows are fetched, embedded and committed per + pass. Committed chunks drop out of the `NOT EXISTS` filter, so the loop needs + no `OFFSET` and always terminates. ### Fixed +- **A KNN index reload transiently doubled index memory** (travsr#736 RCA). The + reload built the replacement index while the previous one was still alive, a + transient 2x of the full index size at exactly the moment a just-finished + reindex had already elevated memory. The old graph is now freed before the + updated file is loaded, for RAM-copy handles where that 2x is the real + concern. +- **ONNX Runtime sized its thread pools from the host, ignoring cgroup limits** + (travsr#736 RCA). Left unset, ORT probes the host's physical core count, so a + container limited to 2 CPUs on a 64-core host got roughly 64 spinning threads + and permanent throttling. Intra-op threads are now capped at + `available_parallelism()` (cgroup quota-aware on Linux) and inter-op at 1. - **Crash residue from a killed reindex could crash or stall the serving sidecar (travsr#735 follow-up).** Four hardening changes: @@ -240,6 +287,8 @@ Initial release of the `travsr-embed` sidecar (RFC-018). ### Changed - Relicensed from MIT to Apache-2.0. +[1.5.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.4.0...v1.5.0 +[1.4.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.3.0...v1.4.0 [1.3.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.2.0...v1.3.0 [1.2.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.1.0...v1.2.0 [1.1.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.0.0...v1.1.0 diff --git a/Cargo.lock b/Cargo.lock index d7bdc59..8baca88 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2256,7 +2256,7 @@ dependencies = [ [[package]] name = "travsr-embed" -version = "1.4.0" +version = "1.5.0" dependencies = [ "anyhow", "dirs", diff --git a/Cargo.toml b/Cargo.toml index 569c9e5..0ca0548 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "travsr-embed" -version = "1.4.0" +version = "1.5.0" edition = "2021" description = "Travsr embed sidecar: BGE family ONNX fp32, runtime model selection via --model-id" license = "Apache-2.0" diff --git a/README.md b/README.md index dd57ed7..489c63f 100644 --- a/README.md +++ b/README.md @@ -11,14 +11,19 @@ users never build it directly. ## Which release asset do I want? -The default asset for your platform always works. GPU assets are opt-in **except -on Apple Silicon**, where acceleration is free (statically linked, nothing to -install). Intel Macs are CPU-only: ONNX Runtime publishes no prebuilt for +The default asset for your platform always works, and GPU assets are opt-in. +Intel Macs are CPU-only: ONNX Runtime publishes no prebuilt for `x86_64-apple-darwin`, so no ORT-based engine can be built for them at all. +Apple Silicon is the exception in both directions. CoreML is compiled into the +default asset (statically linked, nothing to install), but as of 1.5.0 it is no +longer what runs by default: for model families `tract` can run, tract is both +faster and far cheaper in memory than CoreML on this hardware, so it is +preferred. See [Choosing an engine on macOS](#choosing-an-engine-on-macos). + | Your machine | Asset | Accelerator | Host prerequisite | | --- | --- | --- | --- | -| macOS (Apple Silicon) | default | CoreML (ANE + GPU) | none | +| macOS (Apple Silicon) | default | none by default (tract, CPU); CoreML for families tract cannot run, or opt in per model | none | | macOS (Intel) | default | none (tract, CPU) | none | | Linux x86_64 + NVIDIA GPU | `…-x86_64-unknown-linux-gnu-cuda` | CUDA | CUDA runtime + cuDNN, glibc 2.38+, and a Haswell-or-newer CPU (`x86-64-v3`) | | Windows x86_64 + any GPU | `…-x86_64-pc-windows-msvc-directml.exe` | DirectML (Intel, AMD **or** NVIDIA) | a DirectX 12 GPU | @@ -121,6 +126,45 @@ host (notably: no ORT feature builds on a Windows *GNU* toolchain). Backend selection at runtime is capability-based: each registered backend declares which model families it can run, and the sidecar picks the first backend, in preference order, that can serve the model actually being loaded. +Accelerated engines come first, except on macOS (see below). + +### Choosing an engine on macOS + +Since 1.5.0, macOS prefers `tract` over CoreML for families tract can run. +Measured on Apple Silicon with the shipped bge-small model over an identical +30k-document corpus, release builds: tract ran about 2x the throughput at every +point and peaked at 743 MB RSS, finishing in 8m35s, where CoreML peaked near +4.0 GB and never finished the corpus. CoreML loses because ORT fragments the +BERT graph into roughly 97 CoreML/CPU partitions, paying a copy at every seam, +plus a dynamic-shape recompile tax. The flip is gated to macOS, so Linux/CUDA +and Windows/DirectML keep accelerated-first ordering. + +CoreML is still reached on macOS for families `tract` cannot run (ModernBERT, +nomic-bert), and can be opted back into per model with `macos_engine` in that +model's `model.toml`: + +| `macos_engine` | Effect on macOS | +| --- | --- | +| `auto` (default) | Prefer `tract` for families it can run; fall through to ORT/CoreML otherwise. | +| `tract` | Force `tract`, dropping every ORT engine. A loud error for a family tract cannot run, rather than a silent fallback. | +| `ort` | Restore accelerated-first ordering. Set this only for a model benchmarked faster on CoreML. | + +It defaults to `auto`, so existing `model.toml` files need no migration, and it +has no effect off macOS. + +**`macos_engine` does not survive a reinstall.** `travsr embed init` rewrites +`model.toml` from a closed field set that does not yet include it, and does so +even when the model files were already present, so a hand-set value reverts to +`auto` on the next `embed init`. For an override that has to persist, use the +`TRAVSR_EMBED_ENGINE` environment variable instead. + +### Environment variables + +| Variable | Effect | +| --- | --- | +| `TRAVSR_EMBED_ENGINE` | `tract` drops every ORT engine (both accelerated and ORT CPU) from the cascade, forcing the pure-Rust CPU engine regardless of what is compiled in or what the catalog says. `auto`, or unset, keeps the normal cascade. The switch can only ever remove ORT, so an unrecognised value warns and changes nothing. Also reflected in `--capabilities`, so the handshake never advertises acceleration that has been switched off. | +| `TRAVSR_EMBED_TOKEN_BUDGET` | Overrides the derived per-worker padded-token budget, for tuning. By default a total budget of 8,192 is divided across the requested `--parallel` workers, floored at 1,024 and capped at 4,096, which keeps peak memory roughly flat as workers scale. Single-submitter backends (GPU/ORT) keep the whole budget. | +| `ORT_DYLIB_PATH` | Where the `-directml` (and any `ort-dynamic`) build loads `onnxruntime.dll` from, when not beside the executable. | ### Reporting what a build can do