Release 1.5.0

CHANGELOG.md

@@ -5,7 +5,11 @@ All notable changes to `travsr-embed` are documented in this file.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [1.5.0] - 2026-08-22
+
+Hardening and memory release. Reverses 1.4.0's macOS engine choice on measured
+evidence, makes parallel reindex cheap in memory rather than expensive, and
+clears the crash residue that could make a killed reindex poison every later run.
 
 ### Added
 - **`TRAVSR_EMBED_ENGINE` kill-switch.** Setting it to `tract` drops every ORT
@@ -24,6 +28,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   were already present, so a hand-set value reverts to `auto` on the next
   `embed init`. Use the env kill-switch for an override that has to survive a
   reinstall. (Tracked for a CLI-side fix; the env switch is unaffected.)
+- **`TRAVSR_EMBED_TOKEN_BUDGET`** overrides the derived per-worker padded-token
+  budget for tuning. The value is clamped up to one full sequence so a batch can
+  always hold a single item.
 
 ### Changed
 - **macOS now prefers tract over CoreML by default** for families tract can run
@@ -36,8 +43,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   macOS, so Linux/CUDA keeps accelerated-first ordering, and `resolve()` is
   split into a testable `resolve_with(is_macos)` so the ordering is
   deterministic on any CI host.
+- **Inference tokens are budgeted across all workers, not per worker.** The
+  padded-token budget bounds the largest activation tensor **one** worker
+  allocates, and every worker runs one batch at a time, so peak sidecar memory
+  scaled with `workers x budget`. A fixed per-worker 4,096 therefore made memory
+  grow linearly with `-j`, which is what made parallelism expensive: on a
+  1,453-node repo, 8 workers peaked at 634 MB against 345 MB for the 2-worker
+  default, for only a 1.7x speedup. A total budget of 8,192 is now divided by
+  the requested workers, floored at 1,024 and capped at the historical 4,096.
+  Memory stays roughly flat as workers scale and throughput improves rather than
+  degrades, because large batches mostly buy `BatchLongest` padding waste:
+  8 workers measured 45.4s / 316 MB against 48.1s / 634 MB at the old fixed
+  budget. One and two workers keep exactly 4,096, so the configurations where
+  the old default was already right do not regress. Single-submitter backends
+  (GPU/ORT) run one inference loop whatever `-j` says, so they keep the whole
+  budget. Batch composition never changes the vectors (attention masking makes
+  padding inert, verified bit-identical across budgets on texts spanning 12 to
+  480 characters), so this is purely a compute/memory trade-off, not a quality
+  one.
+- **The serving path memory-maps the HNSW index** (usearch `view`) on Linux and
+  macOS instead of copying it into RAM, so the OS pages it in on demand and can
+  evict under pressure. A viewed index is immutable, so the lazy-embed add
+  becomes a no-op (the vector still persists to `embed.db` and enters the index
+  on the next reindex) and reloads re-view. Windows keeps the load fallback,
+  where a live file mapping takes a sharing lock that would break the reindex
+  sidecar's save.
+- **Both reindex paths stop materialising the whole pending corpus.** A `COUNT`
+  query provides totals, then 50k rows are fetched, embedded and committed per
+  pass. Committed chunks drop out of the `NOT EXISTS` filter, so the loop needs
+  no `OFFSET` and always terminates.
 
 ### Fixed
+- **A KNN index reload transiently doubled index memory** (travsr#736 RCA). The
+  reload built the replacement index while the previous one was still alive, a
+  transient 2x of the full index size at exactly the moment a just-finished
+  reindex had already elevated memory. The old graph is now freed before the
+  updated file is loaded, for RAM-copy handles where that 2x is the real
+  concern.
+- **ONNX Runtime sized its thread pools from the host, ignoring cgroup limits**
+  (travsr#736 RCA). Left unset, ORT probes the host's physical core count, so a
+  container limited to 2 CPUs on a 64-core host got roughly 64 spinning threads
+  and permanent throttling. Intra-op threads are now capped at
+  `available_parallelism()` (cgroup quota-aware on Linux) and inter-op at 1.
 - **Crash residue from a killed reindex could crash or stall the serving
   sidecar (travsr#735 follow-up).** Four hardening changes:
 
@@ -240,6 +287,8 @@ Initial release of the `travsr-embed` sidecar (RFC-018).
 ### Changed
 - Relicensed from MIT to Apache-2.0.
 
+[1.5.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.4.0...v1.5.0
+[1.4.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.3.0...v1.4.0
 [1.3.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.2.0...v1.3.0
 [1.2.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.1.0...v1.2.0
 [1.1.0]: https://github.com/Travsr-com/travsr-embed/compare/v1.0.0...v1.1.0

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "travsr-embed"
-version = "1.4.0"
+version = "1.5.0"
 edition = "2021"
 description = "Travsr embed sidecar: BGE family ONNX fp32, runtime model selection via --model-id"
 license = "Apache-2.0"

README.md

@@ -11,14 +11,19 @@ users never build it directly.
 
 ## Which release asset do I want?
 
-The default asset for your platform always works. GPU assets are opt-in **except
-on Apple Silicon**, where acceleration is free (statically linked, nothing to
-install). Intel Macs are CPU-only: ONNX Runtime publishes no prebuilt for
+The default asset for your platform always works, and GPU assets are opt-in.
+Intel Macs are CPU-only: ONNX Runtime publishes no prebuilt for
 `x86_64-apple-darwin`, so no ORT-based engine can be built for them at all.
 
+Apple Silicon is the exception in both directions. CoreML is compiled into the
+default asset (statically linked, nothing to install), but as of 1.5.0 it is no
+longer what runs by default: for model families `tract` can run, tract is both
+faster and far cheaper in memory than CoreML on this hardware, so it is
+preferred. See [Choosing an engine on macOS](#choosing-an-engine-on-macos).
+
 | Your machine | Asset | Accelerator | Host prerequisite |
 | --- | --- | --- | --- |
-| macOS (Apple Silicon) | default | CoreML (ANE + GPU) | none |
+| macOS (Apple Silicon) | default | none by default (tract, CPU); CoreML for families tract cannot run, or opt in per model | none |
 | macOS (Intel) | default | none (tract, CPU) | none |
 | Linux x86_64 + NVIDIA GPU | `…-x86_64-unknown-linux-gnu-cuda` | CUDA | CUDA runtime + cuDNN, glibc 2.38+, and a Haswell-or-newer CPU (`x86-64-v3`) |
 | Windows x86_64 + any GPU | `…-x86_64-pc-windows-msvc-directml.exe` | DirectML (Intel, AMD **or** NVIDIA) | a DirectX 12 GPU |
@@ -121,6 +126,45 @@ host (notably: no ORT feature builds on a Windows *GNU* toolchain).
 Backend selection at runtime is capability-based: each registered backend
 declares which model families it can run, and the sidecar picks the first
 backend, in preference order, that can serve the model actually being loaded.
+Accelerated engines come first, except on macOS (see below).
+
+### Choosing an engine on macOS
+
+Since 1.5.0, macOS prefers `tract` over CoreML for families tract can run.
+Measured on Apple Silicon with the shipped bge-small model over an identical
+30k-document corpus, release builds: tract ran about 2x the throughput at every
+point and peaked at 743 MB RSS, finishing in 8m35s, where CoreML peaked near
+4.0 GB and never finished the corpus. CoreML loses because ORT fragments the
+BERT graph into roughly 97 CoreML/CPU partitions, paying a copy at every seam,
+plus a dynamic-shape recompile tax. The flip is gated to macOS, so Linux/CUDA
+and Windows/DirectML keep accelerated-first ordering.
+
+CoreML is still reached on macOS for families `tract` cannot run (ModernBERT,
+nomic-bert), and can be opted back into per model with `macos_engine` in that
+model's `model.toml`:
+
+| `macos_engine` | Effect on macOS |
+| --- | --- |
+| `auto` (default) | Prefer `tract` for families it can run; fall through to ORT/CoreML otherwise. |
+| `tract` | Force `tract`, dropping every ORT engine. A loud error for a family tract cannot run, rather than a silent fallback. |
+| `ort` | Restore accelerated-first ordering. Set this only for a model benchmarked faster on CoreML. |
+
+It defaults to `auto`, so existing `model.toml` files need no migration, and it
+has no effect off macOS.
+
+**`macos_engine` does not survive a reinstall.** `travsr embed init` rewrites
+`model.toml` from a closed field set that does not yet include it, and does so
+even when the model files were already present, so a hand-set value reverts to
+`auto` on the next `embed init`. For an override that has to persist, use the
+`TRAVSR_EMBED_ENGINE` environment variable instead.
+
+### Environment variables
+
+| Variable | Effect |
+| --- | --- |
+| `TRAVSR_EMBED_ENGINE` | `tract` drops every ORT engine (both accelerated and ORT CPU) from the cascade, forcing the pure-Rust CPU engine regardless of what is compiled in or what the catalog says. `auto`, or unset, keeps the normal cascade. The switch can only ever remove ORT, so an unrecognised value warns and changes nothing. Also reflected in `--capabilities`, so the handshake never advertises acceleration that has been switched off. |
+| `TRAVSR_EMBED_TOKEN_BUDGET` | Overrides the derived per-worker padded-token budget, for tuning. By default a total budget of 8,192 is divided across the requested `--parallel` workers, floored at 1,024 and capped at 4,096, which keeps peak memory roughly flat as workers scale. Single-submitter backends (GPU/ORT) keep the whole budget. |
+| `ORT_DYLIB_PATH` | Where the `-directml` (and any `ort-dynamic`) build loads `onnxruntime.dll` from, when not beside the executable. |
 
 ### Reporting what a build can do