Benchmark and select a multilingual FastEmbed model for semantic search

[phernandez]

Summary

Benchmark and select a multilingual FastEmbed embedding model for Basic Memory semantic and hybrid search, including the Basic Memory Cloud path where FastEmbed generates vectors and PostgreSQL/pgvector stores and retrieves them.

This is deliberately separate from #1294. That issue is scoped to lexical full-text search (FTS) analysis across SQLite and PostgreSQL; changing the embedding model does not fix exact CJK keyword matching.

Current state

• Core defaults to BAAI/bge-small-en-v1.5, an English model.
• Cloud preloads that FastEmbed model into the application image.
• Cloud generates embeddings with FastEmbed and stores/searches vectors in PostgreSQL/pgvector, not sqlite-vec.
• Core already supports explicit embedding provider, model, dimensions, query/document prefixes, model identity, and embedding reindexing.
• fix(core): L2-normalize FastEmbed vectors to satisfy unit-vector contract #843 fixed provider-boundary L2 normalization for models that do not normalize their own output; [BUG] FastEmbed embeddings are not L2-normalized → semantic search silently degrades to FTS-only for non-bge models (e.g. multilingual-mpnet) #1023 documents the earlier failure mode and the required reindex after that fix.

Goal

Choose a multilingual model from evidence rather than switching defaults based on catalog descriptions alone, then define a safe local and Cloud migration plan.

The benchmark should compare the current English BGE baseline with viable FastEmbed multilingual candidates. Initial candidates to evaluate include, but are not limited to:

• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (384 dimensions)
• sentence-transformers/paraphrase-multilingual-mpnet-base-v2 (768 dimensions)
• intfloat/multilingual-e5-large (1024 dimensions; requires query/document prefixes)
• jinaai/jina-embeddings-v3 (1024 dimensions)

Do not preselect a winner. Candidates may be removed or added based on FastEmbed support, licensing, artifact stability, resource limits, and measured quality.

Evaluation requirements

Retrieval quality

Build a reproducible evaluation corpus and query set that covers:

• English baseline/regression
• Chinese, Japanese, and Korean
• additional representative scripts/languages such as Arabic, Russian, Spanish, and Thai
• mixed-language notes
• same-language retrieval
• cross-language retrieval where the query and relevant note use different languages
• short notes, long notes, and matches that cross semantic chunk boundaries
• conceptual/paraphrase queries, not only literal keyword overlap

Report at least recall@k and ranking quality (MRR or nDCG), plus empty-result and material false-positive rates. Keep the dataset and harness in the repository so future model changes can be compared against the same baseline.

Runtime and operational cost

Measure on supported deployment hardware:

• cold model load time
• warm query latency
• document embedding throughput
• peak and steady-state memory
• model artifact and container-image size
• vector dimensions and PostgreSQL storage growth
• pgvector index build/update time
• pgvector query latency and recall under the selected index parameters
• full-project and representative tenant reindex duration

Local sqlite-vec measurements may be useful for the Core default, but Cloud acceptance must use FastEmbed inference with PostgreSQL/pgvector storage.

Model contract

Verify for every candidate:

• FastEmbed support on Basic Memory's supported Python/platform matrix
• deterministic configured dimensions
• correct query/document prefix or input-role behavior
• provider-boundary normalization
• stable embedding identity and stale-vector detection
• cancellation and representative error propagation during batch reindex
• license and redistribution suitability for preloading in the Cloud image

Cloud rollout requirements

The selected model must have an explicit Cloud plan:

1. Configure the same provider, model, dimensions, and prefixes in API and worker processes.
2. Bake/cache the FastEmbed artifact in the Cloud image so workers do not download it at runtime.
3. Measure and provision worker memory and concurrency from benchmark results.
4. Deploy the new model identity without comparing query vectors against vectors from the old model.
5. Rebuild embeddings for every active tenant/project through a retry-safe fleet operation.
6. Define whether the first cutover accepts temporary vector incompleteness or requires blue/green vector indexes.
7. Monitor per-tenant readiness, failures, duration, pgvector size, and query latency before declaring the migration complete.

Changing dimensions requires compatible pgvector schema/index migration. Keeping 384 dimensions avoids that particular schema change but still requires a complete embedding rebuild because vectors from different models are not comparable.

Non-goals

• FTS tokenization or CJK lexical segmentation; tracked by [enhancement] CJK (Chinese/Japanese/Korean) full-text search broken — hardcoded unicode61 tokenizer doesn't segment CJK #1294
• pre-segmenting CJK text before passing it to an embedding model
• selecting a model without reproducible benchmark evidence
• silently mixing embeddings from different model identities

Acceptance criteria

• Reproducible multilingual retrieval benchmark committed
• Current BGE baseline and all retained candidates measured
• Quality, latency, memory, artifact-size, and PostgreSQL/pgvector results documented
• Recommended model and rejected alternatives explained from evidence
• Local default/configuration decision documented
• Cloud capacity, image, configuration, and cutover plan documented
• Fleet reindex and readiness verification plan documented
• Any implementation work is split into bounded follow-up PRs/issues as needed

[claude]

Claude finished @phernandez's task in 43s —— View job

---

Triage

Type: Enhancement — benchmarking/evaluation work to inform a model-selection decision, not a defect fix.

Priority: Medium — no user-facing breakage today (English retrieval works as designed); multilingual retrieval quality is a real gap but not an outage. Priority should rise once the benchmark identifies a concrete regression risk (e.g., current BGE model returning empty/false-positive results for CJK queries in production).

Component: Cloud, with Core implications. This issue explicitly spans both: the Core embedding provider/model configuration and the Cloud FastEmbed + PostgreSQL/pgvector inference, storage, image-baking, and fleet-reindex path. Search/index architecture (services, repository, FastEmbed provider config) is also touched but has no dedicated repo label.

Complexity: Complex — this is a multi-stage research and rollout initiative, not a single PR:

• Building a reproducible multilingual retrieval benchmark corpus + harness (recall@k, MRR/nDCG, false-positive rate) across 7+ languages and cross-language queries
• Runtime/operational benchmarking (latency, memory, artifact size, pgvector index behavior) on both local sqlite-vec and Cloud PostgreSQL/pgvector
• Verifying the model contract (dimensions, prefixes, normalization, embedding identity, cancellation/error propagation, license) for each candidate
• Designing and documenting a Cloud cutover plan (image baking, worker sizing, embedding-identity versioning, fleet-wide reindex, readiness monitoring)

The issue itself acknowledges this and asks for follow-up work to be split into bounded PRs/issues — that's the right approach given the scope.

Related prior work (not duplicates):

• [FEATURE] Add larger Chinese/multilingual embedding models to FastEmbed provider (e.g. bge-base-zh-v1.5, bge-m3) #1295 (closed) — earlier feature request to add specific multilingual FastEmbed models (bge-base-zh-v1.5, bge-m3); this issue supersedes that with an evidence-based benchmark-first approach rather than picking a model from the catalog.
• Investigate Gemini Embedding 2 as embedding provider option #677 (open) — investigate Gemini Embedding 2 as a provider option; different provider/scope, may be worth cross-referencing during candidate selection but doesn't overlap directly.
• [enhancement] CJK (Chinese/Japanese/Korean) full-text search broken — hardcoded unicode61 tokenizer doesn't segment CJK #1294 (open) — CJK lexical FTS tokenization; explicitly called out in this issue's summary as a separate, non-overlapping concern.
• fix(core): L2-normalize FastEmbed vectors to satisfy unit-vector contract #843, [BUG] FastEmbed embeddings are not L2-normalized → semantic search silently degrades to FTS-only for non-bge models (e.g. multilingual-mpnet) #1023, Document query/passage prefixes for asymmetric FastEmbed models #1264, Apply E5 query/passage prefixes via the prefixing provider defaults (FastEmbed asymmetry) #1284, [FEATURE] Add role-specific text prefixes for semantic embedding queries and documents #1008 (closed) — prior normalization/prefix-handling fixes this benchmark builds on.

No open duplicate found. Labels applied: enhancement, cloud (in addition to the existing needs investigation).