Remove all embedding functionality, focus on vector storage and search, clean dead code - #11
Conversation
…d search, clean dead code BREAKING CHANGE: Removed all text embedding generation capabilities to create a focused vector storage and similarity search library. - Remove entire src/Nlp/ directory (Embedder, tokenizers, ONNX model) - Remove tests/Nlp/ directory and all embedding-related tests - Clean up composer.json dependencies (remove onnxruntime, kmeans, polyfills) - Remove composer.lock to allow flexible dependency resolution - Update README.md to reflect vector-only capabilities - Clean unused imports and properties from VectorTable class
|
@maxguru what was your reasoning behind this overhaul |
The embedding generation component should really be in a separate library. Users of the vector storage component are rarely going to need to generate embeddings using the built in model anyway, probably because it will be outdated or wouldn't fit their needs. Also, the embedding generation component has some hefty requirements that I've mentioned in this issue. These requirements are preventing some users (like me) from using the library. I want to mention that I nearly completely rewrote the |
|
I fully support the reasoning of @maxguru, he is right. 👍 Please accept his PR. |
BREAKING CHANGE: Removed all text embedding generation capabilities to create a focused vector storage and similarity search library.