Skip to content

Remove all embedding functionality, focus on vector storage and search, clean dead code - #11

Open
maxguru wants to merge 1 commit into
allanpichardo:mainfrom
maxguru:feature/refactor-to-vector-storage-only-library
Open

Remove all embedding functionality, focus on vector storage and search, clean dead code#11
maxguru wants to merge 1 commit into
allanpichardo:mainfrom
maxguru:feature/refactor-to-vector-storage-only-library

Conversation

@maxguru

@maxguru maxguru commented Jul 19, 2025

Copy link
Copy Markdown

BREAKING CHANGE: Removed all text embedding generation capabilities to create a focused vector storage and similarity search library.

  • Remove entire src/Nlp/ directory (Embedder, tokenizers, ONNX model)
  • Remove tests/Nlp/ directory and all embedding-related tests
  • Clean up composer.json dependencies (remove onnxruntime, kmeans, polyfills)
  • Remove composer.lock to allow flexible dependency resolution
  • Update README.md to reflect vector-only capabilities
  • Clean unused imports and properties from VectorTable class

…d search, clean dead code

BREAKING CHANGE: Removed all text embedding generation capabilities to create a focused vector storage and similarity search library.

- Remove entire src/Nlp/ directory (Embedder, tokenizers, ONNX model)
- Remove tests/Nlp/ directory and all embedding-related tests
- Clean up composer.json dependencies (remove onnxruntime, kmeans, polyfills)
- Remove composer.lock to allow flexible dependency resolution
- Update README.md to reflect vector-only capabilities
- Clean unused imports and properties from VectorTable class
@titonova

titonova commented Nov 1, 2025

Copy link
Copy Markdown

@maxguru what was your reasoning behind this overhaul

@maxguru

maxguru commented Nov 4, 2025

Copy link
Copy Markdown
Author

@maxguru what was your reasoning behind this overhaul

The embedding generation component should really be in a separate library.

Users of the vector storage component are rarely going to need to generate embeddings using the built in model anyway, probably because it will be outdated or wouldn't fit their needs.

Also, the embedding generation component has some hefty requirements that I've mentioned in this issue. These requirements are preventing some users (like me) from using the library.

I want to mention that I nearly completely rewrote the VectorTable class, added a number of very useful features, solved a number of practical problems and released my own fork of this library. You should check it out, it is much better now.

@floriankraemer

Copy link
Copy Markdown

I fully support the reasoning of @maxguru, he is right. 👍 Please accept his PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants