feat(rag): support RAGFlow in a new `RAGFlowKnowledgeBase` class rather than VDB

[DavdGao]

Background

Through recent investigation into RAGFlow, we found that it provides a complete, end-to-end RAG pipeline — including document parsing, chunking, indexing, and retrieval — as a fully managed solution. This makes it fundamentally different from a traditional vector database (VDB).

Forcing RAGFlow into our existing VDB abstraction would be a poor fit: it would either require bypassing RAGFlow's core strengths (e.g., its built-in document processing and indexing pipeline) or lead to an awkward, leaky abstraction. To fully leverage what RAGFlow offers, it should instead be integrated at the Knowledge layer, sitting alongside the existing Knowledge class rather than beneath it.

Changes

Add RAGFlowKnowledge class under the rag module
Implement the same interface as the existing Knowledge class, ensuring consistency and interoperability across different knowledge backends
Encapsulate RAGFlow's document parsing, indexing, and retrieval capabilities within the new class

[MannXo]

I'll take this if no one's working on it yet.

[DavdGao]

@MannXo Thanks for your interest! Since RAGFlow offers a more complete, end-to-end pipeline compared to a traditional VDB, there may well be additional functionalities worth exposing beyond the current Knowledge class interface. If you identify any such capabilities during your exploration, feel free to raise them here — we're happy to discuss and align on the interface design before diving into implementation.

[MannXo]

Thanks, that offer to align first is very welcome, because the exploration turned up more than I expected. Findings below, ending with the decisions I think are yours to make.

Where RAGFlow fits cleanly

The agent-facing surface is smaller than it looks. RAGMiddleware only ever touches name, description and search() on a knowledge base, and search() already takes text rather than a vector. That maps almost exactly onto POST /api/v1/retrieval, so the retrieval half needs no new concepts.

The document lifecycle also lines up better than I expected. KnowledgeDocumentStatus is already pending / parsing / chunking / indexing / ready / error, and RAGFlow reports run, progress, progress_msg, chunk_count and token_count per document. Its server-side parse maps onto states we already model rather than needing new ones.

Where it does not

KnowledgeBase is a concrete class, not an interface. Its constructor binds an embedding model, a vector store, a collection and a metadata filter, and embedding_model / vector_store / collection / metadata_filter / ensure_collection() are all public. None of those have a RAGFlow analogue, since RAGFlow owns embedding and storage itself. So "implement the same interface" needs an interface to exist first. My preference is to extract a narrow ABC holding only what both backends genuinely share, and leave today's class as the VDB implementation beneath it, but that is a call worth making explicitly.

insert_document(chunks: list[Chunk]) is the sharper problem. It takes pre-chunked input because parsing and chunking live one layer up, in the index worker. RAGFlow parses and chunks server-side from a file. Implementing the signature as it stands would mean looping add_chunk per chunk, which is precisely the bypassing of RAGFlow's pipeline this issue sets out to avoid. A file-oriented ingestion path seems unavoidable.

Persistence assumes a VDB too. KnowledgeBaseData requires embedding_model_config and collection_name, and KnowledgeBaseManagerBase takes a vector_store and declares get_dimension_policy() abstract. A RAGFlow-backed record has nothing meaningful to put in any of them.

Capabilities worth exposing beyond the current interface

You asked me to flag these. All are from the HTTP API reference:

• Hybrid retrieval knobs: vector_similarity_weight to balance vector against keyword, plus keyword, similarity_threshold, knn_top_k and rerank_candidates_count.
• Reranking with rerank_id.
• Score decomposition per hit: similarity, vector_similarity and term_similarity come back separately, where VectorSearchResult carries one score.
• Citation material: highlight, positions, document_keyword, and doc_aggs for per-document hit counts.
• Cross-language retrieval via cross_languages.
• Knowledge-graph retrieval via use_kg.
• Retrieval across several datasets in one call, since dataset_ids is a list.
• Chunking strategy per dataset and per document through chunk_method and parser_config.
• Chunk-level curation: important_keywords, questions, and available to disable a chunk without deleting it.
• Native metadata scoping through meta_fields on a document and metadata_condition at retrieval.

That last one matters for a reason beyond features. In #2048 the metadata filter ended up a silent no-op, which broke the scoping the base class relies on. Going through meta_fields and metadata_condition makes that boundary real rather than decorative.

One hard constraint

ragflow-sdk cannot be used. Versions 0.26.1 and up require Python >=3.13,<3.14, and even 0.25.x required >=3.12, while AgentScope supports >=3.11 and CI runs 3.11 only. Adding it to an extra would break pip install on the project's own minimum. It is also requests-based and synchronous, against a fully async RAG layer. I would target the documented HTTP API with httpx, which is already a core dependency, so this adds no new dependency at all.

Decisions I would like your call on

1. Extract an ABC above KnowledgeBase, or keep the two classes independent and duck-typed?
2. Does a RAGFlow knowledge base get a file-oriented ingestion entry point, and if so does that sit on the new interface or only on the RAGFlow class?
3. KnowledgeBaseData needs embedding_model_config and collection_name to become optional for this backend. Acceptable, or would you rather a separate record shape?
4. How should this sit alongside feat(middleware): add ExternalRetrievalMiddleware with RAGFlow backend #2276, which adds a RAGFlow retrieval backend at the middleware layer? Between that and this, RAGFlow would be reachable two ways.
5. Does this supersede feat(vdb): support RAGflow as an optional vector database backend #2038 and feat(rag): add RAGFlow as an optional vector database backend #2048?

Happy to sketch the ABC concretely if that is easier to react to than prose.