Skip to content

feat: add text_match option to HybridSearchConfig for OR keyword matching - #339

Open
ankurjindal (ecomlisters) wants to merge 1 commit into
langchain-ai:mainfrom
jindalankuraws:feat/hybrid-search-text-match
Open

ankurjindal (ecomlisters) wants to merge 1 commit into
langchain-ai:mainfrom
jindalankuraws:feat/hybrid-search-text-match

Conversation

@ecomlisters

Copy link
Copy Markdown

Hybrid search builds its keyword query with plainto_tsquery, which ANDs every word. Natural-language questions rarely contain every word of the passage that answers them, so the keyword half often returns no rows and hybrid search quietly becomes vector-only.

This adds HybridSearchConfig.text_match:

  • "all" (default): unchanged, uses plainto_tsquery.
  • "any": matches documents containing at least one query term, ranked by ts_rank_cd as before.

Any other value raises ValueError when the config is created.

For "any", the terms come from to_tsvector(<tsv_lang>, :fts_query), so stemming and stop words behave exactly as they do for "all". Each term is quoted and cast directly to tsquery. The query stays a bound parameter. The terms are not passed through to_tsquery, because that re-parses them as query syntax: a query containing a URL like example.com/a:b?x=1 would raise syntax error in tsquery.

The GIN index on the TSV column is still used. On a 50,000-row table with aapply_hybrid_search_index(), EXPLAIN ANALYZE of the generated query shows a Bitmap Index Scan on the GIN index for both custom and generic plans. The term list is computed once per query (InitPlan), not per row.

Docs: the hybrid search section of examples/pg_vectorstore_how_to.ipynb documents text_match and corrects the stated tsv_column default (it is ""; init_vectorstore_table names a new column <content_column>_tsv). No execution counts or outputs changed.

Tests

  • tests/unit_tests/v2/test_async_pg_vectorstore_search.py:
    • A multi-word query where only one word is in a document: "all" returns no keyword matches, "any" returns the document (with and without a TSV column).
    • A stopword-only query ("how is the") does not error.
    • "third-party vendor's" does not error and matches.
    • Terms containing tsquery operators (a URL, & ! | ( ) ') do not error with "any".
    • The default config matches exactly the rows plainto_tsquery returns.
  • tests/unit_tests/v2/test_hybrid_search_config.py: "all" and "any" are accepted; other values raise ValueError.

make test: 767 passed. make lint is clean.

Note: this conflicts with #338 on one line, the from typing import ... import in tests/unit_tests/v2/test_async_pg_vectorstore_search.py. Whichever merges second needs the union of both imports.

…hing

Hybrid search builds its keyword query with plainto_tsquery, which ANDs
every word, so natural-language queries often get no keyword matches and
hybrid search silently becomes vector-only.

Add HybridSearchConfig.text_match. "all" (default) keeps plainto_tsquery.
"any" ORs the lexemes of to_tsvector(<tsv_lang>, :fts_query); each lexeme
is quoted and cast directly to tsquery so it is never parsed as tsquery
syntax. Other values raise ValueError.

Document text_match in the how-to notebook and correct the stated
tsv_column default.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant