Skip to content

semantic_query returns identical (alphabetical) results for any query — vector-less nodes win the ranking #915

Description

@frangramirez

Summary

search_graph with semantic_query returns the same result set for any query whatsoever — different phrases, single terms, different languages all return identical hits, in what appears to be alphabetical order. The vector data in the index is fine (verified by computing cosine similarity manually against the SQLite DB); the bug is in the query executor: it returns exactly the nodes that have NO row in node_vectors, which suggests a LEFT JOIN + ORDER BY similarity DESC where the NULL/NaN similarity of vector-less nodes sorts above every real cosine score.

Environment

  • v0.8.1, darwin-arm64 (release binary, checksum verified)
  • Large TypeScript + Java monorepo (~10k files, 154k nodes after mode: "moderate" indexing)
  • Reproduced identically via cli one-shot mode AND via the stdio MCP server (tools/call)

Reproduction

codebase-memory-mcp cli index_repository '{"repo_path":"<repo>","mode":"moderate"}'

# Any of these return the SAME hits (JSON keys from a registry.json, alphabetical):
codebase-memory-mcp cli search_graph '{"project":"<slug>","semantic_query":["where is client debt calculated"],"limit":6}'
codebase-memory-mcp cli search_graph '{"project":"<slug>","semantic_query":["debt"],"limit":6}'
codebase-memory-mcp cli search_graph '{"project":"<slug>","semantic_query":["cost basis FIFO when selling stocks"],"limit":6}'

Every call returns (excerpt):

$behaviors_note        | apps/.../ui/registry.json
$bulkZipDownload_note  | apps/.../ui/registry.json
$candidateExemplar_note| apps/.../ui/registry.json
...

Note the results are query-independent and alphabetically ordered — the classic signature of sorting on NULL/NaN similarity.

Root-cause evidence (from the index DB)

The data side is healthy:

  • node_vectors: 46,606 rows × 768d int8 — populated.
  • token_vectors: 18,049 tokens with vector + idf; all my query terms (debt, overdue, balance, fifo, …) are present.

Manual cosine check (Python over the SQLite DB, query vector = mean of its token vectors):

query "client debt overdue balance" vs CcmaSaldoCalculator  → cos = 0.9492   (the CORRECT hit)
query "client debt overdue balance" vs $behaviors_note      → node has NO row in node_vectors

So the correct ranking exists in the data — CcmaSaldoCalculator (a debt-calculation class) scores 0.949 and should be the top hit. Instead, the executor returns precisely the nodes that are absent from node_vectors (JSON-key nodes from a registry.json, which apparently never get vectors).

Suggested fix

Exclude vector-less nodes from the semantic ranking — INNER JOIN node_vectors (or WHERE nv.vector IS NOT NULL), and/or treat missing similarity as -inf rather than letting NULL/NaN win the sort.

Side note

BM25 (query) and search_code work great on the same index — this report is only about the semantic_query path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions