analyze.py: clustering is lexical, not semantic

[hailcpy]

Two gaps in topic matching:

1. Cross-module, no shared file (the strict-xfail test_crossmodule_clustering_is_a_known_gap): gRPC adopted in two modules shares the concept but no file/deep-dir → split into two candidates. Needs semantic/embedding clustering (deferred v2).
2. Literal token overlap: no stemming/synonyms. "Kafka" vs "event streaming", "redis" vs "caching" wont merge; coincidental word reuse can over-merge.

Fix (v2): embedding-based candidate merging, prototyped offline and proven to beat the deterministic pass before adoption.

[hailcpy]

Additional review detail and v2 direction:

Finding

Embeddings should help, but they should not replace the deterministic graph. The current known gap is cross-module decisions that share a concept but no file/deep-dir overlap. That is a recall problem. Embedding-only clustering would introduce a precision and explainability problem.

Suggested solution

Use embeddings as a fifth edge signal, not as the whole clustering system.

A practical design:

1. Build initial chunks from real PR/merge/squash boundaries.
2. Produce a normalized change summary per chunk: commit/PR text, touched symbols, imports added/removed, dependency/schema/API changes, and a capped diff summary.
3. Embed the normalized summary, not raw diffs alone.
4. Generate candidate semantic edges only above a high threshold.
5. Merge only when semantic similarity is backed by at least one explainable signal, or mark as needs_review in assisted mode.
6. Store edge reasons in JSON: semantic_similarity, shared_symbol, shared_dependency, shared_pr_keyword, etc.

Clustering options worth prototyping:

• weighted graph + Leiden over multi-signal edges,
• HDBSCAN for unknown cluster count and noise handling,
• BERTopic-style topic labels for human-readable cluster names,
• reranker/LLM only as an assisted-mode reviewer, never as a silent merge oracle.

Acceptance criteria

• Add an offline prototype/eval script comparing deterministic-only vs hybrid semantic clustering.
• The xfail cross-module fixture becomes passable under an explicit semantic mode.
• Embedding-only edges are visible and reviewable, not silently treated as strong evidence.
• Regression tests show semantic mode improves recall without obvious precision collapse.