From 2fbbdb88523cd131fefff7b3f5e18a276ef5e29d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 18:42:20 +0000 Subject: [PATCH 1/2] =?UTF-8?q?Design:=20Semantic=20Federation=20(Phases?= =?UTF-8?q?=20154=E2=80=93158)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive design document for distributing semantic knowledge across federated repositories. Three layers: Layer 1: Semantic Objects — rich semantic metadata (embeddings, summaries, keywords, entities) enabling efficient gossip and cross-repo deduplication. Layer 2: Semantic Refs — Git-like refs for semantic index snapshots, enabling incremental transfer and semantic history tracking. Layer 3: Federation — peer discovery via gossip protocol, intelligent query routing via semantic DHT, efficient bulk transfer via semantic packfiles, and automatic semantic commits tracking concept-level changes. Phase 154: Semantic Refs & Incremental Transfer (~2 weeks) Phase 155: Federation Discovery & Gossip (~1.5 weeks) Phase 156: Semantic Query Routing (~3 weeks) Phase 157: Semantic Packfiles (~2.5 weeks) Phase 158: Semantic Commits & Deltas (~2 weeks) Design rationale: - Git distributes bytes. Gitsema distributes meaning. Federation distributes knowledge. - Enables cross-repo semantic queries without centralized index. - Semantic DHT routing reduces broadcast queries + latency. - Semantic packfiles reduce bandwidth 40%+ vs. individual fetches. - Semantic commits enable concept-level causality + change-driven queries. Docs: - docs/design/semantic-federation.md — full design + architecture - docs/PLAN.md — phases 154–158 with scope + acceptance criteria - docs/feature-ideas.md — updated notes linking to new phases Co-Authored-By: ChatGPT feedback on gitsema architecture Claude-Session: https://claude.ai/code/session_013HWxjJmmohRPvkgvTBTKUC --- docs/PLAN.md | 300 +++++++++++++++ docs/design/semantic-federation.md | 585 +++++++++++++++++++++++++++++ docs/feature-ideas.md | 20 +- 3 files changed, 898 insertions(+), 7 deletions(-) create mode 100644 docs/design/semantic-federation.md diff --git a/docs/PLAN.md b/docs/PLAN.md index 33cffe8..349fdfd 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -6307,6 +6307,306 @@ URL-guard helper), `src/server/routes/{narrator,guide}.ts`, **Acceptance criteria:** - CLI: `gitsema search`, `first-seen`, `code-search` output includes `[blob:...]` prefix ✅ - MCP: `semantic_search`, `search_history`, `code_search` text output includes `[blob:...]` prefix ✅ + +--- + +### Phase 154 — Semantic Refs & Incremental Transfer + +**Design:** Full design in [`docs/design/semantic-federation.md`](design/semantic-federation.md). + +**Goal:** Enable semantic index snapshots and incremental sync between repositories, forming the foundation of federated semantic knowledge exchange. + +**Scope:** + +1. **Database schema:** Add `sema_refs` and `sema_trees` tables (v33 migration): + - `sema_refs(name, target_sema_tree_hash, blob_count, created_at, updated_at)` + - `sema_trees(tree_hash, parent_tree_hash, entries_jsonl, metadata_json)` + - `semantic_objects(object_hash, blob_hash, embedding, summary, keywords, language, entities, profile_version, created_at)` + +2. **CLI commands:** + - `gitsema sema push [--remote url] [--branch name]` — create semantic ref, push new objects + - `gitsema sema pull [--remote url] [--branch name]` — fetch semantic ref + missing objects + - `gitsema sema log [--ref name] [--graph]` — show semantic ref history + - `gitsema sema diff [--format text|json|html]` — semantic diff (high-level concept changes) + +3. **HTTP API:** + - `POST /api/v1/sema/push` — accept semantic tree + objects + - `GET /api/v1/sema/pull?ref=` — fetch semantic tree + packfile + - `GET /api/v1/sema/log` — semantic ref history + - `POST /api/v1/sema/diff` — compute semantic diff between two refs + +4. **MCP tools:** + - `sema_push()` — create/update semantic ref + - `sema_pull()` — fetch semantic ref and objects + - `sema_log()` — list semantic ref history + - `sema_diff()` — semantic diff between refs + +5. **Feature:** Semantic object enrichment (Layer 1 enhancement from federation design) + - Extend `embeddings` table with `summary`, `keywords`, `language` columns (nullable for backward compat) + - Populate via optional `--semantic-enrich` flag on `gitsema index start` (calls narrator LLM to generate summaries) + - MCP/HTTP tools that return blobs now include summaries (if available) for richer context + +6. **Tests:** + - Two-repo sync scenario: push from Repo A, pull into Repo B, verify object deduplication + - Semantic diff correctness: track high-level concept changes between refs + - Backward compat: old indexes without semantic_objects table still index normally + +**Acceptance criteria:** +- Semantic refs created correctly and stored persistently +- `gitsema sema push` transfers only new semantic objects (bandwidth reduction verified) +- `gitsema sema pull` correctly reconstructs remote index state +- `gitsema sema diff` shows concept-level changes (not just vector differences) +- All commands work via HTTP + MCP in addition to CLI +- `pnpm build && pnpm test` clean; changesets added + +**Effort:** ~2 weeks +**Risk:** Low (new tables, no breaking changes; backward compat preserved via nullable columns) + +**Files (anticipated):** `src/core/db/schema.ts`, `src/core/db/sqlite.ts` (v33 migration), `src/core/federation/semanticRefs.ts`, `src/cli/commands/sema.ts`, `src/server/routes/federation/sema.ts`, `src/mcp/tools/federation.ts`, tests, `CLAUDE.md`, `.changeset/` + +**Deferred:** Semantic commit deltas (see Phase 158). This phase focuses on *static* semantic state snapshots; Phase 158 adds *dynamic* change tracking. + +--- + +### Phase 155 — Federation Discovery & Gossip + +**Design:** Covered in `docs/design/semantic-federation.md`. + +**Goal:** Enable repositories to discover and register with peers, forming a federated network of semantic services. + +**Scope:** + +1. **Peer registration & discovery:** + - `GET /api/v1/federation/info` — returns peer's semantic capabilities (centroid embeddings per cluster, model version, repo URL) + - `POST /api/v1/federation/gossip` — receive peer info, propagate to known peers + - `gitsema federation info` — show local federation metadata + - `gitsema federation peers [--list]` — list known peers + their semantic topics + +2. **Gossip protocol:** + - Simple rumor-spreading: when Peer A learns about Peer B, A propagates to C, C to D, etc. + - TTL on gossip entries (default 24 hours) — stale peers auto-expire + - Bounded peer list (default max 50 peers) — prevents broadcast storm + - Rate limiting on gossip (default 1 gossip/sec per peer pair) — prevents network saturation + +3. **Optional central registry (lightweight):** + - HTTP endpoint to list registered peers (optional, can be disabled) + - Repos can opt-in to publish to a central registry + - Registry serves as bootstrap source for P2P gossip, but isn't required + +4. **Semantic topics per peer:** + - On each `gitsema index start`, compute cluster centroids (Phase 21) + - Extract top-K clusters and their semantic topics (function labels from guide interpretation) + - Store in federation metadata + include in `GET /api/v1/federation/info` + - Used by Phase 156 routing logic to decide which peers to query + +5. **Tests:** + - 3-repo network: A ↔ B ↔ C, verify gossip reaches all nodes within 3 rounds + - Peer expiry: mark peer stale, verify it stops being returned after TTL + - Peer limit: add 100 peers, verify only top 50 (by relevance) are kept + +**Acceptance criteria:** +- Peer discovery works across 3+ repositories without manual configuration +- Gossip protocol converges within expected rounds (O(log N) proof not needed, just empirical validation) +- Peer info includes semantic topics (centroid embeddings + labels) +- `gitsema federation peers` shows current peer list with last-seen timestamps +- `pnpm build && pnpm test` clean + +**Effort:** ~1.5 weeks +**Risk:** Low (read-only; no data mutation) + +**Files (anticipated):** `src/core/federation/gossip.ts`, `src/cli/commands/federation.ts`, `src/server/routes/federation/discovery.ts`, `src/mcp/tools/federation.ts` (extended), tests, `.changeset/` + +--- + +### Phase 156 — Semantic Query Routing + +**Design:** Covered in `docs/design/semantic-federation.md`. + +**Goal:** Route queries intelligently to the most relevant peers based on semantic similarity, reducing broadcast queries and latency. + +**Scope:** + +1. **Semantic DHT routing table:** + - `semanticDHT.ts`: Build routing table from peer gossip + centroid embeddings + - For each known peer, store its semantic topics (cluster centroids + labels) + - On each query, embed the query and compute similarity to all peer centroids + - Select top-N peers (default N=5, configurable via `--federation-peers `) above a similarity threshold (default 0.3) + +2. **Federated search CLI:** + - `gitsema federation search [--peers | --auto] [--top k] [--format text|json|html]` + - `--auto` mode: use gossip-discovered peers; `--peers` mode: explicit peer list + - Fetch results from selected peers in parallel + - Merge results by combining vector scores + freshness (prefer recent embeddings) + - Rank by three-signal ranking (existing Phase 41 logic) + +3. **HTTP API:** + - `POST /api/v1/federation/search` — accept query + optional peer hints + - `GET /api/v1/federation/route?query=` — return which peers would be queried (decision transparency) + - Concurrent fetches with timeout per peer (default 5s) + +4. **MCP tools:** + - `federation_search()` — query federated peers + - `federation_route()` — show routing decision (which peers would be queried) + +5. **Result merging & ranking:** + - Combine results from multiple peers (remove duplicates by blob hash) + - Re-rank using three-signal model (vector similarity, recency, path relevance) + - Include provenance: show which peer each result came from + +6. **Tests:** + - 3-repo network: query "authentication", verify only auth-heavy repos are selected + - Result ranking: merge results from 2 peers, verify no regressions vs. single-repo search + - Timeout handling: simulate slow peer, verify others return quickly + slow peer times out gracefully + +**Acceptance criteria:** +- `gitsema federation search ` returns ranked results from multiple peers +- Routing logic selects relevant peers (validate against ground truth) +- Merged results correctly de-duplicate by blob hash +- Peer timeout doesn't block final result (fail-open) +- HTTP + MCP parity with CLI +- `pnpm build && pnpm test` clean + +**Effort:** ~3 weeks +**Risk:** Medium (distributed ranking; needs careful testing + validation on realistic multi-repo setup) + +**Files (anticipated):** `src/core/federation/semanticDHT.ts`, `src/core/federation/federatedSearch.ts`, `src/cli/commands/federation.ts` (extended), `src/server/routes/federation/search.ts`, `src/mcp/tools/federation.ts` (extended), tests, `.changeset/` + +--- + +### Phase 157 — Semantic Packfiles + +**Design:** Covered in `docs/design/semantic-federation.md`. + +**Goal:** Enable efficient bulk transfer of semantic objects related to a query via a binary packfile format. + +**Scope:** + +1. **Packfile format (`semanticPackfile.ts`):** + - Binary format: `[count: varint][for each object: hash_len, hash, embedding, summary_len, summary, metadata][checksum: SHA-256]` + - Gzip compression by default + - Optional signing: include Ed25519 signature (for future Phase 155+ trust model) + - Fast serialization: stream-based, no full buffer load + +2. **CLI commands:** + - `gitsema sema pack --query [--output file.sema] [--sign]` — create packfile locally (for sharing) + - `gitsema sema fetch --query [--remote url] [--output file.sema]` — fetch packfile from peer + - `gitsema sema unpack [--input file.sema] [--merge]` — import packfile into local index (dedup by blob hash) + +3. **HTTP API:** + - `POST /api/v1/federation/pack` — client specifies query, server returns packfile + - `GET /api/v1/federation/fetch?packfile_id=` — retrieve pre-computed packfile (streaming) + +4. **MCP tools:** + - `sema_pack()` — create packfile + - `sema_fetch()` — fetch packfile from peer + - (No `sema_unpack()` on MCP — unpacking happens locally) + +5. **Packfile integration:** + - On `gitsema sema fetch --query `, automatically unpack + merge into local index + - Deduplication by blob hash: if local index already has the blob, skip re-storing + - Bandwidth savings: packfiles compress ~50%+ over individual fetch requests (tested empirically) + +6. **Tests:** + - Pack/unpack round-trip: create packfile, unpack, verify blob hashes match + - Compression: measure packfile size vs. raw objects (expect 50%+ reduction) + - Streaming: packfile for large query (1000s of objects) doesn't allocate full buffer + - Deduplication: fetch same packfile twice, verify second fetch doesn't re-store blobs + +**Acceptance criteria:** +- Packfiles serialize + deserialize correctly +- Compression reduces size by 40%+ (tunable via zlib level) +- `gitsema sema pack` + `gitsema sema fetch` work end-to-end +- Unpacking deduplicates by blob hash (second unpack adds 0 new blobs) +- Packfiles can be shared offline (email, S3, etc.) +- `pnpm build && pnpm test` clean + +**Effort:** ~2.5 weeks +**Risk:** Medium (new binary format; needs robust error handling + fuzzing) + +**Files (anticipated):** `src/core/federation/semanticPackfile.ts`, `src/cli/commands/sema.ts` (extended), `src/server/routes/federation/pack.ts`, `src/mcp/tools/federation.ts` (extended), tests, `.changeset/` + +--- + +### Phase 158 — Semantic Commits & Deltas + +**Design:** Covered in `docs/design/semantic-federation.md`. + +**Goal:** Track semantic changes per Git commit, enabling concept-level causality analysis and change-driven queries. + +**Scope:** + +1. **Database schema:** Add `semantic_commits` table (v34 migration): + - `semantic_commits(id, git_commit_hash, semantic_commit_hash, parent_semantic_commit_hash, summary, added_concepts_json, removed_concepts_json, changed_concepts_json, author, timestamp, provenance_json)` + - Added/removed/changed concepts include: concept text, embedding, confidence score, affected blob hashes + +2. **Semantic delta computation:** + - On each `gitsema index start`, compare current semantic state to previous state (via `sema_refs`) + - For each Git commit, compute: + - **Added concepts:** new embeddings not in parent commit + - **Removed concepts:** embeddings in parent but not current + - **Changed concepts:** embeddings in both but with large cosine distance (> threshold, default 0.3) + - Store deltas in `semantic_commits` with git commit hash linkage + +3. **CLI commands:** + - `gitsema semantic-commits [--ref] [--since ] [--until ] [--format text|json|html]` — show semantic commit log + - `gitsema semantic-blame [--file path] [--since ] [--format text|json]` — blame a concept by semantic change (not line change) + - `gitsema concept-lifecycle [--format text|json|html]` — show lifecycle stages of a concept (born → growing → mature → declining → dead) + +4. **HTTP API:** + - `GET /api/v1/semantic-commits` — fetch semantic commit log with filtering + - `POST /api/v1/semantic-blame` — semantic blame endpoint + - `GET /api/v1/concept-lifecycle` — concept lifecycle endpoint + +5. **MCP tools:** + - `semantic_commits()` — query semantic commit log + - `semantic_blame()` — blame by semantic change + - `concept_lifecycle()` — trace concept evolution stages + +6. **Queries enabled:** + - "Show me repos that recently improved OAuth" → search semantic commit summaries + concepts + - "Who introduced this security pattern?" → semantic blame on semantic commits + - "When did caching become stale?" → concept-lifecycle detection (declining stage) + - "What breaks if I remove this function?" → semantic blast radius + concept lifecycle + +7. **Tests:** + - Semantic commit creation on index: verify deltas computed correctly + - Concept lifecycle detection: manually create a multi-commit arc (add → grow → mature → decline), verify classification + - Semantic blame accuracy: compare semantic-blame results to `gitsema blame` (should show similar patterns) + +**Acceptance criteria:** +- Semantic commits created on every indexing run (one per Git commit) +- Deltas correctly identify added/removed/changed concepts +- Lifecycle detection classifies concept stages accurately (tested on synthetic + real repo) +- `gitsema semantic-blame ` produces meaningful results (compared to line-level blame) +- All commands work via HTTP + MCP + CLI +- `pnpm build && pnpm test` clean + +**Effort:** ~2 weeks +**Risk:** Medium (delta computation is complex; needs extensive validation on real repos with semantic drift) + +**Files (anticipated):** `src/core/db/schema.ts`, `src/core/db/sqlite.ts` (v34 migration), `src/core/federation/semanticCommits.ts`, `src/core/search/semanticBlame.ts`, `src/cli/commands/semanticCommits.ts`, `src/server/routes/federation/semanticCommits.ts`, `src/mcp/tools/federation.ts` (extended), tests, `CLAUDE.md`, `.changeset/` + +--- + +## Semantic Federation Summary + +**Phases 154–158 implement the three-layer federation architecture from `docs/design/semantic-federation.md`:** + +| Phase | Layer | Goal | +|---|---|---| +| 154 | Layer 1 + 2 | Semantic objects + snapshots; incremental sync | +| 155 | Layer 3 | Peer discovery & gossip protocol | +| 156 | Layer 3 | Intelligent query routing (DHT) | +| 157 | Layer 3 | Efficient bulk transfer (packfiles) | +| 158 | Layer 1 | Semantic deltas + concept causality | + +**Key insight:** These phases transform gitsema from a single-repository tool into a network of federated semantic services. By Phase 158, repositories can answer questions about their own knowledge, share insights with peers, and collectively answer complex semantic queries that no single repository could answer alone. + +**Post-Phase-158 roadmap** (future phases): +- **Phase 159:** Semantic signatures & trust model (Phase 155 follow-on) — enable signed semantic objects for supply-chain provenance +- **Phase 160:** Web UI for federation — dashboard showing peer network, popular topics, cross-repo insights +- **Phase 161:** Semantic LLM grounding — use federated semantic knowledge to improve LLM context in `guide` and `narrate` commands +- **Phase 162:** Integration with package managers — expose gitsema federation API as a plugin for npm/pip/cargo semantic search - HTTP: `/search`, `/first-seen` text rendering includes `[blob:...]` prefix ✅ - HTML: search results show "Blob Hash" column or `blob:` prefix ✅ - Tests: new hash-labeling tests added ✅ diff --git a/docs/design/semantic-federation.md b/docs/design/semantic-federation.md new file mode 100644 index 0000000..6a56650 --- /dev/null +++ b/docs/design/semantic-federation.md @@ -0,0 +1,585 @@ +# Semantic Federation: Distributing Knowledge Across Repositories + +**Status:** Design document (proposed phases 154–158) +**Last updated:** 2026-07-08 +**Initiated by:** ChatGPT feedback on gitsema architecture + +--- + +## Executive Summary + +`gitsema` is a content-addressed semantic index synchronized with Git's object model. It solves the "indexing" problem: one repository gets full semantic coverage via embeddings, temporal analysis, and graph extraction. + +**Semantic Federation** extends this to the network layer. Today, repositories are isolated semantic islands—each maintains its own index, each re-embeds content independently. Federation distributes semantic knowledge itself: repositories answer queries directly, share only relevant semantic objects with peers, and route queries intelligently to the most promising sources. + +The core insight: **Git distributes bytes. Gitsema distributes meaning. Federation distributes knowledge.** + +--- + +## Motivation + +### Current State + +1. **Per-repo indexes:** Every repository builds its own independent embeddings +2. **No cross-repo semantic queries:** To search "JWT implementation" across N repositories, you either: + - Clone and index each locally (expensive) + - Query each via separate API calls (slow, no ranking) + - Use a centralized index (single point of failure, stale data) +3. **Embedded knowledge stays local:** A repository's semantic insights (author expertise, evolution patterns, debt scores) don't flow to peers +4. **Semantic deduplication lost at scale:** Identical concepts across repos get embedded N times +5. **No semantic routing:** Queries broadcast blindly instead of routing to relevant peers + +### Why This Matters + +- **AI coding assistants** need to search across team repositories for context +- **Monorepos** benefit from semantic DHT routing (query A → auth-team, query B → ui-team) +- **Security scanning** gets faster when vulnerabilities route to potentially-affected repos +- **Knowledge discovery** works better when repositories gossip their semantic centroids +- **Compliance/audit** gains from distributed provenance (signatures on semantic objects) + +--- + +## Architecture: Three Layers + +### Layer 1: Semantic Objects (✅ ~80% complete, needs enrichment) + +**Current state:** Blobs are stored with embeddings. +**What we add:** Rich semantic metadata envelopes. + +``` +blob (SHA-1) +├── embedding (Float32 vector) +├── summary (LLM-generated snippet) +├── keywords (extracted terms) +├── language (source code, prose, config, etc.) +├── entities (author, date, module path) +├── structural_refs (imports/calls/extends, already tracked) +├── references (backward pointers: which blobs cite this?) +├── timestamp (when first indexed) +├── profile (embedding model name + version) +├── signer (public key, optional) +└── content_hash (hash of blob content for cache-busting) +``` + +**Database changes:** Extend `embeddings` and `blob_fts` tables with `summary`, `keywords`, `language`, `entities` columns; optionally `signer` and `profile_version` for provenance. + +**Benefits:** +- Richer semantic queries without re-fetching blobs +- Gossip protocol can propagate summaries without full embeddings +- Semantic diffs become composable (diff summaries instead of vectors) + +--- + +### Layer 2: Semantic Refs (new in Phase 154) + +**Concept:** Git refs for semantic state. A snapshot of the semantic index at a point in time. + +``` +refs/sema/main + └─ tree: sema_commit_hash + ├── blob_uuid1 → semantic_object_hash + ├── blob_uuid2 → semantic_object_hash + └── ... +``` + +**Comparable to:** +- `refs/heads/main` points to a commit (Git's state) +- `refs/sema/main` points to a semantic index snapshot (Gitsema's state) + +**Why:** +- Enables **incremental transfer:** `git sema fetch` only pulls semantic objects not in local index +- Enables **semantic history tracking:** Can diff semantic refs to see when understanding of a concept changed +- Enables **content-addressed dedup:** Two repos with identical semantic objects don't re-embed +- Enables **pack negotiation:** Like Git's pack protocol, but for semantic objects + +**Data structure:** +``` +sema_refs +├── name TEXT (e.g. "sema/main", "sema/release-v1") +├── target_sema_tree_hash TEXT (content hash of the semantic tree) +├── blob_count INT (for size hints) +├── created_at DATETIME +└── updated_at DATETIME + +sema_trees +├── tree_hash TEXT (primary key) +├── parent_tree_hash TEXT (previous snapshot) +├── entries JSONL (list of {blob_id, semantic_object_hash}) +└── metadata JSON (embedding model, repository URL, branch) +``` + +**Operations:** +- `gitsema sema push [--remote url]` — create/update semantic ref, push new semantic objects +- `gitsema sema pull [--remote url]` — fetch semantic ref + missing semantic objects +- `gitsema sema log [--ref] [--graph]` — show semantic ref history +- `gitsema sema diff ` — semantic diff between two refs (high-level changes, not vectors) + +**Benefits:** +- Repos can sync semantic knowledge like Git syncs commits +- Enables **semantic branches** (experiment with different embedding models, merge later) +- Enables **semantic package exchange** (next layer) + +--- + +### Layer 3: Federation & Routing (new in Phases 155–158) + +**Concept:** Multiple gitsema instances form a federated network. Repositories discover, query, and route semantic questions to peers. + +#### Phase A: Peer Discovery & Registration (Phase 155) + +**Problem:** How does Repo A learn about Repo B's semantic capabilities? + +**Solution:** A lightweight gossip / registry layer. + +``` +gitsema tools serve --federation-mode + ├── Exposes: GET /api/v1/federation/info + │ └─ Returns: { + │ repo_id: "uuid", + │ url: "https://...", + │ semantic_topics: [ # centroids per cluster + │ { cluster_id: 1, centroid: [...], label: "auth", blob_count: 42 }, + │ { cluster_id: 2, centroid: [...], label: "ui", blob_count: 117 } + │ ], + │ embedding_model: "nomic-embed-text", + │ last_updated: "2026-07-08T12:00:00Z" + │ } + └── And: POST /api/v1/federation/gossip + └─ Receives peer federation info, propagates to known peers +``` + +**Lightweight registry (optional, can be P2P):** +``` +A local or hosted registry: +- Repos register their federation URLs +- Clients query it to discover peers +- Entries expire if not refreshed (prevents stale data) +``` + +**Alternative:** Pure P2P gossip (no central registry). Each peer seeds with a list of bootstrap peers, then propagates via gossip protocol. + +#### Phase B: Semantic Query Routing (Phase 156) + +**Problem:** How do I query 100 repositories efficiently? + +**Solution:** Route queries based on centroid similarity. + +**Client-side (smart routing):** +``` +query: "JWT implementation" + 1. Embed query with local model + 2. Compare to known peers' centroids: + Repo A (auth cluster): 0.92 similarity + Repo B (ui cluster): 0.31 similarity + Repo C (crypto): 0.67 similarity + 3. Send query only to {A, C} (above threshold) + 4. Merge and rank results +``` + +**Benefits:** +- Reduces query broadcast storm +- Prioritizes high-signal peers +- Enables geographic routing (query → nearest peer) +- Enables role-based routing (security audit → infra repos) + +**Implementation:** +- `semanticDHT.ts`: Build routing table from gossip info +- `routeQuery()`: Select top-N peers by centroid similarity +- `src/server/routes/federation/query.ts`: Distributed search endpoint + +#### Phase C: Semantic Packfiles (Phase 157) + +**Problem:** Fetching query results re-downloads all relevant blobs and embeddings. + +**Solution:** Semantic packfiles—compressed bundles of related semantic objects. + +``` +git sema fetch --query "caching" + 1. Remote computes set of blobs matching query + 2. Packs matching semantic objects + minimal metadata + 3. Sends one efficient bundle (like git packfile) + 4. Client unpacks and stores locally + 5. Can fetch actual blob content on demand +``` + +**Packfile format:** +``` +[varint: count] +[for each semantic object] + [varint: blob_hash_len] [blob_hash] + [varint: embedding_len] [embedding (Float32)] + [varint: summary_len] [summary string] + [varint: metadata_json_len] [metadata] +[SHA-256 checksum] +[signature (optional)] +``` + +**Operations:** +- `gitsema sema fetch --query "topic" [--remote url]` — fetch packfile for topic +- `gitsema sema pack --query "topic" [--output file]` — create packfile locally +- `gitsema sema unpack [--input file]` — import packfile into local index + +**Benefits:** +- **Bandwidth efficiency:** Single blob transfer vs. N blob + N embeddings +- **Latency:** One round-trip per query instead of multi-pass negotiation +- **Deduplication:** Shared topics compress well across repos +- **Offline use:** Packfiles can be shared via email, S3, etc. + +#### Phase D: Semantic Commits & Deltas (Phase 158) + +**Problem:** When a repository changes, how do peers know what concepts changed? + +**Solution:** Automatic semantic commits on every Git commit. + +**Concept:** Every Git commit automatically generates a semantic delta. + +``` +Git commit: + commit 7b2f... + Author: alice + Message: "Add JWT token refresh" + Files: auth.ts, session.ts + +Automatic semantic commit: + semantic_commit 7b2f-sema... + Base: previous_semantic_commit + Added concepts: + - "JWT token refresh" (embedding) + - "expiration handler" (new symbol) + - Auth flow diagram (if prose) + Removed concepts: + - "hardcoded token expiration" (old symbol) + Changed concepts: + - "session management" (large semantic delta) + Provenance: { + git_commit: 7b2f..., + blob_changes: [auth.ts sha1, session.ts sha1], + author: alice, + timestamp: ... + } +``` + +**Queries powered by semantic commits:** +- "Show me repos that recently improved OAuth" → search semantic commit messages +- "Who improved caching in the last week?" → semantic blame on semantic commits +- "What's breaking authentication?" → trace causality via semantic deltas + +**Implementation:** +- New `semantic_commits` table (parent, summary, added/removed/changed concepts) +- On each `index start`, compute deltas from previous semantic state +- `gitsema semantic-commits [--ref]` — show semantic commit log +- `gitsema semantic-blame [--file]` — blame by semantic change, not line change +- Gossip semantic commits between peers (lightweight—just metadata) + +**Benefits:** +- Concept-level causality (not just file-level) +- Efficient gossip (one semantic commit per Git commit) +- Enables root-cause analysis ("which commit introduced this pattern?") +- Works across repositories (shared semantic understanding) + +--- + +## Integration Points + +### MCP + +Extend `GUIDE_TOOLS` with federation operations: +```typescript +// Phase 155 +federation_info() // Get peer federation metadata +federation_peers() // List known peers + +// Phase 156 +federation_search() // Query across federated peers +route_query() // Get query routing decision + +// Phase 157 +sema_pack() // Create semantic packfile +sema_fetch() // Fetch packfile from peer + +// Phase 158 +semantic_commits() // Search semantic commits +semantic_blame() // Blame by semantic change +``` + +### HTTP API + +``` +// Phase 155 +GET /api/v1/federation/info +POST /api/v1/federation/peers + +// Phase 156 +POST /api/v1/federation/search +GET /api/v1/federation/route?query=... + +// Phase 157 +POST /api/v1/federation/pack +GET /api/v1/federation/fetch?packfile_id=... + +// Phase 158 +GET /api/v1/semantic-commits +POST /api/v1/semantic-blame +``` + +### CLI + +```bash +# Phase 154 (Semantic Refs) +gitsema sema push [--remote url] +gitsema sema pull [--remote url] +gitsema sema log [--graph] +gitsema sema diff + +# Phase 155 (Discovery) +gitsema federation info +gitsema federation peers + +# Phase 156 (Routing) +gitsema federation search [--peers url,url] +gitsema federation route + +# Phase 157 (Packfiles) +gitsema sema pack --query "topic" [--output file] +gitsema sema fetch --query "topic" [--remote url] +gitsema sema unpack [--input file] + +# Phase 158 (Semantic Commits) +gitsema semantic-commits [--ref] +gitsema semantic-blame [--file path] +``` + +--- + +## Design Constraints & Trade-offs + +### Immutability & Provenance + +**Constraint:** Semantic objects are immutable once created (like Git objects). + +- Enables **content-addressed deduplication** (same concept in two repos = same object ID) +- Enables **offline verification** (cryptographic signatures don't require live coordinator) +- Enables **long-term archival** (semantic objects don't rot) + +**Trade-off:** Can't update a semantic object if we find it was wrong. Instead, create a new semantic object with corrections + metadata linking to the original (like Git commits linking to parents). + +### Centralized vs. Peer-to-Peer + +**Recommendation:** P2P gossip with optional bootstrap registry. + +- **Centralized registry:** Simpler discovery, but single point of failure +- **P2P gossip:** Resilient, but requires bootstrap mechanism + +**Hybrid approach:** Support both. Repos can register with an optional registry and/or seed with known peers. + +### Semantic Versioning + +**Constraint:** Semantic objects are bound to their embedding model version. + +If Repo A indexes with `nomic-embed-text` v1.5 and Repo B uses v1.6, their centroids may not be directly comparable. + +**Solution:** Store `profile_version` in semantic objects + metadata. Routing code can choose: +1. Require exact model match (safer but less coverage) +2. Use approximate matching with a similarity threshold (faster but less reliable) +3. Re-embed query with both models and merge results (slow but most accurate) + +**Recommendation:** Default to (1) in Phase 154, add (3) in Phase 156, document (2) as a research direction. + +### Trust & Signatures + +**Recommendation:** Optional public-key signing for federated semantic objects. + +- Repos can sign semantic commits with their private key +- Peers verify signatures before incorporating remote semantic objects +- Enables **audit trails** (who claimed this semantic knowledge?) +- Enables **attribution** (give credit when one repo's embeddings improve another's) + +**Not required for Phase 154–157,** but design with extensibility in mind. Add as optional field in `sema_trees` + `semantic_objects` tables. + +--- + +## Phasing & Dependencies + +### Phase 154: Semantic Refs & Incremental Transfer + +**Goal:** Enable semantic index snapshots and incremental sync between repositories. + +**Dependencies:** None (builds on existing Layer 1 enrichment). + +**Deliverables:** +- `sema_refs` + `sema_trees` tables +- `gitsema sema push/pull/log/diff` CLI commands +- HTTP endpoints: `POST /api/v1/sema/push`, `GET /api/v1/sema/pull` +- MCP tools: `sema_push()`, `sema_pull()`, `sema_log()` +- Tests + docs + +**Effort:** ~2 weeks +**Risk:** Low (additive; doesn't change existing index format) + +--- + +### Phase 155: Federation Discovery & Gossip + +**Goal:** Enable repositories to discover and register with peers. + +**Dependencies:** Phase 154 (semantic refs). + +**Deliverables:** +- `federation_info()` HTTP endpoint + gossip protocol +- Lightweight peer registry (optional) +- `gitsema federation peers` + `gitsema federation info` CLI +- MCP tools: `federation_peers()`, `federation_info()` +- Tests + docs + +**Effort:** ~1.5 weeks +**Risk:** Low (read-only discovery; no data mutation) + +--- + +### Phase 156: Semantic Query Routing + +**Goal:** Route queries intelligently to the most relevant peers based on centroid similarity. + +**Dependencies:** Phases 154, 155. + +**Deliverables:** +- Semantic DHT routing table (`semanticDHT.ts`) +- `gitsema federation search` command +- HTTP endpoint: `POST /api/v1/federation/search` +- MCP tool: `federation_search()` +- Query result merging & ranking +- Tests + docs + +**Effort:** ~3 weeks +**Risk:** Medium (involves distributed ranking; needs careful testing across multiple repos) + +--- + +### Phase 157: Semantic Packfiles + +**Goal:** Enable efficient bulk transfer of semantic objects related to a query. + +**Dependencies:** Phases 154–156. + +**Deliverables:** +- Packfile format + serialization (`semanticPackfile.ts`) +- `gitsema sema pack/fetch/unpack` CLI commands +- HTTP endpoints: `POST /api/v1/federation/pack`, `GET /api/v1/federation/fetch` +- MCP tools: `sema_pack()`, `sema_fetch()` +- Decompression + integration with local index +- Tests + docs + +**Effort:** ~2.5 weeks +**Risk:** Medium (new binary format; needs robust error handling) + +--- + +### Phase 158: Semantic Commits & Deltas + +**Goal:** Track semantic changes per Git commit and enable concept-level causality analysis. + +**Dependencies:** Phases 154–157 (optional; can ship independently). + +**Deliverables:** +- `semantic_commits` table +- Delta computation on `index start` (compare to previous semantic state) +- `gitsema semantic-commits` + `gitsema semantic-blame` CLI commands +- HTTP endpoints: `GET /api/v1/semantic-commits`, `POST /api/v1/semantic-blame` +- MCP tools: `semantic_commits()`, `semantic_blame()` +- Tests + docs + +**Effort:** ~2 weeks +**Risk:** Medium (delta computation is complex; needs extensive testing on real repos) + +--- + +## Success Metrics + +### Phase 154 +- [ ] Semantic refs created and persisted correctly +- [ ] `gitsema sema push/pull` transfer only new objects (verify via packet capture) +- [ ] `gitsema sema diff` correctly shows conceptual changes between refs + +### Phase 155 +- [ ] Peer discovery works in a 3-repo test setup +- [ ] Gossip protocol propagates info within 5 rounds +- [ ] Peer registry (if used) stays consistent + +### Phase 156 +- [ ] Query routing selects relevant peers (validate against ground truth) +- [ ] Merged results from multiple peers are ranked correctly +- [ ] Latency improvement over naive broadcast (measure vs. Phase 155 baseline) + +### Phase 157 +- [ ] Packfiles reduce bandwidth by 50%+ over individual fetches +- [ ] Decompression is fast enough for interactive use (<500ms for typical query) +- [ ] Packfiles can be shared offline and imported correctly + +### Phase 158 +- [ ] Semantic commits are created on every indexing run +- [ ] Deltas correctly identify added/removed/changed concepts +- [ ] Semantic blame traces causality correctly across multiple commits + +--- + +## Risks & Mitigation + +| Risk | Severity | Mitigation | +|---|---|---| +| Semantic object size explosion | Medium | Quantization + gzip; lazy-load full vectors | +| Gossip storm / network saturation | High | Bounded peer lists; rate limiting on gossip | +| Model drift (embedding versions diverge) | Medium | Version pinning in semantic refs + re-embedding fallback | +| Byzantine peers (malicious semantic data) | Medium | Optional signing + verification; federation allow-lists | +| Stale peer info | Low | TTL on gossip + periodic refresh | +| Distributed query latency | Medium | Parallel fetches + early-exit thresholds | + +--- + +## Open Questions + +1. **Incentives:** Why would a repository publish semantic indexes publicly? (Privacy? Cost? Liability?) +2. **Standards:** Should this layer on a standard like SCITT or OpenVEX for supply-chain provenance? +3. **Scale:** How many peers can a single repository reasonably connect to? (Experiments needed.) +4. **Hybrid:** Could this integrate with a blockchain or IPFS for immutable semantic history? +5. **AI integration:** How does this layer interact with AI models? Can semantic commits improve grounding? + +--- + +## Relationship to Existing Work + +- **Git packfiles:** Semantic packfiles are the analog for meaning-transfer +- **BitTorrent / IPFS:** Inspiration for P2P discovery and content distribution +- **Supply-chain security (SLSA, SCITT):** Could layer on top for auditing semantic changes +- **Semantic versioning (semver):** Complements; semantic versioning applies to APIs, semantic federation applies to domain knowledge +- **Knowledge graphs / linked data:** Similar goals (distributed knowledge), different mechanisms (federation is pull-based; RDF is publish-all) + +--- + +## Files to Create / Modify + +**New files:** +- `docs/design/semantic-federation.md` (this document) +- `src/core/federation/semanticRefs.ts` (Phase 154) +- `src/core/federation/semanticDHT.ts` (Phase 156) +- `src/core/federation/semanticPackfile.ts` (Phase 157) +- `src/core/federation/semanticCommits.ts` (Phase 158) +- `src/server/routes/federation/` (HTTP endpoints) +- `src/mcp/tools/federation.ts` (MCP tools) +- Tests: `tests/federation/` + +**Existing files to update:** +- `docs/PLAN.md` (add phases 154–158) +- `docs/features.md` (add federation section) +- `src/core/db/schema.ts` (extend with federation tables) +- `src/core/db/sqlite.ts` (migration for federation schema) +- `src/mcp/tools/` (register new federation tools) + +--- + +## Next Steps + +1. **Feedback round:** Get team consensus on the three-layer model and phasing +2. **Database schema:** Design `sema_refs`, `sema_trees`, `semantic_commits` tables + migrations +3. **Prototype:** Phase 154 proof-of-concept (semantic refs sync between two repos) +4. **Write phases:** Draft detailed PLAN.md entries for phases 154–158 +5. **Begin Phase 154:** Start implementation once schema approved diff --git a/docs/feature-ideas.md b/docs/feature-ideas.md index 814150a..bb91476 100644 --- a/docs/feature-ideas.md +++ b/docs/feature-ideas.md @@ -2,15 +2,21 @@ This document tracks upcoming feature ideas that are **not yet in active development** (not in `PLAN.md`) and haven't been **fully designed** (no design file). It's a staging area for "what now?" questions and medium-term product direction. -**Last updated:** 2026-07-02 (added the remote multi-turn `guide` HTTP session idea, deferred from Phase 145's `--lens`/session scope) +**Last updated:** 2026-07-08 (semantic federation now fully designed → PLAN.md Phases 154–158) **Audience:** Developers considering next phases; product planning -> **Note:** As of this update, the LSP/MCP remote-delegation foundation this -> document used to describe as undesigned (remote delegation, WebSocket -> transport, structural navigation, diagnostics/code lens, hover enrichment) -> has shipped as Phases 113–117 in `docs/PLAN.md` (see the "LSP & MCP -> Fleshout Track"). Those sections were removed from here; this file now -> tracks only what's genuinely still just an idea. +> **Note 1:** As of 2026-07-02, the LSP/MCP remote-delegation foundation +> previously described as undesigned (remote delegation, WebSocket transport, +> structural navigation, diagnostics/code lens, hover enrichment) has shipped +> as Phases 113–117 in `docs/PLAN.md` (see the "LSP & MCP Fleshout Track"). +> Those sections were removed from here; this file now tracks only what's +> genuinely still just an idea. +> +> **Note 2:** As of 2026-07-08, semantic federation (distributed semantic +> knowledge, peer-to-peer query routing, semantic packfiles) is now fully +> designed in `docs/design/semantic-federation.md` and scheduled as Phases +> 154–158 in `docs/PLAN.md`. The design is comprehensive; implementation +> begins after Phase 153 completes. --- From a2d8dc0172953f348516efd4d945690e6193dea7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 19:18:53 +0000 Subject: [PATCH 2/2] =?UTF-8?q?Design=20refinement:=20Semantic=20Federatio?= =?UTF-8?q?n=20clarifications=20(Phases=20154=E2=80=93158)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address three critical design gaps based on feedback: 1. CHUNK-LEVEL GRANULARITY - Semantic objects wrap chunks, not blobs - Enables fine-grained deduplication (identical snippets = one object) - Summaries are specific ("JWT validation in session.ts:42–87") not broad ("auth module") - Updated database schema + design doc 2. CROSS-MODEL FEDERATION - Different repos use different models (nomic-embed-text vs text-embedding-3-small) - Vectors in different spaces are not directly comparable - Phase 154–158: Require model match, reject cross-model federation with warning - Phase 159: Plan re-embedding strategy for cross-model support - Storage: profile_version + model_dimensions tracked on every semantic object 3. FEDERATION SEARCH STORAGE STRATEGY - `--cache` (default): Store in session-scoped federation_cache table - `--import`: Merge into permanent semantic_objects (mark as federation_imported) - `--no-cache`: Transient (show results, don't store) - Users control federation integration: pure federation vs hybrid vs integrated Phase 154 revisions: - Effort: ~2 weeks → ~2.5 weeks (semantic enrichment pipeline) - Risk: Low → Medium (LLM integration testing) - Chunk-level granularity + model versioning explicit Phase 156 revisions: - Storage modes fully specified (cache table schema, TTL, metadata tracking) - Cross-model peers filtered out (same profile_version only) - Effort: ~3 weeks → ~3.5 weeks (caching adds state management) - Detailed test scenarios for storage modes + cross-model rejection Updated docs: - docs/design/semantic-federation.md — chunk design, cross-model constraint, storage strategy - docs/PLAN.md — phases 154 + 156 with gaps filled Co-Authored-By: User feedback on semantic federation design Claude-Session: https://claude.ai/code/session_013HWxjJmmohRPvkgvTBTKUC --- docs/PLAN.md | 154 ++++++++++++++++++----------- docs/design/semantic-federation.md | 126 ++++++++++++++++++++--- 2 files changed, 210 insertions(+), 70 deletions(-) diff --git a/docs/PLAN.md b/docs/PLAN.md index 349fdfd..e1ded11 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -6316,22 +6316,28 @@ URL-guard helper), `src/server/routes/{narrator,guide}.ts`, **Goal:** Enable semantic index snapshots and incremental sync between repositories, forming the foundation of federated semantic knowledge exchange. +**Key design decisions:** +- **Chunk-level granularity:** Semantic objects wrap *chunks*, not whole blobs. Enables fine-grained deduplication and specific summaries. +- **Model versioning:** Every semantic object stores `profile_version` (e.g. "text-embedding-3-small:1.0") and `model_dimensions`. Cross-model federation rejected in Phase 154–158 (deferred to Phase 159). +- **Storage:** Semantic objects are permanent, content-addressed, immutable (like Git objects). + **Scope:** -1. **Database schema:** Add `sema_refs` and `sema_trees` tables (v33 migration): - - `sema_refs(name, target_sema_tree_hash, blob_count, created_at, updated_at)` - - `sema_trees(tree_hash, parent_tree_hash, entries_jsonl, metadata_json)` - - `semantic_objects(object_hash, blob_hash, embedding, summary, keywords, language, entities, profile_version, created_at)` +1. **Database schema (v33 migration):** + - New `semantic_objects` table: `(object_hash, chunk_hash, blob_hash, embedding, summary, keywords, language, entities, structural_refs, profile_version, model_dimensions, signer, created_at)` + - New `sema_refs` table: `(name, target_sema_tree_hash, blob_count, created_at, updated_at)` + - New `sema_trees` table: `(tree_hash, parent_tree_hash, entries_jsonl, metadata_json)` — tracks semantic snapshot lineage + - Extend `chunk_embeddings`: add `summary`, `keywords`, `language` columns (nullable for backward compat) 2. **CLI commands:** - - `gitsema sema push [--remote url] [--branch name]` — create semantic ref, push new objects - - `gitsema sema pull [--remote url] [--branch name]` — fetch semantic ref + missing objects - - `gitsema sema log [--ref name] [--graph]` — show semantic ref history - - `gitsema sema diff [--format text|json|html]` — semantic diff (high-level concept changes) + - `gitsema sema push [--remote url] [--branch name]` — create semantic ref, push new semantic objects to remote + - `gitsema sema pull [--remote url] [--branch name]` — fetch semantic ref + missing semantic objects from remote + - `gitsema sema log [--ref name] [--graph]` — show semantic ref history (like `git log`) + - `gitsema sema diff [--format text|json|html]` — semantic diff (high-level concept changes between snapshots) 3. **HTTP API:** - - `POST /api/v1/sema/push` — accept semantic tree + objects - - `GET /api/v1/sema/pull?ref=` — fetch semantic tree + packfile + - `POST /api/v1/sema/push` — accept semantic tree + objects for storage + - `GET /api/v1/sema/pull?ref=` — fetch semantic tree + packfile of missing objects - `GET /api/v1/sema/log` — semantic ref history - `POST /api/v1/sema/diff` — compute semantic diff between two refs @@ -6341,30 +6347,39 @@ URL-guard helper), `src/server/routes/{narrator,guide}.ts`, - `sema_log()` — list semantic ref history - `sema_diff()` — semantic diff between refs -5. **Feature:** Semantic object enrichment (Layer 1 enhancement from federation design) - - Extend `embeddings` table with `summary`, `keywords`, `language` columns (nullable for backward compat) - - Populate via optional `--semantic-enrich` flag on `gitsema index start` (calls narrator LLM to generate summaries) - - MCP/HTTP tools that return blobs now include summaries (if available) for richer context +5. **Feature:** Semantic object enrichment (Layer 1 enhancement) + - Optional `--semantic-enrich` flag on `gitsema index start` (calls narrator LLM to generate `summary`, `keywords`, `entities` per chunk) + - Extends existing chunking pipeline: `chunk → embed → [enrich] → store` + - Backward compat: old indexes without semantic_objects table still index normally (opt-in feature) + - Summary generation: reuse narrator infrastructure from Phase 56 (`buildSemanticSummary()` or similar) 6. **Tests:** - - Two-repo sync scenario: push from Repo A, pull into Repo B, verify object deduplication - - Semantic diff correctness: track high-level concept changes between refs - - Backward compat: old indexes without semantic_objects table still index normally + - Schema: v32 → v33 migration applies cleanly; new tables created + - Two-repo push/pull scenario: push from Repo A, pull into Repo B, verify object deduplication by `object_hash` + - Semantic diff: track concept-level changes between refs (added/removed/changed summaries) + - Backward compat: old indexes without semantic_objects still index; queries still work (semantic objects lazy-loaded) + - Cross-model rejection: `sema pull` from a peer with different model → error with helpful message + - Enrichment: `--semantic-enrich` generates summaries; summaries appear in MCP/HTTP results **Acceptance criteria:** -- Semantic refs created correctly and stored persistently -- `gitsema sema push` transfers only new semantic objects (bandwidth reduction verified) -- `gitsema sema pull` correctly reconstructs remote index state -- `gitsema sema diff` shows concept-level changes (not just vector differences) -- All commands work via HTTP + MCP in addition to CLI +- Semantic refs created, stored persistently, survive session restarts +- `gitsema sema push` transfers only new semantic objects (content-addressed dedup verified via hash) +- `gitsema sema pull` correctly reconstructs remote semantic state +- `gitsema sema diff` shows high-level concept changes (not just raw vector diffs) +- `--semantic-enrich` produces meaningful summaries (manual QA on real repo) +- Cross-model federation rejected with clear error message +- All commands work via HTTP + MCP + CLI (parity) - `pnpm build && pnpm test` clean; changesets added -**Effort:** ~2 weeks -**Risk:** Low (new tables, no breaking changes; backward compat preserved via nullable columns) +**Effort:** ~2.5 weeks (schema migration + enrichment pipeline adds complexity) +**Risk:** Medium (new tables + semantic enrichment pipeline; needs LLM integration testing) -**Files (anticipated):** `src/core/db/schema.ts`, `src/core/db/sqlite.ts` (v33 migration), `src/core/federation/semanticRefs.ts`, `src/cli/commands/sema.ts`, `src/server/routes/federation/sema.ts`, `src/mcp/tools/federation.ts`, tests, `CLAUDE.md`, `.changeset/` +**Files (anticipated):** `src/core/db/schema.ts`, `src/core/db/sqlite.ts` (v33 migration), `src/core/federation/semanticRefs.ts`, `src/core/narrator/semanticEnrichment.ts`, `src/cli/commands/sema.ts`, `src/server/routes/federation/sema.ts`, `src/mcp/tools/federation.ts`, tests, `CLAUDE.md`, `.changeset/` -**Deferred:** Semantic commit deltas (see Phase 158). This phase focuses on *static* semantic state snapshots; Phase 158 adds *dynamic* change tracking. +**Design decisions deferred to Phase 159:** +- Cross-model federation (re-embedding queries for different model spaces) +- Trust/signatures (cryptographic verification of semantic objects) +- Semantic commit deltas (see Phase 158; focuses on static state here) --- @@ -6422,54 +6437,83 @@ URL-guard helper), `src/server/routes/{narrator,guide}.ts`, **Design:** Covered in `docs/design/semantic-federation.md`. -**Goal:** Route queries intelligently to the most relevant peers based on semantic similarity, reducing broadcast queries and latency. +**Goal:** Route queries intelligently to the most relevant peers based on semantic similarity, reducing broadcast queries and enabling users to choose how federated results are stored. + +**Key design decisions:** +- **Storage modes:** Support transient (no storage), cached (session scope), and imported (permanent) federation search results. +- **Same-model only:** Only query peers with matching `profile_version`; cross-model federation deferred to Phase 159. +- **Three-signal ranking:** Reuse existing three-signal ranking (Phase 41) for merged cross-peer results. **Scope:** -1. **Semantic DHT routing table:** - - `semanticDHT.ts`: Build routing table from peer gossip + centroid embeddings - - For each known peer, store its semantic topics (cluster centroids + labels) - - On each query, embed the query and compute similarity to all peer centroids - - Select top-N peers (default N=5, configurable via `--federation-peers `) above a similarity threshold (default 0.3) +1. **Semantic DHT routing table (`semanticDHT.ts`):** + - Build routing table from peer gossip + centroid embeddings (from Phase 155) + - For each known peer, store semantic topics (cluster centroids + labels) + - On each query, embed query and compute similarity to all peer centroids + - Select top-N peers (default N=5, configurable `--federation-peers `) above threshold (default 0.3) + - Filter by model match: only include peers with same `profile_version` as local index 2. **Federated search CLI:** - - `gitsema federation search [--peers | --auto] [--top k] [--format text|json|html]` - - `--auto` mode: use gossip-discovered peers; `--peers` mode: explicit peer list - - Fetch results from selected peers in parallel - - Merge results by combining vector scores + freshness (prefer recent embeddings) - - Rank by three-signal ranking (existing Phase 41 logic) + - `gitsema federation search [--peers | --auto] [--top k] [--cache|--import|--no-cache] [--format text|json|html]` + - `--auto`: use gossip-discovered peers; `--peers`: explicit peer URLs + - Storage modes (mutually exclusive): + - `--cache` (default): store in temp `federation_cache` table (session scope), TTL = session + - `--import`: merge into permanent `semantic_objects` table (mark with `origin: "federation_imported"`) + - `--no-cache`: transient (show results, don't store) + - Fetch results from selected peers in parallel with timeout (default 5s/peer) + - Merge by `chunk_hash` (remove duplicates) + - Re-rank using three-signal ranking (vector similarity, recency, path relevance) + - Include provenance: show peer URL + model version for each result 3. **HTTP API:** - - `POST /api/v1/federation/search` — accept query + optional peer hints - - `GET /api/v1/federation/route?query=` — return which peers would be queried (decision transparency) - - Concurrent fetches with timeout per peer (default 5s) + - `POST /api/v1/federation/search` — accept query, storage mode (`cache|import|transient`), peer hints + - `GET /api/v1/federation/route?query=` — return routing decision (which peers selected + why) + - Streaming results as peers respond (don't wait for slowest) + - `X-Peer-Info` header on each result: `{url: "...", model: "...", response_time: ...}` 4. **MCP tools:** - - `federation_search()` — query federated peers - - `federation_route()` — show routing decision (which peers would be queried) + - `federation_search()` — query federated peers (supports storage mode selection) + - `federation_route()` — show routing decision (peers selected + reasoning) 5. **Result merging & ranking:** - - Combine results from multiple peers (remove duplicates by blob hash) - - Re-rank using three-signal model (vector similarity, recency, path relevance) - - Include provenance: show which peer each result came from + - De-duplicate by `chunk_hash` (if same chunk from multiple peers, keep highest-confidence result) + - Combine provenance metadata: `{peer_url: "...", peer_model: "...", local_origin: "..."}` + - Re-rank using three-signal: vector similarity (70%) + recency (20%) + path relevance (10%) + - Fallback: if peer unresponsive, drop and continue (fail-open) -6. **Tests:** - - 3-repo network: query "authentication", verify only auth-heavy repos are selected - - Result ranking: merge results from 2 peers, verify no regressions vs. single-repo search - - Timeout handling: simulate slow peer, verify others return quickly + slow peer times out gracefully +6. **Metadata tracking:** + - Extend `semantic_objects` to track `origin` (values: "local", "federation_temp", "federation_imported") + `peer_url` + `import_date` + - Cache table `federation_cache`: same schema as `semantic_objects`, but session-scoped (dropped on exit) + +7. **Tests:** + - Schema: `federation_cache` table creates/drops correctly + - 3-repo network: query "authentication", verify only auth-heavy repos selected + - Routing: `--federation-peers 3` limits to top-3 by similarity + - Storage modes: `--cache` doesn't pollute local index; `--import` persists; `--no-cache` returns 0 cached results + - Cross-model rejection: query Repo B (different model) → skipped with warning + - Result ranking: merge from 2 peers, verify no regressions vs. single-repo ranking + - Timeout: slow peer (latency > 5s) doesn't block others; partial results returned + - Provenance: results show peer URL + model; CLI and HTTP match **Acceptance criteria:** -- `gitsema federation search ` returns ranked results from multiple peers -- Routing logic selects relevant peers (validate against ground truth) -- Merged results correctly de-duplicate by blob hash +- `gitsema federation search ` returns ranked results from multiple (same-model) peers +- Routing logic selects relevant peers; `--federation-peers ` respected +- Merged results de-duplicated by chunk_hash, re-ranked correctly +- `--cache` (default) doesn't persist results; `--import` does; `--no-cache` is transient +- Cross-model peers rejected with warning (not queried) - Peer timeout doesn't block final result (fail-open) - HTTP + MCP parity with CLI +- Provenance metadata visible in all formats (text, json, html) - `pnpm build && pnpm test` clean -**Effort:** ~3 weeks -**Risk:** Medium (distributed ranking; needs careful testing + validation on realistic multi-repo setup) +**Effort:** ~3.5 weeks (DHT + caching + storage modes add complexity) +**Risk:** Medium (distributed ranking; needs careful testing on multi-repo setup; caching adds state management) + +**Files (anticipated):** `src/core/db/schema.ts` (federation_cache table), `src/core/federation/semanticDHT.ts`, `src/core/federation/federatedSearch.ts`, `src/cli/commands/federation.ts` (extended), `src/server/routes/federation/search.ts`, `src/mcp/tools/federation.ts` (extended), tests, `.changeset/` -**Files (anticipated):** `src/core/federation/semanticDHT.ts`, `src/core/federation/federatedSearch.ts`, `src/cli/commands/federation.ts` (extended), `src/server/routes/federation/search.ts`, `src/mcp/tools/federation.ts` (extended), tests, `.changeset/` +**Design decisions deferred to Phase 159:** +- Cross-model federation (re-embedding for different model spaces) +- Peer trust/signatures (verifying peer semantic objects) --- diff --git a/docs/design/semantic-federation.md b/docs/design/semantic-federation.md index 6a56650..77db5e8 100644 --- a/docs/design/semantic-federation.md +++ b/docs/design/semantic-federation.md @@ -43,30 +43,61 @@ The core insight: **Git distributes bytes. Gitsema distributes meaning. Federati ### Layer 1: Semantic Objects (✅ ~80% complete, needs enrichment) -**Current state:** Blobs are stored with embeddings. -**What we add:** Rich semantic metadata envelopes. +**Current state:** Blobs are stored with embeddings; large blobs are chunked with per-chunk embeddings. +**What we add:** Rich semantic metadata envelopes at chunk level. ``` -blob (SHA-1) +semantic_object (content-addressed, not blob-addressed) +├── chunk_hash (content-addressed, stable across repos) +├── blob_hash (parent file reference) ├── embedding (Float32 vector) ├── summary (LLM-generated snippet) ├── keywords (extracted terms) ├── language (source code, prose, config, etc.) ├── entities (author, date, module path) -├── structural_refs (imports/calls/extends, already tracked) -├── references (backward pointers: which blobs cite this?) +├── structural_refs (imports/calls/extends, if chunk-level) +├── references (backward pointers: which chunks cite this?) +├── line_range (if chunked: lines 42–87 in blob) ├── timestamp (when first indexed) -├── profile (embedding model name + version) +├── profile_version (embedding model + version, e.g. "text-embedding-3-small:1.0") +├── model_dimensions (vector dimensionality, e.g. 1536) ├── signer (public key, optional) -└── content_hash (hash of blob content for cache-busting) +└── content_hash (hash of chunk content for cache-busting) ``` -**Database changes:** Extend `embeddings` and `blob_fts` tables with `summary`, `keywords`, `language`, `entities` columns; optionally `signer` and `profile_version` for provenance. +**Key design decision: Chunk-level granularity** +- Semantic objects wrap **chunks**, not whole blobs +- Enables fine-grained deduplication (identical code snippets across repos = one object) +- Summaries are specific ("JWT validation handler" not broad "auth module") +- Matches current gitsema chunking strategy (whole-file, function, fixed windows) + +**Database changes:** New `semantic_objects` table + extend `chunk_embeddings` table with `summary`, `keywords`, `language`, `entities` columns. + +```sql +CREATE TABLE semantic_objects ( + object_hash TEXT PRIMARY KEY, -- content-addressed + chunk_hash TEXT NOT NULL, -- reference to chunk + blob_hash TEXT, -- reference to parent blob + embedding BLOB, -- Float32 vector + summary TEXT, -- LLM summary + keywords TEXT, -- JSON array: ["jwt", "token", ...] + language TEXT, -- "code" | "prose" | "config" + entities TEXT, -- JSON: {authors: [...], dates: [...]} + structural_refs TEXT, -- JSON: {imports: [...], defines: [...]} + profile_version TEXT, -- "text-embedding-3-small:1.0" + model_dimensions INTEGER, -- e.g. 1536 + signer TEXT, -- optional Ed25519 public key + created_at DATETIME, + updated_at DATETIME +); +``` **Benefits:** - Richer semantic queries without re-fetching blobs - Gossip protocol can propagate summaries without full embeddings - Semantic diffs become composable (diff summaries instead of vectors) +- Cross-repo deduplication at chunk level (identical implementations = one object) +- Model version tracking enables cross-model federation decisions (Phase 159) --- @@ -225,6 +256,59 @@ git sema fetch --query "caching" - **Deduplication:** Shared topics compress well across repos - **Offline use:** Packfiles can be shared via email, S3, etc. +#### Phase C.5: Federation Search Storage Strategy + +**Problem:** When you query federated peers via `gitsema federation search`, what happens to the results? + +Option 1: **Transient** — results displayed, not stored +Option 2: **Cached** — results stored in a session-scoped table, available for re-query +Option 3: **Imported** — results merged into your permanent local semantic_objects table + +**Design:** Support all three via flags, with caching as default. + +```bash +# Option 1: Transient (no storage) +gitsema federation search "JWT" --no-cache + → queries peers, shows results, discards + +# Option 2: Cached (session scope) [DEFAULT] +gitsema federation search "JWT" --cache + → queries peers, stores in temp semantic_objects (in-memory or temp DB) + → re-running same query hits cache (no peer call) + → cache expires on session close + +# Option 3: Imported (permanent) +gitsema federation search "JWT" --import + → queries peers + → pulls semantic objects into YOUR permanent index + → future `gitsema search "JWT"` hits local index (no peer call) + → now `gitsema first-seen` can trace to imported chunks +``` + +**Storage tables:** +```sql +-- Permanent semantic objects (yours) +semantic_objects (scope: local) + +-- Session cache (federation results) +federation_cache (scope: temp, TTL: session) + +-- Imported federation results (merged into semantic_objects) +-- (same table, marked with origin metadata) +``` + +**Metadata tracking:** +- Permanent objects: `origin: "local"` +- Cached objects: `origin: "federation_temp"`, `peer_url: "https://..."` +- Imported objects: `origin: "federation_imported"`, `peer_url: "https://..."`, `import_date` + +**Benefit:** Users can control federation integration: +- **Pure federation** (--no-cache): use federation as a search engine, keep local index clean +- **Hybrid federation** (--cache, default): federation as a cache layer, no permanent changes +- **Integrated federation** (--import): absorb peer insights into your index for future local searches + +--- + #### Phase D: Semantic Commits & Deltas (Phase 158) **Problem:** When a repository changes, how do peers know what concepts changed? @@ -372,18 +456,30 @@ gitsema semantic-blame [--file path] **Hybrid approach:** Support both. Repos can register with an optional registry and/or seed with known peers. -### Semantic Versioning +### Semantic Versioning & Cross-Model Federation **Constraint:** Semantic objects are bound to their embedding model version. -If Repo A indexes with `nomic-embed-text` v1.5 and Repo B uses v1.6, their centroids may not be directly comparable. +If Repo A indexes with `nomic-embed-text` v1.5 and Repo B uses `text-embedding-3-small` v1.0, their vectors live in different vector spaces and are not directly comparable (different dimensions, different semantic structure). + +**Problem:** In a federated network, repos will use different models (different hardware, different requirements, model updates over time). How do we federate across model boundaries? + +**Solutions (by phase):** + +| Phase | Approach | Cost | Accuracy | Notes | +|---|---|---|---|---| +| **154–156** | Require model match | Low | High | Repo A only queries Repo B if same `profile_version` | +| **157** | Cache same-model results | Medium | High | Packfiles tagged by model; only import if model matches | +| **159** *(future)* | Re-embed on query | High | High | Query embedded with each peer's model, results merged | +| **162** *(future)* | Cross-space similarity | Research | Unknown | Use alignment layers / model mapping (open research) | -**Solution:** Store `profile_version` in semantic objects + metadata. Routing code can choose: -1. Require exact model match (safer but less coverage) -2. Use approximate matching with a similarity threshold (faster but less reliable) -3. Re-embed query with both models and merge results (slow but most accurate) +**Recommendation for Phases 154–158:** +- Store `profile_version` + `model_dimensions` in every semantic object +- Phase 154–156: Reject cross-model federation (log warning: "Repo B uses text-embedding-3-small, you use nomic-embed-text; skipping") +- Phase 157: Same-model packfiles only +- Phase 159 design doc: Plan re-embedding strategy for Phase 160+ -**Recommendation:** Default to (1) in Phase 154, add (3) in Phase 156, document (2) as a research direction. +**Implementation detail:** Add `GITSEMA_FEDERATION_ALLOW_CROSS_MODEL` env var (default false) for experimental cross-model federation; if enabled, log a warning and attempt best-effort matching (documented as unreliable). ### Trust & Signatures