From 751484331b1433dd950d7cc934aa71fce014c137 Mon Sep 17 00:00:00 2001 From: Sarthak Agarwal Date: Wed, 22 Jul 2026 20:15:49 +0530 Subject: [PATCH 1/6] Add connected knowledge cache design doc --- .../local-connected-knowledge-cache-design.md | 1399 +++++++++++++++++ 1 file changed, 1399 insertions(+) create mode 100644 docs/local-connected-knowledge-cache-design.md diff --git a/docs/local-connected-knowledge-cache-design.md b/docs/local-connected-knowledge-cache-design.md new file mode 100644 index 00000000..f35ef1eb --- /dev/null +++ b/docs/local-connected-knowledge-cache-design.md @@ -0,0 +1,1399 @@ +# Local Connected Knowledge Cache Design + +Draft date: July 22, 2026 + +Status: design draft for iteration, not an implementation commitment + +## Executive Summary + +`loc locate ` is an important primitive, but it cannot be the main product +experience. In real workflows, the agent often does not know the exact Notion +URL, Google Doc URL, Gmail thread, Granola note, Slack thread, or issue link. +The agent usually starts with a task: + +```text +Prepare today's scrum update. +Review launch readiness. +Find the customer context behind this bug. +Update the onboarding docs from the latest implementation. +``` + +Locality should therefore become a local connected knowledge cache. Connectors +feed a durable local index and graph. Agents search, inspect, and edit through +files. Hydration happens lazily by relevance, not accidentally by directory +recursion. Sync back remains conservative and reviewable. + +The goal is not to copy Notion search, Obsidian, Glean, or MCP. The goal is to +own a different surface: + +```text +Company knowledge as a local, permission-aware, source-backed filesystem for agents. +``` + +This is the moat: Locality can combine source identity, local files, durable +sync state, freshness, pending edits, push journals, remote provenance, and +agent-readable paths into one system. Generic MCP tools can fetch from apps, but +they do not naturally provide a persistent local working memory that agents can +grep, diff, edit, review, and safely sync back. + +## Strategic Thesis + +The current product is already more than a mount tool, but the user experience +can still feel mount-first: + +- connect source; +- mount source; +- browse files; +- paste URL; +- pull or push. + +That is useful when the user knows exactly where the work lives. The larger +opportunity is workflow-first: + +- user or agent states an objective; +- Locality finds the relevant knowledge across connected sources; +- Locality returns a ranked evidence pack with local paths; +- Locality hydrates only what is useful; +- the agent works in files; +- Locality syncs safe changes back through Review Center and Live Mode. + +In this model, `loc locate --offline ` is a fast identity resolver. +`loc resolve ` is a remote repair path. `loc pull ` is precise body +hydration. `loc context build` is the main agent-facing primitive when the URL is +unknown. + +## Product Principles + +### 1. Filesystem First, Search Assisted + +The winning agent interface remains files, folders, grep, diffs, and normal +editor operations. Search and graph retrieval should produce file paths and +context packs, not replace the filesystem. + +### 2. Local First, Remote Second + +The normal query path should be local and fast. Remote APIs should feed the +cache through scheduled observation, explicit resolve, bounded hydration, and +source-specific sync jobs. Remote APIs should not sit inside every keystroke or +every agent search step. + +### 3. Metadata Before Bodies + +Locality should index cheap metadata broadly and hydrate bodies selectively. +Metadata gives enough signal to rank candidates, explain coverage, and decide +where to spend remote API budget. + +### 4. Relevance Before Recursion + +Directory recursion is a bad default for context. A directory may contain 5 +pages or 5,000 pages. `loc pull ` should be explicit about recursive +policy. Context hydration should rank first, then hydrate only the top useful +items and their most relevant neighbors. + +### 5. Identity And Path Are Different + +Remote identity must be stable even when a visible path changes. Paths are the +human and agent interface. Remote IDs, source URLs, connector IDs, and access +scopes are the durable identity layer. + +### 6. Freshness Must Be Visible + +A good result is not just relevant. It must be honest: + +- ready; +- metadata only; +- stale; +- remote changed; +- disconnected; +- pending local changes; +- conflicted; +- read-only; +- not hydrated yet. + +Agents should not mistake stale or metadata-only context for fresh body content. + +### 7. Source Ownership Stays Clear + +Derived knowledge views should not blur write ownership. The source-backed tree +is writable according to connector rules. Generated knowledge views and context +packs are read-only or explicitly staged until write semantics are clear. + +## The Missing URL Problem + +The product cannot expect agents to know URLs. The user often does not know the +URL either. The system needs to answer: + +```text +Given this goal and this user/workspace, what local knowledge is likely useful? +``` + +That requires a retrieval layer with signals beyond exact ID lookup: + +- title, path, and body lexical matches; +- source URLs and aliases; +- current project or mount; +- recently opened or edited files; +- Review Center pending files; +- Live Mode tracked files; +- recent git commits and changed paths; +- people mentioned in the task; +- dates and time windows; +- document links and backlinks; +- source-specific metadata such as Notion properties, Gmail participants, + Granola attendees, Google Docs owners, Slack channels, issue labels, and + pull request authors; +- graph neighbors around high-confidence matches. + +The UX should let the agent start from a goal, not from a URL. + +## Core Concepts + +### Knowledge Item + +A connector-neutral record for a source-backed object. + +Examples: + +- Notion page; +- Notion database row; +- Google Doc; +- Gmail thread; +- Gmail message; +- Granola note; +- Slack thread; +- Linear issue; +- GitHub pull request; +- local mounted Markdown file. + +Important fields: + +```text +item_id +connector +connection_id +mount_id +remote_id +source_url +kind +title +projected_path +absolute_path +hydration_state +access_state +remote_version +observed_at +indexed_at +updated_at +created_at +actors +source_metadata_json +``` + +### Knowledge Chunk + +A searchable unit inside a knowledge item. + +Examples: + +- page heading section; +- Notion block range; +- Google Docs paragraph group; +- Gmail message body; +- Granola transcript segment; +- code-linked launch note section. + +Important fields: + +```text +chunk_id +item_id +source_block_id +chunk_kind +heading_path +text +frontmatter_json +content_hash +start_offset +end_offset +indexed_at +``` + +### Knowledge Edge + +A relationship between items or chunks. + +Examples: + +- parent-child; +- contains; +- links-to; +- mentioned-by; +- same-person; +- same-project; +- same-date; +- attached-to; +- duplicate-url; +- references-commit; +- references-pr; +- blocks; +- depends-on; +- discussed-in. + +Important fields: + +```text +edge_id +from_item_id +from_chunk_id +to_item_id +to_chunk_id +to_external_ref +edge_type +confidence +source +created_at +``` + +### Context Pack + +A generated, task-specific evidence bundle for an agent. + +The context pack is not a new source of truth. It is a local workspace artifact +that explains what Locality found and why. + +Suggested shape: + +```text +.locality/context/ + 2026-07-22-scrum-update/ + context.md + manifest.json + evidence/ + notion-engineering-wiki-standups.md + granola-product-sync.md + gmail-customer-thread.md + paths.txt + freshness.md + missing-access.md + hydration-plan.json + trace.jsonl +``` + +### Hydration Plan + +A deterministic plan for which metadata-only or stale items should be hydrated +for a workflow. + +The plan should be explicit: + +```text +hydrate_policy: none | metadata | top-k | neighbors | bounded-recursive +max_items +max_depth +max_remote_calls +max_wall_time_ms +source_budgets +reasons +``` + +### Knowledge Cache + +The durable local store composed of: + +- source metadata; +- indexed chunks; +- graph edges; +- freshness state; +- activity signals; +- hydration plans; +- context packs; +- sync and review state. + +This cache is rebuildable where possible, but it must preserve durable source +identity and pending local work. + +## Proposed User And Agent Workflow + +### Workflow: Agent Does Not Know The URL + +Task: + +```text +Generate today's engineering update from recent git changes and relevant company context. +``` + +Expected path: + +1. Agent calls `loc context build` with the task, repository path, and time + window. +2. Locality searches the local knowledge cache first. +3. Locality ranks candidates by text, recency, graph signals, activity, source + confidence, and workflow hints. +4. Locality hydrates only top candidates that need body content, within a bounded + budget. +5. Locality writes a context pack with file paths, snippets, freshness labels, + and missing-access notes. +6. Agent reads `context.md`, `manifest.json`, and the referenced mounted files. +7. Agent writes the output to the correct mounted `page.md` or creates a draft. +8. Live Mode or Review Center handles safe sync back. + +Example command: + +```bash +loc context build \ + --goal "Generate today's engineering update" \ + --repo /Users/me/orgs/research/afs \ + --since 24h \ + --sources notion,granola,gmail,google-docs \ + --hydrate top-k \ + --max-items 25 \ + --out .locality/context/today-engineering-update +``` + +### Workflow: User Gives A URL + +Task: + +```text +Open this Notion page and update the launch risks. +``` + +Expected path: + +1. `loc locate --offline ` checks the local index only. +2. If found and fresh enough, return the local file path immediately. +3. If missing or path looks stale, `loc resolve ` repairs remote identity + and parent path metadata. +4. `loc pull ` hydrates exactly the target page. +5. Agent edits the file. +6. Review Center or Live Mode syncs back. + +This keeps locate, resolve, and hydration measurable and predictable. + +## Command Surface + +### `loc locate --offline ` + +Purpose: + +- fast local resolution; +- no remote connector calls; +- safe for agent loops and desktop typeahead. + +Behavior: + +- returns local file path when known; +- returns metadata-only status when known but not hydrated; +- returns no match when local cache does not know it; +- never performs remote parent repair. + +### `loc resolve ` + +Purpose: + +- remote identity and parent/path repair; +- source-specific URL handling; +- explicit API work. + +Behavior: + +- contacts the connector; +- validates access; +- saves parent/path metadata; +- does not hydrate body content unless asked; +- emits trace spans for remote calls and repaired path entries. + +### `loc pull ` + +Purpose: + +- hydrate exactly one entity body; +- repair missing assets for that entity; +- preserve local dirty or conflicted state. + +Behavior: + +- no recursive descendant hydration by default; +- no mount-wide enumeration; +- safe and measurable. + +### `loc pull ` + +Purpose: + +- explicit directory policy. + +Required flags: + +```bash +loc pull --children +loc pull --recursive --max-depth 2 --max-items 50 +loc pull --hydrate-policy metadata +loc pull --hydrate-policy top-k --query "launch readiness" +``` + +Directory pulls should not accidentally become a broad recursive hydration +benchmark. + +### `loc search ` + +Purpose: + +- search the local knowledge cache. + +Behavior: + +- local by default; +- returns snippets and matched fields; +- returns safety and freshness labels; +- can filter by connector, mount, kind, person, project, date, and status; +- can schedule deeper search through a separate flag. + +Suggested examples: + +```bash +loc search "onboarding live mode" --json +loc search "launch risks" --source notion --kind page --json +loc search "customer renewal" --source gmail --since 30d --json +loc search "review_needed" --status review-needed --json +``` + +### `loc context build` + +Purpose: + +- the main primitive for the missing-URL case; +- converts a task into a ranked evidence pack. + +Behavior: + +- searches local index; +- optionally reads repository state; +- ranks candidates; +- hydrates bounded top candidates; +- writes a context pack; +- explains why each item was included; +- emits trace spans for ranking, hydration, and token-relevant output size. + +Suggested command: + +```bash +loc context build \ + --goal "Prepare launch readiness summary" \ + --cwd "$PWD" \ + --since 7d \ + --sources notion,google-docs,granola,gmail \ + --hydrate top-k \ + --max-items 30 \ + --json +``` + +### `loc context explain ` + +Purpose: + +- make retrieval auditable. + +Behavior: + +- shows query terms; +- shows ranking signals; +- shows hydrated items; +- shows skipped items; +- shows stale or inaccessible sources; +- shows remote calls and timings. + +## Desktop Experience + +### Home + +Home should surface work, not mounts alone: + +- recently opened files; +- recently hydrated files; +- recent context packs; +- Review Center count; +- source health; +- suggested workflows such as "Prepare standup" or "Review launch readiness". + +### Search Or Ask + +The command bar should support: + +- exact URL open; +- title/path search; +- body search; +- "ask a workflow" query; +- add result to context; +- copy path; +- reveal in file browser; +- open remote source; +- hydrate result. + +Result groups: + +- best matches; +- ready files; +- metadata-only matches; +- pending review; +- recent activity; +- online-only candidates; +- disconnected sources, hidden by default. + +### Sources + +Sources should explain cache coverage: + +- connected account; +- mounted paths; +- metadata indexed count; +- body indexed count; +- stale count; +- failed sync count; +- last observation time; +- next scheduled sync; +- per-source "sync metadata" and "hydrate selected" actions. + +### Context Packs + +Desktop should show a small "Context" workspace: + +- created context packs; +- task name; +- sources used; +- files included; +- freshness; +- missing access; +- copy agent prompt; +- open context folder. + +This becomes the human-visible version of `loc context build`. + +## Architecture + +```mermaid +flowchart TD + A[Connectors] --> B[Metadata Observation] + A --> C[Body Hydration] + B --> D[Knowledge Items] + C --> E[Knowledge Chunks] + D --> F[Knowledge Graph] + E --> F + D --> G[Search Index] + E --> G + F --> H[Context Builder] + G --> H + I[User Activity and Review State] --> H + J[Repo and Local Workspace Signals] --> H + H --> K[Hydration Planner] + K --> C + H --> L[Context Pack Files] + L --> M[Agent Reads Files] + M --> N[Mounted Source Files] + N --> O[Review Center and Live Mode] + O --> A +``` + +### Components + +#### Connector Observation Jobs + +Cheap metadata refresh: + +- known objects; +- child containers; +- source roots; +- recent changes where supported; +- webhook or incremental checkpoints where supported. + +Observation should avoid body hydration. + +#### Body Hydration Jobs + +Fetch and render source content: + +- explicit file pull; +- file open; +- context-pack top-k hydration; +- Live Mode remote fast-forward; +- user-selected source sync. + +Hydration should emit spans for connector fetch, render, local store writes, +projection writes, shadow writes, and conflict checks. + +#### Knowledge Indexer + +Builds searchable records from: + +- entity metadata; +- remote observations; +- hydrated shadows; +- visible file contents when safe; +- frontmatter; +- source-specific metadata; +- extracted links and mentions. + +The first implementation should use SQLite FTS5 because the repo already uses +SQLite and has an entity metadata FTS path. Move to a dedicated search engine +only after product-grade field weighting, fuzzy matching, or scale makes it +necessary. + +#### Graph Builder + +Creates edges from: + +- source parent-child structure; +- Markdown links; +- Notion links and mentions; +- Google Docs links; +- Gmail sender/recipient/thread relationships; +- Granola attendees and meeting timestamps; +- Slack channel/thread/message relationships; +- issue IDs; +- pull request URLs; +- commit hashes; +- shared people, projects, labels, and dates. + +Graph edges should be explainable and confidence-scored. + +#### Ranker + +Ranks candidates for search and context packs. + +Initial scoring can be deterministic: + +```text +score = + lexical_match + + title_path_boost + + body_snippet_boost + + recency_boost + + current_workspace_boost + + user_activity_boost + + graph_neighbor_boost + + source_priority_boost + + freshness_boost + - stale_penalty + - disconnected_penalty + - conflict_penalty + - duplicate_penalty +``` + +An LLM reranker can be added later, but the first pass should not require an LLM +to retrieve useful local context. + +#### Context Builder + +Converts a task into an evidence pack. + +Inputs: + +- goal text; +- current working directory; +- mounted source roots; +- optional time window; +- optional people or team; +- optional repo metadata; +- source filters; +- budget. + +Outputs: + +- `context.md`; +- `manifest.json`; +- `paths.txt`; +- `freshness.md`; +- `missing-access.md`; +- `hydration-plan.json`; +- trace events. + +#### Hydration Planner + +Decides whether to hydrate metadata-only or stale candidates. + +Policies: + +- `none`: use only local indexed content; +- `metadata`: refresh cheap metadata only; +- `top-k`: hydrate the best candidates; +- `neighbors`: hydrate direct graph neighbors around high-confidence matches; +- `bounded-recursive`: explicit depth and item cap. + +The planner must respect source budgets and rate limits. + +## Storage Model + +Add a rebuildable knowledge subsystem, while preserving durable identity and +pending local work in existing state. + +Suggested tables: + +```text +knowledge_items +knowledge_chunks +knowledge_edges +knowledge_activity +knowledge_context_packs +knowledge_context_pack_items +knowledge_hydration_plans +knowledge_freshness +knowledge_aliases +knowledge_index_jobs +``` + +### `knowledge_items` + +Source-backed object metadata. This can be rebuilt from mounts, entities, and +observations, but should preserve stable local IDs across rebuilds when remote +identity is unchanged. + +### `knowledge_chunks` + +Searchable text units from hydrated content and source metadata. Chunks are +rebuildable. + +### `knowledge_edges` + +Relationships across items. Most edges are rebuildable. User-pinned or +user-confirmed edges should be durable. + +### `knowledge_activity` + +Signals from: + +- open file; +- copy path; +- reveal; +- context pack inclusion; +- push; +- pull; +- live-mode tracking; +- review; +- recent source sync. + +### `knowledge_context_packs` + +Auditable retrieval sessions. These are valuable for debugging, product metrics, +and enterprise trust. + +## Context Pack Format + +### `context.md` + +Human and agent readable summary: + +```md +# Context Pack: Today's Engineering Update + +Goal: Generate today's engineering update. +Created: 2026-07-22 14:10:00 IST +Coverage: Notion, Granola, Gmail, Google Docs + +## Read First + +1. /Users/me/Library/CloudStorage/Locality/notion/engineering-wiki/standups/page.md + Reason: exact title and recent activity match. + Freshness: observed 3m ago, hydrated 2m ago. + +2. /Users/me/Library/CloudStorage/Locality/granola/product-sync/summary.md + Reason: meeting attendee and date match. + Freshness: indexed 10m ago. + +## Relevant But Not Hydrated + +- Locality Launch QA Notes + Reason: title match, metadata only. + Suggested: loc pull "" + +## Missing Access + +- Gmail account is connected but sync is paused. +``` + +### `manifest.json` + +Machine-readable evidence list: + +```json +{ + "goal": "Generate today's engineering update", + "created_at": "2026-07-22T08:40:00Z", + "items": [ + { + "item_id": "notion:page:...", + "connector": "notion", + "title": "Standups with Locality", + "absolute_path": "/Users/me/Library/CloudStorage/Locality/notion/engineering-wiki/standups/page.md", + "source_url": "https://app.notion.com/...", + "state": "ready", + "freshness": "hydrated", + "score": 92.4, + "reasons": ["title_match", "recent_activity", "graph_neighbor"], + "agent_readable": true + } + ] +} +``` + +## Benchmark Model + +The benchmark phases should match product primitives: + +```text +locate_offline +resolve_remote +hydrate_target +hydrate_context +agent_run +sync_review +push +``` + +This avoids mixing unrelated costs. For example, a single "locate and +prehydrate" number can hide URL repair, remote parent listing, target hydration, +recursive directory hydration, and local projection writes. Those are different +product decisions and should be measured separately. + +Metrics: + +- local search latency; +- remote resolve latency; +- target hydration latency; +- context hydration latency; +- number of remote calls; +- number of hydrated items; +- cache hit rate; +- agent wall time; +- agent tool calls; +- tokens; +- output quality score; +- freshness errors; +- sync conflicts; +- push success rate. + +## Moat + +The defensible product advantage is not just a connector list. Connectors are +necessary but not enough. + +The durable advantage is the combination of: + +- local source-backed filesystem; +- permission-aware source identity; +- source-specific rendering and sync semantics; +- local search over mounted knowledge; +- graph relationships across apps; +- freshness and hydration state; +- pending local edits and Review Center state; +- Live Mode policy; +- push journals and auditability; +- context packs agents can read and cite; +- workflow traces that explain what happened. + +This is difficult for a pure MCP architecture to reproduce because MCP is mostly +a tool invocation layer. It can call APIs, but it does not naturally maintain a +long-lived local graph, source-backed file tree, sync journal, review workflow, +or cache that agents can inspect with normal file tools. + +## Research And Inspiration + +This section is here so reviewers can debate sources and remove anything that +does not fit Locality's direction. The goal is not to copy these systems. The +goal is to understand which ideas are durable and which ideas should remain +outside the product. + +### Glean And Enterprise Knowledge Graphs + +References: + +- https://docs.glean.com/security/knowledge-graph +- https://docs.glean.com/connectors/connectors-power-glean + +Useful ideas: + +- enterprise search needs a permission-aware view of indexed company knowledge; +- people, documents, projects, activity, and source relationships all improve + relevance; +- connectors are not just API wrappers, they feed a normalized index and graph; +- retrieval quality depends on source freshness and access correctness. + +What Locality should not copy blindly: + +- a search/chat-only product center; +- an opaque hosted index as the only user-visible surface; +- treating files as export artifacts rather than the primary agent interface. + +Locality angle: + +```text +Glean-like connected relevance, but with a local source-backed filesystem, +reviewable diffs, Live Mode, and sync back to the system of record. +``` + +### Obsidian And Local Knowledge Work + +Reference: + +- https://obsidian.md/help/plugins/graph + +Useful ideas: + +- backlinks help users discover related knowledge; +- local graph views can reveal nearby context around an active note; +- a local Markdown workspace gives users and agents a simple mental model. + +What Locality should not copy blindly: + +- making graph visualization the main product experience; +- asking users to manually curate tags and links as the core workflow; +- creating a separate knowledge base that drifts away from source apps. + +Locality angle: + +```text +Borrow backlinks, related items, local files, and command-palette navigation. +Do not become a personal notes app. +``` + +### Notion API Search Limits + +References: + +- https://developers.notion.com/reference/post-search +- https://developers.notion.com/reference/search-optimizations-and-limitations + +Useful ideas: + +- source APIs can help with exact locate, metadata discovery, and scoped query; +- source APIs should be part of ingestion and repair paths. + +What Locality should not do: + +- depend on Notion API search as the product's primary search engine; +- block desktop typeahead or agent retrieval on remote API calls; +- imply full workspace body search when only metadata or title coverage exists. + +Locality angle: + +```text +Remote search is a feeder and repair path. Locality search is the product path. +``` + +### MCP + +References: + +- https://modelcontextprotocol.io/specification/2025-06-18 +- https://github.com/modelcontextprotocol/modelcontextprotocol + +Useful ideas: + +- common tool/resource protocol for model clients; +- connector ecosystem and standard integration contracts; +- clear separation between clients, servers, tools, and resources. + +What Locality should not do: + +- reduce itself to a bundle of API tools; +- make agents repeatedly rediscover state through tool calls; +- lose the durable local filesystem, sync journal, and review semantics. + +Locality angle: + +```text +MCP can be an access surface for Locality, but the durable value is the local +cache, graph, files, freshness, review state, and sync back. +``` + +### Retrieval-Augmented Generation + +References: + +- https://arxiv.org/abs/2005.11401 +- https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html + +Useful ideas: + +- models need access to external, updateable knowledge; +- provenance matters for knowledge-intensive tasks; +- retrieval can reduce reliance on model memory. + +What Locality should not do: + +- treat vector retrieval alone as sufficient; +- send private company content to hosted embedding services by default; +- hide provenance behind generated summaries. + +Locality angle: + +```text +Retrieval should return files, snippets, source URLs, freshness, and reasons. +Generation should happen after the agent can inspect evidence. +``` + +### GraphRAG + +References: + +- https://arxiv.org/abs/2404.16130 +- https://www.microsoft.com/en-us/research/publication/from-local-to-global-a-graph-rag-approach-to-query-focused-summarization/ + +Useful ideas: + +- graph indexes can help answer broad questions over large private corpora; +- entity and relationship extraction can improve sensemaking beyond keyword + retrieval; +- community summaries may help when the user asks corpus-level questions. + +What Locality should not do first: + +- require LLM-derived graph construction before basic search works; +- make generated graph summaries authoritative; +- spend high model cost to maintain graph state before the product proves local + lexical and metadata retrieval. + +Locality angle: + +```text +Start with deterministic edges from source structure, links, people, dates, +issues, commits, and activity. Add LLM-derived edges later as optional, +explainable, rebuildable enrichment. +``` + +### SQLite FTS5 And Tantivy + +References: + +- https://sqlite.org/fts5.html +- https://docs.rs/tantivy/latest/tantivy/ +- https://github.com/quickwit-oss/tantivy + +Useful ideas: + +- SQLite FTS5 is a good first implementation path because Locality already uses + SQLite state and can keep packaging simple; +- Tantivy is a strong Rust-native search library if Locality outgrows SQLite FTS + field weighting, snippets, indexing scale, or query performance. + +What Locality should not do: + +- add a separate search engine before product requirements justify it; +- make search index state authoritative user state; +- block sync correctness on a rebuildable search index. + +Locality angle: + +```text +Use SQLite FTS5 first. Treat the search layer as rebuildable. Move to Tantivy +only when fielded search, ranking, snippets, fuzzy matching, or scale require it. +``` + +## Deliberate Non-Goals And Obsolete Paths + +These paths are tempting, but they do not create the product we want. + +### Remote API Search As The Main Product + +Remote search APIs differ by source, rate limit, permissions, and result +quality. They are useful for ingestion and repair. They should not be the normal +agent retrieval loop. + +### Recursive Hydration As Context Retrieval + +Recursive folder pull is not retrieval. It is an expensive traversal. Context +retrieval should rank first, then hydrate a bounded set of useful items. + +### URL-First Agent Workflows + +URL-first workflows work only when the user or agent already knows the target. +The main product should accept goals and produce relevant local context. + +### LLM Memory As The Cache + +Model memory is not a durable company knowledge cache. Locality needs explicit +local state, source identity, freshness, provenance, and reviewable evidence. + +### Vector Search As The Foundation + +Embeddings are useful later, especially for recall. They should not replace +lexical search, source metadata, graph relationships, permissions, and freshness. + +### One Flattened Global Folder + +Flattening every app into one folder creates naming conflicts and write +ambiguity. Keep source-backed trees canonical. Build generated knowledge views +on top. + +### Derived Views As Write Targets + +Generated views such as `knowledge/projects` or context packs should be +read-only first. Writes should happen through source-backed files until ownership +rules are clear. + +### MCP-Only Architecture + +MCP tools can call APIs, but they do not automatically provide persistent local +files, source-backed paths, sync journals, freshness state, review state, or +agent-readable working sets. Locality can expose MCP later without becoming only +an MCP server. + +## Roadmap + +### Phase 0: Clarify Existing Primitives + +Goal: make current behavior measurable and predictable. + +Work: + +- add `loc locate --offline`; +- add `loc resolve `; +- make `loc pull ` single-entity by contract; +- require explicit policy flags for directory recursive hydration; +- split benchmark phases; +- keep current `loc locate` behavior behind compatibility defaults until the + desktop flow migrates. + +Success: + +- exact URL open is faster when cached; +- remote repair is explicit; +- profiling tells us where time is spent. + +### Phase 1: Body-Aware Local Search + +Goal: search ready local knowledge, not only metadata. + +Work: + +- index hydrated shadow body chunks; +- index headings, frontmatter, source URLs, aliases, and snippets; +- return matched field and snippet; +- preserve safety labels; +- expose stable JSON for agents and desktop. + +Success: + +- agents can find relevant hydrated pages without knowing URLs; +- normal search remains local and fast. + +### Phase 2: Context Pack MVP + +Goal: solve the missing-URL workflow. + +Work: + +- add `loc context build`; +- implement deterministic ranker using lexical, path, recency, and activity + signals; +- write `.locality/context/` packs; +- include paths, freshness, missing access, and reasons; +- support `--hydrate none|top-k`; +- add traces for retrieval and hydration. + +Success: + +- scrum update and launch-readiness workflows can start from task text; +- context packs are auditable and repeatable. + +### Phase 3: Connector-Neutral Graph + +Goal: connect knowledge across apps. + +Work: + +- add `knowledge_edges`; +- extract links, mentions, people, dates, issues, PRs, commits, and source + relationships; +- add related-items API; +- add context graph expansion; +- show backlinks and related work in desktop. + +Success: + +- Locality can answer "what else is related to this?" across sources; +- agents discover useful context they did not know to ask for. + +### Phase 4: Background Freshness And Relevance + +Goal: keep useful knowledge hot without broad crawling. + +Work: + +- prioritize active files, pending review, recent context packs, recent mounts, + recent repo activity, and source-specific recent changes; +- add per-source API budgets; +- add cache coverage UI; +- add "hydrate selected" and "sync metadata" actions. + +Success: + +- most agent workflows hit ready local files; +- remote API work is bounded and explainable. + +### Phase 5: Enterprise Controls + +Goal: make the cache trustworthy in company environments. + +Work: + +- source and workspace policy controls; +- cache retention controls; +- local encryption where needed; +- admin-visible source coverage; +- audit export; +- redaction policy for sensitive sources; +- per-connector permission diagnostics. + +Success: + +- teams can understand what Locality cached, why, and who can access it. + +### Phase 6: Optional Semantic Layer + +Goal: improve recall after lexical search and graph are solid. + +Work: + +- chunk embeddings as rebuildable index; +- local embedding option first where practical; +- hosted embedding only with explicit workspace policy; +- semantic reranking for context packs; +- explain semantic matches with source snippets and paths. + +Success: + +- semantic search improves discovery without becoming an opaque source of truth. + +## Immediate Implementation Slice + +The smallest valuable slice: + +1. Add command semantics: + - `loc locate --offline`; + - `loc resolve`; + - single-entity `loc pull ` contract; + - explicit directory hydration flags. +2. Extend local search: + - hydrated body chunks; + - headings; + - snippets; + - field weights; + - JSON reasons. +3. Add `loc context build --hydrate none`: + - local search only; + - writes context pack; + - no new remote calls. +4. Add `loc context build --hydrate top-k`: + - bounded top candidate hydration; + - source budgets; + - trace spans. +5. Add desktop read-only surfacing: + - recent context packs; + - cache coverage; + - "copy agent context path". + +This gives us the product shape without destabilizing sync. + +## Risks And Guardrails + +### Permission Leakage + +Risk: showing stale or disconnected content from old access. + +Guardrail: + +- hide disconnected sources by default; +- preserve access state in every result; +- never include inaccessible content in context packs unless explicitly allowed. + +### Stale Context + +Risk: agent uses outdated content. + +Guardrail: + +- freshness labels in search and context packs; +- stale penalty in ranking; +- `loc context build --require-fresh` for sensitive workflows. + +### API Rate Limits + +Risk: context hydration becomes broad crawling. + +Guardrail: + +- source budgets; +- top-k hydration; +- queue-based background jobs; +- no recursive default. + +### Sync Ambiguity + +Risk: derived knowledge views become confusing write targets. + +Guardrail: + +- source-backed trees are writable; +- context packs and knowledge views are read-only first; +- generated views link back to canonical source files. + +### Local Storage Growth + +Risk: broad body indexing consumes too much disk. + +Guardrail: + +- cache budgets; +- retention policies; +- chunk dedupe by content hash; +- source-specific attachment policies. + +### Opaque Ranking + +Risk: users do not trust why items were included. + +Guardrail: + +- `loc context explain`; +- reasons in manifest; +- trace output; +- desktop "why this result" affordance. + +## Open Questions + +1. Should context packs live under `.locality/context` inside the repo, under the + Locality state root, or under the mounted Locality workspace? +2. Should `loc context build` be a CLI-only beta first, or should desktop expose + it immediately as "Prepare context"? +3. What is the first non-Notion connector that should contribute graph signals: + Gmail, Granola, Google Docs, Slack, Linear, or GitHub? +4. Should we index visible dirty local files directly, or only shadows plus + Review Center state? +5. How should workspace admins define retention and indexing policies? +6. Should source-specific summaries be stored, or should summaries exist only + inside context packs? +7. What quality benchmark should decide whether context retrieval is good: + human rating, answer grounding, remote calls avoided, time saved, or task + completion rate? + +## Recommended Direction + +Build Locality as the local connected knowledge cache for agents. + +The product should keep its current filesystem and sync discipline, but add a +retrieval layer that starts from user intent instead of requiring a URL. The +first version should stay simple and deterministic: local FTS, source metadata, +freshness labels, activity signals, context packs, and bounded hydration. The +graph and semantic layer can follow once the local cache and context-pack +contract are reliable. + +The end state is powerful: + +- users connect company tools once; +- Locality keeps a local, source-backed knowledge cache; +- agents ask for context by goal; +- Locality returns ranked local files and evidence; +- agents work with normal file operations; +- Locality safely syncs back to the systems of record. + +That is the product wedge and the long-term platform. From 2b4cd92c675934c0c8caf92a9ee47c945a3a396d Mon Sep 17 00:00:00 2001 From: Sarthak Agarwal Date: Wed, 22 Jul 2026 20:20:09 +0530 Subject: [PATCH 2/6] Add experiment findings to knowledge cache design --- .../local-connected-knowledge-cache-design.md | 189 ++++++++++++++++++ 1 file changed, 189 insertions(+) diff --git a/docs/local-connected-knowledge-cache-design.md b/docs/local-connected-knowledge-cache-design.md index f35ef1eb..f97c26f1 100644 --- a/docs/local-connected-knowledge-cache-design.md +++ b/docs/local-connected-knowledge-cache-design.md @@ -871,6 +871,195 @@ Metrics: - sync conflicts; - push success rate. +## Experiment Evidence + +This design direction is partly motivated by the Amika comparison experiment we +ran against a Locality-mounted Notion workspace and a Notion MCP workflow. + +Run shape: + +- run id: `traced-20260721T213706Z`; +- model: `gpt-5.6-luna` with low reasoning; +- task: generate a launch-readiness style report from local git state plus + Notion launch context; +- Locality path: locate target page, pull target page, locate context page, + recursively hydrate context directory, let the agent read local files, write a + mounted `page.md`, and inspect `loc diff`; +- MCP path: let the agent use Notion MCP calls for Notion context and local shell + commands for git context; +- Locality trace mode: direct CLI tracing was forced so pull and hydration spans + were visible. + +Topline from this single traced run: + +| Phase | Time | +| --- | ---: | +| Locality setup before agent | 141.2s | +| Locality agent wall time | 49.4s | +| Notion MCP agent wall time | 47.8s | +| Full run through both strategies | 239.5s | + +Agent usage shape: + +| Strategy | Input Tokens | Cached Input | Output Tokens | Tool Shape | +| --- | ---: | ---: | ---: | --- | +| Locality | 196,033 | 152,832 | 4,020 | 12 shell commands, 0 MCP calls | +| Notion MCP | 341,129 | 252,672 | 4,187 | 8 shell commands, 20 MCP calls | + +Important interpretation: + +- The Locality agent phase used fewer input tokens and no MCP calls, which + supports the agent-native file interface thesis. +- The current Locality setup path was expensive. The agent got useful local + files, but the synchronous locate and hydration work before the agent started + dominated the Locality side of the run. +- This is one profile, not a statistical benchmark. It is useful for finding + critical paths, not for claiming a universal win. + +### What Prehydration Showed + +Prehydration is valuable only when it is already done, cheap, or relevance +guided. + +In this run, the Locality path spent `141.2s` preparing context before the agent +started. That setup included URL locate, target pull, context locate, recursive +context hydration, local search, and report target preparation. Once the files +were available, the Locality agent could work through normal shell reads and +file writes. The problem is that current prehydration is still too synchronous +and too broad. + +This points to a product requirement: + +```text +Locality should not make every agent workflow pay the full cost of locating, +repairing, pulling, and recursively hydrating at task time. +``` + +The local connected knowledge cache should move useful work earlier and make +task-time work bounded: + +- metadata should already be indexed; +- known URLs should resolve through `loc locate --offline`; +- body chunks for hot files should already be searchable; +- stale or metadata-only candidates should be ranked before hydration; +- context hydration should use top-k or neighbor expansion, not recursive folder + traversal by default. + +### Critical Path Findings + +The traced run showed these bottlenecks: + +| Area | Finding | Design Implication | +| --- | --- | --- | +| URL locate | Target locate took `44.4s`; context locate took `32.0s`. Final local search was only `1-2ms`; most time was remote parent/path preparation. | Split `loc locate --offline` from `loc resolve`. Offline locate must stay local and fast. Remote path repair should be explicit and measured. | +| Target pull | Pulling the target page took `4.0s`, dominated by one `connector.fetch_render`. | Single-file hydration is acceptable as an explicit operation, but should avoid re-fetching if remote version is unchanged. | +| Context hydration | Context pull took `60.7s`, hydrated `19` pages, and enumerated `18` child entries. | Directory hydration must be policy-driven. Context building should hydrate ranked items, not recursively hydrate a page tree by default. | +| Fetch/render | Context fetch/render spans summed to `34.3s` across `19` calls. | Add skip-on-freshness checks and consider bounded concurrent remote reads once local commit/projection writes remain serial and deterministic. | +| Child listing | Context `list_children` spans summed to `26.1s` across `19` calls. | Avoid listing children unless the context plan needs neighbors. Add deeper connector spans so slow list calls can be separated into pagination, block-child listing, page/database metadata fetch, and rate-limit waits. | +| Agent phase | Locality agent wall time was similar to MCP agent wall time, but used fewer input tokens and no MCP calls. | The opportunity is not only faster agent execution. The bigger opportunity is fewer repeated tool calls, cheaper context, more local cache hits, and safer sync back. | + +### Critical Paths To Improve + +#### 1. Locate And Resolve Split + +Current exact URL locate can do remote parent/path preparation. The product path +should split this: + +```bash +loc locate --offline +loc resolve +loc pull +``` + +This makes the common cached case instant and makes remote repair visible in +traces, benchmarks, and UI. + +#### 2. Local Body Indexing + +Search should not stop at metadata. Hydrated shadows and safe visible content +should produce chunks with snippets, headings, and freshness. This is the first +step toward agents discovering useful context without URLs. + +#### 3. Relevance-Guided Context Hydration + +`loc context build` should use a hydration policy: + +```text +none -> metadata and already-indexed body only +top-k -> hydrate the highest ranked missing body candidates +neighbors -> hydrate graph neighbors around confident matches +bounded-recursive -> explicit depth and item cap +``` + +Recursive directory pull should remain a deliberate operation, not the default +way to prepare agent context. + +#### 4. Freshness-Based Fetch Skips + +If Locality has a hydrated page and remote metadata says the body version has +not changed, context build should not fetch/render that page again. The cache +must make "already good enough" cheap. + +#### 5. Bounded Parallel Remote Reads + +Fetch/render and child listing are currently strong candidates for bounded +parallelism, but only after the operation is split into: + +- parallel remote read/list phase; +- serial deterministic local commit/projection phase. + +This preserves Locality's state and sync safety while reducing wall time. + +#### 6. Deeper Connector Spans + +The current trace tells us `list_children` is expensive, but not always why. Add +spans for: + +- Notion block children pagination; +- retrieve page metadata; +- retrieve database metadata; +- retrieve database rows; +- render canonical Markdown; +- rate-limit waits and retries; +- local SQLite writes; +- visible projection writes. + +This turns "Notion is slow" into actionable work items. + +#### 7. Cache Coverage UI + +Desktop should show whether a source is: + +- metadata indexed; +- body indexed; +- stale; +- hydration queued; +- failed; +- disconnected. + +That gives users a reason to trust or refresh context before asking agents to +work. + +### Experiment Follow-Up + +Before using this comparison in external material, run a repeatable benchmark: + +- `RUNS=5` minimum for each strategy; +- same model and reasoning effort; +- same output format; +- same Notion target and context corpus; +- separate timings for offline locate, resolve, target hydrate, context hydrate, + agent run, diff, and optional push; +- record remote call counts, hydrated item counts, token usage, wall time, + freshness failures, and output quality review. + +The design target is clear even before repeated runs: + +```text +Make the common path local, cache-backed, relevance-ranked, and explicit about +remote work. +``` + ## Moat The defensible product advantage is not just a connector list. Connectors are From f12c8a239715564752bca7af9ecc78f430def6e6 Mon Sep 17 00:00:00 2001 From: Sarthak Agarwal Date: Wed, 22 Jul 2026 22:13:50 +0530 Subject: [PATCH 3/6] Add natural retrieval experiment harness --- experiment/agent-conversation-profile.mjs | 152 ++++- experiment/locality-mcp-comparison/README.md | 30 + .../natural-retrieval-experiment-design.md | 608 ++++++++++++++++++ .../run-natural-retrieval-batch.sh | 33 + .../scripts/run-natural-retrieval-batch.mjs | 550 ++++++++++++++++ 5 files changed, 1357 insertions(+), 16 deletions(-) create mode 100644 experiment/locality-mcp-comparison/natural-retrieval-experiment-design.md create mode 100755 experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh create mode 100755 experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs diff --git a/experiment/agent-conversation-profile.mjs b/experiment/agent-conversation-profile.mjs index 72705468..345b015f 100644 --- a/experiment/agent-conversation-profile.mjs +++ b/experiment/agent-conversation-profile.mjs @@ -9,7 +9,7 @@ import { spawnSync } from "node:child_process"; import { basename, extname, join, resolve } from "node:path"; const DEFAULT_DURATION_MS = 1000; -const TIMESTAMP_KEYS = ["timestamp", "created_at", "time", "ts"]; +const TIMESTAMP_KEYS = ["observed_at_ms", "timestamp", "created_at", "time", "ts"]; const CONTAINER_KEYS = [ "events", "messages", @@ -384,6 +384,19 @@ function partialEventsForRecord(record, target = eventTargetForRecord(record)) { } function eventTargetForRecord(record) { + if (isPlainObject(record.event)) { + const event = record.event; + const item = isPlainObject(event.item) ? event.item : {}; + return mergeEventContext( + { + ...event, + ...item, + event_type: event.type, + type: item.type ?? event.type, + }, + record, + ); + } if (isPlainObject(record.item)) { return mergeEventContext(record.item, record); } @@ -402,6 +415,8 @@ function mergeEventContext(target, parent, options = {}) { created_at: target.created_at ?? parent.created_at, time: target.time ?? parent.time, ts: target.ts ?? parent.ts, + observed_at_ms: target.observed_at_ms ?? parent.observed_at_ms, + event_type: target.event_type ?? parent.event_type, duration_ms: inheritDuration ? target.duration_ms ?? parent.duration_ms : target.duration_ms, @@ -516,6 +531,10 @@ function classifyKind(object, parent) { const rawType = String(object.type ?? object.kind ?? parent?.type ?? "") .toLowerCase() .replace(/\s+/g, "_"); + const eventType = String( + object.event_type ?? object.eventType ?? parent?.event_type ?? parent?.eventType ?? "", + ).toLowerCase(); + const status = String(object.status ?? parent?.status ?? "").toLowerCase(); const role = roleFor(object, parent); if ( @@ -527,17 +546,25 @@ function classifyKind(object, parent) { return "reasoning"; } + if (rawType === "agent_message") { + return "assistant_message"; + } + if ( rawType.includes("tool_use") || rawType.includes("tool_call") || rawType.includes("function_call") || rawType.includes("local_shell_call") || + rawType.includes("command_execution") || rawType === "mcp_call" ) { if ( rawType.includes("output") || rawType.includes("result") || - rawType.includes("response") + rawType.includes("response") || + eventType.endsWith(".completed") || + eventType.endsWith(".done") || + status === "completed" ) { return "tool_result"; } @@ -575,6 +602,12 @@ function roleFor(object, parent) { } function toolNameFor(object, parent) { + const rawType = String(object.type ?? object.kind ?? parent?.type ?? "") + .toLowerCase() + .replace(/\s+/g, "_"); + if (rawType.includes("command_execution")) { + return "bash"; + } const candidate = object.name ?? object.tool_name ?? @@ -755,31 +788,99 @@ function enrichToolEvents(events) { function toolGroupFor(event) { const toolName = event.tool_name ?? "unknown_tool"; if (toolName.toLowerCase() === "bash") { - return bashCommandCallsLoc(event.tool_command) ? "bash_loc" : "bash_other"; + return shellToolGroupFor(event.tool_command); + } + if (toolName.toLowerCase() === "local_shell") { + return shellToolGroupFor(event.tool_command); } return toolName; } +function shellToolGroupFor(command) { + const payload = shellCommandPayload(command); + const segments = payload.split(/(?:&&|\|\||[;|\n])/); + for (const segment of segments) { + const locSubcommand = locSubcommandForSegment(segment); + if (locSubcommand) { + return `loc_${locSubcommand}`; + } + } + + for (const segment of segments) { + const executable = shellSegmentExecutable(segment); + if (!executable) { + continue; + } + if (executable === "git") { + return "git"; + } + if (["rg", "grep", "find", "fd"].includes(executable)) { + return "rg_grep_find"; + } + if ( + ["cat", "sed", "head", "tail", "less", "awk", "nl", "wc", "jq"].includes( + executable, + ) + ) { + return "file_read"; + } + } + + return "bash_other"; +} + +function shellCommandPayload(command) { + if (typeof command !== "string" || command.trim() === "") { + return ""; + } + const trimmed = command.trim(); + const match = trimmed.match( + /(?:^|\s)(?:\S*\/)?(?:zsh|bash|sh)\s+-lc\s+((?:"(?:\\"|[^"])*")|'[^']*')/, + ); + if (!match) { + return trimmed; + } + return stripShellTokenQuotes(match[1]) + .replace(/\\"/g, '"') + .replace(/\\'/g, "'") + .replace(/\\n/g, "\n"); +} + +function locSubcommandForSegment(segment) { + const tokens = shellSegmentTokens(segment); + for (let index = 0; index < tokens.length; index += 1) { + const token = tokens[index]; + if (isAssignmentToken(token) || ["command", "env", "nice", "nohup", "sudo", "time"].includes(token)) { + continue; + } + if (basename(token) !== "loc") { + return null; + } + for (let subIndex = index + 1; subIndex < tokens.length; subIndex += 1) { + const candidate = tokens[subIndex]; + if (!candidate || candidate.startsWith("-") || isAssignmentToken(candidate)) { + continue; + } + return candidate.replace(/[^A-Za-z0-9_-].*$/, "") || "unknown"; + } + return "unknown"; + } + return null; +} + function bashCommandCallsLoc(command) { if (typeof command !== "string" || command.trim() === "") { return false; } - return command + return shellCommandPayload(command) .split(/(?:&&|\|\||[;|\n])/) .some((segment) => shellSegmentExecutable(segment) === "loc"); } function shellSegmentExecutable(segment) { - let remaining = segment.trim(); - while (remaining !== "") { - const token = firstShellToken(remaining); - if (!token) { - return null; - } - const value = stripShellTokenQuotes(token.value); - remaining = remaining.slice(token.end).trimStart(); - - if (/^[A-Za-z_][A-Za-z0-9_]*=.*/.test(value)) { + const tokens = shellSegmentTokens(segment); + for (const value of tokens) { + if (isAssignmentToken(value)) { continue; } if (["command", "env", "nice", "nohup", "sudo", "time"].includes(value)) { @@ -790,6 +891,24 @@ function shellSegmentExecutable(segment) { return null; } +function shellSegmentTokens(segment) { + let remaining = segment.trim(); + const tokens = []; + while (remaining !== "") { + const token = firstShellToken(remaining); + if (!token) { + break; + } + tokens.push(stripShellTokenQuotes(token.value)); + remaining = remaining.slice(token.end).trimStart(); + } + return tokens; +} + +function isAssignmentToken(value) { + return /^[A-Za-z_][A-Za-z0-9_]*=.*/.test(value); +} + function firstShellToken(value) { const match = value.match(/^(?:"(?:\\"|[^"])*"|'[^']*'|\\\s|\S)+/); if (!match) { @@ -907,7 +1026,8 @@ function toolWaitDurationMs(event, events, index) { if (!result) { return event.duration_ms; } - return Math.max(1, result.start_ms - event.start_ms); + const observedWaitMs = result.start_ms - event.start_ms; + return observedWaitMs > 0 ? observedWaitMs : event.duration_ms; } function matchingToolResult(event, events, index) { @@ -1681,7 +1801,7 @@ function renderSummaryMarkdown(summary) { lines.push(""); } - lines.push("## Tool Wait By Group", ""); + lines.push("## Tool Time By Group", ""); for (const conversation of summary.conversations) { lines.push(`### ${markdownHeadingText(conversation.label)}`, ""); if (conversation.tool_groups.length === 0) { diff --git a/experiment/locality-mcp-comparison/README.md b/experiment/locality-mcp-comparison/README.md index 5833b0a3..7b37f73f 100644 --- a/experiment/locality-mcp-comparison/README.md +++ b/experiment/locality-mcp-comparison/README.md @@ -161,6 +161,36 @@ ssh -o StrictHostKeyChecking=accept-new "$SSH_TARGET" ' ' ``` +## Run Natural Retrieval Batch + +The natural retrieval batch tests whether agents can discover relevant context +without receiving known Notion URLs, mounted context paths, or precomputed +inventory files. It writes artifacts under `experiment/runs-2//`. + +```bash +ssh -o StrictHostKeyChecking=accept-new "$SSH_TARGET" ' + export PATH="$HOME/.cargo/bin:$PATH" + cd /home/amika/workspace/locality + NATURAL_RUNS=2 \ + CODEX_MODEL=gpt-5.6-luna \ + CODEX_REASONING_EFFORT=low \ + ./experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh +' +``` + +For a quick smoke test, run only the first scenario, first variant, and first +repeat: + +```bash +NATURAL_SINGLE_PAIR=1 ./experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh +``` + +Optional files-only ablation for the daily-engineering scenario: + +```bash +NATURAL_INCLUDE_FILES_ONLY=1 ./experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh +``` + ## Artifacts Each run writes to: diff --git a/experiment/locality-mcp-comparison/natural-retrieval-experiment-design.md b/experiment/locality-mcp-comparison/natural-retrieval-experiment-design.md new file mode 100644 index 00000000..605268e8 --- /dev/null +++ b/experiment/locality-mcp-comparison/natural-retrieval-experiment-design.md @@ -0,0 +1,608 @@ +# Natural Retrieval Experiment Design + +Draft date: July 22, 2026 + +Status: alignment draft. Do not run this as-is until we agree on scenarios, +prompt count, and publish behavior. + +## Why The Current Benchmark Is Not Enough + +The current launch-readiness benchmark is useful for profiling, but it is too +directive for product evidence. + +Current Locality path: + +- runner locates the target Notion URL; +- runner pulls the target page; +- runner locates and recursively hydrates a known context URL; +- runner gives the agent `CONTEXT_PATHS_FILE`, `CONTEXT_INVENTORY`, and + `CONTEXT_SEARCH_RESULTS`; +- prompt tells the agent which hydrated context inventory/search files to read. + +Current MCP path: + +- prompt tells the agent specific Notion search themes; +- agent uses Notion MCP search/fetch to gather context. + +That answers: + +```text +If context is prepared, can the agent produce a report? +``` + +The next experiment should answer: + +```text +Given a natural work request, can the agent discover relevant company context, +choose useful tools, produce a grounded output, and do it with less cost, +latency, and manual setup? +``` + +## Hypotheses + +### H1: Locality Improves The Agent Work Substrate + +Once useful content is available locally, the agent can use normal shell/file +operations with fewer MCP calls and fewer tokens. + +Expected signal: + +- fewer remote tool calls; +- fewer input tokens; +- more direct evidence paths; +- comparable or better output quality; +- more transparent evidence trail through local files. + +### H2: Current Locality Retrieval Is Not Yet Strong Enough + +If the agent is not given URLs or context paths, it may fall back to `find`, +`grep`, and broad file reads because `loc search` is still not rich enough over +body chunks, snippets, and relevance. + +Expected signal: + +- few or no `loc search` calls; +- repeated shell `find`/`grep` scans; +- missed relevant context that exists in mounted files; +- high time spent in discovery despite local data. + +### H3: Prehydration Should Be Measured But Not Overweighted + +Prehydration is valuable when it happens before the user task or is +relevance-guided. It is not fair to treat all prehydration as agent-run time, but +it is also not fair to ignore synchronous task-time hydration. + +Experiment implication: + +- measure prep/cache state separately; +- measure agent retrieval/execution separately; +- record whether the run was cold, warm metadata, or warm body cache. + +### H4: MCP Will Spend More On Repeated Remote Retrieval + +MCP can be flexible, but natural retrieval will likely require multiple search +and fetch calls, more token traffic, and more source-API dependency. + +Expected signal: + +- more MCP tool calls; +- more input tokens; +- more repeated context transfer; +- output may be more exhaustive if MCP search finds pages Locality has not + indexed or hydrated. + +## Strategies To Compare + +### Strategy A: Locality Natural + +The agent receives a natural task. It has access to: + +- local git checkout; +- mounted Locality source folders; +- `loc` CLI; +- installed Locality guidance semantics. + +The agent does not receive: + +- Notion URLs; +- known page names; +- precomputed context path files; +- precomputed inventory files; +- Notion MCP tools. + +Prompt guidance should say: + +```text +Use Locality-connected files and the `loc` CLI when helpful. For discovery, +prefer `loc search ` first, then inspect mounted files. Use `loc info`, +`loc status`, and `loc diff` when you need source or sync state. Do not use +Notion MCP or direct Notion APIs. +``` + +This intentionally reflects the product guidance installed for local agents. + +### Strategy B: Notion MCP Natural + +The agent receives the same natural task. It has access to: + +- local git checkout; +- Notion MCP tools. + +The agent does not receive: + +- mounted Locality files; +- `loc` CLI; +- precomputed Locality context paths; +- known page URLs. + +Prompt guidance should say: + +```text +Use Notion MCP for company context and local git commands for repository +context. Do not read Locality-mounted files and do not use `loc`. +``` + +### Optional Strategy C: Locality Files-Only Ablation + +This is not the main product path, but it helps answer whether `loc search` and +Locality guidance are adding value beyond mounted files. + +The agent receives: + +- mounted Locality source folders; +- local git checkout. + +The agent does not use: + +- `loc`; +- Notion MCP. + +If this performs as well as Strategy A, then `loc search/info/status` are not yet +pulling enough weight for retrieval. If Strategy A is better, the CLI guidance is +creating measurable value. + +Recommended: run this ablation only for one scenario in the pilot. + +## Cache Conditions + +We should not mix all cache states into one number. + +### Warm Metadata, Natural Body Retrieval + +Recommended primary condition. + +Before the benchmark: + +- connections and mounts exist; +- metadata/index state is allowed to exist from normal product use; +- no benchmark runner gives the agent exact context paths; +- the agent may call `loc search`, open files, and trigger hydration naturally. + +This matches the product vision: Locality has a local knowledge cache, but the +agent still chooses context. + +### Warm Body Cache + +Secondary condition. + +Before the benchmark: + +- relevant content is already hydrated from earlier use; +- agent still receives only the natural prompt. + +This measures the best-case Locality product path after daily use. + +### Cold Setup + +Diagnostic only. + +Before the benchmark: + +- mount exists but target context is not hydrated; +- measure locate/resolve/hydration as setup. + +This is useful for performance engineering, but should not dominate the product +comparison because Locality's strategic goal is a warm local cache. + +## Pilot Scenario Matrix + +To keep cost reasonable, start with two scenarios, two prompt variants, two +strategies, two independent repeats. + +```text +2 scenarios x 2 variants x 2 strategies x 2 repeats = 16 agent runs +``` + +Add the files-only ablation for one scenario if we want a CLI-vs-grep signal: + +```text ++ 2 variants x 1 strategy x 2 repeats = 4 extra runs +``` + +Total pilot with ablation: 20 runs. + +## Natural Scenarios + +### Scenario 1: Daily Engineering Update + +Intent: + +Find recent repository work, connect it to company planning/standup context, and +draft a concise update. + +Prompt A: + +```text +Prepare today's engineering update for the team. Look at recent repository work +and any relevant company context you can access. Summarize what changed, why it +matters, risks, blockers, and suggested next actions. Write the result as a +Markdown draft. Do not publish it remotely. +``` + +Prompt B: + +```text +I need a short standup-style update for Locality based on what changed recently. +Please discover the relevant context yourself, connect code changes to product +or launch work where possible, and produce a grounded Markdown draft. Do not +push or update any remote source. +``` + +Expected evidence classes: + +- recent git commits; +- standup or planning pages; +- launch or product context pages; +- known reliability or platform risks. + +### Scenario 2: Launch Readiness Review + +Intent: + +Assess whether recent work changes launch readiness and identify remaining +release blockers. + +Prompt A: + +```text +We are considering whether Locality is ready for a broader launch. Review recent +engineering work and relevant internal context, then draft a launch-readiness +assessment with evidence, risks, blockers, and the next validation steps. Do not +publish it remotely. +``` + +Prompt B: + +```text +Act like you are preparing a launch gate memo for Locality. Find the relevant +project context and recent code changes, decide what is actually proven, what is +still unverified, and what should block launch. Produce a concise Markdown memo. +Do not push anything. +``` + +Expected evidence classes: + +- launch planning pages; +- install/distribution docs; +- safety/review/push guidance; +- platform provider and live mode context; +- recent code/test commits. + +### Scenario 3: Sync Reliability Bug Triage + +Run after the pilot if the first two are stable. + +Prompt A: + +```text +A user reports that a Notion-mounted page did not sync correctly and that review +state stayed confusing after manual edits. Investigate likely product areas from +recent code and internal context. Draft a technical triage note with suspected +causes, missing evidence, and tests to add. Do not publish it remotely. +``` + +Prompt B: + +```text +Please investigate Locality sync reliability around mounted Notion pages, +conflicts, review state, and Live Mode. Use whatever connected company context +and recent repo work are relevant. Produce a grounded triage memo with concrete +next tests. Do not push anything. +``` + +Expected evidence classes: + +- sync model docs; +- live mode docs; +- conflict/review docs; +- recent commits around pull/push/review/daemon; +- relevant standup or user-report context. + +## Prompt Rules + +All strategies should share the same natural task text except for tool-access +rules. + +Allowed: + +- "Use the tools available to you." +- "Discover relevant context yourself." +- "Do not publish remotely." +- "Write Markdown output to the configured output file." + +Avoid: + +- exact Notion URLs; +- exact page titles; +- exact mounted paths; +- precomputed context inventory; +- search keyword lists that encode the answer; +- telling the agent which specific file to read. + +## Outputs Per Run + +Each run should write: + +```text +experiment/runs// + scenario.json + prompt.md + strategy.md + report-body.md + evidence-manifest.json + agent-trace.md + codex-events.jsonl + codex-events.tsv + codex-summary.json + codex-transcript.md + agent-profile/ + summary.md + summary.json + combined.speedscope.json + locality-agent-locality-trace.jsonl + locality-agent-locality-trace-summary.json + metrics.tsv +``` + +For paired strategy comparison: + +```text +experiment/runs// + comparisons/ + scenario-1-prompt-a-repeat-1.md + scenario-1-prompt-a-repeat-2.md + batch-summary.tsv + batch-summary.md +``` + +## Evidence Manifest + +Every agent should be required to write a machine-readable evidence manifest. +This reduces subjective interpretation after the run. + +Suggested shape: + +```json +{ + "task": "launch_readiness_review", + "strategy": "locality-natural", + "evidence": [ + { + "kind": "git_commit", + "id": "6aa3e9bd", + "reason": "File Provider initial discovery behavior" + }, + { + "kind": "locality_file", + "path": "/home/amika/notion/Go To Market/.../page.md", + "reason": "launch checklist and open blockers" + }, + { + "kind": "notion_mcp_page", + "title": "Locality Launch", + "reason": "launch checklist and open blockers" + } + ], + "limitations": [ + "No CI metadata inspected", + "No remote push attempted" + ] +} +``` + +The manifest lets us score: + +- relevant evidence count; +- unsupported claims; +- missing expected evidence; +- source diversity; +- whether the agent found context without being spoon-fed. + +## Tool-Use Metrics + +The experiment should report these per strategy and run: + +### General + +- total wall time; +- agent wall time; +- input tokens; +- cached input tokens; +- output tokens; +- reasoning output tokens; +- command/tool count; +- errors and retries; +- output word count. + +### Locality + +- `loc search` count and duration; +- `loc info` count and duration; +- `loc status` count and duration; +- `loc locate` / future `loc locate --offline` count and duration; +- `loc pull` count and duration; +- `loc diff` count and duration; +- direct file reads count; +- `find` / `rg` / `grep` count; +- hydrated files touched; +- Locality trace spans emitted by agent-run `loc` commands. + +### MCP + +- Notion MCP search count; +- Notion MCP fetch/read count; +- Notion MCP errors/retries; +- remote pages read; +- duplicated or repeated page fetches; +- tool-result token volume if available. + +### Output Quality + +Manual or semi-automatic rubric: + +- factual grounding; +- correct recent git summary; +- relevant company context found; +- missed important context; +- unsupported claims; +- actionable recommendations; +- concise enough for the target workflow; +- source transparency. + +## Profiler Work Needed Before Running + +`experiment/agent-conversation-profile.mjs` has useful concepts: + +- activity grouping; +- tool wait grouping; +- `bash_loc` vs `bash_other`; +- Perfetto, Speedscope, SnakeViz, and folded-stack outputs; +- summary Markdown. + +However, it currently does not profile the timestamped Codex JSONL from the +comparison runner because those records use `observed_at_ms`, and the script +only recognizes `timestamp`, `created_at`, `time`, and `ts`. + +Before the next experiment, update the profiler or normalize inputs: + +1. Add `observed_at_ms` to supported timestamp keys. +2. Recognize nested Codex records shaped like: + + ```json + {"observed_at_ms": 123, "event": {"type": "item.started", "item": {...}}} + ``` + +3. Classify Codex `command_execution` as shell tool calls. +4. Extract command text from `event.item.command`. +5. Categorize shell commands: + - `loc_search`; + - `loc_info`; + - `loc_status`; + - `loc_locate`; + - `loc_pull`; + - `loc_diff`; + - `git`; + - `rg_grep_find`; + - `file_read`; + - `other_shell`. +6. Categorize MCP calls by server/tool name. +7. Emit per-run and batch-level tool statistics. + +This gives a clearer answer to: + +```text +Did the Locality agent actually use Locality, or did it just grep files? +``` + +## Expected Bottlenecks To Watch + +### Locality + +Likely bottlenecks: + +- poor body-search coverage if hydrated shadows are not indexed well; +- agent falling back to broad filesystem scans; +- `loc search` returning metadata-only results without useful snippets; +- `loc locate` doing remote repair when the user did not give a URL; +- recursive hydration if the agent or runner pulls directories; +- missing `rg` in sandbox, forcing slower `find`/`sed` patterns; +- insufficient agent guidance around when to use `loc info/status/search`. + +Important nuance: + +`loc info` and `loc status` are not discovery tools. They are context and safety +tools once the agent has a candidate path. The main discovery primitive should +be `loc search` now and `loc context build` later. + +### MCP + +Likely bottlenecks: + +- repeated search/fetch calls; +- Notion search API limitations; +- high token volume from tool results; +- remote latency and content-filter stream interruptions; +- no durable local working set unless the agent creates one manually. + +## Success Criteria + +Locality is meaningfully better if: + +- output quality is equal or better; +- the agent finds relevant context without exact URLs; +- fewer tokens are used; +- fewer remote tool calls are used; +- the evidence trail is clearer; +- the agent can write a local draft and inspect sync state; +- setup cost is either amortized by warm cache or visibly reduced by better + retrieval. + +Locality is not yet better if: + +- agent relies mostly on blind `find`/`grep`; +- `loc search` is rarely used or unhelpful; +- relevant context is missed because it was not prehydrated; +- setup/hydration dominates each natural task; +- MCP finds better context with fewer manual hints. + +## Recommended Pilot + +Start with: + +```text +Scenario 1: Daily Engineering Update +Scenario 2: Launch Readiness Review +Prompt variants: A and B for each +Strategies: Locality Natural and Notion MCP Natural +Repeats: 2 independent runs per prompt/strategy +Cache condition: warm metadata, natural body retrieval +Publish: no remote push +``` + +That gives 16 runs and should be enough to expose whether Locality's natural +retrieval loop works. + +Then add: + +```text +Scenario 1 only +Strategy: Locality Files-Only Ablation +Repeats: 2 +``` + +This tells us whether `loc search/info/status` are adding value beyond a mounted +folder and shell search. + +## Alignment Questions + +1. Should the pilot include the files-only ablation now, or keep the first run to + Locality Natural vs Notion MCP Natural? +2. Should the Locality strategy be allowed to trigger hydration by opening files + and running `loc pull`, or should task-time hydration be disabled for the + first pass? +3. Should we publish any output page to Notion, or keep all outputs local until + the scoring rubric is stable? +4. Do we want to use only `gpt-5.6-luna`, or compare Luna vs Terra after the + benchmark harness is stable? +5. Should expected evidence pages be defined privately in a scoring file, so the + prompt remains natural but scoring can check recall? diff --git a/experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh b/experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh new file mode 100755 index 00000000..a6887530 --- /dev/null +++ b/experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="${REPO_DIR:-/home/amika/workspace/locality}" +cd "$REPO_DIR" + +export PATH="$HOME/.cargo/bin:$PATH" + +ENV_FILE="${LOCALITY_EXPERIMENT_ENV:-$HOME/.config/locality-experiment/env}" +if [ -f "$ENV_FILE" ]; then + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a +fi + +if [ -z "${AZURE_OPENAI_API_KEY:-}" ]; then + cat >&2 <<'EOF' +AZURE_OPENAI_API_KEY is missing in the sandbox. +Run the key setup command from your local machine before starting the experiment. +EOF + exit 2 +fi + +export CODEX_MODEL="${CODEX_MODEL:-gpt-5.6-luna}" +export CODEX_REASONING_EFFORT="${CODEX_REASONING_EFFORT:-low}" +export CODEX_EXEC_TIMEOUT_SECONDS="${CODEX_EXEC_TIMEOUT_SECONDS:-900}" +export NATURAL_RUNS="${NATURAL_RUNS:-2}" +export NATURAL_OUT_ROOT="${NATURAL_OUT_ROOT:-$REPO_DIR/experiment/runs-2}" +export LOCALITY_SOURCE_ROOT="${LOCALITY_SOURCE_ROOT:-/home/amika/notion}" + +node "$SCRIPT_DIR/scripts/run-natural-retrieval-batch.mjs" "$@" diff --git a/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs b/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs new file mode 100755 index 00000000..c6b22d37 --- /dev/null +++ b/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs @@ -0,0 +1,550 @@ +#!/usr/bin/env node + +import { + existsSync, + mkdirSync, + readFileSync, + writeFileSync, +} from "node:fs"; +import { spawnSync } from "node:child_process"; +import { dirname, join, relative, resolve } from "node:path"; + +const SCRIPT_DIR = dirname(new URL(import.meta.url).pathname); +const EXPERIMENT_DIR = resolve(SCRIPT_DIR, ".."); +const REPO_DIR = resolve(process.env.REPO_DIR ?? "/home/amika/workspace/locality"); +const OUT_ROOT = resolve(process.env.NATURAL_OUT_ROOT ?? join(REPO_DIR, "experiment/runs-2")); +const BATCH_ID = process.env.NATURAL_BATCH_ID ?? utcStamp(); +const RUNS = positiveInteger(process.env.NATURAL_RUNS ?? "2", "NATURAL_RUNS"); +const CODEX_MODEL = process.env.CODEX_MODEL ?? "gpt-5.6-luna"; +const CODEX_REASONING_EFFORT = process.env.CODEX_REASONING_EFFORT ?? "low"; +const CODEX_EXEC_TIMEOUT_SECONDS = process.env.CODEX_EXEC_TIMEOUT_SECONDS ?? "900"; +const LOC_BIN = resolve(process.env.LOC_BIN ?? join(REPO_DIR, "target/debug/loc")); +const LOCALITY_SOURCE_ROOT = resolve(process.env.LOCALITY_SOURCE_ROOT ?? "/home/amika/notion"); +const INCLUDE_FILES_ONLY = process.env.NATURAL_INCLUDE_FILES_ONLY === "1"; +const SINGLE_PAIR = process.env.NATURAL_SINGLE_PAIR === "1"; + +const scenarios = [ + { + id: "daily-engineering-update", + title: "Daily Engineering Update", + variants: [ + { + id: "a", + prompt: + "Prepare today's engineering update for the team. Look at recent repository work and any relevant company context you can access. Summarize what changed, why it matters, risks, blockers, and suggested next actions. Write the result as a Markdown draft. Do not publish it remotely.", + }, + { + id: "b", + prompt: + "I need a short standup-style update for Locality based on what changed recently. Please discover the relevant context yourself, connect code changes to product or launch work where possible, and produce a grounded Markdown draft. Do not push or update any remote source.", + }, + ], + }, + { + id: "launch-readiness-review", + title: "Launch Readiness Review", + variants: [ + { + id: "a", + prompt: + "We are considering whether Locality is ready for a broader launch. Review recent engineering work and relevant internal context, then draft a launch-readiness assessment with evidence, risks, blockers, and the next validation steps. Do not publish it remotely.", + }, + { + id: "b", + prompt: + "Act like you are preparing a launch gate memo for Locality. Find the relevant project context and recent code changes, decide what is actually proven, what is still unverified, and what should block launch. Produce a concise Markdown memo. Do not push anything.", + }, + ], + }, +]; + +const strategies = [ + { + id: "locality-natural", + label: "Locality Natural", + report: "report-body.md", + allowed: [ + "local git commands in REPO_DIR", + "mounted Locality files under LOCALITY_SOURCE_ROOT", + "`loc` CLI commands", + ], + forbidden: [ + "Notion MCP tools", + "direct Notion API calls", + "publishing or pushing remote changes", + ], + guidance: + "Use Locality-connected files and the `loc` CLI when helpful. For discovery, prefer `loc search ` first, then inspect mounted files. Use `loc info`, `loc status`, and `loc diff` when you need source or sync state. Do not use Notion MCP or direct Notion APIs.", + addDirs: () => [LOCALITY_SOURCE_ROOT], + }, + { + id: "notion-mcp-natural", + label: "Notion MCP Natural", + report: "report-body.md", + allowed: [ + "local git commands in REPO_DIR", + "Notion MCP tools for company context", + ], + forbidden: [ + "mounted Locality files", + "`loc` commands", + "publishing or updating Notion", + ], + guidance: + "Use Notion MCP for company context and local git commands for repository context. Do not read Locality-mounted files and do not use `loc`.", + addDirs: () => [], + }, +]; + +if (INCLUDE_FILES_ONLY) { + strategies.push({ + id: "locality-files-only", + label: "Locality Files Only", + report: "report-body.md", + allowed: [ + "local git commands in REPO_DIR", + "mounted Locality files under LOCALITY_SOURCE_ROOT", + ], + forbidden: [ + "`loc` commands", + "Notion MCP tools", + "direct Notion API calls", + "publishing or pushing remote changes", + ], + guidance: + "Use the mounted Locality files and local git commands only. Do not use `loc`, Notion MCP, or direct Notion APIs.", + addDirs: () => [LOCALITY_SOURCE_ROOT], + }); +} + +function main() { + ensureRepo(); + const batchDir = join(OUT_ROOT, BATCH_ID); + mkdirSync(batchDir, { recursive: true }); + + const selectedScenarios = SINGLE_PAIR ? scenarios.slice(0, 1) : scenarios; + const selectedVariants = (scenario) => + SINGLE_PAIR ? scenario.variants.slice(0, 1) : scenario.variants; + const selectedRuns = SINGLE_PAIR ? 1 : RUNS; + const pairSummaries = []; + + for (const scenario of selectedScenarios) { + for (const variant of selectedVariants(scenario)) { + for (let repeat = 1; repeat <= selectedRuns; repeat += 1) { + const pairDir = join( + batchDir, + scenario.id, + `variant-${variant.id}`, + `repeat-${repeat}`, + ); + mkdirSync(pairDir, { recursive: true }); + writeJson(join(pairDir, "scenario.json"), { + batch_id: BATCH_ID, + scenario: { + id: scenario.id, + title: scenario.title, + }, + variant, + repeat, + model: CODEX_MODEL, + reasoning_effort: CODEX_REASONING_EFFORT, + locality_source_root: LOCALITY_SOURCE_ROOT, + }); + + const runSummaries = []; + for (const strategy of strategiesForPair(scenario, strategies)) { + const runDir = join(pairDir, strategy.id); + mkdirSync(runDir, { recursive: true }); + runSummaries.push(runStrategy({ scenario, variant, repeat, strategy, pairDir, runDir })); + } + + const profileSummary = profilePair(pairDir, runSummaries); + const pairSummary = { + scenario_id: scenario.id, + variant_id: variant.id, + repeat, + pair_dir: relative(REPO_DIR, pairDir), + runs: runSummaries, + profile_summary: profileSummary, + }; + pairSummaries.push(pairSummary); + writeJson(join(pairDir, "pair-summary.json"), pairSummary); + } + } + } + + writeBatchSummary(batchDir, pairSummaries); + console.log(`Natural retrieval batch written to ${batchDir}`); +} + +function strategiesForPair(scenario, allStrategies) { + if (!INCLUDE_FILES_ONLY) { + return allStrategies; + } + if (scenario.id !== "daily-engineering-update") { + return allStrategies.filter((strategy) => strategy.id !== "locality-files-only"); + } + return allStrategies; +} + +function runStrategy({ scenario, variant, repeat, strategy, runDir }) { + const promptPath = join(runDir, "prompt.md"); + const reportPath = join(runDir, strategy.report); + const finalPath = join(runDir, "agent-final.md"); + const tracePath = join(runDir, "agent-trace.md"); + const evidencePath = join(runDir, "evidence-manifest.json"); + const eventsPath = join(runDir, "codex-events.jsonl"); + const errPath = join(runDir, "codex.err"); + const summaryPath = join(runDir, "codex-summary.json"); + const eventsTsvPath = join(runDir, "codex-events.tsv"); + const commandPath = join(runDir, "codex-command.txt"); + const localityTracePath = join(runDir, "agent-locality-trace.jsonl"); + const prompt = renderPrompt({ + scenario, + variant, + repeat, + strategy, + reportPath, + evidencePath, + tracePath, + }); + + writeFileSync(promptPath, prompt); + writeJson(join(runDir, "strategy.json"), { + id: strategy.id, + label: strategy.label, + allowed: strategy.allowed, + forbidden: strategy.forbidden, + guidance: strategy.guidance, + }); + + const addDirs = [runDir, ...strategy.addDirs().filter((dir) => existsSync(dir))]; + const startedAt = Date.now(); + const rc = runCodex({ + promptPath, + finalPath, + eventsPath, + errPath, + commandPath, + localityTracePath, + addDirs, + }); + const endedAt = Date.now(); + const durationMs = endedAt - startedAt; + + runIfExists("python3", [ + join(EXPERIMENT_DIR, "scripts/summarize-codex-events.py"), + eventsPath, + summaryPath, + eventsTsvPath, + ]); + runIfExists("python3", [ + join(EXPERIMENT_DIR, "scripts/codex-events-to-trace.py"), + eventsPath, + join(runDir, "codex"), + ]); + + const codexSummary = readJsonIfExists(summaryPath); + const result = { + strategy_id: strategy.id, + strategy_label: strategy.label, + run_dir: relative(REPO_DIR, runDir), + prompt_path: relative(REPO_DIR, promptPath), + report_path: relative(REPO_DIR, reportPath), + evidence_manifest_path: relative(REPO_DIR, evidencePath), + agent_trace_path: relative(REPO_DIR, tracePath), + events_path: relative(REPO_DIR, eventsPath), + summary_path: relative(REPO_DIR, summaryPath), + exit_code: rc, + status: rc === 0 && existsSync(reportPath) ? "ok" : "failed", + duration_ms: durationMs, + usage: codexSummary?.usage ?? {}, + event_counts: codexSummary?.event_counts ?? {}, + item_counts: codexSummary?.item_counts ?? {}, + tool_counts: codexSummary?.tool_counts ?? {}, + errors: codexSummary?.errors ?? [], + }; + writeJson(join(runDir, "run-summary.json"), result); + return result; +} + +function runCodex({ + promptPath, + finalPath, + eventsPath, + errPath, + commandPath, + localityTracePath, + addDirs, +}) { + const addDirsFile = `${commandPath}.add-dirs`; + writeFileSync(addDirsFile, `${addDirs.join("\n")}\n`); + const bash = String.raw` +set -euo pipefail +prompt="$(cat "$PROMPT_PATH")" +cmd=( + codex exec + --json + --model "$CODEX_MODEL" + -c "model_reasoning_effort=\"$CODEX_REASONING_EFFORT\"" + --dangerously-bypass-approvals-and-sandbox + -C "$REPO_DIR" + --output-last-message "$FINAL_PATH" +) +while IFS= read -r add_dir; do + if [ -n "$add_dir" ]; then + cmd+=(--add-dir "$add_dir") + fi +done < "$ADD_DIRS_FILE" +cmd+=("$prompt") +if [ "$CODEX_EXEC_TIMEOUT_SECONDS" = "0" ]; then + run_cmd=("\${cmd[@]}") +elif command -v timeout >/dev/null 2>&1; then + run_cmd=(timeout --kill-after=30s "\${CODEX_EXEC_TIMEOUT_SECONDS}s" "\${cmd[@]}") +else + run_cmd=(python3 "$EXPERIMENT_DIR/scripts/run-with-timeout.py" "$CODEX_EXEC_TIMEOUT_SECONDS" -- "\${cmd[@]}") +fi +{ + printf 'timeout_seconds=%s\n' "$CODEX_EXEC_TIMEOUT_SECONDS" + printf 'codex_command=' + printf '%q ' "\${cmd[@]}" + printf '\nwrapped_command=' + printf '%q ' "\${run_cmd[@]}" + printf '\n' +} > "$COMMAND_PATH" +set +e +set -o pipefail +LOCALITY_TRACE_FILE="$LOCALITY_TRACE_FILE" LOCALITY_TRACE_RUN_ID="$NATURAL_BATCH_ID" \ + "\${run_cmd[@]}" < /dev/null 2> "$ERR_PATH" | python3 "$EXPERIMENT_DIR/scripts/timestamp-jsonl.py" > "$EVENTS_PATH" +pipe_status=("\${PIPESTATUS[@]}") +rc="\${pipe_status[0]}" +set +o pipefail +set -e +exit "$rc" +`; + const result = spawnSync("bash", ["-lc", bash], { + cwd: REPO_DIR, + env: { + ...process.env, + REPO_DIR, + EXPERIMENT_DIR, + PROMPT_PATH: promptPath, + FINAL_PATH: finalPath, + EVENTS_PATH: eventsPath, + ERR_PATH: errPath, + COMMAND_PATH: commandPath, + ADD_DIRS_FILE: addDirsFile, + LOCALITY_TRACE_FILE: localityTracePath, + NATURAL_BATCH_ID: BATCH_ID, + CODEX_MODEL, + CODEX_REASONING_EFFORT, + CODEX_EXEC_TIMEOUT_SECONDS, + }, + stdio: "inherit", + }); + return result.status ?? 1; +} + +function renderPrompt({ scenario, variant, repeat, strategy, reportPath, evidencePath, tracePath }) { + return `You are participating in the Locality natural retrieval benchmark. + +Scenario: ${scenario.title} +Variant: ${variant.id} +Repeat: ${repeat} + +Natural user request: + +${variant.prompt} + +Allowed context sources: +${strategy.allowed.map((item) => `- ${item}`).join("\n")} + +Forbidden context sources/actions: +${strategy.forbidden.map((item) => `- ${item}`).join("\n")} + +Strategy guidance: + +${strategy.guidance} + +Important benchmark rules: + +- Discover relevant company context yourself. Do not assume a known Notion URL, page title, or mounted path. +- Do not use precomputed context inventories or previous experiment output directories. +- You may inspect recent git history and repository files as needed. +- Be explicit when evidence is missing or only partially verified. +- Keep the final report human, specific, and grounded in inspected evidence. + +Required outputs: + +1. Write the final Markdown report to: + ${reportPath} +2. Write a compact evidence manifest JSON to: + ${evidencePath} +3. Write an agent trace Markdown file to: + ${tracePath} + +Evidence manifest shape: + +{ + "task": "${scenario.id}", + "strategy": "${strategy.id}", + "evidence": [ + { + "kind": "git_commit | locality_file | notion_mcp_page | repo_file | other", + "id": "stable identifier when available", + "path": "local path when available", + "title": "source title when available", + "reason": "why this evidence mattered" + } + ], + "limitations": [ + "what you could not verify" + ] +} +`; +} + +function profilePair(pairDir, runSummaries) { + const locality = runSummaries.find((run) => run.strategy_id === "locality-natural"); + const mcp = runSummaries.find((run) => run.strategy_id === "notion-mcp-natural"); + if (!locality || !mcp) { + return null; + } + const outDir = join(pairDir, "profile-locality-vs-mcp"); + mkdirSync(outDir, { recursive: true }); + const result = spawnSync( + "node", + [ + join(REPO_DIR, "experiment/agent-conversation-profile.mjs"), + "--left", + join(REPO_DIR, locality.events_path), + "--left-label", + "locality-natural", + "--right", + join(REPO_DIR, mcp.events_path), + "--right-label", + "notion-mcp-natural", + "--out", + outDir, + ], + { cwd: REPO_DIR, encoding: "utf8" }, + ); + if (result.status !== 0) { + writeFileSync(join(outDir, "profile-error.log"), `${result.stdout}\n${result.stderr}`); + return { status: "failed", out_dir: relative(REPO_DIR, outDir) }; + } + return { + status: "ok", + out_dir: relative(REPO_DIR, outDir), + summary_md: relative(REPO_DIR, join(outDir, "summary.md")), + summary_json: relative(REPO_DIR, join(outDir, "summary.json")), + }; +} + +function writeBatchSummary(batchDir, pairSummaries) { + writeJson(join(batchDir, "batch-summary.json"), { + batch_id: BATCH_ID, + model: CODEX_MODEL, + reasoning_effort: CODEX_REASONING_EFFORT, + runs_per_prompt: RUNS, + include_files_only: INCLUDE_FILES_ONLY, + generated_at: new Date().toISOString(), + pairs: pairSummaries, + }); + + const rows = [ + [ + "scenario", + "variant", + "repeat", + "strategy", + "status", + "duration_ms", + "input_tokens", + "cached_input_tokens", + "output_tokens", + "mcp_tool_calls", + "report_path", + ], + ]; + for (const pair of pairSummaries) { + for (const run of pair.runs) { + rows.push([ + pair.scenario_id, + pair.variant_id, + String(pair.repeat), + run.strategy_id, + run.status, + String(run.duration_ms), + String(run.usage?.input_tokens ?? ""), + String(run.usage?.cached_input_tokens ?? ""), + String(run.usage?.output_tokens ?? ""), + String(run.tool_counts?.mcp_tool_call ?? ""), + run.report_path, + ]); + } + } + writeFileSync(join(batchDir, "batch-summary.tsv"), rows.map((row) => row.join("\t")).join("\n") + "\n"); + + const md = [ + "# Natural Retrieval Batch Summary", + "", + `Batch: \`${BATCH_ID}\``, + "", + "| Scenario | Variant | Repeat | Strategy | Status | Wall time | Input tokens | Output tokens | MCP calls | Report |", + "| --- | --- | ---: | --- | --- | ---: | ---: | ---: | ---: | --- |", + ]; + for (const pair of pairSummaries) { + for (const run of pair.runs) { + md.push( + `| ${pair.scenario_id} | ${pair.variant_id} | ${pair.repeat} | ${run.strategy_id} | ${run.status} | ${formatMs(run.duration_ms)} | ${run.usage?.input_tokens ?? ""} | ${run.usage?.output_tokens ?? ""} | ${run.tool_counts?.mcp_tool_call ?? ""} | ${run.report_path} |`, + ); + } + } + writeFileSync(join(batchDir, "batch-summary.md"), md.join("\n") + "\n"); +} + +function ensureRepo() { + if (!existsSync(REPO_DIR)) { + throw new Error(`REPO_DIR does not exist: ${REPO_DIR}`); + } +} + +function runIfExists(command, args) { + const result = spawnSync(command, args, { + cwd: REPO_DIR, + encoding: "utf8", + stdio: "inherit", + }); + return result.status ?? 1; +} + +function readJsonIfExists(path) { + if (!existsSync(path)) { + return null; + } + return JSON.parse(readFileSync(path, "utf8")); +} + +function writeJson(path, value) { + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, JSON.stringify(value, null, 2) + "\n"); +} + +function formatMs(ms) { + return ms < 1000 ? `${ms}ms` : `${(ms / 1000).toFixed(1)}s`; +} + +function positiveInteger(value, name) { + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed <= 0) { + throw new Error(`${name} must be a positive integer`); + } + return parsed; +} + +function utcStamp() { + return new Date().toISOString().replace(/[-:]/g, "").replace(/\.\d{3}Z$/, "Z"); +} + +main(); From 5f97e67f7264778205472270d6e2ca39a98d933c Mon Sep 17 00:00:00 2001 From: Sarthak Agarwal Date: Wed, 22 Jul 2026 22:25:18 +0530 Subject: [PATCH 4/6] Fix natural experiment command expansion --- .../scripts/run-natural-retrieval-batch.mjs | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs b/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs index c6b22d37..458ff7dd 100755 --- a/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs +++ b/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs @@ -298,26 +298,26 @@ while IFS= read -r add_dir; do done < "$ADD_DIRS_FILE" cmd+=("$prompt") if [ "$CODEX_EXEC_TIMEOUT_SECONDS" = "0" ]; then - run_cmd=("\${cmd[@]}") + run_cmd=("${"$"}{cmd[@]}") elif command -v timeout >/dev/null 2>&1; then - run_cmd=(timeout --kill-after=30s "\${CODEX_EXEC_TIMEOUT_SECONDS}s" "\${cmd[@]}") + run_cmd=(timeout --kill-after=30s "${"$"}{CODEX_EXEC_TIMEOUT_SECONDS}s" "${"$"}{cmd[@]}") else - run_cmd=(python3 "$EXPERIMENT_DIR/scripts/run-with-timeout.py" "$CODEX_EXEC_TIMEOUT_SECONDS" -- "\${cmd[@]}") + run_cmd=(python3 "$EXPERIMENT_DIR/scripts/run-with-timeout.py" "$CODEX_EXEC_TIMEOUT_SECONDS" -- "${"$"}{cmd[@]}") fi { printf 'timeout_seconds=%s\n' "$CODEX_EXEC_TIMEOUT_SECONDS" printf 'codex_command=' - printf '%q ' "\${cmd[@]}" + printf '%q ' "${"$"}{cmd[@]}" printf '\nwrapped_command=' - printf '%q ' "\${run_cmd[@]}" + printf '%q ' "${"$"}{run_cmd[@]}" printf '\n' } > "$COMMAND_PATH" set +e set -o pipefail LOCALITY_TRACE_FILE="$LOCALITY_TRACE_FILE" LOCALITY_TRACE_RUN_ID="$NATURAL_BATCH_ID" \ - "\${run_cmd[@]}" < /dev/null 2> "$ERR_PATH" | python3 "$EXPERIMENT_DIR/scripts/timestamp-jsonl.py" > "$EVENTS_PATH" -pipe_status=("\${PIPESTATUS[@]}") -rc="\${pipe_status[0]}" + "${"$"}{run_cmd[@]}" < /dev/null 2> "$ERR_PATH" | python3 "$EXPERIMENT_DIR/scripts/timestamp-jsonl.py" > "$EVENTS_PATH" +pipe_status=("${"$"}{PIPESTATUS[@]}") +rc="${"$"}{pipe_status[0]}" set +o pipefail set -e exit "$rc" From 79f16250eb50a0ac10d13a6d821d8ec500ef6926 Mon Sep 17 00:00:00 2001 From: Sarthak Agarwal Date: Wed, 22 Jul 2026 23:05:26 +0530 Subject: [PATCH 5/6] Add normalized natural retrieval summaries --- experiment/locality-mcp-comparison/README.md | 7 + .../scripts/run-natural-retrieval-batch.mjs | 4 + .../scripts/summarize-natural-batch.mjs | 263 ++++++++++++++++++ 3 files changed, 274 insertions(+) create mode 100644 experiment/locality-mcp-comparison/scripts/summarize-natural-batch.mjs diff --git a/experiment/locality-mcp-comparison/README.md b/experiment/locality-mcp-comparison/README.md index 7b37f73f..fb7bcdab 100644 --- a/experiment/locality-mcp-comparison/README.md +++ b/experiment/locality-mcp-comparison/README.md @@ -191,6 +191,13 @@ Optional files-only ablation for the daily-engineering scenario: NATURAL_INCLUDE_FILES_ONLY=1 ./experiment/locality-mcp-comparison/run-natural-retrieval-batch.sh ``` +Natural retrieval batches also write: + +- `batch-summary.md` - raw per-run status, wall time, token, and MCP-call rows. +- `normalized-summary.md` - mean, median, min, max, and pairwise MCP-over-Locality ratios. +- `normalized-summary.json` - machine-readable normalized stats for notebooks or dashboards. +- `profile-locality-vs-mcp/summary.md` under each matched pair - per-pair tool grouping and viewer file links. + ## Artifacts Each run writes to: diff --git a/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs b/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs index 458ff7dd..faaa71d6 100755 --- a/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs +++ b/experiment/locality-mcp-comparison/scripts/run-natural-retrieval-batch.mjs @@ -174,6 +174,10 @@ function main() { } writeBatchSummary(batchDir, pairSummaries); + runIfExists("node", [ + join(EXPERIMENT_DIR, "scripts/summarize-natural-batch.mjs"), + batchDir, + ]); console.log(`Natural retrieval batch written to ${batchDir}`); } diff --git a/experiment/locality-mcp-comparison/scripts/summarize-natural-batch.mjs b/experiment/locality-mcp-comparison/scripts/summarize-natural-batch.mjs new file mode 100644 index 00000000..c7680f7d --- /dev/null +++ b/experiment/locality-mcp-comparison/scripts/summarize-natural-batch.mjs @@ -0,0 +1,263 @@ +#!/usr/bin/env node + +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; + +function main(argv) { + const batchDir = resolve(argv[0] ?? ""); + if (!argv[0] || !existsSync(join(batchDir, "batch-summary.json"))) { + console.error("Usage: node summarize-natural-batch.mjs "); + process.exit(2); + } + + const batch = JSON.parse(readFileSync(join(batchDir, "batch-summary.json"), "utf8")); + const summary = buildNormalizedSummary(batch); + writeFileSync( + join(batchDir, "normalized-summary.json"), + JSON.stringify(summary, null, 2) + "\n", + ); + writeFileSync(join(batchDir, "normalized-summary.md"), renderMarkdown(summary)); + console.log(`Wrote normalized natural retrieval summary to ${batchDir}`); +} + +function buildNormalizedSummary(batch) { + const runs = flattenRuns(batch); + const strategies = groupBy(runs, (run) => run.strategy_id); + const scenarios = groupBy(runs, (run) => `${run.scenario_id}/${run.strategy_id}`); + const pairs = pairRuns(runs); + const pairRatios = pairs + .map((pair) => ratioForPair(pair)) + .filter(Boolean); + + return { + batch_id: batch.batch_id, + model: batch.model, + reasoning_effort: batch.reasoning_effort, + generated_at: new Date().toISOString(), + run_count: runs.length, + strategy_aggregates: Object.fromEntries( + [...strategies.entries()].map(([strategy, rows]) => [ + strategy, + aggregateRuns(rows), + ]), + ), + scenario_strategy_aggregates: Object.fromEntries( + [...scenarios.entries()].map(([key, rows]) => [ + key, + aggregateRuns(rows), + ]), + ), + pairwise_mcp_over_locality: aggregateRatios(pairRatios), + pairwise_samples: pairRatios, + slowest_runs: [...runs] + .sort((left, right) => right.duration_ms - left.duration_ms) + .slice(0, 8), + }; +} + +function flattenRuns(batch) { + return batch.pairs.flatMap((pair) => + pair.runs.map((run) => ({ + scenario_id: pair.scenario_id, + variant_id: pair.variant_id, + repeat: pair.repeat, + strategy_id: run.strategy_id, + status: run.status, + duration_ms: numberOrZero(run.duration_ms), + input_tokens: numberOrZero(run.usage?.input_tokens), + cached_input_tokens: numberOrZero(run.usage?.cached_input_tokens), + output_tokens: numberOrZero(run.usage?.output_tokens), + mcp_tool_calls: numberOrZero(run.tool_counts?.mcp_tool_call), + report_path: run.report_path, + })), + ); +} + +function aggregateRuns(rows) { + return { + count: rows.length, + ok: rows.filter((run) => run.status === "ok").length, + failed: rows.filter((run) => run.status !== "ok").length, + duration_ms: stats(rows.map((run) => run.duration_ms)), + input_tokens: stats(rows.map((run) => run.input_tokens)), + cached_input_tokens: stats(rows.map((run) => run.cached_input_tokens)), + output_tokens: stats(rows.map((run) => run.output_tokens)), + mcp_tool_calls: stats(rows.map((run) => run.mcp_tool_calls)), + }; +} + +function pairRuns(runs) { + const grouped = groupBy( + runs, + (run) => `${run.scenario_id}/${run.variant_id}/${run.repeat}`, + ); + return [...grouped.values()] + .map((rows) => ({ + locality: rows.find((run) => run.strategy_id === "locality-natural"), + mcp: rows.find((run) => run.strategy_id === "notion-mcp-natural"), + })) + .filter((pair) => pair.locality && pair.mcp); +} + +function ratioForPair(pair) { + const { locality, mcp } = pair; + if ( + locality.status !== "ok" || + mcp.status !== "ok" || + locality.duration_ms <= 0 || + locality.input_tokens <= 0 + ) { + return null; + } + return { + scenario_id: locality.scenario_id, + variant_id: locality.variant_id, + repeat: locality.repeat, + duration_ratio: mcp.duration_ms / locality.duration_ms, + input_token_ratio: mcp.input_tokens / locality.input_tokens, + output_token_ratio: + locality.output_tokens > 0 ? mcp.output_tokens / locality.output_tokens : null, + mcp_tool_calls: mcp.mcp_tool_calls, + locality_duration_ms: locality.duration_ms, + mcp_duration_ms: mcp.duration_ms, + locality_input_tokens: locality.input_tokens, + mcp_input_tokens: mcp.input_tokens, + }; +} + +function aggregateRatios(rows) { + return { + count: rows.length, + duration_ratio: stats(rows.map((row) => row.duration_ratio)), + input_token_ratio: stats(rows.map((row) => row.input_token_ratio)), + output_token_ratio: stats( + rows.map((row) => row.output_token_ratio).filter((value) => value !== null), + ), + mcp_tool_calls: stats(rows.map((row) => row.mcp_tool_calls)), + }; +} + +function renderMarkdown(summary) { + const lines = [ + "# Normalized Natural Retrieval Summary", + "", + `Batch: \`${summary.batch_id}\``, + `Model: \`${summary.model}\``, + `Reasoning effort: \`${summary.reasoning_effort}\``, + "", + "## Strategy Aggregates", + "", + "| Strategy | Runs | OK | Mean wall | Median wall | Min wall | Max wall | Mean input | Median input | Mean output | Mean MCP calls |", + "| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ]; + + for (const [strategy, aggregate] of Object.entries(summary.strategy_aggregates)) { + lines.push( + `| ${strategy} | ${aggregate.count} | ${aggregate.ok} | ${formatMs(aggregate.duration_ms.mean)} | ${formatMs(aggregate.duration_ms.median)} | ${formatMs(aggregate.duration_ms.min)} | ${formatMs(aggregate.duration_ms.max)} | ${formatInteger(aggregate.input_tokens.mean)} | ${formatInteger(aggregate.input_tokens.median)} | ${formatInteger(aggregate.output_tokens.mean)} | ${formatNumber(aggregate.mcp_tool_calls.mean)} |`, + ); + } + + lines.push( + "", + "## Pairwise MCP Over Locality", + "", + "Ratios compare the matched MCP run against the Locality run for the same scenario, prompt variant, and repeat. A value above `1.00x` means the MCP run used more of that resource.", + "", + "| Metric | Count | Mean | Median | Min | Max |", + "| --- | ---: | ---: | ---: | ---: | ---: |", + ratioRow("Wall time", summary.pairwise_mcp_over_locality.duration_ratio), + ratioRow("Input tokens", summary.pairwise_mcp_over_locality.input_token_ratio), + ratioRow("Output tokens", summary.pairwise_mcp_over_locality.output_token_ratio), + ratioValueRow("MCP tool calls", summary.pairwise_mcp_over_locality.mcp_tool_calls), + "", + "## Scenario Aggregates", + "", + "| Scenario / Strategy | Runs | Mean wall | Median wall | Mean input | Mean MCP calls |", + "| --- | ---: | ---: | ---: | ---: | ---: |", + ); + + for (const [key, aggregate] of Object.entries(summary.scenario_strategy_aggregates)) { + lines.push( + `| ${key} | ${aggregate.count} | ${formatMs(aggregate.duration_ms.mean)} | ${formatMs(aggregate.duration_ms.median)} | ${formatInteger(aggregate.input_tokens.mean)} | ${formatNumber(aggregate.mcp_tool_calls.mean)} |`, + ); + } + + lines.push( + "", + "## Slowest Runs", + "", + "| Scenario | Variant | Repeat | Strategy | Wall time | Input tokens | MCP calls | Report |", + "| --- | --- | ---: | --- | ---: | ---: | ---: | --- |", + ); + for (const run of summary.slowest_runs) { + lines.push( + `| ${run.scenario_id} | ${run.variant_id} | ${run.repeat} | ${run.strategy_id} | ${formatMs(run.duration_ms)} | ${formatInteger(run.input_tokens)} | ${formatNumber(run.mcp_tool_calls)} | ${run.report_path} |`, + ); + } + + return `${lines.join("\n")}\n`; +} + +function ratioRow(label, metric) { + return `| ${label} | ${metric.count} | ${formatRatio(metric.mean)} | ${formatRatio(metric.median)} | ${formatRatio(metric.min)} | ${formatRatio(metric.max)} |`; +} + +function ratioValueRow(label, metric) { + return `| ${label} | ${metric.count} | ${formatNumber(metric.mean)} | ${formatNumber(metric.median)} | ${formatNumber(metric.min)} | ${formatNumber(metric.max)} |`; +} + +function groupBy(values, keyFn) { + const groups = new Map(); + for (const value of values) { + const key = keyFn(value); + const group = groups.get(key) ?? []; + group.push(value); + groups.set(key, group); + } + return groups; +} + +function stats(values) { + const cleaned = values.filter((value) => Number.isFinite(value)); + if (cleaned.length === 0) { + return { count: 0, mean: 0, median: 0, min: 0, max: 0, stddev: 0 }; + } + const sorted = [...cleaned].sort((left, right) => left - right); + const mean = sorted.reduce((sum, value) => sum + value, 0) / sorted.length; + const variance = + sorted.reduce((sum, value) => sum + (value - mean) ** 2, 0) / sorted.length; + return { + count: sorted.length, + mean, + median: + sorted.length % 2 === 0 + ? (sorted[sorted.length / 2 - 1] + sorted[sorted.length / 2]) / 2 + : sorted[Math.floor(sorted.length / 2)], + min: sorted[0], + max: sorted[sorted.length - 1], + stddev: Math.sqrt(variance), + }; +} + +function numberOrZero(value) { + const number = Number(value); + return Number.isFinite(number) ? number : 0; +} + +function formatMs(ms) { + return `${(ms / 1000).toFixed(1)}s`; +} + +function formatInteger(value) { + return Math.round(value).toLocaleString("en-US"); +} + +function formatNumber(value) { + return Number(value).toFixed(2); +} + +function formatRatio(value) { + return `${Number(value).toFixed(2)}x`; +} + +main(process.argv.slice(2)); From 1c0e8a63086a3febb7edcfbada34b1dd84d4c239 Mon Sep 17 00:00:00 2001 From: Sarthak Agarwal Date: Wed, 22 Jul 2026 23:07:05 +0530 Subject: [PATCH 6/6] Ignore generated experiment runs --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 458131fd..1f1c3eb8 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,8 @@ /target/ node_modules/ dist/ +/experiment/runs/ +/experiment/runs-2/ /.loc/ /.tmp/ /.worktrees/