Fix/pre public accuracy fixes

README.md

@@ -68,7 +68,7 @@ A four-agent scenario (caching architecture migration, Slack signal, GitHub PR c
 
 ## How it works on day one
 
-The guardrail works immediately, without seeding historical context.
+The guardrail works immediately — it requires at least one logged decision to return a meaningful assessment, but no historical seeding beyond that.
 
 **Step 1 — Agents log decisions as they make them.**
 
@@ -92,7 +92,7 @@ Any agent in any session, hours or weeks later, checks the brain before a signif
 
 **Step 3 — Drift alerts fire when signals contradict existing decisions.**
 
-When a Slack message, GitHub PR, or agent signal contradicts a prior decision, the drift detector surfaces a confirmed alert. Agents query it at session start. The alert is visible in the web UI. A webhook delivers it to Slack or a coordinator agent in real time.
+When a Slack message, meeting signal, or agent signal contradicts a prior decision, the drift detector surfaces a confirmed alert. Agents query it at session start. The alert is visible in the web UI. A webhook delivers it to Slack or a coordinator agent in real time. (GitHub PR events are ingested but skipped by drift detection to reduce false positives from PR debate — see Known Limitations.)
 
 That is the full loop. It runs on agent write-back alone. No ADR seeding, no GitHub connector, no Slack listener required to start. Those layers make the guardrail stronger. They are not prerequisites.
 
@@ -144,10 +144,10 @@ A/B comparison: same model, same 3 tasks, same LLM judge. The only difference is
 
 | Metric | Cold start | Brain-assisted | Delta |
 |---|---|---|---|
-| Decision alignment rate | 17% (1/6) | **100% (6/6)** | +5 decisions |
-| Contradiction rate | 67% (4/6) | **0% (0/6)** | −4 contradictions |
+| Decision alignment rate | 17% (1/6) | 100% (6/6) | +5 decisions |
+| Contradiction rate | 67% (4/6) | 0% (0/6) | −4 contradictions |
 
-Without context the agent picked the wrong validation library, wrong rate limiting layer, wrong error format, and wrong auth approach on 4 of 6 relevant decisions. With brain context, all 6 were correct. Caveat: several seeded decisions (JWT auth, Zod validation, RFC 7807 error format) are established best practices the model may already lean toward; the delta may partially reflect model priors, not brain context alone.
+**Interpret with caution.** This eval uses n=6 seeded decisions, several of which (JWT auth, Zod validation, RFC 7807 error format) are established best practices that the model may already lean toward. The delta conflates brain contribution with model prior — a controlled experiment isolating the two would require seeding decisions that contradict the model's defaults. This eval demonstrates the mechanism works; it does not measure production efficacy.
 
 ### Pipeline and retrieval
 
@@ -444,7 +444,7 @@ def run_task(task: str) -> str:
 
 **Key rules:**
 - Use the task description as the query, not a generic "recent decisions". Specific queries return what matters; broad queries flood context.
-- Log immediately after each task, not just at session end. A decision lost when a session crashes is unrecoverable.
+- Log immediately after each task, not just at session end. A decision lost when a session crashes is unrecoverable. Use a distinct `session_id` per task — the endpoint is immutable, a second call with the same `session_id` returns 409.
 - Never raise on brain failure. The brain enhances context; it is not a dependency.
 
 
@@ -505,7 +505,7 @@ Payload includes: `alert_id`, `project_id`, `risk`, `challenged_decision_summary
 
 | | Ollama (default) | Anthropic |
 |---|---|---|
-| LLM | qwen2.5:7b (extraction) + llama3.1:8b (query) | Claude Haiku |
+| LLM | llama3.1:8b (extraction + query) | Claude Haiku 4.5 (extraction) + Claude Sonnet 4.6 (query) |
 | Embeddings | nomic-embed-text:v1.5 | nomic-embed-text:v1.5 (Ollama still required) |
 | Avg query latency | ~14s p50, ~28s p95 | ~2s |
 | Cost | Free | ~$5–15/month |
@@ -524,12 +524,15 @@ bash demo.sh verify    # checks all services, auth, query, CORS
 End-to-end evals:
 
 ```bash
-npm run eval:integration -w apps/api   # 33 checks, full pipeline
-npm run eval:mcp -w apps/mcp           # 8 checks, all MCP tools
-npm run eval:cross-session -w apps/api # 5 queries, cross-session recall
-npm run eval:multi-agent -w apps/api   # 58 checks, guardrail scenario (~4 min)
+npm run eval:integration -w apps/api    # 33 checks, full pipeline (pre-demo smoke test)
+npm run eval:mcp -w apps/mcp            # 8 checks, all MCP tools
+npm run eval:cross-session -w apps/api  # 5 queries, cross-session recall
+npm run eval:multi-agent -w apps/api    # guardrail scenario, 4 agents (~4 min)
+npm run eval:enterprise -w apps/api     # 3 tenants, all ingest paths, isolation (~8 min)
 ```
 
+The load-bearing checks across the suite: `eval-multi-agent` A7 (LLM-confirmed drift alert), A8 (alert challenges ≥2 decisions via graph traversal), A_FR3/A_FR4 (impact analysis surfaces a contradiction for an agent that skipped pre-flight). `eval-enterprise` A23/A24/A42 (cross-tenant isolation on both read paths), A16 (GitHub webhook with bad HMAC → 401). These are the checks hardest to pass with a broken core mechanism.
+
 ---
 
 ## Is purpl-brain the right tool?
@@ -553,7 +556,7 @@ At small scale that works. As the number of agents, sessions, and decisions grow
 ## Known limitations
 
 - **Impact analysis uses a hybrid risk floor.** Any decision with an open drift alert is floored at `high`; any high-confidence decision is floored at `medium`. The LLM can raise tiers above the floor but cannot lower them below it. Decision age and downstream reference count are not yet used as floor inputs.
-- **Drift detection skips GitHub-sourced decisions** to reduce false positives from PR noise.
+- **Drift detection skips GitHub-sourced events** to reduce false positives from PR debate noise. GitHub events are ingested and queryable but do not trigger drift alerts.
 - **Human communication ingestion is partial.** Agent write-back and document ingestion are tested. GitHub webhook ingestion is implemented. Slack ingestion is implemented but thread replies are not fetched, and decision extraction yield from conversational PR threads is low.
 - **The Stop hook catches sessions that close cleanly.** Crashed or force-killed sessions do not fire the hook. Mid-session compaction before close is an open problem.
 - **Logged decision quality depends on timing.** A decision logged when it is made is more complete than one reconstructed at session close.
@@ -565,6 +568,7 @@ At small scale that works. As the number of agents, sessions, and decisions grow
 | Audience | Document |
 |----------|----------|
 | Architecture deep dive | [docs/technical/architecture.md](docs/technical/architecture.md) |
+| Drift detection workflow | [docs/technical/drift-workflow.md](docs/technical/drift-workflow.md) |
 | Agent write-back design | [docs/technical/adrs/004-agent-decision-trails.md](docs/technical/adrs/004-agent-decision-trails.md) |
 | Embedding model selection | [docs/technical/adrs/005-embedding-model.md](docs/technical/adrs/005-embedding-model.md) |
 

apps/api/src/routes/brain.ts

@@ -184,6 +184,11 @@ export const brainRoutes: FastifyPluginAsync = async (fastify) => {
       }
 
       await redis.sadd(PROCESSED_SET, sourceId);
+      // Keep PROCESSED_SET TTL in sync with the webhook dedup path (30 days).
+      // PROCESSED_SET is a shared key — expire resets TTL on the whole set, not
+      // per member. Consistent across all write paths so expiry doesn't depend
+      // on webhook activity alone.
+      await redis.expire(PROCESSED_SET, 60 * 60 * 24 * 30);
 
       fastify.log.info(
         { project_id, title, chunks: chunks.length, format: parsed.format, speakers: parsed.speakers },
@@ -357,7 +362,13 @@ export const brainRoutes: FastifyPluginAsync = async (fastify) => {
       };
 
       await redis.xadd(STREAMS.EXTRACTED, "*", "result", JSON.stringify(extractionResult));
+      // Agent-log dedup is intentionally 409 (not REPLACE like documents) because
+      // a session_id represents a completed reasoning trace — overwriting it would
+      // silently discard the original decisions. Callers that need to update a
+      // session should log a new session with a new session_id and a SUPERSEDES
+      // decision referencing the old one.
       await redis.sadd(PROCESSED_SET, sourceId);
+      await redis.expire(PROCESSED_SET, 60 * 60 * 24 * 30);
 
       fastify.log.info(
         { session_id: log.session_id, decisions: log.decisions.length, project_id: log.project_id },

apps/api/src/routes/ingest.ts

@@ -93,6 +93,7 @@ export const ingestRoutes: FastifyPluginAsync = async (fastify) => {
       }
 
       await redis.sadd(PROCESSED_SET, sourceId);
+      await redis.expire(PROCESSED_SET, 60 * 60 * 24 * 30);
 
       fastify.log.info(
         { project_id, title: resolvedTitle, chunks: chunks.length, type: resolvedType },
@@ -178,6 +179,7 @@ export const ingestRoutes: FastifyPluginAsync = async (fastify) => {
         for (const sid of uniqueSourceIds) {
           await redis.sadd(PROCESSED_SET, sid);
         }
+        await redis.expire(PROCESSED_SET, 60 * 60 * 24 * 30);
 
         fastify.log.info(
           { repo, project_id, files: uniqueSourceIds.length, chunks: events.length },

apps/api/src/scripts/eval/eval-enterprise.ts

@@ -972,13 +972,22 @@ async function main() {
     // we wait one more tick to be sure the extractor has finished.
     await sleep(15000, "extraction + drift settling");
     const whAlerts = await pollForDriftAlerts(TENANT_WAREHOUSE, 1, 20000);
-    // Don't fail hard if extraction didn't pull a Decision from the transcript —
-    // this is best-effort. Assert ≥0 and report what we found.
+    // Hard gate: the endpoint must return a valid response.
     check(
-      "A31: warehouse tenant drift-alerts endpoint returns successfully",
+      "A31: warehouse tenant drift-alerts endpoint returns valid response",
       Array.isArray(whAlerts),
       `alerts=${whAlerts.length}`,
     );
+    // Behavioral gate: at least one drift alert should have fired from the
+    // meeting decision vs. Jira ticket contradiction. This requires the
+    // transcript extractor to have yielded a confirmed Decision node — if
+    // extraction failed (thin corpus, low-signal transcript), this check
+    // will fail and surface the gap.
+    check(
+      "A31: ≥1 drift alert fires from meeting-transcript decision contradicted by Jira",
+      whAlerts.length >= 1,
+      `alerts=${whAlerts.length} — if FAIL: transcript extraction did not yield a confirmed Decision; check corpus stats and extraction yield`
+    );
   }
 
   // ── Phase 12: False-positive drift — should NOT fire ───────────────────────

apps/api/src/scripts/eval/eval-integration.ts

@@ -443,25 +443,33 @@ async function main() {
       ...AGENT_LOG_S2,
       session_id: `sess_int_s2_dedup_${RUN_ID}`,
     };
+    // Capture alert count before re-ingest so we can compare after
+    const alertCountBefore = (await get<{ alerts: Array<Record<string, unknown>> }>(
+      `/brain/drift-alerts?project_id=${encodeURIComponent(PROJECT_ID)}`
+    )).body.alerts?.length ?? 0;
+
     const reingest = await post<{ ok: boolean }>(
       "/brain/agent-log", dedupeLog, true
     );
     check("re-ingest of identical content returns 200 or 202", [200, 202].includes(reingest.status),
       `status=${reingest.status}`);
 
-    // Short wait for the drift detector to process
+    // Wait for the drift detector to process the re-ingested content
     await sleep(30000);
 
-    const alertsAfter = await get<{ alerts: Array<Record<string, unknown>> }>(
-      `/brain/drift-alerts?project_id=${encodeURIComponent(PROJECT_ID)}`
-    );
-    const alertsBefore = (await get<{ alerts: Array<Record<string, unknown>> }>(
+    const alertsAfterBody = (await get<{ alerts: Array<Record<string, unknown>> }>(
       `/brain/drift-alerts?project_id=${encodeURIComponent(PROJECT_ID)}`
     )).body.alerts ?? [];
+    const alertCountAfter = alertsAfterBody.length;
+
+    // Fingerprint dedup: re-ingesting the same content should produce zero new alerts
+    check("re-ingest does not create new drift alerts (fingerprint dedup working)",
+      alertCountAfter <= alertCountBefore,
+      `before=${alertCountBefore} after=${alertCountAfter} — new alerts=${alertCountAfter - alertCountBefore}`);
 
-    const fingerprints = alertsBefore.map((a) => a.fingerprint as string);
+    const fingerprints = alertsAfterBody.map((a) => a.fingerprint as string).filter(Boolean);
     const unique = new Set(fingerprints);
-    check("no duplicate fingerprints after re-ingest", fingerprints.length === unique.size,
+    check("no duplicate fingerprints in alert list", fingerprints.length === unique.size,
       `total=${fingerprints.length} unique=${unique.size}`);
   }
 

apps/api/src/scripts/eval/eval-multi-agent.ts

@@ -688,14 +688,6 @@ async function main() {
           distinctDecisions >= 2,
           `distinct decisions challenged=${distinctDecisions} decisions=${[...new Set(alertLinks.map((r) => r.decision_id))].join(",")}`
         );
-        // Specifically check that the RefactorAgent's decision is also challenged
-        const challengesRefactor = alertLinks.some((r) =>
-          String(r.decision_id ?? "").includes("refactor") ||
-          alertLinks.some(() => refactorLogOk) // softer: if refactor logged, at least check alert exists
-        );
-        check("A8: Neo4j has ≥1 CHALLENGES relationship from a DriftAlert",
-          alertLinks.length >= 1,
-          `alert_links=${alertLinks.length}`);
       } catch (e) {
         check("A8: Neo4j drift alert linkage query succeeded", false, String(e));
       }
@@ -764,10 +756,9 @@ async function main() {
     check("A11: SecurityAuditAgent rejection decision logged (200 or 202)",
       [200, 202].includes(secRejection.status),
       `status=${secRejection.status} body=${JSON.stringify(secRejection.body).slice(0, 80)}`);
-    check("A11: rejection decision has 'REJECT' or 'defer' keyword in body",
-      SECURITY_REJECTION_LOG.decisions[0].description.toLowerCase().includes("reject") ||
-      SECURITY_REJECTION_LOG.decisions[0].rationale.toLowerCase().includes("defer"),
-      "decision description should contain REJECT and rationale should contain defer");
+    check("A11: API reports 1 decision logged",
+      secRejection.body.decisions_logged === 1,
+      `decisions_logged=${secRejection.body.decisions_logged}`);
     securityRejectionOk = [200, 202].includes(secRejection.status);
 
     // A12: Session timeline — query + signal observed + rejection logged in order
@@ -833,9 +824,9 @@ async function main() {
       .join(" ");
     const seesRefactorDecision = /acme.cache|extract|refactor|package/i.test(citationText) ||
       (prQuery.body.citations ?? []).some((c) => String(c.source_url ?? "").includes(REFACTOR_SESSION));
-    check("A13: PRReviewAgent sees RefactorAgent's @acme/cache package decision",
-      seesRefactorDecision || refactorLogOk, // relax if pipeline hasn't propagated yet
-      `citation_text=${citationText.slice(0, 150)}`);
+    check("A13: PRReviewAgent sees RefactorAgent's @acme/cache package decision (cross-agent visibility)",
+      seesRefactorDecision,
+      `citation_text=${citationText.slice(0, 150)} — if FAIL: cross-agent pipeline propagation did not complete in time; increase PIPELINE_WAIT_MS`);
 
     // A14: impact analysis — should name the package extraction as affected
     const prImpact = await post<{
@@ -857,9 +848,9 @@ async function main() {
     check("A14: overall_risk is high or critical",
       ["high", "critical"].includes(prImpact.body.overall_risk ?? ""),
       `overall_risk=${prImpact.body.overall_risk}`);
-    check("A14: ≥3 affected decisions (Redis ADR + ioredis + cache key format at minimum)",
-      (prImpact.body.affected_decisions ?? []).length >= 2,
-      `affected=${prImpact.body.affected_decisions?.length}`);
+    check("A14: ≥3 affected decisions (cache-001 Redis ADR + cache-002 key format + cache-003 ioredis)",
+      (prImpact.body.affected_decisions ?? []).length >= 3,
+      `affected=${prImpact.body.affected_decisions?.length} — if FAIL: one of the three seeded cache decisions was not retrieved; check RELEVANCE_THRESHOLD and PIPELINE_WAIT_MS`);
 
     // A14: PRReviewAgent logs its decision
     const prLog = await post<{ ok: boolean; event_id: string; decisions_logged: number }>(

apps/mcp/src/index.ts

@@ -114,10 +114,8 @@ function buildServer(): McpServer {
         "Project namespace to search (e.g. 'my_org_my_repo'). " +
         "Use underscore-separated org_repo format matching how the project was registered."
       ),
-      mode: z.enum(["project", "expertise", "agent_resume"]).optional().describe(
-        "Query mode: 'project' (default) for general project context, " +
-        "'expertise' for cross-project domain knowledge, " +
-        "'agent_resume' to recall what a previous agent session decided."
+      mode: z.enum(["project"]).optional().describe(
+        "Query mode. Currently only 'project' is active — scopes the query to the given project_id."
       ),
     },
     async ({ query, project_id, mode }) => {
@@ -266,7 +264,7 @@ function buildServer(): McpServer {
 
       if (response.affected_decisions.length > 0) {
         lines.push(`### Affected decisions (${response.affected_decisions.length})`);
-        for (const d of response.affected_decisions) {
+        for (const d of response.affected_decisions.slice(0, 3)) {
           lines.push(`\n**${d.summary}** [${d.status}]`);
           if (d.affected_tickets.length > 0) {
             for (const t of d.affected_tickets) {