Fix/pre public accuracy fixes - #18
Open
skalrn wants to merge 4 commits into
Open
Conversation
- Fix PROCESSED_SET missing expire() on agent-log, transcript, document, and crawl-docs paths — only the webhook path set a TTL; all write paths now reset the 30-day TTL consistently - Add comment explaining 409-vs-REPLACE design asymmetry between agent-log (immutable, 409) and document ingest (REPLACE) so integrators don't guess - Remove unimplemented 'expertise' and 'agent_resume' modes from MCP brain_query enum — they silently degraded to 'project'; schema now only advertises what works - Replace bolded 100%/0% headline in agent-value eval table with an 'Interpret with caution' block — n=6 seeded decisions with model-prior confound; numbers should not be cited as production efficacy evidence - Fix Linear listed as a working source in architecture.md — remove from system diagram and canonical event schema, add explicit 'not yet implemented' row in the ingestion table - Clarify QueryMode comment: distinguish active modes from spec-only modes that degrade silently Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
README: - Remove GitHub PR as a drift trigger source — the drift-detector explicitly skips GitHub events; add clarifying parenthetical and fix the Known Limitations wording from "decisions" to "events" - Add cold brain caveat to "guardrail works immediately" claim - Add batching trap warning to REST key rules: session_id must be distinct per task, same id returns 409 - Fix Ollama LLM model names: both default to llama3.1:8b, not qwen2.5:7b; expand Anthropic column to name both models correctly architecture.md: - Rename "Temporal Index" to "Temporal Filtering"; drop unimplemented "updates create new node versions" claim — nodes are mutated in place, valid_from is a timestamp not a bitemporal range - Replace anomaly engine proactive section with accurate description of drift-detector: two-stage (Qdrant cosine + LLM), source skip rules, DriftAlert write path, webhook; remove two specific checks that were never implemented (PR diff file overlap, Jira epic vs ADR rejection) - Fix impact analysis entry to reflect actual endpoint (POST /brain/query with mode:impact, not /brain/impact-analysis) - Add Status column to query modes table; mark expertise and agent_resume as not yet implemented Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New docs/technical/drift-workflow.md covering the complete end-to-end drift workflow: both entry points (pipeline path and explicit signal path), two-stage detection (Stage A semantic + Stage C LLM), alert creation with fingerprint dedup, resolution semantics (keep / under_review / reopen / escalate), supersedes auto-resolution, and a full 10-step sequence diagram (Slack signal → DriftAlert → agent response) - All details grounded in actual code (drift-detector.ts, signal-engine.ts, neo4j.ts, brain.ts routes) - Link added to architecture.md anomaly engine section and README architecture documents table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
eval-multi-agent: - Delete tautological A11 check that read from the test fixture constant instead of the brain (could never fail unless someone edited the fixture) - Replace with A11 assertion that the API reports decisions_logged=1 - Remove dead challengesRefactor variable + redundant weaker A8 assertion (kept the structural A8: distinctDecisions >= 2 via Neo4j graph query) - Fix A13: remove || refactorLogOk escape hatch on cross-agent visibility — if PRReviewAgent can't see RefactorAgent's decision, the check should fail - Fix A14: assertion was >= 2 but label said "≥3 affected decisions" — align assertion to >= 3 (three cache decisions are seeded: cache-001/002/003) eval-integration: - Fix Phase 8 dedup check: previously fetched alertsBefore and alertsAfter at the same time (both after the wait), so the comparison was the list against itself — re-ingest creating a new alert would have passed - Now captures alert count before re-ingest, waits, then asserts count after <= count before (re-ingest must not produce new alerts) eval-enterprise: - Fix A31: Array.isArray(whAlerts) can never fail (pollForDriftAlerts always returns an array); split into two checks: endpoint functional (Array.isArray) + behavioral gate (alerts >= 1) that surfaces the real question: did transcript extraction yield a confirmed Decision? README: - Add eval-enterprise to the eval command table (1575 lines, 3 tenants, all ingest paths — the most comprehensive eval in the suite, previously not mentioned anywhere in the docs) - Add a note on which specific checks are the load-bearing ones across the suite so readers understand which assertions are hard to fake Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.