Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
118 lines (99 loc) · 7.6 KB

ARCHITECTURE.md

File metadata and controls

118 lines (99 loc) · 7.6 KB
Raw

Sekhmet — Architecture

Why two processes, not one

Node Agent (Go) and Brain (Python) are split because they have opposite constraints. The agent runs on every monitored server: it must be a small, dependency-free static binary that can read /proc and execute a handful of privileged operations with minimal footprint. The Brain does the opposite job — heavyweight reasoning, RAG, graph queries, LLM calls — and only needs to run once, centrally. Go and Python were picked to fit each job, not for novelty: Go for cheap concurrency and low-level system access, Python for the LangGraph/LLM ecosystem.

They talk over gRPC bidirectional streaming with mutual TLS. Streaming (not polling) keeps metric delivery cheap at high frequency; mTLS means the Brain only accepts connections from agents holding a cert signed by the same dev CA, and agents only trust a Brain presenting the matching server cert — this is the entire authentication story for v1, deliberately simple.

The safety model (read this first)

The single most important property of this system: the LLM never executes anything. Two independent layers enforce that:

  1. A closed action schema. sekhmet/models/actions.py defines exactly six Pydantic models — one per whitelisted action. An LLM (or the deterministic fallback) can only ever produce a RemediationPlan whose action string and params validate against one of these. Anything else raises InvalidAction and never reaches the wire.
  2. The node agent re-validates independently. Even if the Brain were compromised or buggy, node-agent/internal/remediate re-checks every ActionCommand against its own local, per-host allowlist (sysctl keys and bounds, service names) before touching the system. The agent does not trust the Brain — it is the last line of defense. This is why the adversarial tests in remediate_test.go exist: out-of-range values, non-allowlisted keys/services, wrong signals, and unset/unknown actions must all be rejected at this layer regardless of what upstream claims.

On top of the closed action set sits the policy gate (sekhmet/policy/gate.py) — plain deterministic code, no LLM involved. LOW risk actions auto-approve; MEDIUM/HIGH pause the LangGraph run via NodeInterrupt until a human approves or rejects through the CLI (brain/cli/approve.py). Blast-radius (Neo4j) can escalate an otherwise auto-approved LOW action to HIGH if enough downstream services depend on the affected host.

Every stage — detection, diagnosis, the plan, the gate's decision, the executed result, verification — is written to the Postgres audit log (audit_log table), independent of whether the LLM path or the deterministic fallback path produced it.

The multi-agent design

The top-level LangGraph state machine (sekhmet/graph/pipeline.py) is:

detect -> enrich -> recall -> diagnose -> plan_step -> prepare_plan -> gate -> execute -> verify -> record
                                                                          |(HIGH/MEDIUM, NodeInterrupt)
                                                                          +-> paused, resumable after restart
                                                            ^--------------------------------------+
                                                            (verify failed once: retry with a fresh plan)

Three roles map onto this graph:

  • Coordinator — the graph itself: gate, the routing functions (gate_router, verify_router), and the retry/escalation logic in verify. It owns when to auto-approve, when to pause for a human, and when a failed verification means "try once more" versus "give up and escalate."
  • Diagnostician (sekhmet/agents/diagnostician.py) — the diagnose node. Given the incident plus RAG recall (Qdrant) and blast-radius context (Neo4j), it asks the LLM for a root cause. A bounded retry loop re-prompts on invalid JSON; after MAX_RETRIES it falls back to a deterministic root-cause mapping keyed by metric type, so the pipeline never blocks on a model that's unavailable or unreliable.
  • Remediation (sekhmet/agents/remediation.py) — the plan_step node. Same shape: LLM proposes an action from the whitelist, validated against models/actions.py, with the same bounded-retry-then-deterministic-fallback behavior. It never executes anything — it only ever hands a RemediationPlan back to the graph.

prepare_plan exists as its own node — separate from gate — for a subtle but important reason: a node that raises NodeInterrupt never returns, so anything it computed is lost, not checkpointed. If plan-ID assignment lived inside gate, every resume-after-approval would regenerate a fresh plan ID and silently duplicate the plan. Splitting it out means the plan ID is committed to the checkpoint before the interrupt can happen, so resuming re-enters gate with the same plan ID and correctly reads its now-decided status from Postgres.

Checkpointing (AsyncSqliteSaver, one thread per incident ID) is what makes the human-approval pause real rather than cosmetic: if the Brain process restarts while a HIGH-risk plan is awaiting approval, the pending plan and the paused graph state both survive in Postgres and the sqlite checkpoint file respectively. brain/cli/approve.py doesn't touch either directly — it talks to the Brain's own loopback admin channel (sekhmet/admin.py) so the resume happens inside the same process that holds the live NodeRegistry (the actual open gRPC streams to node agents).

Why each datastore exists

Store Holds Why not something else
PostgreSQL Nodes, incidents, diagnoses, plans, action results, the append-only audit log, policy config This is the system of record for "what happened and why" — relational integrity (foreign keys from plans to incidents) and transactional writes matter more here than query flexibility.
InfluxDB Raw metric time series per node Purpose-built for high-frequency time-series writes (one point per metric family every 5s per node) and range queries (enrich, verify) — Postgres would work but wastes effort reinventing what a TSDB does natively.
Qdrant Embedded resolved incidents + runbook chunks The recall node needs semantic similarity search, not exact match — a vector index is the right tool. (v1 uses a fixed-dimension hashing embedding rather than a pulled Ollama embedding model, to keep the demo runnable with zero extra model downloads; swapping in a real embedding model is a one-line change in llm/ollama_client.py.)
Neo4j (:Server)-[:RUNS]->(:Service)-[:DEPENDS_ON]->(:Service) Blast-radius is inherently a graph traversal (find everything transitively depending on an affected service) — expressing that in SQL recursive CTEs is possible but far less legible than a 2-line Cypher query.
Ollama Local LLM inference Self-hosted-by-design: incident context (hostnames, process names, internal service topology) never leaves the environment.
LangFuse Traces of every agent decision Answers "what did the agent see, recall, conclude, and do" for a specific incident after the fact — essential for trusting an autonomous remediation system, and the whole point of the observability story.

What's deliberately NOT there

No arbitrary shell execution anywhere in the node agent. No web UI beyond the minimal read-only dashboard (Phase 5). No multi-tenancy or auth beyond mTLS. No k8s deployment mode. These are out of scope by design (see Section 1.3 of Claude.md) — the point of this project is a small, safe, well-traced loop, not breadth.