A self-hosted, agentic AI platform that autonomously monitors and heals a fleet of Linux servers.
See Claude.md for the full build specification and ARCHITECTURE.md for the design rationale — the safety model (whitelist + policy gate) is the part worth reading first.
The fleet dashboard (http://localhost:8080) — nodes, pending approvals,
and the incident timeline, all live:
Click any incident to expand its full record — the pre-fix metrics, the LLM's actual diagnosis and reasoning, the action taken, and the outcome, with a direct link to the full LangFuse trace:
- Phase 0 — monorepo scaffold, shared proto (compiles for Go + Python), Docker Compose stack, mTLS dev CA.
- Phase 1 — Node Agent collectors (CPU, memory, FDs, connections, processes, sysctl) streaming to the Brain over gRPC; persisted to InfluxDB.
- Phase 2 — static-threshold detection, the 6 whitelisted remediation executors with adversarial allowlist tests, the deterministic policy gate, full Postgres audit logging, CLI-based human approval.
- Phase 3 — the full LangGraph agentic core (Coordinator / Diagnostician / Remediation), Ollama-backed diagnosis and planning with bounded-retry and deterministic fallback, restart-survivable checkpointing for the human-approval pause.
- Phase 4 — Qdrant recall (resolved cases + runbooks), Neo4j service-dependency blast-radius (escalates risk for wide-blast-radius incidents), MCP tool servers exposing metrics/graph/action-schema to the LLM.
- Phase 5 — minimal read-only dashboard (fleet, pending approvals, incident timeline).
- Node agent reconnects. A dropped gRPC stream (Brain restart, network blip) no longer kills the agent — it redials with exponential backoff (1s→30s capped) indefinitely, and metrics keep accumulating in the buffered channel while disconnected.
RaiseUlimitNofile's service→PID resolution is real, not a stub — queriessystemctl show -p MainPIDand fails loudly (never a fabricated PID) if systemd isn't present or the unit isn't running.- LangFuse tracing is actually wired up and verified, not just scaffolded — see Observability below.
- Automated integration test suite (
brain/tests/integration/test_end_to_end.py) drives the full pipeline for all the canonical incident shapes against the real Postgres/Qdrant/Neo4j/InfluxDB stack, asserting detect→diagnose→ plan→gate→execute→verify and a complete audit trail — this is what caught a real bug (see below). - Found and fixed while building this: a plan whose params failed
whitelist validation (e.g. no offending PID was resolvable) used to crash
the whole LangGraph run with an uncaught exception instead of being
recorded as a graceful rejection. Fixed in
node_execute; regression- tested in the integration suite.
# 1. Generate mTLS certs (once)
./deploy/certs/gen-certs.sh target-1 target-2
# 2. Bring up backing services (Postgres, Qdrant, Neo4j, InfluxDB, Ollama, LangFuse, 2 target containers)
cd deploy/compose && docker compose up -d && cd ../..
# 3. Build the node agent (it's Linux-only — cross-compile from anywhere)
GOOS=linux GOARCH=amd64 go build -o /tmp/sekhmet-agent ./node-agent/cmd/agent
# 4. Set up the Brain's Python environment
python -m venv brain/.venv
brain/.venv/Scripts/pip install -r brain/requirements.txt # Windows
# brain/.venv/bin/pip install -r brain/requirements.txt # Linux/macOS
# 5. Seed the Neo4j test-fleet topology and ingest the starter runbooks
brain/.venv/Scripts/python.exe -m sekhmet.graphdb.seed
brain/.venv/Scripts/python.exe -m sekhmet.rag.ingest_runbooks
# 6. Start the Brain (gRPC on :7443, admin channel on :7444 loopback, dashboard on :8080)
brain/.venv/Scripts/python.exe brain/app.pyThen drop the built agent + certs into a target container and run it (see
deploy/certs/target-1-agent-config.yaml for the config a containerized
agent needs — note the server_name_override for mTLS across
host.docker.internal).
Open http://localhost:8080 for the fleet dashboard.
Actually exhausting fs.file-max for a real FD-exhaustion demo would be
unsafe (it's a host-wide, non-namespaced kernel counter shared by every
container). Instead, brain/tests/integration/trigger_incident.py drives a
synthetic-but-realistic metric batch through the exact same code path a real
node agent's stream would — detection, diagnosis, planning, the policy gate,
real dispatch to the real connected node agent, and verification:
brain/.venv/Scripts/python.exe brain/tests/integration/trigger_incident.py --host target-1 --type fd # LOW risk, auto-approved
brain/.venv/Scripts/python.exe brain/tests/integration/trigger_incident.py --host target-1 --type memory # MEDIUM risk, auto-approved
brain/.venv/Scripts/python.exe brain/tests/integration/trigger_incident.py --host target-1 --type cpu # HIGH risk, pauses for approval
brain/.venv/Scripts/python.exe brain/tests/integration/trigger_incident.py --host target-1 --type connections # MEDIUM riskFor a HIGH-risk plan, approve or reject it via the CLI (this resumes the paused LangGraph run against the Brain's live connections, even across a Brain restart in between):
brain/.venv/Scripts/python.exe brain/cli/approve.py list
brain/.venv/Scripts/python.exe brain/cli/approve.py approve <plan_id>LangFuse is auto-bootstrapped by deploy/compose/docker-compose.yml's
LANGFUSE_INIT_* env vars on first boot — an org, project, user, and a
fixed dev API key pair are created automatically. brain/sekhmet/config.py
defaults to those same keys, so tracing works with zero manual setup
through the LangFuse web UI. (Note: the server image is langfuse/langfuse:2,
Postgres-only; requirements.txt deliberately pins langfuse<3 on the
Python side, since the v3 SDK only speaks OTLP and needs a ClickHouse-backed
v3 server to ingest into — mixing them silently drops every trace.)
Every incident gets one LangFuse trace, named sekhmet-incident, keyed by
the incident's own UUID, with one span per graph node — this is what "what
did the agent see, recall, conclude, and do" actually looks like for a real
run (captured from a real trace produced by this system):
trace: sekhmet-incident (id = 166e5e48-1c1c-4880-907e-20fcc0920017)
├─ detect 10:19:06.537 → 10:19:06.594 {signature, severity}
├─ enrich 10:19:07.099 → 10:19:07.150 {history_samples: 12}
├─ recall 10:19:07.666 → 10:19:09.182 {similar_cases, runbook_hits, blast_radius}
├─ diagnose 10:19:09.285 → 10:20:38.609 → {root_cause, confidence, reasoning}
├─ plan 10:20:39.090 → 10:20:45.485 → {action: RaiseUlimitNofile, params, rationale}
├─ gate 10:20:45.745 → 10:20:45.745 {gate_status: auto_approved}
├─ execute 10:20:46.248 → 10:20:46.330 {status: rejected, message: "sysctl key not in allowlist: ..."}
├─ verify 10:20:46.835 → 10:21:06.953 {verified: true, attempt: 1}
└─ record 10:21:07.435 → 10:21:07.482 {final_status: resolved}
Every incident's detail panel in the dashboard (http://localhost:8080, click
any incident card) links straight to its full trace with a "View full
trace in LangFuse ↗" button — or browse http://localhost:3000 directly
(the auto-created login is admin@sekhmet.local / sekhmet-dev-password123,
also in the compose file's LANGFUSE_INIT_USER_* vars).
The dashboard itself shows fleet health, pending approvals, and the incident
timeline with the same per-incident detail panel — see Screenshots
above. It also supports a deep link, http://localhost:8080/?expand=first,
which auto-opens the most recent incident's detail panel — handy for sharing
a link straight to "here's what it just did."
# Node agent (Linux-only package: syscall.Kill, golang.org/x/sys/unix — run in a Linux container)
docker run --rm -v "$(pwd):/src" -w /src -e GOTOOLCHAIN=auto golang:1.25 go test ./node-agent/...
# Brain — unit tests (fast, no infra needed)
brain/.venv/Scripts/python.exe -m pytest brain/tests/ -q --ignore=brain/tests/integration
# Brain — integration tests (needs `docker compose up -d`; drives the real
# pipeline end-to-end for each canonical incident shape, asserting the full
# detect→diagnose→plan→gate→execute→verify audit trail; skips cleanly if
# Postgres isn't reachable)
brain/.venv/Scripts/python.exe -m pytest brain/tests/integration/test_end_to_end.py -v# Go
tools/protoc/bin/protoc.exe -I proto \
--go_out=node-agent/internal/grpc/sekhmetpb --go_opt=paths=source_relative \
--go-grpc_out=node-agent/internal/grpc/sekhmetpb --go-grpc_opt=paths=source_relative \
proto/sekhmet.proto
# Python
brain/.venv/Scripts/python.exe -m grpc_tools.protoc -I proto \
--python_out=brain/sekhmet/grpc/sekhmetpb \
--grpc_python_out=brain/sekhmet/grpc/sekhmetpb \
--pyi_out=brain/sekhmet/grpc/sekhmetpb \
proto/sekhmet.proto
# then fix the generated grpc file's import to `from . import sekhmet_pb2 as sekhmet__pb2`