Autonomous Incident Response System

An AI-assisted, multi-agent Site Reliability Engineering (SRE) platform that triages incidents, verifies root-cause hypotheses with tools, executes remediation (when safe), escalates to humans when required, and generates a post-mortem report.

The system combines:

FastAPI backend for streaming incident execution
LangGraph workflow orchestration for autonomous decision loops
Streamlit operations dashboard for real-time observability and Human-in-the-Loop (HITL)
SQLite + ChromaDB memory layers for operational history and retrieval-augmented context
n8n + Telegram bot integration for external automation, alert intake, and incident notifications

Why This Project

Modern incidents are noisy, high-pressure, and expensive when triage is slow. This platform provides a practical blueprint for autonomous incident handling with built-in guardrails:

Rapid intake and classification of raw alerts
Tool-backed verification instead of guess-only reasoning
Escalation and pause/resume for human approval
Execution trace streaming for operator visibility
Persistent incident memory to improve future responses

Core Capabilities

Autonomous incident triage
- Classifies incoming incidents by component and severity (P0-P3)
- Generates structured hypotheses for investigation
RAG-assisted historical memory lookup
- Searches prior incident reports from ChromaDB before deciding next steps
- Encourages reuse of known-good fixes
Verification loop with live tools
- Chooses diagnostic tools dynamically (e.g., DB pool checks, payment latency checks)
- Iterates across hypotheses until one is verified or all are rejected
Automated remediation actions
- Selects and invokes remediation tools (e.g., scale DB connections, rollback deployment, clear cache)
- Captures execution details for post-mortem traceability
Human-in-the-Loop (HITL) escalation
- Interrupts graph execution when confidence is low, hypotheses are exhausted, or incident severity warrants approval
- Supports resume flow with human override decisions
Real-time operator UI
- Streamed event updates and logs in Streamlit
- Live status card, node progress, interrupt payload rendering, final report view
Incident reporting and long-term memory
- Auto-generates markdown post-mortem report
- Persists incidents, hypotheses, actions, traces to SQLite
- Stores report embeddings in ChromaDB for future retrieval
External automation hooks
- Sends interrupt/report payloads to n8n via webhook for notifications/workflows
Telegram group operations
- Supports incident intake from Telegram using /alert <incident details> in a group
- Automatically posts generated incident reports back to the Telegram group via bot workflow
- Supports HITL approvals in Telegram where responders approve/reject fixes and resume workflow execution

System Architecture

UI/API Alert or Telegram /alert -> FastAPI /investigate -> LangGraph Workflow
                                                     -> Triage
                                                     -> Verifier (loop)
                                                        -> Action (if verified)
                                                        -> Escalation (if needed)
                                                     -> Report
                                                     -> Stream events to UI + webhook
                                                     -> Persist to SQLite + ChromaDB

Report/Interrupt Webhook -> n8n -> Telegram Bot -> Telegram Group
Telegram Approve/Reject Action -> n8n -> POST /api/v1/resume/{thread_id}

Graph rendering is available in graph.png.

Workflow Lifecycle

Incident Intake
- Client sends raw_incident to POST /api/v1/investigate directly (UI/API) or through Telegram /alert flow bridged by n8n.
Triage
- Classifies component and severity; queries historical memory; emits hypotheses.
Verification Loop
- Tests each hypothesis with selected diagnostic tools.
Decision Branch
- Verified hypothesis -> remediation action.
- No verified hypothesis / low confidence / guardrail -> escalation.
HITL (if interrupted)
- Human reviews payload in UI or Telegram and resumes via POST /api/v1/resume/{thread_id}.
Reporting + Persistence
- Generates final markdown report, stores full incident lifecycle, and can auto-send report content to Telegram group via webhook automation.

Tech Stack

Backend: FastAPI, Pydantic, LangGraph, LangChain
Frontend: Streamlit
LLM Integrations: OpenAI-compatible endpoint + Ollama models
Memory:
- Relational: SQLite (sre_memory.db) via SQLAlchemy
- Vector: ChromaDB (chroma_data/) using Ollama embeddings
Automation: n8n webhook callbacks
Messaging: Telegram Bot API (typically orchestrated via n8n workflows)
Containerization: Docker, Docker Compose
Package/Runtime Tooling: uv (Python dependency/runtime manager)

Project Structure

backend/
  api/main.py                  # FastAPI app, streaming endpoints, webhook dispatch
  graph/graph.py               # LangGraph construction and routing
  graph/state.py               # Shared workflow state schema
  graph/nodes/                 # Triage, Verify, Action, Escalation, Report nodes
  tools/sre_tools.py           # Diagnostic/action tools + RAG search tool
  core/models.py               # SQLAlchemy data model
  core/schema.py               # Pydantic schemas (hypothesis/action)
  utils/database.py            # SQLite initialization/session management

frontend/
  app.py                       # Streamlit control panel with live stream handling

docker-compose.yml             # n8n + backend + frontend orchestration
bot_test.py                    # Test webhook payload sender (useful for n8n/Telegram flow testing)
graph.png                      # Workflow graph artifact
sre_memory.db                  # SQLite memory store
chroma_data/                   # Chroma vector store persistence

Getting Started

Prerequisites

Python 3.13+
Docker + Docker Compose (for containerized run)
Ollama endpoint accessible by backend/reporting tools
n8n (if using webhook automation path)

Option A: Docker (recommended)

cd "C:\Users\asus\PycharmProjects\Autonomous-Incident-Response-System"
docker compose up --build

Services exposed:

Frontend: http://localhost:8501
Backend API: http://localhost:8000
n8n: http://localhost:5678

Option B: Local Development (without Docker)

cd "C:\Users\asus\PycharmProjects\Autonomous-Incident-Response-System"
uv sync
uv run fastapi run backend/api/main.py

In a separate terminal:

cd "C:\Users\asus\PycharmProjects\Autonomous-Incident-Response-System"
uv run streamlit run frontend/app.py

Configuration

Create a .env file in the project root. Example variable names used by the app:

OLLAMA_API_KEY=
N8N_WEBHOOK_URL=

Environment Variables

Variable	Purpose
`OLLAMA_API_KEY`	API key for OpenAI-compatible calls used by graph nodes
`N8N_WEBHOOK_URL`	Endpoint for async interrupt/report webhook payloads

If you route events to Telegram through n8n, bot-specific values (for example, bot token and target group/chat ID) are typically configured inside n8n credentials/workflow variables rather than this backend.

Running the System

Start backend and frontend (Docker or local).
Open http://localhost:8501.
Paste a raw incident alert into the input area.
Click Start Investigation.
Observe live node logs and status updates.
If interrupted, submit human decision and resume.
Review generated post-mortem report.

Alternative intake path:

Send /alert <incident details> in your configured Telegram group to trigger the same investigation flow via n8n.

Automated messaging path:

Interrupt events and final reports can be forwarded by n8n to your Telegram group through the bot.

Telegram Integration

The project supports Telegram-assisted operations through webhook automation:

Inbound (Telegram -> System)
- In a Telegram group, operators send /alert <incident details>.
- Your n8n Telegram trigger parses the message and calls POST /api/v1/investigate with raw_incident.
Outbound (System -> Telegram)
- Backend emits interrupt and report webhook payloads to N8N_WEBHOOK_URL.
- n8n forwards these payloads to Telegram so the group receives real-time updates and final reports.
HITL approvals from Telegram
- Interrupt payload includes the investigation context and thread_id.
- In Telegram, responders can choose Approve or Reject on the proposed fix.
- n8n maps the choice to API semantics and calls POST /api/v1/resume/{thread_id}:
  - Approve -> decision: "override"
  - Reject -> decision: "ignore"
Recommended n8n mapping
- /alert text -> raw_incident
- report.content -> formatted Telegram message
- interrupt.payload -> approval prompt message for responders
- Telegram action callback -> decision + thread_id for resume API call

API Reference

`POST /api/v1/investigate`

Starts a new incident workflow and streams Server-Sent Events.

Request body:

{
  "raw_incident": "DATADOG ALERT: Database connection pool exhausted..."
}

Response:

text/event-stream
header X-Thread-ID: INC-XXXXXXXX

`POST /api/v1/resume/{thread_id}`

Resumes a paused investigation after human decision.

This endpoint can be invoked from the Streamlit UI or by an n8n Telegram action workflow when a group responder approves/rejects a fix.

Request body:

{
  "decision": "override",
  "action_type": "restart_service",
  "target": "api-server"
}

`GET /`

Basic health-style root response.

Human-in-the-Loop Flow

Escalation is triggered when:

No hypotheses are verified
Confidence/verification path suggests low trust
Severity guardrail requires explicit approval (e.g., P0)

During escalation:

Backend emits interrupt event to frontend
Backend can forward interrupt payload (including thread_id) to n8n webhook
Operator submits override/ignore decision
Graph resumes from interruption point

Telegram-assisted HITL option:

n8n posts an approval message in Telegram group with action buttons
Approve/Reject action triggers POST /api/v1/resume/{thread_id}
n8n maps Telegram action to override/ignore decision payload

Data and Memory Model

The platform persists incident context in two layers:

SQLite (sre_memory.db)
- incidents
- hypotheses
- actions
- traces
- past_incidents
ChromaDB (chroma_data/)
- Embeds final reports into sre_runbooks collection
- Enables semantic similarity retrieval in future triage

Screenshots

1) Dashboard Overview

2) Live Investigation Stream

3) Human Intervention Prompt

4) Final Post-Mortem Report

5) Execution Graph

Troubleshooting

Backend cannot reach model endpoint
- Verify Ollama/OpenAI-compatible URL accessibility and credentials.
Frontend cannot connect to backend
- Ensure API_URL points to backend and port 8000 is reachable.
No webhook events in n8n
- Check N8N_WEBHOOK_URL, tunnel visibility (if using ngrok), and n8n workflow state.
Telegram group does not receive report messages
- Validate n8n Telegram node credentials, bot membership/admin rights in the group, and the correct group/chat ID mapping.
Graph image not visible in UI sidebar
- Confirm graph.png exists and is mounted in Docker (/app/graph.png).
Memory retrieval seems empty
- Verify chroma_data/ persistence and that reports are being generated/embedded.

Security Notes

Store secrets only in .env or a secure secret manager; never hardcode credentials.
If any secrets were committed previously, rotate them immediately.
Restrict CORS origins and harden webhook exposure before production deployment.
Treat mock remediation tools as placeholders before connecting to real infrastructure actions.

Roadmap Ideas

Add authentication and role-based approvals for HITL actions
Replace mock tools with production-safe, audited integrations
Add incident timeline view and search in frontend
Introduce automated tests for node behavior and API streaming
Add CI checks for linting, type checks, and security scanning

License

This project is licensed under the MIT License. See LICENSE for details.

Name		Name	Last commit message	Last commit date
assets		assets
backend		backend
frontend		frontend
.dockerignore		.dockerignore
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
bot_test.py		bot_test.py
docker-compose.yml		docker-compose.yml
graph.png		graph.png
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation