SRE Agent Demo

Warning

This repository is for demo and educational purposes only. It MUST NOT be used in production. The sole purpose of this code is to demonstrate one possible way to implement an SRE agent. It intentionally omits production-grade security hardening, operational controls, testing depth, persistence design, governance, and safety mechanisms that would be required for real systems.

This folder contains a runnable .NET 10 demo implementation for the conference talk Beyond Code: An SRE Agent for Self-Healing Operations.

What It Shows

• Incident ingestion through modular ingestion modules.
• Kubernetes pod-health detection for the FastFood prod namespace.
• Rule-based triage before an AI agent is allowed to work.
• Policy-gated agent modes: ReadOnly, SuggestFix, ApprovalRequired, AutoRemediate, and PostmortemOnly.
• A GitHub Copilot SDK runtime adapter with MCP server configuration.
• A deterministic simulator runtime for rehearsals without credentials.
• Markdown incident reports stored in the demo persistence layer.
• A small ASP.NET Core operations dashboard.

Run The Web Demo

dotnet run --project SreAgent.Web/SreAgent.Web.csproj

Open the URL printed by ASP.NET Core. The default runtime is Simulator, so the demo works without Copilot authentication or live MCP servers.

By default, the web app uses the configured Kubernetes ingestion module rather than a synthetic incident. It polls the prod namespace every 10 seconds with the generated kubeconfig. When pods become unhealthy, it creates an internal incident and starts the agent automatically.

Run The CLI Demo

dotnet run --project SreAgent.Cli/SreAgent.Cli.csproj -- ReadOnly
dotnet run --project SreAgent.Cli/SreAgent.Cli.csproj -- ApprovalRequired

Enable The Copilot Runtime

The demo MCP setup starts with:

• fastfoodmcp over HTTP at http://localhost:5000/mcp
• kubernetes through Docker using ghcr.io/azure/mcp-kubernetes

The Kubernetes MCP server needs a kubeconfig file. Generate a short-lived service-account kubeconfig like this:

./mcp-setup/k8s/create-kubeconfig.sh
export SRE_AGENT_KUBECONFIG="$PWD/mcp-setup/k8s/.generated/kubeconfig-fastfood-copilot.yaml"

The generated kubeconfig is written below .generated/, chmodded to 600, and ignored by Git. Treat it as an ephemeral credential, not as source code.

The web app also uses this .generated path as its default when SRE_AGENT_KUBECONFIG is not set, so after generating the file once you can start SreAgent.Web directly from Rider/F5. Keep the environment variable only when you want to point at a different kubeconfig.

Then switch the runtime:

dotnet run --project SreAgent.Web/SreAgent.Web.csproj -- --SreAgent:Agent:Runtime Copilot

The adapter uses GitHub.Copilot.SDK and maps configured MCP servers into the SDK session. ${SRE_AGENT_KUBECONFIG} placeholders are expanded from environment variables before MCP servers are launched.

Before using write-capable MCP tools against a real cluster, configure Kubernetes RBAC, tool allowlists, and approval policy intentionally.

Copilot SDK Model Provider Modes

The Copilot runtime supports two model-provider modes:

1. GitHubCopilot: uses your authenticated GitHub account and Copilot subscription through the Copilot SDK/CLI runtime.
2. BringYourOwnModel: configures ProviderConfig and sends inference to an OpenAI-compatible, Azure, or Anthropic endpoint.

GitHub Copilot Subscription

Use this when you want to demo the normal Copilot subscription path:

{
  "SreAgent": {
    "Agent": {
      "Runtime": "Copilot",
      "Model": "auto",
      "ModelProvider": {
        "Mode": "GitHubCopilot"
      }
    }
  }
}

Set Model to auto for Copilot model selection or to a specific model id that your Copilot plan allows.

Bring Your Own Local Model

Use this when you want to show local inference through an OpenAI-compatible endpoint:

{
  "SreAgent": {
    "Agent": {
      "Runtime": "Copilot",
      "Model": "Qwen3.6-35B-A3B-8bit",
      "ModelProvider": {
        "Mode": "BringYourOwnModel",
        "Type": "openai",
        "BaseUrl": "http://127.0.0.1:8000/v1",
        "WireApi": "responses",
        "ModelId": "Qwen3.6-35B-A3B-8bit",
        "WireModel": "Qwen3.6-35B-A3B-8bit"
      }
    }
  }
}

For oMLX, keep WireApi as responses when the server exposes an OpenAI-compatible Responses API. This is the better demo path for agentic work because the Copilot SDK uses it for multi-turn state, tool namespacing, and reasoning-capable models.

The API key should be stored in .NET user secrets, not in appsettings.json. The web project already has a UserSecretsId, so Rider/F5 loads these values in Development:

cd SreAgent.Web
dotnet user-secrets set "SreAgent:Agent:Runtime" "Copilot"
dotnet user-secrets set "SreAgent:Agent:Model" "Qwen3.6-35B-A3B-8bit"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:Mode" "BringYourOwnModel"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:Type" "openai"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:BaseUrl" "http://127.0.0.1:8000/v1"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:WireApi" "responses"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:ModelId" "Qwen3.6-35B-A3B-8bit"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:WireModel" "Qwen3.6-35B-A3B-8bit"
dotnet user-secrets set "SreAgent:Agent:ModelProvider:ApiKey" "<your-omlx-api-key>"

To check what is configured locally:

dotnet user-secrets list

For Ollama, start with:

{
  "Mode": "BringYourOwnModel",
  "Type": "openai",
  "BaseUrl": "http://localhost:11434/v1",
  "WireApi": "completions",
  "ModelId": "qwen2.5-coder:14b",
  "WireModel": "qwen2.5-coder:14b"
}

Ollama is documented by GitHub as an OpenAI-compatible provider. Use responses only if the local endpoint/model actually supports the Responses API; otherwise completions is the compatibility fallback.

Example files:

• SreAgent.Web/appsettings.github-copilot.example.json
• SreAgent.Web/appsettings.local-model.example.json

Ingestion Configuration

Ingestion modules are configured through SreAgent:Ingestion:Modules:

{
  "Name": "prod-pod-health",
  "Type": "KubernetesPodHealth",
  "Enabled": true,
  "Environment": "fastfood-demo",
  "Namespace": "prod",
  "Kubeconfig": "${SRE_AGENT_KUBECONFIG}",
  "KubectlPath": "kubectl",
  "MinimumUnhealthyPods": 1,
  "SuggestedMode": "ApprovalRequired"
}

For the current FastFood demo, change the Dapr PubSub or StateStore component to an invalid Redis endpoint. Once the target pods in prod stop becoming ready, the ingestion module emits an incident signal and the SRE Agent run starts.

Grafana ingestion is included in the code, but the default SreAgent:Ingestion:Grafana:BaseUrl is empty in the public demo configuration. Configure your own Grafana URL and token through local configuration or .NET user secrets before enabling that path against a real monitoring stack.

Auto-Remediation Demo

The safe base configuration can detect incidents and run the agent, but it may still stop before writes if RequireApprovalForWrites is true. For the self-healing demo, enable the real write path deliberately:

cd SreAgent.Web
dotnet user-secrets set "SreAgent:Agent:Runtime" "Copilot"
dotnet user-secrets set "SreAgent:Agent:RequireApprovalForWrites" "false"
dotnet user-secrets set "SreAgent:Ingestion:Modules:0:SuggestedMode" "AutoRemediate"
dotnet user-secrets set "SreAgent:Remediation:DaprRedis:ExpectedRedisHost" "redis-ha-haproxy.redis:6379"
dotnet user-secrets set "SreAgent:Remediation:DaprRedis:RedisPasswordSecretName" "redis-ha"
dotnet user-secrets set "SreAgent:Remediation:DaprRedis:RedisPasswordSecretKey" "auth"

In this mode the prompt instructs the agent to inspect Dapr Components in the incident namespace, patch only wrong Redis endpoint metadata, restart affected deployments, and verify pods become Ready. A report that only contains "Recommended Fix" means the agent did not actually remediate.

The Copilot session is configured as MCP-only by default through SreAgent:Agent:EnableBuiltInTools=false. This avoids split-brain cluster access where the agent uses host kubectl against your workstation's default context while the ingestion/MCP path uses the generated SRE kubeconfig.

The base SRE prompt enforces a minimal-mutation contract: read the live object, change only the required field, prefer targeted patches, and verify unrelated fields were preserved. Domain-specific operational guidance lives in standard skill folders under SreAgent.Web/skills/<skill-name>/SKILL.md.

The first skill is dapr-component-remediation, which tells the agent how to inspect and patch Dapr Components safely. It explicitly prevents replacing an entire Dapr Component just to fix one metadata entry.

If Dapr sidecars log NOAUTH Authentication required, the Redis endpoint is reachable but the component is missing Redis password metadata. The expected shape for Redis-backed Dapr components is:

auth:
  secretStore: fastfood-secrets
spec:
  metadata:
  - name: redisHost
    value: redis-ha-haproxy.redis:6379
  - name: redisPassword
    secretKeyRef:
      name: redis-ha
      key: auth

The referenced secret must exist in the same namespace that the Dapr secret store can read.

Useful verification commands:

kubectl -n prod get components.dapr.io -o yaml
kubectl -n prod rollout status deployment --all
kubectl -n prod get pods