AcornOps Agent

Lightweight Kubernetes operations agent with outbound-only connectivity and an MCP bridge for cluster management.

Status

This repository owns the Kubernetes agent code, chart, production image, protocol contract, and agent-level docs. Central platform deployment wiring belongs in acornops-deployment.

Agent-Assisted Development

This repository supports human and agent-assisted development. Start coding agents from this repository root for AgentK-only work, and from the AcornOps workspace cloned from the acornops repository for changes that touch multiple AcornOps repositories.

Contracts

Cross-repo contract documentation lives in docs/contracts/README.md. This repo's only direct platform dependency should be the control-plane protocol documented there. Machine-readable contract data lives in docs/contracts/manifest.json. Run npm run contracts:check to mechanically verify the documented agent/control-plane contract against the implementation.

Coverage is generated in CI with Vitest V8 coverage, uploaded as a workflow artifact, and published to Codecov when CODECOV_TOKEN is configured for the repository.

Documentation

Primary docs:

• AGENTS.md
• ARCHITECTURE.md
• docs/index.md
• docs/DEVELOPMENT.md
• docs/OPERATIONS.md
• Whole-system architecture: ../docs/system-architecture.md

Features

• Rich Snapshots: Modular telemetry collection (Pods, workloads, Services, Ingresses, PVCs, Nodes, Metrics, Events).
• Outbound-only: Initiates secure WebSocket connection to the platform.
• MCP Bridge: Executes "Atomic Tools" and "Orchestrated Remediations" via JSON-RPC.
• Stateless: Zero local persistence; automatic re-handshake on reconnection.
• Safe: Built-in write guards and mandatory audit annotations.

Tech Stack

• Runtime: Node.js 20+
• Language: TypeScript (Strict Mode)
• Core Libraries: @kubernetes/client-node, ws, zod, and pino.

Configuration

The agent is configured via environment variables:

Variable	Description	Default
ACORNOPS_AGENT_PLATFORM_URL	WSS endpoint of the platform	(Required)
ACORNOPS_CLUSTER_ID	Control-plane cluster id this agent key is bound to	(Required)
ACORNOPS_AGENT_KEY	Unique agent authentication token	(Required)
ACORNOPS_AGENT_HANDSHAKE_PROBE_TIMEOUT_MS	Timeout for non-blocking metrics API probe during handshake	3000
ACORNOPS_AGENT_KUBECONFIG_REWRITE_LOOPBACK	Rewrite kubeconfig API server host (0.0.0.0/localhost) to ACORNOPS_AGENT_KUBECONFIG_HOST_ALIAS for containerized local dev	false
ACORNOPS_AGENT_KUBECONFIG_HOST_ALIAS	Hostname used when loopback rewrite is enabled	host.docker.internal
ACORNOPS_AGENT_KUBECONFIG_SKIP_TLS_VERIFY	Skip TLS verification after loopback-host rewrite (local k3d convenience only)	false
ACORNOPS_AGENT_K8S_CONCURRENCY	Process-wide maximum concurrent Kubernetes API list requests during snapshot collection	8
ACORNOPS_AGENT_K8S_LIST_PAGE_LIMIT	Kubernetes API list page size used when collecting large snapshots	500
ACORNOPS_AGENT_TOOL_READ_CONCURRENCY	Maximum concurrent read tool calls	4
ACORNOPS_AGENT_TOOL_WRITE_CONCURRENCY	Maximum concurrent write tool calls	1
ACORNOPS_AGENT_TOOL_QUEUE_LIMIT	Maximum queued calls shared across read and write gates	16
ACORNOPS_AGENT_TOOL_MAX_INPUT_BYTES	Maximum inbound WebSocket/tool request size	1048576
ACORNOPS_AGENT_TOOL_MAX_OUTPUT_BYTES	Maximum serialized tool result size	2097152
ACORNOPS_AGENT_SCALE_MAX_REPLICAS	Maximum accepted scale target (hard ceiling: 100)	100
ACORNOPS_AGENT_ALLOW_SCALE_TO_ZERO	Operator opt-in required before caller-confirmed scale-to-zero	false
ACORNOPS_AGENT_WORKLOAD_PATCH_KINDS	Comma-separated local maximum for patch_workload kinds	Deployment,StatefulSet,DaemonSet
ACORNOPS_AGENT_RESOURCE_PATCH_KINDS	Comma-separated opt-in kinds for Service/Ingress patch_resource	Empty
ACORNOPS_AGENT_ALLOW_ENV_PATCH	Operator opt-in for non-secret workload environment changes	false
ACORNOPS_AGENT_ALLOW_CONFIGMAP_PATCH	Operator opt-in for non-secret ConfigMap data changes	false
ACORNOPS_AGENT_ALLOW_SERVICE_SELECTOR_PATCH	Operator opt-in required before caller-confirmed Service selector changes	false
ACORNOPS_AGENT_RBAC_SCOPE	Local RBAC boundary (cluster or namespace)	cluster
ACORNOPS_AGENT_RBAC_ADDITIONS_JSON	Immutable, control-plane-generated custom-resource additions selected during onboarding	[]
ACORNOPS_AGENT_WATCH_CACHE_ENABLED	Build snapshots from a Kubernetes watch-backed local cache when ready	true
ACORNOPS_AGENT_WATCH_SNAPSHOT_DEBOUNCE_MS	Debounce window for snapshots triggered by watched resource or Warning event changes	5000
ACORNOPS_AGENT_WATCH_CACHE_SYNC_TIMEOUT_MS	Time to wait for watch cache warmup before using list fallback	15000
ACORNOPS_AGENT_WATCH_TIMEOUT_SECONDS	Kubernetes watch stream timeout before clean reconnect	300
ACORNOPS_AGENT_WATCH_NAMESPACES	Comma-separated list of namespaces to watch	All
ACORNOPS_AGENT_EXCLUDE_NAMESPACES	Comma-separated local namespace deny-list	Empty
ACORNOPS_AGENT_WRITE_ENABLED	Set to true to enable mutation tools	false
ACORNOPS_AGENT_LOCAL_FALLBACK_ENABLED	Emit local synthetic node/usage snapshot when Kubernetes API is unreachable (dev only)	false
ACORNOPS_AGENT_LOG_LEVEL	Logging level (info, debug, error, warn, trace)	info
ACORNOPS_AGENT_LEADER_ELECTION_ENABLED	Enable active-passive HA using a Kubernetes Lease	false
ACORNOPS_AGENT_LEASE_NAME	Leader election Lease name	acornops-agent-leader
ACORNOPS_AGENT_LEASE_NAMESPACE	Leader election Lease namespace; empty uses pod namespace	Pod namespace
ACORNOPS_AGENT_LEADER_IDENTITY	Lease holder identity; empty uses pod UID, then pod name	Pod UID/name
ACORNOPS_AGENT_LEASE_DURATION_MS	Lease expiry window before another replica can take over	15000
ACORNOPS_AGENT_RENEW_DEADLINE_MS	Local renew failure deadline before the leader fences itself	10000
ACORNOPS_AGENT_RETRY_PERIOD_MS	Election retry/renew period	2000

Snapshot Scalability

Snapshots are assembled from a Kubernetes watch-backed local cache by default. The agent still emits the existing compact notify/snapshot payload, but it keeps Pods, workloads, Services, Ingresses, PVCs, Nodes, Namespaces, and recent Warning Events fresh through list-then-watch streams instead of listing every resource on each interval.

The list collectors remain the fallback path when the watch cache is disabled, warming, or unhealthy. Initial sync and 410-compaction recovery still use bounded, paginated list calls, so ACORNOPS_AGENT_K8S_CONCURRENCY and ACORNOPS_AGENT_K8S_LIST_PAGE_LIMIT continue to bound API pressure during warmup, scope changes, and watch recovery.

Use snapshot-manager logs to track durationMs, skippedSnapshots, droppedSnapshots, originalBytes, and compressedBytes. If snapshots regularly skip interval ticks, increase the snapshot interval before raising ACORNOPS_AGENT_K8S_CONCURRENCY; higher concurrency can worsen API server pressure when list latency is already the bottleneck.

Deployment

Helm (Recommended)

The supported release install path is the acornops-agentk Helm chart. This is the right install method whether the central AcornOps platform runs on Docker-on-VM or Kubernetes.

Install with the public platform base URL:

helm upgrade --install acornops-agent oci://ghcr.io/acornops/charts/acornops-agentk \
  --namespace acornops \
  --create-namespace \
  --set-string config.platformUrl=https://api.acornops.dev \
  --set-string config.clusterId=YOUR_CLUSTER_ID \
  --set-string config.agentKey=YOUR_AGENT_KEY

config.platformUrl is the public control-plane base URL. The chart derives the agent WebSocket URL as wss://<host>/api/v1/agent/connect. For Docker-on-VM central deployments, this is the API host, for example https://api.acornops.dev; the management console is served separately from https://console.acornops.dev/.

If your WebSocket route is custom, pass it directly:

helm upgrade --install acornops-agent oci://ghcr.io/acornops/charts/acornops-agentk \
  --namespace acornops \
  --create-namespace \
  --set-string config.websocketUrl=wss://api.acornops.dev/api/v1/agent/connect \
  --set-string config.clusterId=YOUR_CLUSTER_ID \
  --set-string config.agentKey=YOUR_AGENT_KEY

Use an existing Kubernetes Secret instead of passing the key in Helm values:

kubectl -n acornops create secret generic acornops-agent-key \
  --from-literal=agent-key=YOUR_AGENT_KEY

helm upgrade --install acornops-agent oci://ghcr.io/acornops/charts/acornops-agentk \
  --namespace acornops \
  --create-namespace \
  --set-string config.platformUrl=https://api.acornops.dev \
  --set-string config.clusterId=YOUR_CLUSTER_ID \
  --set-string existingSecret.name=acornops-agent-key \
  --set-string existingSecret.key=agent-key

To connect to a platform certificate signed by an organization-private CA, either supply a public PEM CA bundle directly to Helm:

--set-file config.tls.additionalCaBundle.inlinePem=/path/to/organization-ca.pem

or reference an existing CA bundle in the AgentK release namespace:

config:
  platformUrl: https://api.acornops.example
  tls:
    additionalCaBundle:
      configMapKeyRef:
        name: organization-platform-trust
        key: ca.crt

Use secretKeyRef with the same name and key fields when the bundle is distributed as a Secret. The three sources are mutually exclusive and fail closed when the selected resource or key is missing. AgentK mounts the bundle read-only and uses NODE_EXTRA_CA_CERTS, which extends Node.js public CA trust without disabling hostname or certificate verification. The resource must be distributed into the release namespace of every affected workload cluster.

Changes supplied through --set-file roll AgentK automatically; changes to an existing ConfigMap or Secret require a restart. Rotate private roots with an old/new overlap, restart AgentK, rotate the platform certificate, then remove the old root and restart again. See the chart guide for Secret configuration, trust-manager compatibility, commands, and failure-mode troubleshooting.

The chart defaults to cluster-wide read-only RBAC. Write tools require an explicit opt-in:

helm upgrade --install acornops-agent oci://ghcr.io/acornops/charts/acornops-agentk \
  --namespace acornops \
  --create-namespace \
  --set-string config.platformUrl=https://api.acornops.dev \
  --set-string config.clusterId=YOUR_CLUSTER_ID \
  --set-string config.agentKey=YOUR_AGENT_KEY \
  --set rbac.write.enabled=true

patch_workload accepts semantic image, non-secret environment, and metadata operations. CronJob support is an explicit patchPolicy.workloadKinds opt-in. Service and Ingress support uses patchPolicy.resourceKinds. patch_configmap requires patchPolicy.allowConfigMapChanges=true; workload environment changes require patchPolicy.allowEnvChanges=true. Literal environment and ConfigMap values are limited to single-line, non-secret configuration of at most 4 KiB, and the caller must set confirm_non_secret_data=true for each request that supplies one. Service selector changes also require patchPolicy.allowServiceSelectorChanges=true and caller confirmation. ConfigMap get/list/watch RBAC is granted only when patchPolicy.allowConfigMapChanges=true; ConfigMap patch RBAC additionally requires rbac.write.enabled=true.

For namespace-scoped installs, create Roles only in the watched namespaces:

helm upgrade --install acornops-agent oci://ghcr.io/acornops/charts/acornops-agentk \
  --namespace acornops \
  --create-namespace \
  --set-string config.platformUrl=https://api.acornops.dev \
  --set-string config.clusterId=YOUR_CLUSTER_ID \
  --set-string config.agentKey=YOUR_AGENT_KEY \
  --set-string rbac.scope=namespace \
  --set-json 'namespaceScope.include=["team-a","team-b"]'

When rbac.scope=namespace, the chart creates Roles in rbac.namespaces when set, otherwise in namespaceScope.include, otherwise in the release namespace.

Active-passive HA is opt-in. Multiple replicas are rejected unless leader election is enabled, because the agent is not active-active safe. In HA mode, only the elected Lease holder connects to the control plane, sends snapshots/heartbeats, and serves tool calls:

helm upgrade --install acornops-agent oci://ghcr.io/acornops/charts/acornops-agentk \
  --namespace acornops \
  --create-namespace \
  --set-string config.platformUrl=https://api.acornops.dev \
  --set-string config.clusterId=YOUR_CLUSTER_ID \
  --set-string config.agentKey=YOUR_AGENT_KEY \
  --set replicaCount=2 \
  --set leaderElection.enabled=true

The default remains replicaCount=1 with leader election disabled. That keeps the common install simple and avoids granting Lease write permissions unless HA is requested. For HA, replicaCount=2 is usually enough to survive one agent pod or node loss. Use replicaCount=3 only when you want two warm standbys or broader scheduling spread; Kubernetes Lease election does not need an odd number of agent replicas.

Shorter Lease timings reduce failover time but increase Kubernetes API churn and sensitivity to transient API delays. Longer timings reduce churn but extend the passive takeover gap.

Check status and logs:

kubectl -n acornops rollout status deployment/acornops-agent
kubectl -n acornops logs -f deployment/acornops-agent

Uninstall:

helm uninstall acornops-agent --namespace acornops

Manual Raw Manifests

The raw manifests under deploy/ are retained for manual development and troubleshooting. Prefer Helm for release installs.

Local Development (Direct Run)

If you have a Kubernetes cluster already running and configured in your KUBECONFIG:

1. Install dependencies:

   npm install

2. Run in development mode:

   export ACORNOPS_AGENT_PLATFORM_URL=wss://api.acornops.dev/api/v1/agent/connect
   export ACORNOPS_CLUSTER_ID=your-cluster-id
   export ACORNOPS_AGENT_KEY=your-test-key
   npm run dev

Local Development and Testing

Compose files are split as:

• docker-compose.yml: base/default runtime for acornops-agentk.
• docker-compose.override.yml: local development additions (mock-platform, host mounts, host ports, local image build).

Run Modes

1. Component-only local development (recommended in this repo):

docker compose up -d --build

This local mode runs the agent with tsx watch and the mock platform with node --watch, so code changes are reflected immediately.

2. Component-only production-style container (agent only):

docker compose -f docker-compose.yml up -d

3. Full AcornOps stack (all components together):

cd ../acornops-deployment
task local-up

This full-stack flow uses the deployment repo Taskfile.yml and requires the task CLI to be installed.

Use full-stack mode when validating the agent against the real control-plane service instead of the local mock platform. Do not run this repository's local compose stack and acornops-deployment local stack at the same time on the same host ports.

If dependencies change (package.json or lockfile), rebuild once:

docker compose up -d --build

For efficient component-level development, run the agent with the local Mock Platform and command-testing endpoint.

The local mock platform provides:

• WebSocket Server: Listens on ws://localhost:3000/agent
• Snapshot Storage: Decompressed snapshots are saved to ./local-snapshots/ in Docker Compose local mode. When the mock platform is run directly, it writes to SNAPSHOT_DIR or ./snapshots/.
• Command Trigger: POST http://localhost:3000/send-command to send MCP requests to the agent.
• Health Endpoint: GET http://localhost:3000/health
• Connection Inspect Endpoint: GET http://localhost:3000/connections
• Swagger UI (Mock Platform): http://localhost:3000/docs
• OpenAPI JSON (Mock Platform): http://localhost:3000/openapi.json

The acornops-agentk process itself is not an HTTP API server; it is a websocket JSON-RPC client. For local developer docs/testing, use the mock platform HTTP API above.

Local k3d Testing Guide

This guide provides step-by-step instructions to set up a local Kubernetes cluster using k3d, build the agent, and deploy it for manual testing.

Prerequisites

Ensure you have the following installed:

• Docker
• k3d
• kubectl
• Node.js 20+

Create a Local Cluster

Create a simple one-node cluster:

k3d cluster create acornops-demo-cluster --no-lb --wait

Build the Agent Image

Build the Docker image locally:

docker build -t acornops/agent:local .

Load Image into k3d

Import the local image into the k3d cluster so it's available without a registry:

k3d image import acornops/agent:local -c acornops-demo-cluster

Prepare Deployment Manifests

Create the acornops namespace:

kubectl create namespace acornops

Create a dummy secret for testing:

kubectl create secret generic acornops-agentk-secret \
  --from-literal=agent-key=test-agent-key-123 \
  -n acornops

Deploy the Agent

Apply the RBAC and Deployment manifests:

# Apply RBAC
kubectl apply -f deploy/rbac.yaml

# Use the local development manifest, which adds least-privilege workload patch
# RBAC because this legacy development deployment explicitly enables write tools
kubectl apply -f deploy/local-development.yaml

Verify and Monitor

Check the mock platform logs to see the agent connecting and sending snapshots:

docker compose logs -f mock-platform

Check the local-snapshots/ directory for saved data in Docker Compose local mode.

Sending Commands

You can test tools by sending commands via the mock platform:

curl -X POST http://localhost:3000/send-command \
  -H "Content-Type: application/json" \
  -d '{
    "method": "get_pod_logs",
    "params": {
      "podName": "acornops-agentk-xxxxxxxxxx-xxxxx",
      "namespace": "acornops"
    }
  }'

Inspect Runtime

Check if the pod is running:

kubectl get pods -n acornops -l app=acornops-agentk

Check the agent logs:

kubectl logs -f -n acornops -l app=acornops-agentk

Cleanup Cluster

When finished, delete the cluster:

k3d cluster delete acornops-demo-cluster

Testing (Automated)

Run unit tests:

npm test

Run E2E tests (requires a running cluster):

npm run test:e2e

Validation

Run the checks that match the change:

• npm run lint
• npm run contracts:check
• npm run harness:check
• npm run validate
• npm test
• npm run test:e2e when lifecycle, websocket, or RBAC behavior changes

Tools Available

• list_resources: List bounded built-in or administrator-approved custom-resource summaries.
• get_resource: Fetch a redacted built-in or administrator-approved custom resource by name.
• get_resource_logs: Read bounded Pod logs (max 1 MiB).
• restart_workload: Guarded rolling restart for Deployments, StatefulSets, and DaemonSets.
• scale_workload: Guarded scaling for Deployments and StatefulSets.
• patch_workload: Apply guarded workload image, single-line non-secret environment, label, or annotation changes.
• patch_resource: Apply guarded Service or Ingress changes, or /spec changes to administrator-approved custom resources.
• patch_configmap: Set or remove bounded single-line non-secret ConfigMap data keys without restarting consumers.