Skip to content

feat: run claw-wrap as a Kubernetes sidecar - #12

Merged
dedene merged 6 commits into
mainfrom
feat/k8s-sidecar
Oct 7, 2026
Merged

dedene merged 6 commits into
mainfrom
feat/k8s-sidecar

Conversation

@dedene

@dedene dedene commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Makes claw-wrap usable as a sidecar next to an AI agent in one pod: the agent container holds no secrets and calls wrapped tools over a socket on a shared emptyDir; the sidecar runs the real binaries with credentials from a Kubernetes Secret.

The guide is docs/KUBERNETES.md, with a manifest in examples/kubernetes/ and a tool config in examples/mcparcel/. I deployed that manifest to a kind cluster (Kubernetes 1.35) with a fake tool that daemonizes and echoes its secret. Allowed and blocked calls, redaction, daemon survival, zombie reaping by tini, audit on stdout, PID isolation and ConfigMap hot reload all behaved as documented.

What would have broken in a pod

  • The caller's cwd doesn't exist in the sidecar, so every call failed with invalid working directory. New per-tool working_dir.
  • ConfigMap updates never reloaded on Linux: kubelet swaps a ..data symlink and the watcher only looked at wrappers.yaml. (It worked on macOS because kqueue follows the symlink.)
  • No backend could read a Secret volume (env: rejects symlinks and root-owned files). New file: backend.
  • A non-root daemon can't chmod the root-owned emptyDir. New CLAW_WRAP_RUNTIME_DIR points both sides at a subdirectory the daemon creates.
  • audit.file: /dev/stdout is refused (symlink). New audit.stdout.

New options

  • match: argv in allowed_args: one anchored regex per argument, exact arity. match: command joins args with spaces, so call front.read --server https://evil.example passes a typical ^call\s+front\.read(\s|$) rule; a test documents this.
  • use_stdin: false: /dev/null stdin and no PTY, so prompts fail fast.
  • request_env: allowlist for caller-supplied env. The agent holds the HMAC secret, so without this it can pass tool-specific variables (a server URL, say).
  • Credential values injected via env/config_file (8+ bytes) are redacted from tool output. This applies to every tool, so it's a behaviour change worth noting in the release.
  • --runtime-gid, --auth-mode, --socket-mode, ported from feature/docker-compose (without its Homebrew/Node image).

Bugs found along the way

  • Output truncation: Wait() closed StdoutPipe readers before the pumpers drained them (8 KB lost out of 200 KB in a test). Now uses owned pipes with an idle-based drain deadline, which also means a tool that leaves a detached daemon holding stdout can't keep a call open.
  • Data race between handleTimeout and cmd.Wait on ProcessState.
  • finalizeOutput returned early on a stdout error and dropped stderr.

Image

A root Dockerfile builds Debian slim with tini as PID 1 (reaps tool daemons that get reparented) and a non-root user. image.yml builds linux/amd64 and linux/arm64, pushes to ghcr.io on v* tags, and only builds on PRs that touch the image.

Testing

  • go vet and go test -race ./... pass on macOS and on Linux (non-root container, like CI).
  • actionlint is clean, goreleaser check passes, and a goreleaser snapshot build produced all four targets.
  • Both image architectures build.

Not addressed

  • On timeout, exec.CommandContext SIGKILLs the leader before the graceful SIGTERM to the group. This predates this PR.
  • TestDaemon_Run_ConfiguresBackendsAndCleansUpOnFailure hangs when run as root. Also existing.

dedene added 6 commits October 7, 2026 11:18
Reads Docker/Kubernetes secret mounts. Follows symlinks (Secret volumes use ..data links), requires owner root or daemon uid, rejects group-writable or other-accessible files, caps size at 64 KiB, trims trailing newlines, supports jq, and is never cached so rotation applies immediately.
Tool options:
- match: argv in allowed_args: one anchored pattern per argument, exact
  arity. Arguments are never joined, so spaces/newlines cannot widen a match.
- working_dir pins the tool cwd (the caller's cwd may not exist daemon-side).
- use_stdin: false gives the tool /dev/null stdin and disables PTY.
- request_env allowlists which env vars the caller may set.
- Injected credential values (>= 8 bytes) are redacted from tool output.

Runtime:
- --runtime-gid, --auth-mode, --socket-mode (ported from
  feature/docker-compose) and strict --uid parsing.
- CLAW_WRAP_RUNTIME_DIR moves socket/auth/proxy token for daemon and wrapper;
  the env credential file stays put.
- Reload config on Kubernetes ConfigMap ..data symlink swaps.
- audit.stdout writes JSONL audit entries to stdout; audit cwd is the
  effective working directory.
- Warn when no tools are configured; log once when caller exe verification
  is unavailable (separate PID namespace).

Fixes:
- Tool output could be truncated: Wait() closed StdoutPipe readers before
  the pumpers drained them. Use owned pipes with an idle-based drain
  deadline, which also keeps detached children from holding a call open.
- Data race between handleTimeout and cmd.Wait on ProcessState.
- finalizeOutput no longer drops stderr when stdout finalization fails.

examples/mcparcel/wrappers.yaml is covered by regression tests.
Debian slim with tini as PID 1 (reaps detached tool daemons) and a non-root claw user. The workflow builds linux/amd64 and linux/arm64, pushes to ghcr.io on v* tags and only builds on PRs touching the image.
Covers the shared runtime dir, UID/GID modes, security model without exe verification, Secrets via file:, ConfigMap reload, probes, audit to stdout and the contract for tools that start background daemons. CONFIG.md gains file:, working_dir, use_pty, use_stdin, request_env, match: argv, audit and daemon flags.
govulncheck flags GO-2026-6218, -6090, -6089, -5972, -5856 (stdlib, fixed in go1.26.5/6) and GO-2026-5026 (x/net, fixed in v0.55.0).
…it secrets

Split the output redactor into two stages. Literal secrets are replaced
first and only a trailing proper prefix of a secret is held back, so output
such as an interactive "Continue? [y/N] " prompt is released immediately
instead of sitting in a 128-byte carry while the tool waits for input.
redact_output regex rules then run on that output only, so they can no
longer rewrite the start of a secret that is still arriving
("key=tok_" + "live_123456" with a tok -> *** rule leaked "_live_123456").
When one secret is a prefix of another, the longer one is awaited.
@dedene
dedene merged commit bc66b03 into main Oct 7, 2026
3 checks passed
@dedene
dedene deleted the feat/k8s-sidecar branch October 7, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant