ci(e2e): Run real-runtime checks on Linux and macOS

[ananos]

Summary

CI drives brig against a stub runtime only. Every v0.3.0 release blocker needed a real one, so each of them was found late, by hand, before the release candidate. This is the first step of #358: an e2e workflow that runs real-runtime checks on three kinds of host, nightly and on demand, and renders the results as one HTML report.

• Linux: a hosted ubuntu-24.04 runner, which exposes nested KVM. script/e2e/canary-linux.sh installs brig with this commit's install.sh as a rootless user, swaps in brig and brigd built from the commit, and runs one gate per past release blocker plus a set of checks.
• macOS: the self-hosted SIP-enabled Macs, mac-dev-01-brig-1 and -2 (label sip-enabled). script/e2e/canary-macos.sh installs hull from the newest hull@main channel release and brig from the commit, and boots real microVMs through them. On schedule and dispatch it also puts the notarized channel build of the commit through Gatekeeper, quarantined the way Homebrew leaves it.
• Homebrew tap: a hosted macos-15 runner runs brew tap and brew readall brig-sh/brig, with no VM.

Each host uploads its own results-<host>.json. A report job merges them with script/e2e/merge-results.py into one results.json, renders report.html and the job summary, and uploads both as e2e-report. The verdict is go when no gate fails on any host. A host whose job uploaded nothing shows as a failure, never as green.

The Mac runners are shared and persistent, so the macOS job leaves them as it found them. It points HOME at a scratch directory under /private/tmp for the whole job. hull's store, ~/.brig, ~/.config/brig and the Go caches all live there, and the runner account's own ~/.hull and ~/.brig are never touched. An always() step (script/e2e/cleanup-macos.sh) stops every process that names that directory, including brig's hull network-gateway, detaches the store's sparse image and removes the directory. A concurrency group keeps one macOS e2e job on those Macs at a time, and pull requests from forks never reach them.

A runtime bundle can be tested before install.sh pins it. workflow_dispatch takes runtime_version, a brig-standalone-linux tag. A plan job checks its form and that the release exists, and then adds a second Linux leg that exports BRIG_RUNTIME_VERSION, which install.sh already reads. Each leg is its own host in the report, and the page names the bundle each one installed, read from its pins.env. A pull request and a scheduled run test the pin alone.

Not in this PR, and next: a failing scheduled run opening or updating one tracking issue.

Related issues

Refs #358

Changes

• script/e2e/canary-linux.sh, the Linux canary. LEVEL=canary is short. LEVEL=nightly boots more often, and adds a parallel block and stop/start churn. The gates:
  • a forced image pull with DOCKER_CONFIG unset (brig from main cannot boot or exec on macOS: the guest HOME reaches hull's own environment #337);
  • 100 short brig sh commands and 100 raw nerdctl exec, counting empty or wrong output;
  • claude-code boots, counting refusals that name .claude;
  • a no-sudo user's install, which must stop before the bundle download and print the commands for root;
  • --cpus, read from the bundle's pins.env: expected to fail on rc9 and older, enforced from rc10.
• Both canaries run guest commands with brig sh <ref> -c, on a pty from script(1). A check about the shared network names --network shared, since the shipped profiles start a new sandbox isolated.
• script/e2e/canary-macos.sh, for the SIP-enabled Macs, written for the bash 3.2 macOS ships. The Gatekeeper gate, and checks for version and doctor, every shipped profile, rm and --home, fix(wrap): Delete the guest home a failed first boot created #344, feat(cli)!: Remove brig shell #341, the Keep the network posture a sandbox was started with #342 postures on hvi (shared, offline and isolated, egress checked with curl from claude-code), vz's refusal of --network isolated, port publishing and the fix(wrap): refuse a symlink on the project path #332 refusal. A published port that takes about 30 s to answer is a warning: a guest that gets an address an earlier guest had comes with a new MAC.
• The checks both platforms share: version and doctor, every shipped profile, rm and --home, fix(wrap): Delete the guest home a failed first boot created #344, port publishing, the Keep the network posture a sandbox was started with #342 postures, feat(cli)!: Remove brig shell #341, the fix(wrap): refuse a symlink on the project path #332 symlink refusal with a canary checksum, and leftovers after every block.
• The boot-asset checks (Bind the boot-asset digest to the kernel and initrd that boot #234, feat(verify): Check the runtime bundle's record, fetch by digest #403), in both canaries. Each is skipped when the commit under test lacks the change it checks.
  • Linux reads the record a runtime bundle keeps from rc12 on: the kernel and initrd match SHA256SUMS, the release's checksums.txt lists it, and the bundle's cosign verifies the signature against install.sh's identity and refuses brig's own. It then changes the installed files one at a time: a changed kernel and a record the release does not list refuse under warn and require, a missing signature is "cannot check", and so is a cosign held by a proxy that never answers.
  • The Mac runs against a scratch HULL_BOOT_ASSETS directory: a first fetch pinned to the verified digest, a changed kernel refused under warn and not fetched over, a named copy refused under require, and an older bundle fetched again by digest.
• The checks for what landed on main since dc36d62:
  • Linux: nerdctl enforcing an egress policy (Add an enforceable egress path on Linux #266), the cases of docs/manual-tests/egress-policy-linux.md on claude-code sandboxes, since a policy attaches to an agent profile only. default: deny and default: allow are each checked against a sandbox with no policy, so a refusal is shown to come from the policy. A resolver killed with SIGKILL must fail closed, the next brig sh must boot the sandbox under a new resolver, a detach must leave no table, and rm must leave no table, resolver or record. This replaces the Linux half of the Refuse a run when the runtime cannot enforce a planned property #237 refusal check, which 784fd4e made obsolete. The Mac keeps it. Setup installs nftables when nft is missing.
  • Both: an unquoted files: mode is refused with a message that says to quote it, the quoted mode parses, and an override of a built-in with an unquoted mode is ignored while the built-in runs (Refuse an unquoted mode: in a files binding instead of applying another mode #453).
• The checks for what landed on main since a82606a, in both canaries:
  • claude-code's credential directory (fix(profile): Keep claude-code's tmpfs off the shared guest home #462): CLAUDE_CONFIG_DIR is /brig/claude on a tmpfs, and a file the guest writes there reaches no host directory. A host mv of the home's .claude, as git clean -fdx makes, leaves the tmpfs in place, and a second file stays off the host too. brig rewrites a hostmounted .claude.json in place (e1df7bd), so the guest's bind sees the trust key. The Linux claude gate now reads /brig/claude, and a refusal that names it counts as one.
  • brig's own telemetry (feat(telemetry): Send brig's own events #459): the block records a yes, points HULL_TELEMETRY_ENDPOINT at a local capture server, and expects one ok command event each for ls, sh and rm, with product brig and the install id brig telemetry status shows. hull adds the sandbox's start and end on macOS only when brig counts its events: a release or an rcN-main channel build from rc31 on. DO_NOT_TRACK=1 sends nothing, nothing stays queued, and the block records a no at the end. No event leaves the runner.
• What a run booted, in the report: on macOS the hvi commit the hull build pins (with hvi-vmm main beside it when they differ) and the boot assets brig fetched, from their bundle.json; on Linux the hull-assets version and the VMMs from pins.env. macOS also checks that hull is the build of hull's main, and skips while the channel has not published it yet.
• script/e2e/tap-check.sh: brew tap brig-sh/brig and brew readall brig-sh/brig, each a pass or a fail.
• .github/workflows/e2e.yml: nightly with LEVEL=nightly, workflow_dispatch with a level and a runtime version, and pull_request on script/e2e/**, the workflow and install.sh. Jobs plan, linux (a matrix of bundle legs: install.sh's pin, the newest bundle release when install.sh does not pin it yet, and a dispatched runtime version), macos, tap and report. The timing baseline is the report of the last scheduled run that passed.
• script/e2e/render-report.py and report-template.html: the renderer of the v0.3.0 release-gate page, with a new --summary mode for $GITHUB_STEP_SUMMARY. Template fixes: high-severity findings sorted after the low ones, an open item lost its links without a detail, and a page with no baseline showed an empty legend entry. The full document now puts the title and the styles in <head>, and the footer links to the workflow run.
• script/e2e/results.py and merge-results.py: how the scripts write results.json, one JSON line per result, and how the report job joins the hosts.
• script/e2e/test-render-report.sh: builds, merges and renders the fixtures, and checks that a bad status, a failing or missing gate and a missing host are caught. CI's shell job runs it, with shellcheck over script/e2e/.
• .github/actionlint.yaml names the sip-enabled runner label. CONTRIBUTING.md describes the new CI step, the e2e workflow and the runtime override.

Testing on this PR

e2e 37521497079 at f9d61f9, rebased on main f685ee9 (v0.4.0): all six jobs pass (plan, linux (linux), linux (linux-rc16), macos, tap and report), the verdict is go, and no check fails. CI and DCO are green on the same head.

• Linux, install.sh's pin v0.1.0-rc15 (hull-assets 0.1.9, urunc f6d54d6): 71 checks pass.
• Linux, the newest bundle v0.1.0-rc16, the leg the plan added on its own: 71 checks pass. rc16 carries brig v0.4.0 and the same runtime as rc15.
• macOS: 69 checks pass and claude-desktop boots is skipped (no console session on a runner). hull came from the channel build of 159772e, v0.1.0-rc31 plus its changelog commit, with hvi c53a857 and boot assets 0.1.10 (kernel 6.18.55, urunc 6c062f3).
• Homebrew tap: brew tap and brew readall brig-sh/brig pass.

The three /brig/claude checks and the telemetry check pass on all three hosts. The channel hull reports the Go pseudo-version v0.1.0-rc31.0.<date>-159772e, which brig's hullVersionAtLeast reads as a source build, so brig suppresses its sandbox events. The check follows that rule.

CodeQL's Analyze (python) failed on this head while uploading its results, after the analysis itself finished, and recorded no analysis. The Python files are the same as on fccb487, where it passed with no alerts.

The previous run, e2e 37509684368 at fccb487 on main c405276, passed the same way with rc14 as the pin and rc15 as the newest leg. The run before it, e2e 37507499577 at c3e1124, failed the telemetry block on every host (no hull on Linux, and the pseudo-version on macOS); both were fixed in fccb487. Its rc15 leg also saw 9.9.9.9:443 closed from a sandbox with no policy, twice, while an allow policy reached it. The same checks have passed on rc15 since.

The new checks also ran locally on this Mac (macOS 26.3.1, SIP off) against the brig they check and against brig from before each change:

check	brig before the change	brig at this head
claude-code credential tmpfs, 3 checks	8e9d2ff (before #462): 3 fail, no CLAUDE_CONFIG_DIR, no /brig/claude	3 pass
.claude.json rewritten in place	b769d28 (before e1df7bd): fail, inode replaced	pass, same inode
telemetry	a82606a (before #459): fail, hull's exec, run, stop instead of brig's ls, sh, rm	pass with hull 9f1d349 (rc30.x, no sandbox events) and with hull v0.1.0-rc31 (start, end)

A probe the guest writes on the home share is found by the same leak search, so the search is not blind.

The egress checks pass on the targets they name. For instance:

nerdctl enforces default: deny, and the allowed name still reaches (#266): under the policy: DNS_example.net=NONE TCP_example.com=OPEN TCP_9.9.9.9=CLOSED. With no policy: DNS_example.net=172.66.175.59 TCP_example.com=OPEN TCP_9.9.9.9=OPEN. Table brig_egress_claude-code-egd
A dead resolver fails closed (#266): 172.66.147.243:443 (example.com, allowed) opened; after a kill -9 of resolver 51961 it was refused within 15 s

Before the push, the Linux canary ran on a fresh Ubuntu 24.04 VM with nested KVM. Its exec gate once got \^@ok-76 from brig sh, a NUL byte ahead of the right output. The same loop on the VM gives it on dc36d62 too, so it predates this branch. It is tracked in #460.

The macOS job leaves nothing behind:

detached /private/tmp/be2e.WV16Si/.hull/store
removed /private/tmp/be2e.WV16Si
left: no process

The runtime_version leg was last run in e2e 36208511977 with rc10, and its --cpus gate passed on that bundle.

Locally: script/e2e/test-render-report.sh and shellcheck 0.9.0, the version CI installs, pass on the final tree.

Checklist

• make all passes (vet, test, build)
• script/smoke.sh passes
• I have added or updated tests covering the change
• I have run go test ./... -race, if the change touches concurrency, subprocesses or the daemon
• I have exercised the change against a real runtime (brig run <agent>), if it touches the run, exec or credential path
• I have updated the affected docs (see docs/README.md for the map)

Credentials and the sandbox boundary

Neither promise changes, since no Go code changes. The canary adds a negative test of the first one on a real runtime, on Linux and on macOS. A sandbox with the parent project read-write plants links to a directory outside it. Three runs through those links must be refused, the real directory next to them must still boot, and a checksum over the outside directory must match before and after.