[erezh16]

Lets a spec keep benchmark gold data out of the optimizer's reach, for every optimizer CLI, and fail closed if that can't be guaranteed. Fixes #875.

Value

• Closes a real leak. Agent-CLI optimizers have a shell, and nothing stopped them reading the files a grader needs. In the AppWorld smoke test (Experiment design: CapEvolve on AppWorld (Suite F), DeepSeek-V3.2 and gpt-oss-120b #804) that is what happened: the claude-code optimizer's subagents read train/dev solutions and ran the evaluator, and the result had to be thrown away.
• Works where Claude Code's sandbox can't. The namespace mode needs only unprivileged user namespaces. It also works for any CLI in the registry (codex, gemini-cli, …), not just claude-code.
• Safe by default, provable before spending. Isolation is off unless the spec asks for it. When it is on but can't be provided, the optimizer call refuses to run. cap-evolve check proves the setup offline, with no model call.

optimizer_isolation: auto            # off (default) | auto | namespace | sandbox
optimizer_hide: [/data/gold, ~/.claude/projects]
optimizer_expose: [/data/gold/public_docs]   # optional: read-only inside a hidden dir

How it works. cap-evolve run validates the keys and exports the config, and run-optimizer applies it to each call:

• namespace (core/cap_evolve/_ns_wrap.sh): each hidden dir becomes an empty tmpfs, with the working dir and any exposed paths bound back, and each hidden file becomes an empty file. The call then runs in a nested user namespace as the same user, so it cannot unmount anything, and /proc/<pid>/root of outside processes is refused.
• sandbox: Claude Code's bubblewrap sandbox with denyRead, allowUnsandboxedCommands: false and failIfUnavailable.
• auto: sandbox for claude-code when a bwrap probe passes, else namespace.

For claude-code, permissions.deny Read(...) rules are passed inline in both modes, because the sandbox only wraps Bash. If the requested mode is unavailable, the call exits 2 with the reason.

---

Evidence

Why it's needed (#875, #804): in hill-climb round 2, 36 subagent tool calls read gold files, copied the data root, ran released solutions and the evaluator with gold traces, and ran find /. Those accepted edits became the champion.

Why not only Claude Code's sandbox: in the dev container used for the smoke test (rootless podman, /dev/null bound from the host), every sandboxed Bash command fails. bwrap can't remount the /dev/null bind it places over protected paths, and that bind can't be remounted at all from a user namespace. A newer bwrap and enableWeakerNestedSandbox fail the same way, and Read/Grep still need separate deny rules. The auto probe detects exactly this case.

This PR, tested in that container:

• core/tests/test_optimizer_isolation.py, 8 tests:
  • config parsing, including YAML off/on booleans;
  • mode choice and fail-closed behaviour;
  • the settings JSON and the command wrapping;
  • a real namespace run: a hidden dir and a hidden file read as empty, an exposed path is readable, the workdir is writable, the cover cannot be unmounted, and the host is untouched;
  • the offline self-check;
  • run-optimizer with the mock optimizer, wrapping a real call and refusing an impossible mode.
• Full core suite: 2,794 passed, 15 skipped. The only failures are 2 test_cli_surface tests that fail identically on current main in a fresh worktree (missing generated manifest).
• End to end with a real claude-code call through run-optimizer with optimizer_isolation: namespace: a random secret was planted in the hidden dir, and the optimizer was told to get it via Bash cat, the Read tool, Grep, find / and a copy. Result: 6 Bash, 1 Read and 1 Write tool calls, 0 occurrences of the secret in the transcript or result, the requested file written in the workdir, and "isolation": "namespace" in the result.
• The same mechanism, as a standalone wrapper, passed the AppWorld example's planted-secret check A9 (6 of 6 secrets hidden, normal work ran; examples(appworld): AppWorld benchmark with cap-evolve vs ALTK-Evolve, and optimizer isolation #874).

Not covered here: network isolation for the namespace mode; macOS (the sandbox mode covers it through Claude Code).

🤖 Generated with Claude Code

[skillberry-bot]

🏷️ Automatic Labeling

I've analyzed this pull request and added the following labels:

• bug - enhancement - honesty - bug - enhancement - honesty - priority-p0

These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually.