[erezh16]

Adds examples/appworld: a ready-to-run AppWorld benchmark for cap-evolve. It includes an ALTK-Evolve arm on the same harness, so cap-evolve can be compared with another learning method under equal conditions. It also closes an isolation hole that let the optimizer read benchmark answers.

Value

• A new benchmark for cap-evolve. AppWorld (everyday multi-app tasks, graded on the final app state) is widely used for agent learning. With this example, cap-evolve can be scored on it in one command per stage.
• A fair comparison by construction. All arms use the same agent harness, model client and limits. Only AppWorld's official splits are used, and all arms are scored together in one interleaved test window. Feedback is gold-free, and candidate lints enforce AppWorld's "no task-specific hints" rule.
• Optimizer isolation that works where Claude Code's sandbox can't start. A claude wrapper runs every optimizer call in a private mount namespace in which the answers simply do not exist. A planted-secret check proves this before a run.

---

Evidence (smoke test, 13–15 Oct 2026: gpt-oss-120b agent, Claude Sonnet 5.5 optimizer)

Detailed log: issue #804 (interim updates 1–6 and summary). Plan, findings (Appendix A) and results (Appendix B): docs/EVAL_SMOKE_TEST.md on the fork branch erezh-eval1. Plain-language summary: docs/EVAL_SMOKE_TEST_SUMMARY.html.

The example ran end to end. Acceptance checks A1–A8 passed:

• the seed prompt is byte-identical to stock;
• a released solution scores 1 and a no-op scores 0;
• persisted records hold no gold values;
• REPL isolation holds;
• the key never reaches a log;
• the longest of 10 dev episodes peaked at 12.2K prompt tokens.

Held-out results (test_normal, 168 tasks × 3 arms × 2 trials = 1,008 episodes, 0 infra errors):

arm	task success	scenario success	Δ task success vs seed [95% CI]
seed (stock ReAct)	35.1%	13.4%	—
cap-evolve hill-climb ⚠	53.6%	34.8%	+18.5 [+10.7, +26.2]
ALTK-Evolve (retrieval dose)	41.1%	23.2%	+6.0 [−0.9, +13.1]

⚠ The first run is labelled isolation breach. In round 2, the optimizer's subagents read train/dev ground truth and ran AppWorld's evaluator; no test data was reached. That breach is why the isolation in this PR exists. A clean re-run with it is planned.

AppWorld gaps found and handled in the harness, for every arm:

• AppWorld's REPL guard allows os, pathlib, io and os.environ. Agent code could list and read files, including ground truth, and read env vars. Those modules are now blocked, and the key is dropped from the environment.
• max_output_tokens_per_task is never enforced at 42b5bcf, so the episode loop enforces it.
• AppWorld's failure trace holds gold values. Only requirement texts and counters are kept, and native output dirs are deleted.
• The PyPI release has no simplified ReAct agent, so the setup builds AppWorld from source at 42b5bcf (pydantic 2.12).

ALTK-Evolve integration:

• The full playbook (1,715 guidelines, about 83K tokens) does not fit a 40K-context model. The dose lint marks it infeasible, so the retrieval dose is used.
• The gateway rejects litellm's forced tool_choice, so a process-local shim routes the library to its own prompt-JSON path. The library itself is unmodified.

Isolation (check A9, plants 6 secrets and runs a real optimizer call that tries cat, Read, Grep, find /, a copy and a sandbox escape):

mechanism	A9 in this dev container
sandbox/claude namespace wrapper	pass: 0 secrets found; files written; lint ran
Claude Code's sandbox (bubblewrap + socat)	cannot start: the container's /dev/null bind cannot be remounted. 0 secrets found, but no shell command can run
the wrapper, from this branch (settings injected by the wrapper)	pass

Tests: python -m pytest examples/appworld/tests gives 15 passed (lints, injection, gold-free record, scoring, ALTK conversion), with no AppWorld install or model needed.

The framework-level gaps that this example works around are filed as separate issues.

🤖 Generated with Claude Code

[skillberry-bot]

🏷️ Automatic Labeling

I've analyzed this pull request and added the following labels:

• enhancement - algorithm - documentation - enhancement - algorithm - documentation - honesty

These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually.

[OsherElhadad]

Code review findings

Reviewed the diff plus the checked-out branch (appworld-example @ a0996fb54) and AppWorld's actually-installed source. Most severe first.

1. Lint safety gate likely broken in the real launch path

examples/appworld/adapters/lints.py:32 reads env var APPWORLD_ROOT (hardcoded fallback /tmp/aw_eval/root_opt), while adapter.py's _appworld_root() (lines 48-49) reads a different env var, AW_APPWORLD_ROOT, falling back to AW_EVAL-derived root_opt. launch_cape_hc.sh — the actual search-launch script — sets neither. Only setup_project.sh sets APPWORLD_ROOT, and only for a one-off cap-evolve check call; final_window.py only sets AW_APPWORLD_ROOT.

On top of that, the optimizer is instructed to run lints.py itself inside its sandboxed mount-namespace (sandbox/claude:22), where root_opt is not mounted — only venv-aw, appworld/src, appworld/experiments, reference, lint_cache.json are allow-listed back in. The lint cache that would let lints.py run without touching root_opt is only ever built by write_cache() / --write-cache, which no script, README step, or CI step calls — only the unit test does.

Net effect: in the realistic launch path, the optimizer's required python lints.py . call hits a cold cache, tries to read root_opt/data/tasks/*/specs.json (invisible inside its sandbox), and raises FileNotFoundError instead of the {"ok": true} its instructions require before stopping — breaking the "lint before you stop" safety gate rather than just degrading it.

2. os-module sandbox bypass via submodule attributes

aw_agent.py:217 (harden_guard) removes os/pathlib/io/shutil/glob from AppWorld's ALLOWED_MODULE_NAMES, but AppWorld's installed safety_guard.py only inspects literal import statements (parse_imports) — it never walks attribute chains. Modules still allowed (e.g. uuid, contextlib) expose a live os submodule as a plain attribute (verified: uuid.os, contextlib.os both resolve to the real os module). Agent code can do import uuid; uuid.os.environ.get(...) and reach os directly, undermining the sandboxing this example's secrets story depends on.

3. Unredacted traceback on the exception path

aw_agent.py:356 — main()'s except branch writes traceback.format_exc()[-4000:] verbatim into the output record, unlike the normal path where evaluation_fields() carefully strips AppWorld's gold-bearing failure trace first. Any internal exception can leak task state/paths into out.json and downstream logs/optimizer feedback.

4. Lint read ordering skips the readiness wait

adapter.py:199 — run_trials() calls lints.lint_candidate(cdir) before _wait_readable(root) (line 206), while every other root_opt read in this file (including _split_ids and run_trials itself a few lines later) waits for readability first. Per the README, root_opt is locked while the optimizer runs; a cold-cache lint call during that window risks a transient read error or reading a half-written tree.

5. Secrets cleanup only covers one key, not all loaded

aw_agent.py:74 — load_secrets() applies os.environ.setdefault for every KEY=VALUE line in the secrets file, but cleanup (aw_agent.py:267, os.environ.pop(KEY_ENV, None)) and adapter.py's subprocess env filter (line 166) only strip the single configured key. A second credential in the secrets file would persist in the episode subprocess's env for the rest of the episode — exploitable via finding #2's os-bypass.

6. Sandbox bind-mount prefix-strip bug

sandbox/claude:54 — rel="${WORK#"$AW"/}" strips $AW/ (with trailing slash) from $WORK (no trailing slash). If the optimizer's cwd is exactly $AW_EVAL itself, the strip doesn't match and rel becomes $AW's own absolute path, producing a nonsense nested bind mount while leaving the real working directory unbound inside the sandboxed namespace.

7. Duplicated bootstrap-CI logic

scripts/analyze.py:50 — bootstrap() reimplements a percentile-bootstrap CI from scratch, duplicating core/cap_evolve/stats.py's existing bootstrap_ci() (the project's designated single source of truth for eval-statistics aggregation).

8. Hand-rolled sandbox reimplements the host's own isolation

examples/appworld/sandbox/claude (whole file, ~75 lines) reimplements mount-namespace/bind-mount isolation that Claude Code's own bubblewrap sandbox should provide, only needed because of a container limitation (/dev/null bind issue). This now requires its own permanent isolation audit (check_optimizer_isolation.sh) instead of fixing the actual gap.

9. Unverified upstream constructor compatibility (lower confidence)

aw_agent.py:252-265 — build_agent_class() passes 8 kwargs into the upstream appworld_agents.code.simplified.react_code_agent.SimplifiedReActCodeAgent constructor via **kwargs. That package isn't vendored and couldn't be installed in review (no registry access), so the signature match couldn't be verified. If upstream renamed/dropped a kwarg, episode construction fails with a generic opaque infra_error. Worth a smoke-test run before merge.

---

Lower-confidence items dropped after verification: minor duplicate progress-logger/token-counter/dotenv code across adapter.py/altk_common.py (cosmetic); a latent joblib cache secret-persistence path in cached_lm_call (unreachable today since use_cache=False); Adapter.tasks() hardcoding ROOT_OPT instead of AW_APPWORLD_ROOT (currently unreachable in the shipped call graph).

🤖 Generated with Claude Code

[erezh16]

Thanks for the review. All nine findings are fixed on this branch, one commit per finding (each with the tests that show it), plus a second pass on #2. Everything was checked against AppWorld at the pin (42b5bcf), installed from source.

#	Finding	Fix
1	Lint gate broken in the launch path	d6799d4
2	os reachable through attributes	f063a9a, 22ba65e
3	Unredacted traceback	22734b9
4	Lint before the readiness wait	1eac7e2
5	Only one secret removed	6074b94
6	Bind-mount prefix strip	3de79f1
7	Duplicated bootstrap CI	a2bcce4
8	Hand-rolled sandbox	86b5fd9
9	Constructor compatibility	5e66af0

1. Lint gate. Confirmed. The step that built the lint cache was lost when this PR was cut. lints.py now reads its data only from the cache (AW_LINT_CACHE) or root_opt, never from the episode root, which is root_full in the final window. make_roots.py builds the cache. setup_project.sh, launch_cape_hc.sh and check_optimizer_isolation.sh refuse to start without it. The optimizer instructions point at the cache, not at root_opt. With no data, the CLI prints "ok": false ("lint data unavailable") and exits 2 instead of raising FileNotFoundError. A real isolation check (A9) through the sandbox now shows the optimizer's lint printing "ok": true.

2. Attribute bypass. Confirmed. harden_guard() now wraps SafetyGuard.is_syntax_safe with an AST check, but a second pass in the real REPL found more routes than uuid.os:

• private aliases such as random._os, collections._sys and enum.bltns;
• importers such as pydoc.locate("os") and unittest.mock.patch("os…");
• yaml tags such as yaml.load("!!python/name:os.environ");
• string walks such as operator.attrgetter("_os.environ") and "{0._os}".format(m).

All of these are refused now:

• the check also refuses private (_x) and dunder attributes, frame access, getattr with a blocked constant or on a module, eval/exec/vars/globals, attrgetter/methodcaller, format fields that walk attributes or indexes, and IPython !/% lines;
• pydoc, unittest and yaml are removed from the allowlist.

In the real REPL, all 42 aliases of a blocked module across AppWorld's allowed modules are refused, and no planted secret leaks. A scripted agent episode doing normal work (API docs, login, listing, json/datetime/re, complete_task) runs with 0 refusals, and the A4 check still passes.

A Python-level guard can't be complete, and the README now says so. The other controls are #5, the per-episode counters guard_blocked/guard_probe (reported by analyze.py), and keeping answers out of root_opt during search.

3. Traceback. Confirmed, though the exposure was narrower than stated. The traceback field was never read and out.json is deleted, but str(e) and the worker's stderr did reach the optimizer's feedback. Errors now report the exception type, the phase and the frames (file:line in func). The message is kept only outside evaluate, with paths and the key replaced and a 300-character cap. A crashed worker's stderr is reduced to its exception type.

4. Readiness wait. Fixed: run_trials now waits, then lints. After #1 the lint reads only the cache, so the order no longer matters at runtime. The stale "root_opt is locked" comments are gone.

5. Secrets. Confirmed, and the bigger leak was the adapter: it gave the worker the whole cap-evolve run environment, including the optimizer's credentials, minus one key. The worker's environment is now built from an allowlist (aw_agent.WORKER_ENV_NAMES). load_secrets() sets only the agent key and the endpoint URL, and drop_secrets() removes the key once the client is built.

6. Prefix strip. Fixed. A working dir equal to $AW_EVAL is refused, AW is resolved with readlink -f (a symlinked AW_EVAL used to fail every workdir), and an empty or absolute rel aborts. The tests run the shim in real user and mount namespaces.

7. Bootstrap. Fixed. With equal-size scenarios (56 × 3), the clustered paired ΔTGC and ΔSGC are means over scenarios of per-scenario differences, so analyze.py now calls cap_evolve.stats.bootstrap_ci on those differences and checks that the scenario sizes are equal.

8. Hand-rolled sandbox. Partly agreed. Claude Code's built-in sandbox is now the default (AW_ISOLATION=sandbox). The namespace code stays as a fallback in sandbox/ns_exec.sh for hosts where bubblewrap can't start (/dev/null bound from the host). sandbox/claude is now a short dispatcher. The isolation check stays either way, because it is the acceptance test for any mechanism, and launch_cape_hc.sh refuses to start without a passing check for its mode that is newer than the rendered settings. A real check passes in both modes here, with 0 of 6 planted secrets leaked.

9. Constructor compatibility. Checked against the source at 42b5bcf. The episode passes 10 kwargs (not 8), and Agent.__init__ or SimplifiedReActCodeAgent.__init__ accepts every one. check_injection.py (A4) now names any kwarg the upstream agent stops accepting, and the episode raises a labelled error instead of an opaque infra_error. A4 passes against the real install.

Items you dropped:

• cached_lm_call: still unreachable, since use_cache=False.
• Adapter.tasks() using ROOT_OPT: intentional (search tasks only), now commented.
• The ALTK duplicate code: gone with the ALTK arm (below).

Other changes in this push:

• The ALTK-Evolve arm is removed. It is a competing optimizer, not a benchmark, so the example now compares seed with cape-hc.
• scripts/itest.sh adds a one-task integration test, modelled on tau2_airline/itest.sh (train task 82e2fac_1, 1 trial, 1 iteration). Its asserter adds two checks to tau2's: no infra-errored or lint-rejected rollout, and no gold-bearing record.
• There is a CI leg in Integration tests (bench: appworld). ci_setup.sh provisions AppWorld in the runner cache, verified from both an empty and a warm cache.
• The tests are laid out one file per module, as in examples/parsec/tests.

Not verified yet: a scored end-to-end run. RITS is unreachable from my machine, so the itest finished every step but scored nothing. Every episode timed out, and the asserter correctly failed it rather than passing a 0.0-vs-0.0 "no regression". The first appworld dispatch on the runner is that run.

🤖 Generated with Claude Code