[aviweit]

Branch: docs/reproduce-tau2-sierra-direct-blackbox → main

Summary

Adds docs/REPRODUCE_tau2_sierra_direct_blackbox.md, a reproduction guide for the two
examples/tau2_sierra_airline runs, one per delivery:

• direct (run_20261013_094928): the runner loads each candidate's tool code into its own process;
• blackbox (run_20261013_121303): each candidate is deployed to the Skillberry Store and the
  Skillberry Proxy-Agent injects it into the agent's LLM calls.

It also adds a Blackbox approach section at the bottom of the site's Results page, with the two runs' dashboard screenshots side by side, linking to the Blackbox tab and to this guide.

What the guide records

Every value comes from the two run directories or the committed specs:

• Results:

  • baseline val, val after each iteration, best candidate and sealed test score;
  • the seed's test score in the same session, the gain, pass^2 and the accepts;
  • optimizer spend, tokens and wall time.

  The sources are final.json, baseline.json, events.jsonl and state.json.

• Pinned τ²-bench: 4ce7c0397c1eb65c9bbe59aeacfe1ca44a1cd699 (TAU2_REFERENCE_SHA).

• Models:

  • agent aws/claude-haiku-4-5, which the blackbox agent reaches through the sentinel
    ibm/skillberry-local and the Proxy-Agent;
  • user simulator aws/claude-sonnet-5;
  • optimizer claude-opus-5-5;
  • all at temperature 0, as recorded in τ²-bench's own results.

• The split: split_ids_budget.json, the same 7 tasks (8, 10, 20, 23, 35, 42, 44) for train, val
  and test, at 5 trials each.

• The run spec: the committed direct/ and blackbox/capevolve.yaml, which differ only in
  intervention, skill_name and protected_paths.

• The .env:

  • placeholders for the gateway;
  • TAU2_AGENT_MODEL and TAU2_USER_MODEL;
  • a note that the blackbox adapter restarts the Proxy-Agent with SPA_MODEL_NAME set to
    TAU2_AGENT_MODEL.

• Reproduce: setup.sh and run.sh for direct and for blackbox, and --smoke for a check with no spend.

• What to compare: which run file answers which question.

Results recorded

	direct	blackbox
Baseline val (seed)	0.286	0.286
Val after iterations 1, 2, 3	0.571, 0.800, 0.914	0.514, 0.600, 0.771
Sealed test (champion cand_0003)	0.943	0.771
Test gain over the seed	+0.600	+0.400

The guide states that this is a no-holdout run (train = val = test), so the test number is a fit
metric. It also gives the two known proxy-side causes of the lower blackbox score.

Checks

• No sensitive data:

  • the .env example uses placeholders (<your gateway>, <your key>);
  • no API keys, tokens or passwords;
  • no internal gateway URL: τ²-bench's raw results contain one, and it is left out;
  • no local paths or user names.

  The only URL is the public Blackbox tab.

• Every linked file exists: examples/tau2_sierra_airline/{direct,blackbox}/capevolve.yaml and
  split_ids_budget.json.

• The specs match: the blackbox run's own project spec matches the committed
  blackbox/capevolve.yaml key for key.

• One gap: the direct run's project spec has since been overwritten, so it is documented from the
  committed direct/capevolve.yaml. Its values agree with what the run's events record (algorithm,
  optimizer, target model, 7 tasks × 5 trials, 3 iterations).

• Scope: docs and site only (this guide, the Results page and two screenshots); no code or CI change.

[skillberry-bot]

❌ Automatic Labeling Failed

An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually.