Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,24 +22,26 @@ pointer, and new work lands here.

## The tests in one paragraph each

- **Tests 1-4 and 7**: agentic coding against the Home Assistant REST API: a CLI, an MCP server
- **Tests 1-4**: agentic coding against the Home Assistant REST API: a CLI, an MCP server
built from a spec, using that server to answer questions about a recorded installation, and
ticket-sized fixes in a Rust project. Agents run in a locked-down container (`harness/sandbox/`)
that can reach only the model server and a replay of a recorded Home Assistant fixture
(`harness/ha-replay.py`), never a live house.
- **Test 7**: a seeded generator for a three-floor labyrinth, checked on unseen seeds.
- **Test 5 (orrery) and test 6 (pagoda)**: one reply, no tools, to a single creative-coding prompt
(`harness/suites/prompt-test5-orrery.txt`, `harness/suites/prompt-test6-pagoda.txt`), rendered
offline and graded on a five-part rubric (`harness/suites/test5/README.md`,
`harness/suites/test6/README.md`).
- **Test 8**: the same two prompts given to an agent with a locked-down browser that can see its own
page. Its runs are evidence and are not published yet (see below).
page. The agent's final page, its rendered frames and the grades are in `oneshots/agent-pagoda/`
and `oneshots/agent-orrery/`; the transcripts and work directories are not published (see below).

## Not here yet

Phase 1 publishes code, reports and the one-shot outputs only. These stay private until a further
scrub review:

- the test evidence: transcripts, work directories, per-case result sets and test 8 agent runs;
- the test evidence: transcripts, work directories and per-case result sets, test 8 included;
- the Home Assistant fixture, including its pseudonymized public version, and the test 3
known-good exercise derived from it.

Expand Down
9 changes: 8 additions & 1 deletion oneshots/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# One-shot replies: orrery (test 5) and pagoda (test 6)
# One-shot replies: orrery (test 5) and pagoda (test 6), and the same tasks as agent runs (test 8)

**Everything under `orrery/*/` and `pagoda/*/` except the frames is model output.** Models wrote these
SVG and HTML files in one reply, without tools, and nobody edited them. The operator read them and judged
Expand All @@ -12,6 +12,13 @@ the code safe to run, and the project site plays them (see below).
page's external URLs). Opening one makes those requests and runs the model's script.
- The frames (`*.webp`) were rendered offline, in a container with no network. Grades come from them.

## Test 8: `agent-orrery/` and `agent-pagoda/`

The same two prompts given to an agent with a sealed browser that can see its own page, a two-hour cap and the same
rubrics. Each case has the file the agent left (`orrery.svg` or `page.html`, unedited), three frames rendered offline,
and a row in that folder's `grades.tsv`. The operator judged these files safe to run too, and the site plays them the
same way. The transcripts and work directories are not published.

## Layout

```
Expand Down
11 changes: 11 additions & 0 deletions oneshots/agent-orrery/grades.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Test 8 grades (agent with a sealed browser; same rubric as tests 5/6), from the private bake-off grades.
case entry cap finish tokens wall_s tok_s A B C D E total note
t8-orrery-uncvllm-r1 qwen3.8-27b-uncensored-vllm agent done 68536.0 5153.6 13.3 2.0 2.0 2.0 2.0 2.0 10.0 Vision off; 22 browser calls, 2 screenshots, 5 compactions, 86 min. Six planets, Mercury to Saturn, at exact period ratios; ring gears and pinions mesh internally; the night side faces away from the sun while the surface turns.
t8-orrery-orcasaq2-r1 lcpp6-orcasaq2-cyber-27b agent done 302501.0 5467.9 55.3 1.0 2.0 2.0 1.0 2.0 8.0 Vision off; 35 browser calls, 7 screenshots, 16 compactions, 91 min. Six planets at exact ratios and everything animates, but plates cover the orbit circles and the gear train is hidden (D 1).
t8-orrery-coder390-r1 lcpp6-q38-coder390 agent done 69964.0 1544.2 45.3 1.0 1.0 2.0 0.0 1.0 5.0 Vision off; 24 browser calls, 5 screenshots, 5 compactions, 26 min. The orbit groups rotate about the top-left corner (the animation replaces the translate), so the planets leave the frame; the gear rings have no visible teeth.
t8-orrery-thinkcap-r1 lcpp6-thinkingcap-27b-heretic time cap 185456.0 7200.1 25.8 2.0 2.0 2.0 1.0 1.0 8.0 Stopped at the 2 h cap and graded as left; 1 browser call, 1 compaction. Eight planets at exact ratios on brass ring gears; the gear train does not demonstrably drive the orbits (D 1); the night shade is fixed because planets do not spin (E 1).
t8-orrery-q38cyberq5-r1 lcpp6-q38-27b-cyber-q5 time cap 192790.0 7200.1 26.8 0.0 0.0 0.0 0.0 0.0 0.0 Stopped at the 2 h cap with 16 compactions and no browser call; orrery.svg was a 0-byte file, so nothing rendered.
t8-orrery-cybertiel-r1 cyber-tiel-coder-35b agent done 92519.0 2199.6 42.1 2.0 2.0 2.0 0.0 2.0 8.0 Vision on; 58 browser calls, 16 screenshots, 37 min. Eight planets at exact ratios; orbits and spins animate; the gears are plain discs with no teeth (D 0).
t8-orrery-heretic2-r1 lcpp6-flashnext-heretic2 agent done 44616.0 3179.0 14.0 1.0 2.0 2.0 1.0 1.0 7.0 Vision off; 5 browser calls, 53 min. Eight planets at exact ratios and the arms orbit, but the gear paths are drawn at the corner and cover the title, so no pair meshes (D 1); fixed crescent shade (E 1).
t8-orrery-abl15-r1 lcpp6-flashnext-abliterated-s15 agent done 14879.0 1091.8 13.6 1.0 1.0 0.0 1.0 0.0 3.0 Vision off; 9 browser calls, 1 screenshot, 18 min. Planet gears pile up at the top-left corner and nothing orbits (C 0); the planets are flat discs with no night side (E 0).
t8-orrery-orcarouter-r1 lcpp6-orcarouter-fn-uncensored agent done 75472.0 5243.4 14.4 2.0 2.0 2.0 2.0 2.0 10.0 Vision off; 19 browser calls, 87 min. Eight planets at exact ratios; each arm's toothed pinion meshes with the sun gear and obeys the planetary-gear law; the night gradient faces away from the sun while the surface turns.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 14 additions & 0 deletions oneshots/agent-orrery/t8-orrery-abl15-r1/orrery.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading