diff --git a/README.md b/README.md index 41ab5fa..c9637d0 100644 --- a/README.md +++ b/README.md @@ -22,24 +22,26 @@ pointer, and new work lands here. ## The tests in one paragraph each -- **Tests 1-4 and 7**: agentic coding against the Home Assistant REST API: a CLI, an MCP server +- **Tests 1-4**: agentic coding against the Home Assistant REST API: a CLI, an MCP server built from a spec, using that server to answer questions about a recorded installation, and ticket-sized fixes in a Rust project. Agents run in a locked-down container (`harness/sandbox/`) that can reach only the model server and a replay of a recorded Home Assistant fixture (`harness/ha-replay.py`), never a live house. +- **Test 7**: a seeded generator for a three-floor labyrinth, checked on unseen seeds. - **Test 5 (orrery) and test 6 (pagoda)**: one reply, no tools, to a single creative-coding prompt (`harness/suites/prompt-test5-orrery.txt`, `harness/suites/prompt-test6-pagoda.txt`), rendered offline and graded on a five-part rubric (`harness/suites/test5/README.md`, `harness/suites/test6/README.md`). - **Test 8**: the same two prompts given to an agent with a locked-down browser that can see its own - page. Its runs are evidence and are not published yet (see below). + page. The agent's final page, its rendered frames and the grades are in `oneshots/agent-pagoda/` + and `oneshots/agent-orrery/`; the transcripts and work directories are not published (see below). ## Not here yet Phase 1 publishes code, reports and the one-shot outputs only. These stay private until a further scrub review: -- the test evidence: transcripts, work directories, per-case result sets and test 8 agent runs; +- the test evidence: transcripts, work directories and per-case result sets, test 8 included; - the Home Assistant fixture, including its pseudonymized public version, and the test 3 known-good exercise derived from it. diff --git a/oneshots/README.md b/oneshots/README.md index c18f47a..6c471a3 100644 --- a/oneshots/README.md +++ b/oneshots/README.md @@ -1,4 +1,4 @@ -# One-shot replies: orrery (test 5) and pagoda (test 6) +# One-shot replies: orrery (test 5) and pagoda (test 6), and the same tasks as agent runs (test 8) **Everything under `orrery/*/` and `pagoda/*/` except the frames is model output.** Models wrote these SVG and HTML files in one reply, without tools, and nobody edited them. The operator read them and judged @@ -12,6 +12,13 @@ the code safe to run, and the project site plays them (see below). page's external URLs). Opening one makes those requests and runs the model's script. - The frames (`*.webp`) were rendered offline, in a container with no network. Grades come from them. +## Test 8: `agent-orrery/` and `agent-pagoda/` + +The same two prompts given to an agent with a sealed browser that can see its own page, a two-hour cap and the same +rubrics. Each case has the file the agent left (`orrery.svg` or `page.html`, unedited), three frames rendered offline, +and a row in that folder's `grades.tsv`. The operator judged these files safe to run too, and the site plays them the +same way. The transcripts and work directories are not published. + ## Layout ``` diff --git a/oneshots/agent-orrery/grades.tsv b/oneshots/agent-orrery/grades.tsv new file mode 100644 index 0000000..f95b688 --- /dev/null +++ b/oneshots/agent-orrery/grades.tsv @@ -0,0 +1,11 @@ +# Test 8 grades (agent with a sealed browser; same rubric as tests 5/6), from the private bake-off grades. +case entry cap finish tokens wall_s tok_s A B C D E total note +t8-orrery-uncvllm-r1 qwen3.8-27b-uncensored-vllm agent done 68536.0 5153.6 13.3 2.0 2.0 2.0 2.0 2.0 10.0 Vision off; 22 browser calls, 2 screenshots, 5 compactions, 86 min. Six planets, Mercury to Saturn, at exact period ratios; ring gears and pinions mesh internally; the night side faces away from the sun while the surface turns. +t8-orrery-orcasaq2-r1 lcpp6-orcasaq2-cyber-27b agent done 302501.0 5467.9 55.3 1.0 2.0 2.0 1.0 2.0 8.0 Vision off; 35 browser calls, 7 screenshots, 16 compactions, 91 min. Six planets at exact ratios and everything animates, but plates cover the orbit circles and the gear train is hidden (D 1). +t8-orrery-coder390-r1 lcpp6-q38-coder390 agent done 69964.0 1544.2 45.3 1.0 1.0 2.0 0.0 1.0 5.0 Vision off; 24 browser calls, 5 screenshots, 5 compactions, 26 min. The orbit groups rotate about the top-left corner (the animation replaces the translate), so the planets leave the frame; the gear rings have no visible teeth. +t8-orrery-thinkcap-r1 lcpp6-thinkingcap-27b-heretic time cap 185456.0 7200.1 25.8 2.0 2.0 2.0 1.0 1.0 8.0 Stopped at the 2 h cap and graded as left; 1 browser call, 1 compaction. Eight planets at exact ratios on brass ring gears; the gear train does not demonstrably drive the orbits (D 1); the night shade is fixed because planets do not spin (E 1). +t8-orrery-q38cyberq5-r1 lcpp6-q38-27b-cyber-q5 time cap 192790.0 7200.1 26.8 0.0 0.0 0.0 0.0 0.0 0.0 Stopped at the 2 h cap with 16 compactions and no browser call; orrery.svg was a 0-byte file, so nothing rendered. +t8-orrery-cybertiel-r1 cyber-tiel-coder-35b agent done 92519.0 2199.6 42.1 2.0 2.0 2.0 0.0 2.0 8.0 Vision on; 58 browser calls, 16 screenshots, 37 min. Eight planets at exact ratios; orbits and spins animate; the gears are plain discs with no teeth (D 0). +t8-orrery-heretic2-r1 lcpp6-flashnext-heretic2 agent done 44616.0 3179.0 14.0 1.0 2.0 2.0 1.0 1.0 7.0 Vision off; 5 browser calls, 53 min. Eight planets at exact ratios and the arms orbit, but the gear paths are drawn at the corner and cover the title, so no pair meshes (D 1); fixed crescent shade (E 1). +t8-orrery-abl15-r1 lcpp6-flashnext-abliterated-s15 agent done 14879.0 1091.8 13.6 1.0 1.0 0.0 1.0 0.0 3.0 Vision off; 9 browser calls, 1 screenshot, 18 min. Planet gears pile up at the top-left corner and nothing orbits (C 0); the planets are flat discs with no night side (E 0). +t8-orrery-orcarouter-r1 lcpp6-orcarouter-fn-uncensored agent done 75472.0 5243.4 14.4 2.0 2.0 2.0 2.0 2.0 10.0 Vision off; 19 browser calls, 87 min. Eight planets at exact ratios; each arm's toothed pinion meshes with the sun gear and obeys the planetary-gear law; the night gradient faces away from the sun while the surface turns. diff --git a/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-100.webp new file mode 100644 index 0000000..df1193b Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-2000.webp new file mode 100644 index 0000000..7ec5032 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-6000.webp new file mode 100644 index 0000000..0648ce7 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-abl15-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-abl15-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-abl15-r1/orrery.svg new file mode 100644 index 0000000..e0fd536 --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-abl15-r1/orrery.svg @@ -0,0 +1,14 @@ +Brass Clockwork OrreryBrass clockwork orrery: planets orbit the sun at their real relative periods; brass gears mesh and turn to drive each orbit; a day-night shadow sweeps each planet. + + + + + + + + + + + + +MERCURYVENUSEARTHMARSJUPITERSATURN \ No newline at end of file diff --git a/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-100.webp new file mode 100644 index 0000000..aca0fd0 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-2000.webp new file mode 100644 index 0000000..8c8c6f3 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-6000.webp new file mode 100644 index 0000000..bdcf055 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-coder390-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-coder390-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-coder390-r1/orrery.svg new file mode 100644 index 0000000..48f74f2 --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-coder390-r1/orrery.svg @@ -0,0 +1,193 @@ + + + Brass clockwork orrery + Six planets orbit a brass sun at true relative periods; each orbit is driven by a brass pinion meshing a fixed ring gear and a stepped sun arbor, and a day/night shadow sweeps each planet as it turns. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + MERCURY + + + + VENUS + + + + EARTH + + + + MARS + + + + JUPITER + + + + SATURN + + + BRASS CLOCKWORK ORRERY — TRUE RELATIVE ORBITAL PERIODS + diff --git a/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-100.webp new file mode 100644 index 0000000..a18f4d9 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-2000.webp new file mode 100644 index 0000000..e144b96 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-6000.webp new file mode 100644 index 0000000..454e5d9 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-cybertiel-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/orrery.svg new file mode 100644 index 0000000..63d1e34 --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-cybertiel-r1/orrery.svg @@ -0,0 +1,544 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-100.webp new file mode 100644 index 0000000..8531117 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-2000.webp new file mode 100644 index 0000000..d5119d4 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-6000.webp new file mode 100644 index 0000000..b580ac8 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-heretic2-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-heretic2-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-heretic2-r1/orrery.svg new file mode 100644 index 0000000..a1a7e9e --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-heretic2-r1/orrery.svg @@ -0,0 +1,241 @@ + + +A Brass Clockwork Orrery +An animated brass clockwork orrery. Each ring gear carries a planet and revolves around the sun with a period proportional to the planet's real orbital period (Mercury 88 d, Venus 225 d, Earth 365 d, Mars 687 d, Jupiter 4333 d, Saturn 10759 d, Uranus 30687 d, Neptune 60190 d), so the planets orbit at their true relative speeds. The toothed wheels visibly mesh and turn as they drive the rings, and the day-and-night shadow sweeps across each planet as it turns. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +HOROLOGIUM COSMICUM +· THE PLANETARY GEAR TRAIN · PLANETS AT THEIR TRUE RELATIVE SPEEDS · + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-100.webp new file mode 100644 index 0000000..ac682eb Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-2000.webp new file mode 100644 index 0000000..b02b14a Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-6000.webp new file mode 100644 index 0000000..55bc180 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-orcarouter-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/orrery.svg new file mode 100644 index 0000000..7336351 --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-orcarouter-r1/orrery.svg @@ -0,0 +1,200 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +CLOCKWORK ORRERY + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-100.webp new file mode 100644 index 0000000..98dfaab Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-2000.webp new file mode 100644 index 0000000..b92f41e Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-6000.webp new file mode 100644 index 0000000..12a125f Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/orrery.svg new file mode 100644 index 0000000..c6f93a5 --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-orcasaq2-r1/orrery.svg @@ -0,0 +1,448 @@ + + +Brass Clockwork Orrery +An orrery of brass clockwork: the planets orbit the sun at their true relative speeds, driven by a meshing gear train; day and night shadows sweep across each turning planet. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +A BRASS CLOCKWORK ORRERY +THE PLANETS ORBIT AT THEIR TRUE RELATIVE SPEEDS + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Mercury +Venus +Earth +Mars +Jupiter +Saturn + \ No newline at end of file diff --git a/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-100.webp new file mode 100644 index 0000000..97d0e49 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-2000.webp new file mode 100644 index 0000000..97d0e49 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-6000.webp new file mode 100644 index 0000000..97d0e49 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-q38cyberq5-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-100.webp new file mode 100644 index 0000000..c7980db Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-2000.webp new file mode 100644 index 0000000..d34eeb0 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-6000.webp new file mode 100644 index 0000000..5972ee0 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-thinkcap-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/orrery.svg new file mode 100644 index 0000000..4d831fc --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-thinkcap-r1/orrery.svg @@ -0,0 +1,447 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +CLOCKWORK ORRERY + \ No newline at end of file diff --git a/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-100.webp b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-100.webp new file mode 100644 index 0000000..11c312c Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-100.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-2000.webp b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-2000.webp new file mode 100644 index 0000000..291c9ea Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-2000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-6000.webp b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-6000.webp new file mode 100644 index 0000000..4358958 Binary files /dev/null and b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/frames/frame-6000.webp differ diff --git a/oneshots/agent-orrery/t8-orrery-uncvllm-r1/orrery.svg b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/orrery.svg new file mode 100644 index 0000000..13bd4b7 --- /dev/null +++ b/oneshots/agent-orrery/t8-orrery-uncvllm-r1/orrery.svg @@ -0,0 +1,425 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +CLOCKWORK ORRERY +brass planetary mechanism · orbits at true relative rates · 1 year ≈ 45 s +planetary 72:8 = 9:1 and mainspring 16:8 = 2:1 · all meshes no-slip · day/night terminators sweep with each planet's 10:1 rotation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +MAINSpring 16:8 ≈ 2:1 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +MERCURY · 88.0 d +VENUS · 224.7 d +EARTH · 365.3 d +MARS · 687.0 d +JUPITER · 11.86 y +SATURN · 29.46 y + \ No newline at end of file diff --git a/oneshots/agent-pagoda/grades.tsv b/oneshots/agent-pagoda/grades.tsv new file mode 100644 index 0000000..3686b5b --- /dev/null +++ b/oneshots/agent-pagoda/grades.tsv @@ -0,0 +1,14 @@ +# Test 8 grades (agent with a sealed browser; same rubric as tests 5/6), from the private bake-off grades. +case entry cap finish tokens wall_s tok_s A B C D E total note +t8-pagoda-uncvllm-r1 qwen3.8-27b-uncensored-vllm agent done 64192.0 5432.2 11.8 2.0 2.0 2.0 2.0 2.0 10.0 Vision off; 42 browser calls, 6 compactions, 90 min. Five cream and red storeys with raised eave blocks, stepped gold spire, stone platform, three cherry trees, pond with footbridge, red torii; orbit by code, petals, shadows. +t8-pagoda-orcasaq2-r1 lcpp6-orcasaq2-cyber-27b agent done 59423.0 1116.7 53.2 1.0 2.0 2.0 2.0 2.0 9.0 Vision off; 30 browser calls, 4 screenshots, 2 compactions, 19 min. Everything draws; the eave tips are detached stepped blocks that float apart from the roofs (A 1). Four cherry trees, pond, torii, lantern. +t8-pagoda-coder390-r1 lcpp6-q38-coder390 agent done 30856.0 632.1 48.8 2.0 2.0 2.0 2.0 2.0 10.0 Vision off; 14 browser calls, 1 compaction, 11 min. Five storeys with raised double eave blocks, stepped gold spire, three trees, round pond, large torii. Title glyphs show as boxes (no CJK font in the renderer; not scored). +t8-pagoda-thinkcap-r1 lcpp6-thinkingcap-27b-heretic agent done 134514.0 6849.8 19.6 2.0 2.0 1.0 2.0 2.0 9.0 Vision off; 32 browser calls, 1 compaction, 114 min. Clean render, ringed red and gold spire; the eave upturns are single cubes level with the upper layer, so none visibly rises (C 1). +t8-pagoda-q38cyber-r1 q38-27b-cyber agent done 60672.0 1974.0 30.7 1.0 2.0 1.0 1.0 2.0 7.0 Vision off; 48 browser calls, 5 screenshots, 33 min. The camera sits too close: only storeys 2 to 4 are in frame, so the top storey and spire never show (C 1); the pond is buried under the platform (D 1). +t8-pagoda-q38cyberq5-r1 lcpp6-q38-27b-cyber-q5 agent done 18525.0 675.6 27.4 1.0 2.0 0.0 2.0 2.0 7.0 Vision off; 25 browser calls, 1 screenshot, 1 compaction, 11 min. Dark sky and fog with the camera so close that blossom cubes block the view; no recognisable pagoda, the spire never shows (C 0). +t8-pagoda-cybertiel-r1 cyber-tiel-coder-35b agent done 124735.0 3140.6 39.7 2.0 1.0 1.0 2.0 2.0 8.0 Vision on; 75 browser calls, 6 screenshots, 52 min. The ground is a plane and the sky a sphere (B 1); the spire is a very long pole that runs out of frame (C 1); two cherry trees, one standing in the pond. +t8-pagoda-heretic2-r1 lcpp6-flashnext-heretic2 agent done 18398.0 1548.0 11.9 1.0 2.0 1.0 1.0 2.0 7.0 Vision off; 11 browser calls, 26 min. The camera aspect is never updated, so the scene is stretched 1.6 times sideways (A 1); the spire is cut off at the top (C 1); the pond is out of frame (D 1). +t8-pagoda-abl15-r1 lcpp6-flashnext-abliterated-s15 agent done 29039.0 3038.0 9.6 1.0 2.0 1.0 2.0 2.0 8.0 Vision off; 21 browser calls, 1 screenshot, 51 min. Wall cubes overlap the wood ring and z-fight (A 1); the camera cuts the spire off (C 1); platform, four cherry trees, pond and torii all visible. +t8-pagoda-orcarouter-r1 lcpp6-orcarouter-fn-uncensored agent done 47168.0 3787.8 12.4 2.0 2.0 1.0 2.0 2.0 9.0 Vision off; 27 browser calls, 2 screenshots, 63 min. Clean render; the gold spire runs out of the top of the frame (C 1); platform, five cherry trees, round pond, torii, two lanterns. +t8-pagoda-tiel-r1 lcpp6-tiel-coder-35b-t06 agent done 66695.0 668.5 99.8 1.0 1.0 2.0 1.0 1.0 6.0 Vision off; 35 browser calls, 2 screenshots, 11 min. Every tree trunk is placed near the origin, so the canopies float and stubs poke out of the pagoda (A 1, D 1); the ground is a plane (B 1); no petal is ever visible (E 1). +t8-pagoda-gyro-s-mtp-r2 strata020-gyro-s-mtp-test agent done 82834.0 969.8 85.41 2.0 1.0 2.0 2.0 2.0 9.0 Vision off (the entry has no vision path); 54 browser calls, 0 screenshots, 0 compactions, 16 min. All frames render a complete scene; most parts are cubes but the roofs use half-height slabs (B 1); five storeys with upturned eaves and spire, stone platform, cherry trees, pond, torii; camera moves and petals fall. diff --git a/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-10000.webp new file mode 100644 index 0000000..69b2d42 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-1500.webp new file mode 100644 index 0000000..c0c4e38 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-5000.webp new file mode 100644 index 0000000..1de3312 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-abl15-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/page.html new file mode 100644 index 0000000..79dcd15 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-abl15-r1/page.html @@ -0,0 +1,218 @@ + + + + +Voxel Pagoda — Japanese Garden + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-10000.webp new file mode 100644 index 0000000..ebf1241 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-1500.webp new file mode 100644 index 0000000..3358974 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-5000.webp new file mode 100644 index 0000000..0511954 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-coder390-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/page.html new file mode 100644 index 0000000..cafa1bb --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-coder390-r1/page.html @@ -0,0 +1,277 @@ + + + + + +Voxel Pagoda — 五重の塔 + + + + +
五重の塔VOXEL PAGODA GARDEN
+ + + diff --git a/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-10000.webp new file mode 100644 index 0000000..2bbef72 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-1500.webp new file mode 100644 index 0000000..8f4ff52 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-5000.webp new file mode 100644 index 0000000..ad3491f Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/page.html new file mode 100644 index 0000000..8be34c4 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-cybertiel-r1/page.html @@ -0,0 +1,432 @@ + + + + + +Voxel Pagoda + + + +
+ + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-10000.webp new file mode 100644 index 0000000..17db9d8 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-1500.webp new file mode 100644 index 0000000..4397461 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-5000.webp new file mode 100644 index 0000000..0c67995 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/page.html b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/page.html new file mode 100644 index 0000000..249d323 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-gyro-s-mtp-r2/page.html @@ -0,0 +1,440 @@ + + + + + +Voxel Pagoda — Japanese Scene + + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-10000.webp new file mode 100644 index 0000000..d05c8ca Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-1500.webp new file mode 100644 index 0000000..85e32d1 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-5000.webp new file mode 100644 index 0000000..b40c63c Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/page.html new file mode 100644 index 0000000..80e048c --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-heretic2-r1/page.html @@ -0,0 +1,305 @@ + + + + + +Voxel Pagoda — voxel art scene + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-10000.webp new file mode 100644 index 0000000..79d4e4d Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-1500.webp new file mode 100644 index 0000000..69a8ad1 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-5000.webp new file mode 100644 index 0000000..a03cdba Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/page.html new file mode 100644 index 0000000..2be3cdf --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-orcarouter-r1/page.html @@ -0,0 +1,281 @@ + + + + +Voxel Pagoda + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-10000.webp new file mode 100644 index 0000000..c9714e9 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-1500.webp new file mode 100644 index 0000000..e2a90a3 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-5000.webp new file mode 100644 index 0000000..e1ea47f Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/page.html new file mode 100644 index 0000000..75f7045 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-orcasaq2-r1/page.html @@ -0,0 +1,352 @@ + + + + + +Voxel Pagoda Garden + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-10000.webp new file mode 100644 index 0000000..b0a52ae Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-1500.webp new file mode 100644 index 0000000..8cc520e Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-5000.webp new file mode 100644 index 0000000..507df95 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/page.html new file mode 100644 index 0000000..8258b84 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-q38cyber-r1/page.html @@ -0,0 +1,250 @@ + + + + + +voxel pagoda + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-10000.webp new file mode 100644 index 0000000..966ad01 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-1500.webp new file mode 100644 index 0000000..702194c Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-5000.webp new file mode 100644 index 0000000..4166968 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/page.html new file mode 100644 index 0000000..947b852 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-q38cyberq5-r1/page.html @@ -0,0 +1,294 @@ + + + + +Voxel Japanese Pagoda + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-10000.webp new file mode 100644 index 0000000..b3cb0b9 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-1500.webp new file mode 100644 index 0000000..7cba8d2 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-5000.webp new file mode 100644 index 0000000..a062cca Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/page.html new file mode 100644 index 0000000..50e4292 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-thinkcap-r1/page.html @@ -0,0 +1,258 @@ + + + + +Voxel Pagoda Garden + + + + + + + diff --git a/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-10000.webp new file mode 100644 index 0000000..513df3f Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-1500.webp new file mode 100644 index 0000000..5b65473 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-5000.webp new file mode 100644 index 0000000..e375fd8 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-tiel-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/page.html new file mode 100644 index 0000000..8c14bf3 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-tiel-r1/page.html @@ -0,0 +1,350 @@ + + + + + +Voxel Japanese Pagoda + + + + +
+ + + diff --git a/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-10000.webp b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-10000.webp new file mode 100644 index 0000000..df1aca3 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-10000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-1500.webp b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-1500.webp new file mode 100644 index 0000000..2725039 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-1500.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-5000.webp b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-5000.webp new file mode 100644 index 0000000..ead4110 Binary files /dev/null and b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/frames/frame-5000.webp differ diff --git a/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/page.html b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/page.html new file mode 100644 index 0000000..a9cc9e7 --- /dev/null +++ b/oneshots/agent-pagoda/t8-pagoda-uncvllm-r1/page.html @@ -0,0 +1,401 @@ + + + + + +Voxel Japanese Pagoda + + + + + + + + diff --git a/site/assets/app.js b/site/assets/app.js index f270016..af3b0fe 100644 --- a/site/assets/app.js +++ b/site/assets/app.js @@ -1,9 +1,10 @@ // The Bake-off page. Fetches data/bakeoff.json and renders every section from it. Each section // renders inside its own try/catch so one odd record never blanks the page; a section that throws // is named in data-render-error on (site/tools/render_smoke.sh looks for it). +// Prose is kept to titles, labels and numbers: the charts carry the story, long text sits behind disclosures. import * as D from './data.js'; -import { el, svg, clear, $, familyChip, verdictChip, kindChip, revealList, repoLink, onReducedMotion } from './dom.js'; -import { speedLadder, struggleStrip, abDumbbell, petalScore, pips, beforeAfter, bigNumeral, runLine, ofMax } from './charts.js'; +import { el, clear, $, familyChip, verdictChip, kindChip, revealList, repoLink, onReducedMotion } from './dom.js'; +import { speedLadder, struggleStrip, abDumbbell, pairDumbbell, petalScore, pips, beforeAfter, bigNumeral, runLine, ofMax } from './charts.js'; import { renderHero } from './hero.js'; import { chapterMotif } from './motifs.js'; import { openViewer } from './viewer.js'; @@ -34,16 +35,17 @@ async function main() { section('oneshots', () => renderOneshots(data)); section('findings', () => renderFindings(data)); section('timeline', () => renderTimeline(data)); - section('log', () => renderLog(data)); - section('caveats', () => renderCaveats(data)); - section('links', () => renderLinks(data)); + section('caveats', () => renderFoot(data)); section('motifs', () => renderMotifs(data)); document.body.dataset.rendered = '1'; // a deep link (#oneshots) was resolved against the empty page; re-aim it now that the chapters have their height if (location.hash && /^#[A-Za-z0-9_-]+$/.test(location.hash)) { const target = $(location.hash.slice(1)); + if (target && target.tagName === 'DETAILS') target.open = true; if (target) target.scrollIntoView({ behavior: onReducedMotion() ? 'auto' : 'instant', block: 'start' }); } + // the caveats live in a closed disclosure in the footer; the nav link opens it + addEventListener('hashchange', () => { if (location.hash === '#caveats') { const d = $('caveats'); if (d) d.open = true; } }); } function section(id, fn) { try { fn(); } catch (e) { @@ -55,6 +57,9 @@ function section(id, fn) { } } const stat = (k, v, unit) => el('div', {}, el('dt', {}, k), el('dd', {}, fmt(v, 0), unit ? el('small', {}, unit) : null)); +// A small closed disclosure for text that should not be read unless asked for. +const disclosure = (label, cls, ...children) => (children.some(Boolean) ? el('details', { class: `fold ${cls || ''}` }, el('summary', {}, label), ...children) : null); +const plural = (n, one, many) => (n === 1 ? `1 ${one}` : `${fmt(n, 0)} ${many || one + 's'}`); // ------------------------------------------------------------------- hero function renderTop(data) { @@ -62,19 +67,24 @@ function renderTop(data) { $('tagline').textContent = data.tagline; const s = D.summary(data); const stats = $('hero-stats'); clear(stats); - stats.append(stat('models', s.models), stat('families', s.families), stat('agent runs', s.runs), stat('passes', s.passes), stat('one-shots', s.oneshots), stat('frames', s.frames)); + stats.append(stat('models', s.models), stat('families', s.families), stat('agent runs', s.runs), stat('passes', s.passes), stat('one-shots', s.oneshots)); + if (s.agentPieces > 0) stats.append(stat('agent pieces', s.agentPieces)); if (s.playable > 0) stats.append(stat('live pieces', s.playable)); if (s.fastest != null) stats.append(stat('fastest run', s.fastest, 'tok/s')); if (s.agentHours > 0) stats.append(stat('agent hours', Math.round(s.agentHours))); - // freshness: the run table is a snapshot; findings are written as they happen and are usually newer + // freshness, derived: the span of everything dated in the file, and when the file was built const gen = $('generated'); clear(gen); const bit = (k, v) => el('span', { class: 'fresh' }, el('small', {}, k), ' ', v); - if (data.generated) gen.appendChild(bit('data generated', data.generated)); - if (data.built) gen.appendChild(bit('site built', data.built)); - if (s.last) gen.appendChild(bit('newest run', fmtDate(s.last))); - if (s.newestFinding) gen.appendChild(bit('newest finding', fmtDate(s.newestFinding))); + if (s.covers.from && s.covers.to) gen.appendChild(bit('covers', s.covers.from === s.covers.to ? fmtDate(s.covers.to) : `${fmtDate(s.covers.from)} – ${fmtDate(s.covers.to)}`)); + if (data.built) gen.appendChild(bit('built', data.built)); + else if (data.generated) gen.appendChild(bit('generated', data.generated)); if (!gen.childNodes.length) gen.textContent = 'Data snapshot'; + // the key to the orrery: four marks, four words each + const key = $('hero-key'); clear(key); + const mark = (cls) => el('i', { class: `mark ${cls}`, 'aria-hidden': 'true' }); + key.append(el('li', {}, mark('speed'), 'speed · best tok/s'), el('li', {}, mark('size'), 'size · runs'), el('li', {}, mark('halo'), 'halo · passed test 2'), el('li', {}, mark('unc'), 'dashed · uncensored')); renderHero(data, $('orrery'), $('hero-caption'), $('hero-legend')); + $('hero-caption').textContent = ''; } // ---------------------------------------------------------------- machine @@ -103,12 +113,12 @@ function renderTests(data) { } function testCard(t, n, undocumented = false) { const kindCls = (t.kind || '').replace(/[^a-z-]/gi, '').toLowerCase(); - return el('article', { class: `test-card ${kindCls}` }, + return el('article', { class: `test-card ${kindCls}${n ? '' : ' pending'}` }, el('span', { class: 'tn', 'aria-hidden': 'true' }, t.number || '?'), el('h3', {}, el('span', { class: 'sr' }, `Test ${t.number}: `), t.name), - t.what ? el('p', { class: 'what' }, t.what) : undocumented ? el('p', { class: 'what empty' }, 'Appears in the results but is not described yet.') : null, - t.graded ? el('p', { class: 'graded' }, t.graded) : null, - el('div', { class: 'meta' }, t.kind ? kindChip(t.kind, kindCls) : null, el('span', { class: 'n' }, n === 1 ? '1 graded run' : `${fmt(n, 0)} graded runs`))); + t.what ? el('p', { class: 'what' }, t.what) : undocumented ? el('p', { class: 'what empty' }, 'Not described yet.') : null, + el('div', { class: 'meta' }, t.kind ? kindChip(t.kind, kindCls) : null, el('span', { class: 'n' }, n ? plural(n, 'graded run') : 'no graded runs yet')), + disclosure('grading', 'small', t.graded ? el('p', {}, t.graded) : null)); } // ------------------------------------------------------------------ picks @@ -121,9 +131,9 @@ function renderPicks(data) { if (!rows.length) fam.appendChild(el('p', { class: 'empty' }, 'No family table yet.')); for (const f of rows) { fam.appendChild(el('div', { class: 'fam-row', style: { '--c': f.color } }, - el('div', { class: 'famhead' }, familyChip(f.family, f.color), el('div', { class: 'count' }, f.models === 1 ? '1 model tried' : `${f.models} models tried`)), - ...f.cells.map(([k, v]) => el('div', { class: 'cell' }, el('small', {}, k), v ? el('b', {}, v) : el('span', { class: 'none' }, 'none yet'))), - f.note ? el('p', { class: 'note' }, f.note) : null)); + el('div', { class: 'famhead' }, familyChip(f.family, f.color), el('div', { class: 'count' }, plural(f.models, 'model'))), + ...f.cells.map(([k, v]) => el('div', { class: 'cell' }, el('small', {}, k), v ? el('b', {}, v) : el('span', { class: 'none' }, '–'))), + disclosure('note', 'small', f.note ? el('p', {}, f.note) : null))); } } function fill(host, items, render, emptyText) { @@ -136,9 +146,9 @@ function pickCard(p) { el('div', { class: 'role' }, p.role), el('h4', {}, p.short || p.model || 'unnamed'), el('div', { class: 'meta' }, familyChip(p.family, p.color), p.uncensored ? kindChip('uncensored') : null, p.quant ? el('span', { class: 'mono' }, p.quant) : null, p.engine ? el('span', { class: 'mono' }, p.engine) : null, p.missing ? kindChip('no model card in data') : null), - p.verdict ? el('p', { class: 'verdict-text' }, p.verdict) : null, - p.tradeoff ? el('p', { class: 'trade' }, p.tradeoff) : null, - p.runs.length ? el('ul', {}, ...p.runs.map(runLine)) : null); + p.verdict ? el('p', { class: 'verdict-text' }, D.firstSentence(p.verdict)) : null, + p.runs.length ? el('ul', {}, ...p.runs.map(runLine)) : null, + disclosure('trade-off', 'small', p.tradeoff ? el('p', {}, p.tradeoff) : null)); } // ------------------------------------------------------------------- runs @@ -157,7 +167,7 @@ function renderRuns(data) { const host = $('ladder'); clear(host); host.appendChild(speedLadder(rows, { limit })); const more = $('ladder-more'); more.hidden = rows.length <= limit; - more.textContent = `Show ${Math.min(40, rows.length - limit)} more rows (${rows.length - limit} left)`; + more.textContent = `Show ${Math.min(40, rows.length - limit)} more (${rows.length - limit} left)`; const tbl = $('run-table'); clear(tbl); tbl.appendChild(runTable(rows)); }; form.addEventListener('input', () => { limit = 40; draw(); }); @@ -195,51 +205,60 @@ function renderStruggle(data) { if (!rows.length) { host.appendChild(el('p', { class: 'empty' }, 'No test-2 runs with a struggle score yet.')); return; } const draw = () => { clear(host); host.appendChild(struggleStrip(rows, host.clientWidth - 16)); }; draw(); - let t; addEventListener('resize', () => { clearTimeout(t); t = setTimeout(draw, 150); }); + onResize(draw); const sum = D.struggleSummary(rows); const row = $('struggle-summary'); clear(row); for (const v of ['pass', 'partial', 'fail']) { if (!sum[v]) continue; - row.appendChild(el('div', { class: 'stat' }, el('div', { class: 'k' }, verdictChip(v), `${sum[v].n} runs`), el('div', { class: 'v' }, fmt(sum[v].median, 1), el('small', {}, ' median errors / kloc')))); + row.appendChild(el('div', { class: 'stat' }, el('div', { class: 'k' }, verdictChip(v), plural(sum[v].n, 'run')), el('div', { class: 'v' }, fmt(sum[v].median, 1), el('small', {}, ' median err / kloc')))); } const c = D.cleanestRun(rows); - if (c) row.appendChild(el('div', { class: 'stat' }, el('div', { class: 'k' }, 'cleanest passing build'), - el('div', { class: 'v' }, c.short, el('small', {}, ` ${fmt(c.struggle.errors_per_kloc, 1)} errors / kloc · ${fmt(c.struggle.compiles, 0)} build${c.struggle.compiles === 1 ? '' : 's'} · ${fmt(c.struggle.loc, 0)} lines`)), - el('div', { class: 'k' }, verdictChip(c.verdict)))); + if (c) row.appendChild(el('div', { class: 'stat wide' }, el('div', { class: 'k' }, 'cleanest pass'), + el('div', { class: 'v' }, c.short, el('small', {}, ` ${fmt(c.struggle.errors_per_kloc, 1)} err / kloc · ${plural(c.struggle.compiles, 'build')} · ${fmt(c.struggle.loc, 0)} lines`)))); +} +let resizeFns = []; +function onResize(fn) { + resizeFns.push(fn); + if (resizeFns.length === 1) { let t; addEventListener('resize', () => { clearTimeout(t); t = setTimeout(() => resizeFns.forEach((f) => f()), 150); }); } } // ----------------------------------------------------------------- test 3 function renderTest3(data) { - const { groups, rubricMax } = D.test3Groups(data); + const { groups, flavors, rubricMax } = D.test3Groups(data); + const lede = $('test3-lede'); + const t = D.testOf(data, '3'); + // the flavours, as the data names them, make the one line under the title + lede.textContent = flavors.length ? flavors.map((f) => `${f}: ${/G$/i.test(f) ? 'known-good server' : /O$/i.test(f) ? "the model's own server" : 'flavour'}`).join(' · ') : (t.what || ''); const host = $('test3-groups'); - const btn = $('test3-more'); - const r = revealList(host, groups, (g) => el('article', { class: 't3', style: { '--c': g.color } }, + const head = el('div', { class: 't3-head', 'aria-hidden': 'true' }, el('span', {}), el('span', {}, 'MCP facts'), el('span', {}, 'API facts'), el('span', {}, 'rubric'), el('span', {}, 'judged'), el('span', { class: 'num' }, 'speed · time')); + revealList(host, groups, (g) => el('article', { class: 't3', style: { '--c': g.color } }, el('div', { class: 'head' }, el('b', {}, g.short), el('div', { class: 'chips' }, familyChip(g.family, g.color), g.uncensored ? kindChip('uncensored') : null)), - el('div', { class: 'flavors' }, ...g.rows.map((row) => el('div', { class: 't3-row' }, - el('span', { class: 'flavor', title: `flavor ${row.flavor}` }, row.flavor), - el('div', { class: 'facts' }, - el('span', {}, 'MCP facts ', pips(row.mcp_facts.got, row.mcp_facts.of), ` ${row.mcp_facts.text}`), - el('span', {}, 'API facts ', pips(row.api_facts.got, row.api_facts.of, 'api'), ` ${row.api_facts.text}`), - el('span', {}, 'rubric ', pips(row.rubric, rubricMax, 'rubric'), el('span', { class: 'rubric-num' }, ` ${ofMax(row.rubric, rubricMax)}`)), - el('span', { class: `judge ${row.judgement_right === true ? 'ok' : row.judgement_right === false ? 'bad' : ''}` }, row.judgement_right === true ? '✓ judged the server right' : row.judgement_right === false ? '✗ misjudged the server' : 'judgement not recorded')), - el('span', { class: 'num' }, `${fmt(row.tps, 1)} tok/s · ${fmtMinutes(row.minutes)}`), - ...noteBlock(row.note, 220))))), { step: 8, label: 'Show more models' }); - btn.hidden = true; // revealList carries its own button - return r; + el('div', { class: 'flavors' }, ...g.rows.map((row) => { + const [note, noteBtn] = noteBlock(row.note); + const judge = row.judgement_right === true ? 'ok' : row.judgement_right === false ? 'bad' : ''; + return el('div', { class: 't3-row' }, + el('span', { class: 'flavor', title: `flavor ${row.flavor}` }, row.flavor), + el('span', { class: 'cell' }, pips(row.mcp_facts.got, row.mcp_facts.of), el('small', {}, row.mcp_facts.text)), + el('span', { class: 'cell' }, pips(row.api_facts.got, row.api_facts.of, 'api'), el('small', {}, row.api_facts.text)), + el('span', { class: 'cell' }, pips(row.rubric, rubricMax, 'rubric'), el('small', { class: 'rubric-num' }, ofMax(row.rubric, rubricMax))), + el('span', { class: `judge ${judge}`, role: 'img', 'aria-label': judge === 'ok' ? 'judged the server right' : judge === 'bad' ? 'misjudged the server' : 'judgement not recorded', title: judge === 'ok' ? 'judged the server right' : judge === 'bad' ? 'misjudged the server' : 'judgement not recorded' }, judge === 'ok' ? '✓' : judge === 'bad' ? '✗' : '·'), + el('span', { class: 'num' }, `${fmt(row.tps, 1)} tok/s · ${fmtMinutes(row.minutes)}`, noteBtn), + note); + }))), { step: 8, label: 'Show more models' }); + if (groups.length) host.insertBefore(head, host.firstChild); } // -------------------------------------------------------------------- A/B function renderAB(data) { const { groups, engines, colors, max } = D.abGroups(data); const legend = $('ab-legend'); clear(legend); - for (const e of engines) legend.appendChild(el('li', {}, el('i', { style: { '--c': colors.get(e) }, 'aria-hidden': 'true' }), `engine ${e}`)); - if (engines.length) legend.appendChild(el('li', {}, 'dot = one probe variant; the line spans the model’s range; ×n is the gain between engines with the same flags')); + for (const e of engines) legend.appendChild(el('li', {}, el('i', { style: { '--c': colors.get(e) }, 'aria-hidden': 'true' }), e)); const host = $('ab-groups'); clear(host); if (!groups.length) { host.appendChild(el('p', { class: 'empty' }, 'No engine probes yet.')); return; } for (const g of groups.filter((g) => !g.allFailed)) host.appendChild(abDumbbell(g, max, colors)); const failed = data.ab.filter((r) => r.decode == null); if (failed.length) { - host.appendChild(el('details', { class: 'ab-errors' }, el('summary', {}, `${failed.length} probe${failed.length === 1 ? '' : 's'} failed to load`), + host.appendChild(el('details', { class: 'ab-errors fold' }, el('summary', {}, `${plural(failed.length, 'probe')} failed to load`), el('ul', {}, ...failed.map((r) => el('li', {}, el('b', {}, D.shortLabel(D.modelOf(data, r.model).label)), ` · ${r.engine} · ${r.variant}: `, el('code', {}, r.error.slice(0, 160) || 'no error text')))))); } } @@ -248,24 +267,16 @@ function renderAB(data) { function renderOneshots(data) { const sets = D.oneshotSets(data); const setRow = $('oneshot-sets'); clear(setRow); + const n = (v, k) => el('span', { class: 'sn' }, el('b', {}, fmt(v, v != null && v < 10 && v % 1 ? 1 : 0)), ' ', k); for (const s of sets) { - setRow.appendChild(el('div', { class: 'set-card' }, el('b', {}, `Test ${s.test} · ${s.name}`), - el('span', {}, `${s.n} replies · ${s.withFrames} rendered · ${s.full} full marks · ${s.zero} scored zero`), - el('span', {}, `${s.cut} cut off by the token cap${s.mean != null ? ` · mean ${fmt(s.mean, 1)}` : ''}`), - s.live > 0 ? el('span', { class: 'live-n' }, el('i', { 'aria-hidden': 'true' }, '▶'), ` ${s.live} can be played live`) : null)); - } - // header line: how many pieces can be played, from the data - const liveLine = $('oneshot-live'); - const playable = sets.reduce((n, s) => n + (s.live || 0), 0), total = sets.reduce((n, s) => n + s.n, 0); - if (liveLine) { - clear(liveLine); - liveLine.hidden = playable <= 0; - if (playable > 0) liveLine.append(el('i', { 'aria-hidden': 'true' }, '▶'), ` ${fmt(playable, 0)} of the ${fmt(total, 0)} replies produced a file you can run here, exactly as the model wrote it: `, - sets.filter((s) => s.live > 0).map((s) => `${fmt(s.live, 0)} ${plural(s.set, s.live)}`).join(' and '), - '. Press “Play live” on a card; each piece runs alone, in a sandboxed frame.'); + setRow.appendChild(el('div', { class: `set-card ${(s.kind || '').replace(/[^a-z-]/gi, '')}` }, + el('div', { class: 'set-head' }, el('b', {}, s.label), kindChip(`test ${s.test}`)), + el('div', { class: 'set-nums' }, n(s.n, 'pieces'), s.live > 0 ? el('span', { class: 'sn live-n' }, el('i', { 'aria-hidden': 'true' }, '▶'), el('b', {}, fmt(s.live, 0)), ' live') : null, + s.mean != null ? n(s.mean, `mean of ${fmt(s.max, 0)}`) : null, n(s.full, 'full'), n(s.zero, 'zero'), s.cut ? n(s.cut, 'cut') : null))); } + renderPairs(data); const tsel = $('o-test'); clear(tsel); tsel.appendChild(el('option', { value: 'all' }, 'all')); - for (const s of sets) tsel.appendChild(el('option', { value: s.test }, `${s.test} · ${s.name}`)); + for (const s of sets) tsel.appendChild(el('option', { value: s.set }, `${s.label} · test ${s.test}`)); const form = $('oneshot-filters'); const grid = $('oneshot-grid'); let current = []; @@ -273,8 +284,8 @@ function renderOneshots(data) { const cards = D.oneshotCards(data, { test: tsel.value, sort: $('o-sort').value, q: $('o-q').value.trim() }); current = cards; const live = cards.filter((c) => c.live).length; - $('o-count').textContent = `${cards.length} replies${live ? ` · ${live} playable` : ''}`; - revealList(grid, cards, (c, i) => shotCard(c, () => openViewerAt(current, i)), { step: 12, label: 'Show more replies' }); + $('o-count').textContent = `${cards.length} pieces${live ? ` · ${live} live` : ''}`; + revealList(grid, cards, (c, i) => shotCard(c, () => openViewerAt(current, i)), { step: 12, label: 'Show more' }); }; const openViewerAt = (cards, i) => { const btn = grid.querySelector(`.shot[data-case="${CSS.escape(cards[i].case)}"] .play`); openViewer(cards, i, btn); }; form.addEventListener('input', draw); @@ -282,18 +293,41 @@ function renderOneshots(data) { draw(); startFlipbooks(grid); } -const plural = (w, n) => (n === 1 ? w : /y$/.test(w) ? `${w.slice(0, -1)}ies` : `${w}s`); -const blankCard = (c) => el('div', { class: 'blank' }, el('span', {}, el('i', { 'aria-hidden': 'true' }, '∅'), c.finish === 'length' ? 'Cut off by the token cap, nothing to render' : c.frames.length ? 'The renders could not be loaded' : 'No render available')); +// One shot against agent: one paired chart per agent set, drawn from the same rows as the gallery. +function renderPairs(data) { + const panels = D.agentPairs(data); + const block = $('agent'), host = $('pairs'), lede = $('pairs-lede'); + clear(host); + block.hidden = !panels.length; + if (!panels.length) return; + lede.textContent = 'Same model, same rubric; best of n one-shots.'; + const draw = () => { + clear(host); + const figs = panels.map((p) => { + const fig = el('figure', { class: 'pair-fig' }); + fig.append(el('figcaption', {}, el('b', {}, p.baseLabel), el('span', { class: 'arrow' }, ' → '), el('b', {}, p.label), + el('span', { class: 'pair-stat' }, `mean ${p.meanOneshot == null ? '–' : fmt(p.meanOneshot)} → ${p.meanAgent == null ? '–' : fmt(p.meanAgent)} · ${p.lifted} of ${p.paired} up`))); + host.appendChild(fig); + return fig; + }); + // every figure is in the grid now, so each chart takes the width its figure really has (one or two columns) + panels.forEach((p, i) => figs[i].appendChild(pairDumbbell(p, Math.max(280, figs[i].clientWidth - 22)))); + }; + draw(); + onResize(draw); +} +const blankCard = (c) => el('div', { class: 'blank' }, el('span', {}, el('i', { 'aria-hidden': 'true' }, '∅'), /length|cap/i.test(c.finish) ? 'cut off, nothing to render' : c.frames.length ? 'renders could not be loaded' : 'no render')); function shotCard(c, onPlay) { const pic = el('div', { class: 'pic' }); const frames = c.frames; + const square = /orrery/i.test(c.set); if (frames.length) { const last = frames.length - 1; let failed = 0; // The last frame is the richest, so it is shown first and fetched eagerly; the others load lazily and the // flipbook only ever advances onto a frame that has really loaded. If none loads the card goes blank. const imgs = frames.map((f, i) => { - const im = el('img', { src: f.url, alt: `${c.alt}, frame at ${f.ms == null ? '?' : fmt(f.ms, 0)} ms`, loading: i === last ? 'eager' : 'lazy', decoding: 'async', width: c.set === 'orrery' ? 1000 : 1280, height: c.set === 'orrery' ? 1000 : 800, class: i === last ? 'on' : '' }); + const im = el('img', { src: f.url, alt: `${c.alt}, frame at ${f.ms == null ? '?' : fmt(f.ms, 0)} ms`, loading: i === last ? 'eager' : 'lazy', decoding: 'async', width: square ? 1000 : 1280, height: square ? 1000 : 800, class: i === last ? 'on' : '' }); im.addEventListener('load', () => { if (im.classList.contains('on')) pic.dataset.loaded = '1'; }); im.addEventListener('error', () => { im.classList.add('broken'); if (++failed === frames.length) { clear(pic); delete pic.dataset.flip; pic.appendChild(blankCard(c)); } }); return im; @@ -318,23 +352,19 @@ function shotCard(c, onPlay) { pic.appendChild(play); if (c.full) pic.appendChild(el('span', { class: 'ribbon', 'aria-hidden': 'true' }, 'live')); } - const [note, noteBtn] = noteBlock(c.note, 160); + const [note, noteBtn] = noteBlock(c.note); const body = el('div', { class: 'body' }, - el('div', {}, el('b', {}, c.short), el('div', { class: 'meta' }, familyChip(c.family, c.color), c.uncensored ? kindChip('uncensored') : null, - el('span', {}, `${fmt(c.tok_s, 1)} tok/s`), el('span', {}, `${fmt(c.tokens, 0)} tokens`), c.finish === 'length' ? el('span', { class: 'cut' }, 'hit the cap') : null)), + el('div', {}, el('b', {}, c.short), el('div', { class: 'meta' }, familyChip(c.family, c.color), kindChip(c.setLabel, c.kind === 'agentic' ? 'agentic' : ''), c.uncensored ? kindChip('uncensored') : null, + el('span', {}, `${fmt(c.tok_s, 1)} tok/s`), /length|cap/i.test(c.finish) ? el('span', { class: 'cut' }, c.finish === 'length' ? 'hit the cap' : c.finish) : null, noteBtn)), petalScore(c), note); - const pmax = c.pointMax > 0 ? c.pointMax : 1; - if (c.points.length) body.appendChild(el('div', { class: 'pts' }, ...c.points.map((p) => { const v = p.value || 0; return el('span', { class: v <= 0 ? 'l0' : v >= pmax ? 'l2' : 'l1' }, `${p.name} ${fmt(p.value)}`); }))); - if (noteBtn) body.appendChild(noteBtn); return el('article', { class: `shot${c.full && frames.length ? ' featured' : ''}${c.live ? ' playable' : ''}`, title: c.case, 'data-case': c.case, style: { '--c': c.color } }, pic, body); } -// A clamped note plus a "full note" toggle when it is long. Returns [p, button|null]. -function noteBlock(text, threshold) { +// A grader's or operator's note, folded away behind a "note" toggle. Returns [p, button] or [null, null]. +function noteBlock(text) { if (!text) return [null, null]; - const p = el('p', { class: 'note' }, text); - if (text.length <= threshold) return [p, null]; - const b = el('button', { type: 'button', class: 'noteb', 'aria-expanded': 'false' }, 'full note'); - b.addEventListener('click', () => { const open = p.classList.toggle('open'); b.setAttribute('aria-expanded', String(open)); b.textContent = open ? 'less' : 'full note'; }); + const p = el('p', { class: 'note', hidden: true }, text); + const b = el('button', { type: 'button', class: 'noteb', 'aria-expanded': 'false' }, 'note'); + b.addEventListener('click', () => { p.hidden = !p.hidden; b.setAttribute('aria-expanded', String(!p.hidden)); b.classList.toggle('open', !p.hidden); }); return [p, b]; } const imgLoaded = (im) => im.complete && im.naturalWidth > 0; @@ -367,68 +397,61 @@ function startFlipbooks(grid) { } // --------------------------------------------------------------- findings +// A card leads with its numbers: the first metric as big numerals, the rest as before/after bars. The summary is +// its first sentence; the full text, settings, quality check and caveat fold away. function renderFindings(data) { - const host = $('findings-list'); clear(host); - if (!data.findings.length) { host.appendChild(el('p', { class: 'empty' }, 'No headline findings yet.')); return; } - for (const f of data.findings) { + const host = $('findings-list'); + revealList(host, data.findings, (f) => { const metrics = D.findingMetrics(f); const right = el('div', { class: 'numbers' }); if (metrics.length) { - // the lead metric is the typographic moment; the rest are bars under it const lead = metrics.find((m) => m.after != null) || metrics[0]; right.appendChild(bigNumeral(lead)); - const rest = metrics.filter((m) => m !== lead); - if (rest.length) { - right.appendChild(el('div', { class: 'ba-key' }, el('span', {}, el('i', { class: 'before' }), 'before'), el('span', {}, el('i', { class: 'after' }), 'after'))); - for (const m of rest) right.appendChild(beforeAfter(m)); - } + for (const m of metrics.filter((m) => m !== lead)) right.appendChild(beforeAfter(m)); } - if (f.quality) right.appendChild(el('p', { class: 'quality', 'data-k': 'quality check' }, f.quality)); - if (f.caveat) right.appendChild(el('p', { class: 'caveat', 'data-k': 'caveat' }, f.caveat)); - host.appendChild(el('article', { class: 'finding' }, + const line = D.firstSentence(f.summary); + const more = disclosure('details', '', + f.summary && f.summary !== line ? el('p', {}, f.summary) : null, + f.settings ? el('p', { class: 'settings' }, 'settings ', el('code', {}, f.settings)) : null, + f.quality ? el('p', { class: 'quality', 'data-k': 'quality check' }, f.quality) : null, + f.caveat ? el('p', { class: 'caveat', 'data-k': 'caveat' }, f.caveat) : null); + return el('article', { class: 'finding' }, el('div', { class: 'when' }, f.date ? el('time', { datetime: f.date }, fmtDate(f.date)) : null, kindChip(f.kind)), el('h3', {}, f.title), - el('div', {}, el('p', { class: 'summary' }, f.summary), f.settings ? el('p', { class: 'settings' }, 'settings ', el('code', {}, f.settings)) : null), - right)); - } + el('div', { class: 'text' }, line ? el('p', { class: 'summary' }, line) : null, more), + right); + }, { step: 8, label: 'Show more findings' }); + if (!data.findings.length) { clear(host); host.appendChild(el('p', { class: 'empty' }, 'No headline findings yet.')); } } // --------------------------------------------------------------- timeline +// Date, title, the what line; a month mark where the month turns; why on hover, focus or the toggle. function renderTimeline(data) { const ol = $('timeline-list'); clear(ol); if (!data.timeline.length) { ol.appendChild(el('li', { class: 'empty' }, 'No timeline yet.')); return; } + let month = ''; for (const t of data.timeline) { - ol.appendChild(el('li', {}, - el('div', { class: 'date' }, t.date ? el('time', { datetime: t.date }, fmtDate(t.date)) : null, t.era ? el('span', { class: 'era' }, t.era) : null), - el('h3', {}, t.title), t.what ? el('p', { class: 'what' }, t.what) : null, t.why ? el('p', { class: 'why' }, t.why) : null)); + const m = t.date.slice(0, 7); + if (m && m !== month) { month = m; ol.appendChild(el('li', { class: 'month', 'aria-hidden': 'true' }, el('span', {}, monthName(m)))); } + const why = t.why ? el('p', { class: 'why' }, t.why) : null; + const btn = why ? el('button', { type: 'button', class: 'noteb why-b', 'aria-expanded': 'false' }, 'why') : null; + const li = el('li', {}, + el('div', { class: 'date' }, t.date ? el('time', { datetime: t.date }, dayLabel(t.date)) : null, t.era ? el('span', { class: 'era' }, t.era) : null), + el('h3', {}, t.title), t.what ? el('p', { class: 'what' }, t.what, ' ', btn) : btn, why); + if (btn) btn.addEventListener('click', () => { const open = li.classList.toggle('open'); btn.setAttribute('aria-expanded', String(open)); }); + ol.appendChild(li); } } +const MONTHS = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']; +const monthName = (ym) => { const m = ym.match(/^(\d{4})-(\d{2})/); return m ? `${MONTHS[+m[2] - 1] || m[2]} ${m[1]}` : ym; }; +const dayLabel = (d) => { const m = d.match(/^\d{4}-(\d{2})-(\d{2})/); return m ? `${(MONTHS[+m[1] - 1] || m[1]).slice(0, 3)} ${+m[2]}` : fmtDate(d); }; -// -------------------------------------------------------------------- log -function renderLog(data) { - const kinds = D.eventKinds(data); - const row = $('event-kinds'); clear(row); - let active = 'all'; - const host = $('events-list'); - const draw = () => { - const items = active === 'all' ? data.events : data.events.filter((e) => e.kind === active); - revealList(host, items, (e) => el('article', { class: 'event' }, - el('div', {}, el('time', {}, e.when), kindChip(e.kind)), el('p', {}, e.text)), { step: 12, label: 'Show more entries' }); - row.querySelectorAll('button').forEach((b) => b.setAttribute('aria-pressed', String(b.dataset.kind === active))); - }; - const mk = (kind, label) => { const b = el('button', { type: 'button', 'data-kind': kind, 'aria-pressed': 'false' }, label); b.addEventListener('click', () => { active = kind; draw(); }); return el('li', {}, b); }; - row.appendChild(mk('all', `all ${data.events.length}`)); - for (const k of kinds) row.appendChild(mk(k.kind, `${k.kind} ${k.n}`)); - draw(); -} - -// ---------------------------------------------------------------- caveats -function renderCaveats(data) { +// ------------------------------------------------------------------ footer +function renderFoot(data) { const ol = $('caveat-list'); clear(ol); + $('caveats-summary').textContent = data.caveats.length ? `${plural(data.caveats.length, 'caveat')}, read before quoting a number` : 'Caveats'; if (!data.caveats.length) ol.appendChild(el('li', {}, 'No caveats are recorded in the data. Treat every number as one sample from a changing harness.')); for (const c of data.caveats) ol.appendChild(el('li', {}, c)); -} -function renderLinks(data) { const ul = $('links'); clear(ul); const names = { repo: 'Repository', cards: 'Model cards', method: 'Method', conclusions: 'Conclusions', log: 'Research log' }; for (const [k, v] of Object.entries(data.links)) ul.appendChild(el('li', {}, repoLink(v, names[k] || k))); @@ -448,4 +471,3 @@ function renderMotifs(data) { } main(); -export { svg }; diff --git a/site/assets/charts.js b/site/assets/charts.js index a377657..d3d10b0 100644 --- a/site/assets/charts.js +++ b/site/assets/charts.js @@ -16,7 +16,9 @@ export function speedLadder(rows, { limit = 40 } = {}) { tick.style.left = `${scale.pos(t) * 100}%`; axis.appendChild(tick); } - wrap.appendChild(el('div', { class: 'ladder-head' }, el('span', {}, 'model · run'), axis, el('span', { class: 'num' }, 'tok/s'))); + // the verdict key rides the head, in the colours the bars use + const key = el('span', { class: 'vkey' }, ...['pass', 'partial', 'fail'].map((v) => el('span', { class: `v-${v}` }, el('span', { 'aria-hidden': 'true' }, VERDICT_GLYPH[v]), ` ${v}`))); + wrap.appendChild(el('div', { class: 'ladder-head' }, key, axis, el('span', { class: 'num' }, 'tok/s'))); const list = el('ol', { class: 'ladder-rows' }); rows.slice(0, limit).forEach((r) => { const pct = Math.max(0.6, scale.pos(r.tps) * 100); @@ -73,6 +75,46 @@ export function struggleStrip(rows, width) { return root; } +// One shot against agent: one row per model, a hollow mark for the best one-shot score and a filled mark (family colour) +// for the agent score on a 0..max axis, joined by a line that is green when the agent did better. Numbers sit at the end +// of every row, so the chart needs no legend beyond the two glyphs in its head. +export function pairDumbbell(panel, width) { + const W = Math.max(320, width || 560), left = Math.min(230, Math.max(120, Math.round(W * 0.38))), right = 72, rowH = 24, top = 30, bottom = 24; + const rows = panel.rows; + const H = top + (rows.length + 1) * rowH + bottom; + const x = (v) => left + (Math.max(0, Math.min(panel.max, v)) / panel.max) * (W - left - right); + const root = svg('svg', { class: 'pairs', viewBox: `0 0 ${W} ${H}`, width: W, height: H, role: 'img', + 'aria-label': `${panel.baseLabel} one-shot against ${panel.label} for ${rows.length} models, scores out of ${fmt(panel.max, 0)}: ` + rows.map((r) => `${r.short} ${r.oneshot == null ? 'no one-shot' : fmt(r.oneshot)} to ${r.agent == null ? 'not graded' : fmt(r.agent)}`).join('; ') }); + const ticks = [0, panel.max / 2, panel.max]; + for (const t of ticks) { + root.appendChild(svg('line', { x1: x(t), x2: x(t), y1: top - 4, y2: H - bottom + 4, class: 'grid' })); + root.appendChild(svg('text', { x: x(t), y: H - 6, class: 'axis', 'text-anchor': 'middle' }, fmt(t, 0))); + } + root.appendChild(svg('text', { x: left - 10, y: 14, class: 'axis dim', 'text-anchor': 'end' }, 'model')); + const head = svg('text', { x: W - 6, y: 14, class: 'axis dim', 'text-anchor': 'end' }); + head.append(svg('tspan', { class: 'one' }, '○'), svg('tspan', {}, ' one-shot → '), svg('tspan', { class: 'agent' }, '●'), svg('tspan', {}, ' agent')); + root.appendChild(head); + const draw = (r, i, cls = '') => { + const cy = top + i * rowH + rowH / 2; + const g = svg('g', { class: `pair ${cls}`, tabindex: 0 }); + g.appendChild(svg('title', {}, `${r.label}: one-shot ${r.oneshot == null ? 'none' : fmt(r.oneshot) + (r.oneshotN > 1 ? ` (best of ${r.oneshotN})` : '')} → agent ${r.agent == null ? 'not graded' : fmt(r.agent)}${r.agentN > 1 ? ` (best of ${r.agentN})` : ''}`)); + const maxChars = Math.max(12, Math.floor((left - 14) / 7.2)); + const name = r.short.length > maxChars ? r.short.slice(0, maxChars - 1).trimEnd() + '…' : r.short; + g.appendChild(svg('text', { x: left - 10, y: cy + 4, class: 'axis lbl', 'text-anchor': 'end' }, name)); + if (r.oneshot != null && r.agent != null) g.appendChild(svg('line', { x1: x(r.oneshot), x2: x(r.agent), y1: cy, y2: cy, class: `link ${r.agent > r.oneshot ? 'up' : r.agent < r.oneshot ? 'down' : 'flat'}` })); + if (r.oneshot != null) g.appendChild(svg('circle', { cx: x(r.oneshot), cy, r: 5, class: 'one' })); + if (r.agent != null) g.appendChild(svg('circle', { cx: x(r.agent), cy, r: 6, class: 'agent', style: { '--c': r.color } })); + g.appendChild(svg('text', { x: W - right + 8, y: cy + 4, class: 'axis num' }, `${r.oneshot == null ? '–' : fmt(r.oneshot)} → ${r.agent == null ? '–' : fmt(r.agent)}`)); + root.appendChild(g); + }; + rows.forEach((r, i) => draw(r, i)); + if (rows.length) { + root.appendChild(svg('line', { x1: 8, x2: W - 8, y1: top + rows.length * rowH + 1, y2: top + rows.length * rowH + 1, class: 'lane' })); + draw({ label: `mean of ${rows.length}`, short: 'mean', oneshot: panel.meanOneshot, agent: panel.meanAgent, color: '#f2c14e', oneshotN: 1, agentN: 1 }, rows.length, 'mean'); + } + return root; +} + // Dumbbell for the A/B speed probes: decode tok/s per engine, same model on one line. Colours come per engine value. export function abDumbbell(group, max, colors) { const W = 100; diff --git a/site/assets/data.js b/site/assets/data.js index cdd47b2..2e5942b 100644 --- a/site/assets/data.js +++ b/site/assets/data.js @@ -5,9 +5,11 @@ // Frame paths: relative, under frames/, every segment starts with a letter/digit/_/- (so "." and ".." segments and // dot-files are impossible), no "//", and the lookahead rejects ".." anywhere at all. export const FRAME_RE = /^(?!.*\.\.)frames\/(?:[A-Za-z0-9_-][A-Za-z0-9._-]*\/)*[A-Za-z0-9_-][A-Za-z0-9._-]*\.webp$/; -// Live pieces (the model's own one-shot file, run only inside a sandboxed iframe): exactly live/// -// with the two known sets and file names, one case segment of safe characters, and never ".." anywhere. -export const LIVE_RE = /^live\/(pagoda|orrery)\/[A-Za-z0-9._-]+\/(page\.html|orrery\.svg)$/; +// Live pieces (the model's own file, run only inside a sandboxed iframe): exactly live/// with the +// four known sets, each paired with its one file name (a pagoda set serves page.html, an orrery set orrery.svg), one +// case segment of safe characters, and never ".." anywhere. A new set must be added here by hand, on purpose. +export const LIVE_SETS = ['pagoda', 'agent-pagoda', 'orrery', 'agent-orrery']; +export const LIVE_RE = /^live\/(?:(?:pagoda|agent-pagoda)\/[A-Za-z0-9._-]+\/page\.html|(?:orrery|agent-orrery)\/[A-Za-z0-9._-]+\/orrery\.svg)$/; export const REPO_PREFIX = 'https://github.com/alexderz/agentic-model-testing'; export const OTHER_FAMILIES = new Set(['other', 'unknown', '']); export const OTHER_COLOR = '#8a94a8'; // neutral silver for Other/Unknown; never handed to a real family @@ -56,6 +58,15 @@ export function fmtDate(s) { return `${months[mi] || m[2]} ${+m[3]}, ${m[1]}`; } export const shortLabel = (label) => str(label).split(' · ')[0].trim(); +// The first sentence of a paragraph (up to the first ". ", "! " or "? "), for one-line summaries. +export function firstSentence(s) { + const t = str(s).trim(); + const m = t.match(/^(.*?[.!?])(?:\s|$)/); + return m ? m[1] : t; +} +// A set key as a label: "agent-pagoda" → "agent pagoda". The base set of an agent set is the same key without "agent-". +export const setLabel = (set) => str(set).replace(/-/g, ' '); +export const baseSet = (set) => (/^agent-/.test(str(set)) ? str(set).slice(6) : null); export const frameMs = (url) => { const m = str(url).match(/(\d+)\.webp$/); return m ? +m[1] : null; }; // Deterministic 0..1 from a string (FNV-1a); used to scatter decorative marks without Math.random. export function hash01(s) { @@ -227,12 +238,31 @@ export const colorOf = (colors, family) => (colors && colors.has(family) ? color // ------------------------------------------------------------------- scales // "n of m" scales come from the data: an optional tests[].max_* field, else the largest value seen. +// Checks: tests[].max_checks, else the largest value seen for the test. When the test's passing runs reach different +// totals (test 4: each ticket has its own hidden-check count) there is no single scale and the count stands alone. export function checksMax(data, test) { const t = testOf(data, test); if (t.max_checks != null) return t.max_checks; const k = testKey(test); - const seen = arr(data.runs).filter((r) => r.test === k || r.flavor === k).map((r) => r.checks).filter((v) => v != null); - return seen.length ? Math.max(...seen) : null; + const rows = arr(data.runs).filter((r) => r.test === k || r.flavor === k); + const seen = rows.map((r) => r.checks).filter((v) => v != null); + if (!seen.length) return null; + const passing = new Set(rows.filter((r) => r.verdict === 'pass' && r.checks != null).map((r) => r.checks)); + return passing.size > 1 ? null : Math.max(...seen); +} +// Names for the one-shot rubric points: tests[].points, else the parenthesised list in tests[].graded that has one name per +// point ("five points of 0-2 (draws, planets, gears, lighting, motion)"), else the base set's names for an agent set. +export function pointNames(data, test, set, n, depth = 0) { + const t = testOf(data, test); + if (t.points.length) return t.points; + for (const m of str(t.graded).matchAll(/\(([^()]+)\)/g)) { + const names = m[1].split(',').map((x) => x.trim()).filter(Boolean); + if (n > 0 && names.length === n && names.every((x) => /^[A-Za-z][A-Za-z -]*$/.test(x))) return names; + } + const base = baseSet(set); + const rows = base ? arr(obj(data.oneshots)[base]) : []; + if (rows.length && depth < 2) return pointNames(data, rows[0].test, base, n, depth + 1); + return []; } export function pointMax(data, test) { const t = testOf(data, test); @@ -262,12 +292,18 @@ export function summary(data) { const dates = [...runs.map((r) => r.finished), ...arr(data.test3).map((r) => r.finished)].filter(Boolean).sort(); const findingDates = arr(data.findings).map((f) => f.date).filter(Boolean).sort(); const families = familyOrder(data).filter((f) => !OTHER_FAMILIES.has(f.toLowerCase())); - const minutes = runs.reduce((s, r) => s + (r.minutes || 0), 0) + arr(data.test3).reduce((s, r) => s + (r.minutes || 0), 0); + const minutes = runs.reduce((s, r) => s + (r.minutes || 0), 0) + arr(data.test3).reduce((s, r) => s + (r.minutes || 0), 0) + + oneshotRows.reduce((s, r) => s + (testOf(data, r.test).kind === 'agentic' && r.wall_s ? r.wall_s / 60 : 0), 0); + // a one-shot set holds one reply each; a set whose test is agentic (test 8) holds agent pieces + const agentPieces = oneshotRows.filter((r) => testOf(data, r.test).kind === 'agentic').length; + // the span of dated things: timeline steps, runs, test 3 rows and findings + const all = [...dates, ...findingDates, ...arr(data.timeline).map((t) => t.date).filter(Boolean)].map((d) => d.slice(0, 10)).sort(); return { models: models.length, families: families.length, runs: runs.length, passes, test3: arr(data.test3).length, - oneshots: oneshotRows.length, frames, playable, uncensored: models.filter((m) => data.models[m].uncensored).length, + oneshots: oneshotRows.length - agentPieces, agentPieces, frames, playable, uncensored: models.filter((m) => data.models[m].uncensored).length, fastest: tps.length ? Math.max(...tps) : null, slowest: tps.length ? Math.min(...tps) : null, first: dates[0] || '', last: dates[dates.length - 1] || '', newestFinding: findingDates[findingDates.length - 1] || '', agentHours: minutes / 60, + covers: { from: all[0] || '', to: all[all.length - 1] || '' }, tests: new Set([...runs.map(effectiveTest), ...arr(data.test3).length ? ['3'] : [], ...oneshotRows.map((r) => r.test)]).size, }; } @@ -439,14 +475,14 @@ export function oneshotCards(data, { test = 'all', sort = 'score', q = '' } = {} for (const r of rows) { const m = modelOf(data, r.entry), t = testOf(data, r.test); if (!pmax.has(r.test)) pmax.set(r.test, pointMax(data, r.test)); - const pointNames = t.points.length ? t.points : []; + const names = pointNames(data, r.test, set, r.points.length); const pointMaxV = pmax.get(r.test); const max = r.points.length * pointMaxV; cards.push({ - ...r, set, modelLabel: m.label, short: shortLabel(m.label), family: m.family, color: colorOf(colors, m.family), uncensored: m.uncensored, + ...r, set, setLabel: setLabel(set), kind: t.kind, modelLabel: m.label, short: shortLabel(m.label), family: m.family, color: colorOf(colors, m.family), uncensored: m.uncensored, testName: t.name, pointMax: pointMaxV, max, full: max > 0 && r.total != null && r.total >= max, - points: r.points.map((p, i) => ({ ...p, name: pointNames[i] || p.key })), - alt: `${t.name || `Test ${r.test}`} by ${shortLabel(m.label)}, test ${r.test}`, + points: r.points.map((p, i) => ({ ...p, name: names[i] || p.key })), + alt: `${setLabel(set)} by ${shortLabel(m.label)}, test ${r.test}`, }); } } @@ -465,11 +501,45 @@ export function oneshotSets(data) { const t = rows.length ? testOf(data, rows[0].test) : null; const pm = rows.length ? pointMax(data, rows[0].test) : 1; const graded = rows.filter((r) => r.total != null); - return { set, test: rows.length ? rows[0].test : '?', name: t ? t.name : set, n: rows.length, withFrames: rows.filter((r) => r.frames.length).length, live: rows.filter((r) => r.live).length, - pointMax: pm, full: graded.filter((r) => r.points.length && r.total >= r.points.length * pm).length, zero: graded.filter((r) => r.total === 0).length, - cut: rows.filter((r) => r.finish === 'length').length, mean: graded.length ? graded.reduce((s, r) => s + r.total, 0) / graded.length : null }; + return { set, label: setLabel(set), base: baseSet(set), kind: t ? t.kind : '', test: rows.length ? rows[0].test : '?', name: t ? t.name : set, n: rows.length, withFrames: rows.filter((r) => r.frames.length).length, live: rows.filter((r) => r.live).length, + pointMax: pm, max: rows.length ? Math.max(0, ...rows.map((r) => r.points.length * pm)) : 0, full: graded.filter((r) => r.points.length && r.total >= r.points.length * pm).length, zero: graded.filter((r) => r.total === 0).length, + cut: rows.filter((r) => /length|cap/i.test(r.finish)).length, mean: graded.length ? graded.reduce((s, r) => s + r.total, 0) / graded.length : null }; }); } +// One shot against agent: for every set named agent- whose base set exists, each agent row is paired with the best +// graded one-shot reply by the same model (best of n, so the comparison favours the one-shot). Rows sort by agent score. +export function agentPairs(data) { + const colors = familyColors(data); + const sets = obj(data.oneshots); + const panels = []; + for (const [set, rows] of Object.entries(sets)) { + const base = baseSet(set); + if (!base || !Array.isArray(sets[base])) continue; + const t = rows.length ? testOf(data, rows[0].test) : null; + const pm = rows.length ? pointMax(data, rows[0].test) : 1; + const max = Math.max(1, ...rows.map((r) => r.points.length * pm)); + const byModel = new Map(); + for (const r of rows) { + if (!byModel.has(r.entry)) byModel.set(r.entry, { agent: [], oneshot: [] }); + byModel.get(r.entry).agent.push(r); + } + for (const r of sets[base]) if (byModel.has(r.entry) && r.total != null) byModel.get(r.entry).oneshot.push(r); + const best = (list) => list.filter((r) => r.total != null).sort((a, b) => b.total - a.total)[0] || null; + const out = []; + for (const [id, g] of byModel) { + const m = modelOf(data, id), a = best(g.agent), o = best(g.oneshot); + out.push({ model: id, label: m.label, short: shortLabel(m.label), family: m.family, color: colorOf(colors, m.family), uncensored: m.uncensored, + agent: a ? a.total : null, agentN: g.agent.length, agentCase: a ? a.case : '', oneshot: o ? o.total : null, oneshotN: g.oneshot.length, oneshotCase: o ? o.case : '' }); + } + out.sort((a, b) => (b.agent ?? -1) - (a.agent ?? -1) || (b.oneshot ?? -1) - (a.oneshot ?? -1) || a.label.localeCompare(b.label)); + const both = out.filter((r) => r.agent != null && r.oneshot != null); // means over the same models on both sides + const mean = (k) => { const vs = both.map((r) => r[k]); return vs.length ? vs.reduce((s, v) => s + v, 0) / vs.length : null; }; + panels.push({ set, base, label: setLabel(set), baseLabel: setLabel(base), test: rows.length ? rows[0].test : '?', name: t ? t.name : set, max, rows: out, + meanAgent: mean('agent'), meanOneshot: mean('oneshot'), lifted: out.filter((r) => r.agent != null && r.oneshot != null && r.agent > r.oneshot).length, paired: out.filter((r) => r.oneshot != null).length }); + } + panels.sort((a, b) => b.rows.length - a.rows.length || a.set.localeCompare(b.set)); + return panels; +} // -------------------------------------------------------------------- picks export function pickCards(data) { @@ -498,11 +568,6 @@ export function findingMetrics(f) { gain: m.before && m.after != null ? m.after / m.before : null }; }); } -export function eventKinds(data) { - const c = new Map(); - for (const e of arr(data.events)) c.set(e.kind, (c.get(e.kind) || 0) + 1); - return [...c.entries()].sort((a, b) => b[1] - a[1]).map(([kind, n]) => ({ kind, n })); -} // ------------------------------------------------------------------- motifs // Numbers behind the decorative chapter motifs. Each entry is a plain list the drawing code turns into SVG, so the @@ -545,11 +610,6 @@ export function motifs(data) { const tl = arr(data.timeline).filter((t) => t.date); const t0 = tl.length ? Date.parse(tl[0].date) : 0, t1 = tl.length ? Date.parse(tl[tl.length - 1].date) : 1; out.timeline = tl.map((t) => { const ms = Date.parse(t.date); return { x: Number.isFinite(ms) && t1 > t0 ? (ms - t0) / (t1 - t0) : 0 }; }); - const kinds = eventKinds(data); - const kmax = Math.max(1, ...kinds.map((k) => k.n)); - const kcols = colorSeries(kinds.length, ENGINE_BASE); - out.log = kinds.map((k, i) => ({ kind: k.kind, h: k.n / kmax, color: kcols[i] })); - out.caveats = arr(data.caveats).map((c, i) => ({ i, len: Math.min(1, c.length / 240) })); return out; } @@ -567,10 +627,12 @@ export function deriveAll(raw) { out.test3 = test3Groups(data); out.ab = abGroups(data); out.oneshots = oneshotCards(data); out.oneshotSets = oneshotSets(data); for (const s of ['speed', 'model', 'newest']) oneshotCards(data, { sort: s, test: '5', q: 'a' }); + oneshotCards(data, { test: 'agent-pagoda' }); out.picks = pickCards(data); out.familyPicks = familyPickRows(data); - out.findings = data.findings.map(findingMetrics); out.eventKinds = eventKinds(data); + out.findings = data.findings.map(findingMetrics); out.pairs = agentPairs(data); + out.pointNames = Object.fromEntries(Object.entries(data.oneshots).map(([set, rows]) => [set, rows.length ? pointNames(data, rows[0].test, set, rows[0].points.length) : []])); out.motifs = motifs(data); - out.fmt = [fmt(1234.5), fmt(null), fmt(2.5), fmt(13.4), fmt(13.4, 1), fmtMinutes(191), fmtMinutes(null), fmtDate('2026-10-22'), fmtDate(''), fmtDate(null)]; + out.fmt = [fmt(1234.5), fmt(null), fmt(2.5), fmt(13.4), fmt(13.4, 1), fmtMinutes(191), fmtMinutes(null), fmtDate('2026-10-22'), fmtDate(''), fmtDate(null), firstSentence('A. B'), firstSentence(null), setLabel('agent-pagoda'), baseSet('x')]; out.series = colorSeries(14); return { data, out }; } diff --git a/site/assets/style.css b/site/assets/style.css index 2f9cfb6..dce5bca 100644 --- a/site/assets/style.css +++ b/site/assets/style.css @@ -51,7 +51,12 @@ h1 { font-size: clamp(2.8rem, 7vw, 5.4rem); line-height: 0.98; background: linea .generated { color: var(--faint); font-size: 0.85rem; display: flex; flex-wrap: wrap; gap: 4px 16px; } .generated .fresh { white-space: nowrap; color: var(--muted); } .generated .fresh small { color: var(--faint); text-transform: uppercase; letter-spacing: 0.1em; font-size: 0.68rem; } -.hero-note { color: var(--muted); font-size: 0.9rem; max-width: 52ch; } +.hero-key { list-style: none; margin: 10px 0 0; padding: 0; display: flex; flex-wrap: wrap; gap: 6px 18px; color: var(--muted); font-size: 0.82rem; } +.hero-key li { display: inline-flex; align-items: center; gap: 7px; } +.hero-key .mark { display: inline-block; width: 12px; height: 12px; border-radius: 50%; background: var(--teal); position: relative; } +.hero-key .mark.speed { background: none; border: 1px solid var(--faint); } .hero-key .mark.speed::after { content: ""; position: absolute; left: 7px; top: -2px; width: 5px; height: 5px; border-radius: 50%; background: var(--gold); } +.hero-key .mark.size { background: var(--sapphire); } .hero-key .mark.halo { box-shadow: 0 0 0 2px var(--ink), 0 0 0 3px var(--teal); background: var(--teal); width: 8px; height: 8px; margin: 0 2px; } +.hero-key .mark.unc { background: var(--magenta); outline: 1px dashed #fff; outline-offset: 2px; width: 8px; height: 8px; margin: 0 2px; } .hero-art { margin: 0; } .orrery-mount { aspect-ratio: 1; max-width: 640px; margin: 0 auto; } .orrery { width: 100%; height: 100%; display: block; overflow: visible; } @@ -88,10 +93,21 @@ h1 { font-size: clamp(2.8rem, 7vw, 5.4rem); line-height: 0.98; background: linea background: repeating-linear-gradient(135deg, transparent 0 7px, rgba(233, 237, 245, 0.012) 7px 8px); } #machine { --accent: var(--teal); } #tests { --accent: var(--violet); } #picks { --accent: var(--gold); } #runs { --accent: var(--sapphire); } #struggle { --accent: var(--magenta); } #test3 { --accent: var(--gold); } #ab { --accent: var(--violet); } #oneshots { --accent: var(--magenta); } -#findings { --accent: var(--teal); } #timeline { --accent: var(--sapphire); } #log { --accent: var(--gold); } #caveats { --accent: var(--fail); } +#findings { --accent: var(--teal); } #timeline { --accent: var(--sapphire); } .chapter-head { display: grid; grid-template-columns: auto 1fr auto; gap: 8px 20px; align-items: baseline; margin-bottom: 26px; } .chapter-head .num { font-family: var(--serif); font-size: 1rem; color: var(--accent); letter-spacing: 0.1em; grid-row: span 2; align-self: start; padding-top: 0.7em; border-top: 2px solid var(--accent); } -.chapter-head .lede { grid-column: 2; color: var(--muted); max-width: 72ch; font-size: 1.02rem; } +.chapter-head .lede { grid-column: 2; color: var(--muted); max-width: 72ch; font-size: 0.95rem; margin: 0; } +.chapter-head .lede:empty { display: none; } +/* folded text: settings, notes, trade-offs, grading. Closed by default; the summary is a small muted word. */ +.fold { font-size: 0.88rem; color: var(--muted); margin-top: 8px; } +.fold > summary { cursor: pointer; color: var(--faint); text-transform: uppercase; letter-spacing: 0.12em; font-size: 0.68rem; list-style: none; display: inline-flex; align-items: center; gap: 6px; } +.fold > summary::-webkit-details-marker { display: none; } +.fold > summary::before { content: "+"; display: inline-grid; place-items: center; width: 14px; height: 14px; border: 1px solid var(--line-2); border-radius: 4px; font-size: 0.7rem; color: var(--gold); } +.fold[open] > summary::before { content: "–"; } +.fold > summary:hover { color: var(--gold); } +.fold > p { margin: 6px 0 0; } +.noteb { background: none; border: 1px solid var(--line-2); border-radius: 999px; color: var(--faint); cursor: pointer; padding: 0 8px; font-size: 0.68rem; text-transform: uppercase; letter-spacing: 0.1em; line-height: 1.6; vertical-align: middle; } +.noteb:hover, .noteb.open, .noteb[aria-expanded="true"] { color: var(--gold); border-color: var(--gold); } .chapter-head .motif { grid-column: 3; grid-row: 1 / span 2; align-self: center; width: clamp(140px, 17vw, 220px); opacity: 0.92; } .motif-svg { display: block; width: 100%; height: auto; overflow: visible; filter: drop-shadow(0 0 10px color-mix(in srgb, var(--accent) 28%, transparent)); } @media (max-width: 860px) { .chapter-head .motif { grid-column: 2; grid-row: auto; width: 150px; justify-self: end; margin-top: -4px; } } @@ -139,8 +155,8 @@ h1 { font-size: clamp(2.8rem, 7vw, 5.4rem); line-height: 0.98; background: linea .test-card .tn { font-family: var(--serif); font-size: 2.4rem; line-height: 1; color: var(--gold); float: right; margin-left: 10px; opacity: 0.9; } .test-card h3 { font-size: 1.2rem; margin-bottom: 6px; } .test-card .what { margin: 8px 0; } -.test-card .graded { color: var(--muted); font-size: 0.9rem; } -.test-card .meta { display: flex; gap: 8px; flex-wrap: wrap; margin-top: 10px; align-items: center; } +.test-card.pending { opacity: 0.75; } +.test-card .meta { display: flex; gap: 8px; flex-wrap: wrap; margin-top: 6px; align-items: center; } .test-card .n { color: var(--muted); font-size: 0.85rem; } /* picks */ @@ -151,7 +167,7 @@ h1 { font-size: clamp(2.8rem, 7vw, 5.4rem); line-height: 0.98; background: linea .pick-card .role { text-transform: uppercase; letter-spacing: 0.16em; font-size: 0.72rem; color: var(--gold); } .pick-card h4 { font-family: var(--serif); font-weight: 500; font-size: 1.35rem; margin: 4px 0 6px; } .pick-card .verdict-text { margin-bottom: 6px; } -.pick-card .trade { color: var(--muted); font-size: 0.92rem; border-left: 2px solid var(--line-2); padding-left: 10px; } +.pick-card .fold > p { border-left: 2px solid var(--line-2); padding-left: 10px; } .pick-card .meta { display: flex; flex-wrap: wrap; gap: 6px; margin: 8px 0; } .pick-card .meta .mono { color: var(--muted); font-size: 0.8rem; } .pick-card ul { list-style: none; padding: 0; margin: 6px 0 0; font-size: 0.9rem; color: var(--muted); } @@ -161,7 +177,7 @@ h1 { font-size: clamp(2.8rem, 7vw, 5.4rem); line-height: 0.98; background: linea .fam-row .cell small { display: block; color: var(--faint); text-transform: uppercase; letter-spacing: 0.1em; font-size: 0.66rem; } .fam-row .cell b { font-weight: 500; } .fam-row .cell .none { color: var(--faint); font-style: italic; } -.fam-row .note { grid-column: 1 / -1; color: var(--muted); font-size: 0.9rem; margin: 0; } +.fam-row .fold { grid-column: 1 / -1; margin-top: 0; } .fam-row .count { font-size: 0.8rem; } @media (max-width: 860px) { .fam-row { grid-template-columns: 1fr 1fr; } .fam-row .famhead { grid-column: 1 / -1; } } @@ -176,6 +192,7 @@ h1 { font-size: clamp(2.8rem, 7vw, 5.4rem); line-height: 0.98; background: linea @media (max-width: 700px) { .ladder { --lbl: 140px; --num: 52px; } } .ladder-head, .ladder-row { display: grid; grid-template-columns: var(--lbl) 1fr var(--num); gap: 10px; align-items: center; } .ladder-head { color: var(--faint); font-size: 0.75rem; text-transform: uppercase; letter-spacing: 0.1em; padding-bottom: 20px; } +.vkey { display: flex; gap: 12px; text-transform: none; letter-spacing: 0; font-size: 0.8rem; } .ladder-axis { position: relative; height: 1px; background: var(--line-2); } .ladder-axis .tick { position: absolute; top: 4px; transform: translateX(-50%); font-size: 0.72rem; color: var(--faint); } .ladder-axis .tick::before { content: ""; position: absolute; left: 50%; top: -5px; width: 1px; height: 4px; background: var(--line-2); } @@ -218,26 +235,40 @@ table.runs td .case { color: var(--faint); font-size: 0.78rem; display: block; } .strip .dot-g { outline: none; } .stat-row { display: flex; flex-wrap: wrap; gap: 12px; margin-top: 14px; } .stat { border-radius: 12px; padding: 10px 14px; min-width: 150px; } +.stat.wide { flex: 1 1 260px; } .stat .k { font-size: 0.72rem; text-transform: uppercase; letter-spacing: 0.12em; color: var(--muted); display: flex; gap: 6px; align-items: center; } .stat .v { font-family: var(--serif); font-size: 1.6rem; line-height: 1.1; } .stat .v small { font-family: var(--sans); font-size: 0.8rem; color: var(--muted); } -/* test 3 */ +/* test 3: one card per model, one row per flavour; the column labels sit once above the list */ .t3-list { display: grid; gap: 10px; } +.t3-list .reveal-list { display: grid; gap: 10px; } +.t3-head, .t3-row { display: grid; grid-template-columns: 30px 1.1fr 1.1fr 1.3fr 28px minmax(130px, auto); gap: 4px 12px; align-items: center; } +.t3-head { padding: 0 14px 0 calc(220px + 18px + 14px); color: var(--faint); font-size: 0.68rem; text-transform: uppercase; letter-spacing: 0.12em; margin-bottom: -4px; } +.t3-head .num, .t3-row > .num { text-align: right; } .t3 { display: grid; grid-template-columns: 220px 1fr; gap: 6px 18px; border-radius: 12px; padding: 12px 14px; } -@media (max-width: 700px) { .t3 { grid-template-columns: 1fr; } .t3-row { grid-template-columns: 30px 1fr; } .t3-row > .num { grid-column: 2; text-align: left; font-size: 0.8rem; color: var(--muted); } .t3-row .note, .t3-row .noteb { grid-column: 1 / -1; } .pips i { width: 5px; height: 9px; } } .t3 .head b { font-weight: 500; display: block; } .t3 .head .chips { display: flex; gap: 6px; flex-wrap: wrap; margin-top: 4px; } .t3 .flavors { display: grid; gap: 8px; } -.t3-row { display: grid; grid-template-columns: 34px 1fr auto; gap: 4px 12px; align-items: center; font-size: 0.88rem; } +.t3-row { font-size: 0.86rem; } .t3-row .flavor { font-family: var(--serif); color: var(--gold); font-size: 1.1rem; } -.t3-row .facts { display: flex; flex-wrap: wrap; gap: 6px 16px; align-items: center; color: var(--muted); } -.t3-row .judge { font-size: 0.85rem; } +.t3-row .cell { display: inline-flex; align-items: center; gap: 8px; color: var(--muted); white-space: nowrap; } +.t3-row .cell small { font-size: 0.78rem; font-variant-numeric: tabular-nums; } +.t3-row .judge { font-size: 1rem; text-align: center; color: var(--faint); } .t3-row .judge.ok { color: var(--pass); } .t3-row .judge.bad { color: var(--fail); } -.t3-row .note { grid-column: 2 / -1; color: var(--muted); font-size: 0.84rem; margin: 0; display: -webkit-box; -webkit-line-clamp: 3; -webkit-box-orient: vertical; overflow: hidden; } -.t3-row .note.open { display: block; } -.noteb { background: none; border: 0; color: var(--teal); cursor: pointer; text-align: left; padding: 0; font-size: 0.8rem; } -.t3-row .noteb { grid-column: 2 / -1; justify-self: start; } +.t3-row > .num { color: var(--muted); font-size: 0.82rem; white-space: nowrap; display: inline-flex; gap: 8px; align-items: center; justify-content: flex-end; } +.t3-row .note { grid-column: 2 / -1; color: var(--muted); font-size: 0.84rem; margin: 0; } +@media (max-width: 900px) { + .t3 { grid-template-columns: 1fr; } .t3-head { display: none; } + .t3-row { grid-template-columns: 30px 24px 1fr; gap: 3px 8px; } + .t3-row .flavor { grid-row: 1 / span 4; align-self: start; } + .t3-row .cell { grid-column: 2 / -1; } + .t3-row .judge { grid-column: 2; grid-row: 4; } + .t3-row > .num { grid-column: 3; grid-row: 4; justify-content: flex-start; } + .t3-row .note { grid-column: 2 / -1; } + .pips i { width: 5px; height: 9px; } + .pair-fig figcaption .pair-stat { margin-left: 0; flex-basis: 100%; } +} .pips { display: inline-flex; gap: 2px; vertical-align: middle; } .pips i { width: 7px; height: 11px; border-radius: 2px; background: rgba(233,237,245,0.1); display: inline-block; } .pips i.on { background: var(--teal); } @@ -265,9 +296,32 @@ table.runs td .case { color: var(--faint); font-size: 0.78rem; display: block; } /* gallery */ .set-row { display: flex; flex-wrap: wrap; gap: 12px; margin-bottom: 18px; } -.set-card { border-radius: 12px; padding: 10px 14px; display: grid; grid-template-columns: auto; gap: 2px; min-width: 200px; } -.set-card b { font-family: var(--serif); font-weight: 500; font-size: 1.15rem; } -.set-card span { color: var(--muted); font-size: 0.85rem; } +.set-card { border-radius: 12px; padding: 10px 14px; display: grid; gap: 6px; min-width: 200px; flex: 1 1 200px; } +.set-card .set-head { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } +.set-card .set-head b { font-family: var(--serif); font-weight: 500; font-size: 1.2rem; text-transform: capitalize; } +.set-card.agentic .set-head b { color: var(--teal); } +.set-card .set-nums { display: flex; flex-wrap: wrap; gap: 4px 14px; color: var(--faint); font-size: 0.78rem; } +.set-card .sn b { font-family: var(--serif); font-weight: 500; font-size: 1.1rem; color: var(--text); } +.set-card .sn.live-n b { color: var(--gold); } +/* one shot → agent */ +.pairs-block { margin: 6px 0 22px; } +.pairs-block .sub { margin-top: 0.4rem; } +.pairs-block .lede { color: var(--muted); font-size: 0.9rem; margin: 0 0 10px; } +.pairs-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 14px; } +.pair-fig { margin: 0; background: var(--card); border: 1px solid var(--line); border-radius: var(--r); padding: 12px 10px 4px; min-width: 0; overflow-x: auto; } +.pair-fig figcaption { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 12px; padding: 0 6px 4px; font-family: var(--serif); font-size: 1.15rem; text-transform: capitalize; } +.pair-fig figcaption b { font-weight: 500; } .pair-fig figcaption b:last-of-type { color: var(--teal); } +.pair-fig figcaption .arrow { color: var(--gold); } +.pair-fig .pair-stat { margin-left: auto; font-family: var(--sans); font-size: 0.8rem; color: var(--muted); text-transform: none; font-variant-numeric: tabular-nums; } +.pairs { display: block; max-width: 100%; height: auto; } +.pairs .grid { stroke: var(--line); } .pairs .lane { stroke: var(--line-2); } +.pairs .axis { font: 12px var(--sans); fill: var(--muted); } .pairs .axis.dim { fill: var(--faint); } .pairs .axis.num { font-variant-numeric: tabular-nums; fill: var(--text); } +.pairs .axis.lbl { fill: var(--text); } +.pairs .link { stroke: var(--faint); stroke-width: 2.5; stroke-linecap: round; } .pairs .link.up { stroke: var(--pass); } .pairs .link.down { stroke: var(--fail); } +.pairs .one { fill: var(--ink); stroke: var(--faint); stroke-width: 1.6; } .pairs tspan.one { fill: var(--faint); } +.pairs .agent { fill: var(--c, var(--teal)); stroke: var(--ink); stroke-width: 1.2; } .pairs tspan.agent { fill: var(--teal); } +.pairs .pair { outline: none; } .pairs .pair:hover .agent, .pairs .pair:focus-visible .agent { stroke: #fff; r: 7; } +.pairs .pair.mean .axis { font-style: italic; } .pairs .pair.mean .agent { fill: var(--gold); } .pairs .pair.mean .link { stroke-dasharray: 3 3; } .gallery { position: relative; padding: 18px; margin: 0 -18px; border-radius: 20px; background: radial-gradient(900px 400px at 50% 0%, color-mix(in srgb, var(--magenta) 10%, transparent), transparent 70%), repeating-linear-gradient(0deg, transparent 0 23px, rgba(233,237,245,0.025) 23px 24px), #0a0f1d; box-shadow: inset 0 0 0 1px var(--line), inset 0 30px 60px rgba(0,0,0,0.35); } .gallery .reveal-list { display: grid; grid-template-columns: repeat(auto-fill, minmax(230px, 1fr)); grid-auto-flow: dense; gap: 18px; } @@ -283,7 +337,6 @@ table.runs td .case { color: var(--faint); font-size: 0.78rem; display: block; } .shot .body::before { content: ""; display: block; height: 2px; margin: 0 0 8px; border-radius: 2px; background: linear-gradient(90deg, transparent, var(--c, var(--gold)) 40%, var(--gold) 60%, transparent); opacity: 0.8; } .shot .pic img.on { opacity: 1; } .shot .pic .blank { position: absolute; inset: 0; display: grid; place-items: center; color: var(--faint); font-size: 0.85rem; text-align: center; padding: 14px; } -.shot .pts span.l0 { color: var(--faint); } .shot .pic .blank i { font-style: normal; font-size: 2rem; display: block; opacity: 0.5; } .shot .frames { position: absolute; left: 8px; bottom: 8px; display: flex; gap: 4px; margin: 0; padding: 0; list-style: none; } .shot .frames button { min-width: 26px; height: 20px; border-radius: 6px; border: 1px solid rgba(255,255,255,0.3); background: rgba(12, 18, 34, 0.7); color: var(--text); font-size: 0.62rem; cursor: pointer; padding: 0 5px; white-space: nowrap; } @@ -292,20 +345,14 @@ table.runs td .case { color: var(--faint); font-size: 0.78rem; display: block; } .shot .body > div:first-child { grid-column: 1; } .shot .body b { font-weight: 500; display: block; line-height: 1.3; font-family: var(--serif); font-size: 1.08rem; } .shot .body .meta { color: var(--muted); font-size: 0.8rem; display: flex; flex-wrap: wrap; gap: 4px 8px; align-items: center; } -.shot .body .note { grid-column: 1 / -1; color: var(--muted); font-size: 0.84rem; margin: 2px 0 0; display: -webkit-box; -webkit-line-clamp: 4; -webkit-box-orient: vertical; overflow: hidden; } -.shot .body .note.open { display: block; } -.shot .body .noteb { grid-column: 1 / -1; } -.shot .pts { grid-column: 1 / -1; display: flex; flex-wrap: wrap; gap: 4px 10px; font-size: 0.74rem; color: var(--faint); margin-top: 2px; } -.shot .pts span.l2 { color: var(--pass); } .shot .pts span.l1 { color: var(--partial); } +.shot .body .note { grid-column: 1 / -1; color: var(--muted); font-size: 0.84rem; margin: 2px 0 0; } .petals .petal-bg { fill: rgba(233,237,245,0.07); } .petals .petal { fill: var(--partial); } .petals .petal.lvl-2 { fill: var(--pass); } .petals .petal-total { font: 600 13px var(--sans); fill: var(--text); } .shot .cut { color: var(--fail); } /* live pieces: the Play control on a card, the badge on full-marks cards, the count line */ -.chapter-head .live-line { grid-column: 2; margin: 2px 0 0; color: var(--text); font-size: 0.98rem; max-width: 72ch; } -.live-line i, .set-card .live-n i, .shot .play i { font-style: normal; color: var(--gold); margin-right: 2px; } -.set-card .live-n { color: var(--gold); } +.set-card .live-n i, .shot .play i { font-style: normal; color: var(--gold); margin-right: 2px; } .shot .play { position: absolute; right: 8px; bottom: 8px; display: inline-flex; align-items: center; gap: 4px; border-radius: 999px; border: 1px solid rgba(242,193,78,0.7); background: rgba(12, 18, 34, 0.82); color: var(--gold); font-size: 0.72rem; font-weight: 600; letter-spacing: 0.02em; padding: 3px 10px 3px 8px; cursor: pointer; backdrop-filter: blur(3px); box-shadow: 0 2px 10px rgba(0,0,0,0.5); transition: background 0.2s, color 0.2s, transform 0.2s; } @@ -373,10 +420,10 @@ dialog.viewer::backdrop { background: rgba(5, 8, 16, 0.85); } } /* findings */ -.findings { display: grid; gap: 16px; } -.finding { border-radius: var(--r); padding: 20px 22px; display: grid; grid-template-columns: minmax(0, 1.1fr) minmax(0, 1fr); gap: 10px 28px; } +.findings .reveal-list { display: grid; gap: 14px; } +.finding { border-radius: var(--r); padding: 18px 22px; display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.1fr); gap: 6px 28px; } /* the typographic moment: lead metric as big numerals */ -.big-num { display: grid; gap: 2px; margin-bottom: 12px; } +.big-num { display: grid; gap: 2px; margin-bottom: 8px; } .big-num .lbl { color: var(--muted); font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.12em; } .big-num .row { display: flex; align-items: baseline; flex-wrap: wrap; gap: 0 10px; font-family: var(--serif); line-height: 1; } .big-num .from { font-size: clamp(1.6rem, 3vw, 2.4rem); color: var(--faint); font-variant-numeric: tabular-nums; } @@ -387,11 +434,11 @@ dialog.viewer::backdrop { background: rgba(5, 8, 16, 0.85); } .big-num .gainbadge { font-family: var(--sans); font-size: 0.82rem; font-weight: 600; color: #0c1222; background: var(--pass); border-radius: 999px; padding: 2px 9px; align-self: center; } @media (max-width: 860px) { .finding { grid-template-columns: 1fr; } } .finding .when { color: var(--gold); font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.14em; display: flex; gap: 10px; align-items: center; } -.finding h3 { font-size: 1.45rem; margin: 4px 0 8px; grid-column: 1 / -1; } -.finding .summary { margin: 0; } -.finding .settings { margin-top: 10px; font-size: 0.8rem; color: var(--muted); word-break: break-word; } +.finding h3 { font-size: 1.35rem; margin: 2px 0 4px; grid-column: 1 / -1; } +.finding .summary { margin: 0; color: var(--muted); font-size: 0.95rem; } +.finding .settings { margin-top: 8px; font-size: 0.8rem; color: var(--muted); word-break: break-word; } .finding .settings code { color: var(--teal); } -.finding .quality, .finding .caveat { font-size: 0.9rem; color: var(--muted); margin: 10px 0 0; } +.finding .quality, .finding .caveat { font-size: 0.88rem; color: var(--muted); margin: 8px 0 0; } .finding .caveat::before, .finding .quality::before { content: attr(data-k); display: block; color: var(--faint); font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.12em; } .finding .caveat { border-left: 2px solid var(--fail); padding-left: 10px; } .ba { display: grid; grid-template-columns: minmax(90px, 0.8fr) 1.4fr auto; gap: 10px; align-items: center; font-size: 0.86rem; margin: 6px 0; } @@ -401,40 +448,36 @@ dialog.viewer::backdrop { background: rgba(5, 8, 16, 0.85); } .ba-bar { display: block; height: 100%; border-radius: 4px; } .ba-bar.before { background: var(--faint); } .ba-bar.after { background: var(--teal); } .ba-num { white-space: nowrap; font-variant-numeric: tabular-nums; } .ba-num small { color: var(--muted); } -.ba-key { color: var(--faint); font-size: 0.75rem; display: flex; gap: 12px; margin-bottom: 4px; } -.ba-key i { display: inline-block; width: 14px; height: 6px; border-radius: 3px; margin-right: 4px; vertical-align: 1px; } -.ba-key i.before { background: var(--faint); } .ba-key i.after { background: var(--teal); } + /* timeline river */ .river { list-style: none; margin: 0; padding: 0 0 0 26px; position: relative; } .river::before { content: ""; position: absolute; left: 7px; top: 6px; bottom: 6px; width: 2px; background: linear-gradient(var(--teal), var(--sapphire), var(--violet), var(--magenta), var(--gold)); border-radius: 2px; } -.river li { position: relative; padding: 0 0 26px 14px; } -.river li::before { content: ""; position: absolute; left: -25px; top: 7px; width: 12px; height: 12px; border-radius: 50%; background: var(--ink); border: 2px solid var(--gold); } -.river .date { color: var(--gold); font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.14em; } -.river .era { color: var(--faint); margin-left: 8px; letter-spacing: 0; text-transform: none; } -.river h3 { font-size: 1.3rem; margin: 2px 0 6px; } -.river .what { margin: 0 0 6px; max-width: 76ch; } -.river .why { color: var(--muted); font-size: 0.92rem; margin: 0; max-width: 76ch; } +.river li { position: relative; padding: 0 0 18px 14px; } +.river li::before { content: ""; position: absolute; left: -25px; top: 6px; width: 12px; height: 12px; border-radius: 50%; background: var(--ink); border: 2px solid var(--gold); } +.river li.month { padding: 6px 0 14px 14px; } +.river li.month::before { width: 8px; height: 8px; left: -23px; top: 10px; background: var(--sapphire); border-color: var(--sapphire); } +.river li.month span { font-family: var(--serif); color: var(--sapphire); font-size: 1rem; letter-spacing: 0.08em; text-transform: uppercase; } +.river .date { color: var(--gold); font-size: 0.74rem; text-transform: uppercase; letter-spacing: 0.14em; display: flex; gap: 10px; align-items: baseline; } +.river .era { color: var(--faint); letter-spacing: 0; text-transform: none; } +.river h3 { font-size: 1.2rem; margin: 1px 0 3px; } +.river .what { margin: 0; max-width: 76ch; color: var(--muted); font-size: 0.95rem; } +.river .what .noteb { margin-left: 6px; } +.river .why { display: none; color: var(--text); font-size: 0.9rem; margin: 6px 0 0; max-width: 76ch; border-left: 2px solid var(--teal); padding-left: 10px; } +.river li:hover .why, .river li:focus-within .why, .river li.open .why { display: block; } .river .why::before { content: "why "; color: var(--teal); text-transform: uppercase; letter-spacing: 0.12em; font-size: 0.7rem; } -/* log */ -.kind-row { list-style: none; margin: 0 0 14px; padding: 0; display: flex; flex-wrap: wrap; gap: 6px; } -.kind-row button { background: var(--card); border: 1px solid var(--line); border-radius: 999px; padding: 3px 10px; cursor: pointer; font-size: 0.82rem; color: var(--muted); } -.kind-row button[aria-pressed="true"] { border-color: var(--gold); color: var(--gold); background: rgba(242,193,78,0.08); } -.events .reveal-list { display: grid; gap: 8px; } -.event { display: grid; grid-template-columns: 130px 1fr; gap: 12px; padding: 10px 12px; border-radius: 10px; font-size: 0.9rem; } -@media (max-width: 600px) { .event { grid-template-columns: 1fr; gap: 4px; } } -.event time { color: var(--faint); font-variant-numeric: tabular-nums; font-size: 0.82rem; display: block; } -.event .chip { margin-top: 4px; } -.event p { margin: 0; } - -/* caveats */ -.caveats { margin: 0; padding: 0; list-style: none; counter-reset: cav; display: grid; gap: 10px; } -.caveats li { counter-increment: cav; border-left: 3px solid var(--fail); border-radius: 12px; padding: 14px 18px 14px 56px; position: relative; } -.caveats li::before { content: counter(cav, decimal-leading-zero); position: absolute; left: 16px; top: 10px; font-family: var(--serif); font-size: 1.5rem; color: var(--fail); } +/* caveats, folded in the footer */ +.caveats-wrap { margin: 14px 0; } +.caveats-wrap > summary { cursor: pointer; color: var(--text); font-family: var(--serif); font-size: 1.05rem; } +.caveats-wrap > summary:hover { color: var(--gold); } +.caveats { margin: 12px 0 0; padding: 0; list-style: none; counter-reset: cav; display: grid; gap: 10px; } +.caveats li { counter-increment: cav; border: 1px solid var(--line); border-left: 3px solid var(--fail); border-radius: 12px; padding: 12px 16px 12px 52px; position: relative; background: var(--card); color: var(--text); } +.caveats li::before { content: counter(cav, decimal-leading-zero); position: absolute; left: 14px; top: 8px; font-family: var(--serif); font-size: 1.4rem; color: var(--fail); } /* footer */ .foot { max-width: var(--maxw); margin: 40px auto 0; padding: 30px clamp(16px, 4vw, 48px) 60px; border-top: 1px solid var(--line); color: var(--muted); font-size: 0.9rem; } +.foot .method { color: var(--muted); } .links { list-style: none; margin: 10px 0; padding: 0; display: flex; flex-wrap: wrap; gap: 8px 20px; } .small { font-size: 0.8rem; color: var(--faint); } diff --git a/site/assets/viewer.js b/site/assets/viewer.js index 3dc4021..9dfd7dd 100644 --- a/site/assets/viewer.js +++ b/site/assets/viewer.js @@ -43,7 +43,7 @@ function build() { const stage = el('div', { class: 'stage', 'data-state': 'idle' }, el('div', { class: 'stage-box' })); const side = el('aside', { class: 'vside' }); dlg.append( - el('header', { class: 'vhead' }, el('span', { class: 'kicker' }, 'Live one-shot'), el('h2', { id: 'viewer-title' }), el('span', { class: 'vpos', 'aria-live': 'polite' }), closeBtn), + el('header', { class: 'vhead' }, el('span', { class: 'kicker' }, 'Live piece'), el('h2', { id: 'viewer-title' }), el('span', { class: 'vpos', 'aria-live': 'polite' }), closeBtn), el('div', { class: 'vmain' }, stage, side), el('footer', { class: 'vfoot' }, prev, el('span', { class: 'vhint' }, 'arrow keys step, Esc closes'), next), ); @@ -82,14 +82,14 @@ function show() { teardownFrame(); const stage = dlg.querySelector('.stage'), box = stage.querySelector('.stage-box'); clear(box); - dlg.querySelector('#viewer-title').textContent = `${c.testName || `Test ${c.test}`} by ${c.short}`; - dlg.querySelector('.vpos').textContent = `${state.index + 1} of ${state.cards.length} playable`; + dlg.querySelector('#viewer-title').textContent = `${c.short} · ${c.setLabel || set0(c.live)}`; + dlg.querySelector('.vpos').textContent = `${state.index + 1} of ${state.cards.length}`; fillSide(dlg.querySelector('.vside'), c); dlg.querySelectorAll('.vnav').forEach((b) => { b.disabled = state.cards.length < 2; }); stage.dataset.state = 'loading'; stage.dataset.kind = /\.svg$/.test(c.live) ? 'svg' : 'page'; // the iframe: sandbox and referrer policy first, src last (dom.js only lets a live path through on an iframe) - const f = el('iframe', { sandbox: 'allow-scripts', referrerpolicy: 'no-referrer', loading: 'lazy', title: `${c.testName || `Test ${c.test}`} by ${c.short}: the model's one-shot output, running`, tabindex: '-1' }); + const f = el('iframe', { sandbox: 'allow-scripts', referrerpolicy: 'no-referrer', loading: 'lazy', title: `${c.setLabel || set0(c.live)} by ${c.short}: the model's own output, running`, tabindex: '-1' }); f.addEventListener('load', () => { if (frame === f) stage.dataset.state = 'running'; }); frame = f; f.fit = { mode: 'fill' }; @@ -100,12 +100,12 @@ function show() { if (!dlg.contains(document.activeElement) || document.activeElement === dlg) dlg.querySelector('.vclose').focus(); } +const set0 = (live) => String(live).split('/')[1] || ''; function fillSide(side, c) { clear(side); const pmax = c.pointMax > 0 ? c.pointMax : 1; - const [set] = String(c.live).split('/').slice(1, 2); side.append( - el('div', { class: 'vmeta' }, familyChip(c.family, c.color), c.uncensored ? kindChip('uncensored') : null, kindChip(`test ${c.test} · ${c.testName || set}`), el('span', { class: 'mono case' }, c.case)), + el('div', { class: 'vmeta' }, familyChip(c.family, c.color), kindChip(`test ${c.test}`, c.kind === 'agentic' ? 'agentic' : ''), c.uncensored ? kindChip('uncensored') : null, el('span', { class: 'mono case' }, c.case)), el('div', { class: 'vscore' }, petalScore(c), el('div', { class: 'pts' }, ...(c.points.length ? c.points.map((p) => { const v = p.value || 0; return el('span', { class: v <= 0 ? 'l0' : v >= pmax ? 'l2' : 'l1' }, el('b', {}, p.name), ` ${fmt(p.value)}`); }) : [el('span', { class: 'l0' }, 'not graded')]), el('span', { class: 'total' }, `${fmt(c.total)} of ${fmt(c.max)}`))), @@ -113,11 +113,11 @@ function fillSide(side, c) { el('div', {}, el('dt', {}, 'speed'), el('dd', {}, fmt(c.tok_s, 1), el('small', {}, ' tok/s'))), el('div', {}, el('dt', {}, 'reply'), el('dd', {}, fmt(c.tokens, 0), el('small', {}, ' tokens'))), c.wall_s != null ? el('div', {}, el('dt', {}, 'wall'), el('dd', {}, fmt(c.wall_s, 0), el('small', {}, ' s'))) : null, - c.finish === 'length' ? el('div', { class: 'cut' }, el('dt', {}, 'finish'), el('dd', {}, 'hit the cap')) : null), + /length|cap/i.test(c.finish) ? el('div', { class: 'cut' }, el('dt', {}, 'finish'), el('dd', {}, c.finish === 'length' ? 'hit the cap' : c.finish)) : null), c.note ? el('p', { class: 'vnote' }, el('span', { class: 'k' }, 'grader'), c.note) : el('p', { class: 'vnote empty' }, 'No grader note.'), c.frames.length ? el('div', { class: 'vstrip', role: 'group', 'aria-label': 'Offline render frames' }, ...c.frames.map((fr) => el('figure', {}, el('img', { src: fr.url, alt: `${c.alt}, offline frame at ${fr.ms == null ? '?' : fmt(fr.ms, 0)} ms`, loading: 'lazy', decoding: 'async' }), el('figcaption', {}, fr.ms == null ? 'frame' : fr.ms >= 1000 ? `${(fr.ms / 1000).toFixed(fr.ms % 1000 ? 1 : 0)} s` : `${fr.ms} ms`)))) : null, - el('p', { class: 'vunedited', id: 'viewer-unedited' }, 'The piece on the stage is the model’s unedited one-shot output, served byte for byte and run inside a sandboxed frame. Slow, blank or broken is part of the result.'), + el('p', { class: 'vunedited', id: 'viewer-unedited' }, 'Unedited model output, byte for byte, in a sandboxed frame.'), ); } diff --git a/site/data/README.md b/site/data/README.md index dbfd90e..d902395 100644 --- a/site/data/README.md +++ b/site/data/README.md @@ -19,7 +19,9 @@ bash site/tools/render_smoke.sh # local only: renders the real file a | `oneshots/{orrery,pagoda}/grades.tsv` + `*/frames/*.webp` + the model's own file | `oneshots.orrery`, `oneshots.pagoda`: frames are copied to `site/frames/`, and the model's own file, byte for byte, to `site/live///page.html` or `orrery.svg` (`live` field) | | `site/data/curated.json` (hand-written) | `title`, `tagline`, `tests`, `findings`, `links` | -Everything that is prose on the page and *not* in this file is a section intro that stays true whatever the data says. +Everything that is prose on the page and *not* in this file is a one-line chapter lede, the method note or the security +note in the footer; they stay true whatever the data says. Long text from the file (grader notes, trade-offs, finding +details, caveats) sits behind small disclosures so the charts carry the page. The ornament in each chapter head is also drawn from the data of that chapter (runs as a constellation, orrery scores as gear teeth, pagoda scores as petals, …); it is decorative and hidden from assistive technology. @@ -36,7 +38,10 @@ as gear teeth, pagoda scores as petals, …); it is decorative and hidden from a Engine colours in the A/B chart work the same way, one per distinct `ab[].engine` value. - Scales: "n of m" numbers come from the data. Checks are shown against the largest `checks` seen for that test, one-shot points against the largest point value seen for that test, the test-3 rubric against the largest rubric - seen; `tests[].max_checks`, `tests[].max_point` and `tests[].max_rubric` override those when present. + seen; `tests[].max_checks`, `tests[].max_point` and `tests[].max_rubric` override those when present. When a test's + passing runs reach different `checks` totals (test 4: 4, 11 or 13 hidden checks per ticket) there is no single + scale and the count is shown alone. Rubric point names come from `tests[].points`, else from the parenthesised list + in `tests[].graded` when it has one name per point, else from the base set's test for an `agent-` set. - Counting: a run is counted on exactly one test card, under its `flavor` when it has one (2T) and under its `test` otherwise. In the run filter, picking a test also shows its flavors; picking a flavor shows only those. - Frames: only relative paths under `frames/` whose every segment starts with a letter, digit, `_` or `-` and ends in @@ -44,8 +49,10 @@ as gear teeth, pagoda scores as petals, …); it is decorative and hidden from a the file name (`frame-2000.webp`) is read as milliseconds for the flipbook label and alt text. The flipbook starts on the last (richest) frame and only ever steps onto frames that have really loaded; if none loads the card is blank. - Live pieces: `oneshots.*[].live` is shown as a "Play live" control only when it matches - `^live/(pagoda|orrery)/[A-Za-z0-9._-]+/(page\.html|orrery\.svg)$` and contains no `..`; anything else (an absolute - URL, `javascript:`, another extension, another set, a deeper path, `null`) gets no control. The viewer rules: + `^live/(?:(?:pagoda|agent-pagoda)//page\.html|(?:orrery|agent-orrery)//orrery\.svg)$` (`` being one + segment of `[A-Za-z0-9._-]+`) and contains no `..`; the four set names are spelled out, a pagoda set only serves + `page.html` and an orrery set only `orrery.svg`. Anything else (an absolute URL, `javascript:`, another extension, + another or a similar-looking set name, a deeper path, `null`) gets no control. The viewer rules: the piece runs in an `