Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions oneshots/orrery/grades.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,5 @@ t5b5-qwen3.8-27b-vllm qwen3.8-27b-vllm 32000 length 32000 361.3 99.3 0 0 0 0 0 0
mx22-flashnext-heretic2-t5-oneshot-orrery-r1 lcpp6-flashnext-heretic2 140000 stop 33547 1513.4 22.3 0 0 0 0 0 0 Final model-supplied standalone SVG is malformed (mismatched tag, line 77). The harness also greedily concatenated two SVG blocks/prose: that packaging error was separated by extracting the final original codeblock verbatim, which still fails XML parsing. Initial valid background-only fragment is not substituted or graded.
t5nm1-lcpp6-ornith-122b-q4kxl lcpp6-ornith-122b-q4kxl 258048 stop 29646 1940.1 15.3 1 1 0 1 0 3 Parses and draws, but planets are left at the central origin instead of translated to arm ends. Only gears visibly move; outer markers stay fixed across sealed frames. Nominal period parameters are not functioning planetary orbits. Toothed wheels turn without credible meshing; no visible planetary day/night.
t5nm1-lcpp-flint-40b-q3kxl lcpp-flint-40b-q3kxl 258048 stop 30402 299.0 101.8 1 2 0 0 1 4 Renders with detached teeth and shadow halos; motion present; orbital periods 80/38/20/40/160/300s incorrectly order inner planets; no credible meshing pairs; offset black shadows rotate incorrectly. Independent original-source/frame review.
mx22-maya1128-ds4-xs-t5-oneshot-orrery-r1 maya1128-ds4-xs 126976 stop 47380 1819.6 26.1 1 2 2 1 1 7 Score basis: 7/10 on the final code block, 0 as delivered. Final model-supplied code block (after the model's own </think> in the reply) graded verbatim. The harness packaging (first <svg> to last </svg>) joined a draft, prose and the final SVG and does not parse (line 3), so as delivered it scores 0. Final block parses and renders: four inner planets only (Mercury, Venus, Earth, Mars) on exact real period ratios (5.78, 14.76, 24, 45.14 s); arms, ring gears and planet spin move between frames; ring gears are dashed circles turning at arbitrary 12/16/20/26 s with unmeshed pinions; night half is a dark half-disc turning at a fixed 5 s, not tied to the sun; title and footer text collide with the outer brass ring. Reply is 54k chars of thinking before the model's </think> and 21k chars of final SVG. Rendered with tools/render_svg.sh, final-block/frames.
mx22-maya1128-ds4-xs-r2-t5-oneshot-orrery-r1 maya1128-ds4-xs 126976 stop 31986 1234.3 26.0 2 2 2 1 2 9 Score basis: 9/10 as delivered. Reply leak did not recur: no </think> in the reply, one HTML document, one SVG, and the harness cut parses (meta.static.parses true). Scored orrery.svg as delivered, frames in frames/. Six planets at exact ratios with Earth = 8 s (1.9264, 4.9216, 8, 15.048, 94.896, 235.656 s); Uranus and Neptune absent. Arms, toothed gears and planet surfaces all move. Gears are 24-tooth symbols but drive periods are 6 to 11 s against a 28 s sun gear, so they do not obey the gear law or drive the orbits (D1). Night gradient is dark on local +x, which stays away from the sun while the surface spins under it (E2). reasoning.txt is still cut (100796 chars, ends mid-token).
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
{
"case": "mx22-maya1128-ds4-xs-r2-t5-oneshot-orrery-r1",
"finish_reason": "stop",
"first_token_s": 4.1,
"max_tokens": 126976,
"model": "maya1128-ds4-xs",
"reasoning_chars": 100796,
"reply_chars": 23352,
"started": "2026-10-26T18:51:46-0500",
"static": {
"svg_found": true,
"parses": true,
"bytes": 21587,
"animateTransform": 19,
"animate": 7,
"animateMotion": 0,
"keyframes": 0,
"script": 0,
"prose_chars_outside_svg": 1765
},
"status": "ok",
"test": "5",
"timings": {
"cache_n": 0,
"prompt_n": 80,
"prompt_ms": 4070.9,
"prompt_per_token_ms": 50.886,
"prompt_per_second": 19.7,
"predicted_n": 31986,
"predicted_ms": 1230218.8,
"predicted_per_token_ms": 38.461,
"predicted_per_second": 26.0,
"draft_n": 25569,
"draft_n_accepted": 16433
},
"tok_s": 26.0,
"usage": {
"prompt_tokens": 80,
"completion_tokens": 31986,
"total_tokens": 32066,
"prompt_tokens_details": {
"cached_tokens": 0
}
},
"wall_s": 1234.3
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"case": "mx22-maya1128-ds4-xs-t5-oneshot-orrery-r1",
"finish_reason": "stop",
"first_token_s": 4.1,
"max_tokens": 126976,
"model": "maya1128-ds4-xs",
"reasoning_chars": 98010,
"reply_chars": 75809,
"started": "2026-10-26T02:40:50-0500",
"status": "ok",
"test": "5",
"timings": {
"cache_n": 0,
"prompt_n": 80,
"prompt_ms": 4076.9,
"prompt_per_token_ms": 50.961,
"prompt_per_second": 19.6,
"predicted_n": 47380,
"predicted_ms": 1815553.4,
"predicted_per_token_ms": 38.319,
"predicted_per_second": 26.1,
"draft_n": 39271,
"draft_n_accepted": 25900
},
"tok_s": 26.1,
"usage": {
"prompt_tokens": 80,
"completion_tokens": 47380,
"total_tokens": 47460,
"prompt_tokens_details": {
"cached_tokens": 0
}
},
"wall_s": 1819.6
}
Loading
Loading