diff --git a/data/README.md b/data/README.md index 5d4dd65b..d57127b6 100644 --- a/data/README.md +++ b/data/README.md @@ -29,6 +29,12 @@ The benchmark input fixtures (silhouette + footprint per device) are `trajectories/` (cohort-pinned to the audited model set; `--all` widens to later runs the trajectories also carry). +`training/qwen-rl-v2/` is the validated evidence for the released model +([`qpaig-mit/pixcell`](https://huggingface.co/qpaig-mit/pixcell), a GRPO-trained +LoRA for Qwen3.6-35B-A3B): the F1–F8 benchmark records and the +attempt–measure–revise deployment-loop record with every champion program; +the render gallery regenerates from it (`rl/track_a/make_gallery.py`). Its +README maps each number to its file. `training/representation-curriculum-v1/` preserves superseded Qwen and Inkling diagnostic runs. Those runs used the retired v1 direct prompt contract and are not canonical baselines or paper results. Their protocol and artifacts remain @@ -135,10 +141,12 @@ Every number in the paper's Results, mapped to its backing file. ### Sec V — verifier-derived training -The repository contains the training recipes under `rl/` and the derived -paper table and gallery under `arxiv/`. The full benchmark-attempt and -training-run ledgers are reserved for the companion training-artifact -release, as stated in the paper's data-availability paragraph. +The repository contains the training recipes under `rl/`, the derived +paper table and gallery under `arxiv/`, and the validated evidence for the +released model under `training/qwen-rl-v2/` (benchmark JSONs, deployment-loop +record, champion programs). Raw step-level training ledgers and +checkpoints stay outside the repository; the released adapter itself is +[`qpaig-mit/pixcell`](https://huggingface.co/qpaig-mit/pixcell). ## Curation notes (read before reusing) diff --git a/data/training/qwen-rl-v2/README.md b/data/training/qwen-rl-v2/README.md new file mode 100644 index 00000000..e465d019 --- /dev/null +++ b/data/training/qwen-rl-v2/README.md @@ -0,0 +1,38 @@ +# Qwen RL v2 — evidence for the released model + +The validated record behind the released PixCell model +([`qpaig-mit/pixcell`](https://huggingface.co/qpaig-mit/pixcell)): a LoRA +adapter for `Qwen/Qwen3.6-35B-A3B` trained with GRPO alone (no SFT) on the +released curriculum L0→L4 at the non-thinking 4096-token operating point — +Run B of the campaign recipe in [`rl/`](../../../rl/README.md). Checkpoint of +record: `tinker://d8269c70-0dd9-5776-9cae-53e8b24e647a:train:0/sampler_weights/final`. + +Per the repository rule, every number here was recomputed from artifacts by +the deterministic evaluator; nothing model-reported enters the record. + +## Files + +| Path | Protocol | Headline | +|---|---|---| +| `bench/base-nothink.json` | F1–F8, 8 attempts @ T=1.0, best-of-8 raw IoU | base model: 0/64 attempts executable | +| `bench/b-final.json` | same | released model: 39/64 executable, mean IoU 0.228, best-of-8 0.467 | +| `bench/a-post-l1.json`, `bench/d-post-l0.json` | same | mid-campaign controls (L0-SFT arm after L1; thinking arm after L0) | +| `agentloop/b-final.json` | attempt–measure–revise loop: 8 attempts, then 3 rounds × 4 revisions of the champion under number-free feedback | **mean champion IoU 0.491**; includes every champion program and per-round trajectory | + +The agent loop is the deployment protocol — the released model is the +searcher inside that loop, and 0.491 is its number. The single-pass benchmark +is the ablation that shows what a bare forward pass gives up. + +## Recompute + +```bash +uv run python -m rl.track_a.benchmark_eval --model-path --out bench.json +uv run python -m rl.track_a.agent_loop --model-path --out agentloop.json +uv run python -m rl.track_a.make_gallery --champions agentloop.json --out-dir gallery/ +``` + +The render gallery (per device: target | champion render | difference map) +is not stored here: `make_gallery` re-executes each stored champion program +from source and re-renders it against the frozen `data/benchmark` fixtures, +so it regenerates from this record alone, no checkpoint required. The +composite ships on the [model card](https://huggingface.co/qpaig-mit/pixcell). diff --git a/data/training/qwen-rl-v2/agentloop/b-final.json b/data/training/qwen-rl-v2/agentloop/b-final.json new file mode 100644 index 00000000..62b26d7c --- /dev/null +++ b/data/training/qwen-rl-v2/agentloop/b-final.json @@ -0,0 +1,281 @@ +{ + "model_path": "tinker://d8269c70-0dd9-5776-9cae-53e8b24e647a:train:0/sampler_weights/final", + "init_attempts": 8, + "revisions_per_round": 4, + "rounds": 3, + "per_target": { + "final_1": { + "champion_iou": 0.518288533514801, + "champion_code": "import gdsfactory as gf\n\n# Physical footprint: [40.0 um x, 1.7 um y]\n# \n# Looking at the image more carefully:\n# - There's a thin horizontal bar in the CENTER that extends from left to about 75-80% of the way\n# - Above and below it are two thicker strips that start parallel to center\n# but curve outward (away from center) toward the right side\n# - The center element is very thin and doesn't extend fully to the right\n# - The top and bottom elements curve away and extend to the far right\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n # Parameters derived from footprint\n total_length = 40.0\n total_height = 1.7\n half_height = total_height / 2\n \n # Center waveguide is very thin\n center_width = 0.1\n \n # Top and bottom elements are thicker\n side_width = 0.35\n \n # Gap parameters\n center_to_side_gap = 0.12\n \n # Layer\n layer = (1, 0)\n \n # Calculate positions\n center_y_offset = center_width / 2\n \n # Top waveguide path: starts at left aligned with above center, \n # runs straight for most of length, then curves upward away from center\n # Center ends at about 80% of total length\n center_end_x = total_length * 0.78\n \n # Top element\n top_y_start = center_y_offset + center_to_side_gap + center_width/2\n top_y_end_offset = half_height - side_width/2\n \n pts_top = [\n (0, top_y_start),\n (25, top_y_start),\n (30, top_y_start + 0.3),\n (35, top_y_end_offset - 0.1),\n (40, top_y_end_offset),\n ]\n \n path_top = gf.Path(pts_top)\n top_comp = gf.path.extrude(p=path_top, cross_section=gf.cross_section.cross_section(width=side_width, layer=layer))\n ref_top = c << top_comp\n \n # Bottom element (mirror of top)\n pts_bottom = [\n (0, -top_y_start),\n (25, -top_y_start),\n (30, -(top_y_start + 0.3)),\n (35, -(top_y_end_offset - 0.1)),\n (40, -top_y_end_offset),\n ]\n \n path_bottom = gf.Path(pts_bottom)\n bottom_comp = gf.path.extrude(p=path_bottom, cross_section=gf.cross_section.cross_section(width=side_width, layer=layer))\n ref_bottom = c << bottom_comp\n \n # Center waveguide\n center_comp = gf.components.straight(\n length=center_end_x,\n cross_section=gf.cross_section.cross_section(width=center_width, layer=layer)\n )\n ref_center = c << center_comp\n \n # Position center waveguide so its top edge is at center_y_offset\n # The straight component is centered by default or we can adjust\n ref_center.move((0, -center_width/2))\n \n return c\n\nif __name__ == \"__main__\":\n device().write_gds(\"device.gds\")", + "footprint_um": [ + 40.0, + 1.7 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.21934628548018628, + "champion_iou": 0.21934628548018628 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.38960463701028564, + "champion_iou": 0.38960463701028564 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.518288533514801, + "champion_iou": 0.518288533514801 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.4847074378015773, + "champion_iou": 0.518288533514801 + } + ] + }, + "final_2": { + "champion_iou": 0.5073776561537096, + "champion_code": "import gdsfactory as gf\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n # The silhouette shows:\n # - A main rectangular body (centered at origin in y)\n # - 4 vertical prongs extending upward from the top surface\n # - 1 vertical prong extending downward from the bottom center\n # \n # Footprint: 25um width x 10um height total\n \n # Key dimensions:\n # Total height = 10 um\n # Let body span from y=-2.5 to y=2.5 (height 5um)\n # Then prongs extend symmetrically: +3.0 up, +2.0 down from body edges? \n # Actually: 5 (body) + a (top extension) + b (bottom extension) = 10\n # So: a + b = 5\n \n # Looking at proportions visually:\n # Top prongs appear longer than bottom prong\n # Let bottom prong extend 1.5um down, top prongs extend 5.0-1.5=3.5um up\n # Check: 5 + 3.5 + 1.5 = 10.0 \u2713\n \n body_height = 5.0\n top_extension = 3.5\n bottom_extension = 1.5\n \n # Width and layout:\n body_width = 25.0\n \n # 4 prongs on top, evenly spaced\n prong_width = 2.0\n # 4 prongs with 3 gaps between them\n # Prong centers at: -7.5, -2.5, 2.5, 7.5 (centered around 0)\n prong_centers = [-7.5, -2.5, 2.5, 7.5]\n \n # Body centered at origin\n body = gf.components.rectangle(\n size=(body_width, body_height),\n layer=(1, 0),\n centered=True\n )\n body_ref = c << body\n \n # Bottom prong - centered at bottom of body, pointing down\n bottom_prong = gf.components.rectangle(\n size=(4.0, bottom_extension),\n layer=(1, 0),\n centered=False\n )\n bottom_ref = c << bottom_prong\n # Position: bottom prong's top should align with body's bottom\n # Body bottom: y = -2.5\n # Bottom prong needs to extend from y=-2.5 to y=-2.5-1.5=-4.0\n bottom_ref.move((0 - 2.0, -2.5))\n \n # Top prongs - extend upward from top of body\n for cx in prong_centers:\n top_prong = gf.components.rectangle(\n size=(prong_width, top_extension),\n layer=(1, 0),\n centered=False\n )\n top_ref = c << top_prong\n # Top prong should start at body top (y=2.5) and extend up to y=2.5+3.5=6.0\n top_ref.move((cx - prong_width/2, 2.5))\n \n return c\n\nif __name__ == \"__main__\":\n device().write_gds(\"device.gds\")", + "footprint_um": [ + 25, + 10 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.49668337732899176, + "champion_iou": 0.49668337732899176 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.4711650909592465, + "champion_iou": 0.49668337732899176 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.49789168188747335, + "champion_iou": 0.49789168188747335 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.5073776561537096, + "champion_iou": 0.5073776561537096 + } + ] + }, + "final_3": { + "champion_iou": 0.33601121463342937, + "champion_code": "import gdsfactory as gf\n\n# Device parameters\ntotal_length = 120 # um\ntotal_height = 4 # um\nwaveguide_width = 0.5 # um\ngap = 0.2 # gap between the two waveguides\n\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n xs = gf.cross_section.cross_section(width=waveguide_width, layer=(1, 0))\n \n # The device consists of two parallel waveguides that both have S-bends\n # on both ends, going from a narrower section in the middle to wider sections\n # at the ends.\n \n # Based on the image:\n # - Left side: both waveguides curve outward (away from center)\n # - Right side: both waveguides curve outward (away from center)\n # - Center: both waveguides run parallel and closer together\n \n # Actually looking more carefully at the silhouette:\n # The waveguides appear to have S-shaped curves on both ends\n # On the left: top goes down, bottom goes up (approaching each other)\n # In the middle: they run parallel close together\n # On the right: top goes up, bottom goes down (moving apart)\n \n # Let me create two paths for the waveguides\n \n # Define the y-coordinates for the center (close together)\n center_y_top = gap/2 + waveguide_width/2\n center_y_bottom = -gap/2 - waveguide_width/2\n \n # Define the y-coordinates for the wide ends\n end_separation = total_height/2\n end_y_top = end_separation\n end_y_bottom = -end_separation\n \n # Top waveguide path: starts at left end (wide), goes to center, then to right end (wide)\n top_path = gf.path.smooth(\n points=[\n (0, end_y_top),\n (total_length/4, center_y_top),\n (total_length*3/4, center_y_top),\n (total_length, end_y_top)\n ],\n radius=10\n )\n \n # Bottom waveguide path: starts at left end (wide), goes to center, then to right end (wide)\n bottom_path = gf.path.smooth(\n points=[\n (0, end_y_bottom),\n (total_length/4, center_y_bottom),\n (total_length*3/4, center_y_bottom),\n (total_length, end_y_bottom)\n ],\n radius=10\n )\n \n # Extrude both paths\n top_wg = c << gf.path.extrude(p=top_path, cross_section=xs)\n bottom_wg = c << gf.path.extrude(p=bottom_path, cross_section=xs)\n \n return c\n\nif __name__ == \"__main__\":\n d = device()\n d.write_gds(\"device.gds\")", + "footprint_um": [ + 120, + 4 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.15880950363708984, + "champion_iou": 0.15880950363708984 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.23123556406976534, + "champion_iou": 0.23123556406976534 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.1896243151629428, + "champion_iou": 0.23123556406976534 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.33601121463342937, + "champion_iou": 0.33601121463342937 + } + ] + }, + "final_4": { + "champion_iou": 0.6509756513451003, + "champion_code": "import gdsfactory as gf\n\n# Physical footprint parameters\nX_LENGTH = 18.0\nY_WIDTH = 3.0\nWAVE_AMPLITUDE = 1.0\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n xs = gf.cross_section.cross_section(width=1.0, layer=(1, 0))\n \n # Create upper wave path\n points_upper = [(0, Y_WIDTH/2), (X_LENGTH/2, 0), (X_LENGTH, Y_WIDTH/2)]\n path_upper = gf.path.smooth(points_upper, radius=4.0)\n comp_upper = gf.path.extrude(path_upper, cross_section=xs)\n \n # Create lower wave path \n points_lower = [(0, -Y_WIDTH/2), (X_LENGTH/2, 0), (X_LENGTH, -Y_WIDTH/2)]\n path_lower = gf.path.smooth(points_lower, radius=4.0)\n comp_lower = gf.path.extrude(path_lower, cross_section=xs)\n \n c << comp_upper\n c << comp_lower\n \n return c\n\nif __name__ == \"__main__\":\n d = device()\n d.write_gds(\"device.gds\")", + "footprint_um": [ + 18, + 3 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.6509756513451003, + "champion_iou": 0.6509756513451003 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.3476650219382819, + "champion_iou": 0.6509756513451003 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.0, + "champion_iou": 0.6509756513451003 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.0, + "champion_iou": 0.6509756513451003 + } + ] + }, + "final_5": { + "champion_iou": 0.29255670007740125, + "champion_code": "import gdsfactory as gf\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n # Define parameters based on footprint\n width = 0.7\n total_length = 18.24\n \n # Cross-section for waveguides\n xs = gf.cross_section.cross_section(width=width, layer=(1, 0))\n \n # Calculate dimensions for U-shapes\n bend_radius = 0.35\n straight_length = (total_length - 2 * bend_radius) / 2\n \n # Build top U-shape (facing down)\n path_top = gf.Path()\n path_top.append(gf.path.straight(length=straight_length))\n path_top.append(gf.path.arc(radius=bend_radius, angle=180))\n path_top.append(gf.path.straight(length=straight_length))\n path_top.append(gf.path.arc(radius=bend_radius, angle=180))\n path_top.append(gf.path.straight(length=straight_length))\n \n wg_top = gf.path.extrude(p=path_top, cross_section=xs)\n ref_top = c << wg_top\n \n # Build bottom U-shape (facing up)\n path_bottom = gf.Path()\n path_bottom.append(gf.path.straight(length=straight_length))\n path_bottom.append(gf.path.arc(radius=bend_radius, angle=-180))\n path_bottom.append(gf.path.straight(length=straight_length))\n path_bottom.append(gf.path.arc(radius=bend_radius, angle=-180))\n path_bottom.append(gf.path.straight(length=straight_length))\n \n wg_bottom = gf.path.extrude(p=path_bottom, cross_section=xs)\n ref_bottom = c << wg_bottom\n # Offset bottom shape to face the top shape\n vertical_offset = -width - 0.2\n ref_bottom.move((0, vertical_offset))\n \n # Add straight bridge/connector in the middle\n bridge_width = 0.15\n bridge_xs = gf.cross_section.cross_section(width=bridge_width, layer=(1, 0))\n \n bridge_path = gf.path.straight(length=total_length)\n wg_bridge = gf.path.extrude(p=bridge_path, cross_section=bridge_xs)\n ref_bridge = c << wg_bridge\n ref_bridge.move((0, -width/2 - 0.1))\n \n # Add gate structures (the vertical rectangular patterns along the center sections)\n gate_count = 60\n gate_width = 0.04\n gate_height = 0.08\n gate_spacing = straight_length / gate_count\n \n for i in range(gate_count):\n x_pos = straight_length/2 + i * gate_spacing\n \n # Create vertical gate structure\n gate = gf.components.rectangle(\n size=(gate_width, gate_height),\n layer=(1, 0),\n centered=True\n )\n ref_gate = c << gate\n \n # Position gate across the gap between strands\n gate_y = -width/2 + vertical_offset/2\n ref_gate.move((x_pos, gate_y))\n \n # Add solid rectangles for the tip regions (no gates in the bent areas)\n tip_rect_xs = rectangle_xs = gf.cross_section.cross_section(width=width, layer=(1, 0))\n\n return c\n\nif __name__ == \"__main__\":\n device().write_gds(\"device.gds\")", + "footprint_um": [ + 18.24, + 0.7 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.2527577514625366, + "champion_iou": 0.2527577514625366 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.24568477234461314, + "champion_iou": 0.2527577514625366 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.2872689688347614, + "champion_iou": 0.2872689688347614 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.29255670007740125, + "champion_iou": 0.29255670007740125 + } + ] + }, + "final_6": { + "champion_iou": 0.715876929571249, + "champion_code": "import gdsfactory as gf\n\n# Device parameters\nlength = 2.35\nwidth = 0.85\nlayer = (1, 0)\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n # The shape appears to be a cross with rounded/elliptical arms and rectangular stubs\n # Looking at the silhouette, it has 4 folded lobes meeting at center with straight extensions\n \n # Create the cross shape using the cross primitive with appropriate dimensions\n # The footprint is 4.7 x 4.7, so half is 2.35\n # The cross should have length matching the full extent\n \n # Using cross primitive with length=4.7, width=0.85 to get the right footprint\n cross = gf.components.cross(length=4.7, width=width, layer=layer)\n c << cross\n \n return c\n\nif __name__ == \"__main__\":\n dev = device()\n dev.write_gds(\"device.gds\")", + "footprint_um": [ + 4.7, + 4.7 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.715876929571249, + "champion_iou": 0.715876929571249 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.5648200901214198, + "champion_iou": 0.715876929571249 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.715876929571249, + "champion_iou": 0.715876929571249 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.715876929571249, + "champion_iou": 0.715876929571249 + } + ] + }, + "final_7": { + "champion_iou": 0.4894129134583224, + "champion_code": "import gdsfactory as gf\n\n# Declare tunable parameters\nfootprint_x = 3.0 # micrometers\nfootprint_y = 5.5 # micrometers\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n # Overall footprint: 3.0 x 5.5 um\n fw = footprint_x # x-dimension\n fh = footprint_y # y-dimension\n \n # Create outer ring\n outer_rect = gf.components.rectangle(\n size=(fw, fh), \n layer=(1, 0), \n centered=True\n )\n \n # Inner hole - asymmetric to match the image proportions\n # The ring is thicker horizontally than vertically\n inner_width = fw * 0.72\n inner_height = fh * 0.15\n \n inner_rect = gf.components.rectangle(\n size=(inner_width, inner_height), \n layer=(1, 0), \n centered=True\n )\n \n outer_ring = gf.boolean(outer_rect, inner_rect, operation=\"not\", layer=(1, 0))\n c << outer_ring\n \n # Central structure: two parallel horizontal bars\n # Each bar has two rectangular segments connected in the middle\n \n # Bar dimensions\n bar_width = fh * 0.13 # vertical thickness of each horizontal bar\n \n # Width of each rectangular segment\n seg_width = fw * 0.14\n seg_height = bar_width\n \n # Spacing between the two segments\n seg_gap = fw * 0.04\n \n # Vertical offset of bars from center\n # Two bars: one above, one below center\n bar_vertical_offset = fh * 0.10\n \n # Position the left end of the composite structure\n total_segment_span = 2 * seg_width + seg_gap\n left_x_start = -total_segment_span / 2\n \n # Top bar - two rectangles\n top_y = bar_vertical_offset + seg_height / 2 # top of top bar\n \n # Top left rectangle\n tl_rect = gf.components.rectangle(\n size=(seg_width, seg_height), \n layer=(1, 0), \n centered=False\n )\n tl_ref = c << tl_rect\n tl_ref.move((left_x_start + seg_width/2, top_y))\n \n # Top right rectangle \n tr_rect = gf.components.rectangle(\n size=(seg_width, seg_height), \n layer=(1, 0), \n centered=False\n )\n tr_ref = c << tr_rect\n tr_ref.move((left_x_start + seg_width + seg_gap + seg_width/2, top_y))\n \n # Bottom bar - two rectangles\n bottom_y = -bar_vertical_offset - seg_height/2\n \n # Bottom left rectangle\n bl_rect = gf.components.rectangle(\n size=(seg_width, seg_height), \n layer=(1, 0), \n centered=False\n )\n bl_ref = c << bl_rect\n bl_ref.move((left_x_start + seg_width/2, bottom_y))\n \n # Bottom right rectangle\n br_rect = gf.components.rectangle(\n size=(seg_width, seg_height), \n layer=(1, 0), \n centered=False\n )\n br_ref = c << br_rect\n br_ref.move((left_x_start + seg_width + seg_gap + seg_width/2, bottom_y))\n \n # Add connecting bars between the segments (thin vertical connectors)\n connector_width = seg_gap * 0.3\n connector_height = seg_width * 0.6\n \n # Top bar center connector\n top_center_x = left_x_start + seg_width + seg_gap/2\n top_center_y = bar_vertical_offset\n \n top_conn = gf.components.rectangle(\n size=(connector_width, connector_height), \n layer=(1, 0), \n centered=True\n )\n top_conn_ref = c << top_conn\n top_conn_ref.move((top_center_x, top_center_y))\n \n # Bottom bar center connector\n bottom_center_x = left_x_start + seg_width + seg_gap/2\n bottom_center_y = -bar_vertical_offset\n \n bottom_conn = gf.components.rectangle(\n size=(connector_width, connector_height), \n layer=(1, 0), \n centered=True\n )\n bottom_conn_ref = c << bottom_conn\n bottom_conn_ref.move((bottom_center_x, bottom_center_y))\n \n return c\n\nif __name__ == \"__main__\":\n device = device()\n device.write_gds(\"device.gds\")", + "footprint_um": [ + 3.0, + 5.5 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.489097544693976, + "champion_iou": 0.489097544693976 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.4544202119971335, + "champion_iou": 0.489097544693976 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.43631382411582176, + "champion_iou": 0.489097544693976 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.4894129134583224, + "champion_iou": 0.4894129134583224 + } + ] + }, + "final_8": { + "champion_iou": 0.4175514291117344, + "champion_code": "import gdsfactory as gf\n\n@gf.cell\ndef device():\n c = gf.Component()\n \n # Analyze the target image carefully:\n # - Top part: A U-shape or rectangular frame with 5 horizontal white lines cutting through the top portion\n # The white lines appear as gaps in the black material\n # - Middle: 5 separate horizontal black lines spanning the width\n # - Bottom: A U-shape or rectangular frame with 5 horizontal white lines cutting through the bottom portion\n # (similar to top but inverted)\n \n # BLACK = material on layer (1,0)\n # WHITE = background (no material)\n \n # The white lines inside the top and bottom blocks mean those regions have NO material on layer (1,0)\n # So I need to construct the black regions, avoiding the white line areas\n \n # Let me define the footprint and structure:\n width_um = 120.0\n height_um = 4.0\n \n # Parameters for the design\n # Top block: solid black frame with 5 horizontal white stripes near the top\n # Middle: 5 horizontal black strips\n # Bottom block: solid black frame with 5 horizontal white stripes near the bottom\n \n # Let me break down vertically (from y=0 at bottom to y=4 at top):\n # Bottom solid region with top portion having 5 white cuts:\n # - Very bottom (y=0 to y~0.3): solid black\n # - Then 5 white horizontal lines with black between them (the white lines are gaps)\n # - Then top solid region up to some y\n \n # Actually, looking again at the image - the \"white lines\" in top and bottom blocks \n # are inside the black, meaning black is ON both sides of each white line.\n # So each white line is a narrow unfilled slot within a larger black region.\n \n # Simpler interpretation: \n # The entire device can be built from horizontal rectangular bars on layer (1,0)\n # The white areas are just empty space between black bars\n \n # Structure (y-coordinates from 0 at bottom to 4 at top):\n # Bottom region: solid black rectangle minus 5 horizontal white slots\n # Middle: 5 separate horizontal black bars\n # Top region: solid black rectangle minus 5 horizontal white slots\n \n # Let me draw it as all black regions (everything is on layer 1,0 if black, nothing if white):\n \n # Bottom rectangular block (from y=0 up to ~1.5, with 5 white horizontal gaps from y~0.5 to y~1.4)\n # This means: \n # - Solid from y=0 to y=0.5 (black)\n # - White lines at specific y positions with small height\n # - Between white lines: thin black strips\n # - Then solid from y~1.4 to y~1.5 (black)\n \n # Let me simplify by defining parameters:\n num_white_slots = 5\n slot_height = 0.08 # height of each white (unfilled) slot\n slot_gap = 0.06 # black material height between slots\n \n # Middle section height occupied by white slots\n middle_slots_height = num_white_slots * slot_height + (num_white_slots - 1) * slot_gap\n \n # Below the slots block: solid black\n below_slots = 0.5 # height of solid region below all white slots\n \n # Above the slots block in bottom region: small solid region\n above_slots_in_block = 0.3\n \n # Bottom block y-range: 0 to below_slots + middle_slots_height + above_slots_in_block\n bottom_block_upper = below_slots + middle_slots_height + above_slots_in_block\n \n # Now add the solid black rectangle for bottom (excluding white slot areas)\n # I'll add the solid parts and skip the white slot regions\n \n # Part 1: Solid bottom (y=0 to y=0.5)\n bottom_solid = gf.components.rectangle(\n size=(width_um, below_slots),\n layer=(1, 0),\n centered=False\n )\n c << bottom_solid\n \n # Part 2: Thin black strips between/around white slots in bottom block\n # Between each pair of white slots: black strip of height slot_gap\n y_current = below_slots\n for i in range(num_white_slots):\n # Black strip above this white slot (or the initial gap before first slot)\n # White slot at y_current to y_current + slot_height\n # Add black strip after white slot if not the last one\n if i < num_white_slots - 1:\n black_strip_between = y_current + slot_height\n black_strip_bottom = y_current + slot_height\n black_strip_height = slot_gap\n \n # Actually let me redo this more carefully\n \n # Let me think of it as: from y=below_slots to y=above_slots_in_block + below_slots\n # Within this range, there are num_white_slots white slots of height slot_height,\n # separated by black strips of height slot_gap\n \n # White slot positions:\n white_slot_starts = []\n y_pos = below_slots\n for i in range(num_white_slots):\n white_slot_starts.append(y_pos)\n y_pos += slot_height + slot_gap\n \n # Now fill in the black regions:\n # Bottom solid already done (0 to below_slots)\n \n # Between white slots: black strips\n for i in range(num_white_slots - 1):\n # Black strip between slot i and slot i+1\n strip_start = white_slot_starts[i] + slot_height\n strip_height = slot_gap\n strip = gf.components.rectangle(\n size=(width_um, strip_height),\n layer=(1, 0),\n centered=False\n )\n ref = c << strip\n ref.move((0, strip_start))\n \n # Top strip of bottom block (above last white slot)\n top_strip_bottom = white_slot_starts[-1] + slot_height\n top_strip_height = above_slots_in_block\n if top_strip_height > 0:\n top_strip = gf.components.rectangle(\n size=(width_um, top_strip_height),\n layer=(1, 0),\n centered=False\n )\n ref = c << top_strip\n ref.move((0, top_strip_bottom))\n \n # Now middle section: 5 separate black bars (above the white slots of bottom block)\n # Middle bars start after the bottom block gap region\n middle_start_y = white_slot_starts[-1] + slot_height + above_slots_in_block\n middle_bar_height = 0.12\n middle_bar_gap = 0.08\n \n for i in range(5): # 5 middle bars\n bar_y = middle_start_y + i * (middle_bar_height + middle_bar_gap)\n bar = gf.components.rectangle(\n size=(width_um, middle_bar_height),\n layer=(1, 0),\n centered=False\n )\n ref = c << bar\n ref.move((0, bar_y))\n \n # Now top block: similar to bottom but at top\n # Top block solid region at top + white slots at bottom of top block\n \n # Leftover space after middle bars\n total_middle_bars_height = 5 * middle_bar_height + 4 * middle_bar_gap\n after_middle = middle_start_y + total_middle_bars_height\n \n # Top block: solid region from some y to 4.0, with white slots at its bottom edge\n top_block_solid_top = 3.5 # solid region from y=3.5 to y=4.0\n \n # White slots in top block (lower part of top block)\n top_block_slots_bottom = top_block_solid_top - (num_white_slots * slot_height + (num_white_slots - 1) * slot_gap)\n \n # Black strip below top block's white slots\n if top_block_slots_bottom > 0:\n bottom_strip_height = top_block_slots_bottom - after_middle\n if bottom_strip_height > 0:\n bottom_strip_2 = gf.components.rectangle(\n size=(width_um, bottom_strip_height),\n layer=(1, 0),\n centered=False\n )\n ref = c << bottom_strip_2\n ref.move((0, after_middle))\n \n # White slots in top block - skip these (they're white/background)\n # White slot positions in top block\n top_white_start = top_block_slots_bottom\n for i in range(num_white_slots):\n # White slot from top_white_start + i*(slot_height+slot_gap) to that + slot_height\n pass # Skip, don't add material here\n \n # Between white slots in top block: black strips\n for i in range(num_white_slots - 1):\n strip_start_2 = top_white_start + i * (slot_height + slot_gap) + slot_height\n strip = gf.components.rectangle(\n size=(width_um, slot_gap),\n layer=(1, 0),\n centered=False\n )\n ref = c << strip\n ref.move((0, strip_start_2))\n \n # Solid top region (already defined to go to y=4.0)\n top_solid = gf.components.rectangle(\n size=(width_um, top_block_solid_top - top_block_slots_bottom),\n layer=(1, 0),\n centered=False\n )\n ref = c << top_solid\n ref.move((0, top_block_slots_bottom))\n \n return c\n\nif __name__ == \"__main__\":\n dev = device()\n dev.write_gds(\"device.gds\")", + "footprint_um": [ + 120.0, + 4.0 + ], + "rounds": [ + { + "round": 0, + "samples": 8, + "round_best": 0.39029444269984703, + "champion_iou": 0.39029444269984703 + }, + { + "round": 1, + "samples": 4, + "round_best": 0.3807243430304511, + "champion_iou": 0.39029444269984703 + }, + { + "round": 2, + "samples": 4, + "round_best": 0.4175514291117344, + "champion_iou": 0.4175514291117344 + }, + { + "round": 3, + "samples": 4, + "round_best": 0.3683442339322844, + "champion_iou": 0.4175514291117344 + } + ] + } + }, + "overall_mean_champion": 0.4910063784832184 +} \ No newline at end of file diff --git a/data/training/qwen-rl-v2/bench/a-post-l1.json b/data/training/qwen-rl-v2/bench/a-post-l1.json new file mode 100644 index 00000000..bf2fd2cc --- /dev/null +++ b/data/training/qwen-rl-v2/bench/a-post-l1.json @@ -0,0 +1,62 @@ +{ + "model_name": "Qwen/Qwen3.6-35B-A3B", + "model_path": "tinker://da5c0ccb-55cb-547c-aa0a-b5169801186a:train:0/sampler_weights/final", + "attempts": 8, + "temperature": 1.0, + "per_target": { + "final_1": { + "executable": 2, + "attempts": 8, + "mean_iou": 0.04667048709503287, + "best_iou": 0.30297820032053774 + }, + "final_2": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_3": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_4": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_5": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_6": { + "executable": 7, + "attempts": 8, + "mean_iou": 0.39151451399245063, + "best_iou": 0.6434092192544019 + }, + "final_7": { + "executable": 6, + "attempts": 8, + "mean_iou": 0.16893688302640955, + "best_iou": 0.45974476777216944 + }, + "final_8": { + "executable": 4, + "attempts": 8, + "mean_iou": 0.18510123992188554, + "best_iou": 0.46525473940028256 + } + }, + "overall": { + "executable": 19, + "total": 64, + "mean_iou": 0.09902789050447232, + "mean_best_of_k": 0.23392336584342394 + } +} \ No newline at end of file diff --git a/data/training/qwen-rl-v2/bench/b-final.json b/data/training/qwen-rl-v2/bench/b-final.json new file mode 100644 index 00000000..c77ea038 --- /dev/null +++ b/data/training/qwen-rl-v2/bench/b-final.json @@ -0,0 +1,62 @@ +{ + "model_name": "Qwen/Qwen3.6-35B-A3B", + "model_path": "tinker://d8269c70-0dd9-5776-9cae-53e8b24e647a:train:0/sampler_weights/final", + "attempts": 8, + "temperature": 1.0, + "per_target": { + "final_1": { + "executable": 7, + "attempts": 8, + "mean_iou": 0.24696953452861198, + "best_iou": 0.43563673082449117 + }, + "final_2": { + "executable": 8, + "attempts": 8, + "mean_iou": 0.42891641561783983, + "best_iou": 0.5225050774837938 + }, + "final_3": { + "executable": 4, + "attempts": 8, + "mean_iou": 0.0815497234827478, + "best_iou": 0.21433973354480168 + }, + "final_4": { + "executable": 5, + "attempts": 8, + "mean_iou": 0.31383181845503727, + "best_iou": 0.6632708820396994 + }, + "final_5": { + "executable": 3, + "attempts": 8, + "mean_iou": 0.06311243742627864, + "best_iou": 0.24391132501379273 + }, + "final_6": { + "executable": 3, + "attempts": 8, + "mean_iou": 0.20205123415863877, + "best_iou": 0.6433213909378293 + }, + "final_7": { + "executable": 2, + "attempts": 8, + "mean_iou": 0.10999817415811014, + "best_iou": 0.48292264524221046 + }, + "final_8": { + "executable": 7, + "attempts": 8, + "mean_iou": 0.3748545833063247, + "best_iou": 0.5264526994671637 + } + }, + "overall": { + "executable": 39, + "total": 64, + "mean_iou": 0.22766049014169865, + "mean_best_of_k": 0.4665450605692228 + } +} \ No newline at end of file diff --git a/data/training/qwen-rl-v2/bench/base-nothink.json b/data/training/qwen-rl-v2/bench/base-nothink.json new file mode 100644 index 00000000..59c096b2 --- /dev/null +++ b/data/training/qwen-rl-v2/bench/base-nothink.json @@ -0,0 +1,62 @@ +{ + "model_name": "Qwen/Qwen3.6-35B-A3B", + "model_path": null, + "attempts": 8, + "temperature": 1.0, + "per_target": { + "final_1": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_2": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_3": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_4": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_5": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_6": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_7": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_8": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + } + }, + "overall": { + "executable": 0, + "total": 64, + "mean_iou": 0.0, + "mean_best_of_k": 0.0 + } +} \ No newline at end of file diff --git a/data/training/qwen-rl-v2/bench/d-post-l0.json b/data/training/qwen-rl-v2/bench/d-post-l0.json new file mode 100644 index 00000000..fc5d609e --- /dev/null +++ b/data/training/qwen-rl-v2/bench/d-post-l0.json @@ -0,0 +1,62 @@ +{ + "model_name": "Qwen/Qwen3.6-35B-A3B", + "model_path": "tinker://b1eeca1b-b1c1-55b8-87c7-fd19a38a3088:train:0/sampler_weights/final", + "attempts": 8, + "temperature": 1.0, + "per_target": { + "final_1": { + "executable": 1, + "attempts": 8, + "mean_iou": 0.01881755829330524, + "best_iou": 0.15054046634644191 + }, + "final_2": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_3": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_4": { + "executable": 6, + "attempts": 8, + "mean_iou": 0.37353030776040386, + "best_iou": 0.6464024496792284 + }, + "final_5": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_6": { + "executable": 5, + "attempts": 8, + "mean_iou": 0.4082847723288638, + "best_iou": 0.764634685414781 + }, + "final_7": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + }, + "final_8": { + "executable": 0, + "attempts": 8, + "mean_iou": 0.0, + "best_iou": 0.0 + } + }, + "overall": { + "executable": 12, + "total": 64, + "mean_iou": 0.10007907979782162, + "mean_best_of_k": 0.19519720018005643 + } +} \ No newline at end of file diff --git a/rl/README.md b/rl/README.md index 1996e16e..0e79c210 100644 --- a/rl/README.md +++ b/rl/README.md @@ -1,59 +1,109 @@ -# PixCell training - -This directory separates measurement shared by every training approach from -the policy choices of each approach. The frozen training data remains under -[`dataset/`](../dataset/); private paper benchmarks, historical rollouts, and -model checkpoints are not part of this tree. - -## Layout - -| Path | Responsibility | -|---|---| -| [`common/`](common/) | Dataset projection, candidate execution, source checking, absolute-scale IoU measurement, output extraction, and bounded evaluation batching | -| [`evaluation/`](evaluation/) | Model-neutral task loading, sampling adapters, immutable attempt records, aggregation, and metrics-only experiment tracking | -| [`studies/`](studies/) | Versioned study protocols, task manifests, and launch entry points | -| [`track_a/`](track_a/) | Self-hostable open model trained by direct SFT, then curriculum G4 RL | -| [`track_b/`](track_b/) | Inkling-specific native TML runtime bridge used by the advanced L4 policy study | -| [`track_c/`](track_c/) | Reserved for the established third training protocol; no implementation is implied yet | -| [`requirements-lock.txt`](requirements-lock.txt) | Shared pinned Python environment | -| [`stack.json`](stack.json) | Shared runtime version evidence | - -The dependency direction is one way: - -```text -PixCell source policy -> rl.common -> rl.evaluation -> rl.studies - \-> rl.track_a +# rl/ — training the blind reconstructor + +Two packages, one training path: + +| Package | Purpose | +|---------|---------| +| [`common/`](common/) | The measurement layer: hash-verified dataset projection ([`dataset_io.py`](common/dataset_io.py), [`contracts.py`](common/contracts.py)), the versioned blind prompt ([`prompt.py`](common/prompt.py), contract `pixcell-direct-reconstruction-v3`), per-candidate Docker-isolated execution and absolute-scale scoring ([`evaluator.py`](common/evaluator.py), [`isolation.py`](common/isolation.py), [`sandbox/`](common/sandbox/)), and the rectangle baselines behind the shaped reward ([`baselines.py`](common/baselines.py), [`geometry.py`](common/geometry.py)) | +| [`track_a/`](track_a/) | The Qwen training path: reward policies, Tinker Cookbook adapters, deterministic curriculum, lean SFT/RL entry points, and the four-run campaign runner | + +The governing invariant is unchanged from the rest of the repository: **the +Python measures; it never searches.** Every reward is recomputed from +artifacts by the deterministic evaluator; no model-reported number enters +the record. + +## The reward — `shaped_v3b` + +The campaign reward is the private library's proven v3-b signal (the reward +behind the recorded learning runs), constants byte-identical: + ``` +iou_n = max(0, (iou − iou_rect) / (1 − iou_rect)) # rectangle-normalized excess IoU +dice_n = likewise +τ = 0.05 · footprint_diagonal_um +t2b = exp(−boundary_chamfer_um / τ) +teacher = max(0, t2b − t2b_rect) / max(1 − t2b_rect, 0.3) + +crash / timeout / no-gds / syntax → 0.0 +source-rejected (non-catalog geometry) → 0.0 +executable + pure, iou_rect < 0.95 → 0.05 + 0.40·iou_n + 0.15·dice_n + 0.40·teacher +executable + pure, iou_rect ≥ 0.95 → 0.05 + 0.40·iou + 0.15·dice + 0.40·t2b +``` + +`iou_rect`/`dice_rect` are what a solid rectangle filling the target's +calibrated ink bounding box scores — precomputed once per task +(`common/baselines.py`) and cached; the dataset is frozen, so they are +constants of the release. Normalizing against the rectangle kills the +area-fill attractor and puts every task on one scale; the 0.05 floor keeps +"valid but wrong" distinguishable from "crashed"; the boundary-chamfer +teacher pays for count/boundary progress IoU cannot see; rectangle-like +targets score raw because normalizing against a rectangle that *is* the +answer would make every program identical. + +## The four runs + +One recipe (Qwen3.6-35B-A3B, LoRA 32, GRPO 8 rollouts × 8 groups/step, +lr 1e-5, temp 1.0, KL 0, importance sampling, 30 steps per level L0→L4 with +the exact 80/20 replay cycle), crossed over ±L0-SFT and thinking budget: + +| Run | Operating point | Arm | +|-----|-----------------|-----| +| A `runA-l0sft-rl` | no-think @ 4096 tokens | L0-SFT → RL L0→L4 | +| B `runB-base-rl` | no-think @ 4096 tokens | base → RL L0→L4 | +| C `runC-think-base-rl` | thinking @ 60000 tokens | base → RL L0→L4 | +| D `runD-think-l0sft-rl` | thinking @ 60000 tokens | L0-SFT → RL L0→L4 (gated) | + +```bash +# price an operating point first (2 GRPO steps, 2 groups): +python -m rl.track_a.run --smoke nothink --confirm-spend PIXCELL_RUNS_V2 +python -m rl.track_a.run --smoke think --confirm-spend PIXCELL_RUNS_V2 + +# then each full campaign, one detached pane each: +python -m rl.track_a.run --run A --confirm-spend PIXCELL_RUNS_V2 +``` + +Each stage is a subprocess over [`track_a/train_sft.py`](track_a/train_sft.py) +/ [`track_a/train_rl.py`](track_a/train_rl.py), chained on the previous +stage's final sampler weights; re-running the command resumes the campaign. +A fixed 80-task probe (16 per level from `depth/validation`, sorted opaque +ids) is evaluated at campaign start — the base-model baseline — and after +every stage, so every point on the curve is comparable. SFT arms must beat +the base probe on L0 before RL spends anything. + +Requirements: `TINKER_API_KEY` and `WANDB_API_KEY`. Candidate programs +execute behind an AST source gate in a resource-limited subprocess by +default; set `PIXCELL_REQUIRE_ISOLATION=1` to demand the Docker tier +(image named by `PIXCELL_EVALUATOR_IMAGE`, built by +[`common/sandbox/build.py`](common/sandbox/build.py)). Run artifacts land +outside the repository (default `~/pixcell-training/runs-v2/`, override with +`--runs-root`); W&B project `pixcell-rl`. The pinned environment is +[`requirements-lock.txt`](requirements-lock.txt) / [`stack.json`](stack.json). + +## Result — the released model + +Run B ran to completion (150 GRPO steps, L0→L4) and is the released model: +[`qpaig-mit/pixcell`](https://huggingface.co/qpaig-mit/pixcell). On the +held-out F1–F8 benchmark (8 attempts at T=1.0) it scores mean IoU 0.228 and +best-of-8 0.467, from a base model that produced 0 executable attempts in 64. +Deployed as intended — inside the attempt–measure–revise loop +([`track_a/agent_loop.py`](track_a/agent_loop.py): 8 attempts, then 3 rounds +of 4 champion revisions under number-free verifier feedback) — it reaches +**mean champion IoU 0.491**. The validated evidence (benchmark records, loop +trajectories, champion programs) is versioned in +[`data/training/qwen-rl-v2/`](../data/training/qwen-rl-v2/); the render +gallery regenerates from it via [`track_a/make_gallery.py`](track_a/make_gallery.py). + +The SFT-seeded arms (Run A and every fork from its checkpoints) entered RL +with low policy entropy and collapsed terminally within ~50–60 RL steps; the +thinking pair (C/D) was still mid-campaign at release time and is not part +of this record. + +## Preflight -`rl.common` measures a program and returns typed evidence. It does not know -which model is being trained, how examples are scheduled, how measurements -become rewards, or how checkpoints are selected. Those decisions belong to -the track or frozen study that owns them. `rl.evaluation` is shared by base -models and trained checkpoints; it does not create training clients or define -policy rewards. - -Track A is the only implemented training path. Its exact model-visible -contract, dataset inputs, SFT schedule, RL recipe, spend locks, and commands -are documented in [`track_a/README.md`](track_a/README.md). - -The active direct-policy baseline protocol is -[`representation_curriculum_v2`](studies/representation_curriculum_v2/README.md). -It measures Qwen and Inkling on F1–F8 under the corrected prompt contract and -performs no training or operating-point selection. - -The active training campaign is -[`representation_training_v1`](studies/representation_training_v1/README.md). -It freezes the shared Qwen L0 checkpoint, mixed and sequential SFT branches, -Qwen curriculum-RL branches, and the bounded Inkling L4 policy-improvement -branch. All training state and sampled programs remain in an immutable -external ledger. - -[`representation_curriculum_v1`](studies/representation_curriculum_v1/README.md) -is a superseded diagnostic record. It compared zero-shot Qwen operating -points, then measured Qwen and Inkling on F1–F8. Its measurements are not -canonical baselines, and the current paid loader rejects every v1 protocol -file. The archived files remain available for audit provenance. - -Paid entry points require their explicit confirmation token and a successful -local preflight. Sample receipts and working state stay outside the repository. -W&B is an observability mirror, not the experiment record. +[`track_a/preflight.py`](track_a/preflight.py) is the zero-spend gate the +runner executes before any Tinker client exists: frozen-release digests for +both dataset layers, clean-worktree check, catalog/dataset/conductor +whitelist identity, per-row reference and source validation, prompt-leak +check, SFT token audit (hard-fails on any truncating label), ground-truth +positive control scored through `shaped_v3b`, and the Docker sandbox +self-test. diff --git a/rl/common/baselines.py b/rl/common/baselines.py new file mode 100644 index 00000000..561aef1c --- /dev/null +++ b/rl/common/baselines.py @@ -0,0 +1,70 @@ +"""Per-task rectangle baselines: the zero point of the shaped reward. + +``iou_rect``/``dice_rect`` are what a solid rectangle filling the target's +calibrated ink bounding box would score — the degenerate answer the old +campaign's policy demonstrably discovered. The shaped reward normalizes +against these so area-filling earns nothing and one reward scale means the +same thing on every task. The dataset is frozen, so a baseline is a constant +of the release; it is computed once per ``target_image_sha256`` and cached. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +import numpy as np + +from rl.common.contracts import VerifierReference +from rl.common.geometry import boundary_mask, chamfer_sym_um + + +@dataclass(frozen=True) +class TaskBaseline: + iou_rect: float + dice_rect: float + chamfer_rect_boundary_um: float | None + diag_um: float + + +def rect_baseline(reference: VerifierReference) -> TaskBaseline: + """Score the bbox-rectangle fill against the target, on the evaluator's + own calibration (one geometry for baseline and score).""" + + from rl.common.evaluator import _prepare_reference + + prepared = _prepare_reference(reference) + array = np.asarray(prepared.target) + threshold = prepared.calibration["white_threshold"] + x0, y0, x1, y1 = prepared.calibration["bbox"] + target = array < threshold + rectangle = np.zeros_like(target) + rectangle[y0:y1, x0:x1] = True + intersection = float(np.logical_and(target, rectangle).sum()) + union = float(np.logical_or(target, rectangle).sum()) + denominator = float(target.sum() + rectangle.sum()) + scale_x, scale_y = prepared.calibration["scale_px_per_um"] + um_per_px = 2.0 / (scale_x + scale_y) + chamfer_b, _, _ = chamfer_sym_um( + boundary_mask(rectangle), + boundary_mask(target), + um_per_px, + ) + length_um, width_um = prepared.calibration["footprint_um"] + return TaskBaseline( + iou_rect=intersection / union if union else 0.0, + dice_rect=2.0 * intersection / denominator if denominator else 0.0, + chamfer_rect_boundary_um=( + None if chamfer_b == float("inf") else float(chamfer_b) + ), + diag_um=float((length_um * length_um + width_um * width_um) ** 0.5), + ) + + +_CACHE: dict[str, TaskBaseline] = {} + + +def cached_rect_baseline(reference: VerifierReference) -> TaskBaseline: + key = reference.target_image_sha256 + if key not in _CACHE: + _CACHE[key] = rect_baseline(reference) + return _CACHE[key] diff --git a/rl/common/evaluator.py b/rl/common/evaluator.py index 4c0581f9..6187b51e 100644 --- a/rl/common/evaluator.py +++ b/rl/common/evaluator.py @@ -40,6 +40,7 @@ ) from .contracts import VerifierReference +from .geometry import boundary_mask, chamfer_sym_um from .isolation import ( ExecutionBoundary, candidate_identity, @@ -795,6 +796,29 @@ def evaluate( prepared.calibration, ) metrics = compute_metrics(raster, prepared.target.copy()) + raster_mask = np.asarray(raster.convert("L")) < 245 + target_mask = np.asarray(prepared.target) < 245 + scale_x, scale_y = prepared.calibration["scale_px_per_um"] + chamfer_b, _, _ = chamfer_sym_um( + boundary_mask(raster_mask), + boundary_mask(target_mask), + 2.0 / (scale_x + scale_y), + ) + _full, full_a_to_b, full_b_to_a = chamfer_sym_um( + raster_mask, target_mask, 2.0 / (scale_x + scale_y) + ) + metrics = { + **metrics, + "chamfer_boundary_um": ( + None if chamfer_b == float("inf") else float(chamfer_b) + ), + "chamfer_pred_to_target_um": ( + None if full_a_to_b == float("inf") else float(full_a_to_b) + ), + "chamfer_target_to_pred_um": ( + None if full_b_to_a == float("inf") else float(full_b_to_a) + ), + } diagnostics = _json_diagnostic( { **metrics, diff --git a/rl/common/geometry.py b/rl/common/geometry.py new file mode 100644 index 00000000..c511be22 --- /dev/null +++ b/rl/common/geometry.py @@ -0,0 +1,43 @@ +"""Distance-transform geometry over boolean masks. Measure-only. + +Ported verbatim from the private library's ``rl/reward/geometry.py`` — the +implementation behind the boundary-chamfer "teacher" term of the shaped +reward that produced the recorded learning runs. +""" + +from __future__ import annotations + +import numpy as np +from scipy.ndimage import binary_erosion, distance_transform_edt + + +def boundary_mask(mask: np.ndarray) -> np.ndarray: + """1px outline of a boolean mask: ``mask & ~erode(mask)``. + + For a 1px-thin mask, erosion is all-False, so the boundary IS the mask; + no special case needed. + """ + + return mask & ~binary_erosion(mask) + + +def chamfer_sym_um( + mask_a: np.ndarray, + mask_b: np.ndarray, + um_per_px: float, +) -> tuple[float, float, float]: + """Symmetric mean nearest-neighbor distance (µm) between two boolean masks. + + Returns ``(chamfer_um, a_to_b_um, b_to_a_um)``. An empty prediction or an + empty target returns ``(inf, inf, inf)`` — callers treat that as worst + case, never as an error. + """ + + if not mask_a.any() or not mask_b.any(): + infinite = float("inf") + return infinite, infinite, infinite + dist_to_b = distance_transform_edt(~mask_b) + dist_to_a = distance_transform_edt(~mask_a) + a_to_b = float(dist_to_b[mask_a].mean()) * um_per_px + b_to_a = float(dist_to_a[mask_b].mean()) * um_per_px + return 0.5 * (a_to_b + b_to_a), a_to_b, b_to_a diff --git a/rl/common/prompt.py b/rl/common/prompt.py index 3d96152b..75d8c659 100644 --- a/rl/common/prompt.py +++ b/rl/common/prompt.py @@ -1,16 +1,30 @@ -"""The versioned direct Phase-A prompt shared by training and evaluation.""" +"""The versioned direct Phase-A prompt shared by training and evaluation. + +v3 adds the anisotropy scale note. The dataset's model image is the +max-visibility view — per-axis magnified, NOT physical aspect. Measured over +depth/train: 50% of rows exceed 1.6x aspect distortion (p90 = 5.0x, L4 max +31.6x). Above the 1.6x threshold the prompt states the per-axis render +scale, computed ONLY from model-visible inputs (the attached image's ink +bounding box and the stated footprint) — honest input, not leakage; the +private library shipped the same note for the same reason. +""" from __future__ import annotations import hashlib +import io import json from pathlib import Path from typing import Any +import numpy as np +from PIL import Image + from rl.common.contracts import ModelObservation -CONTRACT_VERSION = "pixcell-direct-reconstruction-v2" +CONTRACT_VERSION = "pixcell-direct-reconstruction-v3" +SCALE_NOTE_ANISOTROPY_THRESHOLD = 1.6 _REPO_ROOT = Path(__file__).resolve().parents[2] _CATALOG_PATH = ( _REPO_ROOT @@ -39,6 +53,47 @@ def prompt_asset_hashes() -> dict[str, str]: } +_SCALE_NOTE_CACHE: dict[str, str] = {} + + +def scale_note(observation: ModelObservation) -> str: + """The anisotropy note, or an empty string below the threshold. + + Computed only from what the model can already see: the attached image's + ink bounding box and the stated footprint. + """ + + cached = _SCALE_NOTE_CACHE.get(observation.image_sha256) + if cached is not None: + return cached + array = np.asarray( + Image.open(io.BytesIO(observation.image_bytes)).convert("L") + ) + mask = array < 245 + ys, xs = np.where(mask) + if xs.size == 0: + note = "" + else: + bbox_w = int(xs.max()) - int(xs.min()) + 1 + bbox_h = int(ys.max()) - int(ys.min()) + 1 + length_um, width_um = observation.footprint_um + scale_x = bbox_w / length_um + scale_y = bbox_h / width_um + anisotropy = max(scale_x / scale_y, scale_y / scale_x) + if anisotropy <= SCALE_NOTE_ANISOTROPY_THRESHOLD: + note = "" + else: + note = ( + "\nNOTE: the image is NOT drawn to physical scale - it is " + f"rendered at {scale_x:.2f} px/um horizontally and " + f"{scale_y:.2f} px/um vertically. Trust the stated footprint " + "for physical dimensions, not the image's apparent aspect " + "ratio.\n" + ) + _SCALE_NOTE_CACHE[observation.image_sha256] = note + return note + + def build_prompt_text(observation: ModelObservation) -> str: """Build the only text visible to a direct image-to-code policy.""" @@ -54,7 +109,7 @@ def build_prompt_text(observation: ModelObservation) -> str: Physical footprint, where `footprint_um` is `[x-length, y-width]`: {footprint_text} - +{scale_note(observation)} --- ## Exact GDSFactory Extended Primitive Catalogue diff --git a/rl/common/tests/test_baselines.py b/rl/common/tests/test_baselines.py new file mode 100644 index 00000000..0e9b531d --- /dev/null +++ b/rl/common/tests/test_baselines.py @@ -0,0 +1,89 @@ +"""Rectangle-baseline computation: the zero point of the shaped reward.""" + +from __future__ import annotations + +import hashlib +import io +import math + +import numpy as np +import pytest +from PIL import Image + +from rl.common.baselines import cached_rect_baseline, rect_baseline +from rl.common.contracts import VerifierReference +from rl.common.geometry import boundary_mask, chamfer_sym_um + + +def _reference(mask: np.ndarray, footprint_um: tuple[float, float]) -> VerifierReference: + """PNG-encode a boolean ink mask (True = black material) as a target reference.""" + + array = np.where(mask, 0, 255).astype(np.uint8) + image = Image.fromarray(array, mode="L") + buffer = io.BytesIO() + image.save(buffer, format="PNG") + payload = buffer.getvalue() + return VerifierReference( + target_image_bytes=payload, + footprint_um=footprint_um, + target_image_sha256=hashlib.sha256(payload).hexdigest(), + ) + + +def test_solid_rectangle_target_is_degenerate() -> None: + mask = np.zeros((80, 100), dtype=bool) + mask[20:60, 20:80] = True + baseline = rect_baseline(_reference(mask, (30.0, 20.0))) + assert baseline.iou_rect == pytest.approx(1.0) + assert baseline.dice_rect == pytest.approx(1.0) + assert baseline.diag_um == pytest.approx(math.hypot(30.0, 20.0)) + + +def test_l_shape_baseline_matches_hand_computation() -> None: + mask = np.zeros((80, 100), dtype=bool) + mask[20:60, 20:40] = True # vertical bar: 40 x 20 + mask[40:60, 20:80] = True # horizontal bar: 20 x 60, overlap 20 x 20 + ink = float(mask.sum()) + bbox_area = float(40 * 60) # ink bbox: rows 20:60, cols 20:80 + baseline = rect_baseline(_reference(mask, (30.0, 20.0))) + assert baseline.iou_rect == pytest.approx(ink / bbox_area) + assert baseline.dice_rect == pytest.approx(2.0 * ink / (ink + bbox_area)) + assert baseline.chamfer_rect_boundary_um is not None + assert baseline.chamfer_rect_boundary_um > 0.0 + assert baseline.diag_um == pytest.approx(math.hypot(30.0, 20.0)) + + +def test_cache_returns_identical_object_per_sha() -> None: + mask = np.zeros((40, 40), dtype=bool) + mask[10:30, 5:35] = True + reference = _reference(mask, (10.0, 5.0)) + first = cached_rect_baseline(reference) + second = cached_rect_baseline(reference) + assert first is second + + +def test_chamfer_of_shifted_row_is_exact() -> None: + a = np.zeros((5, 5), dtype=bool) + b = np.zeros((5, 5), dtype=bool) + a[2, 1:4] = True + b[3, 1:4] = True + chamfer, a_to_b, b_to_a = chamfer_sym_um(a, b, um_per_px=2.0) + assert a_to_b == pytest.approx(2.0) + assert b_to_a == pytest.approx(2.0) + assert chamfer == pytest.approx(2.0) + + +def test_chamfer_empty_mask_is_infinite() -> None: + empty = np.zeros((4, 4), dtype=bool) + full = np.ones((4, 4), dtype=bool) + chamfer, a_to_b, b_to_a = chamfer_sym_um(empty, full, um_per_px=1.0) + assert chamfer == float("inf") and a_to_b == float("inf") and b_to_a == float("inf") + + +def test_boundary_mask_of_solid_block_is_its_outline() -> None: + mask = np.zeros((6, 6), dtype=bool) + mask[1:5, 1:5] = True + outline = boundary_mask(mask) + assert outline[1, 1] and outline[1, 4] and outline[4, 1] and outline[4, 4] + assert not outline[2, 2] and not outline[3, 3] + assert outline.sum() == 12 diff --git a/rl/common/tests/test_evaluator.py b/rl/common/tests/test_evaluator.py index 1dc8dc04..e44ecb9a 100644 --- a/rl/common/tests/test_evaluator.py +++ b/rl/common/tests/test_evaluator.py @@ -31,6 +31,8 @@ def test_ground_truth_self_scores_at_full_precision(core_row, evaluator): assert result.dice is not None and result.dice > 0.98 assert result.iou != round(result.iou, 4) assert len(result.metrics["render_sha256"]) == 64 + chamfer = result.metrics["chamfer_boundary_um"] + assert chamfer is not None and 0.0 <= chamfer < 1.0 json.dumps(result.metrics, allow_nan=False) assert all( type(value) is bool diff --git a/rl/common/tests/test_prompt_note.py b/rl/common/tests/test_prompt_note.py new file mode 100644 index 00000000..a500b156 --- /dev/null +++ b/rl/common/tests/test_prompt_note.py @@ -0,0 +1,60 @@ +"""The v3 anisotropy scale note: honest input from model-visible state only.""" + +from __future__ import annotations + +import hashlib +import io + +import numpy as np +from PIL import Image + +from rl.common.contracts import ModelObservation +from rl.common.prompt import CONTRACT_VERSION, build_prompt_text, scale_note + + +def _observation( + ink_w: int, + ink_h: int, + footprint_um: tuple[float, float], +) -> ModelObservation: + array = np.full((ink_h + 20, ink_w + 20), 255, dtype=np.uint8) + array[10 : 10 + ink_h, 10 : 10 + ink_w] = 0 + image = Image.fromarray(array, mode="L") + buffer = io.BytesIO() + image.save(buffer, format="PNG") + payload = buffer.getvalue() + return ModelObservation( + image_bytes=payload, + footprint_um=footprint_um, + image_sha256=hashlib.sha256(payload).hexdigest(), + ) + + +def test_contract_is_v3() -> None: + assert CONTRACT_VERSION == "pixcell-direct-reconstruction-v3" + + +def test_isotropic_image_gets_no_note() -> None: + # 200x100 ink for a 20x10 um footprint: 10 px/um on both axes. + observation = _observation(200, 100, (20.0, 10.0)) + assert scale_note(observation) == "" + assert "px/um" not in build_prompt_text(observation) + + +def test_distorted_image_states_both_axis_scales() -> None: + # 200x180 ink for a 20x2 um footprint: 10 px/um in x, 90 px/um in y. + observation = _observation(200, 180, (20.0, 2.0)) + note = scale_note(observation) + assert "NOT drawn to physical scale" in note + assert "10.00 px/um horizontally" in note + assert "90.00 px/um vertically" in note + assert "Trust the stated footprint" in note + assert note in build_prompt_text(observation) + + +def test_note_only_uses_model_visible_state() -> None: + observation = _observation(200, 180, (20.0, 2.0)) + note = scale_note(observation) + assert observation.image_sha256 not in note + assert "target" not in note.casefold() + assert "calibration" not in note.casefold() diff --git a/rl/evaluation/__init__.py b/rl/evaluation/__init__.py deleted file mode 100644 index f91dbeb7..00000000 --- a/rl/evaluation/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Shared, model-agnostic evaluation infrastructure.""" diff --git a/rl/evaluation/attempt_store.py b/rl/evaluation/attempt_store.py deleted file mode 100644 index d9498c6c..00000000 --- a/rl/evaluation/attempt_store.py +++ /dev/null @@ -1,869 +0,0 @@ -"""Crash-safe, immutable storage for baseline sampling attempts. - -The store deliberately lives outside the source repository. A sampling -receipt is the durable boundary: once it exists, a resumed run evaluates that -receipt rather than asking the model for another completion. -""" - -from __future__ import annotations - -import errno -import fcntl -import hashlib -import json -import math -import os -import re -import stat -import threading -import time -import uuid -from collections.abc import Callable, Mapping -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any - - -RECORD_SCHEMA_VERSION = "pixcell-baseline-attempt-record-v1" -RUN_MANIFEST_SCHEMA_VERSION = "pixcell-baseline-run-manifest-v1" -MAX_RECORD_BYTES = 64 * 1024 * 1024 - -_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") -_SHA256 = re.compile(r"^[0-9a-f]{64}$") -_GIT_SHA = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_IMMUTABLE_IMAGE = re.compile( - r"^(?:sha256:[0-9a-f]{64}|[^\s@]+@sha256:[0-9a-f]{64})$" -) -_UNIT_INTERVAL_KEYS = { - "dice", - "executable_rate", - "iou", - "pure_executable_rate", - "raw_absolute_scale_iou", - "raw_iou", - "reward", - "score", - "top_p", -} -_NONNEGATIVE_KEYS = { - "cached_prefill", - "completion_tokens", - "evaluation_seconds", - "input_tokens", - "latency_seconds", - "output_tokens", - "prompt_tokens", - "reasoning_tokens", - "sampling_seconds", - "total_tokens", - "uncached_prefill", -} -_POSITIVE_INTEGER_KEYS = { - "max_output_tokens", - "max_tokens", -} -_RUN_MANIFEST_FIELDS = { - "source_git_sha", - "protocol_sha256", - "task_set_sha256", - "prompt_sha256", - "sandbox_image_ref", - "sandbox_image_id", - "runtime", - "model_binding", -} - - -class AttemptStoreError(RuntimeError): - """Base class for attempt-ledger failures.""" - - -class UnsafeStoreRootError(AttemptStoreError): - """The requested result root overlaps the source repository.""" - - -class InvalidRecordError(AttemptStoreError): - """A record is not valid, canonical, or internally consistent.""" - - -class ImmutableRecordError(AttemptStoreError): - """A caller attempted to replace an existing immutable record.""" - - -class ManifestDriftError(ImmutableRecordError): - """A resumed arm no longer matches its frozen launch manifest.""" - - -class RunLockedError(AttemptStoreError): - """Another process already owns the nonblocking arm lock.""" - - -class RunLockRequiredError(AttemptStoreError): - """Sampling was requested without holding the arm lock.""" - - -@dataclass(frozen=True, order=True) -class RunKey: - study_id: str - hypothesis_id: str - arm_id: str - - def __post_init__(self) -> None: - _validate_identifier(self.study_id, field="study_id") - _validate_identifier(self.hypothesis_id, field="hypothesis_id") - _validate_identifier(self.arm_id, field="arm_id") - - def as_dict(self) -> dict[str, str]: - return { - "study_id": self.study_id, - "hypothesis_id": self.hypothesis_id, - "arm_id": self.arm_id, - } - - -@dataclass(frozen=True, order=True) -class AttemptKey: - study_id: str - hypothesis_id: str - arm_id: str - task_id: str - attempt_index: int - - def __post_init__(self) -> None: - RunKey(self.study_id, self.hypothesis_id, self.arm_id) - _validate_identifier(self.task_id, field="task_id") - if ( - isinstance(self.attempt_index, bool) - or not isinstance(self.attempt_index, int) - or not 1 <= self.attempt_index <= 1_000_000 - ): - raise ValueError("attempt_index must be an integer in [1, 1000000]") - - @property - def run_key(self) -> RunKey: - return RunKey(self.study_id, self.hypothesis_id, self.arm_id) - - def as_dict(self) -> dict[str, Any]: - return { - **self.run_key.as_dict(), - "task_id": self.task_id, - "attempt_index": self.attempt_index, - } - - -@dataclass(frozen=True) -class StoredRecord: - payload: dict[str, Any] - payload_sha256: str - record_sha256: str - relative_path: PurePosixPath - - -@dataclass(frozen=True) -class ReceiptResolution: - record: StoredRecord - resumed: bool - - -def _validate_identifier(value: str, *, field: str) -> None: - if not isinstance(value, str) or not _IDENTIFIER.fullmatch(value): - raise ValueError( - f"{field} must match {_IDENTIFIER.pattern!r}; got {value!r}" - ) - - -def _canonical_bytes(value: Any) -> bytes: - try: - return json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise InvalidRecordError("record is not finite JSON") from exc - - -def _sha256(value: Any) -> str: - return hashlib.sha256(_canonical_bytes(value)).hexdigest() - - -def _validate_json(value: Any, *, path: str = "$", key: str | None = None) -> None: - if value is None or isinstance(value, (str, bool)): - pass - elif isinstance(value, int): - pass - elif isinstance(value, float): - if not math.isfinite(value): - raise InvalidRecordError(f"{path} must be finite") - elif isinstance(value, list): - for index, item in enumerate(value): - _validate_json(item, path=f"{path}[{index}]") - elif isinstance(value, Mapping): - for child_key, item in value.items(): - if not isinstance(child_key, str): - raise InvalidRecordError(f"{path} contains a non-string key") - _validate_json(item, path=f"{path}.{child_key}", key=child_key) - else: - raise InvalidRecordError( - f"{path} has non-JSON type {type(value).__name__}" - ) - - if key is None: - return - if key.endswith("_sha256") and value is not None: - if not isinstance(value, str) or not _SHA256.fullmatch(value): - raise InvalidRecordError(f"{path} must be a lowercase SHA-256") - if value is not None and ( - key in _UNIT_INTERVAL_KEYS - or key.endswith(("_dice", "_fraction", "_iou", "_rate")) - ): - if ( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or not 0.0 <= float(value) <= 1.0 - ): - raise InvalidRecordError(f"{path} must be in [0, 1]") - if key == "temperature" and value is not None: - if ( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or not 0.0 <= float(value) <= 2.0 - ): - raise InvalidRecordError(f"{path} must be in [0, 2]") - if value is not None and ( - key in _NONNEGATIVE_KEYS or key.endswith("_seconds") - ): - if ( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or float(value) < 0.0 - ): - raise InvalidRecordError(f"{path} must be nonnegative") - if key in _POSITIVE_INTEGER_KEYS and value is not None: - if ( - isinstance(value, bool) - or not isinstance(value, int) - or not 1 <= value <= 1_000_000 - ): - raise InvalidRecordError( - f"{path} must be an integer in [1, 1000000]" - ) - - -def _normalize_payload(payload: Mapping[str, Any]) -> dict[str, Any]: - if not isinstance(payload, Mapping): - raise InvalidRecordError("record payload must be a JSON object") - _validate_json(payload) - # The canonical round trip also detaches the immutable record from mutable - # mappings owned by a caller. - normalized = json.loads(_canonical_bytes(payload)) - if not isinstance(normalized, dict): - raise InvalidRecordError("record payload must be a JSON object") - return normalized - - -def _json_object(raw: bytes, *, path: Path) -> dict[str, Any]: - def reject_constant(value: str) -> Any: - raise InvalidRecordError(f"{path} contains non-finite number {value}") - - def reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: - result: dict[str, Any] = {} - for key, value in pairs: - if key in result: - raise InvalidRecordError(f"{path} contains duplicate key {key!r}") - result[key] = value - return result - - try: - value = json.loads( - raw, - parse_constant=reject_constant, - object_pairs_hook=reject_duplicate_keys, - ) - except (UnicodeDecodeError, json.JSONDecodeError) as exc: - raise InvalidRecordError(f"{path} is not valid UTF-8 JSON") from exc - if not isinstance(value, dict): - raise InvalidRecordError(f"{path} must contain a JSON object") - _validate_json(value) - return value - - -def _record_document( - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], - schema_version: str = RECORD_SCHEMA_VERSION, -) -> dict[str, Any]: - normalized = _normalize_payload(payload) - base: dict[str, Any] = { - "schema_version": schema_version, - "record_type": record_type, - "key": dict(key), - "payload": normalized, - "payload_sha256": _sha256(normalized), - } - base["record_sha256"] = _sha256(base) - return base - - -def _validate_record_document( - document: Mapping[str, Any], - *, - expected_type: str, - expected_key: Mapping[str, Any], - expected_schema: str = RECORD_SCHEMA_VERSION, -) -> StoredRecord: - required = { - "schema_version", - "record_type", - "key", - "payload", - "payload_sha256", - "record_sha256", - } - if set(document) != required: - raise InvalidRecordError( - f"{expected_type} record fields differ from the frozen schema" - ) - if document["schema_version"] != expected_schema: - raise InvalidRecordError(f"unexpected {expected_type} schema version") - if document["record_type"] != expected_type: - raise InvalidRecordError(f"expected {expected_type} record") - if document["key"] != dict(expected_key): - raise InvalidRecordError(f"{expected_type} key does not match its path") - payload = document["payload"] - if not isinstance(payload, dict): - raise InvalidRecordError(f"{expected_type} payload must be an object") - payload_sha = _sha256(payload) - if document["payload_sha256"] != payload_sha: - raise InvalidRecordError(f"{expected_type} payload hash mismatch") - unsigned = dict(document) - record_sha = unsigned.pop("record_sha256") - if record_sha != _sha256(unsigned): - raise InvalidRecordError(f"{expected_type} record hash mismatch") - return StoredRecord( - payload=dict(payload), - payload_sha256=payload_sha, - record_sha256=str(record_sha), - relative_path=PurePosixPath("."), - ) - - -class _RunLock: - def __init__(self, store: AttemptStore, run_key: RunKey, path: Path) -> None: - self._store = store - self._run_key = run_key - self._path = path - self._fd: int | None = None - - def __enter__(self) -> _RunLock: - self._path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - flags = os.O_RDWR | os.O_CREAT - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(self._path, flags, 0o600) - fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - except BlockingIOError as exc: - if "fd" in locals(): - os.close(fd) - raise RunLockedError( - f"run {self._run_key} is already active" - ) from exc - except OSError: - if "fd" in locals(): - os.close(fd) - raise - self._fd = fd - try: - self._store._mark_lock_held(self._run_key) - except Exception: - fcntl.flock(fd, fcntl.LOCK_UN) - os.close(fd) - self._fd = None - raise - return self - - def __exit__(self, *_: object) -> None: - if self._fd is None: - return - self._store._mark_lock_released(self._run_key) - fcntl.flock(self._fd, fcntl.LOCK_UN) - os.close(self._fd) - self._fd = None - - -class AttemptStore: - """External, create-only attempt ledger.""" - - def __init__(self, *, repo_root: Path, external_root: Path) -> None: - self.repo_root = Path(repo_root).resolve(strict=True) - requested_root = Path(external_root).expanduser().resolve(strict=False) - if ( - requested_root == self.repo_root - or requested_root.is_relative_to(self.repo_root) - or self.repo_root.is_relative_to(requested_root) - ): - raise UnsafeStoreRootError( - "attempt output must not overlap the source repository" - ) - requested_root.mkdir(parents=True, exist_ok=True, mode=0o700) - self.external_root = requested_root.resolve(strict=True) - if ( - self.external_root == self.repo_root - or self.external_root.is_relative_to(self.repo_root) - or self.repo_root.is_relative_to(self.external_root) - ): - raise UnsafeStoreRootError( - "attempt output resolves across the source repository" - ) - self._held_locks: set[RunKey] = set() - self._lock_state_guard = threading.Lock() - - @staticmethod - def run_relative_path(run_key: RunKey) -> PurePosixPath: - return PurePosixPath( - "studies", - run_key.study_id, - "hypotheses", - run_key.hypothesis_id, - "arms", - run_key.arm_id, - ) - - @classmethod - def attempt_relative_path(cls, key: AttemptKey) -> PurePosixPath: - return cls.run_relative_path(key.run_key) / PurePosixPath( - "tasks", - key.task_id, - "attempts", - f"{key.attempt_index:06d}", - ) - - @classmethod - def run_manifest_relative_path(cls, run_key: RunKey) -> PurePosixPath: - return cls.run_relative_path(run_key) / "run_manifest.json" - - @classmethod - def sampling_receipt_relative_path(cls, key: AttemptKey) -> PurePosixPath: - return cls.attempt_relative_path(key) / "sampling_receipt.json" - - @classmethod - def evaluation_relative_path(cls, key: AttemptKey) -> PurePosixPath: - return cls.attempt_relative_path(key) / "evaluation.json" - - def _path(self, relative: PurePosixPath) -> Path: - path = self.external_root.joinpath(*relative.parts) - resolved_parent = path.parent.resolve(strict=False) - if resolved_parent != self.external_root and not resolved_parent.is_relative_to( - self.external_root - ): - raise UnsafeStoreRootError("record path escaped the external root") - return path - - def acquire_run_lock(self, run_key: RunKey) -> _RunLock: - digest = hashlib.sha256( - _canonical_bytes(run_key.as_dict()) - ).hexdigest() - return _RunLock( - self, - run_key, - self.external_root / ".locks" / f"{digest}.lock", - ) - - def _mark_lock_held(self, run_key: RunKey) -> None: - with self._lock_state_guard: - if run_key in self._held_locks: - raise RunLockedError(f"run {run_key} is already locked here") - self._held_locks.add(run_key) - - def _mark_lock_released(self, run_key: RunKey) -> None: - with self._lock_state_guard: - self._held_locks.discard(run_key) - - def _require_run_lock(self, run_key: RunKey) -> None: - with self._lock_state_guard: - if run_key not in self._held_locks: - raise RunLockRequiredError( - "sampling requires the caller to hold the arm run lock" - ) - - def create_or_verify_run_manifest( - self, - run_key: RunKey, - manifest: Mapping[str, Any], - ) -> StoredRecord: - normalized = _normalize_payload(manifest) - missing = sorted(_RUN_MANIFEST_FIELDS - set(normalized)) - if missing: - raise InvalidRecordError( - f"run manifest is missing launch bindings: {missing}" - ) - if not _GIT_SHA.fullmatch(str(normalized["source_git_sha"])): - raise InvalidRecordError("source_git_sha must be a Git object SHA") - for field in ("protocol_sha256", "task_set_sha256", "prompt_sha256"): - if not _SHA256.fullmatch(str(normalized[field])): - raise InvalidRecordError(f"{field} must be a lowercase SHA-256") - if not _IMMUTABLE_IMAGE.fullmatch( - str(normalized["sandbox_image_ref"]) - ): - raise InvalidRecordError( - "sandbox_image_ref must be an immutable image digest" - ) - if not re.fullmatch( - r"sha256:[0-9a-f]{64}", str(normalized["sandbox_image_id"]) - ): - raise InvalidRecordError( - "sandbox_image_id must be an immutable local image ID" - ) - if ( - not isinstance(normalized["model_binding"], dict) - or not normalized["model_binding"] - ): - raise InvalidRecordError("model_binding must be a non-empty object") - runtime = normalized["runtime"] - if ( - not isinstance(runtime, dict) - or not isinstance(runtime.get("python"), str) - or not runtime["python"] - or not isinstance(runtime.get("packages"), dict) - or not runtime["packages"] - or not _GIT_SHA.fullmatch( - str(runtime.get("tinker_cookbook_commit", "")) - ) - ): - raise InvalidRecordError("runtime must contain the pinned observed stack") - - expected = _record_document( - record_type="run_manifest", - key=run_key.as_dict(), - payload=normalized, - schema_version=RUN_MANIFEST_SCHEMA_VERSION, - ) - relative = self.run_manifest_relative_path(run_key) - try: - return self._create_or_verify( - relative=relative, - expected=expected, - record_type="run_manifest", - key=run_key.as_dict(), - schema_version=RUN_MANIFEST_SCHEMA_VERSION, - ) - except ImmutableRecordError as exc: - raise ManifestDriftError( - "run manifest differs from the immutable arm binding" - ) from exc - - def load_run_manifest(self, run_key: RunKey) -> StoredRecord | None: - return self._load( - relative=self.run_manifest_relative_path(run_key), - record_type="run_manifest", - key=run_key.as_dict(), - schema_version=RUN_MANIFEST_SCHEMA_VERSION, - ) - - def write_sampling_receipt( - self, - key: AttemptKey, - payload: Mapping[str, Any], - ) -> StoredRecord: - return self._write_attempt_record( - key=key, - payload=payload, - record_type="sampling_receipt", - relative=self.sampling_receipt_relative_path(key), - ) - - def load_sampling_receipt(self, key: AttemptKey) -> StoredRecord | None: - return self._load( - relative=self.sampling_receipt_relative_path(key), - record_type="sampling_receipt", - key=key.as_dict(), - ) - - def write_evaluation( - self, - key: AttemptKey, - payload: Mapping[str, Any], - ) -> StoredRecord: - if self.load_sampling_receipt(key) is None: - raise InvalidRecordError( - "evaluation cannot precede its durable sampling receipt" - ) - return self._write_attempt_record( - key=key, - payload=payload, - record_type="evaluation", - relative=self.evaluation_relative_path(key), - ) - - def load_evaluation(self, key: AttemptKey) -> StoredRecord | None: - return self._load( - relative=self.evaluation_relative_path(key), - record_type="evaluation", - key=key.as_dict(), - ) - - def write_evaluator_failure( - self, - key: AttemptKey, - payload: Mapping[str, Any], - ) -> StoredRecord: - """Preserve a verifier diagnostic without poisoning canonical results.""" - - if self.load_sampling_receipt(key) is None: - raise InvalidRecordError( - "evaluator failure cannot precede its sampling receipt" - ) - normalized = _normalize_payload(payload) - failure_id = _sha256(normalized) - relative = ( - self.attempt_relative_path(key) - / "evaluator_failures" - / f"{failure_id}.json" - ) - return self._write_attempt_record( - key=key, - payload=normalized, - record_type="evaluator_failure", - relative=relative, - ) - - def list_evaluator_failures( - self, - key: AttemptKey, - ) -> tuple[StoredRecord, ...]: - directory = self._path( - self.attempt_relative_path(key) / "evaluator_failures" - ) - if not directory.exists(): - return () - result: list[StoredRecord] = [] - for path in sorted(directory.glob("*.json")): - relative = PurePosixPath( - path.relative_to(self.external_root).as_posix() - ) - record = self._load( - relative=relative, - record_type="evaluator_failure", - key=key.as_dict(), - ) - if record is not None: - result.append(record) - return tuple(result) - - def resume_or_sample( - self, - key: AttemptKey, - sampler: Callable[[], Mapping[str, Any]], - ) -> ReceiptResolution: - existing = self.load_sampling_receipt(key) - if existing is not None: - return ReceiptResolution(record=existing, resumed=True) - self._require_run_lock(key.run_key) - if self.load_run_manifest(key.run_key) is None: - raise InvalidRecordError( - "sampling requires a frozen per-arm run manifest" - ) - # Recheck after validating ownership and the launch binding. - existing = self.load_sampling_receipt(key) - if existing is not None: - return ReceiptResolution(record=existing, resumed=True) - sampled = sampler() - record = self.write_sampling_receipt(key, sampled) - return ReceiptResolution(record=record, resumed=False) - - def write_tracking_error( - self, - key: AttemptKey, - payload: Mapping[str, Any], - ) -> StoredRecord: - normalized = _normalize_payload(payload) - normalized.setdefault("occurred_at_unix_ns", time.time_ns()) - error_id = _sha256(normalized) - relative = ( - self.attempt_relative_path(key) - / "tracking_errors" - / f"{error_id}.json" - ) - return self._write_attempt_record( - key=key, - payload=normalized, - record_type="tracking_error", - relative=relative, - ) - - def list_tracking_errors(self, key: AttemptKey) -> tuple[StoredRecord, ...]: - directory = self._path( - self.attempt_relative_path(key) / "tracking_errors" - ) - if not directory.exists(): - return () - result: list[StoredRecord] = [] - for path in sorted(directory.glob("*.json")): - relative = PurePosixPath(path.relative_to(self.external_root).as_posix()) - record = self._load( - relative=relative, - record_type="tracking_error", - key=key.as_dict(), - ) - if record is not None: - result.append(record) - return tuple(result) - - def _write_attempt_record( - self, - *, - key: AttemptKey, - payload: Mapping[str, Any], - record_type: str, - relative: PurePosixPath, - ) -> StoredRecord: - expected = _record_document( - record_type=record_type, - key=key.as_dict(), - payload=payload, - ) - return self._create_or_verify( - relative=relative, - expected=expected, - record_type=record_type, - key=key.as_dict(), - ) - - def _create_or_verify( - self, - *, - relative: PurePosixPath, - expected: Mapping[str, Any], - record_type: str, - key: Mapping[str, Any], - schema_version: str = RECORD_SCHEMA_VERSION, - ) -> StoredRecord: - path = self._path(relative) - path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - raw = _canonical_bytes(expected) + b"\n" - temporary = path.parent / ( - f".{path.name}.tmp-{os.getpid()}-{uuid.uuid4().hex}" - ) - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(temporary, flags, 0o600) - try: - with os.fdopen(fd, "wb", closefd=False) as stream: - stream.write(raw) - stream.flush() - os.fsync(stream.fileno()) - finally: - os.close(fd) - try: - os.link(temporary, path) - self._fsync_directory(path.parent) - except FileExistsError: - observed = self._load( - relative=relative, - record_type=record_type, - key=key, - schema_version=schema_version, - ) - if observed is None: - raise InvalidRecordError( - f"{path} disappeared while verifying immutability" - ) - if _canonical_bytes(expected) != _canonical_bytes( - self._read_document(path) - ): - raise ImmutableRecordError( - f"{relative} already contains different data" - ) - return observed - finally: - try: - temporary.unlink() - except FileNotFoundError: - pass - record = _validate_record_document( - expected, - expected_type=record_type, - expected_key=key, - expected_schema=schema_version, - ) - return StoredRecord( - payload=record.payload, - payload_sha256=record.payload_sha256, - record_sha256=record.record_sha256, - relative_path=relative, - ) - - def _load( - self, - *, - relative: PurePosixPath, - record_type: str, - key: Mapping[str, Any], - schema_version: str = RECORD_SCHEMA_VERSION, - ) -> StoredRecord | None: - path = self._path(relative) - if not path.exists(): - return None - document = self._read_document(path) - record = _validate_record_document( - document, - expected_type=record_type, - expected_key=key, - expected_schema=schema_version, - ) - return StoredRecord( - payload=record.payload, - payload_sha256=record.payload_sha256, - record_sha256=record.record_sha256, - relative_path=relative, - ) - - @staticmethod - def _read_document(path: Path) -> dict[str, Any]: - flags = os.O_RDONLY - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(path, flags) - except OSError as exc: - if exc.errno == errno.ELOOP: - raise InvalidRecordError(f"{path} must not be a symlink") from exc - raise - try: - metadata = os.fstat(fd) - if not stat.S_ISREG(metadata.st_mode): - raise InvalidRecordError(f"{path} is not a regular file") - if metadata.st_size > MAX_RECORD_BYTES: - raise InvalidRecordError(f"{path} exceeds the record size limit") - chunks: list[bytes] = [] - remaining = MAX_RECORD_BYTES + 1 - while remaining > 0: - chunk = os.read(fd, min(1024 * 1024, remaining)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - if len(raw) > MAX_RECORD_BYTES: - raise InvalidRecordError(f"{path} exceeds the record size limit") - finally: - os.close(fd) - return _json_object(raw, path=path) - - @staticmethod - def _fsync_directory(path: Path) -> None: - flags = os.O_RDONLY - if hasattr(os, "O_DIRECTORY"): - flags |= os.O_DIRECTORY - fd = os.open(path, flags) - try: - os.fsync(fd) - finally: - os.close(fd) diff --git a/rl/evaluation/protocol.py b/rl/evaluation/protocol.py deleted file mode 100644 index cedd8bd7..00000000 --- a/rl/evaluation/protocol.py +++ /dev/null @@ -1,508 +0,0 @@ -"""Frozen protocol loading and exact baseline-launch binding.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import os -import subprocess -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from rl.common.isolation import require_execution_boundary -from rl.common.preprocess import IMAGE_PREPROCESS_VERSION -from rl.common.prompt import CONTRACT_VERSION, prompt_asset_hashes -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.tasks import ( - canonical_json_sha256, - load_task_manifest, - load_task_set, -) - - -PROTOCOL_SCHEMA = "pixcell-representation-curriculum-protocol-v1" -STUDY_RELATIVE_PATH = Path("rl/studies/representation_curriculum_v2") -PROTOCOL_FILENAME = "protocol.json" -TASK_MANIFEST_FILENAME = "task_manifest.json" -ACTIVE_STUDY_ID = "representation-curriculum-v2" -ACTIVE_IMAGE_LONG_EDGE = 1920 -_SHA_RE = __import__("re").compile(r"^[0-9a-f]{40}$") - - -@dataclass(frozen=True) -class ArmSpec: - arm_id: str - provider: str - model: str - renderer: str - thinking: bool - thinking_effort: float | None - max_output_tokens: int - context_tokens: int - max_image_long_edge: int - - -@dataclass(frozen=True) -class JobSpec: - hypothesis_id: str - arm_id: str - task_id: str - attempt_index: int - seed: int - - -def _logical_document(document: dict[str, Any]) -> dict[str, Any]: - return {key: value for key, value in document.items() if key != "logical_sha256"} - - -def seal_protocol(path: Path) -> dict[str, Any]: - document = json.loads(path.read_text(encoding="utf-8")) - document["logical_sha256"] = canonical_json_sha256(_logical_document(document)) - path.write_text( - json.dumps(document, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - return document - - -def protocol_path(repo_root: Path, protocol_file: Path | None = None) -> Path: - root = repo_root.expanduser().resolve() - if protocol_file is None: - return root / STUDY_RELATIVE_PATH / PROTOCOL_FILENAME - candidate = protocol_file.expanduser().resolve(strict=True) - if not candidate.is_relative_to(root): - raise ValueError("baseline protocol must live inside the source repository") - return candidate - - -def task_manifest_path(repo_root: Path) -> Path: - return ( - repo_root.expanduser().resolve() - / STUDY_RELATIVE_PATH - / TASK_MANIFEST_FILENAME - ) - - -def _require_protocol_at_head(repo_root: Path, path: Path) -> None: - root = repo_root.expanduser().resolve() - candidate = path.expanduser().resolve(strict=True) - if not candidate.is_relative_to(root): - raise ValueError("baseline protocol must live inside the source repository") - relative = candidate.relative_to(root).as_posix() - try: - subprocess.check_output( - ["git", "ls-files", "--error-unmatch", "--", relative], - cwd=root, - stderr=subprocess.DEVNULL, - ) - committed = subprocess.check_output( - ["git", "show", f"HEAD:{relative}"], - cwd=root, - stderr=subprocess.DEVNULL, - ) - except (OSError, subprocess.CalledProcessError) as exc: - raise ValueError( - "baseline protocol must be tracked by the paid-launch commit" - ) from exc - if candidate.read_bytes() != committed: - raise ValueError("baseline protocol bytes differ from the paid-launch commit") - - -def _validate_arm(arm_id: str, value: dict[str, Any]) -> ArmSpec: - effort = value.get("thinking_effort") - if effort is not None: - effort = float(effort) - if not math.isfinite(effort) or not 0 <= effort < 1: - raise ValueError(f"{arm_id} has invalid thinking effort") - result = ArmSpec( - arm_id=arm_id, - provider=str(value["provider"]), - model=str(value["model"]), - renderer=str(value["renderer"]), - thinking=bool(value["thinking"]), - thinking_effort=effort, - max_output_tokens=int(value["max_output_tokens"]), - context_tokens=int(value["context_tokens"]), - max_image_long_edge=int(value["max_image_long_edge"]), - ) - if result.provider != "tinker": - raise ValueError(f"{arm_id} has unsupported provider {result.provider!r}") - if result.max_output_tokens < 1 or result.context_tokens < 1: - raise ValueError(f"{arm_id} has invalid token limits") - if result.max_output_tokens >= result.context_tokens: - raise ValueError(f"{arm_id} output cap exhausts the context window") - if result.max_image_long_edge < 1: - raise ValueError(f"{arm_id} has an invalid image-size policy") - if result.model == "thinkingmachines/Inkling": - if result.renderer != "tml_v0" or result.thinking_effort is None: - raise ValueError(f"{arm_id} has an invalid Inkling renderer contract") - elif result.model == "Qwen/Qwen3.6-35B-A3B": - expected = "qwen3_5" if result.thinking else "qwen3_5_disable_thinking" - if result.renderer != expected or result.thinking_effort is not None: - raise ValueError(f"{arm_id} has an invalid Qwen renderer contract") - else: - raise ValueError(f"{arm_id} uses an unregistered model") - return result - - -def load_protocol( - repo_root: Path, - *, - protocol_file: Path | None = None, -) -> dict[str, Any]: - """Load a protocol only when it matches the active paid-launch contract.""" - - root = repo_root.expanduser().resolve() - document = load_archived_protocol(root, protocol_file=protocol_file) - if document.get("contract_version") != CONTRACT_VERSION: - raise ValueError("protocol prompt contract does not match the runtime") - if document.get("study_id") != ACTIVE_STUDY_ID: - raise ValueError("protocol is not the active v2 baseline study") - arms = { - arm_id: _validate_arm(arm_id, value) - for arm_id, value in document["arms"].items() - } - if len(arms) != len(document["arms"]): - raise ValueError("duplicate arm IDs") - if any( - arm.max_image_long_edge != ACTIVE_IMAGE_LONG_EDGE - for arm in arms.values() - ): - raise ValueError( - "active v2 baselines require the frozen 1920px image policy" - ) - for hypothesis_id, hypothesis in document["hypotheses"].items(): - if not hypothesis_id.startswith("RC-H"): - raise ValueError(f"invalid hypothesis ID: {hypothesis_id!r}") - if hypothesis["status"] not in {"frozen", "conditional"}: - raise ValueError(f"{hypothesis_id} is not launchable") - if int(hypothesis["attempts_per_task"]) < 1: - raise ValueError(f"{hypothesis_id} has no attempts") - for arm_id in hypothesis.get("arm_ids", ()): - if arm_id not in arms: - raise ValueError(f"{hypothesis_id} names unknown arm {arm_id}") - for wave_name, wave in hypothesis["waves"].items(): - indices = [int(value) for value in wave["attempt_indices"]] - if wave_name not in {"smoke", "complete"}: - raise ValueError(f"{hypothesis_id} has unsupported wave {wave_name}") - if not indices or min(indices) < 1: - raise ValueError(f"{hypothesis_id}/{wave_name} has invalid attempts") - if max(indices) > int(hypothesis["attempts_per_task"]): - raise ValueError(f"{hypothesis_id}/{wave_name} exceeds attempt count") - if int(wave["task_count"]) < 1: - raise ValueError(f"{hypothesis_id}/{wave_name} has no tasks") - - manifest = load_task_manifest(task_manifest_path(root), repo_root=root) - expected_manifest_sha = document["task_manifest"]["logical_sha256"] - if manifest["logical_sha256"] != expected_manifest_sha: - raise ValueError("protocol does not bind the frozen task manifest") - dataset_binding = document.get("dataset") - if dataset_binding is not None: - freeze = json.loads( - (root / "dataset/depth-v1/factory/freeze.json").read_text( - encoding="utf-8" - ) - ) - if ( - freeze["logical_release_sha256"] - != dataset_binding["logical_release_sha256"] - ): - raise ValueError("protocol does not bind the local depth release") - return document - - -def load_archived_protocol( - repo_root: Path, - *, - protocol_file: Path | None = None, -) -> dict[str, Any]: - """Inspect a sealed historical protocol without making it launchable. - - This validates repository containment, schema, and the protocol's own - logical digest. Paid entry points must use :func:`load_protocol`, which - additionally requires the active model-visible contract. - """ - - root = repo_root.expanduser().resolve() - document = json.loads( - protocol_path(root, protocol_file).read_text(encoding="utf-8") - ) - if document.get("schema_version") != PROTOCOL_SCHEMA: - raise ValueError("unsupported representation-curriculum protocol schema") - observed = canonical_json_sha256(_logical_document(document)) - if document.get("logical_sha256") != observed: - raise ValueError("protocol logical SHA-256 mismatch") - return document - - -def arm_spec(protocol: dict[str, Any], arm_id: str) -> ArmSpec: - try: - value = protocol["arms"][arm_id] - except KeyError as exc: - raise ValueError(f"unknown arm: {arm_id!r}") from exc - return _validate_arm(arm_id, value) - - -def _validate_selection( - protocol: dict[str, Any], - selection: dict[str, Any], -) -> str: - if selection.get("schema_version") != "pixcell-operating-point-selection-v1": - raise ValueError("unsupported operating-point selection schema") - if selection.get("study_id") != protocol["study_id"]: - raise ValueError("operating-point selection belongs to another study") - if selection.get("protocol_sha256") != protocol["logical_sha256"]: - raise ValueError("operating-point selection has stale protocol provenance") - if selection.get("source_hypothesis") != "RC-H00": - raise ValueError("operating-point selection has the wrong source hypothesis") - selected = str(selection.get("selected_arm", "")) - allowed = set(protocol["hypotheses"]["RC-H00"]["arm_ids"]) - if selection.get("rescue_run"): - allowed.update(protocol["hypotheses"]["RC-H00R"]["arm_ids"]) - if selected not in allowed: - raise ValueError("operating-point selection names an ineligible arm") - if not selection.get("complete"): - raise ValueError("operating-point selection is not complete") - return selected - - -def resolved_arm_ids( - protocol: dict[str, Any], - hypothesis_id: str, - *, - selection: dict[str, Any] | None, -) -> list[str]: - try: - hypothesis = protocol["hypotheses"][hypothesis_id] - except KeyError as exc: - raise ValueError(f"unknown hypothesis: {hypothesis_id!r}") from exc - if hypothesis_id == "RC-H01": - if selection is None: - raise ValueError("RC-H01 requires the frozen operating-point selection") - selected = _validate_selection(protocol, selection) - incumbent = str(hypothesis["arm_roles"]["incumbent"]) - return [incumbent] if selected == incumbent else [incumbent, selected] - if hypothesis_id == "RC-H00R": - if selection is None or not selection.get("rescue_required"): - raise ValueError("RC-H00R is forbidden unless RC-H00 triggers rescue") - return [str(value) for value in hypothesis["arm_ids"]] - - -def deterministic_seed( - protocol: dict[str, Any], - *, - task_id: str, - attempt_index: int, -) -> int: - namespace = str(protocol["sampling"]["seed_namespace"]) - digest = hashlib.sha256( - f"{namespace}\0{task_id}\0{attempt_index}".encode("utf-8") - ).digest() - return int.from_bytes(digest[:4], "big") & 0x7FFFFFFF - - -def jobs_for_wave( - protocol: dict[str, Any], - *, - hypothesis_id: str, - wave_name: str, - task_ids: list[str], - selection: dict[str, Any] | None = None, -) -> list[JobSpec]: - hypothesis = protocol["hypotheses"].get(hypothesis_id) - if hypothesis is None: - raise ValueError(f"unknown hypothesis: {hypothesis_id!r}") - wave = hypothesis["waves"].get(wave_name) - if wave is None: - raise ValueError(f"{hypothesis_id} has no wave {wave_name!r}") - count = int(wave["task_count"]) - if count > len(task_ids): - raise ValueError( - f"{hypothesis_id}/{wave_name} asks for {count} of {len(task_ids)} tasks" - ) - arms = resolved_arm_ids(protocol, hypothesis_id, selection=selection) - return [ - JobSpec( - hypothesis_id=hypothesis_id, - arm_id=arm_id, - task_id=task_id, - attempt_index=int(attempt_index), - seed=deterministic_seed( - protocol, - task_id=task_id, - attempt_index=int(attempt_index), - ), - ) - for attempt_index in wave["attempt_indices"] - for task_id in task_ids[:count] - for arm_id in arms - ] - - -def _git_state(repo_root: Path) -> dict[str, Any]: - head = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo_root, - text=True, - ).strip() - status = subprocess.check_output( - ["git", "status", "--porcelain", "--untracked-files=all"], - cwd=repo_root, - text=True, - ).strip() - return {"head": head, "clean": not bool(status)} - - -def _external_output_root(repo_root: Path, value: Path) -> Path: - root = value.expanduser().resolve() - repository = repo_root.expanduser().resolve() - if ( - root == repository - or root.is_relative_to(repository) - or repository.is_relative_to(root) - ): - raise ValueError("mutable study output must not overlap the Git repository") - root.mkdir(parents=True, exist_ok=True) - if not root.is_dir(): - raise ValueError("mutable study output is not a directory") - return root - - -def build_launch_binding( - *, - repo_root: Path, - hypothesis_id: str, - wave_name: str, - expected_source_sha: str, - external_root: Path, - selection: dict[str, Any] | None = None, - protocol_file: Path | None = None, -) -> tuple[dict[str, Any], list[Any], list[JobSpec]]: - """Resolve every semantic launch input before a remote client exists.""" - - root = repo_root.expanduser().resolve() - if not _SHA_RE.fullmatch(expected_source_sha): - raise ValueError("expected_source_sha must be a full lowercase Git SHA") - state = _git_state(root) - if not state["clean"]: - raise ValueError("paid baseline launch requires a clean worktree") - if state["head"] != expected_source_sha: - raise ValueError( - f"source SHA mismatch: {state['head']} != {expected_source_sha}" - ) - selected_protocol = protocol_path(root, protocol_file) - _require_protocol_at_head(root, selected_protocol) - protocol = load_protocol(root, protocol_file=selected_protocol) - runtime = validate_runtime_stack(root) - manifest = load_task_manifest(task_manifest_path(root), repo_root=root) - hypothesis = protocol["hypotheses"].get(hypothesis_id) - if hypothesis is None: - raise ValueError(f"unknown hypothesis: {hypothesis_id!r}") - task_set_name = str(hypothesis["task_set"]) - tasks = load_task_set( - repo_root=root, - manifest=manifest, - task_set=task_set_name, - ) - jobs = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name=wave_name, - task_ids=[task.task_id for task in tasks], - selection=selection, - ) - boundary = require_execution_boundary() - output_root = _external_output_root(root, external_root) - assets = prompt_asset_hashes() - prompt_binding = { - "contract_version": CONTRACT_VERSION, - "preprocess_version": IMAGE_PREPROCESS_VERSION, - **assets, - } - task_entries = manifest["task_sets"][task_set_name] - arms = { - arm_id: protocol["arms"][arm_id] - for arm_id in resolved_arm_ids( - protocol, - hypothesis_id, - selection=selection, - ) - } - binding = { - "schema_version": "pixcell-baseline-launch-binding-v1", - "study_id": protocol["study_id"], - "hypothesis_id": hypothesis_id, - "wave": wave_name, - "source_git_sha": state["head"], - "protocol_path": selected_protocol.relative_to(root).as_posix(), - "protocol_sha256": protocol["logical_sha256"], - "task_manifest_sha256": manifest["logical_sha256"], - "task_set": task_set_name, - "task_set_sha256": canonical_json_sha256(task_entries), - "task_count": len({job.task_id for job in jobs}), - "job_count": len(jobs), - "prompt": prompt_binding, - "prompt_sha256": canonical_json_sha256(prompt_binding), - "arms": arms, - "sampling": protocol["sampling"], - "runtime": runtime, - "sandbox": { - "runtime_path": str(boundary.runtime_path), - "daemon_endpoint": boundary.daemon_endpoint, - "image_ref": boundary.image_ref, - "image_id": boundary.image_id, - "workspace_root": str(boundary.workspace_root), - }, - "external_root": str(output_root), - "selection_sha256": ( - canonical_json_sha256(selection) if selection is not None else None - ), - "jobs_sha256": canonical_json_sha256( - [ - { - "hypothesis_id": job.hypothesis_id, - "arm_id": job.arm_id, - "task_id": job.task_id, - "attempt_index": job.attempt_index, - "seed": job.seed, - } - for job in jobs - ] - ), - } - binding["binding_sha256"] = canonical_json_sha256(binding) - return binding, tasks, jobs - - -def required_credentials(protocol: dict[str, Any]) -> tuple[str, ...]: - result = ["TINKER_API_KEY"] - if protocol["tracking"]["mode"] == "online": - result.append("WANDB_API_KEY") - return tuple(result) - - -def validate_credentials(protocol: dict[str, Any]) -> None: - missing = [key for key in required_credentials(protocol) if not os.getenv(key)] - if missing: - raise ValueError(f"missing launch credentials: {', '.join(missing)}") - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--seal", type=Path) - return result - - -def main() -> None: - args = parser().parse_args() - if args.seal is None: - raise SystemExit("--seal is required") - document = seal_protocol(args.seal) - print(document["logical_sha256"]) - - -if __name__ == "__main__": - main() diff --git a/rl/evaluation/runner.py b/rl/evaluation/runner.py deleted file mode 100644 index 539126ad..00000000 --- a/rl/evaluation/runner.py +++ /dev/null @@ -1,1240 +0,0 @@ -"""Resumable, candidate-level baseline sampling and deterministic evaluation.""" - -from __future__ import annotations - -import asyncio -import hashlib -import json -import math -import statistics -import time -from contextlib import ExitStack -from pathlib import Path -from typing import Any - -from rl.common.evaluator import Attribution, EvaluationStatus, PixCellEvaluator -from rl.common.output import extract_code -from rl.common.preprocess import model_image -from rl.common.prompt import build_prompt_text -from rl.evaluation.attempt_store import ( - AttemptKey, - AttemptStore, - RunKey, - StoredRecord, -) -from rl.evaluation.protocol import ( - ArmSpec, - JobSpec, - arm_spec, - build_launch_binding, - jobs_for_wave, - load_protocol, - resolved_arm_ids, - task_manifest_path, - validate_credentials, -) -from rl.evaluation.samplers import ( - SamplerTarget, - SamplingRequest, - TinkerSampler, -) -from rl.evaluation.summarize import ( - atomic_write_json, - choose_operating_point, - summarize_arm, -) -from rl.evaluation.tasks import ( - EvaluationTask, - canonical_json_sha256, - load_task_manifest, - load_task_set, -) -from rl.evaluation.tracking import ( - WandbMetricsMirror, - WandbRunSession, - validate_wandb_access, -) - - -_FORBIDDEN_PROMPT_FRAGMENTS = ( - "DISPLAY NOTE", - "magnification x:y", - "px/um", - "target_image", - "calibration.json", -) - - -def _sha256_text(value: str) -> str: - return hashlib.sha256(value.encode("utf-8")).hexdigest() - - -def _messages(task: EvaluationTask, arm: ArmSpec) -> list[Any]: - from tinker_cookbook.renderers import ImagePart, Message, TextPart - - text = build_prompt_text(task.observation) - leaked = [value for value in _FORBIDDEN_PROMPT_FRAGMENTS if value in text] - if leaked: - raise ValueError(f"model prompt leaked forbidden fields: {leaked}") - return [ - Message( - role="user", - content=[ - ImagePart( - type="image", - image=model_image( - task.observation, - max_long_edge=arm.max_image_long_edge, - ), - ), - TextPart(type="text", text=text), - ], - ) - ] - - -def _sampler_target(arm: ArmSpec) -> SamplerTarget: - return SamplerTarget( - model_name=arm.model, - renderer_name=arm.renderer, - base_model=arm.model, - effort=arm.thinking_effort, - ) - - -def _safe_json(value: Any) -> Any: - if value is None or isinstance(value, (str, int, float, bool)): - return value - if isinstance(value, dict): - return {str(key): _safe_json(item) for key, item in value.items()} - if isinstance(value, (list, tuple)): - return [_safe_json(item) for item in value] - if hasattr(value, "item"): - return _safe_json(value.item()) - return str(value) - - -def _attempt_key(study_id: str, job: JobSpec) -> AttemptKey: - return AttemptKey( - study_id=study_id, - hypothesis_id=job.hypothesis_id, - arm_id=job.arm_id, - task_id=job.task_id, - attempt_index=job.attempt_index, - ) - - -def _run_manifest(binding: dict[str, Any], arm_id: str) -> dict[str, Any]: - return { - "source_git_sha": binding["source_git_sha"], - "protocol_sha256": binding["protocol_sha256"], - "task_set_sha256": binding["task_set_sha256"], - "prompt_sha256": binding["prompt_sha256"], - "sandbox_image_ref": binding["sandbox"]["image_ref"], - "sandbox_image_id": binding["sandbox"]["image_id"], - "runtime": binding["runtime"], - "model_binding": { - "arm_id": arm_id, - **binding["arms"][arm_id], - "temperature": binding["sampling"]["temperature"], - "top_p": binding["sampling"]["top_p"], - }, - } - - -def _selection_document(path: Path | None) -> dict[str, Any] | None: - if path is None: - return None - value = json.loads(path.expanduser().resolve(strict=True).read_text(encoding="utf-8")) - if not isinstance(value, dict): - raise ValueError("selection receipt must contain a JSON object") - claimed = value.get("logical_sha256") - unsigned = {key: item for key, item in value.items() if key != "logical_sha256"} - if claimed != canonical_json_sha256(unsigned): - raise ValueError("selection receipt logical SHA-256 mismatch") - return value - - -def _canonical_selection_path( - *, - external_root: Path, - study_id: str, - hypothesis_id: str, -) -> Path | None: - root = ( - external_root.expanduser().resolve() - / "studies" - / study_id - / "selections" - ) - if hypothesis_id == "RC-H00R": - return root / "operating_point.preliminary.json" - if hypothesis_id == "RC-H01": - return root / "operating_point.json" - return None - - -def _cost_estimate( - protocol: dict[str, Any], - arm: ArmSpec, - *, - prompt_tokens: int, - completion_tokens: int, -) -> dict[str, float]: - rates = protocol["pricing"]["models"][arm.model] - sample = completion_tokens * float(rates["sample"]) / 1_000_000 - return { - "cached_prefill": ( - prompt_tokens * float(rates["prefill_cached"]) / 1_000_000 + sample - ), - "uncached_prefill": ( - prompt_tokens * float(rates["prefill_uncached"]) / 1_000_000 + sample - ), - } - - -def _tracking_step( - job: JobSpec, - *, - tasks: list[EvaluationTask], -) -> int: - task_index = {task.task_id: index for index, task in enumerate(tasks)} - return (job.attempt_index - 1) * len(tasks) + task_index[job.task_id] - - -def _summary_path(store: AttemptStore, run_key: RunKey) -> Path: - relative = store.run_relative_path(run_key) / "summary.json" - return store.external_root.joinpath(*relative.parts) - - -def _selection_paths(store: AttemptStore, study_id: str) -> tuple[Path, Path]: - root = store.external_root / "studies" / study_id / "selections" - return ( - root / "operating_point.preliminary.json", - root / "operating_point.json", - ) - - -def _preflight_renderers( - *, - tasks: list[EvaluationTask], - arms: dict[str, ArmSpec], - samplers: dict[str, TinkerSampler], -) -> dict[tuple[str, str], list[Any]]: - messages: dict[tuple[str, str], list[Any]] = {} - for arm_id, arm in arms.items(): - sampler = samplers[arm_id] - for task in tasks: - rendered_messages = _messages(task, arm) - prompt = sampler.build_prompt(rendered_messages) - if int(prompt.length) + arm.max_output_tokens > arm.context_tokens: - raise ValueError( - f"{arm_id}/{task.task_id} exceeds its context window: " - f"{prompt.length}+{arm.max_output_tokens}>{arm.context_tokens}" - ) - messages[(arm_id, task.task_id)] = rendered_messages - return messages - - -def _preflight_references(tasks: list[EvaluationTask], evaluator: PixCellEvaluator) -> None: - for task in tasks: - evidence = evaluator.validate_reference(task.reference) - if evidence["target_image_sha256"] != task.reference.target_image_sha256: - raise ValueError(f"{task.task_id} reference validation changed its digest") - - -def _expected_request( - *, - arm: ArmSpec, - protocol: dict[str, Any], - job: JobSpec, -) -> dict[str, int | float]: - return { - "max_tokens": arm.max_output_tokens, - "temperature": float(protocol["sampling"]["temperature"]), - "top_p": float(protocol["sampling"]["top_p"]), - "seed": job.seed, - } - - -def _finite_number(value: Any, *, field: str, key: AttemptKey) -> float: - if ( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or not math.isfinite(float(value)) - ): - raise ValueError(f"{key}: {field} must be a finite number") - return float(value) - - -def _unit_interval(value: Any, *, field: str, key: AttemptKey) -> float: - result = _finite_number(value, field=field, key=key) - if not 0.0 <= result <= 1.0: - raise ValueError(f"{key}: {field} must be in [0, 1]") - return result - - -def _validate_existing_receipt( - *, - key: AttemptKey, - job: JobSpec, - arm: ArmSpec, - protocol: dict[str, Any], - receipt: dict[str, Any], - expected_prompt_tokens: int, - run_manifest_record_sha256: str, -) -> None: - expected_fields = { - "request", - "sample", - "sampling_seconds", - "raw_completion_sha256", - "completion_sha256", - "cost_estimate_usd", - "run_manifest_record_sha256", - } - if set(receipt) != expected_fields: - raise ValueError( - f"{key}: sampling receipt fields differ from the launch schema" - ) - if receipt["run_manifest_record_sha256"] != run_manifest_record_sha256: - raise ValueError( - f"{key}: sampling receipt belongs to another launch manifest" - ) - if receipt["request"] != _expected_request( - arm=arm, - protocol=protocol, - job=job, - ): - raise ValueError( - f"{key}: sampling request differs from the exact current job" - ) - - sample = receipt["sample"] - required_sample_fields = { - "prompt_tokens", - "completion_tokens", - "stop_reason", - "cap_hit", - "raw_text", - "completion_text", - "reasoning_text", - "answer_text", - "reasoning_tokens_exact", - "answer_tokens_exact", - "reasoning_tokens_estimate", - "answer_tokens_estimate", - "channel_token_count_basis", - "channel_parse_complete", - } - if not isinstance(sample, dict) or set(sample) != required_sample_fields: - raise ValueError( - f"{key}: sampled completion fields differ from the launch schema" - ) - prompt_tokens = sample["prompt_tokens"] - completion_tokens = sample["completion_tokens"] - if ( - isinstance(prompt_tokens, bool) - or not isinstance(prompt_tokens, int) - or prompt_tokens != expected_prompt_tokens - ): - raise ValueError( - f"{key}: stored prompt length differs from the current renderer" - ) - if ( - isinstance(completion_tokens, bool) - or not isinstance(completion_tokens, int) - or not 0 <= completion_tokens <= arm.max_output_tokens - or prompt_tokens + arm.max_output_tokens > arm.context_tokens - ): - raise ValueError(f"{key}: stored completion has invalid token accounting") - if ( - not isinstance(sample["stop_reason"], str) - or not isinstance(sample["cap_hit"], bool) - or sample["cap_hit"] != (sample["stop_reason"] == "length") - or (sample["cap_hit"] and completion_tokens != arm.max_output_tokens) - ): - raise ValueError( - f"{key}: stored stop reason and cap state are inconsistent" - ) - for field in ("raw_text", "completion_text", "answer_text"): - if not isinstance(sample[field], str): - raise ValueError(f"{key}: sample.{field} must be text") - if sample["reasoning_text"] is not None and not isinstance( - sample["reasoning_text"], - str, - ): - raise ValueError(f"{key}: sample.reasoning_text must be text or null") - if ( - not isinstance(sample["channel_token_count_basis"], str) - or not sample["channel_token_count_basis"] - ): - raise ValueError( - f"{key}: sample.channel_token_count_basis must be non-empty text" - ) - if not isinstance(sample["channel_parse_complete"], bool): - raise ValueError( - f"{key}: sample.channel_parse_complete must be boolean" - ) - - reasoning_exact = sample["reasoning_tokens_exact"] - answer_exact = sample["answer_tokens_exact"] - if (reasoning_exact is None) != (answer_exact is None): - raise ValueError(f"{key}: exact channel token accounting is partial") - if reasoning_exact is not None and ( - isinstance(reasoning_exact, bool) - or not isinstance(reasoning_exact, int) - or reasoning_exact < 0 - or isinstance(answer_exact, bool) - or not isinstance(answer_exact, int) - or answer_exact < 0 - or reasoning_exact + answer_exact != completion_tokens - ): - raise ValueError( - f"{key}: exact channel tokens do not partition completion" - ) - for field in ("reasoning_tokens_estimate", "answer_tokens_estimate"): - value = sample[field] - if value is not None and ( - isinstance(value, bool) or not isinstance(value, int) or value < 0 - ): - raise ValueError( - f"{key}: sample.{field} must be nonnegative or null" - ) - - raw_text = sample["raw_text"] - completion_text = sample["completion_text"] - if receipt["raw_completion_sha256"] != _sha256_text(raw_text): - raise ValueError(f"{key}: raw completion digest mismatch") - if receipt["completion_sha256"] != _sha256_text(completion_text): - raise ValueError(f"{key}: completion digest mismatch") - if _finite_number( - receipt["sampling_seconds"], - field="sampling_seconds", - key=key, - ) < 0.0: - raise ValueError(f"{key}: sampling_seconds must be nonnegative") - expected_cost = _cost_estimate( - protocol, - arm, - prompt_tokens=prompt_tokens, - completion_tokens=completion_tokens, - ) - if receipt["cost_estimate_usd"] != expected_cost: - raise ValueError(f"{key}: cost evidence does not reproduce") - - -def _validate_existing_evaluation( - *, - key: AttemptKey, - task: EvaluationTask, - receipt: dict[str, Any], - evaluation: dict[str, Any], -) -> None: - required_fields = { - "status", - "attribution", - "pure_executable", - "measurement_available", - "measured_iou", - "measured_dice", - "aggregation_iou", - "program", - "program_sha256", - "reference_sha256", - "violations", - "error", - "retryable", - "evaluation_seconds", - "diagnostics", - } - if not required_fields.issubset(evaluation) or not set(evaluation).issubset( - required_fields | {"render_sha256"} - ): - raise ValueError(f"{key}: evaluation fields differ from the launch schema") - try: - attribution = Attribution(str(evaluation["attribution"])) - status = EvaluationStatus(str(evaluation["status"])) - except ValueError as exc: - raise ValueError( - f"{key}: evaluation has an unknown status or attribution" - ) from exc - if attribution is not Attribution.MODEL: - raise RuntimeError( - f"{key}: existing evaluation is {attribution.value}/{status.value}; " - "refusing to buy more samples" - ) - - completion = str(receipt["sample"]["completion_text"]) - program = evaluation["program"] - if not isinstance(program, str) or program != extract_code(completion): - raise ValueError( - f"{key}: stored program differs from deterministic extraction" - ) - if evaluation["program_sha256"] != _sha256_text(program): - raise ValueError(f"{key}: program digest mismatch") - if evaluation["reference_sha256"] != task.reference.target_image_sha256: - raise ValueError(f"{key}: evaluation belongs to another reference") - if ( - not isinstance(evaluation["pure_executable"], bool) - or evaluation["pure_executable"] != (status is EvaluationStatus.OK) - ): - raise ValueError(f"{key}: pure-executable flag is inconsistent") - if not isinstance(evaluation["measurement_available"], bool): - raise ValueError( - f"{key}: measurement-availability flag must be boolean" - ) - measured_iou = evaluation["measured_iou"] - measured_dice = evaluation["measured_dice"] - if (measured_iou is None) != (measured_dice is None): - raise ValueError(f"{key}: geometry metrics are partial") - if evaluation["measurement_available"] != (measured_iou is not None): - raise ValueError( - f"{key}: measurement-availability flag is inconsistent" - ) - if evaluation["measurement_available"] != ( - status is EvaluationStatus.OK - ): - raise ValueError( - f"{key}: successful execution and geometry measurement disagree" - ) - if not isinstance(evaluation["diagnostics"], dict): - raise ValueError(f"{key}: evaluation diagnostics must be an object") - expected_aggregation = 0.0 - if measured_iou is not None: - expected_aggregation = _unit_interval( - measured_iou, - field="measured_iou", - key=key, - ) - _unit_interval(measured_dice, field="measured_dice", key=key) - render_sha = evaluation.get("render_sha256") - if ( - not isinstance(render_sha, str) - or len(render_sha) != 64 - or any( - character not in "0123456789abcdef" - for character in render_sha - ) - ): - raise ValueError( - f"{key}: measured geometry has no render digest" - ) - if evaluation["diagnostics"].get("render_sha256") != render_sha: - raise ValueError( - f"{key}: render digest differs from evaluator diagnostics" - ) - if ( - evaluation["diagnostics"].get("reference_sha256") - != task.reference.target_image_sha256 - ): - raise ValueError( - f"{key}: diagnostics belong to another reference" - ) - aggregation_iou = _unit_interval( - evaluation["aggregation_iou"], - field="aggregation_iou", - key=key, - ) - if aggregation_iou != expected_aggregation: - raise ValueError(f"{key}: aggregation IoU does not reproduce") - if ( - not isinstance(evaluation["violations"], list) - or any( - not isinstance(item, str) for item in evaluation["violations"] - ) - or not isinstance(evaluation["retryable"], bool) - or not isinstance(evaluation["diagnostics"], dict) - ): - raise ValueError( - f"{key}: evaluation diagnostics have an invalid shape" - ) - if evaluation["retryable"]: - raise ValueError( - f"{key}: model-attributed evaluation cannot be retryable" - ) - if evaluation["error"] is not None and not isinstance( - evaluation["error"], - str, - ): - raise ValueError(f"{key}: evaluation error must be text or null") - if _finite_number( - evaluation["evaluation_seconds"], - field="evaluation_seconds", - key=key, - ) < 0.0: - raise ValueError(f"{key}: evaluation_seconds must be nonnegative") - - -def _validate_existing_attempts_before_sampling( - *, - store: AttemptStore, - study_id: str, - jobs: list[JobSpec], - tasks: list[EvaluationTask], - arms: dict[str, ArmSpec], - protocol: dict[str, Any], - samplers: dict[str, TinkerSampler], - messages: dict[tuple[str, str], list[Any]], - run_manifest_record_sha256_by_arm: dict[str, str], - require_receipt_evaluations: bool = True, -) -> None: - """Bind every resumed record before any remote sampling is possible.""" - - if not jobs: - raise ValueError("baseline wave contains no jobs") - task_by_id = {task.task_id: task for task in tasks} - hypothesis_ids = {job.hypothesis_id for job in jobs} - if len(hypothesis_ids) != 1: - raise ValueError("baseline wave contains multiple hypotheses") - hypothesis_id = next(iter(hypothesis_ids)) - prompt_lengths: dict[tuple[str, str], int] = {} - for arm_id in arms: - run_key = RunKey(study_id, hypothesis_id, arm_id) - manifest = store.load_run_manifest(run_key) - expected_manifest_sha = run_manifest_record_sha256_by_arm[arm_id] - if manifest is None or manifest.record_sha256 != expected_manifest_sha: - raise ValueError(f"{run_key}: launch manifest is missing or stale") - for task in tasks: - prompt = samplers[arm_id].build_prompt( - messages[(arm_id, task.task_id)] - ) - prompt_lengths[(arm_id, task.task_id)] = int(prompt.length) - - seen: set[AttemptKey] = set() - for job in jobs: - key = _attempt_key(study_id, job) - if key in seen: - raise ValueError(f"duplicate launch job: {key}") - seen.add(key) - receipt = store.load_sampling_receipt(key) - evaluation = store.load_evaluation(key) - if receipt is None: - if evaluation is not None: - raise ValueError( - f"{key}: evaluation exists without a sampling receipt" - ) - continue - _validate_existing_receipt( - key=key, - job=job, - arm=arms[job.arm_id], - protocol=protocol, - receipt=receipt.payload, - expected_prompt_tokens=prompt_lengths[ - (job.arm_id, job.task_id) - ], - run_manifest_record_sha256=( - run_manifest_record_sha256_by_arm[job.arm_id] - ), - ) - if evaluation is None: - if require_receipt_evaluations: - raise RuntimeError( - f"{key}: existing receipt has not been evaluated; " - "refusing to buy more samples" - ) - continue - _validate_existing_evaluation( - key=key, - task=task_by_id[job.task_id], - receipt=receipt.payload, - evaluation=evaluation.payload, - ) - - -async def _sample_missing( - *, - store: AttemptStore, - study_id: str, - jobs: list[JobSpec], - tasks: list[EvaluationTask], - arms: dict[str, ArmSpec], - protocol: dict[str, Any], - samplers: dict[str, TinkerSampler], - messages: dict[tuple[str, str], list[Any]], - run_manifest_record_sha256_by_arm: dict[str, str], -) -> None: - semaphore = asyncio.Semaphore(int(protocol["sampling"]["sampling_concurrency"])) - - async def sample_one(job: JobSpec) -> None: - key = _attempt_key(study_id, job) - if store.load_sampling_receipt(key) is not None: - return - arm = arms[job.arm_id] - request = SamplingRequest( - max_tokens=arm.max_output_tokens, - temperature=float(protocol["sampling"]["temperature"]), - top_p=float(protocol["sampling"]["top_p"]), - seed=job.seed, - ) - started = time.monotonic() - async with semaphore: - sample = await samplers[job.arm_id].sample( - messages[(job.arm_id, job.task_id)], - request, - ) - elapsed = time.monotonic() - started - sample_payload = sample.to_dict() - payload = { - "request": { - "max_tokens": request.max_tokens, - "temperature": request.temperature, - "top_p": request.top_p, - "seed": request.seed, - }, - "sample": sample_payload, - "sampling_seconds": elapsed, - "raw_completion_sha256": _sha256_text(sample.raw_text), - "completion_sha256": _sha256_text(sample.completion_text), - "run_manifest_record_sha256": run_manifest_record_sha256_by_arm[ - job.arm_id - ], - "cost_estimate_usd": _cost_estimate( - protocol, - arm, - prompt_tokens=sample.prompt_tokens, - completion_tokens=sample.completion_tokens, - ), - } - store.write_sampling_receipt(key, payload) - - outcomes = await asyncio.gather( - *(sample_one(job) for job in jobs), - return_exceptions=True, - ) - failures = [ - f"{jobs[index].arm_id}/{jobs[index].task_id}/" - f"attempt-{jobs[index].attempt_index}: {type(outcome).__name__}" - for index, outcome in enumerate(outcomes) - if isinstance(outcome, BaseException) - ] - if failures: - raise RuntimeError( - "one or more model requests failed after all sibling requests " - "settled and successful receipts were persisted:\n" - + "\n".join(failures) - ) - - -def _evaluate_missing( - *, - store: AttemptStore, - study_id: str, - jobs: list[JobSpec], - tasks: list[EvaluationTask], - evaluator: PixCellEvaluator, - tracker_by_arm: dict[str, WandbRunSession] | None, -) -> None: - task_by_id = {task.task_id: task for task in tasks} - pending: list[tuple[JobSpec, AttemptKey, StoredRecord]] = [] - for job in jobs: - key = _attempt_key(study_id, job) - if store.load_evaluation(key) is not None: - continue - receipt = store.load_sampling_receipt(key) - if receipt is None: - raise RuntimeError(f"missing sampling receipt for {key}") - pending.append((job, key, receipt)) - - failures: list[str] = [] - for job in jobs: - key = _attempt_key(study_id, job) - existing = store.load_evaluation(key) - if ( - existing is not None - and existing.payload.get("attribution") != Attribution.MODEL.value - ): - failures.append( - f"{key}: {existing.payload.get('attribution')}/" - f"{existing.payload.get('status')}; " - f"record={existing.relative_path}" - ) - batch_size = evaluator.max_workers * 4 - for start in range(0, len(pending), batch_size): - batch = pending[start : start + batch_size] - requests = [ - ( - task_by_id[job.task_id].reference, - receipt.payload["sample"]["completion_text"], - ) - for job, _key, receipt in batch - ] - results = evaluator.evaluate_batch(requests) - for (job, key, receipt), result in zip(batch, results, strict=True): - receipt_payload = receipt.payload - completion = str(receipt_payload["sample"]["completion_text"]) - program = extract_code(completion) - measured_iou = float(result.iou) if result.iou is not None else None - measured_dice = float(result.dice) if result.dice is not None else None - aggregation_iou = ( - measured_iou - if result.attribution is Attribution.MODEL and measured_iou is not None - else 0.0 - ) - pure = result.status is EvaluationStatus.OK - payload: dict[str, Any] = { - "status": result.status.value, - "attribution": result.attribution.value, - "pure_executable": pure, - "measurement_available": result.iou is not None, - "measured_iou": measured_iou, - "measured_dice": measured_dice, - "aggregation_iou": aggregation_iou, - "program": program, - "program_sha256": _sha256_text(program), - "reference_sha256": task_by_id[ - job.task_id - ].reference.target_image_sha256, - "violations": list(result.violations), - "error": result.error, - "retryable": bool(result.retryable), - "evaluation_seconds": float(result.latency_seconds), - "diagnostics": _safe_json(result.metrics), - } - render_sha256 = result.metrics.get("render_sha256") - if render_sha256 is not None: - payload["render_sha256"] = render_sha256 - if result.attribution is not Attribution.MODEL: - diagnostic = store.write_evaluator_failure( - key, - { - "status": result.status.value, - "attribution": result.attribution.value, - "error": result.error, - "retryable": bool(result.retryable), - "violations": list(result.violations), - "program_sha256": _sha256_text(program), - "reference_sha256": task_by_id[ - job.task_id - ].reference.target_image_sha256, - "sampling_record_sha256": receipt.record_sha256, - }, - ) - failures.append( - f"{key}: {result.attribution.value}/" - f"{result.status.value}; " - f"diagnostic={diagnostic.relative_path}" - ) - continue - store.write_evaluation(key, payload) - tracking_metrics = { - "attempt/iou": aggregation_iou, - "attempt/pure_executable": int(pure), - "attempt/cap_hit": int( - receipt_payload["sample"]["cap_hit"] - ), - "attempt/prompt_tokens": int( - receipt_payload["sample"]["prompt_tokens"] - ), - "attempt/completion_tokens": int( - receipt_payload["sample"]["completion_tokens"] - ), - "attempt/sampling_seconds": float( - receipt_payload["sampling_seconds"] - ), - "attempt/evaluation_seconds": float(result.latency_seconds), - "attempt/estimated_uncached_cost_usd": float( - receipt_payload["cost_estimate_usd"][ - "uncached_prefill" - ] - ), - } - if tracker_by_arm is not None: - tracker_by_arm[job.arm_id].mirror_saved_metrics( - key=key, - source="evaluation", - metrics=tracking_metrics, - step=_tracking_step(job, tasks=tasks), - ) - if failures: - raise RuntimeError( - "baseline evaluation aborted on verifier/reference failures:\n" - + "\n".join(failures) - ) - - -def _validate_selection_provenance( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, - protocol: dict[str, Any], - hypothesis_id: str, - selection: dict[str, Any] | None, -) -> None: - """Recompute a canonical operating-point receipt from immutable attempts.""" - - if hypothesis_id not in {"RC-H00R", "RC-H01"}: - if selection is not None: - raise ValueError(f"{hypothesis_id} must not receive a selection receipt") - return - if selection is None: - raise ValueError(f"{hypothesis_id} requires a selection receipt") - if selection.get("source_git_sha") != expected_source_sha: - raise ValueError("operating-point selection source SHA differs from launch SHA") - - store = AttemptStore(repo_root=repo_root, external_root=external_root) - manifest = load_task_manifest(task_manifest_path(repo_root), repo_root=repo_root) - tasks = load_task_set( - repo_root=repo_root, - manifest=manifest, - task_set=protocol["hypotheses"]["RC-H00"]["task_set"], - ) - task_ids = [task.task_id for task in tasks] - h00_jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H00", - wave_name="complete", - task_ids=task_ids, - ) - h00_binding, _bound_tasks, _bound_jobs = build_launch_binding( - repo_root=repo_root, - hypothesis_id="RC-H00", - wave_name="complete", - expected_source_sha=expected_source_sha, - external_root=external_root, - ) - summaries: dict[str, dict[str, Any]] = {} - for arm_id in protocol["hypotheses"]["RC-H00"]["arm_ids"]: - run_key = RunKey(protocol["study_id"], "RC-H00", arm_id) - run_manifest = store.load_run_manifest(run_key) - if run_manifest is None or run_manifest.payload != _run_manifest( - h00_binding, - arm_id, - ): - raise ValueError(f"RC-H00/{arm_id} run manifest is missing or stale") - summaries[arm_id] = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id="RC-H00", - arm_id=arm_id, - jobs=h00_jobs, - tasks=tasks, - ) - - rescue_summary = None - if selection.get("rescue_run"): - rescue_arm = str(protocol["hypotheses"]["RC-H00R"]["arm_ids"][0]) - h00r_jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H00R", - wave_name="complete", - task_ids=task_ids, - selection=selection, - ) - h00r_binding, _bound_tasks, _bound_jobs = build_launch_binding( - repo_root=repo_root, - hypothesis_id="RC-H00R", - wave_name="complete", - expected_source_sha=expected_source_sha, - external_root=external_root, - selection=selection, - ) - run_key = RunKey(protocol["study_id"], "RC-H00R", rescue_arm) - run_manifest = store.load_run_manifest(run_key) - if run_manifest is None or run_manifest.payload != _run_manifest( - h00r_binding, - rescue_arm, - ): - raise ValueError("RC-H00R run manifest is missing or stale") - rescue_summary = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id="RC-H00R", - arm_id=rescue_arm, - jobs=h00r_jobs, - tasks=tasks, - ) - - expected = choose_operating_point( - protocol=protocol, - source_git_sha=expected_source_sha, - source_summaries=summaries, - rescue_summary=rescue_summary, - ) - if selection != expected: - raise ValueError( - "operating-point selection does not reproduce from immutable attempts" - ) - if hypothesis_id == "RC-H00R" and ( - not expected["rescue_required"] or expected["rescue_run"] - ): - raise ValueError("RC-H00R requires the incomplete triggered preliminary receipt") - if hypothesis_id == "RC-H01" and not expected["complete"]: - raise ValueError("RC-H01 requires a complete operating-point selection") - - -def _write_summaries_and_selection( - *, - store: AttemptStore, - protocol: dict[str, Any], - binding: dict[str, Any], - jobs: list[JobSpec], - tasks: list[EvaluationTask], -) -> dict[str, dict[str, Any]]: - hypothesis_id = str(binding["hypothesis_id"]) - summaries: dict[str, dict[str, Any]] = {} - for arm_id in binding["arms"]: - run_key = RunKey(protocol["study_id"], hypothesis_id, arm_id) - summary = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id=hypothesis_id, - arm_id=arm_id, - jobs=jobs, - tasks=tasks, - ) - atomic_write_json(_summary_path(store, run_key), summary) - summaries[arm_id] = summary - - if binding["wave"] != "complete": - return summaries - preliminary_path, final_path = _selection_paths(store, protocol["study_id"]) - if hypothesis_id == "RC-H00": - selection = choose_operating_point( - protocol=protocol, - source_git_sha=binding["source_git_sha"], - source_summaries=summaries, - ) - atomic_write_json(preliminary_path, selection) - if selection["complete"]: - atomic_write_json(final_path, selection) - elif hypothesis_id == "RC-H00R": - manifest = json.loads(preliminary_path.read_text(encoding="utf-8")) - original = { - arm_id: value - for arm_id, value in manifest["arm_summaries"].items() - if arm_id in protocol["hypotheses"]["RC-H00"]["arm_ids"] - } - if len(summaries) != 1: - raise ValueError("RC-H00R must contain exactly one rescue arm") - selection = choose_operating_point( - protocol=protocol, - source_git_sha=binding["source_git_sha"], - source_summaries=original, - rescue_summary=next(iter(summaries.values())), - ) - atomic_write_json(final_path, selection) - return summaries - - -async def run_baseline( - *, - repo_root: Path, - hypothesis_id: str, - wave_name: str, - expected_source_sha: str, - external_root: Path, - confirmation: str, - selection_path: Path | None = None, - protocol_file: Path | None = None, -) -> dict[str, Any]: - """Run one frozen wave. No training client is ever constructed.""" - - root = repo_root.expanduser().resolve() - protocol = ( - load_protocol(root) - if protocol_file is None - else load_protocol(root, protocol_file=protocol_file) - ) - canonical_selection = _canonical_selection_path( - external_root=external_root, - study_id=protocol["study_id"], - hypothesis_id=hypothesis_id, - ) - if canonical_selection is None: - if selection_path is not None: - raise ValueError(f"{hypothesis_id} does not accept a selection receipt") - else: - if selection_path is None: - raise ValueError( - f"{hypothesis_id} requires --selection {canonical_selection}" - ) - observed_selection = selection_path.expanduser().resolve(strict=True) - if observed_selection != canonical_selection: - raise ValueError( - "selection receipt must be the canonical file under the study root" - ) - expected_confirmation = protocol["launch"]["confirmation_token"] - if confirmation != expected_confirmation: - raise ValueError( - f"paid baseline launch requires --confirm-spend {expected_confirmation}" - ) - selection = _selection_document(selection_path) - binding, tasks, jobs = build_launch_binding( - repo_root=root, - hypothesis_id=hypothesis_id, - wave_name=wave_name, - expected_source_sha=expected_source_sha, - external_root=external_root, - selection=selection, - protocol_file=protocol_file, - ) - _validate_selection_provenance( - repo_root=root, - external_root=external_root, - expected_source_sha=expected_source_sha, - protocol=protocol, - hypothesis_id=hypothesis_id, - selection=selection, - ) - validate_credentials(protocol) - validate_wandb_access(expected_entity=protocol["tracking"].get("entity")) - arms = { - arm_id: arm_spec(protocol, arm_id) - for arm_id in resolved_arm_ids( - protocol, - hypothesis_id, - selection=selection, - ) - } - store = AttemptStore(repo_root=root, external_root=external_root) - shared_service: dict[str, Any] = {} - - def service_factory() -> Any: - if "client" not in shared_service: - import tinker - - shared_service["client"] = tinker.ServiceClient( - user_metadata={ - "study": protocol["study_id"], - "hypothesis": hypothesis_id, - "source_sha": expected_source_sha, - } - ) - return shared_service["client"] - - samplers = { - arm_id: TinkerSampler( - _sampler_target(arm), - service_client_factory=service_factory, - ) - for arm_id, arm in arms.items() - } - # Renderer/tokenizer construction, complete context accounting, reference - # validation, launch manifests, and locks all precede the first ServiceClient. - messages = _preflight_renderers( - tasks=tasks, - arms=arms, - samplers=samplers, - ) - with PixCellEvaluator( - max_workers=int(protocol["sampling"]["evaluator_workers"]), - evaluator_retries=1, - require_isolation=True, - ) as evaluator: - _preflight_references(tasks, evaluator) - with ExitStack() as stack: - run_manifest_record_sha256_by_arm: dict[str, str] = {} - for arm_id in arms: - run_key = RunKey(protocol["study_id"], hypothesis_id, arm_id) - stack.enter_context(store.acquire_run_lock(run_key)) - manifest = store.create_or_verify_run_manifest( - run_key, - _run_manifest(binding, arm_id), - ) - run_manifest_record_sha256_by_arm[arm_id] = ( - manifest.record_sha256 - ) - _validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=tasks, - arms=arms, - protocol=protocol, - samplers=samplers, - messages=messages, - run_manifest_record_sha256_by_arm=( - run_manifest_record_sha256_by_arm - ), - require_receipt_evaluations=False, - ) - resumed_jobs = [ - job - for job in jobs - if store.load_sampling_receipt( - _attempt_key(protocol["study_id"], job) - ) - is not None - ] - _evaluate_missing( - store=store, - study_id=protocol["study_id"], - jobs=resumed_jobs, - tasks=tasks, - evaluator=evaluator, - tracker_by_arm=None, - ) - _validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=tasks, - arms=arms, - protocol=protocol, - samplers=samplers, - messages=messages, - run_manifest_record_sha256_by_arm=( - run_manifest_record_sha256_by_arm - ), - require_receipt_evaluations=True, - ) - await _sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=tasks, - arms=arms, - protocol=protocol, - samplers=samplers, - messages=messages, - run_manifest_record_sha256_by_arm=( - run_manifest_record_sha256_by_arm - ), - ) - mirrors = { - arm_id: WandbMetricsMirror( - store=store, - project=protocol["tracking"]["project"], - source_git_sha=expected_source_sha, - entity=protocol["tracking"].get("entity"), - group=protocol["tracking"].get("group"), - ) - for arm_id in arms - } - trackers = { - arm_id: stack.enter_context( - mirror.session( - RunKey(protocol["study_id"], hypothesis_id, arm_id) - ) - ) - for arm_id, mirror in mirrors.items() - } - _evaluate_missing( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=tasks, - evaluator=evaluator, - tracker_by_arm=trackers, - ) - summaries = _write_summaries_and_selection( - store=store, - protocol=protocol, - binding=binding, - jobs=jobs, - tasks=tasks, - ) - return { - "binding": binding, - "summaries": summaries, - "mean_arm_iou": ( - statistics.fmean( - value["mean_iou_at_1"] for value in summaries.values() - ) - if summaries - else 0.0 - ), - } diff --git a/rl/evaluation/samplers.py b/rl/evaluation/samplers.py deleted file mode 100644 index 2c51cf9b..00000000 --- a/rl/evaluation/samplers.py +++ /dev/null @@ -1,531 +0,0 @@ -"""Lazy, one-completion Tinker samplers for frozen baseline evaluations. - -This module owns transport and renderer bookkeeping only. Callers supply the -already-built cookbook ``Message`` conversation, and the evaluation runner owns -task selection, persistence, and geometry scoring. -""" - -from __future__ import annotations - -import base64 -import io -import math -from collections.abc import Callable, Sequence -from dataclasses import asdict, dataclass -from importlib.metadata import PackageNotFoundError, version -from typing import Any, Protocol - - -QWEN_MODEL = "Qwen/Qwen3.6-35B-A3B" -INKLING_MODEL = "thinkingmachines/Inkling" -QWEN_RENDERERS = frozenset({"qwen3_5", "qwen3_5_disable_thinking"}) -INKLING_RENDERER = "tml_v0" -TML_RENDERERS_VERSION = "0.1.0" - - -class SamplerConfigurationError(ValueError): - """A target or sampling request is not fully and unambiguously bound.""" - - -class SamplingProtocolError(RuntimeError): - """Tinker returned a response outside the one-completion contract.""" - - -@dataclass(frozen=True) -class SamplerTarget: - """Exact model location plus the renderer needed to interpret its tokens. - - ``model_name`` identifies the base family for tokenizer/renderer loading. - Exactly one of ``base_model`` and ``model_path`` is the Tinker sampling - location. A checkpoint therefore remains explicitly tied to its base - family instead of trying to infer that family from an opaque URI. - """ - - model_name: str - renderer_name: str - base_model: str | None = None - model_path: str | None = None - effort: float | None = None - - def __post_init__(self) -> None: - locations = int(self.base_model is not None) + int(self.model_path is not None) - if locations != 1: - raise SamplerConfigurationError("exactly one of base_model or model_path must be set") - if self.base_model is not None and self.base_model != self.model_name: - raise SamplerConfigurationError( - "base_model must exactly match model_name; aliases are not accepted" - ) - if self.model_path is not None and not self.model_path.strip(): - raise SamplerConfigurationError("model_path must be non-empty") - - if self.model_name == QWEN_MODEL: - if self.renderer_name not in QWEN_RENDERERS: - raise SamplerConfigurationError( - f"{QWEN_MODEL} requires one of {sorted(QWEN_RENDERERS)}" - ) - if self.effort is not None: - raise SamplerConfigurationError( - "effort is a TML/Inkling setting and must be omitted for Qwen" - ) - return - - if self.model_name == INKLING_MODEL: - if self.renderer_name != INKLING_RENDERER: - raise SamplerConfigurationError( - f"{INKLING_MODEL} requires renderer_name={INKLING_RENDERER!r}" - ) - if ( - self.effort is None - or isinstance(self.effort, bool) - or not math.isfinite(self.effort) - or not 0.0 <= self.effort < 1.0 - ): - raise SamplerConfigurationError( - "Inkling effort must be an explicit finite number in [0, 1)" - ) - return - - raise SamplerConfigurationError(f"unsupported model_name: {self.model_name!r}") - - @property - def client_kwargs(self) -> dict[str, str]: - if self.base_model is not None: - return {"base_model": self.base_model} - assert self.model_path is not None - return {"model_path": self.model_path} - - -@dataclass(frozen=True) -class SamplingRequest: - """Every stochastic and length control sent to Tinker, with no defaults.""" - - max_tokens: int - temperature: float - top_p: float - seed: int - - def __post_init__(self) -> None: - if ( - isinstance(self.max_tokens, bool) - or not isinstance(self.max_tokens, int) - or self.max_tokens <= 0 - ): - raise SamplerConfigurationError("max_tokens must be a positive integer") - if ( - isinstance(self.temperature, bool) - or not math.isfinite(self.temperature) - or self.temperature < 0.0 - ): - raise SamplerConfigurationError("temperature must be a finite non-negative number") - if ( - isinstance(self.top_p, bool) - or not math.isfinite(self.top_p) - or not 0.0 < self.top_p <= 1.0 - ): - raise SamplerConfigurationError("top_p must be finite and in (0, 1]") - if isinstance(self.seed, bool) or not isinstance(self.seed, int) or self.seed < 0: - raise SamplerConfigurationError("seed must be a non-negative integer") - - -@dataclass(frozen=True) -class SampleResult: - """One sampled completion with transport and channel accounting. - - Qwen channel counts are exact sampled-token partitions. Inkling's TML - parser identifies the channels exactly, but its public parser does not - expose per-channel token spans. Its ``*_estimate`` fields therefore count - only re-encoded channel text and deliberately exclude TML framing tokens. - """ - - prompt_tokens: int - completion_tokens: int - stop_reason: str - cap_hit: bool - raw_text: str - completion_text: str - reasoning_text: str | None - answer_text: str - reasoning_tokens_exact: int | None - answer_tokens_exact: int | None - reasoning_tokens_estimate: int | None - answer_tokens_estimate: int | None - channel_token_count_basis: str - channel_parse_complete: bool - - def to_dict(self) -> dict[str, Any]: - return asdict(self) - - -@dataclass(frozen=True) -class _DecodedCompletion: - raw_text: str - completion_text: str - reasoning_text: str | None - answer_text: str - reasoning_tokens_exact: int | None - answer_tokens_exact: int | None - reasoning_tokens_estimate: int | None - answer_tokens_estimate: int | None - channel_token_count_basis: str - channel_parse_complete: bool - - -class _Codec(Protocol): - def build_prompt(self, messages: Sequence[Any]) -> Any: ... - - def stop_sequences(self) -> list[int]: ... - - def decode(self, tokens: Sequence[int]) -> _DecodedCompletion: ... - - -class _QwenCodec: - def __init__(self, renderer: Any, tokenizer: Any, renderer_name: str) -> None: - self._renderer = renderer - self._tokenizer = tokenizer - self._renderer_name = renderer_name - self._think_close = list(tokenizer.encode("", add_special_tokens=False)) - if not self._think_close: - raise SamplingProtocolError("Qwen tokenizer produced no tokens") - - @classmethod - def load(cls, target: SamplerTarget) -> _QwenCodec: - from tinker_cookbook.image_processing_utils import get_image_processor - from tinker_cookbook.renderers import get_renderer - from tinker_cookbook.tokenizer_utils import get_tokenizer - - tokenizer = get_tokenizer(target.model_name) - image_processor = get_image_processor(target.model_name) - renderer = get_renderer( - target.renderer_name, - tokenizer, - image_processor, - model_name=target.model_name, - ) - return cls(renderer, tokenizer, target.renderer_name) - - def build_prompt(self, messages: Sequence[Any]) -> Any: - return self._renderer.build_generation_prompt(list(messages)) - - def stop_sequences(self) -> list[int]: - return list(self._renderer.get_stop_sequences()) - - @staticmethod - def _first_subsequence(tokens: Sequence[int], needle: Sequence[int]) -> int | None: - limit = len(tokens) - len(needle) + 1 - for index in range(max(0, limit)): - if list(tokens[index : index + len(needle)]) == list(needle): - return index - return None - - def decode(self, tokens: Sequence[int]) -> _DecodedCompletion: - sampled = list(tokens) - raw_text = str(self._tokenizer.decode(sampled)) - if self._renderer_name == "qwen3_5_disable_thinking": - return _DecodedCompletion( - raw_text=raw_text, - completion_text=raw_text, - reasoning_text="", - answer_text=raw_text, - reasoning_tokens_exact=0, - answer_tokens_exact=len(sampled), - reasoning_tokens_estimate=None, - answer_tokens_estimate=None, - channel_token_count_basis=("qwen_disabled_thinking_all_sampled_tokens_are_answer"), - channel_parse_complete=True, - ) - - close_at = self._first_subsequence(sampled, self._think_close) - if close_at is None: - # The generation prompt has already opened the thinking channel. - # Until is sampled, every sampled token is in that channel. - return _DecodedCompletion( - raw_text=raw_text, - completion_text=f"\n{raw_text}", - reasoning_text=raw_text, - answer_text="", - reasoning_tokens_exact=len(sampled), - answer_tokens_exact=0, - reasoning_tokens_estimate=None, - answer_tokens_estimate=None, - channel_token_count_basis=("qwen_thinking_open_channel_without_closing_delimiter"), - channel_parse_complete=False, - ) - - answer_at = close_at + len(self._think_close) - reasoning_tokens = sampled[:close_at] - answer_tokens = sampled[answer_at:] - reasoning_text = str(self._tokenizer.decode(reasoning_tokens)) - answer_text = str(self._tokenizer.decode(answer_tokens)).lstrip() - return _DecodedCompletion( - raw_text=raw_text, - completion_text=f"\n{reasoning_text}\n{answer_text}", - reasoning_text=reasoning_text, - answer_text=answer_text, - reasoning_tokens_exact=answer_at, - answer_tokens_exact=len(sampled) - answer_at, - reasoning_tokens_estimate=None, - answer_tokens_estimate=None, - channel_token_count_basis=( - "qwen_sampled_tokens_partitioned_at_first_think_close_" - "closing_delimiter_assigned_to_reasoning" - ), - channel_parse_complete=True, - ) - - -def _pil_to_data_uri(image: Any) -> str: - buffer = io.BytesIO() - image.convert("RGB").save(buffer, format="PNG") - payload = base64.b64encode(buffer.getvalue()).decode("ascii") - return f"data:image/png;base64,{payload}" - - -def _normalize_tml_media(messages: Sequence[Any]) -> list[dict[str, Any]]: - """Convert cookbook ImagePart values to the OSS image_url shape TML reads.""" - - normalized: list[dict[str, Any]] = [] - for message in messages: - item = dict(message) - content = item.get("content") - if not isinstance(content, list): - normalized.append(item) - continue - parts: list[Any] = [] - for part in content: - if isinstance(part, dict) and part.get("type") == "image": - image = part.get("image") - if isinstance(image, str): - url = image - elif hasattr(image, "convert") and hasattr(image, "save"): - url = _pil_to_data_uri(image) - else: - raise TypeError( - "TML ImagePart.image must be a path/data URI string or PIL image" - ) - parts.append({"type": "image_url", "image_url": {"url": url}}) - else: - parts.append(part) - normalized.append({**item, "content": parts}) - return normalized - - -class _InklingCodec: - def __init__(self, native_renderer: Any, tokenizer: Any, effort: float) -> None: - self._renderer = native_renderer - self._tokenizer = tokenizer - self._effort = effort - - @classmethod - def load(cls, target: SamplerTarget) -> _InklingCodec: - try: - installed = version("tml-renderers") - except PackageNotFoundError as exc: - raise SamplerConfigurationError( - f"tml-renderers=={TML_RENDERERS_VERSION} is required for Inkling" - ) from exc - if installed != TML_RENDERERS_VERSION: - raise SamplerConfigurationError( - f"Inkling requires tml-renderers=={TML_RENDERERS_VERSION}; found {installed}" - ) - - from tml_renderers import tokenizers as tml_tokenizers - from tml_renderers import v0 as tml_v0 - - tokenizer = tml_tokenizers.o200k_base_chat() - return cls(tml_v0.Renderer(tokenizer), tokenizer, float(target.effort)) - - def _render_input(self, messages: Sequence[Any]) -> list[Any]: - from tml_renderers import chat as tml_chat - - return list(tml_chat.OpenAIMessage.from_oss_messages(_normalize_tml_media(messages))) - - def build_prompt(self, messages: Sequence[Any]) -> Any: - from tml_renderers import tinker as tml_tinker - - spans, _parser = self._renderer.render_for_completion_with_effort( - self._render_input(messages), - self._effort, - ) - return tml_tinker.token_spans_to_tinker_model_input(spans) - - def stop_sequences(self) -> list[int]: - return list(self._renderer.stop()) - - @staticmethod - def _raw_decode_fallback(raw_text: str, *, reason: str) -> _DecodedCompletion: - """Keep truncated TML output evaluable without inventing channel facts. - - A capped Inkling response can end inside TML framing. Depending on the - exact boundary, the native parser either raises or returns no textual - channel at all. The historical TML shim preserved the tokenizer's raw - decode in both cases, allowing downstream code extraction a best-effort - recovery path. The channel partition is unknowable in that state, so - channel-specific text/counts remain unset and the parse is incomplete. - """ - - return _DecodedCompletion( - raw_text=raw_text, - completion_text=raw_text, - reasoning_text=None, - answer_text="", - reasoning_tokens_exact=None, - answer_tokens_exact=None, - reasoning_tokens_estimate=None, - answer_tokens_estimate=None, - channel_token_count_basis=( - f"tml_{reason}_raw_decode_fallback_channel_partition_unknown" - ), - channel_parse_complete=False, - ) - - def decode(self, tokens: Sequence[int]) -> _DecodedCompletion: - from tml_renderers import chat as tml_chat - - sampled = list(tokens) - raw_text = str(self._tokenizer.decode(sampled)) - _spans, parser = self._renderer.render_for_completion([]) - try: - parsed = list(parser.parse_tokens(sampled)) - except Exception: - return self._raw_decode_fallback( - raw_text, - reason="parse_failed", - ) - - reasoning_parts: list[str] = [] - answer_parts: list[str] = [] - saw_end = False - for message in parsed: - content = message.content - if isinstance(content, tml_chat.ModelEndSampling): - saw_end = True - elif isinstance(content, tml_chat.Thinking): - reasoning_parts.append(getattr(content, "text", str(content))) - elif isinstance(content, tml_chat.Text): - answer_parts.append(getattr(content, "text", str(content))) - else: - # Tools are not part of PixCell's contract. Keeping an unknown - # model channel visible in the answer is safer than dropping it. - answer_parts.append(str(content)) - - reasoning_text = "\n".join(reasoning_parts) - answer_text = "\n".join(answer_parts) - reasoning_estimate = sum( - len(self._tokenizer.encode_ordinary(part)) for part in reasoning_parts - ) - answer_estimate = sum(len(self._tokenizer.encode_ordinary(part)) for part in answer_parts) - completion_text = ( - "".join(f"{part}\n" for part in reasoning_parts) + answer_text - ) - if not any(part.strip() for part in (*reasoning_parts, *answer_parts)): - return self._raw_decode_fallback( - raw_text, - reason="parse_had_no_usable_channel_text", - ) - return _DecodedCompletion( - raw_text=raw_text, - completion_text=completion_text, - reasoning_text=reasoning_text, - answer_text=answer_text, - reasoning_tokens_exact=None, - answer_tokens_exact=None, - reasoning_tokens_estimate=reasoning_estimate, - answer_tokens_estimate=answer_estimate, - channel_token_count_basis=( - "estimate_o200k_encode_ordinary_of_parsed_channel_text_excludes_tml_framing" - ), - channel_parse_complete=saw_end, - ) - - -def _build_codec(target: SamplerTarget) -> _Codec: - if target.model_name == QWEN_MODEL: - return _QwenCodec.load(target) - if target.model_name == INKLING_MODEL: - return _InklingCodec.load(target) - raise AssertionError(f"unvalidated sampler target: {target.model_name}") - - -def _default_service_client_factory() -> Any: - import tinker - - return tinker.ServiceClient() - - -class TinkerSampler: - """A lazy sampling client with a strict one-request/one-completion surface.""" - - def __init__( - self, - target: SamplerTarget, - *, - service_client_factory: Callable[[], Any] | None = None, - ) -> None: - self.target = target - self._service_client_factory = service_client_factory or _default_service_client_factory - self._codec: _Codec | None = None - self._sampling_client: Any | None = None - - def _get_codec(self) -> _Codec: - if self._codec is None: - self._codec = _build_codec(self.target) - return self._codec - - def build_prompt(self, messages: Sequence[Any]) -> Any: - if not messages: - raise SamplerConfigurationError("messages must not be empty") - return self._get_codec().build_prompt(messages) - - def _get_sampling_client(self) -> Any: - if self._sampling_client is None: - service = self._service_client_factory() - self._sampling_client = service.create_sampling_client(**self.target.client_kwargs) - return self._sampling_client - - async def sample( - self, - messages: Sequence[Any], - request: SamplingRequest, - ) -> SampleResult: - """Sample exactly one completion; this is the first method that creates a client.""" - - import tinker - - codec = self._get_codec() - prompt = self.build_prompt(messages) - response = await self._get_sampling_client().sample_async( - prompt=prompt, - num_samples=1, - sampling_params=tinker.SamplingParams( - max_tokens=request.max_tokens, - temperature=request.temperature, - top_p=request.top_p, - seed=request.seed, - stop=codec.stop_sequences(), - ), - ) - sequences = list(response.sequences) - if len(sequences) != 1: - raise SamplingProtocolError( - f"expected exactly one completion, received {len(sequences)}" - ) - sequence = sequences[0] - tokens = list(sequence.tokens) - stop_reason = str(sequence.stop_reason) - decoded = codec.decode(tokens) - return SampleResult( - prompt_tokens=int(prompt.length), - completion_tokens=len(tokens), - stop_reason=stop_reason, - cap_hit=stop_reason == "length", - raw_text=decoded.raw_text, - completion_text=decoded.completion_text, - reasoning_text=decoded.reasoning_text, - answer_text=decoded.answer_text, - reasoning_tokens_exact=decoded.reasoning_tokens_exact, - answer_tokens_exact=decoded.answer_tokens_exact, - reasoning_tokens_estimate=decoded.reasoning_tokens_estimate, - answer_tokens_estimate=decoded.answer_tokens_estimate, - channel_token_count_basis=decoded.channel_token_count_basis, - channel_parse_complete=decoded.channel_parse_complete, - ) diff --git a/rl/evaluation/summarize.py b/rl/evaluation/summarize.py deleted file mode 100644 index 450592c7..00000000 --- a/rl/evaluation/summarize.py +++ /dev/null @@ -1,269 +0,0 @@ -"""Deterministic aggregation and Qwen operating-point selection.""" - -from __future__ import annotations - -import json -import math -import os -import statistics -import uuid -from collections import Counter, defaultdict -from pathlib import Path -from typing import Any - -from rl.evaluation.attempt_store import AttemptKey, AttemptStore -from rl.evaluation.protocol import JobSpec -from rl.evaluation.tasks import EvaluationTask, canonical_json_sha256 - - -SUMMARY_SCHEMA = "pixcell-baseline-summary-v1" -SELECTION_SCHEMA = "pixcell-operating-point-selection-v1" - - -def _percentile(values: list[int | float], fraction: float) -> float: - if not values: - return 0.0 - ordered = sorted(float(value) for value in values) - position = fraction * (len(ordered) - 1) - lower = math.floor(position) - upper = math.ceil(position) - if lower == upper: - return ordered[lower] - weight = position - lower - return ordered[lower] * (1 - weight) + ordered[upper] * weight - - -def _attempt_key(study_id: str, job: JobSpec) -> AttemptKey: - return AttemptKey( - study_id=study_id, - hypothesis_id=job.hypothesis_id, - arm_id=job.arm_id, - task_id=job.task_id, - attempt_index=job.attempt_index, - ) - - -def summarize_arm( - *, - store: AttemptStore, - study_id: str, - hypothesis_id: str, - arm_id: str, - jobs: list[JobSpec], - tasks: list[EvaluationTask], -) -> dict[str, Any]: - relevant = [ - job - for job in jobs - if job.hypothesis_id == hypothesis_id and job.arm_id == arm_id - ] - task_by_id = {task.task_id: task for task in tasks} - attempts: list[dict[str, Any]] = [] - missing: list[dict[str, Any]] = [] - for job in relevant: - key = _attempt_key(study_id, job) - receipt = store.load_sampling_receipt(key) - evaluation = store.load_evaluation(key) - if receipt is None or evaluation is None: - missing.append(key.as_dict()) - continue - if evaluation.payload.get("attribution") != "model": - raise ValueError( - f"{key} is not a model-attributed result and cannot be summarized" - ) - task = task_by_id[job.task_id] - attempts.append( - { - "task_id": job.task_id, - "level": task.level, - "representation_id": task.representation_id, - "attempt_index": job.attempt_index, - "receipt": receipt.payload, - "evaluation": evaluation.payload, - } - ) - - by_task: dict[str, list[dict[str, Any]]] = defaultdict(list) - by_representation: dict[str, list[dict[str, Any]]] = defaultdict(list) - by_level: dict[str, list[dict[str, Any]]] = defaultdict(list) - for attempt in attempts: - by_task[attempt["task_id"]].append(attempt) - by_representation[attempt["representation_id"]].append(attempt) - by_level[attempt["level"]].append(attempt) - - def score(value: dict[str, Any]) -> float: - return float(value["evaluation"]["aggregation_iou"]) - - task_means = [ - statistics.fmean(score(value) for value in values) - for values in by_task.values() - ] - task_bests = [ - max(score(value) for value in values) - for values in by_task.values() - ] - representation_means = [ - statistics.fmean(score(value) for value in values) - for values in by_representation.values() - ] - completion_tokens = [ - int(value["receipt"]["sample"]["completion_tokens"]) for value in attempts - ] - prompt_tokens = [ - int(value["receipt"]["sample"]["prompt_tokens"]) for value in attempts - ] - sampling_seconds = [ - float(value["receipt"]["sampling_seconds"]) for value in attempts - ] - evaluation_seconds = [ - float(value["evaluation"]["evaluation_seconds"]) for value in attempts - ] - pure_count = sum( - bool(value["evaluation"]["pure_executable"]) for value in attempts - ) - cap_count = sum(bool(value["receipt"]["sample"]["cap_hit"]) for value in attempts) - cached_cost = sum( - float(value["receipt"]["cost_estimate_usd"]["cached_prefill"]) - for value in attempts - ) - uncached_cost = sum( - float(value["receipt"]["cost_estimate_usd"]["uncached_prefill"]) - for value in attempts - ) - statuses = Counter( - str(value["evaluation"]["status"]) for value in attempts - ) - levels: dict[str, Any] = {} - for level, values in sorted(by_level.items()): - level_tasks: dict[str, list[dict[str, Any]]] = defaultdict(list) - for value in values: - level_tasks[value["task_id"]].append(value) - levels[level] = { - "attempts": len(values), - "tasks": len(level_tasks), - "mean_iou_at_1": statistics.fmean( - statistics.fmean(score(item) for item in task_values) - for task_values in level_tasks.values() - ), - "pure_executable_rate": ( - sum(bool(value["evaluation"]["pure_executable"]) for value in values) - / len(values) - ), - } - - count = len(attempts) - summary = { - "schema_version": SUMMARY_SCHEMA, - "study_id": study_id, - "hypothesis_id": hypothesis_id, - "arm_id": arm_id, - "complete": not missing and len(attempts) == len(relevant), - "expected_attempts": len(relevant), - "attempts": count, - "tasks": len(by_task), - "representations": len(by_representation), - "mean_iou_at_1": statistics.fmean(task_means) if task_means else 0.0, - "mean_best_at_k_iou": statistics.fmean(task_bests) if task_bests else 0.0, - "representation_macro_mean_iou": ( - statistics.fmean(representation_means) - if representation_means - else 0.0 - ), - "pure_executable_rate": pure_count / count if count else 0.0, - "cap_hit_rate": cap_count / count if count else 0.0, - "completion_tokens_total": sum(completion_tokens), - "completion_tokens_median": ( - statistics.median(completion_tokens) if completion_tokens else 0.0 - ), - "completion_tokens_p95": _percentile(completion_tokens, 0.95), - "prompt_tokens_total": sum(prompt_tokens), - "sampling_seconds_total": sum(sampling_seconds), - "evaluation_seconds_total": sum(evaluation_seconds), - "estimated_cached_cost_usd": cached_cost, - "estimated_uncached_cost_usd": uncached_cost, - "statuses": dict(sorted(statuses.items())), - "levels": levels, - "missing": missing, - } - summary["logical_sha256"] = canonical_json_sha256(summary) - return summary - - -def choose_operating_point( - *, - protocol: dict[str, Any], - source_git_sha: str, - source_summaries: dict[str, dict[str, Any]], - rescue_summary: dict[str, Any] | None = None, -) -> dict[str, Any]: - expected = set(protocol["hypotheses"]["RC-H00"]["arm_ids"]) - if set(source_summaries) != expected: - raise ValueError("operating-point selection requires every RC-H00 arm") - if not all(value["complete"] for value in source_summaries.values()): - raise ValueError("operating-point selection requires complete RC-H00 summaries") - trigger = protocol["hypotheses"]["RC-H00R"]["trigger"] - source_arm = str(trigger["source_arm"]) - rescue_required = ( - float(source_summaries[source_arm][trigger["metric"]]) - > float(trigger["threshold"]) - ) - candidates = dict(source_summaries) - rescue_run = rescue_summary is not None - if rescue_summary is not None: - if not rescue_summary["complete"]: - raise ValueError("rescue summary is incomplete") - candidates[str(rescue_summary["arm_id"])] = rescue_summary - complete = not rescue_required or rescue_run - selected: str | None = None - if complete: - primary = protocol["operating_point_selection"]["primary_metric"] - best = max(float(value[primary]) for value in candidates.values()) - tie_band = float(protocol["operating_point_selection"]["metric_tie_band"]) - eligible = [ - (arm_id, value) - for arm_id, value in candidates.items() - if float(value[primary]) >= best - tie_band - ] - eligible.sort( - key=lambda item: ( - -float(item[1]["pure_executable_rate"]), - float(item[1]["estimated_uncached_cost_usd"]), - float(item[1]["completion_tokens_median"]), - item[0], - ) - ) - selected = eligible[0][0] - document = { - "schema_version": SELECTION_SCHEMA, - "study_id": protocol["study_id"], - "protocol_sha256": protocol["logical_sha256"], - "source_git_sha": source_git_sha, - "source_hypothesis": "RC-H00", - "complete": complete, - "rescue_required": rescue_required, - "rescue_run": rescue_run, - "selected_arm": selected, - "selection_rule": protocol["operating_point_selection"], - "arm_summaries": candidates, - } - document["logical_sha256"] = canonical_json_sha256(document) - return document - - -def atomic_write_json(path: Path, document: dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - raw = json.dumps(document, indent=2, sort_keys=True) + "\n" - temporary = path.parent / f".{path.name}.tmp-{os.getpid()}-{uuid.uuid4().hex}" - try: - with temporary.open("x", encoding="utf-8") as handle: - handle.write(raw) - handle.flush() - os.fsync(handle.fileno()) - os.replace(temporary, path) - directory_fd = os.open(path.parent, os.O_RDONLY) - try: - os.fsync(directory_fd) - finally: - os.close(directory_fd) - finally: - temporary.unlink(missing_ok=True) diff --git a/rl/evaluation/tasks.py b/rl/evaluation/tasks.py deleted file mode 100644 index 94ba31c1..00000000 --- a/rl/evaluation/tasks.py +++ /dev/null @@ -1,346 +0,0 @@ -"""Frozen evaluation-task selection and model/reference-safe loading.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from PIL import Image - -from rl.common.contracts import ( - ModelObservation, - VerifierReference, - observation_from_row, - reference_from_row, - sampler_from_row, -) -from rl.common.dataset_io import parquet_paths - - -TASK_MANIFEST_SCHEMA = "pixcell-evaluation-task-manifest-v1" -OPERATING_POINT_TASK_SET = "qwen_operating_point" -BENCHMARK_TASK_SET = "f1_f8" -_SELECTION_SEED = "representation-curriculum-v1-operating-point-20260727" -_DEV_ROWS_PER_LEVEL = 8 -_LEVELS = ("L0", "L1", "L2", "L3", "L4") -_DEV_COLUMNS = ( - "id", - "level", - "image", - "target_image", - "footprint_um", - "representation_id", - "leakage_group_id", - "image_sha256", - "target_image_sha256", - "realization_slot", - "split_role", -) - - -@dataclass(frozen=True) -class EvaluationTask: - """A task with no supervised label in memory.""" - - task_id: str - level: str - representation_id: str - observation: ModelObservation - reference: VerifierReference - - -def canonical_json_sha256(value: Any) -> str: - payload = json.dumps( - value, - sort_keys=True, - separators=(",", ":"), - ensure_ascii=True, - ).encode("utf-8") - return hashlib.sha256(payload).hexdigest() - - -def _read_depth_validation_rows(dataset_root: Path) -> list[dict[str, Any]]: - import pyarrow.parquet as pq - - rows: list[dict[str, Any]] = [] - for path in parquet_paths( - dataset_root, - configuration="depth", - split="validation", - ): - schema = set(pq.read_schema(path).names) - missing = set(_DEV_COLUMNS) - schema - if missing: - raise ValueError( - f"{path} is missing evaluation columns: {sorted(missing)}" - ) - rows.extend(pq.read_table(path, columns=list(_DEV_COLUMNS)).to_pylist()) - return rows - - -def _row_rank(row: dict[str, Any], *, namespace: str) -> str: - payload = ( - f"{_SELECTION_SEED}\0{namespace}\0{row['representation_id']}\0{row['id']}" - ) - return hashlib.sha256(payload.encode("utf-8")).hexdigest() - - -def _dev_manifest_entries(dataset_root: Path) -> list[dict[str, Any]]: - rows = _read_depth_validation_rows(dataset_root) - by_level_representation: dict[str, dict[str, list[dict[str, Any]]]] = { - level: {} for level in _LEVELS - } - for row in rows: - level = str(row["level"]).upper() - if level not in by_level_representation: - raise ValueError(f"unexpected validation level: {level!r}") - by_level_representation[level].setdefault( - str(row["representation_id"]), - [], - ).append(row) - - selected: list[dict[str, Any]] = [] - for level in _LEVELS: - representations = by_level_representation[level] - ranked_representations = sorted( - representations, - key=lambda value: hashlib.sha256( - f"{_SELECTION_SEED}\0{level}\0{value}".encode() - ).hexdigest(), - ) - if len(ranked_representations) < _DEV_ROWS_PER_LEVEL: - raise ValueError( - f"{level} has only {len(ranked_representations)} representations" - ) - for representation_id in ranked_representations[:_DEV_ROWS_PER_LEVEL]: - row = min( - representations[representation_id], - key=lambda value: _row_rank(value, namespace=level), - ) - sampler = sampler_from_row(row) - observation = observation_from_row(row) - reference = reference_from_row(row) - selected.append( - { - "task_id": sampler.opaque_id, - "source": "depth/validation", - "level": level, - "representation_id": representation_id, - "image_sha256": observation.image_sha256, - "target_image_sha256": reference.target_image_sha256, - "footprint_um": list(observation.footprint_um), - } - ) - return selected - - -def _benchmark_entry(repo_root: Path, index: int) -> dict[str, Any]: - relative = Path("data") / "benchmark" / f"final_{index}" - root = repo_root / relative - image_path = root / "device_bw.png" - footprint_path = root / "footprint.json" - meta_path = root / "meta.json" - image = image_path.read_bytes() - footprint_doc = json.loads(footprint_path.read_text(encoding="utf-8")) - meta = json.loads(meta_path.read_text(encoding="utf-8")) - footprint = [float(value) for value in footprint_doc["footprint_um"]] - digest = hashlib.sha256(image).hexdigest() - with Image.open(image_path) as raster: - image_size = [int(raster.width), int(raster.height)] - return { - "task_id": f"F{index}", - "source": str(relative), - "source_id": str(meta["id"]), - "level": "BENCHMARK", - "representation_id": f"benchmark-f{index}", - "image_sha256": digest, - "target_image_sha256": digest, - "footprint_um": footprint, - "image_size_px": image_size, - } - - -def build_task_manifest(repo_root: Path) -> dict[str, Any]: - root = repo_root.expanduser().resolve() - manifest = { - "schema_version": TASK_MANIFEST_SCHEMA, - "selection": { - "seed": _SELECTION_SEED, - "algorithm": ( - "sha256-rank representations per level, then sha256-rank held-out " - "realizations within each selected representation" - ), - "depth_validation_rows_per_level": _DEV_ROWS_PER_LEVEL, - }, - "task_sets": { - OPERATING_POINT_TASK_SET: _dev_manifest_entries(root / "dataset"), - BENCHMARK_TASK_SET: [ - _benchmark_entry(root, index) for index in range(1, 9) - ], - }, - } - manifest["logical_sha256"] = canonical_json_sha256(manifest) - return manifest - - -def build_benchmark_task_manifest(repo_root: Path) -> dict[str, Any]: - """Bind only the fixed repository benchmark without touching dataset rows.""" - - root = repo_root.expanduser().resolve() - manifest = { - "schema_version": TASK_MANIFEST_SCHEMA, - "selection": { - "algorithm": ( - "ordered fixed repository fixtures " - "data/benchmark/final_1 through final_8" - ), - "seed": None, - }, - "task_sets": { - BENCHMARK_TASK_SET: [ - _benchmark_entry(root, index) for index in range(1, 9) - ], - }, - } - manifest["logical_sha256"] = canonical_json_sha256(manifest) - return manifest - - -def load_task_manifest(path: Path, *, repo_root: Path) -> dict[str, Any]: - manifest = json.loads(path.read_text(encoding="utf-8")) - if manifest.get("schema_version") != TASK_MANIFEST_SCHEMA: - raise ValueError("unsupported evaluation task-manifest schema") - logical_sha = manifest.pop("logical_sha256", None) - observed_sha = canonical_json_sha256(manifest) - manifest["logical_sha256"] = logical_sha - if logical_sha != observed_sha: - raise ValueError("evaluation task-manifest logical SHA-256 mismatch") - task_sets = set(manifest.get("task_sets", {})) - if task_sets == {BENCHMARK_TASK_SET}: - expected = build_benchmark_task_manifest(repo_root) - elif task_sets == {OPERATING_POINT_TASK_SET, BENCHMARK_TASK_SET}: - expected = build_task_manifest(repo_root) - else: - raise ValueError( - f"unsupported evaluation task-set collection: {sorted(task_sets)}" - ) - if manifest != expected: - raise ValueError("evaluation task manifest does not match frozen sources") - return manifest - - -def _task_from_depth_row(row: dict[str, Any]) -> EvaluationTask: - sampler = sampler_from_row(row) - return EvaluationTask( - task_id=sampler.opaque_id, - level=sampler.level.upper(), - representation_id=sampler.representation_id, - observation=observation_from_row(row), - reference=reference_from_row(row), - ) - - -def _load_depth_tasks(dataset_root: Path) -> dict[str, EvaluationTask]: - return { - task.task_id: task - for task in ( - _task_from_depth_row(row) - for row in _read_depth_validation_rows(dataset_root) - ) - } - - -def _load_benchmark_task(repo_root: Path, entry: dict[str, Any]) -> EvaluationTask: - root = repo_root / entry["source"] - image = (root / "device_bw.png").read_bytes() - digest = hashlib.sha256(image).hexdigest() - footprint = tuple(float(value) for value in entry["footprint_um"]) - if digest != entry["image_sha256"]: - raise ValueError(f"{entry['task_id']} benchmark image digest changed") - observation = ModelObservation( - image_bytes=image, - footprint_um=footprint, - image_sha256=digest, - ) - reference = VerifierReference( - target_image_bytes=image, - footprint_um=footprint, - target_image_sha256=digest, - ) - return EvaluationTask( - task_id=str(entry["task_id"]), - level="BENCHMARK", - representation_id=str(entry["representation_id"]), - observation=observation, - reference=reference, - ) - - -def load_task_set( - *, - repo_root: Path, - manifest: dict[str, Any], - task_set: str, -) -> list[EvaluationTask]: - entries = list(manifest["task_sets"].get(task_set, ())) - if not entries: - raise ValueError(f"unknown or empty task set: {task_set!r}") - if task_set == OPERATING_POINT_TASK_SET: - available = _load_depth_tasks(repo_root / "dataset") - tasks = [] - for entry in entries: - task = available.get(str(entry["task_id"])) - if task is None: - raise ValueError(f"missing depth task {entry['task_id']}") - tasks.append(task) - elif task_set == BENCHMARK_TASK_SET: - tasks = [_load_benchmark_task(repo_root, entry) for entry in entries] - else: - raise ValueError(f"unsupported task set: {task_set!r}") - - for entry, task in zip(entries, tasks, strict=True): - observed = { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "image_sha256": task.observation.image_sha256, - "target_image_sha256": task.reference.target_image_sha256, - "footprint_um": list(task.observation.footprint_um), - } - for key, value in observed.items(): - if entry.get(key) != value: - raise ValueError( - f"{task.task_id} manifest mismatch for {key}: " - f"{entry.get(key)!r} != {value!r}" - ) - return tasks - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, required=True) - result.add_argument("--output", type=Path, required=True) - result.add_argument("--benchmark-only", action="store_true") - return result - - -def main() -> None: - args = parser().parse_args() - document = ( - build_benchmark_task_manifest(args.repo_root) - if args.benchmark_only - else build_task_manifest(args.repo_root) - ) - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text( - json.dumps(document, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - - -if __name__ == "__main__": - main() diff --git a/rl/evaluation/tests/test_attempt_store.py b/rl/evaluation/tests/test_attempt_store.py deleted file mode 100644 index e085765e..00000000 --- a/rl/evaluation/tests/test_attempt_store.py +++ /dev/null @@ -1,239 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -from pathlib import Path - -import pytest - -from rl.evaluation.attempt_store import ( - AttemptKey, - AttemptStore, - ImmutableRecordError, - InvalidRecordError, - ManifestDriftError, - RunKey, - RunLockedError, - RunLockRequiredError, - UnsafeStoreRootError, -) - - -def _digest(value: str) -> str: - return hashlib.sha256(value.encode("utf-8")).hexdigest() - - -def _manifest(**updates: object) -> dict: - value = { - "source_git_sha": "a" * 40, - "protocol_sha256": _digest("protocol"), - "task_set_sha256": _digest("tasks"), - "prompt_sha256": _digest("prompt"), - "sandbox_image_ref": "sha256:" + "b" * 64, - "sandbox_image_id": "sha256:" + "b" * 64, - "runtime": { - "python": "3.13.14", - "packages": {"tinker": "0.22.7"}, - "tinker_cookbook_commit": "c" * 40, - }, - "model_binding": { - "provider": "tinker", - "model": "Qwen/Qwen3.6-35B-A3B", - "renderer": "qwen3_5_disable_thinking", - "temperature": 1.0, - "top_p": 1.0, - "max_output_tokens": 4096, - }, - } - value.update(updates) - return value - - -@pytest.fixture -def roots(tmp_path: Path) -> tuple[Path, Path]: - repo = tmp_path / "repo" - repo.mkdir() - return repo, tmp_path / "external-record" - - -def test_store_rejects_repository_output_and_unsafe_keys( - roots: tuple[Path, Path], -) -> None: - repo, _ = roots - with pytest.raises(UnsafeStoreRootError): - AttemptStore(repo_root=repo, external_root=repo / "runs") - with pytest.raises(UnsafeStoreRootError): - AttemptStore(repo_root=repo, external_root=repo.parent) - with pytest.raises(ValueError): - AttemptKey("study", "hypothesis", "arm", "../escape", 1) - with pytest.raises(ValueError): - AttemptKey("study", "hypothesis", "arm", "task", 0) - - -def test_manifest_is_create_or_verify_and_paths_are_stable( - roots: tuple[Path, Path], -) -> None: - repo, external = roots - store = AttemptStore(repo_root=repo, external_root=external) - run_key = RunKey("study-v1", "H-QWEN-ZERO", "off-4k") - first = store.create_or_verify_run_manifest(run_key, _manifest()) - second = store.create_or_verify_run_manifest(run_key, _manifest()) - - assert first == second - assert str(first.relative_path) == ( - "studies/study-v1/hypotheses/H-QWEN-ZERO/" - "arms/off-4k/run_manifest.json" - ) - with pytest.raises(ManifestDriftError): - store.create_or_verify_run_manifest( - run_key, - _manifest(prompt_sha256=_digest("changed prompt")), - ) - assert store.load_run_manifest(run_key) == first - - -def test_resume_uses_receipt_without_calling_sampler_and_records_evaluation( - roots: tuple[Path, Path], -) -> None: - repo, external = roots - store = AttemptStore(repo_root=repo, external_root=external) - key = AttemptKey("study-v1", "H-QWEN-ZERO", "off-4k", "F1", 1) - store.create_or_verify_run_manifest(key.run_key, _manifest()) - calls = 0 - - def sample() -> dict: - nonlocal calls - calls += 1 - return { - "raw_completion": "print('first')", - "completion_sha256": _digest("print('first')"), - "finish_reason": "stop", - "output_tokens": 7, - "latency_seconds": 0.25, - } - - with pytest.raises(RunLockRequiredError): - store.resume_or_sample(key, sample) - with store.acquire_run_lock(key.run_key): - created = store.resume_or_sample(key, sample) - resumed = store.resume_or_sample( - key, - lambda: pytest.fail("a saved receipt must never be resampled"), - ) - - assert calls == 1 - assert created.resumed is False - assert resumed.resumed is True - assert resumed.record == created.record - assert str(created.record.relative_path).endswith( - "tasks/F1/attempts/000001/sampling_receipt.json" - ) - - evaluation = store.write_evaluation( - key, - { - "extracted_code": "print('first')", - "raw_iou": 0.75, - "measured_dice": None, - "render_sha256": None, - "pure_executable_rate": 1.0, - }, - ) - assert store.load_evaluation(key) == evaluation - with pytest.raises(ImmutableRecordError): - store.write_evaluation( - key, - { - "extracted_code": "print('different')", - "raw_iou": 0.8, - "pure_executable_rate": 1.0, - }, - ) - - -def test_evaluator_failures_are_diagnostic_and_do_not_block_final_evaluation( - roots: tuple[Path, Path], -) -> None: - repo, external = roots - store = AttemptStore(repo_root=repo, external_root=external) - key = AttemptKey("study-v1", "H-QWEN-ZERO", "off-4k", "F1", 1) - store.write_sampling_receipt(key, {"raw_completion": "print('first')"}) - diagnostic = { - "status": "worker_error", - "attribution": "evaluator", - "error": "container unavailable", - "retryable": True, - "program_sha256": _digest("print('first')"), - "reference_sha256": _digest("reference"), - "sampling_record_sha256": _digest("receipt"), - } - first = store.write_evaluator_failure(key, diagnostic) - repeated = store.write_evaluator_failure(key, diagnostic) - - assert first == repeated - assert store.load_evaluation(key) is None - assert store.list_evaluator_failures(key) == (first,) - - final = store.write_evaluation( - key, - { - "status": "syntax_error", - "attribution": "model", - "aggregation_iou": 0.0, - }, - ) - assert store.load_evaluation(key) == final - assert store.list_evaluator_failures(key) == (first,) - - -def test_run_lock_is_nonblocking_across_store_instances( - roots: tuple[Path, Path], -) -> None: - repo, external = roots - first = AttemptStore(repo_root=repo, external_root=external) - second = AttemptStore(repo_root=repo, external_root=external) - run_key = RunKey("study-v1", "H-QWEN-ZERO", "off-4k") - - with first.acquire_run_lock(run_key): - with pytest.raises(RunLockedError): - with second.acquire_run_lock(run_key): - pass - with second.acquire_run_lock(run_key): - pass - - -@pytest.mark.parametrize( - "payload", - [ - {"raw_completion": "x", "raw_iou": float("nan")}, - {"raw_completion": "x", "raw_iou": 1.01}, - {"raw_completion": "x", "completion_sha256": "not-a-hash"}, - {"raw_completion": "x", "output_tokens": -1}, - {"raw_completion": "x", "temperature": 2.01}, - ], -) -def test_receipts_reject_nonfinite_out_of_range_and_bad_hashes( - roots: tuple[Path, Path], - payload: dict, -) -> None: - repo, external = roots - store = AttemptStore(repo_root=repo, external_root=external) - key = AttemptKey("study-v1", "H-QWEN-ZERO", "off-4k", "F1", 1) - with pytest.raises(InvalidRecordError): - store.write_sampling_receipt(key, payload) - - -def test_read_detects_tampering_through_record_hash( - roots: tuple[Path, Path], -) -> None: - repo, external = roots - store = AttemptStore(repo_root=repo, external_root=external) - key = AttemptKey("study-v1", "H-QWEN-ZERO", "off-4k", "F1", 1) - stored = store.write_sampling_receipt(key, {"raw_completion": "original"}) - path = external / Path(*stored.relative_path.parts) - document = json.loads(path.read_text(encoding="utf-8")) - document["payload"]["raw_completion"] = "tampered" - path.write_text(json.dumps(document), encoding="utf-8") - - with pytest.raises(InvalidRecordError, match="payload hash mismatch"): - store.load_sampling_receipt(key) diff --git a/rl/evaluation/tests/test_protocol.py b/rl/evaluation/tests/test_protocol.py deleted file mode 100644 index 8700b464..00000000 --- a/rl/evaluation/tests/test_protocol.py +++ /dev/null @@ -1,215 +0,0 @@ -from __future__ import annotations - -import json -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.evaluation.protocol import ( - deterministic_seed, - jobs_for_wave, - load_archived_protocol, - load_protocol, - resolved_arm_ids, -) -from rl.evaluation.tasks import load_task_manifest -from rl.studies.representation_curriculum_v1 import ( - run_baseline as archived_runner, -) - - -REPO_ROOT = Path(__file__).resolve().parents[3] -TASK_MANIFEST = ( - REPO_ROOT / "rl/studies/representation_curriculum_v1/task_manifest.json" -) -BASE_PROTOCOL = TASK_MANIFEST.with_name("protocol.json") -QWEN32_PROTOCOL = ( - TASK_MANIFEST.parent / "extensions/qwen32/protocol.json" -) -QWEN60_PROTOCOL = ( - TASK_MANIFEST.parent / "extensions/qwen60/protocol.json" -) - - -def _protocol() -> dict: - return load_archived_protocol(REPO_ROOT, protocol_file=BASE_PROTOCOL) - - -def _selection( - selected: str, - *, - complete: bool = True, - protocol: dict | None = None, -) -> dict: - protocol = protocol or _protocol() - return { - "schema_version": "pixcell-operating-point-selection-v1", - "study_id": protocol["study_id"], - "protocol_sha256": protocol["logical_sha256"], - "source_hypothesis": "RC-H00", - "selected_arm": selected, - "rescue_required": False, - "rescue_run": False, - "complete": complete, - } - - -def _task_ids(task_set: str) -> list[str]: - manifest = load_task_manifest(TASK_MANIFEST, repo_root=REPO_ROOT) - return [value["task_id"] for value in manifest["task_sets"][task_set]] - - -def test_protocol_freezes_baseline_and_forbids_unfrozen_training(): - protocol = _protocol() - assert protocol["study_id"] == "representation-curriculum-v1" - assert protocol["hypotheses"]["RC-H00"]["status"] == "frozen" - assert protocol["hypotheses"]["RC-H01"]["status"] == "frozen" - assert protocol["hypotheses"]["RC-H02"]["status"] == "frozen" - assert all( - value["status"] == "launch_forbidden_until_protocol_frozen" - for value in protocol["future_hypothesis_registry"].values() - ) - - -def test_qwen_factorial_is_matched_and_seed_paired(): - protocol = _protocol() - arms = [ - protocol["arms"][value] - for value in protocol["hypotheses"]["RC-H00"]["arm_ids"] - ] - assert {(value["thinking"], value["max_output_tokens"]) for value in arms} == { - (False, 4096), - (False, 16384), - (True, 4096), - (True, 16384), - } - first = deterministic_seed(protocol, task_id="same", attempt_index=1) - assert first == deterministic_seed( - protocol, - task_id="same", - attempt_index=1, - ) - assert first != deterministic_seed( - protocol, - task_id="same", - attempt_index=2, - ) - - -def test_wave_job_counts_are_exact(): - protocol = _protocol() - dev = _task_ids("qwen_operating_point") - bench = _task_ids("f1_f8") - assert len( - jobs_for_wave( - protocol, - hypothesis_id="RC-H00", - wave_name="smoke", - task_ids=dev, - ) - ) == 4 - assert len( - jobs_for_wave( - protocol, - hypothesis_id="RC-H00", - wave_name="complete", - task_ids=dev, - ) - ) == 320 - assert len( - jobs_for_wave( - protocol, - hypothesis_id="RC-H02", - wave_name="smoke", - task_ids=bench, - ) - ) == 8 - assert len( - jobs_for_wave( - protocol, - hypothesis_id="RC-H02", - wave_name="complete", - task_ids=bench, - ) - ) == 32 - - -def test_qwen_baseline_keeps_incumbent_control_when_winner_differs(): - protocol = _protocol() - assert resolved_arm_ids( - protocol, - "RC-H01", - selection=_selection("qwen-off-4096", protocol=protocol), - ) == ["qwen-off-4096"] - assert resolved_arm_ids( - protocol, - "RC-H01", - selection=_selection("qwen-on-16384", protocol=protocol), - ) == ["qwen-off-4096", "qwen-on-16384"] - - -def test_conditional_rescue_is_fail_closed(): - protocol = _protocol() - with pytest.raises(ValueError, match="forbidden"): - resolved_arm_ids( - protocol, - "RC-H00R", - selection=_selection("qwen-on-16384", protocol=protocol), - ) - triggered = _selection( - "qwen-on-16384", - complete=False, - protocol=protocol, - ) - triggered["rescue_required"] = True - assert resolved_arm_ids( - protocol, - "RC-H00R", - selection=triggered, - ) == ["qwen-on-32768-confirmation"] - - -@pytest.mark.parametrize( - "protocol_file", - (BASE_PROTOCOL, QWEN32_PROTOCOL, QWEN60_PROTOCOL), -) -def test_superseded_v1_protocols_are_rejected_for_paid_launch( - protocol_file: Path, -): - with pytest.raises( - ValueError, - match="prompt contract does not match the runtime", - ): - load_protocol(REPO_ROOT, protocol_file=protocol_file) - - -def test_archived_runner_binds_its_own_protocol( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, - capsys: pytest.CaptureFixture[str], -): - captured: dict[str, object] = {} - - async def fake_run_baseline(**kwargs): - captured.update(kwargs) - return {"summaries": {}} - - class FakeParser: - @staticmethod - def parse_args(): - return SimpleNamespace( - hypothesis="RC-H02", - wave="smoke", - expected_source_sha="a" * 40, - confirm_spend="PIXCELL_PHASE1_BASELINE", - external_root=tmp_path, - selection=None, - ) - - monkeypatch.setattr(archived_runner, "run_baseline", fake_run_baseline) - monkeypatch.setattr(archived_runner, "parser", FakeParser) - archived_runner.main() - assert captured["protocol_file"] == BASE_PROTOCOL - assert captured["hypothesis_id"] == "RC-H02" - assert json.loads(capsys.readouterr().out) == {} diff --git a/rl/evaluation/tests/test_runner.py b/rl/evaluation/tests/test_runner.py deleted file mode 100644 index 495035aa..00000000 --- a/rl/evaluation/tests/test_runner.py +++ /dev/null @@ -1,1104 +0,0 @@ -from __future__ import annotations - -import asyncio -import hashlib -import io -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest -from PIL import Image - -from rl.common.contracts import ModelObservation, VerifierReference -from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus -from rl.evaluation import runner -from rl.evaluation.attempt_store import AttemptKey, AttemptStore, RunKey -from rl.evaluation.protocol import ArmSpec, JobSpec -from rl.evaluation.samplers import SampleResult -from rl.evaluation.tasks import EvaluationTask - - -def _image_bytes() -> bytes: - image = Image.new("L", (4, 4), color=255) - image.putpixel((1, 1), 0) - buffer = io.BytesIO() - image.save(buffer, format="PNG") - return buffer.getvalue() - - -def _task(task_id: str = "F1") -> EvaluationTask: - image = _image_bytes() - digest = hashlib.sha256(image).hexdigest() - return EvaluationTask( - task_id=task_id, - level="BENCHMARK", - representation_id=f"benchmark-{task_id.lower()}", - observation=ModelObservation( - image_bytes=image, - footprint_um=(4.0, 4.0), - image_sha256=digest, - ), - reference=VerifierReference( - target_image_bytes=image, - footprint_um=(4.0, 4.0), - target_image_sha256=digest, - ), - ) - - -def _arm(arm_id: str = "qwen-off-4096") -> ArmSpec: - return ArmSpec( - arm_id=arm_id, - provider="tinker", - model="Qwen/Qwen3.6-35B-A3B", - renderer="qwen3_5_disable_thinking", - thinking=False, - thinking_effort=None, - max_output_tokens=4096, - context_tokens=65536, - max_image_long_edge=1440, - ) - - -def _job( - *, - arm_id: str = "qwen-off-4096", - task_id: str = "F1", - attempt_index: int = 1, -) -> JobSpec: - return JobSpec( - hypothesis_id="RC-H01", - arm_id=arm_id, - task_id=task_id, - attempt_index=attempt_index, - seed=20260727, - ) - - -def _protocol() -> dict[str, Any]: - return { - "study_id": "representation-curriculum-v1", - "launch": {"confirmation_token": "RUN_PHASE1_BASELINES"}, - "sampling": { - "sampling_concurrency": 2, - "evaluator_workers": 1, - "temperature": 1.0, - "top_p": 1.0, - }, - "pricing": { - "models": { - "Qwen/Qwen3.6-35B-A3B": { - "prefill_cached": 0.108, - "prefill_uncached": 0.54, - "sample": 1.335, - } - } - }, - "tracking": { - "project": "pixcell-test", - "entity": None, - }, - } - - -def _store(tmp_path: Path) -> AttemptStore: - repo = tmp_path / "repo" - repo.mkdir() - return AttemptStore( - repo_root=repo, - external_root=tmp_path / "external", - ) - - -def _sample_result(program: str = "print('sample')") -> SampleResult: - return SampleResult( - prompt_tokens=100, - completion_tokens=10, - stop_reason="stop", - cap_hit=False, - raw_text=program, - completion_text=program, - reasoning_text="", - answer_text=program, - reasoning_tokens_exact=0, - answer_tokens_exact=10, - reasoning_tokens_estimate=None, - answer_tokens_estimate=None, - channel_token_count_basis="test", - channel_parse_complete=True, - ) - - -def _binding( - *, - protocol: dict[str, Any], - arm: ArmSpec, - hypothesis_id: str, - source_sha: str = "a" * 40, -) -> dict[str, Any]: - return { - "hypothesis_id": hypothesis_id, - "source_git_sha": source_sha, - "protocol_sha256": "b" * 64, - "task_set_sha256": "c" * 64, - "prompt_sha256": "d" * 64, - "arms": { - arm.arm_id: { - "model": arm.model, - "renderer": arm.renderer, - "thinking": arm.thinking, - "thinking_effort": arm.thinking_effort, - "max_output_tokens": arm.max_output_tokens, - "context_tokens": arm.context_tokens, - "max_image_long_edge": arm.max_image_long_edge, - } - }, - "sampling": protocol["sampling"], - "runtime": { - "python": "3.13.14", - "packages": {"tinker": "0.22.7"}, - "tinker_cookbook_commit": "f" * 40, - }, - "sandbox": { - "image_ref": "sha256:" + "e" * 64, - "image_id": "sha256:" + "e" * 64, - }, - } - - -def _manifest_record( - *, - store: AttemptStore, - protocol: dict[str, Any], - arm: ArmSpec, - hypothesis_id: str, - binding: dict[str, Any], -) -> Any: - run_key = RunKey(protocol["study_id"], hypothesis_id, arm.arm_id) - with store.acquire_run_lock(run_key): - return store.create_or_verify_run_manifest( - run_key, - runner._run_manifest(binding, arm.arm_id), - ) - - -def _receipt_payload( - *, - protocol: dict[str, Any], - arm: ArmSpec, - job: JobSpec, - manifest_sha: str, - prompt_tokens: int = 100, -) -> dict[str, Any]: - sample = _sample_result().to_dict() - sample["prompt_tokens"] = prompt_tokens - return { - "request": runner._expected_request( - arm=arm, - protocol=protocol, - job=job, - ), - "sample": sample, - "sampling_seconds": 0.1, - "raw_completion_sha256": runner._sha256_text(sample["raw_text"]), - "completion_sha256": runner._sha256_text(sample["completion_text"]), - "run_manifest_record_sha256": manifest_sha, - "cost_estimate_usd": runner._cost_estimate( - protocol, - arm, - prompt_tokens=prompt_tokens, - completion_tokens=sample["completion_tokens"], - ), - } - - -def _evaluation_payload( - *, - task: EvaluationTask, - attribution: Attribution = Attribution.MODEL, - status: EvaluationStatus = EvaluationStatus.SYNTAX_ERROR, -) -> dict[str, Any]: - program = runner.extract_code(_sample_result().completion_text) - return { - "status": status.value, - "attribution": attribution.value, - "pure_executable": status is EvaluationStatus.OK, - "measurement_available": False, - "measured_iou": None, - "measured_dice": None, - "aggregation_iou": 0.0, - "program": program, - "program_sha256": runner._sha256_text(program), - "reference_sha256": task.reference.target_image_sha256, - "violations": [], - "error": "expected test failure", - "retryable": attribution is not Attribution.MODEL, - "evaluation_seconds": 0.1, - "diagnostics": {}, - } - - -class _CountingSampler: - def __init__(self) -> None: - self.calls = 0 - - def build_prompt(self, _messages: list[Any]) -> SimpleNamespace: - return SimpleNamespace(length=100) - - async def sample( - self, - _messages: list[Any], - _request: Any, - ) -> SampleResult: - self.calls += 1 - return _sample_result() - - -class _SavedRecordTracker: - def __init__(self, store: AttemptStore) -> None: - self.store = store - self.calls = 0 - - def mirror_saved_metrics( - self, - *, - key: AttemptKey, - source: str, - metrics: dict[str, int | float], - step: int, - ) -> SimpleNamespace: - assert source == "evaluation" - assert self.store.load_evaluation(key) is not None - assert metrics["attempt/iou"] >= 0.0 - assert step >= 0 - self.calls += 1 - return SimpleNamespace(mirrored=True) - - -def test_reference_preflight_failure_happens_before_remote_client_creation( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -) -> None: - """A local reference failure must remain a zero-spend failure.""" - - protocol = _protocol() - task = _task() - arm = _arm() - job = JobSpec( - hypothesis_id="RC-H02", - arm_id=arm.arm_id, - task_id=task.task_id, - attempt_index=1, - seed=20260727, - ) - source_sha = "a" * 40 - client_calls = 0 - repo_root = tmp_path / "repo" - repo_root.mkdir() - - binding = { - "hypothesis_id": job.hypothesis_id, - "source_git_sha": source_sha, - "protocol_sha256": "b" * 64, - "task_set_sha256": "c" * 64, - "prompt_sha256": "d" * 64, - "arms": {arm.arm_id: {"model": arm.model}}, - "sampling": protocol["sampling"], - "runtime": { - "python": "3.13.14", - "packages": {"tinker": "0.22.7"}, - "tinker_cookbook_commit": "f" * 40, - }, - "sandbox": { - "image_ref": "sha256:" + "e" * 64, - "image_id": "sha256:" + "e" * 64, - }, - } - - class FakeSampler: - def __init__( - self, - _target: Any, - *, - service_client_factory: Any, - ) -> None: - self.service_client_factory = service_client_factory - - def build_prompt(self, _messages: list[Any]) -> SimpleNamespace: - return SimpleNamespace(length=100) - - async def sample(self, _messages: list[Any], _request: Any) -> Any: - nonlocal client_calls - client_calls += 1 - self.service_client_factory() - raise AssertionError("sampling must not begin") - - class BadReferenceEvaluator: - max_workers = 1 - - def __init__(self, **kwargs: Any) -> None: - assert kwargs["require_isolation"] is True - - def __enter__(self) -> BadReferenceEvaluator: - return self - - def __exit__(self, *_args: object) -> None: - pass - - def validate_reference(self, _reference: VerifierReference) -> dict[str, Any]: - raise ValueError("broken local reference") - - monkeypatch.setattr(runner, "load_protocol", lambda _root, **_kwargs: protocol) - monkeypatch.setattr( - runner, - "build_launch_binding", - lambda **_kwargs: (binding, [task], [job]), - ) - monkeypatch.setattr(runner, "validate_credentials", lambda _protocol: None) - monkeypatch.setattr( - runner, - "validate_wandb_access", - lambda **_kwargs: {"username": "test", "entity": None}, - ) - monkeypatch.setattr( - runner, - "resolved_arm_ids", - lambda *_args, **_kwargs: [arm.arm_id], - ) - monkeypatch.setattr(runner, "arm_spec", lambda *_args, **_kwargs: arm) - monkeypatch.setattr(runner, "TinkerSampler", FakeSampler) - monkeypatch.setattr(runner, "PixCellEvaluator", BadReferenceEvaluator) - monkeypatch.setattr(runner, "_messages", lambda _task, _arm: [{"role": "user"}]) - - with pytest.raises(ValueError, match="broken local reference"): - asyncio.run( - runner.run_baseline( - repo_root=repo_root, - hypothesis_id=job.hypothesis_id, - wave_name="smoke", - expected_source_sha=source_sha, - external_root=tmp_path / "external", - confirmation=protocol["launch"]["confirmation_token"], - ) - ) - - assert client_calls == 0 - assert list((tmp_path / "external").rglob("sampling_receipt.json")) == [] - - -def test_receipt_precedes_evaluation_and_resume_never_resamples( - tmp_path: Path, -) -> None: - store = _store(tmp_path) - task = _task() - arm = _arm() - job = _job() - sampler = _CountingSampler() - protocol = _protocol() - key = runner._attempt_key(protocol["study_id"], job) - messages = {(arm.arm_id, task.task_id): [{"role": "user"}]} - - asyncio.run( - runner._sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=[job], - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm={arm.arm_id: "a" * 64}, - ) - ) - first = store.load_sampling_receipt(key) - assert first is not None - - asyncio.run( - runner._sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=[job], - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm={arm.arm_id: "a" * 64}, - ) - ) - assert sampler.calls == 1 - assert store.load_sampling_receipt(key) == first - - class ReceiptAwareEvaluator: - max_workers = 1 - - def evaluate_batch( - self, - requests: list[tuple[VerifierReference, str]], - ) -> list[EvaluationResult]: - assert store.load_sampling_receipt(key) == first - assert len(requests) == 1 - return [ - EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=0.75, - dice=0.8, - metrics={"render_sha256": "f" * 64}, - latency_seconds=0.1, - ) - ] - - tracker = _SavedRecordTracker(store) - runner._evaluate_missing( - store=store, - study_id=protocol["study_id"], - jobs=[job], - tasks=[task], - evaluator=ReceiptAwareEvaluator(), - tracker_by_arm={arm.arm_id: tracker}, - ) - assert store.load_evaluation(key) is not None - assert tracker.calls == 1 - - -def test_smoke_expands_to_complete_without_duplicate_sampling( - tmp_path: Path, -) -> None: - """The complete wave must reuse its smoke subset and add only missing jobs.""" - - store = _store(tmp_path) - protocol = _protocol() - tasks = [_task("F1"), _task("F2")] - arms = { - arm_id: _arm(arm_id) - for arm_id in ("qwen-off-4096", "qwen-off-16384") - } - samplers = {arm_id: _CountingSampler() for arm_id in arms} - messages = { - (arm_id, task.task_id): [{"role": "user"}] - for arm_id in arms - for task in tasks - } - smoke = [ - _job(arm_id=arm_id, task_id="F1", attempt_index=1) - for arm_id in arms - ] - complete = [ - _job( - arm_id=arm_id, - task_id=task.task_id, - attempt_index=attempt_index, - ) - for attempt_index in (1, 2) - for task in tasks - for arm_id in arms - ] - - for jobs in (smoke, complete, complete): - asyncio.run( - runner._sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=tasks, - arms=arms, - protocol=protocol, - samplers=samplers, - messages=messages, - run_manifest_record_sha256_by_arm={ - arm_id: "a" * 64 for arm_id in arms - }, - ) - ) - - assert {arm_id: sampler.calls for arm_id, sampler in samplers.items()} == { - "qwen-off-4096": 4, - "qwen-off-16384": 4, - } - assert len(list(store.external_root.rglob("sampling_receipt.json"))) == 8 - - -def test_sampling_wave_persists_siblings_before_raising_transport_failure( - tmp_path: Path, -) -> None: - """One failed request must not cancel paid sibling generations.""" - - store = _store(tmp_path) - protocol = _protocol() - protocol["sampling"]["sampling_concurrency"] = 4 - task = _task() - arm = _arm() - jobs = [ - JobSpec( - hypothesis_id="RC-H01", - arm_id=arm.arm_id, - task_id=task.task_id, - attempt_index=index, - seed=20260727 + index, - ) - for index in range(1, 5) - ] - started: list[int] = [] - - class OneFailureSampler(_CountingSampler): - async def sample( - self, - _messages: list[Any], - request: Any, - ) -> SampleResult: - self.calls += 1 - started.append(request.seed) - await asyncio.sleep(0) - if request.seed == jobs[0].seed: - raise ConnectionError("simulated transport failure") - return _sample_result() - - sampler = OneFailureSampler() - with pytest.raises(RuntimeError, match="sibling requests settled"): - asyncio.run( - runner._sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages={ - (arm.arm_id, task.task_id): [{"role": "user"}] - }, - run_manifest_record_sha256_by_arm={ - arm.arm_id: "a" * 64 - }, - ) - ) - - assert started == [job.seed for job in jobs] - assert store.load_sampling_receipt( - runner._attempt_key(protocol["study_id"], jobs[0]) - ) is None - assert all( - store.load_sampling_receipt( - runner._attempt_key(protocol["study_id"], job) - ) - is not None - for job in jobs[1:] - ) - - -def test_resume_preflight_rejects_stale_request_before_new_sampling( - tmp_path: Path, -) -> None: - """An immutable receipt cannot be silently adopted by another job.""" - - store = _store(tmp_path) - protocol = _protocol() - task = _task() - arm = _arm() - jobs = [_job(attempt_index=1), _job(attempt_index=2)] - binding = _binding( - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - ) - manifest = _manifest_record( - store=store, - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - binding=binding, - ) - stale = _receipt_payload( - protocol=protocol, - arm=arm, - job=jobs[0], - manifest_sha=manifest.record_sha256, - ) - stale["request"]["seed"] += 1 - store.write_sampling_receipt( - runner._attempt_key(protocol["study_id"], jobs[0]), - stale, - ) - sampler = _CountingSampler() - messages = {(arm.arm_id, task.task_id): [{"role": "user"}]} - - with pytest.raises(ValueError, match="exact current job"): - runner._validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm={ - arm.arm_id: manifest.record_sha256 - }, - ) - - assert sampler.calls == 0 - assert store.load_sampling_receipt( - runner._attempt_key(protocol["study_id"], jobs[1]) - ) is None - - -def test_complete_resume_rejects_saved_verifier_failure_before_sampling( - tmp_path: Path, -) -> None: - """A bad smoke evaluation must stop a complete wave before attempts 2-4.""" - - store = _store(tmp_path) - protocol = _protocol() - task = _task() - arm = _arm() - jobs = [_job(attempt_index=1), _job(attempt_index=2)] - binding = _binding( - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - ) - manifest = _manifest_record( - store=store, - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - binding=binding, - ) - key = runner._attempt_key(protocol["study_id"], jobs[0]) - store.write_sampling_receipt( - key, - _receipt_payload( - protocol=protocol, - arm=arm, - job=jobs[0], - manifest_sha=manifest.record_sha256, - ), - ) - store.write_evaluation( - key, - _evaluation_payload( - task=task, - attribution=Attribution.EVALUATOR, - status=EvaluationStatus.WORKER_ERROR, - ), - ) - sampler = _CountingSampler() - messages = {(arm.arm_id, task.task_id): [{"role": "user"}]} - - with pytest.raises(RuntimeError, match="refusing to buy more samples"): - runner._validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm={ - arm.arm_id: manifest.record_sha256 - }, - ) - - assert sampler.calls == 0 - assert store.load_sampling_receipt( - runner._attempt_key(protocol["study_id"], jobs[1]) - ) is None - - -def test_run_baseline_complete_fails_before_client_on_bad_smoke_record( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -) -> None: - """The paid entrypoint applies the resume audit before sampling.""" - - protocol = _protocol() - task = _task() - arm = _arm() - jobs = [ - JobSpec( - hypothesis_id="RC-H05", - arm_id=arm.arm_id, - task_id=task.task_id, - attempt_index=index, - seed=20260727 + index, - ) - for index in (1, 2) - ] - source_sha = "a" * 40 - binding = _binding( - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - source_sha=source_sha, - ) - repo_root = tmp_path / "repo" - repo_root.mkdir() - external_root = tmp_path / "external" - store = AttemptStore(repo_root=repo_root, external_root=external_root) - manifest = _manifest_record( - store=store, - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - binding=binding, - ) - key = runner._attempt_key(protocol["study_id"], jobs[0]) - store.write_sampling_receipt( - key, - _receipt_payload( - protocol=protocol, - arm=arm, - job=jobs[0], - manifest_sha=manifest.record_sha256, - ), - ) - store.write_evaluation( - key, - _evaluation_payload( - task=task, - attribution=Attribution.REFERENCE, - status=EvaluationStatus.REFERENCE_ERROR, - ), - ) - client_calls = 0 - - class FakeSampler: - def __init__( - self, - _target: Any, - *, - service_client_factory: Any, - ) -> None: - self.service_client_factory = service_client_factory - - def build_prompt(self, _messages: list[Any]) -> SimpleNamespace: - return SimpleNamespace(length=100) - - async def sample(self, _messages: list[Any], _request: Any) -> Any: - nonlocal client_calls - client_calls += 1 - self.service_client_factory() - raise AssertionError("resume audit must stop before sampling") - - class ValidReferenceEvaluator: - max_workers = 1 - - def __init__(self, **kwargs: Any) -> None: - assert kwargs["require_isolation"] is True - - def __enter__(self) -> ValidReferenceEvaluator: - return self - - def __exit__(self, *_args: object) -> None: - pass - - def validate_reference( - self, - reference: VerifierReference, - ) -> dict[str, Any]: - return {"target_image_sha256": reference.target_image_sha256} - - def evaluate_batch(self, _requests: Any) -> Any: - raise AssertionError("resume audit must stop before evaluation") - - monkeypatch.setattr(runner, "load_protocol", lambda _root, **_kwargs: protocol) - monkeypatch.setattr( - runner, - "build_launch_binding", - lambda **_kwargs: (binding, [task], jobs), - ) - monkeypatch.setattr(runner, "validate_credentials", lambda _protocol: None) - monkeypatch.setattr( - runner, - "validate_wandb_access", - lambda **_kwargs: {"username": "test", "entity": None}, - ) - monkeypatch.setattr( - runner, - "resolved_arm_ids", - lambda *_args, **_kwargs: [arm.arm_id], - ) - monkeypatch.setattr(runner, "arm_spec", lambda *_args, **_kwargs: arm) - monkeypatch.setattr(runner, "TinkerSampler", FakeSampler) - monkeypatch.setattr(runner, "PixCellEvaluator", ValidReferenceEvaluator) - monkeypatch.setattr( - runner, - "_messages", - lambda _task, _arm: [{"role": "user"}], - ) - - with pytest.raises(RuntimeError, match="refusing to buy more samples"): - asyncio.run( - runner.run_baseline( - repo_root=repo_root, - hypothesis_id=jobs[0].hypothesis_id, - wave_name="complete", - expected_source_sha=source_sha, - external_root=external_root, - confirmation=protocol["launch"]["confirmation_token"], - ) - ) - - assert client_calls == 0 - assert store.load_sampling_receipt( - runner._attempt_key(protocol["study_id"], jobs[1]) - ) is None - - -def test_complete_resume_reuses_valid_model_failure_and_samples_only_missing( - tmp_path: Path, -) -> None: - """A legitimate model failure remains evidence and does not poison resume.""" - - store = _store(tmp_path) - protocol = _protocol() - task = _task() - arm = _arm() - jobs = [_job(attempt_index=1), _job(attempt_index=2)] - binding = _binding( - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - ) - manifest = _manifest_record( - store=store, - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - binding=binding, - ) - key = runner._attempt_key(protocol["study_id"], jobs[0]) - store.write_sampling_receipt( - key, - _receipt_payload( - protocol=protocol, - arm=arm, - job=jobs[0], - manifest_sha=manifest.record_sha256, - ), - ) - store.write_evaluation(key, _evaluation_payload(task=task)) - sampler = _CountingSampler() - messages = {(arm.arm_id, task.task_id): [{"role": "user"}]} - manifest_shas = {arm.arm_id: manifest.record_sha256} - - runner._validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm=manifest_shas, - ) - asyncio.run( - runner._sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm=manifest_shas, - ) - ) - - assert sampler.calls == 1 - assert len(list(store.external_root.rglob("sampling_receipt.json"))) == 2 - - -def test_partial_receipt_is_evaluated_before_complete_wave_expands( - tmp_path: Path, -) -> None: - """A crash between sampling and scoring resumes without deadlock or spend.""" - - store = _store(tmp_path) - protocol = _protocol() - task = _task() - arm = _arm() - jobs = [_job(attempt_index=1), _job(attempt_index=2)] - binding = _binding( - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - ) - manifest = _manifest_record( - store=store, - protocol=protocol, - arm=arm, - hypothesis_id=jobs[0].hypothesis_id, - binding=binding, - ) - first_key = runner._attempt_key(protocol["study_id"], jobs[0]) - store.write_sampling_receipt( - first_key, - _receipt_payload( - protocol=protocol, - arm=arm, - job=jobs[0], - manifest_sha=manifest.record_sha256, - ), - ) - sampler = _CountingSampler() - messages = {(arm.arm_id, task.task_id): [{"role": "user"}]} - manifest_shas = {arm.arm_id: manifest.record_sha256} - - runner._validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm=manifest_shas, - require_receipt_evaluations=False, - ) - - class ModelFailureEvaluator: - max_workers = 1 - - def evaluate_batch( - self, - requests: list[tuple[VerifierReference, str]], - ) -> list[EvaluationResult]: - assert len(requests) == 1 - return [ - EvaluationResult( - status=EvaluationStatus.SYNTAX_ERROR, - attribution=Attribution.MODEL, - error="invalid syntax", - latency_seconds=0.1, - ) - ] - - runner._evaluate_missing( - store=store, - study_id=protocol["study_id"], - jobs=[jobs[0]], - tasks=[task], - evaluator=ModelFailureEvaluator(), - tracker_by_arm=None, - ) - assert store.load_evaluation(first_key) is not None - - runner._validate_existing_attempts_before_sampling( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm=manifest_shas, - require_receipt_evaluations=True, - ) - asyncio.run( - runner._sample_missing( - store=store, - study_id=protocol["study_id"], - jobs=jobs, - tasks=[task], - arms={arm.arm_id: arm}, - protocol=protocol, - samplers={arm.arm_id: sampler}, - messages=messages, - run_manifest_record_sha256_by_arm=manifest_shas, - ) - ) - assert sampler.calls == 1 - assert len(list(store.external_root.rglob("sampling_receipt.json"))) == 2 - - -def test_saved_verifier_failure_is_diagnostic_and_receipt_can_be_re_evaluated( - tmp_path: Path, -) -> None: - """Infrastructure failure is preserved without poisoning the paid receipt.""" - - store = _store(tmp_path) - task = _task() - arm = _arm() - job = _job() - protocol = _protocol() - key = runner._attempt_key(protocol["study_id"], job) - store.write_sampling_receipt( - key, - { - "sample": _sample_result().to_dict(), - "sampling_seconds": 0.1, - "cost_estimate_usd": { - "cached_prefill": 0.0, - "uncached_prefill": 0.0, - }, - }, - ) - - class FailingEvaluator: - max_workers = 1 - - def evaluate_batch( - self, - _requests: list[tuple[VerifierReference, str]], - ) -> list[EvaluationResult]: - return [ - EvaluationResult( - status=EvaluationStatus.WORKER_ERROR, - attribution=Attribution.EVALUATOR, - error="worker crashed", - latency_seconds=0.1, - retryable=True, - ) - ] - - tracker = _SavedRecordTracker(store) - with pytest.raises(RuntimeError, match="verifier/reference failures"): - runner._evaluate_missing( - store=store, - study_id=protocol["study_id"], - jobs=[job], - tasks=[task], - evaluator=FailingEvaluator(), - tracker_by_arm={arm.arm_id: tracker}, - ) - assert store.load_evaluation(key) is None - assert len(store.list_evaluator_failures(key)) == 1 - assert tracker.calls == 0 - - class RecoveredEvaluator: - max_workers = 1 - - def evaluate_batch( - self, - _requests: list[tuple[VerifierReference, str]], - ) -> list[EvaluationResult]: - return [ - EvaluationResult( - status=EvaluationStatus.SYNTAX_ERROR, - attribution=Attribution.MODEL, - error="model syntax error", - latency_seconds=0.1, - ) - ] - - runner._evaluate_missing( - store=store, - study_id=protocol["study_id"], - jobs=[job], - tasks=[task], - evaluator=RecoveredEvaluator(), - tracker_by_arm={arm.arm_id: tracker}, - ) - assert store.load_evaluation(key) is not None - assert len(store.list_evaluator_failures(key)) == 1 - assert tracker.calls == 1 diff --git a/rl/evaluation/tests/test_samplers.py b/rl/evaluation/tests/test_samplers.py deleted file mode 100644 index e1d3d1b5..00000000 --- a/rl/evaluation/tests/test_samplers.py +++ /dev/null @@ -1,450 +0,0 @@ -from __future__ import annotations - -import asyncio -from types import SimpleNamespace - -import pytest - -from rl.evaluation import samplers -from rl.evaluation.samplers import ( - INKLING_MODEL, - QWEN_MODEL, - SampleResult, - SamplerConfigurationError, - SamplerTarget, - SamplingProtocolError, - SamplingRequest, - TinkerSampler, -) - - -def test_target_binds_exactly_one_location_and_renderer_contract() -> None: - qwen = SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5_disable_thinking", - base_model=QWEN_MODEL, - ) - assert qwen.client_kwargs == {"base_model": QWEN_MODEL} - - checkpoint = SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5", - model_path="tinker://run/sampler_weights/000001", - ) - assert checkpoint.client_kwargs == {"model_path": "tinker://run/sampler_weights/000001"} - - with pytest.raises(SamplerConfigurationError, match="exactly one"): - SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5", - ) - with pytest.raises(SamplerConfigurationError, match="exactly one"): - SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5", - base_model=QWEN_MODEL, - model_path="tinker://run/sampler_weights/1", - ) - with pytest.raises(SamplerConfigurationError, match="exactly match"): - SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5", - base_model="Qwen/Qwen3.6-35B-A3B-alias", - ) - with pytest.raises(SamplerConfigurationError, match="requires one of"): - SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qml_v0", - base_model=QWEN_MODEL, - ) - - -def test_inkling_requires_explicit_valid_effort() -> None: - target = SamplerTarget( - model_name=INKLING_MODEL, - renderer_name="tml_v0", - base_model=INKLING_MODEL, - effort=0.9, - ) - assert target.effort == 0.9 - - for effort in (None, -0.1, 1.0, float("nan"), True): - with pytest.raises(SamplerConfigurationError, match="explicit finite"): - SamplerTarget( - model_name=INKLING_MODEL, - renderer_name="tml_v0", - base_model=INKLING_MODEL, - effort=effort, - ) - - -@pytest.mark.parametrize( - ("kwargs", "message"), - [ - ({"max_tokens": 0, "temperature": 1.0, "top_p": 1.0, "seed": 1}, "max_tokens"), - ({"max_tokens": 1, "temperature": -0.1, "top_p": 1.0, "seed": 1}, "temperature"), - ({"max_tokens": 1, "temperature": 1.0, "top_p": 0.0, "seed": 1}, "top_p"), - ({"max_tokens": 1, "temperature": 1.0, "top_p": 1.0, "seed": -1}, "seed"), - ], -) -def test_sampling_request_has_no_implicit_invalid_controls(kwargs, message) -> None: - with pytest.raises(SamplerConfigurationError, match=message): - SamplingRequest(**kwargs) - - -class _FakeTokenizer: - _decode = { - 1: "plan ", - 2: "carefully", - 99: "", - 3: "\n\n```python\npass\n```", - } - - def encode(self, text, *, add_special_tokens): - assert add_special_tokens is False - assert text == "" - return [99] - - def decode(self, tokens): - return "".join(self._decode[token] for token in tokens) - - -class _FakeRenderer: - def build_generation_prompt(self, messages): - return SimpleNamespace(length=17, messages=messages) - - def get_stop_sequences(self): - return [777] - - -def test_qwen_channel_accounting_is_an_exact_sampled_token_partition() -> None: - thinking = samplers._QwenCodec(_FakeRenderer(), _FakeTokenizer(), "qwen3_5") - decoded = thinking.decode([1, 2, 99, 3]) - assert decoded.reasoning_text == "plan carefully" - assert decoded.answer_text == "```python\npass\n```" - assert decoded.reasoning_tokens_exact == 3 - assert decoded.answer_tokens_exact == 1 - assert decoded.reasoning_tokens_exact + decoded.answer_tokens_exact == 4 - assert decoded.channel_parse_complete is True - - truncated = thinking.decode([1, 2]) - assert truncated.reasoning_tokens_exact == 2 - assert truncated.answer_tokens_exact == 0 - assert truncated.answer_text == "" - assert truncated.channel_parse_complete is False - - direct = samplers._QwenCodec( - _FakeRenderer(), - _FakeTokenizer(), - "qwen3_5_disable_thinking", - ).decode([3]) - assert direct.reasoning_tokens_exact == 0 - assert direct.answer_tokens_exact == 1 - assert direct.channel_parse_complete is True - - -def test_sampler_is_lazy_and_sends_exactly_one_explicit_sample(monkeypatch) -> None: - tinker = pytest.importorskip( - "tinker", - reason="Tinker SDK env (rl/requirements-lock.txt) not installed", - ) - - calls: dict[str, object] = {"factory": 0} - - class FakeCodec: - def build_prompt(self, messages): - calls["messages"] = messages - return SimpleNamespace(length=123) - - def stop_sequences(self): - return [456] - - def decode(self, tokens): - calls["decoded"] = list(tokens) - return samplers._DecodedCompletion( - raw_text="raw", - completion_text="answer", - reasoning_text="", - answer_text="answer", - reasoning_tokens_exact=0, - answer_tokens_exact=len(tokens), - reasoning_tokens_estimate=None, - answer_tokens_estimate=None, - channel_token_count_basis="test", - channel_parse_complete=True, - ) - - class FakeSamplingClient: - async def sample_async(self, **kwargs): - calls["sample"] = kwargs - return SimpleNamespace( - sequences=[SimpleNamespace(tokens=[10, 11], stop_reason="length")] - ) - - class FakeService: - def create_sampling_client(self, **kwargs): - calls["client"] = kwargs - return FakeSamplingClient() - - def factory(): - calls["factory"] = int(calls["factory"]) + 1 - return FakeService() - - monkeypatch.setattr(samplers, "_build_codec", lambda _target: FakeCodec()) - target = SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5_disable_thinking", - base_model=QWEN_MODEL, - ) - sampler = TinkerSampler(target, service_client_factory=factory) - assert calls["factory"] == 0 - - # Prompt validation and tokenization remain zero-client operations. - assert sampler.build_prompt([{"role": "user", "content": "prompt"}]).length == 123 - assert calls["factory"] == 0 - - result = asyncio.run( - sampler.sample( - [{"role": "user", "content": "prompt"}], - SamplingRequest( - max_tokens=8192, - temperature=1.0, - top_p=0.95, - seed=20260727, - ), - ) - ) - assert isinstance(result, SampleResult) - assert calls["factory"] == 1 - assert calls["client"] == {"base_model": QWEN_MODEL} - sample_call = calls["sample"] - assert sample_call["num_samples"] == 1 - params = sample_call["sampling_params"] - assert isinstance(params, tinker.SamplingParams) - assert params.max_tokens == 8192 - assert params.temperature == 1.0 - assert params.top_p == 0.95 - assert params.seed == 20260727 - assert params.stop == [456] - assert result.prompt_tokens == 123 - assert result.completion_tokens == 2 - assert result.stop_reason == "length" - assert result.cap_hit is True - assert result.raw_text == "raw" - assert result.answer_text == "answer" - - -def test_sampler_rejects_backend_multi_completion(monkeypatch) -> None: - pytest.importorskip( - "tinker", - reason="Tinker SDK env (rl/requirements-lock.txt) not installed", - ) - - class FakeCodec: - def build_prompt(self, _messages): - return SimpleNamespace(length=1) - - def stop_sequences(self): - return [2] - - class FakeSamplingClient: - async def sample_async(self, **_kwargs): - return SimpleNamespace( - sequences=[ - SimpleNamespace(tokens=[1], stop_reason="stop"), - SimpleNamespace(tokens=[2], stop_reason="stop"), - ] - ) - - class FakeService: - def create_sampling_client(self, **_kwargs): - return FakeSamplingClient() - - monkeypatch.setattr(samplers, "_build_codec", lambda _target: FakeCodec()) - sampler = TinkerSampler( - SamplerTarget( - model_name=QWEN_MODEL, - renderer_name="qwen3_5", - base_model=QWEN_MODEL, - ), - service_client_factory=FakeService, - ) - with pytest.raises(SamplingProtocolError, match="exactly one"): - asyncio.run( - sampler.sample( - [{"role": "user", "content": "prompt"}], - SamplingRequest( - max_tokens=8, - temperature=1.0, - top_p=1.0, - seed=1, - ), - ) - ) - - -def test_tml_renderer_uses_instance_effort_and_channel_estimates() -> None: - pytest.importorskip("tml_renderers") - pytest.importorskip( - "tinker", - reason="Tinker SDK env (rl/requirements-lock.txt) not installed", - ) - from tml_renderers import chat as tml_chat - from tml_renderers import tokenizers as tml_tokenizers - from tml_renderers import v0 as tml_v0 - - tokenizer = tml_tokenizers.o200k_base_chat() - codec = samplers._InklingCodec(tml_v0.Renderer(tokenizer), tokenizer, 0.9) - prompt = codec.build_prompt([{"role": "user", "content": "draw it"}]) - assert prompt.length > 0 - - model = tml_chat.Author(tml_chat.AuthorKind.Model) - messages = [ - tml_chat.Message(tml_chat.Thinking("inspect geometry"), model), - tml_chat.Message(tml_chat.Text("```python\npass\n```"), model), - tml_chat.Message(tml_chat.ModelEndSampling(), model), - ] - example = codec._renderer.render_for_sft(messages)[0] - sampled: list[int] = [] - for wrapper in example.input_token_spans: - inner = wrapper.span - if hasattr(inner, "tokens"): - sampled.extend(inner.tokens) - - decoded = codec.decode(sampled) - assert decoded.reasoning_text == "inspect geometry" - assert decoded.answer_text == "```python\npass\n```" - assert decoded.reasoning_tokens_exact is None - assert decoded.answer_tokens_exact is None - assert decoded.reasoning_tokens_estimate == len(tokenizer.encode_ordinary("inspect geometry")) - assert decoded.answer_tokens_estimate == len(tokenizer.encode_ordinary("```python\npass\n```")) - assert decoded.channel_token_count_basis.startswith("estimate_") - assert "excludes_tml_framing" in decoded.channel_token_count_basis - assert decoded.channel_parse_complete is True - - -class _FakeTmlTokenizer: - def __init__(self, raw_text: str) -> None: - self.raw_text = raw_text - - def decode(self, tokens): - assert list(tokens) == [11, 12] - return self.raw_text - - def encode_ordinary(self, text): - return list(text.encode("utf-8")) - - -class _FakeTmlParser: - def __init__(self, parsed=None, error: Exception | None = None) -> None: - self.parsed = [] if parsed is None else parsed - self.error = error - - def parse_tokens(self, tokens): - assert list(tokens) == [11, 12] - if self.error is not None: - raise self.error - return self.parsed - - -class _FakeTmlRenderer: - def __init__(self, parser: _FakeTmlParser) -> None: - self.parser = parser - - def render_for_completion(self, messages): - assert messages == [] - return [], self.parser - - -@pytest.mark.parametrize( - ("parsed", "error", "expected_basis"), - [ - ( - None, - ValueError("truncated TML object"), - "tml_parse_failed_raw_decode_fallback_channel_partition_unknown", - ), - ( - [], - None, - ( - "tml_parse_had_no_usable_channel_text_raw_decode_fallback_" - "channel_partition_unknown" - ), - ), - ], -) -def test_inkling_raw_decode_fallback_preserves_unparseable_completion( - parsed, - error, - expected_basis, -) -> None: - pytest.importorskip("tml_renderers") - raw = "partial framing followed by ```python\nprint('recover me')\n```" - codec = samplers._InklingCodec( - _FakeTmlRenderer(_FakeTmlParser(parsed=parsed, error=error)), - _FakeTmlTokenizer(raw), - 0.9, - ) - - decoded = codec.decode([11, 12]) - - assert decoded.raw_text == raw - assert decoded.completion_text == raw - assert decoded.reasoning_text is None - assert decoded.answer_text == "" - assert decoded.reasoning_tokens_exact is None - assert decoded.answer_tokens_exact is None - assert decoded.reasoning_tokens_estimate is None - assert decoded.answer_tokens_estimate is None - assert decoded.channel_token_count_basis == expected_basis - assert decoded.channel_parse_complete is False - - -def test_inkling_raw_decode_fallback_rejects_terminal_or_whitespace_as_usable() -> None: - pytest.importorskip("tml_renderers") - from tml_renderers import chat as tml_chat - - model = tml_chat.Author(tml_chat.AuthorKind.Model) - parsed = [ - tml_chat.Message(tml_chat.Thinking(" \n\t"), model), - tml_chat.Message(tml_chat.Text(" "), model), - tml_chat.Message(tml_chat.ModelEndSampling(), model), - ] - raw = "```python\nprint('raw survives')\n```" - codec = samplers._InklingCodec( - _FakeTmlRenderer(_FakeTmlParser(parsed=parsed)), - _FakeTmlTokenizer(raw), - 0.9, - ) - - decoded = codec.decode([11, 12]) - - assert decoded.completion_text == raw - assert decoded.channel_parse_complete is False - assert "parse_had_no_usable_channel_text" in decoded.channel_token_count_basis - - -def test_inkling_preserves_usable_parsed_channels_without_terminal_marker() -> None: - pytest.importorskip("tml_renderers") - from tml_renderers import chat as tml_chat - - model = tml_chat.Author(tml_chat.AuthorKind.Model) - parsed = [ - tml_chat.Message(tml_chat.Thinking("inspect"), model), - tml_chat.Message(tml_chat.Text("```python\npass\n```"), model), - ] - codec = samplers._InklingCodec( - _FakeTmlRenderer(_FakeTmlParser(parsed=parsed)), - _FakeTmlTokenizer("raw TML serialization"), - 0.9, - ) - - decoded = codec.decode([11, 12]) - - assert decoded.completion_text == "inspect\n```python\npass\n```" - assert decoded.reasoning_text == "inspect" - assert decoded.answer_text == "```python\npass\n```" - assert decoded.reasoning_tokens_estimate == len("inspect".encode("utf-8")) - assert decoded.answer_tokens_estimate == len("```python\npass\n```".encode("utf-8")) - assert decoded.channel_parse_complete is False diff --git a/rl/evaluation/tests/test_tasks.py b/rl/evaluation/tests/test_tasks.py deleted file mode 100644 index 427e1559..00000000 --- a/rl/evaluation/tests/test_tasks.py +++ /dev/null @@ -1,72 +0,0 @@ -from __future__ import annotations - -from collections import Counter -from pathlib import Path - -from rl.evaluation.tasks import ( - BENCHMARK_TASK_SET, - OPERATING_POINT_TASK_SET, - build_task_manifest, - load_task_manifest, - load_task_set, -) - - -REPO_ROOT = Path(__file__).resolve().parents[3] -MANIFEST_PATH = ( - REPO_ROOT / "rl/studies/representation_curriculum_v1/task_manifest.json" -) - - -def test_frozen_task_manifest_rebuilds_exactly(): - frozen = load_task_manifest(MANIFEST_PATH, repo_root=REPO_ROOT) - assert frozen == build_task_manifest(REPO_ROOT) - dev = frozen["task_sets"][OPERATING_POINT_TASK_SET] - benchmark = frozen["task_sets"][BENCHMARK_TASK_SET] - assert len(dev) == 40 - assert Counter(value["level"] for value in dev) == { - "L0": 8, - "L1": 8, - "L2": 8, - "L3": 8, - "L4": 8, - } - assert len({value["representation_id"] for value in dev}) == 40 - assert [value["task_id"] for value in benchmark] == [ - f"F{index}" for index in range(1, 9) - ] - - -def test_evaluation_loader_has_no_supervised_label(): - manifest = load_task_manifest(MANIFEST_PATH, repo_root=REPO_ROOT) - for task_set, expected in ( - (OPERATING_POINT_TASK_SET, 40), - (BENCHMARK_TASK_SET, 8), - ): - tasks = load_task_set( - repo_root=REPO_ROOT, - manifest=manifest, - task_set=task_set, - ) - assert len(tasks) == expected - assert set(tasks[0].__dataclass_fields__) == { - "task_id", - "level", - "representation_id", - "observation", - "reference", - } - assert not hasattr(tasks[0], "label") - - -def test_benchmark_model_and_reference_bytes_are_same_but_separate(): - manifest = load_task_manifest(MANIFEST_PATH, repo_root=REPO_ROOT) - tasks = load_task_set( - repo_root=REPO_ROOT, - manifest=manifest, - task_set=BENCHMARK_TASK_SET, - ) - for task in tasks: - assert task.observation.image_bytes == task.reference.target_image_bytes - assert task.observation.image_sha256 == task.reference.target_image_sha256 - assert task.observation is not task.reference diff --git a/rl/evaluation/tests/test_tracking.py b/rl/evaluation/tests/test_tracking.py deleted file mode 100644 index 1b56594f..00000000 --- a/rl/evaluation/tests/test_tracking.py +++ /dev/null @@ -1,382 +0,0 @@ -from __future__ import annotations - -import hashlib -import os -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.evaluation.attempt_store import AttemptKey, AttemptStore -from rl.evaluation.tracking import ( - TrackingContractError, - WandbMetricsMirror, - deterministic_wandb_run_id, - validate_wandb_access, -) - -SOURCE_GIT_SHA = "a" * 40 - - -class FakeSettings: - def __init__(self, **kwargs: object) -> None: - self.kwargs = kwargs - - -class FakeRun: - def __init__(self, owner: FakeWandb) -> None: - self.owner = owner - - def log( - self, - metrics: dict[str, int | float], - *, - step: int, - commit: bool, - ) -> None: - if self.owner.fail_log: - raise RuntimeError("log failed") - self.owner.logs.append((metrics, step, commit)) - - def finish(self) -> None: - self.owner.finished += 1 - if self.owner.fail_finish: - raise RuntimeError("finish failed") - - -class FakeWandb: - Settings = FakeSettings - - def __init__( - self, - *, - fail: bool = False, - fail_log: bool = False, - fail_finish: bool = False, - ) -> None: - self.fail = fail - self.fail_log = fail_log - self.fail_finish = fail_finish - self.init_calls: list[dict] = [] - self.logs: list[tuple[dict[str, int | float], int, bool]] = [] - self.finished = 0 - self.environment: dict[str, str | None] = {} - - def init(self, **kwargs: object) -> FakeRun: - self.init_calls.append(kwargs) - self.environment = { - name: os.environ.get(name) - for name in ( - "WANDB_DIR", - "WANDB_DISABLE_GIT", - "WANDB_DISABLE_CODE", - "WANDB_SAVE_CODE", - ) - } - if self.fail: - raise ConnectionError("offline") - return FakeRun(self) - - -def _store(tmp_path: Path) -> AttemptStore: - repo = tmp_path / "repo" - repo.mkdir() - return AttemptStore( - repo_root=repo, - external_root=tmp_path / "external-record", - ) - - -def _key() -> AttemptKey: - return AttemptKey("study-v1", "H-QWEN-ZERO", "off-4k", "F1", 1) - - -def test_wandb_run_id_is_bound_to_the_exact_source_commit() -> None: - first = deterministic_wandb_run_id( - _key().run_key, - source_git_sha="a" * 40, - ) - second = deterministic_wandb_run_id( - _key().run_key, - source_git_sha="b" * 40, - ) - - assert first != second - with pytest.raises(TrackingContractError, match="Git object SHA"): - deterministic_wandb_run_id( - _key().run_key, - source_git_sha="not-a-sha", - ) - - -def test_wandb_access_preflight_binds_the_expected_entity() -> None: - class FakeAccess: - @staticmethod - def Api(*, timeout: int) -> SimpleNamespace: - assert timeout == 20 - return SimpleNamespace( - viewer=SimpleNamespace(username="aadarwal", entity="qpaig-mit"), - default_entity="qpaig-mit", - ) - - assert validate_wandb_access( - expected_entity="qpaig-mit", - wandb_module=FakeAccess, - ) == {"username": "aadarwal", "entity": "qpaig-mit"} - with pytest.raises(TrackingContractError, match="entity mismatch"): - validate_wandb_access( - expected_entity="another-team", - wandb_module=FakeAccess, - ) - - -def test_wandb_access_preflight_fails_closed() -> None: - class OfflineAccess: - @staticmethod - def Api(*, timeout: int) -> None: - assert timeout == 20 - raise ConnectionError("offline") - - with pytest.raises(TrackingContractError, match="credential preflight"): - validate_wandb_access( - expected_entity=None, - wandb_module=OfflineAccess, - ) - - -def test_tracking_refuses_to_run_before_local_record_exists( - tmp_path: Path, -) -> None: - store = _store(tmp_path) - fake = FakeWandb() - mirror = WandbMetricsMirror( - store=store, - project="pixcell", - source_git_sha=SOURCE_GIT_SHA, - wandb_module=fake, - ) - - with pytest.raises(TrackingContractError, match="before local state"): - mirror.mirror_saved_metrics( - key=_key(), - source="sampling_receipt", - metrics={"output_tokens": 10}, - step=0, - ) - assert fake.init_calls == [] - - -def test_metrics_only_mirror_uses_resumable_id_and_disables_source_upload( - tmp_path: Path, -) -> None: - store = _store(tmp_path) - key = _key() - store.write_sampling_receipt( - key, - { - "raw_completion": "secret program", - "output_tokens": 10, - }, - ) - fake = FakeWandb() - mirror = WandbMetricsMirror( - store=store, - project="pixcell-baselines", - source_git_sha=SOURCE_GIT_SHA, - entity="qpaig-mit", - group="phase1-zero-shot", - wandb_module=fake, - ) - - result = mirror.mirror_saved_metrics( - key=key, - source="sampling_receipt", - metrics={"output_tokens": 10, "latency_seconds": 1.25}, - step=3, - ) - - assert result.mirrored is True - assert result.wandb_run_id == deterministic_wandb_run_id( - key.run_key, - source_git_sha=SOURCE_GIT_SHA, - ) - assert fake.logs == [ - ({"output_tokens": 10, "latency_seconds": 1.25}, 3, True) - ] - assert fake.finished == 1 - call = fake.init_calls[0] - assert call["id"] == result.wandb_run_id - assert call["reinit"] == "create_new" - assert call["resume"] == "allow" - assert call["group"] == "phase1-zero-shot" - assert call["dir"] == str(store.external_root / ".wandb") - assert "config" not in call - assert "secret program" not in repr(call) - assert call["settings"].kwargs == { - "disable_code": True, - "disable_git": True, - "disable_job_creation": True, - "save_code": False, - "x_disable_machine_info": True, - "x_disable_stats": True, - "x_save_requirements": False, - } - assert fake.environment == { - "WANDB_DIR": str(store.external_root / ".wandb"), - "WANDB_DISABLE_GIT": "true", - "WANDB_DISABLE_CODE": "true", - "WANDB_SAVE_CODE": "false", - } - - -def test_persistent_arm_session_initializes_once_and_finishes_once( - tmp_path: Path, -) -> None: - store = _store(tmp_path) - first = _key() - second = AttemptKey( - first.study_id, - first.hypothesis_id, - first.arm_id, - "F2", - 1, - ) - store.write_sampling_receipt(first, {"raw_completion": "first"}) - store.write_sampling_receipt(second, {"raw_completion": "second"}) - fake = FakeWandb() - mirror = WandbMetricsMirror( - store=store, - project="pixcell-baselines", - source_git_sha=SOURCE_GIT_SHA, - wandb_module=fake, - ) - - with mirror.session(first.run_key) as session: - first_result = session.mirror_saved_metrics( - key=first, - source="sampling_receipt", - metrics={"output_tokens": 10}, - step=0, - ) - second_result = session.mirror_saved_metrics( - key=second, - source="sampling_receipt", - metrics={"output_tokens": 20}, - step=1, - ) - - assert first_result.mirrored and second_result.mirrored - assert len(fake.init_calls) == 1 - assert len(fake.logs) == 2 - assert fake.finished == 1 - - -def test_tracking_failure_is_a_local_error_and_keeps_saved_sample( - tmp_path: Path, -) -> None: - store = _store(tmp_path) - key = _key() - saved = store.write_sampling_receipt( - key, - { - "raw_completion": "durable sample", - "output_tokens": 10, - }, - ) - mirror = WandbMetricsMirror( - store=store, - project="pixcell-baselines", - source_git_sha=SOURCE_GIT_SHA, - wandb_module=FakeWandb(fail=True), - ) - - result = mirror.mirror_saved_metrics( - key=key, - source="sampling_receipt", - metrics={"output_tokens": 10}, - step=0, - ) - - assert result.mirrored is False - assert store.load_sampling_receipt(key) == saved - errors = store.list_tracking_errors(key) - assert len(errors) == 1 - assert result.tracking_error_relative_path == str(errors[0].relative_path) - payload = errors[0].payload - assert payload["error_type"] == "ConnectionError" - assert payload["operation"] == "init" - assert payload["source_record_sha256"] == saved.record_sha256 - assert payload["metrics_sha256"] == hashlib.sha256( - b'{"output_tokens":10}' - ).hexdigest() - assert "durable sample" not in repr(payload) - - -@pytest.mark.parametrize( - ("fake", "operation"), - [ - (FakeWandb(fail_log=True), "log"), - (FakeWandb(fail_finish=True), "finish"), - ], -) -def test_log_and_finish_failures_each_create_local_error_records( - tmp_path: Path, - fake: FakeWandb, - operation: str, -) -> None: - store = _store(tmp_path) - key = _key() - saved = store.write_sampling_receipt(key, {"raw_completion": "durable"}) - mirror = WandbMetricsMirror( - store=store, - project="pixcell-baselines", - source_git_sha=SOURCE_GIT_SHA, - wandb_module=fake, - ) - - with mirror.session(key.run_key) as session: - result = session.mirror_saved_metrics( - key=key, - source="sampling_receipt", - metrics={"output_tokens": 10}, - step=0, - ) - - errors = store.list_tracking_errors(key) - assert len(errors) == 1 - assert errors[0].payload["operation"] == operation - assert store.load_sampling_receipt(key) == saved - if operation == "log": - assert result.mirrored is False - else: - # The metric reached W&B; only finalization failed afterward. - assert result.mirrored is True - - -def test_tracking_contract_rejects_bad_metrics_and_repo_wandb_dir( - tmp_path: Path, -) -> None: - store = _store(tmp_path) - with pytest.raises(TrackingContractError, match="outside"): - WandbMetricsMirror( - store=store, - project="pixcell", - source_git_sha=SOURCE_GIT_SHA, - wandb_dir=store.repo_root / "wandb", - wandb_module=FakeWandb(), - ) - key = _key() - store.write_sampling_receipt(key, {"raw_completion": "saved"}) - mirror = WandbMetricsMirror( - store=store, - project="pixcell", - source_git_sha=SOURCE_GIT_SHA, - wandb_module=FakeWandb(), - ) - with pytest.raises(TrackingContractError): - mirror.mirror_saved_metrics( - key=key, - source="sampling_receipt", - metrics={"iou": float("inf")}, - step=0, - ) diff --git a/rl/evaluation/tracking.py b/rl/evaluation/tracking.py deleted file mode 100644 index 0456116a..00000000 --- a/rl/evaluation/tracking.py +++ /dev/null @@ -1,430 +0,0 @@ -"""Metrics-only W&B mirroring for immutable local attempt records.""" - -from __future__ import annotations - -import hashlib -import importlib -import json -import math -import os -import re -import threading -from collections.abc import Mapping -from contextlib import contextmanager -from dataclasses import dataclass -from pathlib import Path -from types import ModuleType -from typing import Any, Literal - -from .attempt_store import AttemptKey, AttemptStore, RunKey, StoredRecord - - -TRACKING_SCHEMA_VERSION = "pixcell-baseline-wandb-mirror-v2" -_METRIC_NAME = re.compile(r"^[A-Za-z][A-Za-z0-9_./-]{0,127}$") -_SOURCE_GIT_SHA = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") -_WANDB_ENVIRONMENT_LOCK = threading.RLock() - - -class TrackingContractError(RuntimeError): - """Tracking was requested without a valid saved record or metric set.""" - - -@dataclass(frozen=True) -class TrackingResult: - mirrored: bool - wandb_run_id: str - source_record_sha256: str - tracking_error_relative_path: str | None = None - - -def deterministic_wandb_run_id( - run_key: RunKey, - *, - source_git_sha: str, -) -> str: - """Return the stable W&B run ID used for create-or-resume.""" - - if not _SOURCE_GIT_SHA.fullmatch(source_git_sha): - raise TrackingContractError("source_git_sha must be a Git object SHA") - canonical = json.dumps( - { - "schema_version": TRACKING_SCHEMA_VERSION, - "source_git_sha": source_git_sha, - **run_key.as_dict(), - }, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - return "pxc-" + hashlib.sha256(canonical).hexdigest()[:28] - - -def validate_wandb_access( - *, - expected_entity: str | None, - wandb_module: ModuleType | Any | None = None, -) -> dict[str, str | None]: - """Verify the configured W&B identity before any paid sampling begins.""" - - module = wandb_module or importlib.import_module("wandb") - try: - api = module.Api(timeout=20) - viewer = api.viewer - username = getattr(viewer, "username", None) - entity = getattr(viewer, "entity", None) or getattr( - api, - "default_entity", - None, - ) - except Exception as exc: - raise TrackingContractError("W&B credential preflight failed") from exc - if not isinstance(username, str) or not username.strip(): - raise TrackingContractError("W&B credential resolved no authenticated user") - if expected_entity is not None and entity != expected_entity: - raise TrackingContractError( - f"W&B entity mismatch: {entity!r} != {expected_entity!r}" - ) - return {"username": username, "entity": entity} - - -def _metrics_payload(metrics: Mapping[str, int | float]) -> dict[str, int | float]: - if not isinstance(metrics, Mapping) or not metrics: - raise TrackingContractError("tracking metrics must be a non-empty mapping") - result: dict[str, int | float] = {} - for key, value in metrics.items(): - if not isinstance(key, str) or not _METRIC_NAME.fullmatch(key): - raise TrackingContractError(f"invalid metric name: {key!r}") - if ( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or not math.isfinite(float(value)) - ): - raise TrackingContractError( - f"metric {key!r} must be a finite integer or float" - ) - result[key] = value - return result - - -def _metrics_sha256(metrics: Mapping[str, int | float]) -> str: - raw = json.dumps( - metrics, - allow_nan=False, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - return hashlib.sha256(raw).hexdigest() - - -@contextmanager -def _wandb_environment(wandb_dir: Path): - names = ("WANDB_DIR", "WANDB_DISABLE_GIT", "WANDB_DISABLE_CODE", "WANDB_SAVE_CODE") - previous = {name: os.environ.get(name) for name in names} - os.environ.update( - { - "WANDB_DIR": str(wandb_dir), - "WANDB_DISABLE_GIT": "true", - "WANDB_DISABLE_CODE": "true", - "WANDB_SAVE_CODE": "false", - } - ) - try: - yield - finally: - for name, value in previous.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value - - -class WandbMetricsMirror: - """Mirror metrics only after their source attempt record is durable. - - W&B is deliberately an observability mirror, never the experiment record. - Any W&B exception is converted into an immutable local tracking-error - record while the sampling receipt/evaluation remains valid. - """ - - def __init__( - self, - *, - store: AttemptStore, - project: str, - source_git_sha: str, - entity: str | None = None, - group: str | None = None, - wandb_dir: Path | None = None, - wandb_module: ModuleType | Any | None = None, - ) -> None: - if not project.strip(): - raise TrackingContractError("W&B project must be non-empty") - if not _SOURCE_GIT_SHA.fullmatch(source_git_sha): - raise TrackingContractError("source_git_sha must be a Git object SHA") - self.store = store - self.project = project - self.source_git_sha = source_git_sha - self.entity = entity - self.group = group - if self.group is not None and not self.group.strip(): - raise TrackingContractError("W&B group must be non-empty when provided") - requested = ( - Path(wandb_dir).expanduser() - if wandb_dir is not None - else store.external_root / ".wandb" - ).resolve(strict=False) - if requested == store.repo_root or requested.is_relative_to(store.repo_root): - raise TrackingContractError("WANDB_DIR must be outside the repository") - requested.mkdir(parents=True, exist_ok=True, mode=0o700) - self.wandb_dir = requested.resolve(strict=True) - if self.wandb_dir == store.repo_root or self.wandb_dir.is_relative_to( - store.repo_root - ): - raise TrackingContractError("WANDB_DIR resolves inside the repository") - self._wandb_module = wandb_module - - def mirror_saved_metrics( - self, - *, - key: AttemptKey, - source: Literal["sampling_receipt", "evaluation"], - metrics: Mapping[str, int | float], - step: int, - ) -> TrackingResult: - """Compatibility helper for one record and one short-lived session.""" - - with self.session(key.run_key) as session: - return session.mirror_saved_metrics( - key=key, - source=source, - metrics=metrics, - step=step, - ) - - def session(self, run_key: RunKey) -> WandbRunSession: - """Create a lazy, persistent one-init-per-arm tracking session.""" - - return WandbRunSession(mirror=self, run_key=run_key) - - def _load_saved_source( - self, - *, - key: AttemptKey, - source: Literal["sampling_receipt", "evaluation"], - ) -> StoredRecord | None: - if source == "sampling_receipt": - return self.store.load_sampling_receipt(key) - return self.store.load_evaluation(key) - - def _settings(self, module: Any) -> Any: - return module.Settings( - disable_code=True, - disable_git=True, - disable_job_creation=True, - save_code=False, - x_disable_machine_info=True, - x_disable_stats=True, - x_save_requirements=False, - ) - - def _init_kwargs(self, run_key: RunKey, run_id: str, module: Any) -> dict[str, Any]: - result: dict[str, Any] = { - "project": self.project, - "id": run_id, - "name": ( - f"{run_key.study_id}/{run_key.hypothesis_id}/{run_key.arm_id}" - f"@{self.source_git_sha[:12]}" - ), - # The baseline runner keeps one session per arm open while arms - # execute concurrently. W&B otherwise returns the process-global - # active run, silently collapsing every arm into the first run. - "reinit": "create_new", - "resume": "allow", - "dir": str(self.wandb_dir), - "settings": self._settings(module), - } - if self.entity is not None: - result["entity"] = self.entity - if self.group is not None: - result["group"] = self.group - return result - - def _module(self) -> Any: - if self._wandb_module is not None: - return self._wandb_module - return importlib.import_module("wandb") - - def _record_tracking_error( - self, - *, - key: AttemptKey, - saved: StoredRecord, - source: Literal["sampling_receipt", "evaluation"], - run_id: str, - step: int, - metrics_sha256: str, - operation: Literal["init", "log", "finish"], - error: Exception, - ) -> StoredRecord: - return self.store.write_tracking_error( - key, - { - "schema_version": TRACKING_SCHEMA_VERSION, - "source_record_type": source, - "source_record_relative_path": str(saved.relative_path), - "source_record_sha256": saved.record_sha256, - "wandb_run_id": run_id, - "wandb_step": step, - "metrics_sha256": metrics_sha256, - "operation": operation, - "error_type": type(error).__name__, - "error_message": str(error)[:4096], - }, - ) - - -class WandbRunSession: - """One lazily initialized W&B run for one immutable PixCell arm.""" - - def __init__(self, *, mirror: WandbMetricsMirror, run_key: RunKey) -> None: - self.mirror = mirror - self.run_key = run_key - self.run_id = deterministic_wandb_run_id( - run_key, - source_git_sha=mirror.source_git_sha, - ) - self._run: Any | None = None - self._closed = False - self._last_context: tuple[ - AttemptKey, - Literal["sampling_receipt", "evaluation"], - StoredRecord, - int, - str, - ] | None = None - - def __enter__(self) -> WandbRunSession: - if self._closed: - raise TrackingContractError("a closed W&B session cannot be reopened") - # Initialization remains lazy so the first external tracking action - # cannot precede its immutable local source record. - return self - - def __exit__(self, *_: object) -> None: - self.close() - - def mirror_saved_metrics( - self, - *, - key: AttemptKey, - source: Literal["sampling_receipt", "evaluation"], - metrics: Mapping[str, int | float], - step: int, - ) -> TrackingResult: - if self._closed: - raise TrackingContractError("W&B session is closed") - if key.run_key != self.run_key: - raise TrackingContractError( - "attempt key belongs to a different W&B arm session" - ) - if isinstance(step, bool) or not isinstance(step, int) or step < 0: - raise TrackingContractError("W&B step must be a nonnegative integer") - clean_metrics = _metrics_payload(metrics) - saved = self.mirror._load_saved_source(key=key, source=source) - if saved is None: - raise TrackingContractError( - f"cannot mirror {source} metrics before local state is saved" - ) - metrics_sha = _metrics_sha256(clean_metrics) - self._last_context = (key, source, saved, step, metrics_sha) - try: - if self._run is None: - self._initialize() - except Exception as exc: - error = self.mirror._record_tracking_error( - key=key, - saved=saved, - source=source, - run_id=self.run_id, - step=step, - metrics_sha256=metrics_sha, - operation="init", - error=exc, - ) - return TrackingResult( - mirrored=False, - wandb_run_id=self.run_id, - source_record_sha256=saved.record_sha256, - tracking_error_relative_path=str(error.relative_path), - ) - - try: - with _WANDB_ENVIRONMENT_LOCK, _wandb_environment( - self.mirror.wandb_dir - ): - # No task IDs, prompts, completions, programs, configs, - # artifacts, tables, or source files cross this boundary. - self._run.log(dict(clean_metrics), step=step, commit=True) - except Exception as exc: - error = self.mirror._record_tracking_error( - key=key, - saved=saved, - source=source, - run_id=self.run_id, - step=step, - metrics_sha256=metrics_sha, - operation="log", - error=exc, - ) - return TrackingResult( - mirrored=False, - wandb_run_id=self.run_id, - source_record_sha256=saved.record_sha256, - tracking_error_relative_path=str(error.relative_path), - ) - - return TrackingResult( - mirrored=True, - wandb_run_id=self.run_id, - source_record_sha256=saved.record_sha256, - ) - - def _initialize(self) -> None: - module = self.mirror._module() - init_kwargs = self.mirror._init_kwargs(self.run_key, self.run_id, module) - # os.environ is process-global. Serialize this short section so - # parallel samplers cannot restore WANDB_DIR beneath another init. - with _WANDB_ENVIRONMENT_LOCK, _wandb_environment(self.mirror.wandb_dir): - run = module.init(**init_kwargs) - if run is None: - raise RuntimeError("wandb.init returned no run") - self._run = run - - def close(self) -> None: - if self._closed: - return - self._closed = True - if self._run is None: - return - try: - with _WANDB_ENVIRONMENT_LOCK, _wandb_environment( - self.mirror.wandb_dir - ): - self._run.finish() - except Exception as exc: - if self._last_context is None: - raise - key, source, saved, step, metrics_sha = self._last_context - self.mirror._record_tracking_error( - key=key, - saved=saved, - source=source, - run_id=self.run_id, - step=step, - metrics_sha256=metrics_sha, - operation="finish", - error=exc, - ) - finally: - self._run = None diff --git a/rl/stack.json b/rl/stack.json index f86af299..2b67d972 100644 --- a/rl/stack.json +++ b/rl/stack.json @@ -8,10 +8,10 @@ "pyarrow": "24.0.0", "tinker": "0.22.7", "tinker-cookbook": "0.4.3.dev2+gca232b084", - "tml-renderers": "0.1.0", "torch": "2.12.1", "transformers": "5.5.3", - "wandb": "0.28.0" + "wandb": "0.28.0", + "tml-renderers": "0.1.0" }, "tinker_cookbook_commit": "ca232b084411029df565a36095642918a097fa7f" } diff --git a/rl/studies/__init__.py b/rl/studies/__init__.py deleted file mode 100644 index c4bba494..00000000 --- a/rl/studies/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Versioned training and evaluation studies.""" diff --git a/rl/studies/representation_curriculum_v1/README.md b/rl/studies/representation_curriculum_v1/README.md deleted file mode 100644 index 35896ee4..00000000 --- a/rl/studies/representation_curriculum_v1/README.md +++ /dev/null @@ -1,160 +0,0 @@ -# Representation curriculum v1 - -> **Superseded diagnostic record.** This study and its Qwen 32k/60k -> extensions used the `pixcell-blind-phase-a-direct-v1` prompt contract. That -> one-shot contract did not state the output-file requirement enforced by the -> evaluator, so its measurements are not canonical baselines or paper -> results. The protocol JSON and study files remain unchanged for provenance. -> Current paid launch code rejects this prompt contract. - -This archived study recorded preliminary zero-shot measurements before any -curriculum training began. It had two merge boundaries: - -1. Phase 0 freezes and validates the protocol, task manifest, prompt, sampling - adapters, evaluator, result schema, and source SHA without creating a - Tinker client. -2. Phase 1 samples and evaluates the frozen base models, then commits a - validated evidence export in a separate PR. - -The SFT and RL hypotheses reserved in [`protocol.json`](protocol.json) are not -launchable in either phase. - -## Fixed model contract - -Each model receives one maximum-visibility image, the physical footprint, the -PixCell primitive catalogue, and the existing blind Phase-A programming task. -It returns one Python program. Display calibration, the target raster, -representation labels, expected code, and verifier diagnostics are not -model-visible. - -Every completion passes through the same strict source gate, isolated -execution boundary, GDS render, and absolute-scale IoU evaluator. A model -failure contributes zero IoU to aggregation. A reference or evaluator failure -aborts the wave instead of becoming a zero-scored model attempt. - -## Frozen hypotheses - -| ID | Purpose | Tasks and attempts | -|---|---|---| -| `RC-H00` | Select the Qwen inference operating point | 40 fixed `depth/validation` tasks, eight per level L0–L4 and one representation per task; thinking on/off × 4,096/16,384 output tokens; two paired attempts per task | -| `RC-H00R` | Resolve truncation at the largest registered Qwen thinking budget | Conditional 32,768-token confirmation only when `qwen-on-16384` has a cap-hit rate above 5% | -| `RC-H01` | Qwen zero-shot F1–F8 baseline | The incumbent `qwen-off-4096` and the selected `RC-H00` arm, omitting a duplicate arm; four attempts per figure | -| `RC-H02` | Inkling zero-shot F1–F8 baseline | Thinking effort 0.9, 20,000 output tokens; four attempts per figure | - -The `RC-H00` arms use identical task-attempt seeds. Selection first maximizes -representation-macro mean raw IoU. Arms within 0.01 of the best are ordered by -pure executable rate, estimated uncached cost, median completion tokens, then -arm ID. The rule and all model, renderer, image, context, sampling, pricing, -task, and tracking settings are sealed in -[`protocol.json`](protocol.json). [`task_manifest.json`](task_manifest.json) -binds the exact tasks and source digests. - -## Historical launch order - -The commands below document how the superseded record was produced. They are -not runnable against the current prompt contract and must not be used to spend -or report a canonical baseline. - -The launch must come from a clean checkout at the exact Phase 0 merge SHA. -Mutable output must be outside the repository. Build the committed evaluator -image and export the immutable image ID it prints: - -```bash -python rl/common/sandbox/build.py -export PIXCELL_EVALUATOR_IMAGE=sha256: - -export PIXCELL_STUDY_ROOT=/absolute/path/outside/PixCell -export TINKER_API_KEY= -export WANDB_API_KEY= -SOURCE_SHA="$(git rev-parse HEAD)" - -PYTHONPATH=src:. python -m \ - rl.studies.representation_curriculum_v1.audit_phase0 \ - --expected-source-sha "$SOURCE_SHA" -``` - -The audit builds every frozen renderer/prompt combination without creating a -Tinker client, validates all 48 references, executes one accepted program in -the isolated evaluator, and round-trips the attempt-record schema. Only after -it passes, run the Qwen operating-point smoke and complete waves: - -```bash -PYTHONPATH=src:. python -m rl.studies.representation_curriculum_v1.run_baseline \ - --hypothesis RC-H00 --wave smoke \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE - -PYTHONPATH=src:. python -m rl.studies.representation_curriculum_v1.run_baseline \ - --hypothesis RC-H00 --wave complete \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE -``` - -The smoke wave records attempt 1 for the first fixed development task in each -of the four arms. The complete wave reuses those receipts and fills two -attempts for all 40 tasks. If its preliminary selection requires `RC-H00R`, run -the conditional confirmation before continuing: - -```bash -PRELIMINARY="$PIXCELL_STUDY_ROOT/studies/representation-curriculum-v1/selections/operating_point.preliminary.json" - -PYTHONPATH=src:. python -m rl.studies.representation_curriculum_v1.run_baseline \ - --hypothesis RC-H00R --wave complete --selection "$PRELIMINARY" \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE -``` - -Then start the Qwen and Inkling F1–F8 smoke waves concurrently. Each records -attempt 1 for all eight figures. Inspect and validate those 8-attempt results -per arm before starting the complete waves, which reuse attempt 1 and fill -attempts 2–4: - -```bash -SELECTION="$PIXCELL_STUDY_ROOT/studies/representation-curriculum-v1/selections/operating_point.json" - -PYTHONPATH=src:. python -m rl.studies.representation_curriculum_v1.run_baseline \ - --hypothesis RC-H01 --wave smoke --selection "$SELECTION" \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE - -PYTHONPATH=src:. python -m rl.studies.representation_curriculum_v1.run_baseline \ - --hypothesis RC-H02 --wave smoke \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE -``` - -Use the same commands with `--wave complete` only after both smoke records pass. -The immutable local receipt is the sampling boundary: restarting a wave -evaluates an existing completion or fills a missing attempt; it does not -resample a saved attempt. - -After every complete wave validates, freeze and revalidate the compact Phase 1 -record: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_curriculum_v1.freeze_phase1 \ - --external-root "$PIXCELL_STUDY_ROOT" \ - --expected-source-sha "$SOURCE_SHA" - -PYTHONPATH=src:. python -m \ - rl.studies.representation_curriculum_v1.freeze_phase1 \ - --external-root "$PIXCELL_STUDY_ROOT" \ - --expected-source-sha "$SOURCE_SHA" \ - --validate -``` - -The freezer accepts source changes only inside the fixed Phase 1 evidence -directory. The resulting files form the separate Phase 1 PR. - -## Record ownership - -The external attempt ledger is authoritative while the run is active. It -contains immutable sampling receipts, exact programs, evaluator evidence, -summaries, and the operating-point selection. W&B receives metrics only after -the corresponding local record is durable. A W&B outage is recorded locally -and cannot change an evaluation. - -The Phase 1 PR freezes the validated release subset under -[`data/training/representation-curriculum-v1/`](../../../data/training/representation-curriculum-v1/). -Secrets, mutable W&B files, model caches, and checkpoints are never committed. diff --git a/rl/studies/representation_curriculum_v1/__init__.py b/rl/studies/representation_curriculum_v1/__init__.py deleted file mode 100644 index 23b991f7..00000000 --- a/rl/studies/representation_curriculum_v1/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""The frozen representation-curriculum-v1 study.""" diff --git a/rl/studies/representation_curriculum_v1/audit_phase0.py b/rl/studies/representation_curriculum_v1/audit_phase0.py deleted file mode 100644 index 998a299d..00000000 --- a/rl/studies/representation_curriculum_v1/audit_phase0.py +++ /dev/null @@ -1,322 +0,0 @@ -#!/usr/bin/env python3 -"""Run the zero-spend audit required before a baseline launch is committed.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import subprocess -import tempfile -from pathlib import Path -from typing import Any - -from rl.common.dataset_io import parquet_paths -from rl.common.evaluator import Attribution, EvaluationStatus, PixCellEvaluator -from rl.common.output import extract_code -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.attempt_store import AttemptKey, AttemptStore, RunKey -from rl.evaluation.protocol import arm_spec, load_protocol, task_manifest_path -from rl.evaluation.runner import ( - _FORBIDDEN_PROMPT_FRAGMENTS, - _preflight_references, - _preflight_renderers, -) -from rl.evaluation.samplers import TinkerSampler -from rl.evaluation.tasks import ( - BENCHMARK_TASK_SET, - OPERATING_POINT_TASK_SET, - load_task_manifest, - load_task_set, -) - - -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") - - -REPO_ROOT = Path(__file__).resolve().parents[3] -AUDIT_SCHEMA = "pixcell-representation-curriculum-phase0-audit-v1" - - -def _git_state(repo_root: Path) -> tuple[str, bool]: - head = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo_root, - text=True, - ).strip() - status = subprocess.check_output( - ["git", "status", "--porcelain", "--untracked-files=all"], - cwd=repo_root, - text=True, - ).strip() - return head, not bool(status) - - -def _exact_code(dataset_root: Path, opaque_id: str) -> str: - import pyarrow.parquet as pq - - matches: list[str] = [] - for path in parquet_paths( - dataset_root, - configuration="depth", - split="validation", - ): - for row in pq.read_table(path, columns=["id", "code"]).to_pylist(): - row_id = str(row["id"]) - observed = hashlib.sha256(row_id.encode("utf-8")).hexdigest()[:20] - if observed == opaque_id: - matches.append(str(row["code"])) - if len(matches) != 1 or not matches[0].strip(): - raise ValueError( - f"expected one supervised label for audit task {opaque_id}; " - f"found {len(matches)}" - ) - return matches[0] - - -def _samplers(protocol: dict[str, Any]) -> dict[str, TinkerSampler]: - def forbidden_client() -> Any: - raise AssertionError("Phase 0 audit attempted to create a Tinker client") - - from rl.evaluation.runner import _sampler_target - - return { - arm_id: TinkerSampler( - _sampler_target(arm_spec(protocol, arm_id)), - service_client_factory=forbidden_client, - ) - for arm_id in protocol["arms"] - } - - -def _schema_smoke( - *, - repo_root: Path, - protocol: dict[str, Any], - sandbox: Any, -) -> dict[str, str]: - with tempfile.TemporaryDirectory(prefix="pixcell-phase0-audit-") as temporary: - store = AttemptStore( - repo_root=repo_root, - external_root=Path(temporary) / "records", - ) - run_key = RunKey(protocol["study_id"], "RC-H02", "inkling-e09-20000") - manifest = { - "source_git_sha": "0" * 40, - "protocol_sha256": protocol["logical_sha256"], - "task_set_sha256": "1" * 64, - "prompt_sha256": "2" * 64, - "sandbox_image_ref": sandbox.image_ref, - "sandbox_image_id": sandbox.image_id, - "runtime": validate_runtime_stack(repo_root), - "model_binding": {"model": "thinkingmachines/Inkling"}, - } - with store.acquire_run_lock(run_key): - saved_manifest = store.create_or_verify_run_manifest(run_key, manifest) - key = AttemptKey( - protocol["study_id"], - "RC-H02", - "inkling-e09-20000", - "F1", - 1, - ) - completion = "```python\nprint('audit')\n```" - receipt = store.write_sampling_receipt( - key, - { - "request": { - "max_tokens": 20000, - "temperature": 1.0, - "top_p": 1.0, - "seed": 1, - }, - "sample": { - "prompt_tokens": 100, - "completion_tokens": 10, - "stop_reason": "stop", - "cap_hit": False, - "raw_text": completion, - "completion_text": completion, - "reasoning_text": "", - "answer_text": completion, - "reasoning_tokens_exact": 0, - "answer_tokens_exact": 10, - "reasoning_tokens_estimate": None, - "answer_tokens_estimate": None, - "channel_token_count_basis": "phase0_schema_audit", - "channel_parse_complete": True, - }, - "sampling_seconds": 0.1, - "raw_completion_sha256": hashlib.sha256( - completion.encode("utf-8") - ).hexdigest(), - "completion_sha256": hashlib.sha256( - completion.encode("utf-8") - ).hexdigest(), - "cost_estimate_usd": { - "cached_prefill": 0.0, - "uncached_prefill": 0.0, - }, - }, - ) - evaluation = store.write_evaluation( - key, - { - "status": EvaluationStatus.SYNTAX_ERROR.value, - "attribution": Attribution.MODEL.value, - "pure_executable": False, - "measurement_available": False, - "measured_iou": None, - "measured_dice": None, - "aggregation_iou": 0.0, - "program": "print('audit')", - "program_sha256": hashlib.sha256( - b"print('audit')" - ).hexdigest(), - "reference_sha256": "3" * 64, - "violations": [], - "error": "schema audit", - "retryable": False, - "evaluation_seconds": 0.1, - "diagnostics": {}, - }, - ) - if store.load_sampling_receipt(key) != receipt: - raise AssertionError("sampling receipt did not round-trip") - if store.load_evaluation(key) != evaluation: - raise AssertionError("evaluation record did not round-trip") - if store.load_run_manifest(run_key) != saved_manifest: - raise AssertionError("run manifest did not round-trip") - return { - "run_manifest_record_sha256": saved_manifest.record_sha256, - "sampling_record_sha256": receipt.record_sha256, - "evaluation_record_sha256": evaluation.record_sha256, - } - - -def run_audit( - *, - repo_root: Path, - expected_source_sha: str | None = None, -) -> dict[str, Any]: - root = repo_root.expanduser().resolve(strict=True) - head, clean = _git_state(root) - if expected_source_sha is not None: - if head != expected_source_sha: - raise ValueError(f"source SHA mismatch: {head} != {expected_source_sha}") - if not clean: - raise ValueError("source worktree must be clean for the sealed audit") - - protocol = load_protocol(root, protocol_file=PROTOCOL_FILE) - manifest = load_task_manifest(task_manifest_path(root), repo_root=root) - task_sets = { - name: load_task_set(repo_root=root, manifest=manifest, task_set=name) - for name in (OPERATING_POINT_TASK_SET, BENCHMARK_TASK_SET) - } - samplers = _samplers(protocol) - dev_arm_ids = [ - arm_id for arm_id in protocol["arms"] if arm_id.startswith("qwen-") - ] - benchmark_arm_ids = list(protocol["arms"]) - renderer_cases = 0 - for task_set, arm_ids in ( - (task_sets[OPERATING_POINT_TASK_SET], dev_arm_ids), - (task_sets[BENCHMARK_TASK_SET], benchmark_arm_ids), - ): - arms = {arm_id: arm_spec(protocol, arm_id) for arm_id in arm_ids} - rendered = _preflight_renderers( - tasks=task_set, - arms=arms, - samplers={arm_id: samplers[arm_id] for arm_id in arm_ids}, - ) - renderer_cases += len(rendered) - for (arm_id, task_id), messages in rendered.items(): - visible = repr(messages) - leaked = [ - fragment - for fragment in _FORBIDDEN_PROMPT_FRAGMENTS - if fragment in visible - ] - if leaked: - raise ValueError( - f"{arm_id}/{task_id} leaked verifier-only prompt text: {leaked}" - ) - - from rl.common.isolation import require_execution_boundary - - sandbox = require_execution_boundary() - all_tasks = [ - *task_sets[OPERATING_POINT_TASK_SET], - *task_sets[BENCHMARK_TASK_SET], - ] - with PixCellEvaluator( - max_workers=1, - evaluator_retries=1, - require_isolation=True, - ) as evaluator: - _preflight_references(all_tasks, evaluator) - audit_task = task_sets[OPERATING_POINT_TASK_SET][0] - exact_code = _exact_code(root / "dataset", audit_task.task_id) - wrapped = f"audit\n```python\n{exact_code}\n```" - if extract_code(wrapped) != exact_code.strip(): - raise ValueError("output extractor changed the known audit program") - result = evaluator.evaluate(audit_task.reference, wrapped) - if ( - result.status is not EvaluationStatus.OK - or result.attribution is not Attribution.MODEL - or result.iou is None - ): - raise ValueError( - "isolated evaluator failed the accepted-program audit: " - f"{result.status.value}/{result.attribution.value}: {result.error}" - ) - - runtime = validate_runtime_stack(root) - - report = { - "schema_version": AUDIT_SCHEMA, - "source_git_sha": head, - "source_worktree_clean": clean, - "protocol_sha256": protocol["logical_sha256"], - "task_manifest_sha256": manifest["logical_sha256"], - "task_counts": { - name: len(tasks) for name, tasks in sorted(task_sets.items()) - }, - "renderer_prompt_cases": renderer_cases, - "model_client_created": False, - "reference_count": len(all_tasks), - "extractor_exact": True, - "evaluator": { - "status": result.status.value, - "iou": result.iou, - "sandbox_image_ref": sandbox.image_ref, - "sandbox_image_id": sandbox.image_id, - }, - "record_schema": _schema_smoke( - repo_root=root, - protocol=protocol, - sandbox=sandbox, - ), - "runtime": runtime, - } - return report - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=REPO_ROOT) - result.add_argument("--expected-source-sha") - return result - - -def main() -> None: - args = parser().parse_args() - report = run_audit( - repo_root=args.repo_root, - expected_source_sha=args.expected_source_sha, - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/extensions/__init__.py b/rl/studies/representation_curriculum_v1/extensions/__init__.py deleted file mode 100644 index 06149dc8..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Registered extensions to the frozen representation-curriculum study.""" diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/README.md b/rl/studies/representation_curriculum_v1/extensions/qwen32/README.md deleted file mode 100644 index 3292bd2a..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Qwen 32k F1-F8 extension - -> **Superseded diagnostic record.** This extension used the retired v1 prompt -> contract. Its measurements are not canonical baselines or paper results, -> and current paid launch code rejects the protocol. The files are retained -> for provenance. - -This is the quality-first extension to the frozen Phase 1 baseline study. It -was registered after the `RC-H01` smoke wave showed that 7 of 8 Qwen -thinking-on/16k completions reached the output cap. - -`RC-H03` keeps the original F1-F8 tasks, blind prompt, renderer, image policy, -temperature, seeds, evaluator, and four attempts per figure. Its only changed -model setting is the 32,768-token output cap. The extension uses its own study -ID, source commit, and external ledger. It does not rewrite the original -operating-point selection or `RC-H01` evidence. - -The historical launch command was: - -```bash -python -m rl.studies.representation_curriculum_v1.extensions.qwen32.run_baseline \ - --wave smoke \ - --external-root "$PIXCELL_STUDY_ROOT" \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE -``` - -After verifying all eight sampling and evaluation records, repeat with -`--wave complete`. The complete wave is resumable and contains 32 samples. diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/__init__.py b/rl/studies/representation_curriculum_v1/extensions/qwen32/__init__.py deleted file mode 100644 index 20b83d83..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Quality-first Qwen 32k F1-F8 baseline extension.""" diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/freeze.py b/rl/studies/representation_curriculum_v1/extensions/qwen32/freeze.py deleted file mode 100644 index df2684ec..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/freeze.py +++ /dev/null @@ -1,228 +0,0 @@ -#!/usr/bin/env python3 -"""Freeze or validate the tracked Qwen 32k F1-F8 baseline addendum.""" - -from __future__ import annotations - -import argparse -import json -import re -from pathlib import Path, PurePosixPath -from typing import Any - -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.attempt_store import AttemptStore, RunKey -from rl.evaluation.protocol import jobs_for_wave, load_protocol, task_manifest_path -from rl.evaluation.tasks import canonical_json_sha256, load_task_manifest -from rl.evaluation.tracking import deterministic_wandb_run_id -from rl.studies.representation_curriculum_v1.freeze_phase1 import ( - Phase1FreezeError, - _assert_exact_output, - _collect_run_attempts, - _git_head, - _json_bytes, - _jsonl_bytes, - _load_run, - _prompt_binding, - _require_source_clean_except_output, - _sha256_bytes, - _task_set, - _validate_roots, - _write_create_or_verify, -) - - -REPO_ROOT = Path(__file__).resolve().parents[5] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") -OUTPUT_RELATIVE_ROOT = Path( - "data/training/representation-curriculum-v1/phase1-qwen32" -) -RELEASE_SCHEMA = "pixcell-representation-curriculum-qwen32-release-v1" -_SOURCE_SHA = re.compile(r"^[0-9a-f]{40}$") - - -def _output_root(repo_root: Path, requested: Path | None) -> Path: - root = repo_root.expanduser().resolve(strict=True) - expected = (root / OUTPUT_RELATIVE_ROOT).resolve(strict=False) - if expected == root or not expected.is_relative_to(root): - raise Phase1FreezeError( - "the tracked Qwen 32k path resolves outside the source repository" - ) - observed = ( - expected - if requested is None - else requested.expanduser().resolve(strict=False) - ) - if observed != expected: - raise Phase1FreezeError( - f"Qwen 32k evidence must be written exactly to {expected}" - ) - return expected - - -def _collect_bundle( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, -) -> tuple[dict[PurePosixPath, bytes], dict[str, Any]]: - if not _SOURCE_SHA.fullmatch(expected_source_sha): - raise Phase1FreezeError( - "expected_source_sha must be a full lowercase Git SHA" - ) - if _git_head(repo_root) != expected_source_sha: - raise Phase1FreezeError("source HEAD differs from the paid-launch commit") - - protocol = load_protocol(repo_root, protocol_file=PROTOCOL_FILE) - runtime = validate_runtime_stack(repo_root) - manifest = load_task_manifest( - task_manifest_path(repo_root), - repo_root=repo_root, - ) - prompt_binding, prompt_sha = _prompt_binding() - tasks, task_set_sha = _task_set( - repo_root=repo_root, - protocol=protocol, - manifest=manifest, - hypothesis_id="RC-H03", - ) - jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H03", - wave_name="complete", - task_ids=[task.task_id for task in tasks], - ) - arm_id = str(protocol["hypotheses"]["RC-H03"]["arm_ids"][0]) - store = AttemptStore(repo_root=repo_root, external_root=external_root) - evidence, sandbox = _load_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H03", - arm_id=arm_id, - source_git_sha=expected_source_sha, - tasks=tasks, - jobs=jobs, - task_set_sha256=task_set_sha, - prompt_sha256=prompt_sha, - sandbox=None, - runtime=runtime, - ) - attempts, programs = _collect_run_attempts( - protocol=protocol, - store=store, - evidence=evidence, - ) - attempts.sort(key=lambda row: (row["task_id"], row["attempt_index"])) - - summary_document = { - "schema_version": "pixcell-representation-curriculum-qwen32-summary-v1", - "study_id": protocol["study_id"], - "hypothesis_id": "RC-H03", - "arm_id": arm_id, - "summary": evidence.summary, - } - summary_document["logical_sha256"] = canonical_json_sha256(summary_document) - files: dict[PurePosixPath, bytes] = { - PurePosixPath("summary.json"): _json_bytes(summary_document), - PurePosixPath("attempts.jsonl"): _jsonl_bytes(attempts), - **programs, - } - file_inventory = [ - { - "path": path.as_posix(), - "sha256": _sha256_bytes(content), - "bytes": len(content), - } - for path, content in sorted(files.items(), key=lambda item: item[0]) - ] - release_manifest: dict[str, Any] = { - "schema_version": RELEASE_SCHEMA, - "study_id": protocol["study_id"], - "phase": "phase1-quality-first-qwen32-addendum", - "source_git_sha": expected_source_sha, - "protocol_path": str(PROTOCOL_FILE.relative_to(repo_root)), - "protocol_sha256": protocol["logical_sha256"], - "base_study": protocol["base_study"], - "decision": protocol["decision"], - "task_manifest_sha256": manifest["logical_sha256"], - "dataset": protocol["dataset"], - "prompt": prompt_binding, - "prompt_sha256": prompt_sha, - "sandbox": { - "image_ref": sandbox[0], - "image_id": sandbox[1], - }, - "runtime": runtime, - "run": { - "hypothesis_id": "RC-H03", - "arm_id": arm_id, - "attempts": len(attempts), - "run_manifest_record_sha256": evidence.manifest.record_sha256, - "wandb_run_id": deterministic_wandb_run_id( - RunKey(protocol["study_id"], "RC-H03", arm_id), - source_git_sha=expected_source_sha, - ), - }, - "files": file_inventory, - } - release_manifest["logical_sha256"] = canonical_json_sha256(release_manifest) - return files, release_manifest - - -def freeze( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, - output: Path | None = None, - validate_only: bool = False, -) -> dict[str, Any]: - output_root = _output_root(repo_root, output) - source, external = _validate_roots( - repo_root=repo_root, - external_root=external_root, - output_root=output_root, - ) - _require_source_clean_except_output( - repo_root=source, - output_root=output_root, - ) - files, manifest = _collect_bundle( - repo_root=source, - external_root=external, - expected_source_sha=expected_source_sha, - ) - expected = { - PurePosixPath("release_manifest.json"): _json_bytes(manifest), - **files, - } - if validate_only: - _assert_exact_output(output_root, expected) - else: - _write_create_or_verify(output_root, expected) - return manifest - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=REPO_ROOT) - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--output", type=Path) - result.add_argument("--validate", action="store_true") - return result - - -def main() -> None: - args = parser().parse_args() - report = freeze( - repo_root=args.repo_root, - external_root=args.external_root, - expected_source_sha=args.expected_source_sha, - output=args.output, - validate_only=args.validate, - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/protocol.json b/rl/studies/representation_curriculum_v1/extensions/qwen32/protocol.json deleted file mode 100644 index a2f7919b..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/protocol.json +++ /dev/null @@ -1,116 +0,0 @@ -{ - "arms": { - "qwen-on-32768": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 32768, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": true, - "thinking_effort": null - } - }, - "base_study": { - "hypothesis": "RC-H01", - "protocol_sha256": "1e2869e6c640463e4cf4233a2806dadb5368c4a22a24634c755bb1d3dcc8e349", - "source_git_sha": "b4b56ce2929298aa0d9334d7cdaf69f227329349", - "study_id": "representation-curriculum-v1" - }, - "contract_version": "pixcell-blind-phase-a-direct-v1", - "dataset": { - "configuration": "depth", - "development_split": "validation", - "logical_release_sha256": "676c49134d4d044c7d84426cf8eeecf09302e74ca4aa548956f14b7631a4d80b", - "repository": "qpaig-mit/pixcell", - "revision": "v2.0.0" - }, - "decision": { - "basis": "RC-H01 smoke observed a 0.875 cap-hit rate for qwen-on-16384", - "canonical_selection_logical_sha256": "14e1cd1b71cf345afece19fe110c15e9c734879e3149545489e67df801e6928b", - "policy": "quality_first_user_directed_extension", - "prior_32k_confirmation": { - "arm_id": "qwen-on-32768-confirmation", - "attempts": 80, - "cap_hit_rate": 0.0, - "hypothesis_id": "RC-H00R", - "summary_logical_sha256": "45c291e95c3b6f1f1c7145190c36f532b4e9acb713c0f11a66225af394b2e67f" - }, - "trigger_evidence": { - "arm_id": "qwen-on-16384", - "attempts": 8, - "cap_hit_rate": 0.875, - "hypothesis_id": "RC-H01", - "summary_logical_sha256": "241912e11d77ae5d96a9b5d7814300ff3a03dfe673670c591c5235c39e31d59c", - "wave": "smoke" - } - }, - "hypotheses": { - "RC-H03": { - "arm_ids": [ - "qwen-on-32768" - ], - "attempts_per_task": 4, - "name": "Qwen quality-first 32k zero-shot F1-F8 baseline", - "status": "frozen", - "task_set": "f1_f8", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2, - 3, - 4 - ], - "task_count": 8 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 8 - } - } - } - }, - "launch": { - "confirmation_token": "PIXCELL_PHASE1_BASELINE", - "mutable_output_environment": "PIXCELL_STUDY_ROOT" - }, - "logical_sha256": "bd51768349f512741080fe4f080f343d311e81c5946186a9541565737db9da33", - "pricing": { - "as_of": "2026-07-27", - "currency": "USD", - "models": { - "Qwen/Qwen3.6-35B-A3B": { - "prefill_cached": 0.108, - "prefill_uncached": 0.54, - "sample": 1.335 - } - }, - "source": "https://tinker-docs.thinkingmachines.ai/tinker/models/", - "unit": "per_million_tokens" - }, - "sampling": { - "evaluator_workers": 8, - "require_candidate_isolation": true, - "sampling_concurrency": 8, - "seed_namespace": "pixcell-representation-curriculum-v1", - "temperature": 1.0, - "top_p": 1.0 - }, - "schema_version": "pixcell-representation-curriculum-protocol-v1", - "study_id": "representation-curriculum-v1-qwen32", - "task_manifest": { - "logical_sha256": "e32a0a1522fba3c90163e5687c65ab5e92fdb77f4f875a73cbfcdaa432b44272", - "path": "task_manifest.json" - }, - "tracking": { - "entity": "aadarwal-massachusetts-institute-of-technology", - "group": "phase1-qwen32", - "local_record_is_authoritative": true, - "mode": "online", - "project": "pixcell-representation-curriculum-v1", - "provider": "wandb" - } -} diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/run_baseline.py b/rl/studies/representation_curriculum_v1/extensions/qwen32/run_baseline.py deleted file mode 100644 index a625111d..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/run_baseline.py +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env python3 -"""Run the fixed Qwen thinking-on/32k F1-F8 baseline extension.""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from pathlib import Path - -from rl.evaluation.runner import run_baseline - - -REPO_ROOT = Path(__file__).resolve().parents[5] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--wave", required=True, choices=("smoke", "complete")) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--confirm-spend", default="") - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_STUDY_ROOT"]) - if os.environ.get("PIXCELL_STUDY_ROOT") - else None - ), - ) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit( - "set PIXCELL_STUDY_ROOT or provide --external-root outside the repository" - ) - report = asyncio.run( - run_baseline( - repo_root=REPO_ROOT, - protocol_file=PROTOCOL_FILE, - hypothesis_id="RC-H03", - wave_name=args.wave, - expected_source_sha=args.expected_source_sha, - external_root=args.external_root, - confirmation=args.confirm_spend, - ) - ) - print(json.dumps(report["summaries"], indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/test_freeze.py b/rl/studies/representation_curriculum_v1/extensions/qwen32/test_freeze.py deleted file mode 100644 index 6fcbf334..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/test_freeze.py +++ /dev/null @@ -1,99 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.evaluation.attempt_store import RunKey -from rl.evaluation.tracking import deterministic_wandb_run_id -from rl.studies.representation_curriculum_v1.extensions.qwen32 import freeze -from rl.studies.representation_curriculum_v1.freeze_phase1 import Phase1FreezeError - - -REPO_ROOT = Path(__file__).resolve().parents[5] - - -def test_collect_bundle_uses_canonical_wandb_identity( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -): - source_sha = "a" * 40 - protocol = { - "study_id": "representation-curriculum-v1-qwen32", - "logical_sha256": "b" * 64, - "hypotheses": { - "RC-H03": { - "arm_ids": ["qwen-on-32768"], - "task_set": "f1_f8", - } - }, - "base_study": {"study_id": "representation-curriculum-v1"}, - "decision": {"policy": "quality_first_user_directed_extension"}, - "dataset": {"revision": "v2.0.0"}, - } - evidence = SimpleNamespace( - summary={"complete": True}, - manifest=SimpleNamespace(record_sha256="c" * 64), - ) - task = SimpleNamespace(task_id="F1") - job = SimpleNamespace(task_id="F1", attempt_index=1) - - monkeypatch.setattr(freeze, "_git_head", lambda _root: source_sha) - monkeypatch.setattr( - freeze, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr(freeze, "validate_runtime_stack", lambda _root: {}) - monkeypatch.setattr( - freeze, - "load_task_manifest", - lambda *_args, **_kwargs: {"logical_sha256": "d" * 64}, - ) - monkeypatch.setattr(freeze, "_prompt_binding", lambda: ({}, "e" * 64)) - monkeypatch.setattr( - freeze, - "_task_set", - lambda **_kwargs: ([task], "f" * 64), - ) - monkeypatch.setattr(freeze, "jobs_for_wave", lambda *_args, **_kwargs: [job]) - monkeypatch.setattr( - freeze, - "_load_run", - lambda **_kwargs: (evidence, ("sha256:image", "sha256:image")), - ) - monkeypatch.setattr( - freeze, - "_collect_run_attempts", - lambda **_kwargs: ( - [{"task_id": "F1", "attempt_index": 1}], - {}, - ), - ) - - _files, manifest = freeze._collect_bundle( - repo_root=REPO_ROOT, - external_root=tmp_path, - expected_source_sha=source_sha, - ) - assert manifest["run"]["wandb_run_id"] == deterministic_wandb_run_id( - RunKey( - "representation-curriculum-v1-qwen32", - "RC-H03", - "qwen-on-32768", - ), - source_git_sha=source_sha, - ) - - -def test_output_root_rejects_symlink_escape(tmp_path: Path): - repo = tmp_path / "repo" - parent = repo / "data/training/representation-curriculum-v1" - outside = tmp_path / "outside" - parent.mkdir(parents=True) - outside.mkdir() - (parent / "phase1-qwen32").symlink_to(outside, target_is_directory=True) - - with pytest.raises(Phase1FreezeError, match="outside the source repository"): - freeze._output_root(repo, None) diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen32/test_protocol.py b/rl/studies/representation_curriculum_v1/extensions/qwen32/test_protocol.py deleted file mode 100644 index 62f0bdce..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen32/test_protocol.py +++ /dev/null @@ -1,109 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import pytest - -from rl.evaluation.protocol import ( - _require_protocol_at_head, - deterministic_seed, - jobs_for_wave, - load_archived_protocol, - load_protocol, -) -from rl.evaluation.tasks import load_task_manifest - - -REPO_ROOT = Path(__file__).resolve().parents[5] -BASE_PROTOCOL = ( - REPO_ROOT / "rl/studies/representation_curriculum_v1/protocol.json" -) -EXTENSION_PROTOCOL = Path(__file__).with_name("protocol.json") -TASK_MANIFEST = ( - REPO_ROOT / "rl/studies/representation_curriculum_v1/task_manifest.json" -) - - -def _benchmark_task_ids() -> list[str]: - manifest = load_task_manifest(TASK_MANIFEST, repo_root=REPO_ROOT) - return [value["task_id"] for value in manifest["task_sets"]["f1_f8"]] - - -def test_extension_is_one_fixed_quality_first_arm(): - protocol = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - assert protocol["study_id"] == "representation-curriculum-v1-qwen32" - assert protocol["base_study"]["protocol_sha256"] == load_archived_protocol( - REPO_ROOT, - protocol_file=BASE_PROTOCOL, - )["logical_sha256"] - assert protocol["hypotheses"]["RC-H03"]["arm_ids"] == ["qwen-on-32768"] - assert protocol["arms"]["qwen-on-32768"] == { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 32768, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": True, - "thinking_effort": None, - } - - -def test_extension_smoke_and_complete_are_exact_f1_f8_g4_jobs(): - protocol = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - task_ids = _benchmark_task_ids() - smoke = jobs_for_wave( - protocol, - hypothesis_id="RC-H03", - wave_name="smoke", - task_ids=task_ids, - ) - complete = jobs_for_wave( - protocol, - hypothesis_id="RC-H03", - wave_name="complete", - task_ids=task_ids, - ) - assert len(smoke) == 8 - assert len(complete) == 32 - assert {job.arm_id for job in complete} == {"qwen-on-32768"} - assert {job.task_id for job in smoke} == set(task_ids) - assert {job.attempt_index for job in complete} == {1, 2, 3, 4} - - -def test_extension_keeps_base_seed_pairing(): - base = load_archived_protocol(REPO_ROOT, protocol_file=BASE_PROTOCOL) - extension = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - for task_id in _benchmark_task_ids(): - for attempt_index in range(1, 5): - assert deterministic_seed( - extension, - task_id=task_id, - attempt_index=attempt_index, - ) == deterministic_seed( - base, - task_id=task_id, - attempt_index=attempt_index, - ) - - -def test_protocol_override_must_be_source_controlled(tmp_path: Path): - outside = tmp_path / "protocol.json" - outside.write_text(EXTENSION_PROTOCOL.read_text(encoding="utf-8"), encoding="utf-8") - with pytest.raises(ValueError, match="inside the source repository"): - load_protocol(REPO_ROOT, protocol_file=outside) - - -def test_paid_protocol_must_be_tracked_by_head(): - _require_protocol_at_head(REPO_ROOT, BASE_PROTOCOL) - with pytest.raises(ValueError, match="tracked by the paid-launch commit"): - _require_protocol_at_head(REPO_ROOT, REPO_ROOT / ".git") diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/README.md b/rl/studies/representation_curriculum_v1/extensions/qwen60/README.md deleted file mode 100644 index 9da38e57..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Qwen 60k F1-F8 extension - -> **Superseded diagnostic record.** This extension used the retired v1 prompt -> contract. Its measurements are not canonical baselines or paper results, -> and current paid launch code rejects the protocol. The files are retained -> for provenance. - -This is the near-context-limit extension to the frozen Phase 1 baseline study. -It follows the accepted Qwen 32k extension in pull request 7 and records the -user's quality-first decision to test the strongest practical setting for a -self-hostable model. - -Qwen has a 65,536-token context window shared by the prompt and completion, so -a literal 65,536-token completion cannot be requested. Under the frozen -renderer, the largest F1-F8 prompt is 3,685 tokens. The 60,000-token cap -therefore fits every task and leaves at least 1,851 tokens of context reserve. - -`RC-H04` keeps the original F1-F8 tasks, blind prompt, renderer, image policy, -temperature, seed namespace, evaluator, and four attempts per figure. Its only -changed model setting relative to `RC-H03` is the output cap. It has its own -study ID, W&B run identity, external ledger, and tracked evidence directory, -so it cannot overwrite the 16k or 32k records. - -The historical launch command was: - -```bash -python -m rl.studies.representation_curriculum_v1.extensions.qwen60.run_baseline \ - --wave smoke \ - --external-root "$PIXCELL_STUDY_ROOT" \ - --expected-source-sha "$SOURCE_SHA" \ - --confirm-spend PIXCELL_PHASE1_BASELINE -``` - -After verifying all eight sampling and evaluation records, repeat with -`--wave complete`. The complete wave is resumable and contains 32 samples. diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/__init__.py b/rl/studies/representation_curriculum_v1/extensions/qwen60/__init__.py deleted file mode 100644 index ec4c4493..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Near-context-limit Qwen 60k F1-F8 baseline extension.""" diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/freeze.py b/rl/studies/representation_curriculum_v1/extensions/qwen60/freeze.py deleted file mode 100644 index 05326ca7..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/freeze.py +++ /dev/null @@ -1,228 +0,0 @@ -#!/usr/bin/env python3 -"""Freeze or validate the tracked Qwen 60k F1-F8 baseline addendum.""" - -from __future__ import annotations - -import argparse -import json -import re -from pathlib import Path, PurePosixPath -from typing import Any - -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.attempt_store import AttemptStore, RunKey -from rl.evaluation.protocol import jobs_for_wave, load_protocol, task_manifest_path -from rl.evaluation.tasks import canonical_json_sha256, load_task_manifest -from rl.evaluation.tracking import deterministic_wandb_run_id -from rl.studies.representation_curriculum_v1.freeze_phase1 import ( - Phase1FreezeError, - _assert_exact_output, - _collect_run_attempts, - _git_head, - _json_bytes, - _jsonl_bytes, - _load_run, - _prompt_binding, - _require_source_clean_except_output, - _sha256_bytes, - _task_set, - _validate_roots, - _write_create_or_verify, -) - - -REPO_ROOT = Path(__file__).resolve().parents[5] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") -OUTPUT_RELATIVE_ROOT = Path( - "data/training/representation-curriculum-v1/phase1-qwen60" -) -RELEASE_SCHEMA = "pixcell-representation-curriculum-qwen60-release-v1" -_SOURCE_SHA = re.compile(r"^[0-9a-f]{40}$") - - -def _output_root(repo_root: Path, requested: Path | None) -> Path: - root = repo_root.expanduser().resolve(strict=True) - expected = (root / OUTPUT_RELATIVE_ROOT).resolve(strict=False) - if expected == root or not expected.is_relative_to(root): - raise Phase1FreezeError( - "the tracked Qwen 60k path resolves outside the source repository" - ) - observed = ( - expected - if requested is None - else requested.expanduser().resolve(strict=False) - ) - if observed != expected: - raise Phase1FreezeError( - f"Qwen 60k evidence must be written exactly to {expected}" - ) - return expected - - -def _collect_bundle( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, -) -> tuple[dict[PurePosixPath, bytes], dict[str, Any]]: - if not _SOURCE_SHA.fullmatch(expected_source_sha): - raise Phase1FreezeError( - "expected_source_sha must be a full lowercase Git SHA" - ) - if _git_head(repo_root) != expected_source_sha: - raise Phase1FreezeError("source HEAD differs from the paid-launch commit") - - protocol = load_protocol(repo_root, protocol_file=PROTOCOL_FILE) - runtime = validate_runtime_stack(repo_root) - manifest = load_task_manifest( - task_manifest_path(repo_root), - repo_root=repo_root, - ) - prompt_binding, prompt_sha = _prompt_binding() - tasks, task_set_sha = _task_set( - repo_root=repo_root, - protocol=protocol, - manifest=manifest, - hypothesis_id="RC-H04", - ) - jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H04", - wave_name="complete", - task_ids=[task.task_id for task in tasks], - ) - arm_id = str(protocol["hypotheses"]["RC-H04"]["arm_ids"][0]) - store = AttemptStore(repo_root=repo_root, external_root=external_root) - evidence, sandbox = _load_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H04", - arm_id=arm_id, - source_git_sha=expected_source_sha, - tasks=tasks, - jobs=jobs, - task_set_sha256=task_set_sha, - prompt_sha256=prompt_sha, - sandbox=None, - runtime=runtime, - ) - attempts, programs = _collect_run_attempts( - protocol=protocol, - store=store, - evidence=evidence, - ) - attempts.sort(key=lambda row: (row["task_id"], row["attempt_index"])) - - summary_document = { - "schema_version": "pixcell-representation-curriculum-qwen60-summary-v1", - "study_id": protocol["study_id"], - "hypothesis_id": "RC-H04", - "arm_id": arm_id, - "summary": evidence.summary, - } - summary_document["logical_sha256"] = canonical_json_sha256(summary_document) - files: dict[PurePosixPath, bytes] = { - PurePosixPath("summary.json"): _json_bytes(summary_document), - PurePosixPath("attempts.jsonl"): _jsonl_bytes(attempts), - **programs, - } - file_inventory = [ - { - "path": path.as_posix(), - "sha256": _sha256_bytes(content), - "bytes": len(content), - } - for path, content in sorted(files.items(), key=lambda item: item[0]) - ] - release_manifest: dict[str, Any] = { - "schema_version": RELEASE_SCHEMA, - "study_id": protocol["study_id"], - "phase": "phase1-quality-first-qwen60-addendum", - "source_git_sha": expected_source_sha, - "protocol_path": str(PROTOCOL_FILE.relative_to(repo_root)), - "protocol_sha256": protocol["logical_sha256"], - "base_study": protocol["base_study"], - "decision": protocol["decision"], - "task_manifest_sha256": manifest["logical_sha256"], - "dataset": protocol["dataset"], - "prompt": prompt_binding, - "prompt_sha256": prompt_sha, - "sandbox": { - "image_ref": sandbox[0], - "image_id": sandbox[1], - }, - "runtime": runtime, - "run": { - "hypothesis_id": "RC-H04", - "arm_id": arm_id, - "attempts": len(attempts), - "run_manifest_record_sha256": evidence.manifest.record_sha256, - "wandb_run_id": deterministic_wandb_run_id( - RunKey(protocol["study_id"], "RC-H04", arm_id), - source_git_sha=expected_source_sha, - ), - }, - "files": file_inventory, - } - release_manifest["logical_sha256"] = canonical_json_sha256(release_manifest) - return files, release_manifest - - -def freeze( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, - output: Path | None = None, - validate_only: bool = False, -) -> dict[str, Any]: - output_root = _output_root(repo_root, output) - source, external = _validate_roots( - repo_root=repo_root, - external_root=external_root, - output_root=output_root, - ) - _require_source_clean_except_output( - repo_root=source, - output_root=output_root, - ) - files, manifest = _collect_bundle( - repo_root=source, - external_root=external, - expected_source_sha=expected_source_sha, - ) - expected = { - PurePosixPath("release_manifest.json"): _json_bytes(manifest), - **files, - } - if validate_only: - _assert_exact_output(output_root, expected) - else: - _write_create_or_verify(output_root, expected) - return manifest - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=REPO_ROOT) - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--output", type=Path) - result.add_argument("--validate", action="store_true") - return result - - -def main() -> None: - args = parser().parse_args() - report = freeze( - repo_root=args.repo_root, - external_root=args.external_root, - expected_source_sha=args.expected_source_sha, - output=args.output, - validate_only=args.validate, - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/protocol.json b/rl/studies/representation_curriculum_v1/extensions/qwen60/protocol.json deleted file mode 100644 index 43d5ccdb..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/protocol.json +++ /dev/null @@ -1,123 +0,0 @@ -{ - "arms": { - "qwen-on-60000": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 60000, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": true, - "thinking_effort": null - } - }, - "base_study": { - "arm_id": "qwen-on-32768", - "github_pull_request": 7, - "hypothesis": "RC-H03", - "merge_git_sha": "53de41ad43c0eeec36aa73bf4ae466f199711dfc", - "protocol_sha256": "bd51768349f512741080fe4f080f343d311e81c5946186a9541565737db9da33", - "source_git_sha": "53de41ad43c0eeec36aa73bf4ae466f199711dfc", - "study_id": "representation-curriculum-v1-qwen32" - }, - "contract_version": "pixcell-blind-phase-a-direct-v1", - "dataset": { - "configuration": "depth", - "development_split": "validation", - "logical_release_sha256": "676c49134d4d044c7d84426cf8eeecf09302e74ca4aa548956f14b7631a4d80b", - "repository": "qpaig-mit/pixcell", - "revision": "v2.0.0" - }, - "decision": { - "context_fit": { - "context_tokens": 65536, - "largest_frozen_f1_f8_prompt_tokens": 3685, - "minimum_context_reserve_tokens": 1851, - "minimum_f1_f8_completion_capacity_tokens": 61851, - "selected_max_output_tokens": 60000 - }, - "policy": "quality_first_user_directed_near_context_limit_extension", - "rationale": "The 65536-token context includes the rendered prompt, so a literal 64k completion cannot fit. A 60000-token cap is the largest round-number quality arm with at least 1851 tokens of reserve on every frozen F1-F8 task.", - "trigger_evidence": { - "arm_id": "qwen-on-32768", - "attempts": 8, - "cap_hit_rate": 0.125, - "hypothesis_id": "RC-H03", - "mean_iou": 0.02505331854632315, - "pure_executable_rate": 0.125, - "run_manifest_record_sha256": "d6ed8fc91e5a18bcc364bbd6fb167b12188a9ac3c08a8886de10774405aa42d5", - "source_git_sha": "53de41ad43c0eeec36aa73bf4ae466f199711dfc", - "summary_logical_sha256": "144769709b59150270487f72f5a9b84e95f5a4d8f6ec0cfcba4378190f79da95", - "wave": "smoke" - }, - "user_direction": "Test the strongest practical self-hostable Qwen setting even when it costs more." - }, - "hypotheses": { - "RC-H04": { - "arm_ids": [ - "qwen-on-60000" - ], - "attempts_per_task": 4, - "name": "Qwen quality-first 60k zero-shot F1-F8 baseline", - "status": "frozen", - "task_set": "f1_f8", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2, - 3, - 4 - ], - "task_count": 8 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 8 - } - } - } - }, - "launch": { - "confirmation_token": "PIXCELL_PHASE1_BASELINE", - "mutable_output_environment": "PIXCELL_STUDY_ROOT" - }, - "logical_sha256": "84d78f49ac16d3e1a13ccb73a8bddfb8837dad7abd61f6a93bb99ec027b490a1", - "pricing": { - "as_of": "2026-07-27", - "currency": "USD", - "models": { - "Qwen/Qwen3.6-35B-A3B": { - "prefill_cached": 0.108, - "prefill_uncached": 0.54, - "sample": 1.335 - } - }, - "source": "https://tinker-docs.thinkingmachines.ai/tinker/models/", - "unit": "per_million_tokens" - }, - "sampling": { - "evaluator_workers": 8, - "require_candidate_isolation": true, - "sampling_concurrency": 8, - "seed_namespace": "pixcell-representation-curriculum-v1", - "temperature": 1.0, - "top_p": 1.0 - }, - "schema_version": "pixcell-representation-curriculum-protocol-v1", - "study_id": "representation-curriculum-v1-qwen60", - "task_manifest": { - "logical_sha256": "e32a0a1522fba3c90163e5687c65ab5e92fdb77f4f875a73cbfcdaa432b44272", - "path": "task_manifest.json" - }, - "tracking": { - "entity": "aadarwal-massachusetts-institute-of-technology", - "group": "phase1-qwen60", - "local_record_is_authoritative": true, - "mode": "online", - "project": "pixcell-representation-curriculum-v1", - "provider": "wandb" - } -} diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/run_baseline.py b/rl/studies/representation_curriculum_v1/extensions/qwen60/run_baseline.py deleted file mode 100644 index b899f43b..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/run_baseline.py +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env python3 -"""Run the fixed Qwen thinking-on/60k F1-F8 baseline extension.""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from pathlib import Path - -from rl.evaluation.runner import run_baseline - - -REPO_ROOT = Path(__file__).resolve().parents[5] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--wave", required=True, choices=("smoke", "complete")) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--confirm-spend", default="") - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_STUDY_ROOT"]) - if os.environ.get("PIXCELL_STUDY_ROOT") - else None - ), - ) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit( - "set PIXCELL_STUDY_ROOT or provide --external-root outside the repository" - ) - report = asyncio.run( - run_baseline( - repo_root=REPO_ROOT, - protocol_file=PROTOCOL_FILE, - hypothesis_id="RC-H04", - wave_name=args.wave, - expected_source_sha=args.expected_source_sha, - external_root=args.external_root, - confirmation=args.confirm_spend, - ) - ) - print(json.dumps(report["summaries"], indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/test_freeze.py b/rl/studies/representation_curriculum_v1/extensions/qwen60/test_freeze.py deleted file mode 100644 index 55cd9416..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/test_freeze.py +++ /dev/null @@ -1,119 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.evaluation.attempt_store import RunKey -from rl.evaluation.tracking import deterministic_wandb_run_id -from rl.studies.representation_curriculum_v1.extensions.qwen60 import freeze -from rl.studies.representation_curriculum_v1.freeze_phase1 import Phase1FreezeError - - -REPO_ROOT = Path(__file__).resolve().parents[5] - - -def test_collect_bundle_uses_distinct_canonical_wandb_identity( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -): - source_sha = "a" * 40 - protocol = { - "study_id": "representation-curriculum-v1-qwen60", - "logical_sha256": "b" * 64, - "hypotheses": { - "RC-H04": { - "arm_ids": ["qwen-on-60000"], - "task_set": "f1_f8", - } - }, - "base_study": {"study_id": "representation-curriculum-v1-qwen32"}, - "decision": { - "policy": "quality_first_user_directed_near_context_limit_extension" - }, - "dataset": {"revision": "v2.0.0"}, - } - evidence = SimpleNamespace( - summary={"complete": True}, - manifest=SimpleNamespace(record_sha256="c" * 64), - ) - task = SimpleNamespace(task_id="F1") - job = SimpleNamespace(task_id="F1", attempt_index=1) - - monkeypatch.setattr(freeze, "_git_head", lambda _root: source_sha) - monkeypatch.setattr( - freeze, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr(freeze, "validate_runtime_stack", lambda _root: {}) - monkeypatch.setattr( - freeze, - "load_task_manifest", - lambda *_args, **_kwargs: {"logical_sha256": "d" * 64}, - ) - monkeypatch.setattr(freeze, "_prompt_binding", lambda: ({}, "e" * 64)) - monkeypatch.setattr( - freeze, - "_task_set", - lambda **_kwargs: ([task], "f" * 64), - ) - monkeypatch.setattr(freeze, "jobs_for_wave", lambda *_args, **_kwargs: [job]) - monkeypatch.setattr( - freeze, - "_load_run", - lambda **_kwargs: (evidence, ("sha256:image", "sha256:image")), - ) - monkeypatch.setattr( - freeze, - "_collect_run_attempts", - lambda **_kwargs: ( - [{"task_id": "F1", "attempt_index": 1}], - {}, - ), - ) - - _files, manifest = freeze._collect_bundle( - repo_root=REPO_ROOT, - external_root=tmp_path, - expected_source_sha=source_sha, - ) - expected = deterministic_wandb_run_id( - RunKey( - "representation-curriculum-v1-qwen60", - "RC-H04", - "qwen-on-60000", - ), - source_git_sha=source_sha, - ) - qwen32 = deterministic_wandb_run_id( - RunKey( - "representation-curriculum-v1-qwen32", - "RC-H03", - "qwen-on-32768", - ), - source_git_sha=source_sha, - ) - assert manifest["run"]["wandb_run_id"] == expected - assert manifest["run"]["wandb_run_id"] != qwen32 - - -def test_output_root_rejects_symlink_escape(tmp_path: Path): - repo = tmp_path / "repo" - parent = repo / "data/training/representation-curriculum-v1" - outside = tmp_path / "outside" - parent.mkdir(parents=True) - outside.mkdir() - (parent / "phase1-qwen60").symlink_to(outside, target_is_directory=True) - - with pytest.raises(Phase1FreezeError, match="outside the source repository"): - freeze._output_root(repo, None) - - -def test_output_root_is_fixed_and_cannot_overwrite_qwen32(tmp_path: Path): - repo = tmp_path / "repo" - repo.mkdir() - qwen32 = repo / "data/training/representation-curriculum-v1/phase1-qwen32" - with pytest.raises(Phase1FreezeError, match="must be written exactly"): - freeze._output_root(repo, qwen32) diff --git a/rl/studies/representation_curriculum_v1/extensions/qwen60/test_protocol.py b/rl/studies/representation_curriculum_v1/extensions/qwen60/test_protocol.py deleted file mode 100644 index 6c9e4ecb..00000000 --- a/rl/studies/representation_curriculum_v1/extensions/qwen60/test_protocol.py +++ /dev/null @@ -1,225 +0,0 @@ -from __future__ import annotations - -import json -import subprocess -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.evaluation.protocol import ( - _require_protocol_at_head, - deterministic_seed, - jobs_for_wave, - load_archived_protocol, - load_protocol, - task_manifest_path, -) -from rl.evaluation.tasks import load_task_manifest, load_task_set -from rl.studies.representation_curriculum_v1.extensions.qwen60 import ( - run_baseline as fixed_runner, -) - - -REPO_ROOT = Path(__file__).resolve().parents[5] -BASE_PROTOCOL = ( - REPO_ROOT / "rl/studies/representation_curriculum_v1/protocol.json" -) -QWEN32_PROTOCOL = ( - REPO_ROOT - / "rl/studies/representation_curriculum_v1/extensions/qwen32/protocol.json" -) -EXTENSION_PROTOCOL = Path(__file__).with_name("protocol.json") -QWEN32_MERGE_SHA = "53de41ad43c0eeec36aa73bf4ae466f199711dfc" - - -def _benchmark_tasks(): - manifest = load_task_manifest(task_manifest_path(REPO_ROOT), repo_root=REPO_ROOT) - return load_task_set( - repo_root=REPO_ROOT, - manifest=manifest, - task_set="f1_f8", - ) - - -def test_extension_is_one_fixed_near_context_limit_arm(): - protocol = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - qwen32 = load_archived_protocol( - REPO_ROOT, - protocol_file=QWEN32_PROTOCOL, - ) - assert protocol["study_id"] == "representation-curriculum-v1-qwen60" - assert protocol["base_study"] == { - "arm_id": "qwen-on-32768", - "github_pull_request": 7, - "hypothesis": "RC-H03", - "merge_git_sha": QWEN32_MERGE_SHA, - "protocol_sha256": qwen32["logical_sha256"], - "source_git_sha": QWEN32_MERGE_SHA, - "study_id": "representation-curriculum-v1-qwen32", - } - assert protocol["hypotheses"]["RC-H04"]["arm_ids"] == ["qwen-on-60000"] - assert protocol["arms"]["qwen-on-60000"] == { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 60000, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": True, - "thinking_effort": None, - } - for field in ( - "contract_version", - "dataset", - "launch", - "pricing", - "sampling", - "task_manifest", - ): - assert protocol[field] == qwen32[field] - qwen32_arm = dict(qwen32["arms"]["qwen-on-32768"]) - qwen32_arm["max_output_tokens"] = 60000 - assert protocol["arms"]["qwen-on-60000"] == qwen32_arm - assert protocol["decision"]["trigger_evidence"] == { - "arm_id": "qwen-on-32768", - "attempts": 8, - "cap_hit_rate": 0.125, - "hypothesis_id": "RC-H03", - "mean_iou": 0.02505331854632315, - "pure_executable_rate": 0.125, - "run_manifest_record_sha256": ( - "d6ed8fc91e5a18bcc364bbd6fb167b12188a9ac3c08a8886de10774405aa42d5" - ), - "source_git_sha": QWEN32_MERGE_SHA, - "summary_logical_sha256": ( - "144769709b59150270487f72f5a9b84e95f5a4d8f6ec0cfcba4378190f79da95" - ), - "wave": "smoke", - } - - -def test_qwen32_base_commit_contains_the_bound_protocol_and_pr_merge(): - relative = QWEN32_PROTOCOL.relative_to(REPO_ROOT).as_posix() - committed = subprocess.check_output( - ["git", "show", f"{QWEN32_MERGE_SHA}:{relative}"], - cwd=REPO_ROOT, - ) - assert json.loads(committed) == json.loads(QWEN32_PROTOCOL.read_bytes()) - subject = subprocess.check_output( - ["git", "show", "-s", "--format=%s", QWEN32_MERGE_SHA], - cwd=REPO_ROOT, - text=True, - ).strip() - assert subject == "Merge pull request #7 from QPG-MIT/agent/qwen32-f1f8-baseline" - - -def test_extension_smoke_and_complete_are_exact_f1_f8_g4_jobs(): - protocol = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - task_ids = [task.task_id for task in _benchmark_tasks()] - smoke = jobs_for_wave( - protocol, - hypothesis_id="RC-H04", - wave_name="smoke", - task_ids=task_ids, - ) - complete = jobs_for_wave( - protocol, - hypothesis_id="RC-H04", - wave_name="complete", - task_ids=task_ids, - ) - assert len(smoke) == 8 - assert len(complete) == 32 - assert {job.arm_id for job in complete} == {"qwen-on-60000"} - assert {job.task_id for job in smoke} == set(task_ids) - assert {job.attempt_index for job in complete} == {1, 2, 3, 4} - - -def test_extension_keeps_base_seed_pairing(): - base = load_archived_protocol(REPO_ROOT, protocol_file=BASE_PROTOCOL) - extension = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - for task in _benchmark_tasks(): - for attempt_index in range(1, 5): - assert deterministic_seed( - extension, - task_id=task.task_id, - attempt_index=attempt_index, - ) == deterministic_seed( - base, - task_id=task.task_id, - attempt_index=attempt_index, - ) - - -def test_archived_60k_context_arithmetic_is_internally_consistent(): - protocol = load_archived_protocol( - REPO_ROOT, - protocol_file=EXTENSION_PROTOCOL, - ) - arm = protocol["arms"]["qwen-on-60000"] - context_fit = protocol["decision"]["context_fit"] - largest_prompt = context_fit["largest_frozen_f1_f8_prompt_tokens"] - context_tokens = arm["context_tokens"] - max_output_tokens = arm["max_output_tokens"] - minimum_reserve = context_tokens - largest_prompt - max_output_tokens - assert context_tokens - largest_prompt == context_fit[ - "minimum_f1_f8_completion_capacity_tokens" - ] - assert minimum_reserve == context_fit["minimum_context_reserve_tokens"] - assert minimum_reserve >= 0 - assert context_fit["selected_max_output_tokens"] == max_output_tokens - assert largest_prompt + 64_000 > context_tokens - assert largest_prompt + max_output_tokens <= context_tokens - - -def test_protocol_override_must_be_source_controlled(tmp_path: Path): - outside = tmp_path / "protocol.json" - outside.write_text(EXTENSION_PROTOCOL.read_text(encoding="utf-8"), encoding="utf-8") - with pytest.raises(ValueError, match="inside the source repository"): - load_protocol(REPO_ROOT, protocol_file=outside) - - -def test_fixed_runner_forwards_the_extension_protocol_to_the_head_guard( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, - capsys: pytest.CaptureFixture[str], -): - captured: dict[str, object] = {} - - async def fake_run_baseline(**kwargs): - captured.update(kwargs) - return {"summaries": {}} - - class FakeParser: - @staticmethod - def parse_args(): - return SimpleNamespace( - wave="smoke", - expected_source_sha="a" * 40, - confirm_spend="PIXCELL_PHASE1_BASELINE", - external_root=tmp_path, - ) - - monkeypatch.setattr(fixed_runner, "run_baseline", fake_run_baseline) - monkeypatch.setattr(fixed_runner, "parser", FakeParser) - fixed_runner.main() - assert captured["protocol_file"] == EXTENSION_PROTOCOL - assert captured["hypothesis_id"] == "RC-H04" - assert captured["wave_name"] == "smoke" - assert json.loads(capsys.readouterr().out) == {} - - -def test_paid_protocol_guard_accepts_head_and_rejects_non_protocol_path(): - _require_protocol_at_head(REPO_ROOT, BASE_PROTOCOL) - with pytest.raises(ValueError, match="tracked by the paid-launch commit"): - _require_protocol_at_head(REPO_ROOT, REPO_ROOT / ".git") diff --git a/rl/studies/representation_curriculum_v1/freeze_phase1.py b/rl/studies/representation_curriculum_v1/freeze_phase1.py deleted file mode 100644 index 47a83c88..00000000 --- a/rl/studies/representation_curriculum_v1/freeze_phase1.py +++ /dev/null @@ -1,1027 +0,0 @@ -#!/usr/bin/env python3 -"""Freeze or validate the compact, tracked Phase 1 baseline evidence.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import os -import re -import subprocess -import uuid -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any - -from rl.common.output import extract_code -from rl.common.preprocess import IMAGE_PREPROCESS_VERSION -from rl.common.prompt import CONTRACT_VERSION, prompt_asset_hashes -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.attempt_store import ( - AttemptKey, - AttemptStore, - RunKey, - StoredRecord, -) -from rl.evaluation.protocol import ( - JobSpec, - jobs_for_wave, - load_protocol, - resolved_arm_ids, - task_manifest_path, -) -from rl.evaluation.summarize import choose_operating_point, summarize_arm -from rl.evaluation.tasks import ( - EvaluationTask, - canonical_json_sha256, - load_task_manifest, - load_task_set, -) -from rl.evaluation.tracking import deterministic_wandb_run_id - - -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PHASE1_RELATIVE_ROOT = Path( - "data/training/representation-curriculum-v1/phase1" -) -RELEASE_SCHEMA = "pixcell-representation-curriculum-phase1-release-v1" -SUMMARIES_SCHEMA = "pixcell-representation-curriculum-phase1-summaries-v1" -ATTEMPT_INDEX_SCHEMA = "pixcell-representation-curriculum-attempt-v1" -_SOURCE_SHA = re.compile(r"^[0-9a-f]{40}$") -_HYPOTHESIS_ORDER = ("RC-H00", "RC-H00R", "RC-H01", "RC-H02") - - -class Phase1FreezeError(RuntimeError): - """The external ledger cannot certify the tracked Phase 1 release.""" - - -@dataclass(frozen=True) -class _RunEvidence: - hypothesis_id: str - arm_id: str - tasks: tuple[EvaluationTask, ...] - jobs: tuple[JobSpec, ...] - manifest: StoredRecord - summary: dict[str, Any] - - -@dataclass(frozen=True) -class _Bundle: - files: dict[PurePosixPath, bytes] - release_manifest: dict[str, Any] - - @property - def all_files(self) -> dict[PurePosixPath, bytes]: - return { - PurePosixPath("release_manifest.json"): _json_bytes( - self.release_manifest - ), - **self.files, - } - - -def _json_bytes(value: Any) -> bytes: - try: - return ( - json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - indent=2, - sort_keys=True, - ) - + "\n" - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise Phase1FreezeError("release evidence is not finite JSON") from exc - - -def _jsonl_bytes(values: list[dict[str, Any]]) -> bytes: - try: - return b"".join( - ( - json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ) - + "\n" - ).encode("utf-8") - for value in values - ) - except (TypeError, ValueError) as exc: - raise Phase1FreezeError("attempt index is not finite JSON") from exc - - -def _sha256_bytes(value: bytes) -> str: - return hashlib.sha256(value).hexdigest() - - -def _load_json(path: Path) -> dict[str, Any]: - def reject_constant(value: str) -> Any: - raise Phase1FreezeError(f"{path} contains non-finite number {value}") - - def reject_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: - result: dict[str, Any] = {} - for key, value in pairs: - if key in result: - raise Phase1FreezeError( - f"{path} contains duplicate key {key!r}" - ) - result[key] = value - return result - - try: - value = json.loads( - path.read_bytes(), - parse_constant=reject_constant, - object_pairs_hook=reject_duplicates, - ) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise Phase1FreezeError(f"cannot read canonical JSON from {path}") from exc - if not isinstance(value, dict): - raise Phase1FreezeError(f"{path} must contain a JSON object") - return value - - -def _git_head(repo_root: Path) -> str: - try: - return subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo_root, - text=True, - ).strip() - except (OSError, subprocess.CalledProcessError) as exc: - raise Phase1FreezeError("repository HEAD cannot be resolved") from exc - - -def _changed_paths(repo_root: Path) -> set[Path]: - commands = ( - ["git", "diff", "--name-only", "-z"], - ["git", "diff", "--cached", "--name-only", "-z"], - ["git", "ls-files", "--others", "--exclude-standard", "-z"], - ) - result: set[Path] = set() - for command in commands: - try: - output = subprocess.check_output(command, cwd=repo_root) - except (OSError, subprocess.CalledProcessError) as exc: - raise Phase1FreezeError("repository worktree cannot be inspected") from exc - for raw in output.split(b"\0"): - if raw: - result.add((repo_root / os.fsdecode(raw)).resolve(strict=False)) - return result - - -def _require_source_clean_except_output( - *, - repo_root: Path, - output_root: Path, -) -> None: - drift = sorted( - str(path.relative_to(repo_root)) - for path in _changed_paths(repo_root) - if path != output_root and not path.is_relative_to(output_root) - ) - if drift: - raise Phase1FreezeError( - "source worktree differs from the paid-launch commit outside the " - f"tracked Phase 1 output: {drift}" - ) - - -def _phase1_output_root(repo_root: Path, requested: Path | None) -> Path: - root = repo_root.expanduser().resolve(strict=True) - expected = (root / PHASE1_RELATIVE_ROOT).resolve(strict=False) - if expected == root or not expected.is_relative_to(root): - raise Phase1FreezeError( - "the tracked Phase 1 path resolves outside the source repository" - ) - observed = ( - expected - if requested is None - else requested.expanduser().resolve(strict=False) - ) - if observed != expected: - raise Phase1FreezeError( - f"Phase 1 evidence must be written exactly to {expected}" - ) - return expected - - -def _validate_roots( - *, - repo_root: Path, - external_root: Path, - output_root: Path, -) -> tuple[Path, Path]: - source = repo_root.expanduser().resolve(strict=True) - external = external_root.expanduser().resolve(strict=True) - if external == source or external.is_relative_to(source): - raise Phase1FreezeError( - "the mutable attempt ledger must remain outside the source repository" - ) - if ( - output_root == external - or output_root.is_relative_to(external) - or external.is_relative_to(output_root) - ): - raise Phase1FreezeError("source ledger and tracked output overlap") - return source, external - - -def _prompt_binding() -> tuple[dict[str, str], str]: - binding = { - "contract_version": CONTRACT_VERSION, - "preprocess_version": IMAGE_PREPROCESS_VERSION, - **prompt_asset_hashes(), - } - return binding, canonical_json_sha256(binding) - - -def _task_set( - *, - repo_root: Path, - protocol: dict[str, Any], - manifest: dict[str, Any], - hypothesis_id: str, -) -> tuple[list[EvaluationTask], str]: - task_set_name = str(protocol["hypotheses"][hypothesis_id]["task_set"]) - tasks = load_task_set( - repo_root=repo_root, - manifest=manifest, - task_set=task_set_name, - ) - task_set_sha = canonical_json_sha256(manifest["task_sets"][task_set_name]) - return tasks, task_set_sha - - -def _expected_run_manifest( - *, - protocol: dict[str, Any], - hypothesis_id: str, - arm_id: str, - source_git_sha: str, - task_set_sha256: str, - prompt_sha256: str, - sandbox_image_ref: str, - sandbox_image_id: str, - runtime: dict[str, Any], -) -> dict[str, Any]: - if hypothesis_id != "RC-H01": - allowed = resolved_arm_ids( - protocol, - hypothesis_id, - selection=( - {"rescue_required": True} - if hypothesis_id == "RC-H00R" - else None - ), - ) - if arm_id not in allowed: - raise Phase1FreezeError(f"{hypothesis_id} does not register {arm_id}") - return { - "source_git_sha": source_git_sha, - "protocol_sha256": protocol["logical_sha256"], - "task_set_sha256": task_set_sha256, - "prompt_sha256": prompt_sha256, - "sandbox_image_ref": sandbox_image_ref, - "sandbox_image_id": sandbox_image_id, - "runtime": runtime, - "model_binding": { - "arm_id": arm_id, - **protocol["arms"][arm_id], - "temperature": protocol["sampling"]["temperature"], - "top_p": protocol["sampling"]["top_p"], - }, - } - - -def _load_run( - *, - store: AttemptStore, - protocol: dict[str, Any], - hypothesis_id: str, - arm_id: str, - source_git_sha: str, - tasks: list[EvaluationTask], - jobs: list[JobSpec], - task_set_sha256: str, - prompt_sha256: str, - sandbox: tuple[str, str] | None, - runtime: dict[str, Any], -) -> tuple[_RunEvidence, tuple[str, str]]: - run_key = RunKey(protocol["study_id"], hypothesis_id, arm_id) - manifest = store.load_run_manifest(run_key) - if manifest is None: - raise Phase1FreezeError(f"missing run manifest for {hypothesis_id}/{arm_id}") - observed_sandbox = ( - str(manifest.payload.get("sandbox_image_ref", "")), - str(manifest.payload.get("sandbox_image_id", "")), - ) - if sandbox is not None and observed_sandbox != sandbox: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} used a different evaluator sandbox" - ) - expected = _expected_run_manifest( - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - source_git_sha=source_git_sha, - task_set_sha256=task_set_sha256, - prompt_sha256=prompt_sha256, - sandbox_image_ref=observed_sandbox[0], - sandbox_image_id=observed_sandbox[1], - runtime=runtime, - ) - if manifest.payload != expected: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} run manifest does not match the frozen launch" - ) - summary = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id=hypothesis_id, - arm_id=arm_id, - jobs=jobs, - tasks=tasks, - ) - if not summary["complete"]: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} is incomplete: {summary['missing']}" - ) - external_summary = ( - store.external_root.joinpath(*store.run_relative_path(run_key).parts) - / "summary.json" - ) - if not external_summary.is_file(): - raise Phase1FreezeError( - f"missing canonical summary for {hypothesis_id}/{arm_id}" - ) - if _load_json(external_summary) != summary: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} summary does not reproduce from attempts" - ) - return ( - _RunEvidence( - hypothesis_id=hypothesis_id, - arm_id=arm_id, - tasks=tuple(tasks), - jobs=tuple( - job - for job in jobs - if job.hypothesis_id == hypothesis_id and job.arm_id == arm_id - ), - manifest=manifest, - summary=summary, - ), - observed_sandbox, - ) - - -def _validate_receipt_and_evaluation( - *, - protocol: dict[str, Any], - store: AttemptStore, - task: EvaluationTask, - job: JobSpec, -) -> tuple[StoredRecord, StoredRecord, str]: - key = AttemptKey( - protocol["study_id"], - job.hypothesis_id, - job.arm_id, - job.task_id, - job.attempt_index, - ) - receipt = store.load_sampling_receipt(key) - evaluation = store.load_evaluation(key) - if receipt is None or evaluation is None: - raise Phase1FreezeError(f"missing immutable attempt record for {key}") - - request = receipt.payload.get("request") - sample = receipt.payload.get("sample") - if not isinstance(request, dict) or not isinstance(sample, dict): - raise Phase1FreezeError(f"{key} has an invalid sampling payload") - arm = protocol["arms"][job.arm_id] - expected_request = { - "max_tokens": int(arm["max_output_tokens"]), - "temperature": protocol["sampling"]["temperature"], - "top_p": protocol["sampling"]["top_p"], - "seed": job.seed, - } - if request != expected_request: - raise Phase1FreezeError(f"{key} sampling request differs from protocol") - required_sample = { - "prompt_tokens", - "completion_tokens", - "stop_reason", - "cap_hit", - "raw_text", - "completion_text", - "reasoning_text", - "answer_text", - "reasoning_tokens_exact", - "answer_tokens_exact", - "reasoning_tokens_estimate", - "answer_tokens_estimate", - "channel_token_count_basis", - "channel_parse_complete", - } - if set(sample) != required_sample: - raise Phase1FreezeError(f"{key} sample fields differ from the frozen schema") - prompt_tokens = sample["prompt_tokens"] - completion_tokens = sample["completion_tokens"] - if ( - isinstance(prompt_tokens, bool) - or not isinstance(prompt_tokens, int) - or prompt_tokens < 1 - or isinstance(completion_tokens, bool) - or not isinstance(completion_tokens, int) - or completion_tokens < 0 - or completion_tokens > int(request["max_tokens"]) - ): - raise Phase1FreezeError(f"{key} has invalid token accounting") - if bool(sample["cap_hit"]) != (sample["stop_reason"] == "length"): - raise Phase1FreezeError(f"{key} cap-hit flag differs from its stop reason") - if sample["cap_hit"] and completion_tokens != int(request["max_tokens"]): - raise Phase1FreezeError(f"{key} length stop did not reach the output cap") - reasoning_exact = sample["reasoning_tokens_exact"] - answer_exact = sample["answer_tokens_exact"] - if (reasoning_exact is None) != (answer_exact is None): - raise Phase1FreezeError(f"{key} has a partial exact channel partition") - if reasoning_exact is not None and ( - isinstance(reasoning_exact, bool) - or not isinstance(reasoning_exact, int) - or reasoning_exact < 0 - or isinstance(answer_exact, bool) - or not isinstance(answer_exact, int) - or answer_exact < 0 - or reasoning_exact + answer_exact != completion_tokens - ): - raise Phase1FreezeError(f"{key} exact channel tokens do not partition output") - raw_text = str(sample["raw_text"]) - completion = str(sample["completion_text"]) - if receipt.payload.get("raw_completion_sha256") != hashlib.sha256( - raw_text.encode("utf-8") - ).hexdigest(): - raise Phase1FreezeError(f"{key} raw completion digest mismatch") - if receipt.payload.get("completion_sha256") != hashlib.sha256( - completion.encode("utf-8") - ).hexdigest(): - raise Phase1FreezeError(f"{key} completion digest mismatch") - if not math.isfinite(float(receipt.payload.get("sampling_seconds", -1))): - raise Phase1FreezeError(f"{key} has non-finite sampling duration") - costs = receipt.payload.get("cost_estimate_usd") - if not isinstance(costs, dict) or set(costs) != { - "cached_prefill", - "uncached_prefill", - }: - raise Phase1FreezeError(f"{key} has invalid cost evidence") - rates = protocol["pricing"]["models"][arm["model"]] - sample_cost = completion_tokens * float(rates["sample"]) / 1_000_000 - expected_costs = { - "cached_prefill": ( - prompt_tokens * float(rates["prefill_cached"]) / 1_000_000 - + sample_cost - ), - "uncached_prefill": ( - prompt_tokens * float(rates["prefill_uncached"]) / 1_000_000 - + sample_cost - ), - } - if costs != expected_costs: - raise Phase1FreezeError(f"{key} cost evidence does not reproduce") - - measured = evaluation.payload - required_evaluation = { - "status", - "attribution", - "pure_executable", - "measurement_available", - "measured_iou", - "measured_dice", - "aggregation_iou", - "program", - "program_sha256", - "reference_sha256", - "violations", - "error", - "retryable", - "evaluation_seconds", - "diagnostics", - } - allowed_evaluation = required_evaluation | {"render_sha256"} - if not required_evaluation.issubset(measured) or not set(measured).issubset( - allowed_evaluation - ): - raise Phase1FreezeError( - f"{key} evaluation fields differ from the frozen schema" - ) - if measured["attribution"] != "model": - raise Phase1FreezeError( - f"{key} is an evaluator/reference failure, not a model result" - ) - program = str(measured["program"]) - if program != extract_code(completion): - raise Phase1FreezeError(f"{key} stored program does not match extraction") - if measured["program_sha256"] != hashlib.sha256( - program.encode("utf-8") - ).hexdigest(): - raise Phase1FreezeError(f"{key} program digest mismatch") - if measured["reference_sha256"] != task.reference.target_image_sha256: - raise Phase1FreezeError(f"{key} reference digest mismatch") - aggregation_iou = float(measured["aggregation_iou"]) - if not math.isfinite(aggregation_iou) or not 0 <= aggregation_iou <= 1: - raise Phase1FreezeError(f"{key} has invalid aggregation IoU") - measured_iou = measured["measured_iou"] - expected_iou = 0.0 if measured_iou is None else float(measured_iou) - if aggregation_iou != expected_iou: - raise Phase1FreezeError(f"{key} aggregation IoU is not the raw model IoU") - if bool(measured["measurement_available"]) != (measured_iou is not None): - raise Phase1FreezeError(f"{key} measurement-availability flag is inconsistent") - if measured_iou is not None and "render_sha256" not in measured: - raise Phase1FreezeError(f"{key} measured geometry has no render digest") - if bool(measured["pure_executable"]) != (measured["status"] == "ok"): - raise Phase1FreezeError(f"{key} executable flag is inconsistent with status") - return receipt, evaluation, program - - -def _collect_run_attempts( - *, - protocol: dict[str, Any], - store: AttemptStore, - evidence: _RunEvidence, -) -> tuple[list[dict[str, Any]], dict[PurePosixPath, bytes]]: - tasks = {task.task_id: task for task in evidence.tasks} - rows: list[dict[str, Any]] = [] - programs: dict[PurePosixPath, bytes] = {} - for job in evidence.jobs: - task = tasks[job.task_id] - receipt, evaluation, program = _validate_receipt_and_evaluation( - protocol=protocol, - store=store, - task=task, - job=job, - ) - program_path = PurePosixPath( - "programs", - job.hypothesis_id, - job.arm_id, - job.task_id, - f"attempt_{job.attempt_index:02d}.py", - ) - program_bytes = program.encode("utf-8") - if program_path in programs and programs[program_path] != program_bytes: - raise Phase1FreezeError(f"program-path collision at {program_path}") - programs[program_path] = program_bytes - sample = receipt.payload["sample"] - measured = evaluation.payload - rows.append( - { - "schema_version": ATTEMPT_INDEX_SCHEMA, - "study_id": protocol["study_id"], - "hypothesis_id": job.hypothesis_id, - "arm_id": job.arm_id, - "task_id": job.task_id, - "level": task.level, - "representation_id": task.representation_id, - "attempt_index": job.attempt_index, - "seed": job.seed, - "status": measured["status"], - "violations": measured["violations"], - "error": measured["error"], - "retryable": measured["retryable"], - "pure_executable": measured["pure_executable"], - "measurement_available": measured["measurement_available"], - "aggregation_iou": measured["aggregation_iou"], - "measured_iou": measured["measured_iou"], - "measured_dice": measured["measured_dice"], - "cap_hit": sample["cap_hit"], - "stop_reason": sample["stop_reason"], - "prompt_tokens": sample["prompt_tokens"], - "completion_tokens": sample["completion_tokens"], - "reasoning_tokens_exact": sample["reasoning_tokens_exact"], - "answer_tokens_exact": sample["answer_tokens_exact"], - "reasoning_tokens_estimate": sample[ - "reasoning_tokens_estimate" - ], - "answer_tokens_estimate": sample["answer_tokens_estimate"], - "channel_token_count_basis": sample[ - "channel_token_count_basis" - ], - "channel_parse_complete": sample["channel_parse_complete"], - "estimated_cost_usd": receipt.payload["cost_estimate_usd"], - "sampling_seconds": receipt.payload["sampling_seconds"], - "evaluation_seconds": measured["evaluation_seconds"], - "diagnostics": measured["diagnostics"], - "raw_completion_sha256": receipt.payload[ - "raw_completion_sha256" - ], - "completion_sha256": receipt.payload["completion_sha256"], - "sampling_record_sha256": receipt.record_sha256, - "evaluation_record_sha256": evaluation.record_sha256, - "program_path": program_path.as_posix(), - "program_sha256": measured["program_sha256"], - "reference_sha256": measured["reference_sha256"], - **( - {"render_sha256": measured["render_sha256"]} - if measured.get("render_sha256") is not None - else {} - ), - } - ) - return rows, programs - - -def _selection_path(store: AttemptStore, name: str) -> Path: - return ( - store.external_root - / "studies" - / "representation-curriculum-v1" - / "selections" - / name - ) - - -def _require_selection(path: Path, expected: dict[str, Any]) -> None: - if not path.is_file(): - raise Phase1FreezeError(f"missing canonical selection receipt: {path}") - if _load_json(path) != expected: - raise Phase1FreezeError( - f"{path.name} does not reproduce from immutable attempts" - ) - - -def _collect_bundle( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, -) -> _Bundle: - if not _SOURCE_SHA.fullmatch(expected_source_sha): - raise Phase1FreezeError( - "expected source SHA must be a full lowercase Git commit SHA" - ) - if _git_head(repo_root) != expected_source_sha: - raise Phase1FreezeError("current source HEAD differs from the launch SHA") - protocol = load_protocol(repo_root, protocol_file=PROTOCOL_FILE) - manifest = load_task_manifest(task_manifest_path(repo_root), repo_root=repo_root) - prompt_binding, prompt_sha = _prompt_binding() - runtime = validate_runtime_stack(repo_root) - store = AttemptStore(repo_root=repo_root, external_root=external_root) - source_git_sha = expected_source_sha - sandbox: tuple[str, str] | None = None - runs: list[_RunEvidence] = [] - - dev_tasks, dev_task_sha = _task_set( - repo_root=repo_root, - protocol=protocol, - manifest=manifest, - hypothesis_id="RC-H00", - ) - dev_ids = [task.task_id for task in dev_tasks] - h00_jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H00", - wave_name="complete", - task_ids=dev_ids, - ) - h00_summaries: dict[str, dict[str, Any]] = {} - for arm_id in protocol["hypotheses"]["RC-H00"]["arm_ids"]: - evidence, sandbox = _load_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H00", - arm_id=str(arm_id), - source_git_sha=source_git_sha, - tasks=dev_tasks, - jobs=h00_jobs, - task_set_sha256=dev_task_sha, - prompt_sha256=prompt_sha, - sandbox=sandbox, - runtime=runtime, - ) - runs.append(evidence) - h00_summaries[str(arm_id)] = evidence.summary - - preliminary = choose_operating_point( - protocol=protocol, - source_git_sha=source_git_sha, - source_summaries=h00_summaries, - ) - _require_selection( - _selection_path(store, "operating_point.preliminary.json"), - preliminary, - ) - final_selection = preliminary - if preliminary["rescue_required"]: - rescue_arm = str(protocol["hypotheses"]["RC-H00R"]["arm_ids"][0]) - h00r_jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H00R", - wave_name="complete", - task_ids=dev_ids, - selection=preliminary, - ) - rescue, sandbox = _load_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H00R", - arm_id=rescue_arm, - source_git_sha=source_git_sha, - tasks=dev_tasks, - jobs=h00r_jobs, - task_set_sha256=dev_task_sha, - prompt_sha256=prompt_sha, - sandbox=sandbox, - runtime=runtime, - ) - runs.append(rescue) - final_selection = choose_operating_point( - protocol=protocol, - source_git_sha=source_git_sha, - source_summaries=h00_summaries, - rescue_summary=rescue.summary, - ) - if not final_selection["complete"]: - raise Phase1FreezeError("operating-point selection is not complete") - _require_selection( - _selection_path(store, "operating_point.json"), - final_selection, - ) - - benchmark_tasks, benchmark_task_sha = _task_set( - repo_root=repo_root, - protocol=protocol, - manifest=manifest, - hypothesis_id="RC-H01", - ) - benchmark_ids = [task.task_id for task in benchmark_tasks] - for hypothesis_id in ("RC-H01", "RC-H02"): - jobs = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name="complete", - task_ids=benchmark_ids, - selection=final_selection if hypothesis_id == "RC-H01" else None, - ) - for arm_id in resolved_arm_ids( - protocol, - hypothesis_id, - selection=final_selection if hypothesis_id == "RC-H01" else None, - ): - evidence, sandbox = _load_run( - store=store, - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - source_git_sha=source_git_sha, - tasks=benchmark_tasks, - jobs=jobs, - task_set_sha256=benchmark_task_sha, - prompt_sha256=prompt_sha, - sandbox=sandbox, - runtime=runtime, - ) - runs.append(evidence) - - attempts: list[dict[str, Any]] = [] - programs: dict[PurePosixPath, bytes] = {} - for evidence in runs: - run_attempts, run_programs = _collect_run_attempts( - protocol=protocol, - store=store, - evidence=evidence, - ) - attempts.extend(run_attempts) - for path, content in run_programs.items(): - if path in programs and programs[path] != content: - raise Phase1FreezeError(f"duplicate program evidence at {path}") - programs[path] = content - attempts.sort( - key=lambda value: ( - _HYPOTHESIS_ORDER.index(value["hypothesis_id"]), - value["arm_id"], - value["task_id"], - value["attempt_index"], - ) - ) - - summaries_document = { - "schema_version": SUMMARIES_SCHEMA, - "study_id": protocol["study_id"], - "summaries": { - f"{run.hypothesis_id}/{run.arm_id}": run.summary for run in runs - }, - } - summaries_document["logical_sha256"] = canonical_json_sha256( - summaries_document - ) - files: dict[PurePosixPath, bytes] = { - PurePosixPath("operating_point_selection.json"): _json_bytes( - final_selection - ), - PurePosixPath("summaries.json"): _json_bytes(summaries_document), - PurePosixPath("attempts.jsonl"): _jsonl_bytes(attempts), - **programs, - } - file_inventory = [ - { - "path": path.as_posix(), - "sha256": _sha256_bytes(content), - "bytes": len(content), - } - for path, content in sorted(files.items(), key=lambda item: item[0]) - ] - release_manifest: dict[str, Any] = { - "schema_version": RELEASE_SCHEMA, - "study_id": protocol["study_id"], - "phase": "phase1-zero-shot-baselines", - "source_git_sha": source_git_sha, - "protocol_sha256": protocol["logical_sha256"], - "task_manifest_sha256": manifest["logical_sha256"], - "dataset": protocol["dataset"], - "prompt": prompt_binding, - "prompt_sha256": prompt_sha, - "sandbox": { - "image_ref": sandbox[0] if sandbox is not None else "", - "image_id": sandbox[1] if sandbox is not None else "", - }, - "selection_sha256": final_selection["logical_sha256"], - "selected_qwen_arm": final_selection["selected_arm"], - "attempt_count": len(attempts), - "program_count": len(programs), - "runs": [ - { - "hypothesis_id": run.hypothesis_id, - "arm_id": run.arm_id, - "expected_attempts": len(run.jobs), - "run_manifest_payload": run.manifest.payload, - "run_manifest_payload_sha256": run.manifest.payload_sha256, - "run_manifest_record_sha256": run.manifest.record_sha256, - "summary_sha256": run.summary["logical_sha256"], - "wandb_run_id": deterministic_wandb_run_id( - RunKey( - protocol["study_id"], - run.hypothesis_id, - run.arm_id, - ), - source_git_sha=source_git_sha, - ), - } - for run in runs - ], - "tracking": { - "provider": protocol["tracking"]["provider"], - "project": protocol["tracking"]["project"], - "entity": protocol["tracking"].get("entity"), - "group": protocol["tracking"].get("group"), - "local_record_is_authoritative": protocol["tracking"][ - "local_record_is_authoritative" - ], - }, - "files": file_inventory, - "excluded_external_state": [ - "raw_completion_text", - "reasoning_text", - "answer_text", - "wandb_state", - "tracking_errors", - ], - } - release_manifest["logical_sha256"] = canonical_json_sha256( - release_manifest - ) - return _Bundle(files=files, release_manifest=release_manifest) - - -def _assert_exact_output(output_root: Path, expected: dict[PurePosixPath, bytes]) -> None: - if not output_root.is_dir(): - raise Phase1FreezeError(f"tracked Phase 1 release is missing: {output_root}") - observed_paths = { - PurePosixPath(path.relative_to(output_root).as_posix()) - for path in output_root.rglob("*") - if path.is_file() - } - expected_paths = set(expected) - if observed_paths != expected_paths: - raise Phase1FreezeError( - "tracked Phase 1 file inventory differs: " - f"missing={sorted(str(value) for value in expected_paths - observed_paths)}, " - f"extra={sorted(str(value) for value in observed_paths - expected_paths)}" - ) - for relative, content in expected.items(): - path = output_root.joinpath(*relative.parts) - if path.is_symlink() or path.read_bytes() != content: - raise Phase1FreezeError(f"tracked evidence differs at {relative}") - - -def _write_create_or_verify( - output_root: Path, - expected: dict[PurePosixPath, bytes], -) -> None: - output_root.mkdir(parents=True, exist_ok=True) - observed_files = [ - path for path in output_root.rglob("*") if path.is_file() or path.is_symlink() - ] - expected_paths = { - output_root.joinpath(*relative.parts): content - for relative, content in expected.items() - } - extras = sorted( - str(path.relative_to(output_root)) - for path in observed_files - if path not in expected_paths - ) - if extras: - raise Phase1FreezeError( - f"refusing to overwrite a non-canonical Phase 1 directory: {extras}" - ) - for path, content in expected_paths.items(): - if path.exists() or path.is_symlink(): - if path.is_symlink() or path.read_bytes() != content: - raise Phase1FreezeError( - f"refusing to replace existing evidence at {path}" - ) - continue - path.parent.mkdir(parents=True, exist_ok=True) - temporary = path.parent / f".{path.name}.tmp-{os.getpid()}-{uuid.uuid4().hex}" - try: - with temporary.open("xb") as handle: - handle.write(content) - handle.flush() - os.fsync(handle.fileno()) - os.replace(temporary, path) - finally: - temporary.unlink(missing_ok=True) - _assert_exact_output(output_root, expected) - - -def freeze_phase1( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, - output: Path | None = None, - validate_only: bool = False, -) -> dict[str, Any]: - """Recompute the canonical record, then write or validate exact bytes.""" - - output_root = _phase1_output_root(repo_root, output) - source, external = _validate_roots( - repo_root=repo_root, - external_root=external_root, - output_root=output_root, - ) - _require_source_clean_except_output( - repo_root=source, - output_root=output_root, - ) - bundle = _collect_bundle( - repo_root=source, - external_root=external, - expected_source_sha=expected_source_sha, - ) - expected = bundle.all_files - if validate_only: - _assert_exact_output(output_root, expected) - else: - _write_create_or_verify(output_root, expected) - return bundle.release_manifest - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=REPO_ROOT) - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--output", type=Path) - result.add_argument( - "--validate", - action="store_true", - help="validate an existing tracked release without writing", - ) - return result - - -def main() -> None: - args = parser().parse_args() - report = freeze_phase1( - repo_root=args.repo_root, - external_root=args.external_root, - expected_source_sha=args.expected_source_sha, - output=args.output, - validate_only=args.validate, - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/protocol.json b/rl/studies/representation_curriculum_v1/protocol.json deleted file mode 100644 index addde724..00000000 --- a/rl/studies/representation_curriculum_v1/protocol.json +++ /dev/null @@ -1,260 +0,0 @@ -{ - "arms": { - "inkling-e09-20000": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 20000, - "model": "thinkingmachines/Inkling", - "provider": "tinker", - "renderer": "tml_v0", - "thinking": true, - "thinking_effort": 0.9 - }, - "qwen-off-16384": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 16384, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5_disable_thinking", - "thinking": false, - "thinking_effort": null - }, - "qwen-off-4096": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 4096, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5_disable_thinking", - "thinking": false, - "thinking_effort": null - }, - "qwen-on-16384": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 16384, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": true, - "thinking_effort": null - }, - "qwen-on-32768-confirmation": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 32768, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": true, - "thinking_effort": null - }, - "qwen-on-4096": { - "context_tokens": 65536, - "max_image_long_edge": 1440, - "max_output_tokens": 4096, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": true, - "thinking_effort": null - } - }, - "contract_version": "pixcell-blind-phase-a-direct-v1", - "dataset": { - "configuration": "depth", - "development_split": "validation", - "logical_release_sha256": "676c49134d4d044c7d84426cf8eeecf09302e74ca4aa548956f14b7631a4d80b", - "repository": "qpaig-mit/pixcell", - "revision": "v2.0.0" - }, - "future_hypothesis_registry": { - "RC-H10": { - "name": "mixed full-depth SFT", - "status": "launch_forbidden_until_protocol_frozen" - }, - "RC-H11": { - "name": "sequential L0-to-L4 SFT", - "status": "launch_forbidden_until_protocol_frozen" - }, - "RC-H12": { - "name": "L0 SFT followed by L1-to-L4 curriculum RL", - "status": "launch_forbidden_until_protocol_frozen" - }, - "RC-H13": { - "name": "full-depth SFT followed by curriculum RL", - "status": "launch_forbidden_until_protocol_frozen" - }, - "RC-H20": { - "name": "Inkling advanced-component on-policy RL", - "status": "launch_forbidden_until_protocol_frozen" - } - }, - "hypotheses": { - "RC-H00": { - "arm_ids": [ - "qwen-off-4096", - "qwen-off-16384", - "qwen-on-4096", - "qwen-on-16384" - ], - "attempts_per_task": 2, - "name": "Qwen zero-shot renderer by output-budget operating point", - "status": "frozen", - "task_set": "qwen_operating_point", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2 - ], - "task_count": 40 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 1 - } - } - }, - "RC-H00R": { - "arm_ids": [ - "qwen-on-32768-confirmation" - ], - "attempts_per_task": 2, - "name": "Qwen thinking-budget truncation rescue", - "status": "conditional", - "task_set": "qwen_operating_point", - "trigger": { - "metric": "cap_hit_rate", - "operator": ">", - "source_arm": "qwen-on-16384", - "source_hypothesis": "RC-H00", - "threshold": 0.05 - }, - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2 - ], - "task_count": 40 - } - } - }, - "RC-H01": { - "arm_roles": { - "incumbent": "qwen-off-4096", - "selected": "RC-H00" - }, - "attempts_per_task": 4, - "name": "Qwen zero-shot F1-F8 baseline", - "skip_duplicate_selected_arm": true, - "status": "frozen", - "task_set": "f1_f8", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2, - 3, - 4 - ], - "task_count": 8 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 8 - } - } - }, - "RC-H02": { - "arm_ids": [ - "inkling-e09-20000" - ], - "attempts_per_task": 4, - "name": "Inkling zero-shot F1-F8 baseline", - "status": "frozen", - "task_set": "f1_f8", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2, - 3, - 4 - ], - "task_count": 8 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 8 - } - } - } - }, - "launch": { - "confirmation_token": "PIXCELL_PHASE1_BASELINE", - "mutable_output_environment": "PIXCELL_STUDY_ROOT" - }, - "logical_sha256": "1e2869e6c640463e4cf4233a2806dadb5368c4a22a24634c755bb1d3dcc8e349", - "operating_point_selection": { - "conditional_rescue_hypothesis": "RC-H00R", - "metric_tie_band": 0.01, - "model_failures_score_as_iou": 0.0, - "primary_metric": "representation_macro_mean_iou", - "source_hypothesis": "RC-H00", - "tie_breakers": [ - "pure_executable_rate_desc", - "estimated_uncached_cost_usd_asc", - "completion_tokens_median_asc", - "arm_id_asc" - ] - }, - "pricing": { - "as_of": "2026-07-27", - "currency": "USD", - "models": { - "Qwen/Qwen3.6-35B-A3B": { - "prefill_cached": 0.108, - "prefill_uncached": 0.54, - "sample": 1.335 - }, - "thinkingmachines/Inkling": { - "prefill_cached": 0.374, - "prefill_uncached": 1.87, - "sample": 4.68 - } - }, - "source": "https://tinker-docs.thinkingmachines.ai/tinker/models/", - "unit": "per_million_tokens" - }, - "sampling": { - "evaluator_workers": 8, - "require_candidate_isolation": true, - "sampling_concurrency": 8, - "seed_namespace": "pixcell-representation-curriculum-v1", - "temperature": 1.0, - "top_p": 1.0 - }, - "schema_version": "pixcell-representation-curriculum-protocol-v1", - "study_id": "representation-curriculum-v1", - "task_manifest": { - "logical_sha256": "e32a0a1522fba3c90163e5687c65ab5e92fdb77f4f875a73cbfcdaa432b44272", - "path": "task_manifest.json" - }, - "tracking": { - "entity": "aadarwal-massachusetts-institute-of-technology", - "group": "phase1-zero-shot", - "local_record_is_authoritative": true, - "mode": "online", - "project": "pixcell-representation-curriculum-v1", - "provider": "wandb" - } -} diff --git a/rl/studies/representation_curriculum_v1/run_baseline.py b/rl/studies/representation_curriculum_v1/run_baseline.py deleted file mode 100644 index 8485dbb5..00000000 --- a/rl/studies/representation_curriculum_v1/run_baseline.py +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env python3 -"""Run one frozen, resumable Phase 1 baseline wave.""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from pathlib import Path - -from rl.evaluation.runner import run_baseline - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument( - "--hypothesis", - required=True, - choices=("RC-H00", "RC-H00R", "RC-H01", "RC-H02"), - ) - result.add_argument("--wave", required=True, choices=("smoke", "complete")) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--confirm-spend", default="") - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_STUDY_ROOT"]) - if os.environ.get("PIXCELL_STUDY_ROOT") - else None - ), - ) - result.add_argument("--selection", type=Path) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit( - "set PIXCELL_STUDY_ROOT or provide --external-root outside the repository" - ) - report = asyncio.run( - run_baseline( - repo_root=REPO_ROOT, - protocol_file=PROTOCOL_FILE, - hypothesis_id=args.hypothesis, - wave_name=args.wave, - expected_source_sha=args.expected_source_sha, - external_root=args.external_root, - confirmation=args.confirm_spend, - selection_path=args.selection, - ) - ) - print(json.dumps(report["summaries"], indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v1/task_manifest.json b/rl/studies/representation_curriculum_v1/task_manifest.json deleted file mode 100644 index 69ce1026..00000000 --- a/rl/studies/representation_curriculum_v1/task_manifest.json +++ /dev/null @@ -1,631 +0,0 @@ -{ - "logical_sha256": "e32a0a1522fba3c90163e5687c65ab5e92fdb77f4f875a73cbfcdaa432b44272", - "schema_version": "pixcell-evaluation-task-manifest-v1", - "selection": { - "algorithm": "sha256-rank representations per level, then sha256-rank held-out realizations within each selected representation", - "depth_validation_rows_per_level": 8, - "seed": "representation-curriculum-v1-operating-point-20260727" - }, - "task_sets": { - "f1_f8": [ - { - "footprint_um": [ - 40.0, - 1.7 - ], - "image_sha256": "0fbd3bb59c59f81b606be1b5d8d948b941b73ca240d4bb31ba9c9d72e6808d3c", - "image_size_px": [ - 1616, - 656 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f1", - "source": "data/benchmark/final_1", - "source_id": "final_1", - "target_image_sha256": "0fbd3bb59c59f81b606be1b5d8d948b941b73ca240d4bb31ba9c9d72e6808d3c", - "task_id": "F1" - }, - { - "footprint_um": [ - 25.0, - 10.0 - ], - "image_sha256": "f9aac3ccc82c4528647f741d619e246e003e42aab13a0b5687d6a9811068576b", - "image_size_px": [ - 1888, - 2272 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f2", - "source": "data/benchmark/final_2", - "source_id": "final_2", - "target_image_sha256": "f9aac3ccc82c4528647f741d619e246e003e42aab13a0b5687d6a9811068576b", - "task_id": "F2" - }, - { - "footprint_um": [ - 120.0, - 4.0 - ], - "image_sha256": "4962daf2f6b430a3124ba83f0702742b019c24121272f111363c991d847e3c3e", - "image_size_px": [ - 2688, - 1568 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f3", - "source": "data/benchmark/final_3", - "source_id": "final_3", - "target_image_sha256": "4962daf2f6b430a3124ba83f0702742b019c24121272f111363c991d847e3c3e", - "task_id": "F3" - }, - { - "footprint_um": [ - 18.0, - 3.0 - ], - "image_sha256": "a57c3d7acbffa63f691d580bba31ada077ad552c94f360271160912b7e30d701", - "image_size_px": [ - 1584, - 672 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f4", - "source": "data/benchmark/final_4", - "source_id": "final_4", - "target_image_sha256": "a57c3d7acbffa63f691d580bba31ada077ad552c94f360271160912b7e30d701", - "task_id": "F4" - }, - { - "footprint_um": [ - 18.24, - 0.7 - ], - "image_sha256": "6e850e58b0768aa18e828b7b0a4d1b38db4460cef7f3fef0782a58909bf457a3", - "image_size_px": [ - 3712, - 1152 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f5", - "source": "data/benchmark/final_5", - "source_id": "final_5", - "target_image_sha256": "6e850e58b0768aa18e828b7b0a4d1b38db4460cef7f3fef0782a58909bf457a3", - "task_id": "F5" - }, - { - "footprint_um": [ - 4.7, - 4.7 - ], - "image_sha256": "bacae5d13895fd2462fb77fda14d06c56080dd408f362323bc5a7ef2618596a5", - "image_size_px": [ - 1424, - 752 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f6", - "source": "data/benchmark/final_6", - "source_id": "final_6", - "target_image_sha256": "bacae5d13895fd2462fb77fda14d06c56080dd408f362323bc5a7ef2618596a5", - "task_id": "F6" - }, - { - "footprint_um": [ - 3.0, - 5.5 - ], - "image_sha256": "91847bf5797234e985065e7828fbbc306c92c26e281c685175e0efaa4749ce18", - "image_size_px": [ - 1612, - 809 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f7", - "source": "data/benchmark/final_7", - "source_id": "final_7", - "target_image_sha256": "91847bf5797234e985065e7828fbbc306c92c26e281c685175e0efaa4749ce18", - "task_id": "F7" - }, - { - "footprint_um": [ - 120.0, - 4.0 - ], - "image_sha256": "6654508aff5e3fb52f0cecb66a0caa055049e82f6c6ad296386c9e44c341758b", - "image_size_px": [ - 4800, - 3584 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f8", - "source": "data/benchmark/final_8", - "source_id": "final_8", - "target_image_sha256": "6654508aff5e3fb52f0cecb66a0caa055049e82f6c6ad296386c9e44c341758b", - "task_id": "F8" - } - ], - "qwen_operating_point": [ - { - "footprint_um": [ - 92.15, - 104.078 - ], - "image_sha256": "d818a5f411ff7cdac6313bd7f615eccf6fd3e5f52c28a705dba845734c360b19", - "level": "L0", - "representation_id": "g_bba3dc449fc04ddf0f53", - "source": "depth/validation", - "target_image_sha256": "0e0ac78937a8f7989a471d16b341f494531445557b8e77fcf0de3eaeaf45af75", - "task_id": "1cc03a87be7d7c8de25b" - }, - { - "footprint_um": [ - 17.115, - 19.041 - ], - "image_sha256": "0122bf7271a3043a52440aa723be1ac3edde9dea64a7f025ce28f041d2249d97", - "level": "L0", - "representation_id": "g_cdbd6b287f4f7bb8cf76", - "source": "depth/validation", - "target_image_sha256": "7f8ca719a944f6e6aefd7b84261054a8f78c385a323bf19666b9b1148cc8768a", - "task_id": "04ff216228a8cadd5dc7" - }, - { - "footprint_um": [ - 25.76, - 24.2 - ], - "image_sha256": "b495013c61326334569ee226f56f2bdb7baeda631e0750d5924bfbc3b11d8f59", - "level": "L0", - "representation_id": "g_d77719f938c1d0c48b58", - "source": "depth/validation", - "target_image_sha256": "cde3757f49d9f6018ecc58fd972bd001af626e905260203e87822a6809980245", - "task_id": "944288dd187db218cd6f" - }, - { - "footprint_um": [ - 40.8, - 40.8 - ], - "image_sha256": "0656fcd8e862ce86dbe3742ffdcfacad92aa15487a404c8a8dccb8a6a324943a", - "level": "L0", - "representation_id": "g_dc110418f141e49ff242", - "source": "depth/validation", - "target_image_sha256": "cd3caa1b6212225f05a1cbd359a67d38861c1ff61de0f2087e7bea495ca53c74", - "task_id": "b23766e71b76e14c2846" - }, - { - "footprint_um": [ - 31.722, - 31.722 - ], - "image_sha256": "57b821b62b39757b60b33ea0cf88cfbabce5f25d7434448f65672f4df65bd641", - "level": "L0", - "representation_id": "g_a6bad992eb625a2f064c", - "source": "depth/validation", - "target_image_sha256": "191c74a14f53f71cd1badae8b79f73d9b495c703e9ae1f737b25d5c003cdb4b7", - "task_id": "b91220c0502769d1d329" - }, - { - "footprint_um": [ - 23.92, - 23.92 - ], - "image_sha256": "156ccda452c44c1e89e79417caac9a2225a900735c3211a26c32691dcf8a735f", - "level": "L0", - "representation_id": "g_a2b970c7d908676a5762", - "source": "depth/validation", - "target_image_sha256": "92bef39a8335d0696f9bc5ad748971e1a9ea9deb91b9b31937ce1bf0dd4ed9ac", - "task_id": "7faabac2422512d26cfe" - }, - { - "footprint_um": [ - 52.0, - 14.0 - ], - "image_sha256": "48caad2b61c37a96418014d4878b29e9a30a2ab20ae3a6cb3f4328ba63219894", - "level": "L0", - "representation_id": "g_6c5077ce576eaa655f87", - "source": "depth/validation", - "target_image_sha256": "3a0b683bfe1f61840b0cfd05d6e9dd1e67a3f26e5c1cb13fa95187e6f7329086", - "task_id": "70615fe17db58ec7c80c" - }, - { - "footprint_um": [ - 21.358, - 20.313 - ], - "image_sha256": "0b8b425ec323d694285c75b403b5a3e70f915df01d6fe0ac62ee4c9f6724d0c7", - "level": "L0", - "representation_id": "g_6d9c9f9ad2271b0b5b30", - "source": "depth/validation", - "target_image_sha256": "80c3b14b96e895165392a40f70ee57349d8f5e979f7c5a9dba32c8232752ee67", - "task_id": "acd51c0009fad78c40e8" - }, - { - "footprint_um": [ - 53.368, - 24.0 - ], - "image_sha256": "15be832ed53459cbbad68cb684ea1f1671e1575739a224a0858e53fbb514afc1", - "level": "L1", - "representation_id": "l1_13_boolean_or__a2", - "source": "depth/validation", - "target_image_sha256": "9c94e4ae883904806b9a75874c0a48becffb07c84311f513a1e1beb4a3b86556", - "task_id": "dbd325f0cf9e071eff47" - }, - { - "footprint_um": [ - 107.52, - 30.912 - ], - "image_sha256": "e095afae53c57286e30f1b3c08cc98f7b578bb484ff35b1cdc967c81d93fc8dc", - "level": "L1", - "representation_id": "l1_23_conditional_offset__a2", - "source": "depth/validation", - "target_image_sha256": "7d6adf53659d184f5d67cb2f57c092157cf6e0c5b8f508090616e72af59074dd", - "task_id": "59b100b767c6941e3ba1" - }, - { - "footprint_um": [ - 75.08, - 30.9 - ], - "image_sha256": "b65f3ccf157f7f24bd05337585956d5e92d3ed4252b2a57910dd3fd116c63db7", - "level": "L1", - "representation_id": "l1_19_array_2d__b1", - "source": "depth/validation", - "target_image_sha256": "89f285395572f8b25a526a26be680962ad91efa515090095e303ff152fdd6c0d", - "task_id": "8a5fcf0f92f31748fe27" - }, - { - "footprint_um": [ - 25.7, - 17.0 - ], - "image_sha256": "c21a376ee128a9faf911e667a0d97617a68108519f44f1c98f53743d54b3c30e", - "level": "L1", - "representation_id": "l1_14_boolean_and__a2", - "source": "depth/validation", - "target_image_sha256": "49b652a026e1ba64c00004b5d1d0e68542c6c1008cf0aef4337f1158d56e8571", - "task_id": "e2c4efe6d88f3908f029" - }, - { - "footprint_um": [ - 46.0, - 30.52 - ], - "image_sha256": "db17221e75d68418a1b7c5659393abf8eb965a8052e753848ac0b81876715eb0", - "level": "L1", - "representation_id": "l1_06_gap__c1", - "source": "depth/validation", - "target_image_sha256": "373e3bb1a212e1c5526eaff978fdc9af94746844eb5b8bd64ae41565a9bf62c9", - "task_id": "253ee622b93f9055566f" - }, - { - "footprint_um": [ - 58.12, - 23.0 - ], - "image_sha256": "a0f7bcdd1ae28ec1628a3d318b6d90ccbfe3f8400ed3840d052b94c26a3c6de0", - "level": "L1", - "representation_id": "l1_05_edge_align__b1", - "source": "depth/validation", - "target_image_sha256": "d2b563ec515fb2a89ad672f436bdaa9e3f72fc8b00343baaf7d5dccb27fae1c9", - "task_id": "59fb4e9c65ed13124dfd" - }, - { - "footprint_um": [ - 75.08, - 18.0 - ], - "image_sha256": "c24e700d8df20cb6e1e4474ad2e8306174dcd3a008ec10924db4f2c4aee16dba", - "level": "L1", - "representation_id": "l1_17_linear_repetition__a1", - "source": "depth/validation", - "target_image_sha256": "bbab4448234f709048cde8651fdc3a32225b0161b019282698d173523785ba5f", - "task_id": "dd832178b0a6cc1fb4d6" - }, - { - "footprint_um": [ - 37.496, - 37.496 - ], - "image_sha256": "a653e826cb93159c4bd26d1a80de72f3a59ec01f4dbbd5f6cd1f043fbe025161", - "level": "L1", - "representation_id": "l1_19_array_2d__a1", - "source": "depth/validation", - "target_image_sha256": "cb8ef4faff0d72c51ba925f524437ae8cef91bf146370f625b53960bfef40bd4", - "task_id": "b8200b92a7a9c8024dae" - }, - { - "footprint_um": [ - 81.76, - 52.56 - ], - "image_sha256": "61d15c01fcb4a891f5c7c5706ba4943030094cb8309b5a1c7d6a66a11b7caeaf", - "level": "L2", - "representation_id": "l2_06_aligned_cutout__s01", - "source": "depth/validation", - "target_image_sha256": "51217f0f12f4683d77a494b5e5d5f0bc7b4084654aa1526df0f9d2136355d0c6", - "task_id": "c7216e842d2ec8ce4477" - }, - { - "footprint_um": [ - 57.04, - 33.782 - ], - "image_sha256": "2cb5ba75253e23a69f3ad7c0f57b66470ec192af04fab815cd684b98e8c73041", - "level": "L2", - "representation_id": "l2_06_aligned_cutout__s05", - "source": "depth/validation", - "target_image_sha256": "e17cc77ccbdc8aa4358246b3cfb4e62fa70d89d6a6be15c731746c220cd73457", - "task_id": "c236e7fc63568023a544" - }, - { - "footprint_um": [ - 91.8, - 52.1 - ], - "image_sha256": "28bccb907e1a53b1c808fc22c39e4158c60b52b56eb62d1c9f755329086480fc", - "level": "L2", - "representation_id": "l2_18_mirrored_graded_bank__s02", - "source": "depth/validation", - "target_image_sha256": "2a8147e6bc8c27c7df2ee6e26961120cd57f011dabd1b139e28d323b2ebab6ce", - "task_id": "0fb87a48ecec5932bc13" - }, - { - "footprint_um": [ - 61.9, - 26.0 - ], - "image_sha256": "d70e855f0450336dc6771e97b5330687b83c625147f10625fc939280816ed06a", - "level": "L2", - "representation_id": "l2_07_overlap_extraction__s03", - "source": "depth/validation", - "target_image_sha256": "9ed23b09546292c2bf364f40c6f60e20321c03d7343c6e177f2355ba7523acc2", - "task_id": "f5c01ad6addd2d7be3e7" - }, - { - "footprint_um": [ - 88.0, - 16.2 - ], - "image_sha256": "223f14d63435f7e4d9e0539a2ecbdcdcf69bb0fd2004897c21f486ecb6590c95", - "level": "L2", - "representation_id": "l2_09_width_changing_connected_chain__s02", - "source": "depth/validation", - "target_image_sha256": "43eb7f49fd22e76f1c949fd0ce05fadb4ba5297205ed495112ce4ff318deaa76", - "task_id": "5270b249e71b13822b93" - }, - { - "footprint_um": [ - 127.28, - 22.08 - ], - "image_sha256": "c92b0af4bb2544642ff5a8a5c628e598db1c8c48fbea161cd1b07ef381bf959e", - "level": "L2", - "representation_id": "l2_16_repeated_carrier_combination__s06", - "source": "depth/validation", - "target_image_sha256": "cc4bf1dce0050813eb930c088a2b8b969621b6b081d3dc6c99efbe7684de048f", - "task_id": "4a6fed60986a7309666e" - }, - { - "footprint_um": [ - 113.0, - 40.8 - ], - "image_sha256": "264329e7029dc13bfa767807586ed23733cb1a2fe145e767b16f1a4984bae406", - "level": "L2", - "representation_id": "l2_18_mirrored_graded_bank__s04", - "source": "depth/validation", - "target_image_sha256": "848f320fa995186b22f3f91cafecb22217ec26ecad2da234b29c4bd266471fee", - "task_id": "b3f40852049294740eca" - }, - { - "footprint_um": [ - 83.479, - 44.674 - ], - "image_sha256": "cc3afddb083cc6f0d66419312b334608683c7bda4aef6a23880dde90412723af", - "level": "L2", - "representation_id": "l2_14_local_branch_junction__s03", - "source": "depth/validation", - "target_image_sha256": "1af7a2627a79f7784346fa03d3fcfe702b3d7a3ecc30dd7053578920b8442015", - "task_id": "512e7c8aa6067b337a65" - }, - { - "footprint_um": [ - 190.0, - 130.5 - ], - "image_sha256": "fd35c409250ea4e0514b5b17d515493a13d285c0e9e28d4a4e02e649cfc2b3ca", - "level": "L3", - "representation_id": "l3_10_obstacle_constrained_route_bundle_s06", - "source": "depth/validation", - "target_image_sha256": "7bef8bf01de866f611ae68472d17e36629d24ce1573913f8b9346324e2ebbe73", - "task_id": "af1d61009d8481936231" - }, - { - "footprint_um": [ - 188.0, - 26.0 - ], - "image_sha256": "1467b9d19220aa3b18a15932b072a6f5b9fb76e67d9c41390539cbb604bfe77f", - "level": "L3", - "representation_id": "l3_11_transition_repeated_transition_spine_s01", - "source": "depth/validation", - "target_image_sha256": "542ec11c5f456a6710dd3bf281808fd1d758f29662043741fb450c39e5c928c6", - "task_id": "238a4885beef2d00cf35" - }, - { - "footprint_um": [ - 99.84, - 67.5 - ], - "image_sha256": "68d77f06b1df56f121356c2a91cb8744079ea1100c4448ac56b2c0a2d3720fdc", - "level": "L3", - "representation_id": "l3_03_recursive_asymmetric_branch_hierarchy_s03", - "source": "depth/validation", - "target_image_sha256": "27927b9e555d36a0f6b080ca3ac45c4bcfbb994a08223379004cb8e9aab5dfab", - "task_id": "4d9bda68604856b089de" - }, - { - "footprint_um": [ - 146.0, - 58.0 - ], - "image_sha256": "f12e7f35dc49d63e519b15e4cc2565f6f838af37438c06cab0dd0527ae946f0c", - "level": "L3", - "representation_id": "l3_20_carrier_through_aperture_system_s03", - "source": "depth/validation", - "target_image_sha256": "e9b2f4a86224eb3c12a6af6d1f165aaff1667ac6fdb9397e17ae5e839e2447f7", - "task_id": "8ef7187761cdf8cf2da7" - }, - { - "footprint_um": [ - 154.0, - 26.4 - ], - "image_sha256": "06a986100eed46525a5303eb60190e844d5e3b4bcbce504fb2375ed33a7e5cf6", - "level": "L3", - "representation_id": "l3_13_dual_bank_local_defect_s01", - "source": "depth/validation", - "target_image_sha256": "2b03c20a23957abab4ea30ee7851aec610d5ef9bca53e3cf9ddb3323a7b65a76", - "task_id": "6bcd359aead7c84bf19b" - }, - { - "footprint_um": [ - 123.6, - 37.0 - ], - "image_sha256": "7ce7935b1228614c1475f7ef61b316d728a89ccaf567306a959b4ffd6608083e", - "level": "L3", - "representation_id": "l3_02_parallel_reconvergent_bundle_s01", - "source": "depth/validation", - "target_image_sha256": "52a77af689506e1fd8e5a6231b3e680a62e8c40f8072e7c719ad95c7420012fe", - "task_id": "2154d2b81a0d7124167d" - }, - { - "footprint_um": [ - 172.0, - 27.92 - ], - "image_sha256": "270146d58dd251d95b7ce78c2f2215ae60429b3e8f689035cee6e8df324704db", - "level": "L3", - "representation_id": "l3_13_dual_bank_local_defect_s06", - "source": "depth/validation", - "target_image_sha256": "46eab00cd12a04489cceae8d0de7e4efd0e71eb57c80adea74b5ddd8e6d38938", - "task_id": "e5989db4a43fbb73b02e" - }, - { - "footprint_um": [ - 104.548, - 83.329 - ], - "image_sha256": "14973b83d83139e3943727258ed0b5f7abf02890b6542bf7990361e7b27aed48", - "level": "L3", - "representation_id": "l3_21_branch_coupled_repeated_media_s01", - "source": "depth/validation", - "target_image_sha256": "c366eee994a3892e030eb5cc4a8a82abb285310303ec50286a847517c6dd6b69", - "task_id": "9479b1f103cd3cde5f0f" - }, - { - "footprint_um": [ - 58.9, - 17.0 - ], - "image_sha256": "032ecd5749bab626607a5b92cfe133ed4304b5c96cc6fbec69f1a67bec077391", - "level": "L4", - "representation_id": "chirped_apodized_tapered_grating_1x0", - "source": "depth/validation", - "target_image_sha256": "fe872bf270d4345e3972ce4785e5297f5e789ca7a327224c55088f99543e5269", - "task_id": "61fc7dbc77c0869b02aa" - }, - { - "footprint_um": [ - 48.94, - 16.55 - ], - "image_sha256": "25d480b8ff3366d187850ddc2f483934b5bdb0250d6f8d5db13a90d6bf6a73a1", - "level": "L4", - "representation_id": "teardrop_hub_splitter_1x2", - "source": "depth/validation", - "target_image_sha256": "dc1baee573732a52b459ca569eb0e2ecb00296e2dc4d4350a349e2a15de7bdf2", - "task_id": "07086e1f5d6a5cb1356c" - }, - { - "footprint_um": [ - 60.0, - 20.68 - ], - "image_sha256": "7bec50f947c1c39e26cd73a11d72216f1accfda86875e398e9506f3d2b311580", - "level": "L4", - "representation_id": "bragg_loaded_racetrack_resonator_1x1", - "source": "depth/validation", - "target_image_sha256": "4d094d8c25bf97160a8fcc1e9b258d8bd0327ae0b6d6321e3f61321edf5d793a", - "task_id": "889c0fa3ee688a215b1c" - }, - { - "footprint_um": [ - 49.5, - 6.25 - ], - "image_sha256": "03c0f3408a7642d195c0b1afe5173343b6d94e7c79b385a6680da0af32e139f4", - "level": "L4", - "representation_id": "staggered_double_row_swg_1x1", - "source": "depth/validation", - "target_image_sha256": "be8208dbf2946b0a812a8b3e5be5a0ba0a5731c138e03d21e1a8d4a4beb664b8", - "task_id": "f280eed9ee51da5dc90d" - }, - { - "footprint_um": [ - 61.575, - 9.0 - ], - "image_sha256": "bd3e2c09830a88c660ac6f55623a4bfaa47e610efe6d2d17160024735e5574c5", - "level": "L4", - "representation_id": "slot_to_swg_converter_1x1", - "source": "depth/validation", - "target_image_sha256": "a0fd87992e07481b75154886623f7c439cef9aa5143a52ee2a4c7067e5defa31", - "task_id": "6e084f5fe6b625a72441" - }, - { - "footprint_um": [ - 38.68, - 24.688 - ], - "image_sha256": "232b46e3dd70f61a4672f24393daf7b3870faf82d45ef8705c4a8dacdf7ea703", - "level": "L4", - "representation_id": "slotted_ring_bus_1x1", - "source": "depth/validation", - "target_image_sha256": "ddb401718d965b584439ef296ddb2ccb71cb5cb43fc096fb0a7da4b5916c620c", - "task_id": "92d3261039f70d857fa1" - }, - { - "footprint_um": [ - 40.35, - 40.35 - ], - "image_sha256": "228ac96deb178ce2a7dfa0021e7f4d98f651579ebf253256ad4398131a98643d", - "level": "L4", - "representation_id": "crossing_etched_annular", - "source": "depth/validation", - "target_image_sha256": "78b0fbf618c8a8b497d4735deff9f3b6e3330063513d75862132ae3d69e5a43b", - "task_id": "e456dbe43672c1647130" - }, - { - "footprint_um": [ - 77.0, - 14.122 - ], - "image_sha256": "59901eaeeda82e531733691301c6b133df75862309e7c22633559ad4af4180f1", - "level": "L4", - "representation_id": "adiabatic_mode_evolution_coupler_1x2", - "source": "depth/validation", - "target_image_sha256": "ce045224ba076942e9b3ed76f1a2ec23b35c02d970cada95848d19846731a514", - "task_id": "488f1589d0166bbbb86e" - } - ] - } -} diff --git a/rl/studies/representation_curriculum_v1/tests/test_freeze_phase1.py b/rl/studies/representation_curriculum_v1/tests/test_freeze_phase1.py deleted file mode 100644 index f4a031b9..00000000 --- a/rl/studies/representation_curriculum_v1/tests/test_freeze_phase1.py +++ /dev/null @@ -1,607 +0,0 @@ -from __future__ import annotations - -import hashlib -from pathlib import Path -from typing import Any - -import pytest - -from rl.common.contracts import ModelObservation, VerifierReference -from rl.common.output import extract_code -from rl.evaluation.attempt_store import AttemptKey, AttemptStore, RunKey -from rl.evaluation.protocol import jobs_for_wave, resolved_arm_ids -from rl.evaluation.summarize import ( - atomic_write_json, - choose_operating_point, - summarize_arm, -) -from rl.evaluation.tasks import EvaluationTask, canonical_json_sha256 -from rl.studies.representation_curriculum_v1 import freeze_phase1 as freezer - - -SOURCE_SHA = "a" * 40 -PROMPT_SHA = "b" * 64 -SANDBOX = ("sha256:" + "c" * 64, "sha256:" + "c" * 64) -RUNTIME = { - "python": "3.13.14", - "packages": {"tinker": "0.22.7"}, - "tinker_cookbook_commit": "f" * 40, -} - - -def _arm(model: str, max_tokens: int) -> dict[str, Any]: - return { - "provider": "tinker", - "model": model, - "renderer": f"renderer-{model}", - "thinking": False, - "thinking_effort": None, - "max_output_tokens": max_tokens, - "context_tokens": 65536, - "max_image_long_edge": 1440, - } - - -def _protocol() -> dict[str, Any]: - return { - "study_id": "representation-curriculum-v1", - "logical_sha256": "d" * 64, - "dataset": { - "repository": "example/pixcell", - "revision": "test", - "configuration": "depth", - }, - "arms": { - "q0": _arm("qwen", 4096), - "q1": _arm("qwen", 16384), - "qr": _arm("qwen", 32768), - "ink": _arm("inkling", 20000), - }, - "hypotheses": { - "RC-H00": { - "arm_ids": ["q0", "q1"], - "attempts_per_task": 1, - "task_set": "dev", - "waves": { - "complete": { - "attempt_indices": [1], - "task_count": 1, - } - }, - }, - "RC-H00R": { - "arm_ids": ["qr"], - "attempts_per_task": 1, - "task_set": "dev", - "trigger": { - "source_arm": "q1", - "metric": "cap_hit_rate", - "operator": ">", - "threshold": 0.05, - }, - "waves": { - "complete": { - "attempt_indices": [1], - "task_count": 1, - } - }, - }, - "RC-H01": { - "arm_roles": {"incumbent": "q0", "selected": "RC-H00"}, - "attempts_per_task": 1, - "task_set": "bench", - "waves": { - "complete": { - "attempt_indices": [1], - "task_count": 1, - } - }, - }, - "RC-H02": { - "arm_ids": ["ink"], - "attempts_per_task": 1, - "task_set": "bench", - "waves": { - "complete": { - "attempt_indices": [1], - "task_count": 1, - } - }, - }, - }, - "sampling": { - "temperature": 1.0, - "top_p": 1.0, - "seed_namespace": "phase1-freezer-test", - }, - "pricing": { - "models": { - "qwen": { - "prefill_cached": 0.1, - "prefill_uncached": 0.5, - "sample": 1.3, - }, - "inkling": { - "prefill_cached": 0.3, - "prefill_uncached": 1.8, - "sample": 4.6, - }, - } - }, - "tracking": { - "provider": "wandb", - "project": "pixcell-test", - "entity": "test", - "group": "phase1", - "local_record_is_authoritative": True, - }, - "operating_point_selection": { - "primary_metric": "representation_macro_mean_iou", - "metric_tie_band": 0.01, - "tie_breakers": [ - "pure_executable_rate_desc", - "estimated_uncached_cost_usd_asc", - "completion_tokens_median_asc", - "arm_id_asc", - ], - }, - } - - -def _task(task_id: str, level: str) -> EvaluationTask: - image = f"image-{task_id}".encode() - digest = hashlib.sha256(image).hexdigest() - return EvaluationTask( - task_id=task_id, - level=level, - representation_id=f"representation-{task_id}", - observation=ModelObservation( - image_bytes=image, - footprint_um=(10.0, 5.0), - image_sha256=digest, - ), - reference=VerifierReference( - target_image_bytes=image, - footprint_um=(10.0, 5.0), - target_image_sha256=digest, - ), - ) - - -def _manifest(dev: EvaluationTask, bench: EvaluationTask) -> dict[str, Any]: - value = { - "task_sets": { - "dev": [{"task_id": dev.task_id}], - "bench": [{"task_id": bench.task_id}], - } - } - value["logical_sha256"] = canonical_json_sha256(value) - return value - - -def _write_attempt( - *, - store: AttemptStore, - protocol: dict[str, Any], - task: EvaluationTask, - job: Any, - score: float, - cap_hit: bool, -) -> None: - key = AttemptKey( - protocol["study_id"], - job.hypothesis_id, - job.arm_id, - job.task_id, - job.attempt_index, - ) - completion = f"```python\nprint('{job.hypothesis_id}-{job.arm_id}')\n```" - program = extract_code(completion) - raw = f"private reasoning\n{completion}" - rates = protocol["pricing"]["models"][ - protocol["arms"][job.arm_id]["model"] - ] - prompt_tokens = 100 - completion_tokens = ( - protocol["arms"][job.arm_id]["max_output_tokens"] - if cap_hit - else 20 - ) - sample_cost = completion_tokens * rates["sample"] / 1_000_000 - store.write_sampling_receipt( - key, - { - "request": { - "max_tokens": protocol["arms"][job.arm_id][ - "max_output_tokens" - ], - "temperature": 1.0, - "top_p": 1.0, - "seed": job.seed, - }, - "sample": { - "prompt_tokens": prompt_tokens, - "completion_tokens": completion_tokens, - "stop_reason": "length" if cap_hit else "stop", - "cap_hit": cap_hit, - "raw_text": raw, - "completion_text": completion, - "reasoning_text": "private reasoning", - "answer_text": completion, - "reasoning_tokens_exact": 2, - "answer_tokens_exact": completion_tokens - 2, - "reasoning_tokens_estimate": None, - "answer_tokens_estimate": None, - "channel_token_count_basis": "test", - "channel_parse_complete": True, - }, - "sampling_seconds": 0.5, - "raw_completion_sha256": hashlib.sha256(raw.encode()).hexdigest(), - "completion_sha256": hashlib.sha256(completion.encode()).hexdigest(), - "cost_estimate_usd": { - "cached_prefill": ( - prompt_tokens * rates["prefill_cached"] / 1_000_000 - + sample_cost - ), - "uncached_prefill": ( - prompt_tokens * rates["prefill_uncached"] / 1_000_000 - + sample_cost - ), - }, - }, - ) - store.write_evaluation( - key, - { - "status": "ok", - "attribution": "model", - "pure_executable": True, - "measurement_available": True, - "measured_iou": score, - "measured_dice": min(1.0, score + 0.1), - "aggregation_iou": score, - "program": program, - "program_sha256": hashlib.sha256(program.encode()).hexdigest(), - "reference_sha256": task.reference.target_image_sha256, - "violations": [], - "error": None, - "retryable": False, - "evaluation_seconds": 0.25, - "diagnostics": {}, - "render_sha256": "e" * 64, - }, - ) - - -def _write_run( - *, - store: AttemptStore, - protocol: dict[str, Any], - hypothesis_id: str, - arm_id: str, - tasks: list[EvaluationTask], - jobs: list[Any], - task_set_sha: str, - score: float, - cap_hit: bool = False, -) -> dict[str, Any]: - run_key = RunKey(protocol["study_id"], hypothesis_id, arm_id) - store.create_or_verify_run_manifest( - run_key, - freezer._expected_run_manifest( - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - source_git_sha=SOURCE_SHA, - task_set_sha256=task_set_sha, - prompt_sha256=PROMPT_SHA, - sandbox_image_ref=SANDBOX[0], - sandbox_image_id=SANDBOX[1], - runtime=RUNTIME, - ), - ) - task_by_id = {task.task_id: task for task in tasks} - for job in jobs: - if job.arm_id == arm_id: - _write_attempt( - store=store, - protocol=protocol, - task=task_by_id[job.task_id], - job=job, - score=score, - cap_hit=cap_hit, - ) - summary = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id=hypothesis_id, - arm_id=arm_id, - jobs=jobs, - tasks=tasks, - ) - path = ( - store.external_root.joinpath(*store.run_relative_path(run_key).parts) - / "summary.json" - ) - atomic_write_json(path, summary) - return summary - - -def _prepare( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - *, - trigger_rescue: bool = False, - include_rescue: bool = True, -) -> tuple[Path, Path]: - repo = tmp_path / "repo" - repo.mkdir() - external = tmp_path / "external" - protocol = _protocol() - dev = _task("D1", "L0") - bench = _task("F1", "BENCHMARK") - manifest = _manifest(dev, bench) - - monkeypatch.setattr(freezer, "_git_head", lambda _root: SOURCE_SHA) - monkeypatch.setattr(freezer, "_changed_paths", lambda _root: set()) - monkeypatch.setattr( - freezer, - "validate_runtime_stack", - lambda _root: RUNTIME, - ) - monkeypatch.setattr( - freezer, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - freezer, - "load_task_manifest", - lambda _path, *, repo_root: manifest, - ) - monkeypatch.setattr( - freezer, - "load_task_set", - lambda *, repo_root, manifest, task_set: ( - [dev] if task_set == "dev" else [bench] - ), - ) - monkeypatch.setattr( - freezer, - "_prompt_binding", - lambda: ({"contract_version": "test"}, PROMPT_SHA), - ) - - store = AttemptStore(repo_root=repo, external_root=external) - dev_sha = canonical_json_sha256(manifest["task_sets"]["dev"]) - bench_sha = canonical_json_sha256(manifest["task_sets"]["bench"]) - h00_jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H00", - wave_name="complete", - task_ids=[dev.task_id], - ) - h00 = { - "q0": _write_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H00", - arm_id="q0", - tasks=[dev], - jobs=h00_jobs, - task_set_sha=dev_sha, - score=0.2, - ), - "q1": _write_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H00", - arm_id="q1", - tasks=[dev], - jobs=h00_jobs, - task_set_sha=dev_sha, - score=0.4, - cap_hit=trigger_rescue, - ), - } - preliminary = choose_operating_point( - protocol=protocol, - source_git_sha=SOURCE_SHA, - source_summaries=h00, - ) - selection_root = ( - external - / "studies" - / protocol["study_id"] - / "selections" - ) - atomic_write_json( - selection_root / "operating_point.preliminary.json", - preliminary, - ) - final = preliminary - if trigger_rescue and include_rescue: - rescue_jobs = jobs_for_wave( - protocol, - hypothesis_id="RC-H00R", - wave_name="complete", - task_ids=[dev.task_id], - selection=preliminary, - ) - rescue_summary = _write_run( - store=store, - protocol=protocol, - hypothesis_id="RC-H00R", - arm_id="qr", - tasks=[dev], - jobs=rescue_jobs, - task_set_sha=dev_sha, - score=0.5, - ) - final = choose_operating_point( - protocol=protocol, - source_git_sha=SOURCE_SHA, - source_summaries=h00, - rescue_summary=rescue_summary, - ) - atomic_write_json(selection_root / "operating_point.json", final) - if trigger_rescue and not include_rescue: - return repo, external - - for hypothesis_id in ("RC-H01", "RC-H02"): - jobs = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name="complete", - task_ids=[bench.task_id], - selection=final if hypothesis_id == "RC-H01" else None, - ) - for arm_id in resolved_arm_ids( - protocol, - hypothesis_id, - selection=final if hypothesis_id == "RC-H01" else None, - ): - _write_run( - store=store, - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - tasks=[bench], - jobs=jobs, - task_set_sha=bench_sha, - score=0.3, - ) - return repo, external - - -def test_freeze_is_deterministic_idempotent_and_excludes_raw_text( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - report = freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - output = repo / freezer.PHASE1_RELATIVE_ROOT - first = { - path.relative_to(output): path.read_bytes() - for path in output.rglob("*") - if path.is_file() - } - assert report["attempt_count"] == 5 - assert report["selected_qwen_arm"] == "q1" - assert b"private reasoning" not in b"".join(first.values()) - assert not any(b"```python" in value for value in first.values()) - - repeated = freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - assert repeated == report - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - validate_only=True, - ) - assert first == { - path.relative_to(output): path.read_bytes() - for path in output.rglob("*") - if path.is_file() - } - - -def test_validate_rejects_tampered_tracked_evidence( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - attempts = repo / freezer.PHASE1_RELATIVE_ROOT / "attempts.jsonl" - attempts.write_bytes(attempts.read_bytes() + b"{}\n") - with pytest.raises(freezer.Phase1FreezeError, match="differs"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - validate_only=True, - ) - - -def test_triggered_rescue_is_mandatory( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare( - tmp_path, - monkeypatch, - trigger_rescue=True, - include_rescue=False, - ) - with pytest.raises(freezer.Phase1FreezeError, match="missing run manifest"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_triggered_rescue_is_included_in_the_canonical_release( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare( - tmp_path, - monkeypatch, - trigger_rescue=True, - include_rescue=True, - ) - report = freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - assert report["selected_qwen_arm"] == "qr" - assert report["attempt_count"] == 6 - assert any( - run["hypothesis_id"] == "RC-H00R" for run in report["runs"] - ) - - -def test_output_location_is_fixed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - with pytest.raises(freezer.Phase1FreezeError, match="exactly"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - output=repo / "somewhere-else", - ) - - -def test_source_drift_outside_the_release_directory_is_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - monkeypatch.setattr( - freezer, - "_changed_paths", - lambda _root: {repo / "rl" / "unexpected.py"}, - ) - with pytest.raises(freezer.Phase1FreezeError, match="source worktree differs"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) diff --git a/rl/studies/representation_curriculum_v2/README.md b/rl/studies/representation_curriculum_v2/README.md deleted file mode 100644 index 302ad422..00000000 --- a/rl/studies/representation_curriculum_v2/README.md +++ /dev/null @@ -1,114 +0,0 @@ -# Direct-policy zero-shot baselines v2 - -This study measures the untrained Qwen and Inkling policies on the fixed F1–F8 -benchmark under the corrected `pixcell-direct-reconstruction-v2` contract. It -does not select an operating point and does not train a model. - -## What the policy receives - -Each attempt is one user turn, ordered as: - -1. the benchmark's `device_bw.png`, decoded as RGB and downscaled only when - needed with aspect ratio preserved and a 1920-pixel longest edge; -2. the image semantics: black is material on layer `(1, 0)`, white is - background; -3. the benchmark's physical `[x-length, y-width]` footprint; -4. the exact extended primitive catalogue from - `src/michaelangelo/reference/gds_factory_function_catalogue_extended.md`; -5. the direct programming requirements in - `rl/common/assets/phase_a_direct_v2.md`. - -There is no system message. For F1–F8, `device_bw.png` is both the visible -input and the silhouette against which the verifier later measures the -candidate; there is no separate hidden reference raster. The policy receives -the image once, in the user message. It never receives the verifier's -calibration, reference code, IoU, or answer-derived metadata. - -The image and footprint are the normal Phase-A inputs defined by -`src/homi/briefs/phase_a_recreation.md`. The primitive catalogue is the -repository's permitted programming vocabulary. The direct programming -requirements retain the normal Phase-A source, parameterization, and output -rules while making the one-shot transport boundary explicit: the answer must -be a complete program that writes `device.gds`. -`rl/common/preprocess.py` is the authority for the deterministic image -transformation. - -## Frozen arms - -| Hypothesis | Policy | Reasoning | Output ceiling | Attempts | -|---|---|---|---:|---:| -| `RC-H05` | `Qwen/Qwen3.6-35B-A3B` | `qwen3_5`, thinking on | 60,000 | 4 per figure | -| `RC-H06` | `thinkingmachines/Inkling` | `tml_v0`, effort 0.9 | 60,000 | 4 per figure | - -Both arms use a 65,536-token context, a 1920-pixel maximum image edge, -temperature 1.0, top-p 1.0, sampling concurrency 8, and the strict isolated -evaluator. A local renderer preflight measures every complete prompt and -rejects the launch if `prompt tokens + 60,000` exceeds the context. -The frozen 1920-pixel edge leaves positive context headroom for every Qwen and -Inkling F1–F8 input at the 60,000-token ceiling. - -`smoke` means attempt index 1 for each of F1–F8: eight independent model -requests per arm. Those eight receipts are already the first quarter of the -declared best-at-4 baseline. After they execute and evaluate correctly, -`complete` resumes the same run and adds attempt indices 2–4; it does not -discard or repeat attempt 1. - -## Before launch - -The launch path is deliberately fail-closed. It requires a clean, exact source -commit; sealed protocol and task-manifest digests; the pinned runtime; the -immutable candidate-execution boundary; exact renderer context accounting; -valid references; credentials; an external attempt ledger; and the explicit -`PIXCELL_BASELINE_V2` spend confirmation. No Tinker service client exists until -those checks pass. - -Inspect exact local renderer token accounting without making a remote request: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_curriculum_v2.audit_context -``` - -After committing the launch source, run the complete zero-spend audit with the -immutable evaluator image already selected: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_curriculum_v2.audit_phase0 \ - --expected-source-sha "$(git rev-parse HEAD)" -``` - -That audit needs no model or tracking credentials. It works offline, renders -all 16 arm/figure contexts, validates all eight references, executes one -fixed primitive-only probe against F1 inside the required sandbox, and -round-trips the run-manifest, sampling-receipt, and evaluation-record schemas. -A paid smoke launch is not ready unless this sealed audit passes. - -Paid launch commands are intentionally not presented as casual copy-paste -examples. Use `run_baseline.py --help`, review the sealed commit, run `smoke`, -inspect all eight records, and only then resume `complete`. - -## Freeze completed evidence - -After both 32-attempt arms are complete, freeze the external ledger into the -small tracked record: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_curriculum_v2.freeze_phase1 \ - --external-root "$PIXCELL_STUDY_ROOT" \ - --expected-source-sha "$(git rev-parse HEAD)" -``` - -The command reproduces both summaries from immutable attempt receipts, -re-renders every prompt offline, and re-executes all 64 extracted programs in -the exact launch-bound sandbox before accepting their scores. It writes -`release_manifest.json`, `checksums.sha256`, `summaries.json`, -`attempts.jsonl`, and `programs.jsonl` under -`data/training/representation-curriculum-v2/phase1`. `programs.jsonl` contains -only programs that passed the source, execution, and geometry measurement -path, making every nonzero reported geometry independently re-evaluable. -Failed-output fallback text is not published. Raw completions, separate -reasoning and answer channels, mutable W&B state, and credentials remain -outside Git. Add `--validate` to prove that an existing tracked record matches -the authoritative ledger byte for byte. diff --git a/rl/studies/representation_curriculum_v2/__init__.py b/rl/studies/representation_curriculum_v2/__init__.py deleted file mode 100644 index 0cdf1d6b..00000000 --- a/rl/studies/representation_curriculum_v2/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Canonical direct-policy zero-shot baselines under the v2 prompt contract.""" diff --git a/rl/studies/representation_curriculum_v2/audit_context.py b/rl/studies/representation_curriculum_v2/audit_context.py deleted file mode 100644 index 17690a28..00000000 --- a/rl/studies/representation_curriculum_v2/audit_context.py +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env python3 -"""Render every v2 baseline input and prove that it fits without remote access.""" - -from __future__ import annotations - -import json -import os -from contextlib import contextmanager -from pathlib import Path -from typing import Any, Iterator - -from rl.evaluation.protocol import arm_spec, load_protocol -from rl.evaluation.runner import _messages, _sampler_target -from rl.evaluation.samplers import TinkerSampler -from rl.evaluation.tasks import load_task_manifest, load_task_set - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") -TASK_MANIFEST_FILE = Path(__file__).with_name("task_manifest.json") - - -@contextmanager -def _offline_model_assets() -> Iterator[None]: - """Fail closed rather than fetching a missing renderer asset.""" - - names = ("HF_HUB_OFFLINE", "TRANSFORMERS_OFFLINE") - previous = {name: os.environ.get(name) for name in names} - os.environ.update({name: "1" for name in names}) - try: - yield - finally: - for name, value in previous.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value - - -def audit_context() -> dict[str, Any]: - """Return exact renderer lengths; raise before any remote client can exist.""" - - protocol = load_protocol(REPO_ROOT, protocol_file=PROTOCOL_FILE) - manifest = load_task_manifest(TASK_MANIFEST_FILE, repo_root=REPO_ROOT) - tasks = load_task_set( - repo_root=REPO_ROOT, - manifest=manifest, - task_set="f1_f8", - ) - arm_ids = [ - arm_id - for hypothesis in protocol["hypotheses"].values() - for arm_id in hypothesis["arm_ids"] - ] - rows: list[dict[str, Any]] = [] - summaries: dict[str, dict[str, int]] = {} - - def forbidden_remote_client() -> Any: - raise AssertionError("context audit must not create a remote client") - - with _offline_model_assets(): - for arm_id in arm_ids: - arm = arm_spec(protocol, arm_id) - sampler = TinkerSampler( - _sampler_target(arm), - service_client_factory=forbidden_remote_client, - ) - arm_rows: list[dict[str, Any]] = [] - for task in tasks: - prompt = sampler.build_prompt(_messages(task, arm)) - prompt_tokens = int(prompt.length) - headroom = ( - arm.context_tokens - arm.max_output_tokens - prompt_tokens - ) - if headroom < 0: - raise ValueError( - f"{arm_id}/{task.task_id} exceeds its context window: " - f"{prompt_tokens}+{arm.max_output_tokens}>" - f"{arm.context_tokens}" - ) - arm_rows.append( - { - "arm_id": arm_id, - "task_id": task.task_id, - "prompt_tokens": prompt_tokens, - "max_output_tokens": arm.max_output_tokens, - "context_tokens": arm.context_tokens, - "headroom_tokens": headroom, - } - ) - rows.extend(arm_rows) - summaries[arm_id] = { - "task_count": len(arm_rows), - "max_prompt_tokens": max( - row["prompt_tokens"] for row in arm_rows - ), - "min_headroom_tokens": min( - row["headroom_tokens"] for row in arm_rows - ), - } - return { - "schema_version": "pixcell-direct-context-audit-v1", - "study_id": protocol["study_id"], - "protocol_sha256": protocol["logical_sha256"], - "task_manifest_sha256": manifest["logical_sha256"], - "remote_client_created": False, - "summaries": summaries, - "rows": rows, - } - - -def main() -> None: - print(json.dumps(audit_context(), indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v2/audit_phase0.py b/rl/studies/representation_curriculum_v2/audit_phase0.py deleted file mode 100644 index 1e868389..00000000 --- a/rl/studies/representation_curriculum_v2/audit_phase0.py +++ /dev/null @@ -1,492 +0,0 @@ -#!/usr/bin/env python3 -"""Prove the v2 baseline launch boundary locally without making a model request.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import os -import subprocess -import tempfile -from contextlib import contextmanager -from pathlib import Path -from typing import Any, Iterator - -from rl.common.evaluator import Attribution, EvaluationStatus, PixCellEvaluator -from rl.common.isolation import ExecutionBoundary, require_execution_boundary -from rl.common.output import extract_code -from rl.common.preprocess import IMAGE_PREPROCESS_VERSION -from rl.common.prompt import ( - CONTRACT_VERSION, - build_prompt_text, - prompt_asset_hashes, -) -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.attempt_store import AttemptKey, AttemptStore, RunKey -from rl.evaluation.protocol import arm_spec, load_protocol -from rl.evaluation.runner import _messages, _preflight_references, _sampler_target -from rl.evaluation.samplers import TinkerSampler -from rl.evaluation.tasks import ( - BENCHMARK_TASK_SET, - canonical_json_sha256, - load_task_manifest, - load_task_set, -) - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") -TASK_MANIFEST_FILE = Path(__file__).with_name("task_manifest.json") -AUDIT_SCHEMA = "pixcell-representation-curriculum-phase0-audit-v2" -EXPECTED_STUDY_ID = "representation-curriculum-v2" -EXPECTED_ARMS = ("inkling-e09-60000", "qwen-on-60000") -EXPECTED_IMAGE_LONG_EDGE = 1920 -_AUDIT_PROBE_PROGRAM = """\ -import gdsfactory as gf - -length = 40.0 -width = 1.7 -layer = (1, 0) - - -@gf.cell -def device(): - component = gf.Component() - component << gf.components.rectangle( - size=(length, width), - layer=layer, - centered=True, - ) - return component - - -if __name__ == "__main__": - device().write_gds("device.gds") -""" -_FORBIDDEN_MODEL_TEXT = ( - "calibration", - "display note", - "ground truth", - "iou", - "leakage_group_id", - "magnification", - "px/um", - "realization_slot", - "reference code", - "representation_id", - "scale_px_per_um", - "split_role", - "target raster", - "target_image", -) - - -def _git_state(repo_root: Path) -> tuple[str, bool]: - head = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo_root, - text=True, - ).strip() - status = subprocess.check_output( - ["git", "status", "--porcelain", "--untracked-files=all"], - cwd=repo_root, - text=True, - ).strip() - return head, not bool(status) - - -def _assert_prompt_is_model_safe(text: str, *, case_id: str) -> None: - normalized = text.casefold() - leaked = [ - fragment - for fragment in _FORBIDDEN_MODEL_TEXT - if fragment.casefold() in normalized - ] - if leaked: - raise ValueError( - f"{case_id} leaked verifier/answer-only prompt fields: {leaked}" - ) - - -@contextmanager -def _offline_model_assets() -> Iterator[None]: - """Forbid tokenizer/image-processor cache misses from reaching the network.""" - - names = ("HF_HUB_OFFLINE", "TRANSFORMERS_OFFLINE") - previous = {name: os.environ.get(name) for name in names} - os.environ.update({name: "1" for name in names}) - try: - yield - finally: - for name, value in previous.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value - - -def _renderer_audit( - *, - protocol: dict[str, Any], - tasks: list[Any], -) -> dict[str, Any]: - remote_client_calls = 0 - - def forbidden_remote_client() -> Any: - nonlocal remote_client_calls - remote_client_calls += 1 - raise AssertionError("Phase 0 audit attempted to create a Tinker client") - - rows: list[dict[str, Any]] = [] - summaries: dict[str, dict[str, int]] = {} - with _offline_model_assets(): - for arm_id in EXPECTED_ARMS: - arm = arm_spec(protocol, arm_id) - if arm.max_image_long_edge != EXPECTED_IMAGE_LONG_EDGE: - raise ValueError( - f"{arm_id} must use the frozen " - f"{EXPECTED_IMAGE_LONG_EDGE}px image policy" - ) - sampler = TinkerSampler( - _sampler_target(arm), - service_client_factory=forbidden_remote_client, - ) - arm_rows: list[dict[str, Any]] = [] - for task in tasks: - text = build_prompt_text(task.observation) - _assert_prompt_is_model_safe( - text, - case_id=f"{arm_id}/{task.task_id}", - ) - prompt = sampler.build_prompt(_messages(task, arm)) - prompt_tokens = int(prompt.length) - headroom = ( - arm.context_tokens - - arm.max_output_tokens - - prompt_tokens - ) - if headroom < 0: - raise ValueError( - f"{arm_id}/{task.task_id} exceeds context: " - f"{prompt_tokens}+{arm.max_output_tokens}>" - f"{arm.context_tokens}" - ) - arm_rows.append( - { - "arm_id": arm_id, - "task_id": task.task_id, - "prompt_tokens": prompt_tokens, - "headroom_tokens": headroom, - } - ) - rows.extend(arm_rows) - summaries[arm_id] = { - "prompt_cases": len(arm_rows), - "prompt_tokens_min": min( - row["prompt_tokens"] for row in arm_rows - ), - "prompt_tokens_max": max( - row["prompt_tokens"] for row in arm_rows - ), - "minimum_headroom_tokens": min( - row["headroom_tokens"] for row in arm_rows - ), - } - - expected_cases = len(EXPECTED_ARMS) * len(tasks) - if len(rows) != expected_cases or expected_cases != 16: - raise ValueError( - f"expected exactly 16 arm/task prompt cases, observed {len(rows)}" - ) - if remote_client_calls: - raise AssertionError("renderer audit created a remote model client") - return { - "model_client_created": False, - "prompt_cases": len(rows), - "arms": summaries, - "rows": rows, - } - - -def _run_manifest( - *, - protocol: dict[str, Any], - arm_id: str, - source_git_sha: str, - task_set_sha256: str, - prompt_sha256: str, - runtime: dict[str, Any], - boundary: ExecutionBoundary, -) -> dict[str, Any]: - return { - "source_git_sha": source_git_sha, - "protocol_sha256": protocol["logical_sha256"], - "task_set_sha256": task_set_sha256, - "prompt_sha256": prompt_sha256, - "sandbox_image_ref": boundary.image_ref, - "sandbox_image_id": boundary.image_id, - "runtime": runtime, - "model_binding": { - "arm_id": arm_id, - **protocol["arms"][arm_id], - "temperature": protocol["sampling"]["temperature"], - "top_p": protocol["sampling"]["top_p"], - }, - } - - -def _schema_smoke( - *, - repo_root: Path, - protocol: dict[str, Any], - source_git_sha: str, - task_set_sha256: str, - prompt_sha256: str, - runtime: dict[str, Any], - boundary: ExecutionBoundary, -) -> dict[str, dict[str, str]]: - hypothesis_by_arm = { - hypothesis["arm_ids"][0]: hypothesis_id - for hypothesis_id, hypothesis in protocol["hypotheses"].items() - } - records: dict[str, dict[str, str]] = {} - with tempfile.TemporaryDirectory(prefix="pixcell-phase0-v2-") as temporary: - store = AttemptStore( - repo_root=repo_root, - external_root=Path(temporary) / "records", - ) - for arm_id in EXPECTED_ARMS: - hypothesis_id = hypothesis_by_arm[arm_id] - run_key = RunKey(protocol["study_id"], hypothesis_id, arm_id) - key = AttemptKey( - protocol["study_id"], - hypothesis_id, - arm_id, - "F1", - 1, - ) - completion = "```python\nprint('phase0 schema audit')\n```" - with store.acquire_run_lock(run_key): - run = store.create_or_verify_run_manifest( - run_key, - _run_manifest( - protocol=protocol, - arm_id=arm_id, - source_git_sha=source_git_sha, - task_set_sha256=task_set_sha256, - prompt_sha256=prompt_sha256, - runtime=runtime, - boundary=boundary, - ), - ) - receipt = store.write_sampling_receipt( - key, - { - "request": { - "max_tokens": protocol["arms"][arm_id][ - "max_output_tokens" - ], - "temperature": protocol["sampling"]["temperature"], - "top_p": protocol["sampling"]["top_p"], - "seed": 1, - }, - "sample": { - "prompt_tokens": 100, - "completion_tokens": 10, - "stop_reason": "stop", - "cap_hit": False, - "raw_text": completion, - "completion_text": completion, - "reasoning_text": "", - "answer_text": completion, - "reasoning_tokens_exact": 0, - "answer_tokens_exact": 10, - "reasoning_tokens_estimate": None, - "answer_tokens_estimate": None, - "channel_token_count_basis": "phase0_schema_audit", - "channel_parse_complete": True, - }, - "sampling_seconds": 0.0, - "raw_completion_sha256": hashlib.sha256( - completion.encode("utf-8") - ).hexdigest(), - "completion_sha256": hashlib.sha256( - completion.encode("utf-8") - ).hexdigest(), - "run_manifest_record_sha256": run.record_sha256, - "cost_estimate_usd": { - "cached_prefill": 0.0, - "uncached_prefill": 0.0, - }, - }, - ) - evaluation = store.write_evaluation( - key, - { - "status": EvaluationStatus.SYNTAX_ERROR.value, - "attribution": Attribution.MODEL.value, - "pure_executable": False, - "measurement_available": False, - "measured_iou": None, - "measured_dice": None, - "aggregation_iou": 0.0, - "program": "print('phase0 schema audit')", - "program_sha256": hashlib.sha256( - b"print('phase0 schema audit')" - ).hexdigest(), - "reference_sha256": "3" * 64, - "violations": [], - "error": "schema audit", - "retryable": False, - "evaluation_seconds": 0.0, - "diagnostics": {}, - }, - ) - if store.load_run_manifest(run_key) != run: - raise AssertionError("run manifest did not round-trip") - if store.load_sampling_receipt(key) != receipt: - raise AssertionError("sampling receipt did not round-trip") - if store.load_evaluation(key) != evaluation: - raise AssertionError("evaluation record did not round-trip") - records[arm_id] = { - "run_manifest_record_sha256": run.record_sha256, - "sampling_record_sha256": receipt.record_sha256, - "evaluation_record_sha256": evaluation.record_sha256, - } - return records - - -def run_audit( - *, - repo_root: Path, - expected_source_sha: str | None = None, -) -> dict[str, Any]: - root = repo_root.expanduser().resolve(strict=True) - head, clean = _git_state(root) - if expected_source_sha is not None: - if head != expected_source_sha: - raise ValueError( - f"source SHA mismatch: {head} != {expected_source_sha}" - ) - if not clean: - raise ValueError("source worktree must be clean for the sealed audit") - - protocol_file = root / PROTOCOL_FILE.relative_to(REPO_ROOT) - manifest_file = root / TASK_MANIFEST_FILE.relative_to(REPO_ROOT) - protocol = load_protocol(root, protocol_file=protocol_file) - if protocol["study_id"] != EXPECTED_STUDY_ID: - raise ValueError("Phase 0 loaded the wrong study") - if tuple(sorted(protocol["arms"])) != EXPECTED_ARMS: - raise ValueError("Phase 0 loaded an unexpected baseline arm set") - manifest = load_task_manifest(manifest_file, repo_root=root) - tasks = load_task_set( - repo_root=root, - manifest=manifest, - task_set=BENCHMARK_TASK_SET, - ) - if [task.task_id for task in tasks] != [ - f"F{index}" for index in range(1, 9) - ]: - raise ValueError("Phase 0 did not resolve exact F1-F8 order") - - runtime = validate_runtime_stack(root) - renderer = _renderer_audit(protocol=protocol, tasks=tasks) - boundary = require_execution_boundary() - with PixCellEvaluator( - max_workers=1, - evaluator_retries=1, - require_isolation=True, - ) as evaluator: - if evaluator.execution_boundary is None: - raise AssertionError("accepted-program audit is not isolated") - if evaluator.execution_boundary.image_id != boundary.image_id: - raise ValueError("sandbox identity changed during Phase 0") - _preflight_references(tasks, evaluator) - probe = _AUDIT_PROBE_PROGRAM.strip() - wrapped = f"```python\n{probe}\n```" - if extract_code(wrapped) != probe: - raise ValueError("output extractor changed the fixed audit probe") - result = evaluator.evaluate(tasks[0].reference, wrapped) - if ( - result.status is not EvaluationStatus.OK - or result.attribution is not Attribution.MODEL - or result.iou is None - ): - raise ValueError( - "isolated evaluator rejected the fixed primitive-only probe: " - f"{result.status.value}/{result.attribution.value}: {result.error}" - ) - - prompt_binding = { - "contract_version": CONTRACT_VERSION, - "preprocess_version": IMAGE_PREPROCESS_VERSION, - **prompt_asset_hashes(), - } - prompt_sha256 = canonical_json_sha256(prompt_binding) - task_set_sha256 = canonical_json_sha256( - manifest["task_sets"][BENCHMARK_TASK_SET] - ) - return { - "schema_version": AUDIT_SCHEMA, - "study_id": protocol["study_id"], - "source_git_sha": head, - "source_worktree_clean": clean, - "protocol_sha256": protocol["logical_sha256"], - "task_manifest_sha256": manifest["logical_sha256"], - "task_set_sha256": task_set_sha256, - "prompt": prompt_binding, - "prompt_sha256": prompt_sha256, - "renderer": renderer, - "reference_count": len(tasks), - "runtime": runtime, - "sandbox": { - "image_ref": boundary.image_ref, - "image_id": boundary.image_id, - }, - "evaluator_probe": { - "task_id": tasks[0].task_id, - "program_sha256": hashlib.sha256( - probe.encode("utf-8") - ).hexdigest(), - "status": result.status.value, - "iou": result.iou, - }, - "record_schema": _schema_smoke( - repo_root=root, - protocol=protocol, - source_git_sha=head, - task_set_sha256=task_set_sha256, - prompt_sha256=prompt_sha256, - runtime=runtime, - boundary=boundary, - ), - "model_client_created": False, - "network_required": False, - "credentials_required": False, - } - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=REPO_ROOT) - result.add_argument("--expected-source-sha") - return result - - -def main() -> None: - args = parser().parse_args() - print( - json.dumps( - run_audit( - repo_root=args.repo_root, - expected_source_sha=args.expected_source_sha, - ), - indent=2, - sort_keys=True, - ) - ) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v2/freeze_phase1.py b/rl/studies/representation_curriculum_v2/freeze_phase1.py deleted file mode 100644 index f7f98fe4..00000000 --- a/rl/studies/representation_curriculum_v2/freeze_phase1.py +++ /dev/null @@ -1,1414 +0,0 @@ -#!/usr/bin/env python3 -"""Freeze or validate the compact Phase 1 evidence for the v2 baselines.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import os -import re -import subprocess -import uuid -from contextlib import contextmanager -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any, Iterator - -from rl.common.evaluator import Attribution, EvaluationResult, PixCellEvaluator -from rl.common.output import extract_code -from rl.common.preprocess import IMAGE_PREPROCESS_VERSION -from rl.common.prompt import CONTRACT_VERSION, prompt_asset_hashes -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.attempt_store import ( - AttemptKey, - AttemptStore, - RunKey, - StoredRecord, -) -from rl.evaluation.protocol import ( - JobSpec, - arm_spec, - jobs_for_wave, - load_protocol, -) -from rl.evaluation.runner import _messages, _sampler_target -from rl.evaluation.samplers import TinkerSampler -from rl.evaluation.summarize import summarize_arm -from rl.evaluation.tasks import ( - EvaluationTask, - canonical_json_sha256, - load_task_manifest, - load_task_set, -) -from rl.evaluation.tracking import deterministic_wandb_run_id - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") -TASK_MANIFEST_FILE = Path(__file__).with_name("task_manifest.json") -PHASE1_RELATIVE_ROOT = Path( - "data/training/representation-curriculum-v2/phase1" -) -RELEASE_SCHEMA = "pixcell-representation-curriculum-phase1-release-v2" -SUMMARIES_SCHEMA = "pixcell-representation-curriculum-phase1-summaries-v2" -ATTEMPT_INDEX_SCHEMA = "pixcell-representation-curriculum-attempt-v2" -PROGRAM_INDEX_SCHEMA = "pixcell-representation-curriculum-program-v2" -EXPECTED_CONTRACT = "pixcell-direct-reconstruction-v2" -EXPECTED_STUDY = "representation-curriculum-v2" -EXPECTED_HYPOTHESES = { - "RC-H05": "qwen-on-60000", - "RC-H06": "inkling-e09-60000", -} -EXPECTED_TASK_IDS = tuple(f"F{index}" for index in range(1, 9)) -EXPECTED_ATTEMPTS = tuple(range(1, 5)) -_SOURCE_SHA = re.compile(r"^[0-9a-f]{40}$") -_SHA256 = re.compile(r"^[0-9a-f]{64}$") - - -class Phase1FreezeError(RuntimeError): - """The external ledger cannot certify the tracked v2 Phase 1 release.""" - - -@dataclass(frozen=True) -class _RunEvidence: - hypothesis_id: str - arm_id: str - tasks: tuple[EvaluationTask, ...] - jobs: tuple[JobSpec, ...] - manifest: StoredRecord - summary: dict[str, Any] - - -@dataclass(frozen=True) -class _Bundle: - files: dict[PurePosixPath, bytes] - release_manifest: dict[str, Any] - - @property - def all_files(self) -> dict[PurePosixPath, bytes]: - release_path = PurePosixPath("release_manifest.json") - release_bytes = _json_bytes(self.release_manifest) - checksummed = {release_path: release_bytes, **self.files} - checksums = b"".join( - ( - f"{_sha256_bytes(content)} {path.as_posix()}\n" - ).encode("ascii") - for path, content in sorted( - checksummed.items(), - key=lambda item: item[0], - ) - ) - return { - **checksummed, - PurePosixPath("checksums.sha256"): checksums, - } - - -@dataclass(frozen=True) -class _AttemptEvidence: - row: dict[str, Any] - program_row: dict[str, Any] - task: EvaluationTask - stored_evaluation: dict[str, Any] - - -def _json_bytes(value: Any) -> bytes: - try: - return ( - json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - indent=2, - sort_keys=True, - ) - + "\n" - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise Phase1FreezeError("release evidence is not finite JSON") from exc - - -def _jsonl_bytes(values: list[dict[str, Any]]) -> bytes: - try: - return b"".join( - ( - json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ) - + "\n" - ).encode("utf-8") - for value in values - ) - except (TypeError, ValueError) as exc: - raise Phase1FreezeError("attempt index is not finite JSON") from exc - - -def _sha256_bytes(value: bytes) -> str: - return hashlib.sha256(value).hexdigest() - - -def _load_json(path: Path) -> dict[str, Any]: - def reject_constant(value: str) -> Any: - raise Phase1FreezeError(f"{path} contains non-finite number {value}") - - def reject_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: - result: dict[str, Any] = {} - for key, value in pairs: - if key in result: - raise Phase1FreezeError( - f"{path} contains duplicate key {key!r}" - ) - result[key] = value - return result - - try: - value = json.loads( - path.read_bytes(), - parse_constant=reject_constant, - object_pairs_hook=reject_duplicates, - ) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise Phase1FreezeError( - f"cannot read canonical JSON from {path}" - ) from exc - if not isinstance(value, dict): - raise Phase1FreezeError(f"{path} must contain a JSON object") - return value - - -def _git_head(repo_root: Path) -> str: - try: - return subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo_root, - text=True, - ).strip() - except (OSError, subprocess.CalledProcessError) as exc: - raise Phase1FreezeError("repository HEAD cannot be resolved") from exc - - -def _require_launch_source_state( - *, - repo_root: Path, - output_root: Path, - expected_source_sha: str, - validate_only: bool, -) -> None: - """Bind source to launch SHA while allowing a later data-only commit.""" - - head = _git_head(repo_root) - if head == expected_source_sha: - return - if not validate_only: - raise Phase1FreezeError("current source HEAD differs from the launch SHA") - try: - subprocess.run( - [ - "git", - "merge-base", - "--is-ancestor", - expected_source_sha, - head, - ], - cwd=repo_root, - check=True, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - changed = subprocess.check_output( - [ - "git", - "diff", - "--name-only", - "-z", - f"{expected_source_sha}..{head}", - ], - cwd=repo_root, - ) - except (OSError, subprocess.CalledProcessError) as exc: - raise Phase1FreezeError( - "the launch SHA is not an ancestor of the evidence commit" - ) from exc - drift = [] - for raw in changed.split(b"\0"): - if not raw: - continue - path = (repo_root / os.fsdecode(raw)).resolve(strict=False) - if path != output_root and not path.is_relative_to(output_root): - drift.append(str(path.relative_to(repo_root))) - if drift: - raise Phase1FreezeError( - "the evidence commit changes source beyond the frozen Phase 1 " - f"record: {sorted(drift)}" - ) - - -def _changed_paths(repo_root: Path) -> set[Path]: - commands = ( - ["git", "diff", "--name-only", "-z"], - ["git", "diff", "--cached", "--name-only", "-z"], - ["git", "ls-files", "--others", "--exclude-standard", "-z"], - ) - result: set[Path] = set() - for command in commands: - try: - output = subprocess.check_output(command, cwd=repo_root) - except (OSError, subprocess.CalledProcessError) as exc: - raise Phase1FreezeError( - "repository worktree cannot be inspected" - ) from exc - for raw in output.split(b"\0"): - if raw: - result.add( - (repo_root / os.fsdecode(raw)).resolve(strict=False) - ) - return result - - -def _require_source_clean_except_output( - *, - repo_root: Path, - output_root: Path, -) -> None: - drift = sorted( - str(path.relative_to(repo_root)) - for path in _changed_paths(repo_root) - if path != output_root and not path.is_relative_to(output_root) - ) - if drift: - raise Phase1FreezeError( - "source worktree differs from the paid-launch commit outside the " - f"tracked v2 Phase 1 output: {drift}" - ) - - -def _phase1_output_root(repo_root: Path, requested: Path | None) -> Path: - root = repo_root.expanduser().resolve(strict=True) - expected = (root / PHASE1_RELATIVE_ROOT).resolve(strict=False) - if expected == root or not expected.is_relative_to(root): - raise Phase1FreezeError( - "the tracked v2 Phase 1 path resolves outside the repository" - ) - observed = ( - expected - if requested is None - else requested.expanduser().resolve(strict=False) - ) - if observed != expected: - raise Phase1FreezeError( - f"v2 Phase 1 evidence must be written exactly to {expected}" - ) - return expected - - -def _validate_roots( - *, - repo_root: Path, - external_root: Path, - output_root: Path, -) -> tuple[Path, Path]: - source = repo_root.expanduser().resolve(strict=True) - external = external_root.expanduser().resolve(strict=True) - if external == source or external.is_relative_to(source): - raise Phase1FreezeError( - "the mutable attempt ledger must remain outside the repository" - ) - if ( - output_root == external - or output_root.is_relative_to(external) - or external.is_relative_to(output_root) - ): - raise Phase1FreezeError("source ledger and tracked output overlap") - return source, external - - -def _prompt_binding() -> tuple[dict[str, str], str]: - binding = { - "contract_version": CONTRACT_VERSION, - "preprocess_version": IMAGE_PREPROCESS_VERSION, - **prompt_asset_hashes(), - } - if binding["contract_version"] != EXPECTED_CONTRACT: - raise Phase1FreezeError( - "the active prompt is not the direct reconstruction v2 contract" - ) - return binding, canonical_json_sha256(binding) - - -@contextmanager -def _offline_model_assets() -> Iterator[None]: - """Prevent prompt certification from consulting mutable remote assets.""" - - names = ( - "HF_DATASETS_OFFLINE", - "HF_HUB_OFFLINE", - "TRANSFORMERS_OFFLINE", - ) - previous = {name: os.environ.get(name) for name in names} - os.environ.update({name: "1" for name in names}) - try: - yield - finally: - for name, value in previous.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value - - -def _expected_prompt_lengths( - protocol: dict[str, Any], - tasks: list[EvaluationTask], -) -> dict[tuple[str, str], int]: - """Re-render every bound input without allowing a remote model client.""" - - def forbidden_remote_client() -> Any: - raise Phase1FreezeError( - "prompt certification must not create a remote model client" - ) - - result: dict[tuple[str, str], int] = {} - with _offline_model_assets(): - for arm_id in EXPECTED_HYPOTHESES.values(): - arm = protocol["arms"][arm_id] - resolved_arm = arm_spec(protocol, arm_id) - sampler = TinkerSampler( - _sampler_target(resolved_arm), - service_client_factory=forbidden_remote_client, - ) - for task in tasks: - prompt = sampler.build_prompt(_messages(task, resolved_arm)) - prompt_tokens = int(prompt.length) - if ( - prompt_tokens + int(arm["max_output_tokens"]) - > int(arm["context_tokens"]) - ): - raise Phase1FreezeError( - f"{arm_id}/{task.task_id} no longer fits its context" - ) - result[(arm_id, task.task_id)] = prompt_tokens - expected_count = len(EXPECTED_HYPOTHESES) * len(tasks) - if len(result) != expected_count: - raise Phase1FreezeError( - "prompt certification did not cover every arm/task input" - ) - return result - - -def _validate_protocol_shape(protocol: dict[str, Any]) -> None: - if protocol.get("study_id") != EXPECTED_STUDY: - raise Phase1FreezeError("the protocol is not representation-curriculum-v2") - if protocol.get("contract_version") != EXPECTED_CONTRACT: - raise Phase1FreezeError("the protocol does not bind the v2 prompt") - if set(protocol.get("hypotheses", {})) != set(EXPECTED_HYPOTHESES): - raise Phase1FreezeError("the v2 Phase 1 hypothesis registry changed") - if set(protocol.get("arms", {})) != set(EXPECTED_HYPOTHESES.values()): - raise Phase1FreezeError("the v2 Phase 1 arm registry changed") - if protocol.get("sampling", {}).get("temperature") != 1.0: - raise Phase1FreezeError("v2 Phase 1 requires temperature 1.0") - if protocol.get("sampling", {}).get("top_p") != 1.0: - raise Phase1FreezeError("v2 Phase 1 requires top-p 1.0") - - qwen = protocol["arms"]["qwen-on-60000"] - if { - "model": qwen.get("model"), - "renderer": qwen.get("renderer"), - "thinking": qwen.get("thinking"), - "thinking_effort": qwen.get("thinking_effort"), - "max_output_tokens": qwen.get("max_output_tokens"), - "context_tokens": qwen.get("context_tokens"), - "max_image_long_edge": qwen.get("max_image_long_edge"), - } != { - "model": "Qwen/Qwen3.6-35B-A3B", - "renderer": "qwen3_5", - "thinking": True, - "thinking_effort": None, - "max_output_tokens": 60000, - "context_tokens": 65536, - "max_image_long_edge": 1920, - }: - raise Phase1FreezeError("the Qwen 60k arm changed") - - inkling = protocol["arms"]["inkling-e09-60000"] - if { - "model": inkling.get("model"), - "renderer": inkling.get("renderer"), - "thinking": inkling.get("thinking"), - "thinking_effort": inkling.get("thinking_effort"), - "max_output_tokens": inkling.get("max_output_tokens"), - "context_tokens": inkling.get("context_tokens"), - "max_image_long_edge": inkling.get("max_image_long_edge"), - } != { - "model": "thinkingmachines/Inkling", - "renderer": "tml_v0", - "thinking": True, - "thinking_effort": 0.9, - "max_output_tokens": 60000, - "context_tokens": 65536, - "max_image_long_edge": 1920, - }: - raise Phase1FreezeError("the Inkling 60k arm changed") - - for hypothesis_id, arm_id in EXPECTED_HYPOTHESES.items(): - hypothesis = protocol["hypotheses"][hypothesis_id] - expected_waves = { - "smoke": {"attempt_indices": [1], "task_count": 8}, - "complete": { - "attempt_indices": list(EXPECTED_ATTEMPTS), - "task_count": 8, - }, - } - if ( - hypothesis.get("arm_ids") != [arm_id] - or hypothesis.get("attempts_per_task") != 4 - or hypothesis.get("task_set") != "f1_f8" - or hypothesis.get("status") != "frozen" - or hypothesis.get("waves") != expected_waves - ): - raise Phase1FreezeError(f"{hypothesis_id} changed") - - -def _expected_run_manifest( - *, - protocol: dict[str, Any], - hypothesis_id: str, - arm_id: str, - source_git_sha: str, - task_set_sha256: str, - prompt_sha256: str, - sandbox_image_ref: str, - sandbox_image_id: str, - runtime: dict[str, Any], -) -> dict[str, Any]: - if EXPECTED_HYPOTHESES.get(hypothesis_id) != arm_id: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} is not a registered v2 Phase 1 run" - ) - return { - "source_git_sha": source_git_sha, - "protocol_sha256": protocol["logical_sha256"], - "task_set_sha256": task_set_sha256, - "prompt_sha256": prompt_sha256, - "sandbox_image_ref": sandbox_image_ref, - "sandbox_image_id": sandbox_image_id, - "runtime": runtime, - "model_binding": { - "arm_id": arm_id, - **protocol["arms"][arm_id], - "temperature": protocol["sampling"]["temperature"], - "top_p": protocol["sampling"]["top_p"], - }, - } - - -def _load_run( - *, - store: AttemptStore, - protocol: dict[str, Any], - hypothesis_id: str, - arm_id: str, - source_git_sha: str, - tasks: list[EvaluationTask], - jobs: list[JobSpec], - task_set_sha256: str, - prompt_sha256: str, - sandbox: tuple[str, str] | None, - runtime: dict[str, Any], -) -> tuple[_RunEvidence, tuple[str, str]]: - run_key = RunKey(protocol["study_id"], hypothesis_id, arm_id) - manifest = store.load_run_manifest(run_key) - if manifest is None: - raise Phase1FreezeError( - f"missing run manifest for {hypothesis_id}/{arm_id}" - ) - observed_sandbox = ( - str(manifest.payload.get("sandbox_image_ref", "")), - str(manifest.payload.get("sandbox_image_id", "")), - ) - if sandbox is not None and observed_sandbox != sandbox: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} used a different evaluator sandbox" - ) - expected = _expected_run_manifest( - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - source_git_sha=source_git_sha, - task_set_sha256=task_set_sha256, - prompt_sha256=prompt_sha256, - sandbox_image_ref=observed_sandbox[0], - sandbox_image_id=observed_sandbox[1], - runtime=runtime, - ) - if manifest.payload != expected: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} run manifest does not match its launch" - ) - - run_jobs = tuple( - job - for job in jobs - if job.hypothesis_id == hypothesis_id and job.arm_id == arm_id - ) - if len(run_jobs) != 32: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} does not resolve to exactly 32 attempts" - ) - summary = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id=hypothesis_id, - arm_id=arm_id, - jobs=jobs, - tasks=tasks, - ) - if ( - not summary["complete"] - or summary["expected_attempts"] != 32 - or summary["attempts"] != 32 - or summary["tasks"] != 8 - or summary["missing"] - ): - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} is not a complete F1-F8 G4 run" - ) - external_summary = ( - store.external_root.joinpath(*store.run_relative_path(run_key).parts) - / "summary.json" - ) - if not external_summary.is_file(): - raise Phase1FreezeError( - f"missing canonical summary for {hypothesis_id}/{arm_id}" - ) - if _load_json(external_summary) != summary: - raise Phase1FreezeError( - f"{hypothesis_id}/{arm_id} summary does not reproduce from attempts" - ) - return ( - _RunEvidence( - hypothesis_id=hypothesis_id, - arm_id=arm_id, - tasks=tuple(tasks), - jobs=run_jobs, - manifest=manifest, - summary=summary, - ), - observed_sandbox, - ) - - -def _finite_unit_interval( - value: Any, - *, - key: AttemptKey, - field: str, -) -> float: - if isinstance(value, bool) or not isinstance(value, (int, float)): - raise Phase1FreezeError(f"{key} has invalid {field}") - result = float(value) - if not math.isfinite(result) or not 0.0 <= result <= 1.0: - raise Phase1FreezeError(f"{key} has invalid {field}") - return result - - -def _validate_receipt_and_evaluation( - *, - protocol: dict[str, Any], - store: AttemptStore, - task: EvaluationTask, - job: JobSpec, - expected_prompt_tokens: int, - run_manifest_record_sha256: str, -) -> tuple[StoredRecord, StoredRecord]: - key = AttemptKey( - protocol["study_id"], - job.hypothesis_id, - job.arm_id, - job.task_id, - job.attempt_index, - ) - receipt = store.load_sampling_receipt(key) - evaluation = store.load_evaluation(key) - if receipt is None or evaluation is None: - raise Phase1FreezeError(f"missing immutable attempt record for {key}") - - required_receipt = { - "request", - "sample", - "sampling_seconds", - "raw_completion_sha256", - "completion_sha256", - "cost_estimate_usd", - "run_manifest_record_sha256", - } - if set(receipt.payload) != required_receipt: - raise Phase1FreezeError( - f"{key} sampling receipt fields differ from the frozen schema" - ) - if ( - receipt.payload["run_manifest_record_sha256"] - != run_manifest_record_sha256 - ): - raise Phase1FreezeError( - f"{key} sampling receipt belongs to another run manifest" - ) - request = receipt.payload.get("request") - sample = receipt.payload.get("sample") - if not isinstance(request, dict) or not isinstance(sample, dict): - raise Phase1FreezeError(f"{key} has an invalid sampling payload") - arm = protocol["arms"][job.arm_id] - expected_request = { - "max_tokens": int(arm["max_output_tokens"]), - "temperature": protocol["sampling"]["temperature"], - "top_p": protocol["sampling"]["top_p"], - "seed": job.seed, - } - if request != expected_request: - raise Phase1FreezeError(f"{key} sampling request differs from protocol") - required_sample = { - "prompt_tokens", - "completion_tokens", - "stop_reason", - "cap_hit", - "raw_text", - "completion_text", - "reasoning_text", - "answer_text", - "reasoning_tokens_exact", - "answer_tokens_exact", - "reasoning_tokens_estimate", - "answer_tokens_estimate", - "channel_token_count_basis", - "channel_parse_complete", - } - if set(sample) != required_sample: - raise Phase1FreezeError( - f"{key} sample fields differ from the frozen schema" - ) - prompt_tokens = sample["prompt_tokens"] - completion_tokens = sample["completion_tokens"] - if ( - isinstance(prompt_tokens, bool) - or not isinstance(prompt_tokens, int) - or prompt_tokens != expected_prompt_tokens - or isinstance(completion_tokens, bool) - or not isinstance(completion_tokens, int) - or completion_tokens < 0 - or completion_tokens > int(request["max_tokens"]) - or prompt_tokens + int(request["max_tokens"]) - > int(arm["context_tokens"]) - ): - raise Phase1FreezeError( - f"{key} has invalid or non-reproducible token accounting" - ) - if not isinstance(sample["cap_hit"], bool): - raise Phase1FreezeError(f"{key} has a non-boolean cap-hit flag") - if sample["cap_hit"] != (sample["stop_reason"] == "length"): - raise Phase1FreezeError( - f"{key} cap-hit flag differs from its stop reason" - ) - if sample["cap_hit"] and completion_tokens != int(request["max_tokens"]): - raise Phase1FreezeError( - f"{key} length stop did not reach the output cap" - ) - - reasoning_exact = sample["reasoning_tokens_exact"] - answer_exact = sample["answer_tokens_exact"] - if (reasoning_exact is None) != (answer_exact is None): - raise Phase1FreezeError(f"{key} has a partial exact channel partition") - if reasoning_exact is not None and ( - isinstance(reasoning_exact, bool) - or not isinstance(reasoning_exact, int) - or reasoning_exact < 0 - or isinstance(answer_exact, bool) - or not isinstance(answer_exact, int) - or answer_exact < 0 - or reasoning_exact + answer_exact != completion_tokens - ): - raise Phase1FreezeError( - f"{key} exact channel tokens do not partition output" - ) - for field in ("reasoning_tokens_estimate", "answer_tokens_estimate"): - value = sample[field] - if value is not None and ( - isinstance(value, bool) - or not isinstance(value, int) - or value < 0 - ): - raise Phase1FreezeError(f"{key} has invalid {field}") - if not isinstance(sample["channel_parse_complete"], bool): - raise Phase1FreezeError(f"{key} has invalid channel parse state") - - raw_text = str(sample["raw_text"]) - completion = str(sample["completion_text"]) - if receipt.payload.get("raw_completion_sha256") != _sha256_bytes( - raw_text.encode("utf-8") - ): - raise Phase1FreezeError(f"{key} raw completion digest mismatch") - if receipt.payload.get("completion_sha256") != _sha256_bytes( - completion.encode("utf-8") - ): - raise Phase1FreezeError(f"{key} completion digest mismatch") - sampling_seconds = receipt.payload.get("sampling_seconds") - if ( - isinstance(sampling_seconds, bool) - or not isinstance(sampling_seconds, (int, float)) - or not math.isfinite(float(sampling_seconds)) - or float(sampling_seconds) < 0 - ): - raise Phase1FreezeError(f"{key} has invalid sampling duration") - costs = receipt.payload.get("cost_estimate_usd") - if not isinstance(costs, dict) or set(costs) != { - "cached_prefill", - "uncached_prefill", - }: - raise Phase1FreezeError(f"{key} has invalid cost evidence") - rates = protocol["pricing"]["models"][arm["model"]] - sample_cost = completion_tokens * float(rates["sample"]) / 1_000_000 - expected_costs = { - "cached_prefill": ( - prompt_tokens * float(rates["prefill_cached"]) / 1_000_000 - + sample_cost - ), - "uncached_prefill": ( - prompt_tokens * float(rates["prefill_uncached"]) / 1_000_000 - + sample_cost - ), - } - if costs != expected_costs: - raise Phase1FreezeError(f"{key} cost evidence does not reproduce") - - measured = evaluation.payload - required_evaluation = { - "status", - "attribution", - "pure_executable", - "measurement_available", - "measured_iou", - "measured_dice", - "aggregation_iou", - "program", - "program_sha256", - "reference_sha256", - "violations", - "error", - "retryable", - "evaluation_seconds", - "diagnostics", - } - allowed_evaluation = required_evaluation | {"render_sha256"} - if not required_evaluation.issubset(measured) or not set(measured).issubset( - allowed_evaluation - ): - raise Phase1FreezeError( - f"{key} evaluation fields differ from the frozen schema" - ) - if measured["attribution"] != "model": - raise Phase1FreezeError( - f"{key} is an evaluator/reference failure, not a model result" - ) - program = str(measured["program"]) - if program != extract_code(completion): - raise Phase1FreezeError( - f"{key} stored program does not match deterministic extraction" - ) - if measured["program_sha256"] != _sha256_bytes( - program.encode("utf-8") - ): - raise Phase1FreezeError(f"{key} program digest mismatch") - if measured["reference_sha256"] != task.reference.target_image_sha256: - raise Phase1FreezeError(f"{key} reference digest mismatch") - if not _SHA256.fullmatch(str(measured["program_sha256"])): - raise Phase1FreezeError(f"{key} has invalid program digest") - - aggregation_iou = _finite_unit_interval( - measured["aggregation_iou"], - key=key, - field="aggregation_iou", - ) - measured_iou = measured["measured_iou"] - measured_dice = measured["measured_dice"] - if (measured_iou is None) != (measured_dice is None): - raise Phase1FreezeError(f"{key} has partial geometry metrics") - expected_iou = 0.0 - if measured_iou is not None: - expected_iou = _finite_unit_interval( - measured_iou, - key=key, - field="measured_iou", - ) - _finite_unit_interval( - measured_dice, - key=key, - field="measured_dice", - ) - if aggregation_iou != expected_iou: - raise Phase1FreezeError( - f"{key} aggregation IoU is not the raw model IoU" - ) - if ( - not isinstance(measured["measurement_available"], bool) - or measured["measurement_available"] != (measured_iou is not None) - ): - raise Phase1FreezeError( - f"{key} measurement-availability flag is inconsistent" - ) - if measured["measurement_available"] != (measured["status"] == "ok"): - raise Phase1FreezeError( - f"{key} successful execution and geometry measurement disagree" - ) - if not isinstance(measured["diagnostics"], dict): - raise Phase1FreezeError(f"{key} has invalid evaluator diagnostics") - if measured_iou is not None: - render_sha = measured.get("render_sha256") - if not isinstance(render_sha, str) or not _SHA256.fullmatch(render_sha): - raise Phase1FreezeError( - f"{key} measured geometry has no valid render digest" - ) - if measured["diagnostics"].get("render_sha256") != render_sha: - raise Phase1FreezeError( - f"{key} render digest differs from evaluator diagnostics" - ) - if ( - measured["diagnostics"].get("reference_sha256") - != task.reference.target_image_sha256 - ): - raise Phase1FreezeError( - f"{key} diagnostics belong to another reference" - ) - if ( - not isinstance(measured["pure_executable"], bool) - or measured["pure_executable"] != (measured["status"] == "ok") - ): - raise Phase1FreezeError( - f"{key} executable flag is inconsistent with status" - ) - if measured["retryable"] is not False: - raise Phase1FreezeError( - f"{key} model-attributed evaluation cannot be retryable" - ) - evaluation_seconds = measured["evaluation_seconds"] - if ( - isinstance(evaluation_seconds, bool) - or not isinstance(evaluation_seconds, (int, float)) - or not math.isfinite(float(evaluation_seconds)) - or float(evaluation_seconds) < 0 - ): - raise Phase1FreezeError(f"{key} has invalid evaluation duration") - return receipt, evaluation - - -def _attempt_row( - *, - protocol: dict[str, Any], - store: AttemptStore, - evidence: _RunEvidence, - job: JobSpec, - expected_prompt_tokens: int, -) -> _AttemptEvidence: - tasks = {task.task_id: task for task in evidence.tasks} - task = tasks[job.task_id] - receipt, evaluation = _validate_receipt_and_evaluation( - protocol=protocol, - store=store, - task=task, - job=job, - expected_prompt_tokens=expected_prompt_tokens, - run_manifest_record_sha256=evidence.manifest.record_sha256, - ) - sample = receipt.payload["sample"] - measured = evaluation.payload - row = { - "schema_version": ATTEMPT_INDEX_SCHEMA, - "study_id": protocol["study_id"], - "hypothesis_id": job.hypothesis_id, - "arm_id": job.arm_id, - "task_id": job.task_id, - "attempt_index": job.attempt_index, - "seed": job.seed, - "status": measured["status"], - "pure_executable": measured["pure_executable"], - "measurement_available": measured["measurement_available"], - "aggregation_iou": measured["aggregation_iou"], - "measured_iou": measured["measured_iou"], - "measured_dice": measured["measured_dice"], - "violations": measured["violations"], - "cap_hit": sample["cap_hit"], - "stop_reason": sample["stop_reason"], - "prompt_tokens": sample["prompt_tokens"], - "completion_tokens": sample["completion_tokens"], - "reasoning_tokens_exact": sample["reasoning_tokens_exact"], - "answer_tokens_exact": sample["answer_tokens_exact"], - "reasoning_tokens_estimate": sample[ - "reasoning_tokens_estimate" - ], - "answer_tokens_estimate": sample["answer_tokens_estimate"], - "channel_token_count_basis": sample["channel_token_count_basis"], - "channel_parse_complete": sample["channel_parse_complete"], - "estimated_cost_usd": receipt.payload["cost_estimate_usd"], - "sampling_seconds": receipt.payload["sampling_seconds"], - "evaluation_seconds": measured["evaluation_seconds"], - "raw_completion_sha256": receipt.payload[ - "raw_completion_sha256" - ], - "completion_sha256": receipt.payload["completion_sha256"], - "program_sha256": measured["program_sha256"], - "reference_sha256": measured["reference_sha256"], - "render_sha256": measured.get("render_sha256"), - "sampling_record_sha256": receipt.record_sha256, - "evaluation_record_sha256": evaluation.record_sha256, - } - program = str(measured["program"]) - return _AttemptEvidence( - row=row, - program_row={ - "schema_version": PROGRAM_INDEX_SCHEMA, - "study_id": protocol["study_id"], - "hypothesis_id": job.hypothesis_id, - "arm_id": job.arm_id, - "task_id": job.task_id, - "attempt_index": job.attempt_index, - "program": program, - "program_sha256": measured["program_sha256"], - "reference_sha256": measured["reference_sha256"], - }, - task=task, - stored_evaluation=measured, - ) - - -def _same_optional_metric(observed: float | None, expected: Any) -> bool: - if observed is None or expected is None: - return observed is None and expected is None - return math.isclose( - float(observed), - float(expected), - rel_tol=0.0, - abs_tol=1e-12, - ) - - -def _compare_replayed_evaluation( - evidence: _AttemptEvidence, - replayed: EvaluationResult, -) -> None: - """Require deterministic recomputation to reproduce the stored verdict.""" - - row = evidence.row - stored = evidence.stored_evaluation - identity = ( - f"{row['hypothesis_id']}/{row['arm_id']}/" - f"{row['task_id']}/attempt-{row['attempt_index']}" - ) - if replayed.attribution is not Attribution.MODEL: - raise Phase1FreezeError( - f"{identity} replay produced an evaluator/reference failure: " - f"{replayed.attribution.value}/{replayed.status.value}" - ) - if replayed.status.value != stored["status"]: - raise Phase1FreezeError( - f"{identity} replay status differs: " - f"{replayed.status.value} != {stored['status']}" - ) - if tuple(replayed.violations) != tuple(stored["violations"]): - raise Phase1FreezeError(f"{identity} replay purity verdict differs") - if not _same_optional_metric(replayed.iou, stored["measured_iou"]): - raise Phase1FreezeError(f"{identity} replay IoU differs") - if not _same_optional_metric(replayed.dice, stored["measured_dice"]): - raise Phase1FreezeError(f"{identity} replay Dice differs") - - replay_measurement = replayed.iou is not None - replay_iou = float(replayed.iou) if replayed.iou is not None else 0.0 - if replay_measurement != stored["measurement_available"]: - raise Phase1FreezeError( - f"{identity} replay measurement availability differs" - ) - if not math.isclose( - replay_iou, - float(stored["aggregation_iou"]), - rel_tol=0.0, - abs_tol=1e-12, - ): - raise Phase1FreezeError(f"{identity} replay aggregation IoU differs") - if (replayed.status.value == "ok") != stored["pure_executable"]: - raise Phase1FreezeError(f"{identity} replay executable verdict differs") - - replay_render = replayed.metrics.get("render_sha256") - if replay_render != stored.get("render_sha256"): - raise Phase1FreezeError(f"{identity} replay render digest differs") - replay_reference = replayed.metrics.get("reference_sha256") - if replay_measurement and ( - replay_reference != evidence.task.reference.target_image_sha256 - ): - raise Phase1FreezeError(f"{identity} replay reference digest differs") - - -def _replay_attempts( - attempts: list[_AttemptEvidence], - *, - expected_sandbox: tuple[str, str], -) -> None: - """Re-execute all recorded programs inside the immutable sandbox.""" - - requests = [ - (evidence.task.reference, evidence.program_row["program"]) - for evidence in attempts - ] - with PixCellEvaluator( - max_workers=8, - evaluator_retries=1, - require_isolation=True, - ) as evaluator: - boundary = evaluator.execution_boundary - if boundary is None or ( - boundary.image_ref, - boundary.image_id, - ) != expected_sandbox: - raise Phase1FreezeError( - "certification sandbox differs from the paid launch sandbox" - ) - replayed = evaluator.evaluate_batch(requests) - if len(replayed) != len(attempts): - raise Phase1FreezeError("evaluator replay returned the wrong row count") - for evidence, result in zip(attempts, replayed, strict=True): - _compare_replayed_evaluation(evidence, result) - - -def _published_program_rows( - attempts: list[_AttemptEvidence], -) -> list[dict[str, Any]]: - """Publish executable code, never raw fallback text from failed outputs.""" - - return [ - evidence.program_row - for evidence in attempts - if evidence.row["status"] == "ok" - ] - - -def _collect_bundle( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, -) -> _Bundle: - if not _SOURCE_SHA.fullmatch(expected_source_sha): - raise Phase1FreezeError( - "expected source SHA must be a full lowercase Git commit SHA" - ) - protocol = load_protocol(repo_root, protocol_file=PROTOCOL_FILE) - _validate_protocol_shape(protocol) - manifest = load_task_manifest(TASK_MANIFEST_FILE, repo_root=repo_root) - prompt_binding, prompt_sha = _prompt_binding() - runtime = validate_runtime_stack(repo_root) - store = AttemptStore(repo_root=repo_root, external_root=external_root) - - tasks = load_task_set( - repo_root=repo_root, - manifest=manifest, - task_set="f1_f8", - ) - task_ids = tuple(task.task_id for task in tasks) - if task_ids != EXPECTED_TASK_IDS: - raise Phase1FreezeError( - f"the benchmark is not ordered exactly F1-F8: {task_ids}" - ) - task_set_sha = canonical_json_sha256(manifest["task_sets"]["f1_f8"]) - prompt_lengths = _expected_prompt_lengths(protocol, tasks) - - sandbox: tuple[str, str] | None = None - runs: list[_RunEvidence] = [] - for hypothesis_id, arm_id in EXPECTED_HYPOTHESES.items(): - jobs = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name="complete", - task_ids=list(task_ids), - ) - evidence, sandbox = _load_run( - store=store, - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - source_git_sha=expected_source_sha, - tasks=tasks, - jobs=jobs, - task_set_sha256=task_set_sha, - prompt_sha256=prompt_sha, - sandbox=sandbox, - runtime=runtime, - ) - runs.append(evidence) - - attempt_evidence = [ - _attempt_row( - protocol=protocol, - store=store, - evidence=run, - job=job, - expected_prompt_tokens=prompt_lengths[ - (job.arm_id, job.task_id) - ], - ) - for run in runs - for job in run.jobs - ] - attempt_evidence.sort( - key=lambda value: ( - value.row["hypothesis_id"], - value.row["task_id"], - value.row["attempt_index"], - ) - ) - if len(attempt_evidence) != 64: - raise Phase1FreezeError("v2 Phase 1 must contain exactly 64 attempts") - if sandbox is None: - raise Phase1FreezeError("v2 Phase 1 has no bound evaluator sandbox") - _replay_attempts( - attempt_evidence, - expected_sandbox=sandbox, - ) - attempts = [evidence.row for evidence in attempt_evidence] - programs = _published_program_rows(attempt_evidence) - - summaries_document = { - "schema_version": SUMMARIES_SCHEMA, - "study_id": protocol["study_id"], - "summaries": { - f"{run.hypothesis_id}/{run.arm_id}": run.summary for run in runs - }, - } - summaries_document["logical_sha256"] = canonical_json_sha256( - summaries_document - ) - files: dict[PurePosixPath, bytes] = { - PurePosixPath("attempts.jsonl"): _jsonl_bytes(attempts), - PurePosixPath("programs.jsonl"): _jsonl_bytes(programs), - PurePosixPath("summaries.json"): _json_bytes(summaries_document), - } - file_inventory = [ - { - "path": path.as_posix(), - "sha256": _sha256_bytes(content), - "bytes": len(content), - } - for path, content in sorted(files.items(), key=lambda item: item[0]) - ] - release_manifest: dict[str, Any] = { - "schema_version": RELEASE_SCHEMA, - "study_id": protocol["study_id"], - "phase": "phase1-zero-shot-baselines", - "source_git_sha": expected_source_sha, - "protocol_sha256": protocol["logical_sha256"], - "task_manifest_sha256": manifest["logical_sha256"], - "task_set_sha256": task_set_sha, - "task_ids": list(task_ids), - "prompt": prompt_binding, - "prompt_sha256": prompt_sha, - "runtime": runtime, - "sandbox": { - "image_ref": sandbox[0] if sandbox is not None else "", - "image_id": sandbox[1] if sandbox is not None else "", - }, - "attempt_count": len(attempts), - "program_count": len(programs), - "certification": { - "prompt_token_counts_recomputed_offline": True, - "programs_reexecuted_in_bound_sandbox": len(attempts), - "stored_and_replayed_metrics_must_match": True, - "published_programs_are_successful_executables_only": True, - }, - "runs": [ - { - "hypothesis_id": run.hypothesis_id, - "arm_id": run.arm_id, - "expected_attempts": len(run.jobs), - "run_manifest_payload": run.manifest.payload, - "run_manifest_payload_sha256": run.manifest.payload_sha256, - "run_manifest_record_sha256": run.manifest.record_sha256, - "summary_sha256": run.summary["logical_sha256"], - "wandb_run_id": deterministic_wandb_run_id( - RunKey( - protocol["study_id"], - run.hypothesis_id, - run.arm_id, - ), - source_git_sha=expected_source_sha, - ), - } - for run in runs - ], - "tracking": { - "provider": protocol["tracking"]["provider"], - "project": protocol["tracking"]["project"], - "entity": protocol["tracking"].get("entity"), - "group": protocol["tracking"].get("group"), - "local_record_is_authoritative": protocol["tracking"][ - "local_record_is_authoritative" - ], - }, - "files": file_inventory, - "excluded_external_state": [ - "raw_completion_text", - "completion_text", - "reasoning_text", - "answer_text", - "wandb_state", - "tracking_errors", - ], - } - release_manifest["logical_sha256"] = canonical_json_sha256( - release_manifest - ) - return _Bundle(files=files, release_manifest=release_manifest) - - -def _assert_exact_output( - output_root: Path, - expected: dict[PurePosixPath, bytes], -) -> None: - if not output_root.is_dir(): - raise Phase1FreezeError( - f"tracked v2 Phase 1 release is missing: {output_root}" - ) - observed_paths = { - PurePosixPath(path.relative_to(output_root).as_posix()) - for path in output_root.rglob("*") - if path.is_file() - } - expected_paths = set(expected) - if observed_paths != expected_paths: - raise Phase1FreezeError( - "tracked v2 Phase 1 file inventory differs: " - f"missing={sorted(str(value) for value in expected_paths - observed_paths)}, " - f"extra={sorted(str(value) for value in observed_paths - expected_paths)}" - ) - for relative, content in expected.items(): - path = output_root.joinpath(*relative.parts) - if path.is_symlink() or path.read_bytes() != content: - raise Phase1FreezeError(f"tracked evidence differs at {relative}") - - -def _write_create_or_verify( - output_root: Path, - expected: dict[PurePosixPath, bytes], -) -> None: - output_root.mkdir(parents=True, exist_ok=True) - observed_files = [ - path - for path in output_root.rglob("*") - if path.is_file() or path.is_symlink() - ] - expected_paths = { - output_root.joinpath(*relative.parts): content - for relative, content in expected.items() - } - extras = sorted( - str(path.relative_to(output_root)) - for path in observed_files - if path not in expected_paths - ) - if extras: - raise Phase1FreezeError( - "refusing to overwrite a non-canonical v2 Phase 1 directory: " - f"{extras}" - ) - for path, content in expected_paths.items(): - if path.exists() or path.is_symlink(): - if path.is_symlink() or path.read_bytes() != content: - raise Phase1FreezeError( - f"refusing to replace existing evidence at {path}" - ) - continue - path.parent.mkdir(parents=True, exist_ok=True) - temporary = ( - path.parent / f".{path.name}.tmp-{os.getpid()}-{uuid.uuid4().hex}" - ) - try: - with temporary.open("xb") as handle: - handle.write(content) - handle.flush() - os.fsync(handle.fileno()) - os.replace(temporary, path) - finally: - temporary.unlink(missing_ok=True) - _assert_exact_output(output_root, expected) - - -def freeze_phase1( - *, - repo_root: Path, - external_root: Path, - expected_source_sha: str, - output: Path | None = None, - validate_only: bool = False, -) -> dict[str, Any]: - """Recompute the canonical record, then write or validate exact bytes.""" - - output_root = _phase1_output_root(repo_root, output) - source, external = _validate_roots( - repo_root=repo_root, - external_root=external_root, - output_root=output_root, - ) - _require_source_clean_except_output( - repo_root=source, - output_root=output_root, - ) - if not _SOURCE_SHA.fullmatch(expected_source_sha): - raise Phase1FreezeError( - "expected source SHA must be a full lowercase Git commit SHA" - ) - _require_launch_source_state( - repo_root=source, - output_root=output_root, - expected_source_sha=expected_source_sha, - validate_only=validate_only, - ) - bundle = _collect_bundle( - repo_root=source, - external_root=external, - expected_source_sha=expected_source_sha, - ) - expected = bundle.all_files - if validate_only: - _assert_exact_output(output_root, expected) - else: - _write_create_or_verify(output_root, expected) - return bundle.release_manifest - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=REPO_ROOT) - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--output", type=Path) - result.add_argument( - "--validate", - action="store_true", - help="validate an existing tracked release without writing", - ) - return result - - -def main() -> None: - args = parser().parse_args() - report = freeze_phase1( - repo_root=args.repo_root, - external_root=args.external_root, - expected_source_sha=args.expected_source_sha, - output=args.output, - validate_only=args.validate, - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v2/protocol.json b/rl/studies/representation_curriculum_v2/protocol.json deleted file mode 100644 index a1db4eed..00000000 --- a/rl/studies/representation_curriculum_v2/protocol.json +++ /dev/null @@ -1,129 +0,0 @@ -{ - "arms": { - "inkling-e09-60000": { - "context_tokens": 65536, - "max_image_long_edge": 1920, - "max_output_tokens": 60000, - "model": "thinkingmachines/Inkling", - "provider": "tinker", - "renderer": "tml_v0", - "thinking": true, - "thinking_effort": 0.9 - }, - "qwen-on-60000": { - "context_tokens": 65536, - "max_image_long_edge": 1920, - "max_output_tokens": 60000, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": true, - "thinking_effort": null - } - }, - "contract_version": "pixcell-direct-reconstruction-v2", - "hypotheses": { - "RC-H05": { - "arm_ids": [ - "qwen-on-60000" - ], - "attempts_per_task": 4, - "name": "Qwen thinking-on 60k zero-shot F1-F8 baseline", - "status": "frozen", - "task_set": "f1_f8", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2, - 3, - 4 - ], - "task_count": 8 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 8 - } - } - }, - "RC-H06": { - "arm_ids": [ - "inkling-e09-60000" - ], - "attempts_per_task": 4, - "name": "Inkling effort-0.9 60k zero-shot F1-F8 baseline", - "status": "frozen", - "task_set": "f1_f8", - "waves": { - "complete": { - "attempt_indices": [ - 1, - 2, - 3, - 4 - ], - "task_count": 8 - }, - "smoke": { - "attempt_indices": [ - 1 - ], - "task_count": 8 - } - } - } - }, - "launch": { - "confirmation_token": "PIXCELL_BASELINE_V2", - "mutable_output_environment": "PIXCELL_STUDY_ROOT" - }, - "logical_sha256": "ee1493e7932ae8f549ba419d7859315617b492e6c4b6c3404f6853090c374dca", - "pricing": { - "as_of": "2026-07-27", - "currency": "USD", - "models": { - "Qwen/Qwen3.6-35B-A3B": { - "prefill_cached": 0.108, - "prefill_uncached": 0.54, - "sample": 1.335 - }, - "thinkingmachines/Inkling": { - "prefill_cached": 0.374, - "prefill_uncached": 1.87, - "sample": 4.68 - } - }, - "source": "https://tinker-docs.thinkingmachines.ai/tinker/models/", - "unit": "per_million_tokens" - }, - "purpose": { - "kind": "direct-policy-zero-shot-baseline", - "operating_point_selection": false, - "training": false - }, - "sampling": { - "evaluator_workers": 8, - "require_candidate_isolation": true, - "sampling_concurrency": 8, - "seed_namespace": "pixcell-representation-curriculum-v2-direct-baselines", - "temperature": 1.0, - "top_p": 1.0 - }, - "schema_version": "pixcell-representation-curriculum-protocol-v1", - "study_id": "representation-curriculum-v2", - "task_manifest": { - "logical_sha256": "05ca50526df05a3a79aa785504e36c7538106e5abdb72b64e86b20cc6f94a890", - "path": "task_manifest.json" - }, - "tracking": { - "entity": "aadarwal-massachusetts-institute-of-technology", - "group": "direct-zero-shot-v2", - "local_record_is_authoritative": true, - "mode": "online", - "project": "pixcell-representation-curriculum-v2", - "provider": "wandb" - } -} diff --git a/rl/studies/representation_curriculum_v2/run_baseline.py b/rl/studies/representation_curriculum_v2/run_baseline.py deleted file mode 100644 index 24216ad4..00000000 --- a/rl/studies/representation_curriculum_v2/run_baseline.py +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/env python3 -"""Run one frozen, resumable v2 direct-policy baseline wave.""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from pathlib import Path - -from rl.evaluation.runner import run_baseline - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PROTOCOL_FILE = Path(__file__).with_name("protocol.json") - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument( - "--hypothesis", - required=True, - choices=("RC-H05", "RC-H06"), - ) - result.add_argument("--wave", required=True, choices=("smoke", "complete")) - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--confirm-spend", default="") - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_STUDY_ROOT"]) - if os.environ.get("PIXCELL_STUDY_ROOT") - else None - ), - ) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit( - "set PIXCELL_STUDY_ROOT or provide --external-root outside the repository" - ) - report = asyncio.run( - run_baseline( - repo_root=REPO_ROOT, - protocol_file=PROTOCOL_FILE, - hypothesis_id=args.hypothesis, - wave_name=args.wave, - expected_source_sha=args.expected_source_sha, - external_root=args.external_root, - confirmation=args.confirm_spend, - ) - ) - print(json.dumps(report["summaries"], indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_curriculum_v2/task_manifest.json b/rl/studies/representation_curriculum_v2/task_manifest.json deleted file mode 100644 index d7e64b81..00000000 --- a/rl/studies/representation_curriculum_v2/task_manifest.json +++ /dev/null @@ -1,148 +0,0 @@ -{ - "logical_sha256": "05ca50526df05a3a79aa785504e36c7538106e5abdb72b64e86b20cc6f94a890", - "schema_version": "pixcell-evaluation-task-manifest-v1", - "selection": { - "algorithm": "ordered fixed repository fixtures data/benchmark/final_1 through final_8", - "seed": null - }, - "task_sets": { - "f1_f8": [ - { - "footprint_um": [ - 40.0, - 1.7 - ], - "image_sha256": "0fbd3bb59c59f81b606be1b5d8d948b941b73ca240d4bb31ba9c9d72e6808d3c", - "image_size_px": [ - 1616, - 656 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f1", - "source": "data/benchmark/final_1", - "source_id": "final_1", - "target_image_sha256": "0fbd3bb59c59f81b606be1b5d8d948b941b73ca240d4bb31ba9c9d72e6808d3c", - "task_id": "F1" - }, - { - "footprint_um": [ - 25.0, - 10.0 - ], - "image_sha256": "f9aac3ccc82c4528647f741d619e246e003e42aab13a0b5687d6a9811068576b", - "image_size_px": [ - 1888, - 2272 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f2", - "source": "data/benchmark/final_2", - "source_id": "final_2", - "target_image_sha256": "f9aac3ccc82c4528647f741d619e246e003e42aab13a0b5687d6a9811068576b", - "task_id": "F2" - }, - { - "footprint_um": [ - 120.0, - 4.0 - ], - "image_sha256": "4962daf2f6b430a3124ba83f0702742b019c24121272f111363c991d847e3c3e", - "image_size_px": [ - 2688, - 1568 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f3", - "source": "data/benchmark/final_3", - "source_id": "final_3", - "target_image_sha256": "4962daf2f6b430a3124ba83f0702742b019c24121272f111363c991d847e3c3e", - "task_id": "F3" - }, - { - "footprint_um": [ - 18.0, - 3.0 - ], - "image_sha256": "a57c3d7acbffa63f691d580bba31ada077ad552c94f360271160912b7e30d701", - "image_size_px": [ - 1584, - 672 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f4", - "source": "data/benchmark/final_4", - "source_id": "final_4", - "target_image_sha256": "a57c3d7acbffa63f691d580bba31ada077ad552c94f360271160912b7e30d701", - "task_id": "F4" - }, - { - "footprint_um": [ - 18.24, - 0.7 - ], - "image_sha256": "6e850e58b0768aa18e828b7b0a4d1b38db4460cef7f3fef0782a58909bf457a3", - "image_size_px": [ - 3712, - 1152 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f5", - "source": "data/benchmark/final_5", - "source_id": "final_5", - "target_image_sha256": "6e850e58b0768aa18e828b7b0a4d1b38db4460cef7f3fef0782a58909bf457a3", - "task_id": "F5" - }, - { - "footprint_um": [ - 4.7, - 4.7 - ], - "image_sha256": "bacae5d13895fd2462fb77fda14d06c56080dd408f362323bc5a7ef2618596a5", - "image_size_px": [ - 1424, - 752 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f6", - "source": "data/benchmark/final_6", - "source_id": "final_6", - "target_image_sha256": "bacae5d13895fd2462fb77fda14d06c56080dd408f362323bc5a7ef2618596a5", - "task_id": "F6" - }, - { - "footprint_um": [ - 3.0, - 5.5 - ], - "image_sha256": "91847bf5797234e985065e7828fbbc306c92c26e281c685175e0efaa4749ce18", - "image_size_px": [ - 1612, - 809 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f7", - "source": "data/benchmark/final_7", - "source_id": "final_7", - "target_image_sha256": "91847bf5797234e985065e7828fbbc306c92c26e281c685175e0efaa4749ce18", - "task_id": "F7" - }, - { - "footprint_um": [ - 120.0, - 4.0 - ], - "image_sha256": "6654508aff5e3fb52f0cecb66a0caa055049e82f6c6ad296386c9e44c341758b", - "image_size_px": [ - 4800, - 3584 - ], - "level": "BENCHMARK", - "representation_id": "benchmark-f8", - "source": "data/benchmark/final_8", - "source_id": "final_8", - "target_image_sha256": "6654508aff5e3fb52f0cecb66a0caa055049e82f6c6ad296386c9e44c341758b", - "task_id": "F8" - } - ] - } -} diff --git a/rl/studies/representation_curriculum_v2/tests/__init__.py b/rl/studies/representation_curriculum_v2/tests/__init__.py deleted file mode 100644 index c7eb3129..00000000 --- a/rl/studies/representation_curriculum_v2/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Tests for the active representation-curriculum-v2 study.""" diff --git a/rl/studies/representation_curriculum_v2/tests/test_audit_phase0.py b/rl/studies/representation_curriculum_v2/tests/test_audit_phase0.py deleted file mode 100644 index 612a029c..00000000 --- a/rl/studies/representation_curriculum_v2/tests/test_audit_phase0.py +++ /dev/null @@ -1,221 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from homi.conductor.source_policy import strict_purity_violations -from rl.common.isolation import ExecutionBoundary -from rl.studies.representation_curriculum_v2 import audit_phase0 - - -REPO_ROOT = Path(__file__).resolve().parents[4] - - -def _protocol() -> dict: - return { - "study_id": "representation-curriculum-v2", - "logical_sha256": "1" * 64, - "arms": { - "inkling-e09-60000": { - "model": "thinkingmachines/Inkling", - "renderer": "tml_v0", - "thinking": True, - "thinking_effort": 0.9, - "context_tokens": 65_536, - "max_output_tokens": 60_000, - "max_image_long_edge": 1920, - "provider": "tinker", - }, - "qwen-on-60000": { - "model": "Qwen/Qwen3.6-35B-A3B", - "renderer": "qwen3_5", - "thinking": True, - "thinking_effort": None, - "context_tokens": 65_536, - "max_output_tokens": 60_000, - "max_image_long_edge": 1920, - "provider": "tinker", - }, - }, - "hypotheses": { - "RC-H05": {"arm_ids": ["qwen-on-60000"]}, - "RC-H06": {"arm_ids": ["inkling-e09-60000"]}, - }, - "sampling": {"temperature": 1.0, "top_p": 1.0}, - } - - -def _boundary(tmp_path: Path) -> ExecutionBoundary: - return ExecutionBoundary( - runtime_path=Path("/usr/bin/docker"), - daemon_endpoint="unix:///var/run/docker.sock", - image_ref="sha256:" + "a" * 64, - image_id="sha256:" + "a" * 64, - workspace_root=tmp_path, - ) - - -def test_forbidden_prompt_fields_fail_closed() -> None: - audit_phase0._assert_prompt_is_model_safe( - "image, footprint, catalogue, and program", - case_id="clean", - ) - with pytest.raises(ValueError, match="verifier/answer-only"): - audit_phase0._assert_prompt_is_model_safe( - "Here is calibration.json and the target raster.", - case_id="leak", - ) - - -def test_evaluator_probe_is_fixed_and_strictly_primitive_only() -> None: - probe = audit_phase0._AUDIT_PROBE_PROGRAM - assert strict_purity_violations(probe) == [] - assert 'write_gds("device.gds")' in probe - assert "gf.components.rectangle" in probe - assert "dataset" not in probe.casefold() - - -def test_renderer_audit_covers_exactly_sixteen_contexts( - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = _protocol() - tasks = [ - SimpleNamespace( - task_id=f"F{index}", - observation=SimpleNamespace(footprint_um=(1.0, 1.0)), - ) - for index in range(1, 9) - ] - - class FakeSampler: - def __init__(self, _target, *, service_client_factory): - self.service_client_factory = service_client_factory - - @staticmethod - def build_prompt(messages): - return SimpleNamespace(length=messages) - - def fake_arm_spec(document, arm_id): - return SimpleNamespace( - **document["arms"][arm_id], - arm_id=arm_id, - ) - - monkeypatch.setattr(audit_phase0, "TinkerSampler", FakeSampler) - monkeypatch.setattr(audit_phase0, "arm_spec", fake_arm_spec) - monkeypatch.setattr(audit_phase0, "_sampler_target", lambda arm: arm) - monkeypatch.setattr( - audit_phase0, - "_messages", - lambda task, _arm: 4000 + int(task.task_id[1:]), - ) - monkeypatch.setattr( - audit_phase0, - "build_prompt_text", - lambda _observation: "image, footprint, catalogue, and program", - ) - - report = audit_phase0._renderer_audit( - protocol=protocol, - tasks=tasks, - ) - assert report["model_client_created"] is False - assert report["prompt_cases"] == 16 - assert set(report["arms"]) == set(audit_phase0.EXPECTED_ARMS) - for summary in report["arms"].values(): - assert summary == { - "prompt_cases": 8, - "prompt_tokens_min": 4001, - "prompt_tokens_max": 4008, - "minimum_headroom_tokens": 1528, - } - - -def test_renderer_audit_rejects_negative_headroom( - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = _protocol() - tasks = [ - SimpleNamespace( - task_id=f"F{index}", - observation=SimpleNamespace(footprint_um=(1.0, 1.0)), - ) - for index in range(1, 9) - ] - - class OversizeSampler: - def __init__(self, _target, *, service_client_factory): - self.service_client_factory = service_client_factory - - @staticmethod - def build_prompt(_messages): - return SimpleNamespace(length=5537) - - monkeypatch.setattr(audit_phase0, "TinkerSampler", OversizeSampler) - monkeypatch.setattr( - audit_phase0, - "arm_spec", - lambda document, arm_id: SimpleNamespace( - **document["arms"][arm_id], - arm_id=arm_id, - ), - ) - monkeypatch.setattr(audit_phase0, "_sampler_target", lambda arm: arm) - monkeypatch.setattr(audit_phase0, "_messages", lambda _task, _arm: []) - monkeypatch.setattr( - audit_phase0, - "build_prompt_text", - lambda _observation: "image, footprint, catalogue, and program", - ) - with pytest.raises(ValueError, match="exceeds context"): - audit_phase0._renderer_audit(protocol=protocol, tasks=tasks) - - -def test_schema_smoke_round_trips_both_arms(tmp_path: Path) -> None: - protocol = _protocol() - runtime = { - "python": "3.13.5", - "packages": {"gdsfactory": "9.20.7"}, - "tinker_cookbook_commit": "b" * 40, - } - report = audit_phase0._schema_smoke( - repo_root=REPO_ROOT, - protocol=protocol, - source_git_sha="c" * 40, - task_set_sha256="d" * 64, - prompt_sha256="e" * 64, - runtime=runtime, - boundary=_boundary(tmp_path), - ) - assert set(report) == set(audit_phase0.EXPECTED_ARMS) - assert all( - set(records) - == { - "run_manifest_record_sha256", - "sampling_record_sha256", - "evaluation_record_sha256", - } - for records in report.values() - ) - - -def test_sealed_audit_rejects_dirty_or_wrong_source( - monkeypatch: pytest.MonkeyPatch, -) -> None: - monkeypatch.setattr( - audit_phase0, - "_git_state", - lambda _root: ("a" * 40, False), - ) - with pytest.raises(ValueError, match="source worktree must be clean"): - audit_phase0.run_audit( - repo_root=REPO_ROOT, - expected_source_sha="a" * 40, - ) - with pytest.raises(ValueError, match="source SHA mismatch"): - audit_phase0.run_audit( - repo_root=REPO_ROOT, - expected_source_sha="b" * 40, - ) diff --git a/rl/studies/representation_curriculum_v2/tests/test_freeze_phase1.py b/rl/studies/representation_curriculum_v2/tests/test_freeze_phase1.py deleted file mode 100644 index e11762db..00000000 --- a/rl/studies/representation_curriculum_v2/tests/test_freeze_phase1.py +++ /dev/null @@ -1,842 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -from rl.common.contracts import ModelObservation, VerifierReference -from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus -from rl.common.output import extract_code -from rl.evaluation.attempt_store import AttemptKey, AttemptStore, RunKey -from rl.evaluation.protocol import jobs_for_wave -from rl.evaluation.summarize import atomic_write_json, summarize_arm -from rl.evaluation.tasks import EvaluationTask, canonical_json_sha256 -from rl.studies.representation_curriculum_v2 import freeze_phase1 as freezer - - -SOURCE_SHA = "a" * 40 -PROMPT_SHA = "b" * 64 -SANDBOX = ("sha256:" + "c" * 64, "sha256:" + "c" * 64) -RUNTIME = { - "python": "3.13.14", - "packages": {"tinker": "0.22.7"}, - "tinker_cookbook_commit": "f" * 40, -} - - -def _arm( - *, - model: str, - renderer: str, - effort: float | None, -) -> dict[str, Any]: - return { - "provider": "tinker", - "model": model, - "renderer": renderer, - "thinking": True, - "thinking_effort": effort, - "max_output_tokens": 60000, - "context_tokens": 65536, - "max_image_long_edge": 1920, - } - - -def _protocol() -> dict[str, Any]: - hypotheses = { - "RC-H05": { - "name": "Qwen thinking-on 60k zero-shot F1-F8 baseline", - "arm_ids": ["qwen-on-60000"], - "attempts_per_task": 4, - "task_set": "f1_f8", - "status": "frozen", - "waves": { - "smoke": {"attempt_indices": [1], "task_count": 8}, - "complete": { - "attempt_indices": [1, 2, 3, 4], - "task_count": 8, - }, - }, - }, - "RC-H06": { - "name": "Inkling effort-0.9 60k zero-shot F1-F8 baseline", - "arm_ids": ["inkling-e09-60000"], - "attempts_per_task": 4, - "task_set": "f1_f8", - "status": "frozen", - "waves": { - "smoke": {"attempt_indices": [1], "task_count": 8}, - "complete": { - "attempt_indices": [1, 2, 3, 4], - "task_count": 8, - }, - }, - }, - } - return { - "study_id": freezer.EXPECTED_STUDY, - "contract_version": freezer.EXPECTED_CONTRACT, - "logical_sha256": "d" * 64, - "dataset": { - "repository": "example/pixcell", - "revision": "test", - "configuration": "depth", - }, - "arms": { - "qwen-on-60000": _arm( - model="Qwen/Qwen3.6-35B-A3B", - renderer="qwen3_5", - effort=None, - ), - "inkling-e09-60000": _arm( - model="thinkingmachines/Inkling", - renderer="tml_v0", - effort=0.9, - ), - }, - "hypotheses": hypotheses, - "sampling": { - "temperature": 1.0, - "top_p": 1.0, - "seed_namespace": "phase1-v2-freezer-test", - }, - "pricing": { - "models": { - "Qwen/Qwen3.6-35B-A3B": { - "prefill_cached": 0.1, - "prefill_uncached": 0.5, - "sample": 1.3, - }, - "thinkingmachines/Inkling": { - "prefill_cached": 0.3, - "prefill_uncached": 1.8, - "sample": 4.6, - }, - } - }, - "tracking": { - "provider": "wandb", - "project": "pixcell-test", - "entity": "test", - "group": "phase1", - "local_record_is_authoritative": True, - }, - } - - -def _task(index: int) -> EvaluationTask: - task_id = f"F{index}" - image = f"image-{task_id}".encode() - digest = hashlib.sha256(image).hexdigest() - return EvaluationTask( - task_id=task_id, - level="BENCHMARK", - representation_id=f"benchmark-f{index}", - observation=ModelObservation( - image_bytes=image, - footprint_um=(10.0, 5.0), - image_sha256=digest, - ), - reference=VerifierReference( - target_image_bytes=image, - footprint_um=(10.0, 5.0), - target_image_sha256=digest, - ), - ) - - -def _manifest(tasks: list[EvaluationTask]) -> dict[str, Any]: - value = { - "task_sets": { - "f1_f8": [{"task_id": task.task_id} for task in tasks], - } - } - value["logical_sha256"] = canonical_json_sha256(value) - return value - - -def _write_attempt( - *, - store: AttemptStore, - protocol: dict[str, Any], - task: EvaluationTask, - job: Any, - manifest_sha: str, - bad_program_hash: bool = False, -) -> None: - key = AttemptKey( - protocol["study_id"], - job.hypothesis_id, - job.arm_id, - job.task_id, - job.attempt_index, - ) - completion = ( - "```python\n" - f"print('{job.hypothesis_id}-{job.arm_id}-{job.task_id}')\n" - "```" - ) - program = extract_code(completion) - raw = f"secret raw reasoning\n{completion}" - rates = protocol["pricing"]["models"][ - protocol["arms"][job.arm_id]["model"] - ] - prompt_tokens = 100 - completion_tokens = 20 - sample_cost = completion_tokens * rates["sample"] / 1_000_000 - store.write_sampling_receipt( - key, - { - "request": { - "max_tokens": 60000, - "temperature": 1.0, - "top_p": 1.0, - "seed": job.seed, - }, - "sample": { - "prompt_tokens": prompt_tokens, - "completion_tokens": completion_tokens, - "stop_reason": "stop", - "cap_hit": False, - "raw_text": raw, - "completion_text": completion, - "reasoning_text": "secret raw reasoning", - "answer_text": completion, - "reasoning_tokens_exact": 2, - "answer_tokens_exact": 18, - "reasoning_tokens_estimate": None, - "answer_tokens_estimate": None, - "channel_token_count_basis": "test", - "channel_parse_complete": True, - }, - "sampling_seconds": 0.5, - "raw_completion_sha256": hashlib.sha256(raw.encode()).hexdigest(), - "completion_sha256": hashlib.sha256( - completion.encode() - ).hexdigest(), - "run_manifest_record_sha256": manifest_sha, - "cost_estimate_usd": { - "cached_prefill": ( - prompt_tokens * rates["prefill_cached"] / 1_000_000 - + sample_cost - ), - "uncached_prefill": ( - prompt_tokens * rates["prefill_uncached"] / 1_000_000 - + sample_cost - ), - }, - }, - ) - store.write_evaluation( - key, - { - "status": "ok", - "attribution": "model", - "pure_executable": True, - "measurement_available": True, - "measured_iou": 0.25, - "measured_dice": 0.4, - "aggregation_iou": 0.25, - "program": program, - "program_sha256": ( - "0" * 64 - if bad_program_hash - else hashlib.sha256(program.encode()).hexdigest() - ), - "reference_sha256": task.reference.target_image_sha256, - "violations": [], - "error": None, - "retryable": False, - "evaluation_seconds": 0.25, - "diagnostics": { - "render_sha256": "e" * 64, - "reference_sha256": task.reference.target_image_sha256, - }, - "render_sha256": "e" * 64, - }, - ) - - -def _write_run( - *, - store: AttemptStore, - protocol: dict[str, Any], - hypothesis_id: str, - arm_id: str, - tasks: list[EvaluationTask], - task_set_sha: str, - omit_last: bool = False, - bad_program_hash: bool = False, -) -> None: - jobs = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name="complete", - task_ids=[task.task_id for task in tasks], - ) - manifest = store.create_or_verify_run_manifest( - RunKey(protocol["study_id"], hypothesis_id, arm_id), - freezer._expected_run_manifest( - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - source_git_sha=SOURCE_SHA, - task_set_sha256=task_set_sha, - prompt_sha256=PROMPT_SHA, - sandbox_image_ref=SANDBOX[0], - sandbox_image_id=SANDBOX[1], - runtime=RUNTIME, - ), - ) - tasks_by_id = {task.task_id: task for task in tasks} - selected = jobs[:-1] if omit_last else jobs - for index, job in enumerate(selected): - _write_attempt( - store=store, - protocol=protocol, - task=tasks_by_id[job.task_id], - job=job, - manifest_sha=manifest.record_sha256, - bad_program_hash=bad_program_hash and index == 0, - ) - summary = summarize_arm( - store=store, - study_id=protocol["study_id"], - hypothesis_id=hypothesis_id, - arm_id=arm_id, - jobs=jobs, - tasks=tasks, - ) - path = ( - store.external_root.joinpath( - *store.run_relative_path( - RunKey(protocol["study_id"], hypothesis_id, arm_id) - ).parts - ) - / "summary.json" - ) - atomic_write_json(path, summary) - - -def _prepare( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - *, - omit_last: bool = False, - bad_program_hash: bool = False, -) -> tuple[Path, Path]: - repo = tmp_path / "repo" - repo.mkdir() - external = tmp_path / "external" - protocol = _protocol() - tasks = [_task(index) for index in range(1, 9)] - manifest = _manifest(tasks) - - monkeypatch.setattr(freezer, "_git_head", lambda _root: SOURCE_SHA) - monkeypatch.setattr(freezer, "_changed_paths", lambda _root: set()) - monkeypatch.setattr( - freezer, - "validate_runtime_stack", - lambda _root: RUNTIME, - ) - monkeypatch.setattr( - freezer, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - freezer, - "load_task_manifest", - lambda _path, *, repo_root: manifest, - ) - monkeypatch.setattr( - freezer, - "load_task_set", - lambda *, repo_root, manifest, task_set: tasks, - ) - monkeypatch.setattr( - freezer, - "_prompt_binding", - lambda: ( - { - "contract_version": freezer.EXPECTED_CONTRACT, - "preprocess_version": "test", - }, - PROMPT_SHA, - ), - ) - monkeypatch.setattr( - freezer, - "_expected_prompt_lengths", - lambda _protocol, tasks: { - (arm_id, task.task_id): 100 - for arm_id in freezer.EXPECTED_HYPOTHESES.values() - for task in tasks - }, - ) - monkeypatch.setattr( - freezer, - "_replay_attempts", - lambda _attempts, *, expected_sandbox: None, - ) - - store = AttemptStore(repo_root=repo, external_root=external) - task_set_sha = canonical_json_sha256(manifest["task_sets"]["f1_f8"]) - for index, (hypothesis_id, arm_id) in enumerate( - freezer.EXPECTED_HYPOTHESES.items() - ): - _write_run( - store=store, - protocol=protocol, - hypothesis_id=hypothesis_id, - arm_id=arm_id, - tasks=tasks, - task_set_sha=task_set_sha, - omit_last=omit_last and index == 0, - bad_program_hash=bad_program_hash and index == 0, - ) - return repo, external - - -def test_freeze_is_exact_idempotent_and_excludes_private_text( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - report = freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - output = repo / freezer.PHASE1_RELATIVE_ROOT - first = { - path.relative_to(output): path.read_bytes() - for path in output.rglob("*") - if path.is_file() - } - assert report["attempt_count"] == 64 - assert report["program_count"] == 64 - assert set(first) == { - Path("attempts.jsonl"), - Path("programs.jsonl"), - Path("summaries.json"), - Path("release_manifest.json"), - Path("checksums.sha256"), - } - combined = b"".join(first.values()) - assert b"secret raw reasoning" not in combined - assert b"```python" not in combined - programs = [ - json.loads(line) - for line in first[Path("programs.jsonl")].decode().splitlines() - ] - assert len(programs) == 64 - assert all("program" in row for row in programs) - assert all( - hashlib.sha256(row["program"].encode()).hexdigest() - == row["program_sha256"] - for row in programs - ) - - checksums = first[Path("checksums.sha256")].decode().splitlines() - assert len(checksums) == 4 - for line in checksums: - digest, relative = line.split(" ", 1) - assert digest == hashlib.sha256( - first[Path(relative)] - ).hexdigest() - - repeated = freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - assert repeated == report - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - validate_only=True, - ) - assert first == { - path.relative_to(output): path.read_bytes() - for path in output.rglob("*") - if path.is_file() - } - - -def test_freeze_rejects_an_incomplete_32_attempt_arm( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare( - tmp_path, - monkeypatch, - omit_last=True, - ) - with pytest.raises( - freezer.Phase1FreezeError, - match="not a complete F1-F8 G4 run", - ): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_freeze_rejects_a_program_digest_that_does_not_match_code( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare( - tmp_path, - monkeypatch, - bad_program_hash=True, - ) - with pytest.raises(freezer.Phase1FreezeError, match="program digest mismatch"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_validate_rejects_tampered_tracked_evidence( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - attempts = repo / freezer.PHASE1_RELATIVE_ROOT / "attempts.jsonl" - attempts.write_bytes(attempts.read_bytes() + b"{}\n") - with pytest.raises(freezer.Phase1FreezeError, match="differs"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - validate_only=True, - ) - - -def test_freeze_rejects_prompt_binding_drift( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - monkeypatch.setattr( - freezer, - "_prompt_binding", - lambda: ( - {"contract_version": freezer.EXPECTED_CONTRACT}, - "9" * 64, - ), - ) - with pytest.raises( - freezer.Phase1FreezeError, - match="run manifest does not match its launch", - ): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_freeze_rejects_runtime_binding_drift( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - monkeypatch.setattr( - freezer, - "validate_runtime_stack", - lambda _root: { - **RUNTIME, - "python": "3.13.15", - }, - ) - with pytest.raises( - freezer.Phase1FreezeError, - match="run manifest does not match its launch", - ): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_freeze_rejects_a_summary_not_reproduced_from_attempts( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - path = ( - external - / "studies" - / freezer.EXPECTED_STUDY - / "hypotheses" - / "RC-H05" - / "arms" - / "qwen-on-60000" - / "summary.json" - ) - summary = json.loads(path.read_text()) - summary["attempts"] = 31 - path.write_text(json.dumps(summary)) - with pytest.raises( - freezer.Phase1FreezeError, - match="summary does not reproduce", - ): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_source_drift_outside_release_is_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - monkeypatch.setattr( - freezer, - "_changed_paths", - lambda _root: {repo / "rl" / "unexpected.py"}, - ) - with pytest.raises(freezer.Phase1FreezeError, match="source worktree differs"): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_protocol_shape_rejects_a_non_60k_arm() -> None: - protocol = _protocol() - protocol["arms"]["qwen-on-60000"]["max_output_tokens"] = 59999 - with pytest.raises(freezer.Phase1FreezeError, match="Qwen 60k arm changed"): - freezer._validate_protocol_shape(protocol) - - -def test_release_manifest_is_finite_json( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - release = json.loads( - ( - repo - / freezer.PHASE1_RELATIVE_ROOT - / "release_manifest.json" - ).read_text() - ) - assert release["runtime"] == RUNTIME - assert release["task_ids"] == list(freezer.EXPECTED_TASK_IDS) - assert [run["expected_attempts"] for run in release["runs"]] == [32, 32] - assert release["certification"] == { - "programs_reexecuted_in_bound_sandbox": 64, - "prompt_token_counts_recomputed_offline": True, - "published_programs_are_successful_executables_only": True, - "stored_and_replayed_metrics_must_match": True, - } - - -def test_freeze_rejects_prompt_token_count_that_does_not_reproduce( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, external = _prepare(tmp_path, monkeypatch) - monkeypatch.setattr( - freezer, - "_expected_prompt_lengths", - lambda _protocol, tasks: { - (arm_id, task.task_id): 101 - for arm_id in freezer.EXPECTED_HYPOTHESES.values() - for task in tasks - }, - ) - with pytest.raises( - freezer.Phase1FreezeError, - match="non-reproducible token accounting", - ): - freezer.freeze_phase1( - repo_root=repo, - external_root=external, - expected_source_sha=SOURCE_SHA, - ) - - -def test_replay_comparison_rejects_metric_drift() -> None: - task = _task(1) - evidence = freezer._AttemptEvidence( - row={ - "hypothesis_id": "RC-H05", - "arm_id": "qwen-on-60000", - "task_id": "F1", - "attempt_index": 1, - }, - program_row={ - "program": "print('test')", - "program_sha256": hashlib.sha256( - b"print('test')" - ).hexdigest(), - "reference_sha256": task.reference.target_image_sha256, - }, - task=task, - stored_evaluation={ - "status": "ok", - "attribution": "model", - "pure_executable": True, - "measurement_available": True, - "measured_iou": 0.25, - "measured_dice": 0.4, - "aggregation_iou": 0.25, - "violations": [], - "render_sha256": "e" * 64, - }, - ) - replayed = EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=0.26, - dice=0.4, - metrics={ - "render_sha256": "e" * 64, - "reference_sha256": task.reference.target_image_sha256, - }, - ) - with pytest.raises(freezer.Phase1FreezeError, match="replay IoU differs"): - freezer._compare_replayed_evaluation(evidence, replayed) - - -def test_failed_fallback_text_is_not_published_as_program_source() -> None: - task = _task(1) - private_fallback = "private reasoning without an answer" - evidence = freezer._AttemptEvidence( - row={ - "hypothesis_id": "RC-H05", - "arm_id": "qwen-on-60000", - "task_id": "F1", - "attempt_index": 1, - "status": "syntax_error", - }, - program_row={ - "program": private_fallback, - "program_sha256": hashlib.sha256( - private_fallback.encode() - ).hexdigest(), - "reference_sha256": task.reference.target_image_sha256, - }, - task=task, - stored_evaluation={}, - ) - assert freezer._published_program_rows([evidence]) == [] - - -def test_validation_allows_a_descendant_commit_containing_only_evidence( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo = tmp_path / "repo" - output = repo / freezer.PHASE1_RELATIVE_ROOT - output.mkdir(parents=True) - monkeypatch.setattr(freezer, "_git_head", lambda _root: "b" * 40) - monkeypatch.setattr( - freezer.subprocess, - "run", - lambda *_args, **_kwargs: SimpleNamespace(returncode=0), - ) - monkeypatch.setattr( - freezer.subprocess, - "check_output", - lambda *_args, **_kwargs: ( - b"data/training/representation-curriculum-v2/" - b"phase1/attempts.jsonl\0" - ), - ) - freezer._require_launch_source_state( - repo_root=repo, - output_root=output, - expected_source_sha=SOURCE_SHA, - validate_only=True, - ) - - -def test_validation_rejects_source_drift_after_launch( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo = tmp_path / "repo" - output = repo / freezer.PHASE1_RELATIVE_ROOT - output.mkdir(parents=True) - monkeypatch.setattr(freezer, "_git_head", lambda _root: "b" * 40) - monkeypatch.setattr( - freezer.subprocess, - "run", - lambda *_args, **_kwargs: SimpleNamespace(returncode=0), - ) - monkeypatch.setattr( - freezer.subprocess, - "check_output", - lambda *_args, **_kwargs: b"rl/common/prompt.py\0", - ) - with pytest.raises( - freezer.Phase1FreezeError, - match="changes source beyond", - ): - freezer._require_launch_source_state( - repo_root=repo, - output_root=output, - expected_source_sha=SOURCE_SHA, - validate_only=True, - ) - - -def test_replay_rejects_a_different_immutable_sandbox( - monkeypatch: pytest.MonkeyPatch, -) -> None: - class WrongSandboxEvaluator: - def __init__(self, **_kwargs: Any) -> None: - self.execution_boundary = SimpleNamespace( - image_ref="sha256:" + "1" * 64, - image_id="sha256:" + "1" * 64, - ) - - def __enter__(self) -> WrongSandboxEvaluator: - return self - - def __exit__(self, *_args: object) -> None: - pass - - def evaluate_batch(self, _requests: Any) -> Any: - raise AssertionError("sandbox mismatch must precede evaluation") - - monkeypatch.setattr( - freezer, - "PixCellEvaluator", - WrongSandboxEvaluator, - ) - with pytest.raises( - freezer.Phase1FreezeError, - match="certification sandbox differs", - ): - freezer._replay_attempts( - [], - expected_sandbox=SANDBOX, - ) diff --git a/rl/studies/representation_curriculum_v2/tests/test_v2_protocol.py b/rl/studies/representation_curriculum_v2/tests/test_v2_protocol.py deleted file mode 100644 index b0e038d5..00000000 --- a/rl/studies/representation_curriculum_v2/tests/test_v2_protocol.py +++ /dev/null @@ -1,201 +0,0 @@ -from __future__ import annotations - -import json -import os -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.evaluation import tasks as task_loader -from rl.evaluation.protocol import jobs_for_wave, load_protocol -from rl.evaluation.tasks import ( - build_benchmark_task_manifest, - load_task_manifest, -) -from rl.studies.representation_curriculum_v2 import ( - audit_context as context_auditor, -) -from rl.studies.representation_curriculum_v2 import ( - run_baseline as active_runner, -) - - -REPO_ROOT = Path(__file__).resolve().parents[4] -STUDY_ROOT = REPO_ROOT / "rl/studies/representation_curriculum_v2" -PROTOCOL_FILE = STUDY_ROOT / "protocol.json" -TASK_MANIFEST_FILE = STUDY_ROOT / "task_manifest.json" - - -def _protocol() -> dict: - return load_protocol(REPO_ROOT, protocol_file=PROTOCOL_FILE) - - -def _task_ids() -> list[str]: - manifest = load_task_manifest(TASK_MANIFEST_FILE, repo_root=REPO_ROOT) - return [entry["task_id"] for entry in manifest["task_sets"]["f1_f8"]] - - -def test_v2_is_only_two_direct_policy_zero_shot_baselines(): - protocol = _protocol() - assert protocol["study_id"] == "representation-curriculum-v2" - assert protocol["contract_version"] == "pixcell-direct-reconstruction-v2" - assert protocol["purpose"] == { - "kind": "direct-policy-zero-shot-baseline", - "operating_point_selection": False, - "training": False, - } - assert set(protocol["hypotheses"]) == {"RC-H05", "RC-H06"} - assert "dataset" not in protocol - assert protocol["hypotheses"]["RC-H05"]["arm_ids"] == ["qwen-on-60000"] - assert protocol["hypotheses"]["RC-H06"]["arm_ids"] == [ - "inkling-e09-60000" - ] - assert "operating_point_selection" not in protocol - assert "future_hypothesis_registry" not in protocol - - -def test_v2_model_and_sampling_bindings_are_exact(): - protocol = _protocol() - assert protocol["arms"] == { - "inkling-e09-60000": { - "context_tokens": 65536, - "max_image_long_edge": 1920, - "max_output_tokens": 60000, - "model": "thinkingmachines/Inkling", - "provider": "tinker", - "renderer": "tml_v0", - "thinking": True, - "thinking_effort": 0.9, - }, - "qwen-on-60000": { - "context_tokens": 65536, - "max_image_long_edge": 1920, - "max_output_tokens": 60000, - "model": "Qwen/Qwen3.6-35B-A3B", - "provider": "tinker", - "renderer": "qwen3_5", - "thinking": True, - "thinking_effort": None, - }, - } - assert protocol["sampling"] == { - "evaluator_workers": 8, - "require_candidate_isolation": True, - "sampling_concurrency": 8, - "seed_namespace": ( - "pixcell-representation-curriculum-v2-direct-baselines" - ), - "temperature": 1.0, - "top_p": 1.0, - } - assert protocol["launch"]["confirmation_token"] == "PIXCELL_BASELINE_V2" - - -def test_v2_uses_the_exact_frozen_f1_f8_tasks( - monkeypatch: pytest.MonkeyPatch, -): - active = json.loads(TASK_MANIFEST_FILE.read_text(encoding="utf-8")) - canonical = build_benchmark_task_manifest(REPO_ROOT) - assert active == canonical - assert ( - active["logical_sha256"] - == "05ca50526df05a3a79aa785504e36c7538106e5abdb72b64e86b20cc6f94a890" - ) - - def reject_depth_read(_root: Path) -> list[dict]: - raise AssertionError("v2 benchmark manifest must not read depth rows") - - monkeypatch.setattr( - task_loader, - "_read_depth_validation_rows", - reject_depth_read, - ) - assert _task_ids() == [f"F{index}" for index in range(1, 9)] - - -@pytest.mark.parametrize("hypothesis_id", ("RC-H05", "RC-H06")) -def test_v2_smoke_is_attempt_one_and_complete_is_best_at_four( - hypothesis_id: str, -): - protocol = _protocol() - task_ids = _task_ids() - smoke = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name="smoke", - task_ids=task_ids, - ) - complete = jobs_for_wave( - protocol, - hypothesis_id=hypothesis_id, - wave_name="complete", - task_ids=task_ids, - ) - assert len(smoke) == 8 - assert {job.attempt_index for job in smoke} == {1} - assert len(complete) == 32 - assert {job.attempt_index for job in complete} == {1, 2, 3, 4} - assert set(smoke).issubset(set(complete)) - - -def test_exact_renderer_contexts_fit_without_a_remote_client(): - pytest.importorskip("tinker") - pytest.importorskip("tml_renderers") - report = context_auditor.audit_context() - assert report["remote_client_created"] is False - assert report["summaries"] == { - "inkling-e09-60000": { - "task_count": 8, - "max_prompt_tokens": 4159, - "min_headroom_tokens": 1377, - }, - "qwen-on-60000": { - "task_count": 8, - "max_prompt_tokens": 5357, - "min_headroom_tokens": 179, - }, - } - - -def test_context_audit_forces_offline_assets_and_restores_environment( - monkeypatch: pytest.MonkeyPatch, -): - monkeypatch.delenv("HF_HUB_OFFLINE", raising=False) - monkeypatch.setenv("TRANSFORMERS_OFFLINE", "previous") - with context_auditor._offline_model_assets(): - assert os.environ["HF_HUB_OFFLINE"] == "1" - assert os.environ["TRANSFORMERS_OFFLINE"] == "1" - assert "HF_HUB_OFFLINE" not in os.environ - assert os.environ["TRANSFORMERS_OFFLINE"] == "previous" - - -def test_active_runner_binds_v2_and_accepts_no_selection( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, - capsys: pytest.CaptureFixture[str], -): - captured: dict[str, object] = {} - - async def fake_run_baseline(**kwargs): - captured.update(kwargs) - return {"summaries": {}} - - class FakeParser: - @staticmethod - def parse_args(): - return SimpleNamespace( - hypothesis="RC-H05", - wave="smoke", - expected_source_sha="a" * 40, - confirm_spend="PIXCELL_BASELINE_V2", - external_root=tmp_path, - ) - - monkeypatch.setattr(active_runner, "run_baseline", fake_run_baseline) - monkeypatch.setattr(active_runner, "parser", FakeParser) - active_runner.main() - assert captured["protocol_file"] == PROTOCOL_FILE - assert captured["hypothesis_id"] == "RC-H05" - assert "selection_path" not in captured - assert json.loads(capsys.readouterr().out) == {} diff --git a/rl/studies/representation_training_v1/README.md b/rl/studies/representation_training_v1/README.md deleted file mode 100644 index 70db5ea6..00000000 --- a/rl/studies/representation_training_v1/README.md +++ /dev/null @@ -1,384 +0,0 @@ -# Representation training v1 - -This study owns the first training campaign over the released PixCell -curriculum. It compares five Qwen learning paths and one bounded Inkling -policy-improvement path while keeping the model input, evaluator, dataset, and -F1–F8 benchmark fixed. - -No command in this directory runs at import time. A paid launch requires the -committed protocol, an exact clean source commit, the pinned runtime, the -immutable candidate sandbox, valid Tinker and W&B identities, an external -ledger, and the explicit `PIXCELL_TRAINING_V1` confirmation. - -## Branches - -```text -Base Qwen -├── qwen-base-rl-l0 → l1 → l2 → l3 → l4 -├── qwen-mixed-sft -│ └── qwen-mixed-rl-l{first unmastered ... L4} -└── qwen-l0-sft - ├── qwen-sequential-sft-l1 → l2 → l3 → l4 - └── qwen-l0-rl-l1 → l2 → l3 → l4 - -Base Inkling -└── inkling-l4-rl -``` - -The pure-RL branch tests whether verifier-only learning can acquire the full -primitive-to-component curriculum directly from base Qwen. The L0 SFT -checkpoint is shared unchanged by the sequential-SFT and -composition-by-RL hypotheses. Mixed SFT supplies the checkpoint for the -strongest-open-model hypothesis. Each new SFT level loads its parent weights -with a fresh optimizer and restarts the linear schedule; that reset is part of -the sequential-SFT hypothesis. The fixed L0→RL path always enters at L1 after -its reviewed L0 checkpoint; it is not blocked by an invented mastery threshold. -The mixed-SFT path evaluates L0–L4 and enters RL at the first level below the -protocol-sealed thresholds of 0.80 mean raw IoU or 0.95 pure-executable rate. - -Each Qwen RL level is one optimizer run with sealed ceilings at steps 1, 5, 10, -15, 20, 25, and 30. In the pure-RL branch, L0 begins directly from base Qwen. -Its step-1 smoke establishes rollout health, and an immutable rollout-health -receipt is required before step 5. Starting at step 5, progress is measured on -the same fixed held-out panel: realization slot 6, with exactly one row for -every representation at the current level. A report from that panel gates each -later ceiling. Entry into each pure-RL L1–L4 stage requires two reports from -the exact preceding-level terminal checkpoint: its prior-level slot-6 report -and its pre-update slot-6 baseline on the new level. Both reports are bound -into one approval and canonically recomputed at launch. F1–F8 remains a -periodic transfer report and never gates this curriculum. - -The L0-SFT→RL branch uses the same rollout-health and fixed slot-6 cadence from -L1 onward. Its L1 entry is intentionally different: the L0-SFT checkpoint was -produced by an earlier frozen source, so smoke requires its exact parent -receipt plus a reviewed source transition. It does not require the obsolete -1,092-row depth report, F1–F8, or a fabricated same-source level-panel record. -L2–L4 entry requires the preceding completed level's slot-6 report. - -In the pure-RL branch, each level first evaluates its actual parent policy on -that level's slot-6 panel, then reuses the identical tasks after steps 5, 10, -15, 20, 25, and 30. For L0, that pre-update parent is base Qwen and the panel -has 93 tasks. For L1 through L4, it is the preceding level's completed -checkpoint evaluated on the new level before its first update. This separates -learning from changes in sampled training-task difficulty. Final checkpoint -selection uses the separate 546-row slot-7 panel across L0–L4. Inkling uses -sealed ceilings at steps 1, 6, and 14. These are continuation waves, not -independently initialized runs. - -The 21 exact stage definitions, parent edges, wave ceilings, models, recipes, -dataset release, and evaluation identities are in -[`protocol.json`](protocol.json). - -## Policy input and objective - -Every policy receives the established direct Phase-A interface: - -1. the maximum-visibility component image; -2. the physical footprint; -3. black/white layer semantics; -4. the permitted extended primitive catalogue; -5. the direct one-shot programming contract. - -The prompt is assembled by [`rl/common/prompt.py`](../../common/prompt.py). -The image transform is owned by -[`rl/common/preprocess.py`](../../common/preprocess.py). Calibration, -`target_image`, reference code, representation labels, split metadata, and IoU -remain verifier-only. - -Qwen SFT places loss on exactly the Python program and the assistant end token. -Prompt tokens, image tokens, and the empty thinking header have zero weight. -RL receives no answer program and optimizes raw absolute-scale IoU. Model -failures receive zero; reference or evaluator failures abort rather than -becoming negative examples. Ports, counts, and structural witnesses remain -diagnostics. - -## Frozen recipes - -| Policy | Training | Important settings | -|---|---|---| -| Qwen | mixed or level SFT | Qwen3.6 35B A3B, thinking renderer, LoRA 32, batch 64, one physical pass, Adam `1e-4`, linear decay | -| Qwen | curriculum RL | G4, 8 groups per step, `1e-5`, importance sampling, KL 0, 20% earlier-level replay, 30 steps per level | -| Inkling | advanced L4 RL | native TMLv0, effort 0.9, G4, 8 groups per step, `1e-5`, importance sampling, KL 0 | - -Qwen and Inkling both use a 1920-pixel longest edge, temperature 1, top-p 1, -and a 60,000-token output ceiling inside the 65,536-token context. The pinned -Cookbook exposes temperature but not top-p in its RL config; the pinned Tinker -SDK default of top-p 1 is asserted by a contract test. Every prompt is rendered -locally before launch and must retain positive context headroom. - -SFT stages run continuously to their complete one-pass checkpoint. They are -not stopped after a one-step smoke because the pinned Cookbook finalizes the -SFT epoch state when `max_steps` is reached. Initial health is confirmed from -the first local/W&B telemetry while the same run continues. - -RL has no built-in sampling validation. This prevents a one-step 60k smoke -from silently creating dozens of extra completions. F1–F8 and held-out depth -evaluation are explicit checkpoint jobs. - -Tinker reuses the mutable alias `final` when a run continues. The study never -uses that alias as an RL promotion identity. Every wave receipt instead binds -the unique numeric checkpoint at that exact ceiling. SFT receipts inventory -the numeric quarter checkpoints and terminal `final`; all mixed-SFT inventory -checkpoints are evaluated on the full 1,092-row depth validation split. The -selected parent is the finite report with the highest representation-macro -mean IoU, then pure-executable rate, then later progress. - -## Records and W&B - -Set `PIXCELL_TRAINING_ROOT` to storage outside the repository. The external -ledger is authoritative: - -```text -$PIXCELL_TRAINING_ROOT/ -└── studies/representation-training-v1/ - └── stages//runs// - ├── run_manifest.json - ├── invocations/ - ├── candidates/ - ├── waves/ - ├── tinker/ - └── wandb/ -``` - -Each manifest binds the source commit, protocol, prompt assets, all frozen -dataset artifacts, runtime and Cookbook checkout, sandbox image, exact finite -task schedule, model, recipe, parent checkpoint, and tracking identity. -Candidate records retain the exact sampled token IDs, raw response, separated -reasoning/final channels, extracted program, stop state, and deterministic -measurement. A crash retry gets a new invocation ID, so it cannot overwrite -earlier paid samples. - -`r0-retry1` is reserved for the L0-SFT→L1–L4 RL path after its original `r0` -smoke stopped before Tinker client creation. The original run has no samples, -checkpoints, metrics, or wave receipt and remains excluded from results. The -retry consumes the exact historical `qwen-l0-sft/r0` checkpoint; it is an -operational identity, not a second statistical replicate. - -W&B is the live observability mirror: - -- entity: `aadarwal-massachusetts-institute-of-technology` -- project: `pixcell-representation-training-v1` -- name: `representation-training-v1//@` -- group: `representation-training-v1/qwen-open` or - `representation-training-v1/inkling` -- tags: stage, model, kind, replicate, and hypothesis IDs - -The run ID is deterministic in source commit, protocol digest, stage, and -replicate. RL continuation uses `WANDB_RESUME=must`; a fresh stage uses -`WANDB_RESUME=never`. Git and source-code upload are disabled. -Checkpoint and fixed level-panel evaluations are authoritative external-ledger -jobs, not separate W&B training runs. W&B mirrors live optimizer runs; the -immutable evaluation records supply checkpoint-selection and promotion -evidence. - -## Zero-spend gate - -After this source is committed, run: - -```bash -PIXCELL_EVALUATOR_IMAGE='' \ -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.audit_phase0 \ - --expected-source-sha "$(git rev-parse HEAD)" -``` - -It checks all four initial roots: - -- `qwen-mixed-sft` over all 3,468 depth training rows; -- `qwen-l0-sft` over all 642 L0 training rows; -- `qwen-base-rl-l0/smoke` over the 642 L0 training rows; -- `inkling-l4-rl/smoke` over the 648 L4 training rows. - -The audit makes no model request. It hashes every frozen release artifact, -tokenizes every selected policy input, verifies every source and reference, -seals the exact task batches, checks the installed Cookbook source commit, and -executes a known-good program inside the immutable sandbox. - -## Initial start order - -The first paid starts may run in parallel: - -1. start `qwen-base-rl-l0/smoke` directly from base Qwen. Its first 8×G4 - rollout batch is the untouched policy's on-policy starting measurement, - immediately followed by one optimizer update; -2. start `qwen-mixed-sft/complete`, confirm its first metric and local - checkpoint activity without stopping it; -3. start `qwen-l0-sft/complete`, confirm the same; -4. start `inkling-l4-rl/smoke`, which ends after one update; -5. evaluate selected checkpoints only when the result will govern a - continuation or final comparison. - -The optional `level-progress-l0` base panel remains available for reporting, -but it does not gate the direct RL branch. - -The guarded launcher exposes no model, data, optimizer, token, schedule, or -W&B override: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.run_stage \ - --stage \ - --wave \ - --replicate r0 \ - --expected-source-sha "$(git rev-parse HEAD)" \ - --confirm-spend PIXCELL_TRAINING_V1 -``` - -Dependent stages fail closed until their parent checkpoint and required -evaluation or promotion receipt exist. Repeating a completed command verifies -its immutable receipt and exits without touching Tinker or W&B. - -## Evaluation and promotion - -The progress panel is one attempt on each F1–F8 figure. The depth panel is the -full 1,092-row held-out-parameter split. The Inkling promotion panel contains -36 L4 anchors, six anchors from each earlier level, and F1–F8. Reference -rasters, calibration, labels, and target code never enter model messages. - -Evaluate one checkpoint named in its wave receipt: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.evaluate_checkpoint \ - --repo-root "$(pwd)" \ - --stage \ - --wave \ - --checkpoint-name \ - --replicate r0 \ - --panel \ - --source-git-sha "$(git rev-parse HEAD)" \ - --wave-receipt-record-sha256 \ - --external-root "$PIXCELL_TRAINING_ROOT" \ - --confirm-spend PIXCELL_TRAINING_V1 -``` - -The final F1–F8 benchmark is a separate, reporting-only v2 panel; it does not -change progress, depth, or promotion records. It makes four independently -seeded requests per figure (32 total), resumes only missing attempts, and -reports mean@1 plus per-figure and aggregate best@4. Its records live under -`evaluations///final-benchmark-best4-v2/`: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.evaluate_final_benchmark \ - --repo-root "$(pwd)" \ - --stage \ - --wave \ - --checkpoint-name \ - --replicate r0 \ - --source-git-sha "$(git rev-parse HEAD)" \ - --wave-receipt-record-sha256 \ - --external-root "$PIXCELL_TRAINING_ROOT" \ - --confirm-spend PIXCELL_TRAINING_V1 -``` - -The mixed-SFT renderer postmortem is a fixed, non-canonical diagnostic. It -samples only the final checkpoint with `qwen3_5_disable_thinking` and steps -14, 28, and 42 with `qwen3_5`, once each on F1–F8. Its 32 rows live under -`diagnostics/qwen-sft-renderer-diagnostic-v1/`, use a diagnostic-only schema, -and are explicitly ineligible for checkpoint promotion. Sampling the three -periodic checkpoints is a deliberate postmortem-only exception to the study's -terminal-checkpoint evaluation rule: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.evaluate_renderer_diagnostic \ - --repo-root "$(pwd)" \ - --evaluator-source-git-sha "$(git rev-parse HEAD)" \ - --producer-wave-receipt-record-sha256 \ - --external-root "$PIXCELL_TRAINING_ROOT" \ - --confirm-spend PIXCELL_TRAINING_V1 -``` - -After reviewing that immutable report, bind it to exactly one dependent wave: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.record_approval \ - --stage \ - --wave \ - --replicate r0 \ - --evaluation-stage \ - --evaluation-wave \ - --evaluation-checkpoint \ - --evaluation-report-record-sha256 \ - --expected-source-sha "$(git rev-parse HEAD)" \ - --confirm-approval PIXCELL_APPROVE_TRAINING_V1 -``` - -Approvals bind the report, source commit, protocol digest, task panel, -checkpoint inventory, training receipt, and selection decision. They cannot be -reused for a different wave or checkpoint. The launcher treats the stored -approval as an index and canonically revalidates its bound report and metrics -before every paid continuation. - -Pure-RL L1–L4 entry evaluates the same parent terminal checkpoint twice: once -on the preceding level's slot-6 panel and once on the new level's slot-6 panel. -Pass the preceding-level report as the primary evaluation above and the -new-level report as: - -```bash - --entry-baseline-report-record-sha256 -``` - -The combined approval fails unless both reports resolve to the same parent -stage, wave, checkpoint, checkpoint inventory, run manifest, and wave receipt. - -The L0-SFT→L1-RL smoke is the one receipt-only exception. It does not invent or -wait for an evaluation report. At the clean consumer commit, explicitly review -and enumerate every path changed since the source commit in the immutable -L0-SFT manifest: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.record_parent_transition \ - --replicate r0 \ - --expected-source-sha "$(git rev-parse HEAD)" \ - --approve-changed-path \ - --approve-changed-path \ - --confirm-approval PIXCELL_APPROVE_TRAINING_V1 -``` - -This approval recomputes the parent manifest, terminal receipt, invocation, -checkpoint inventory, local artifact hashes, unchanged prompt, dataset, model, -SFT recipe and programming contract, plus the full ancestor-to-consumer Git -transition. The L1 smoke launcher revalidates all of it before W&B or Tinker. - -An evaluation must always run from the same clean commit as its training -manifest. To adopt that already-reviewed evidence from a later descendant, -record a v2 approval at the clean consumer commit and explicitly enumerate the -complete reviewed Git path scope: - -```bash -PYTHONPATH=src:. python -m \ - rl.studies.representation_training_v1.record_approval \ - --stage \ - --wave \ - --replicate r0 \ - --evaluation-stage \ - --evaluation-wave \ - --evaluation-checkpoint \ - --evaluation-report-record-sha256 \ - --evidence-source-sha \ - --expected-source-sha "$(git rev-parse HEAD)" \ - --approve-changed-path \ - --approve-changed-path \ - --confirm-approval PIXCELL_APPROVE_TRAINING_V1 -``` - -Every path from `git diff --name-status --no-renames HEAD` must -appear exactly once. The approval records both commit trees, ancestry, and a -plumbing-level inventory of every path, status, mode, and old/new object ID. -It also binds the producer report and evaluation manifest, training receipt -and checkpoint inventory, and all protocol, prompt, dataset, model/renderer, -sampling, and sandbox evidence. The dependent child manifest embeds that one -stage/wave/replicate transition. A same-source approval remains byte-for-byte -v1 and must omit both transition flags. - -## Baseline completion - -Attempts two through four of the zero-shot Qwen and Inkling F1–F8 baselines -remain the separate `representation-curriculum-v2` study. They are evaluation -only and do not block training. They may resume in the background with that -study's `complete` wave, but local Docker evaluation should not contend with a -live RL rollout unless a host-wide worker budget is in place. diff --git a/rl/studies/representation_training_v1/__init__.py b/rl/studies/representation_training_v1/__init__.py deleted file mode 100644 index 9a7298b9..00000000 --- a/rl/studies/representation_training_v1/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Sealed post-training study over the PixCell representation curriculum.""" diff --git a/rl/studies/representation_training_v1/audit_phase0.py b/rl/studies/representation_training_v1/audit_phase0.py deleted file mode 100644 index ae36d890..00000000 --- a/rl/studies/representation_training_v1/audit_phase0.py +++ /dev/null @@ -1,82 +0,0 @@ -#!/usr/bin/env python3 -"""Run every initial training preflight without credentials or model spend.""" - -from __future__ import annotations - -import argparse -import json -from pathlib import Path -from typing import Any - -from .preflight import stage_preflight -from .protocol import ( - INITIAL_STAGE_IDS, - load_protocol, - repository_root, - stage_spec, - validate_source_sha, -) - - -def run_audit( - *, - repo_root: Path, - expected_source_sha: str | None = None, -) -> dict[str, Any]: - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol( - root, - require_committed=expected_source_sha is not None, - ) - source_git_sha = ( - validate_source_sha(root, expected_source_sha) - if expected_source_sha is not None - else None - ) - reports: dict[str, Any] = {} - for stage_id in INITIAL_STAGE_IDS: - stage = stage_spec(protocol, stage_id) - wave = min( - stage.waves, - key=lambda name: int(stage.waves[name]["max_steps"]), - ) - reports[stage_id] = stage_preflight( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name=wave, - require_sandbox=stage.kind == "rl", - ) - return { - "schema_version": "pixcell-representation-training-phase0-audit-v1", - "study_id": protocol["study_id"], - "source_git_sha": source_git_sha, - "model_client_created": False, - "paid_request_made": False, - "initial_stages": reports, - } - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, default=repository_root()) - result.add_argument("--expected-source-sha") - return result - - -def main() -> None: - args = parser().parse_args() - print( - json.dumps( - run_audit( - repo_root=args.repo_root, - expected_source_sha=args.expected_source_sha, - ), - indent=2, - sort_keys=True, - ) - ) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/dataset_binding.py b/rl/studies/representation_training_v1/dataset_binding.py deleted file mode 100644 index f9ee50d8..00000000 --- a/rl/studies/representation_training_v1/dataset_binding.py +++ /dev/null @@ -1,63 +0,0 @@ -"""Frozen dataset provenance shared by launch and promotion gates.""" - -from __future__ import annotations - -import hashlib -import json -from pathlib import Path -from typing import Any - -from rl.common.dataset_io import load_rows, parquet_paths - -from .protocol import file_sha256 - - -def _dataset_binding(dataset_root: Path, protocol: dict[str, Any]) -> dict[str, Any]: - freeze_path = dataset_root / "depth-v1" / "factory" / "freeze.json" - freeze = json.loads(freeze_path.read_text(encoding="utf-8")) - expected = protocol["dataset"]["logical_release_sha256"] - if freeze.get("logical_release_sha256") != expected: - raise RuntimeError("local depth release differs from the training protocol") - train = load_rows(dataset_root, configuration="depth", split="train") - validation = load_rows(dataset_root, configuration="depth", split="validation") - if ( - len(train) != protocol["dataset"]["train_rows"] - or len(validation) != protocol["dataset"]["validation_rows"] - ): - raise RuntimeError("local depth row counts differ from the training protocol") - reference_artifacts = freeze.get("reference_artifacts") - if not isinstance(reference_artifacts, dict) or not reference_artifacts: - raise RuntimeError("depth freeze has no reference artifact inventory") - observed_artifacts: dict[str, str] = {} - release_root = freeze_path.parents[1] - for relative, expected_sha in sorted(reference_artifacts.items()): - path = (release_root / relative).resolve(strict=True) - if not path.is_relative_to(release_root.resolve()): - raise RuntimeError("depth reference artifact escaped its release") - observed = file_sha256(path) - if observed != expected_sha: - raise RuntimeError(f"depth reference artifact changed: {relative}") - observed_artifacts[str(relative)] = observed - return { - "logical_release_sha256": expected, - "freeze_file_sha256": file_sha256(freeze_path), - "train_rows": len(train), - "validation_rows": len(validation), - "parquet_shards": { - str(path.relative_to(dataset_root)): file_sha256(path) - for split in ("train", "validation") - for path in parquet_paths( - dataset_root, - configuration="depth", - split=split, - ) - }, - "reference_artifacts_sha256": hashlib.sha256( - json.dumps( - observed_artifacts, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - ).hexdigest(), - "reference_artifact_count": len(observed_artifacts), - } diff --git a/rl/studies/representation_training_v1/evaluate_checkpoint.py b/rl/studies/representation_training_v1/evaluate_checkpoint.py deleted file mode 100644 index 789e3d72..00000000 --- a/rl/studies/representation_training_v1/evaluate_checkpoint.py +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env python3 -"""CLI entry point for the sealed representation-training checkpoint evaluator.""" - -from rl.studies.representation_training_v1.evaluation import main - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/evaluate_final_benchmark.py b/rl/studies/representation_training_v1/evaluate_final_benchmark.py deleted file mode 100644 index b44a65f4..00000000 --- a/rl/studies/representation_training_v1/evaluate_final_benchmark.py +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env python3 -"""CLI entry point for the versioned final-checkpoint best@4 evaluator.""" - -from rl.studies.representation_training_v1.final_benchmark import main - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/evaluate_renderer_diagnostic.py b/rl/studies/representation_training_v1/evaluate_renderer_diagnostic.py deleted file mode 100644 index 52027426..00000000 --- a/rl/studies/representation_training_v1/evaluate_renderer_diagnostic.py +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env python3 -"""CLI for the sealed, non-canonical Qwen SFT renderer diagnostic.""" - -from rl.studies.representation_training_v1.renderer_diagnostic import main - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/evaluation.py b/rl/studies/representation_training_v1/evaluation.py deleted file mode 100644 index 01feb81a..00000000 --- a/rl/studies/representation_training_v1/evaluation.py +++ /dev/null @@ -1,2219 +0,0 @@ -"""Receipt-bound checkpoint evaluation for representation-training-v1. - -The evaluator is deliberately narrower than the general baseline runner: - -* the sampler checkpoint can only come from an immutable training-wave receipt; -* the model, renderer, effort, image limit, context, and sampling policy come - from the committed study protocol; -* every prompt and verifier reference is checked before a Tinker client exists; -* sampled output is persisted before execution, so an infrastructure fault can - be retried without paying for another completion; and -* candidate execution always uses the pinned isolation boundary. -""" - -from __future__ import annotations - -import argparse -import asyncio -import errno -import fcntl -import hashlib -import json -import math -import os -import re -import stat -import subprocess -import uuid -from collections import Counter, defaultdict -from collections.abc import Callable, Mapping, Sequence -from contextlib import AbstractContextManager -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any - -from rl.common.contracts import ( - ModelObservation, - observation_from_row, - reference_from_row, - sampler_from_row, -) -from rl.common.dataset_io import parquet_paths -from rl.common.evaluator import ( - Attribution, - EvaluationResult, - EvaluationStatus, - PixCellEvaluator, -) -from rl.common.prompt import build_prompt_text, prompt_asset_hashes -from rl.evaluation.tasks import ( - BENCHMARK_TASK_SET, - EvaluationTask, - canonical_json_sha256, - load_task_manifest, - load_task_set, -) -from rl.track_a.tinker_data import _message, _renderer - -from .dataset_binding import _dataset_binding -from .protocol import ( - STUDY_ID, - StageSpec, - file_sha256, - load_protocol, - protocol_path, - stage_spec, - validate_stage_replicate, - validate_source_sha, -) -from .store import StageKey, TrainingRecord, TrainingStore - - -EVALUATION_SCHEMA_VERSION = "pixcell-training-checkpoint-evaluation-v1" -EVALUATION_RECORD_SCHEMA_VERSION = "pixcell-training-evaluation-record-v1" -PANEL_PROGRESS = "progress" -PANEL_DEPTH_VALIDATION = "depth-validation" -PANEL_INKLING_PROMOTION = "inkling-promotion" -LEVEL_PROGRESS_PANELS = { - "L0": "level-progress-l0", - "L1": "level-progress-l1", - "L2": "level-progress-l2", - "L3": "level-progress-l3", - "L4": "level-progress-l4", -} -PANEL_DEPTH_FINAL_SELECTION = "depth-final-selection" -LEVEL_PROGRESS_ROWS = { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108, -} -_MODEL_EVALUATION_STATUSES = frozenset( - { - EvaluationStatus.OK.value, - EvaluationStatus.SYNTAX_ERROR.value, - EvaluationStatus.SOURCE_REJECTED.value, - EvaluationStatus.RUNTIME_TIMEOUT.value, - EvaluationStatus.RUNTIME_ERROR.value, - EvaluationStatus.NO_GDS.value, - EvaluationStatus.BAD_GDS.value, - } -) -PANELS = ( - PANEL_PROGRESS, - PANEL_DEPTH_VALIDATION, - PANEL_INKLING_PROMOTION, -) -SUPPORTED_PANELS = ( - *PANELS, - *LEVEL_PROGRESS_PANELS.values(), - PANEL_DEPTH_FINAL_SELECTION, -) -MAX_IMAGE_LONG_EDGE = 1920 -MAX_OUTPUT_TOKENS = 60_000 -TEMPERATURE = 1.0 -TOP_P = 1.0 -SAMPLE_CONCURRENCY = 8 -EVALUATION_BATCH_SIZE = 32 -EVALUATOR_WORKERS = 8 -_PANEL_SELECTION_SEED = ( - "pixcell-representation-training-v1-inkling-promotion-20260728" -) -_LEVEL_PROGRESS_SELECTION = "validation-realization-slot-6" -_DEPTH_FINAL_SELECTION = "validation-realization-slot-7" -_DEPTH_COLUMNS = ( - "id", - "level", - "image", - "target_image", - "footprint_um", - "representation_id", - "leakage_group_id", - "image_sha256", - "target_image_sha256", - "realization_slot", - "split_role", -) -_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") -_TINKER_PATH = re.compile(r"^tinker://[^\s]+$") -_SHA256 = re.compile(r"^[0-9a-f]{64}$") -_MAX_RECORD_BYTES = 32 * 1024 * 1024 -_FORBIDDEN_PROMPT_FRAGMENTS = ( - "DISPLAY NOTE", - "magnification x:y", - "px/um", - "target_image", - "calibration.json", -) - - -class CheckpointEvaluationError(RuntimeError): - """The requested checkpoint evaluation is not safe or reproducible.""" - - -class CheckpointReceiptError(CheckpointEvaluationError): - """The requested checkpoint is not bound to an exact training receipt.""" - - -class SamplingInfrastructureFault(CheckpointEvaluationError): - """Tinker sampling failed outside the model's generated program.""" - - -class ReferencePanelFault(CheckpointEvaluationError): - """A private verifier reference is invalid or changed.""" - - -class EvaluationInfrastructureFault(CheckpointEvaluationError): - """The isolated evaluator failed independently of model output.""" - - -class ImmutableEvaluationRecordError(CheckpointEvaluationError): - """A create-only evaluation record already contains different bytes.""" - - -@dataclass(frozen=True) -class CheckpointBinding: - key: StageKey - stage: StageSpec - wave: str - checkpoint_name: str - checkpoint: dict[str, Any] - checkpoint_inventory_sha256: str - sampler_path: str - run_manifest: TrainingRecord - wave_receipt: TrainingRecord - - -@dataclass(frozen=True) -class BaseModelBinding: - """An explicit base-model sampler identity with no fictitious receipt.""" - - key: StageKey - stage: StageSpec - wave: str - checkpoint_name: str - checkpoint: dict[str, Any] - checkpoint_inventory_sha256: str - sampler_path: str - binding_sha256: str - - -EvaluationBinding = CheckpointBinding | BaseModelBinding - - -@dataclass(frozen=True) -class PreparedTask: - task: EvaluationTask - prompt_tokens: int - prompt_text_sha256: str - reference_evidence: dict[str, Any] - - -def _canonical_bytes(value: Any) -> bytes: - try: - return json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise CheckpointEvaluationError( - "evaluation record is not finite JSON" - ) from exc - - -def _canonical_sha256(value: Any) -> str: - return hashlib.sha256(_canonical_bytes(value)).hexdigest() - - -def _safe_json(value: Any) -> Any: - if value is None or isinstance(value, (str, int, bool)): - return value - if isinstance(value, float): - if not math.isfinite(value): - raise EvaluationInfrastructureFault( - "evaluator returned a non-finite number" - ) - return value - if isinstance(value, Mapping): - return {str(key): _safe_json(item) for key, item in value.items()} - if isinstance(value, (list, tuple)): - return [_safe_json(item) for item in value] - if hasattr(value, "item"): - return _safe_json(value.item()) - return str(value) - - -def _record_document( - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], -) -> dict[str, Any]: - normalized = json.loads(_canonical_bytes(payload)) - document = { - "schema_version": EVALUATION_RECORD_SCHEMA_VERSION, - "record_type": record_type, - "key": dict(key), - "payload": normalized, - "payload_sha256": _canonical_sha256(normalized), - } - document["record_sha256"] = _canonical_sha256(document) - return document - - -def _validate_record( - value: Mapping[str, Any], - *, - record_type: str, - key: Mapping[str, Any], -) -> dict[str, Any]: - required = { - "schema_version", - "record_type", - "key", - "payload", - "payload_sha256", - "record_sha256", - } - if set(value) != required: - raise CheckpointEvaluationError("evaluation record schema changed") - if value["schema_version"] != EVALUATION_RECORD_SCHEMA_VERSION: - raise CheckpointEvaluationError("evaluation record has a foreign schema") - if value["record_type"] != record_type or value["key"] != dict(key): - raise CheckpointEvaluationError( - "evaluation record key differs from its canonical path" - ) - payload = value["payload"] - if not isinstance(payload, dict): - raise CheckpointEvaluationError("evaluation record payload is not an object") - if value["payload_sha256"] != _canonical_sha256(payload): - raise CheckpointEvaluationError("evaluation payload digest mismatch") - unsigned = dict(value) - observed = str(unsigned.pop("record_sha256")) - if observed != _canonical_sha256(unsigned) or not _SHA256.fullmatch(observed): - raise CheckpointEvaluationError("evaluation envelope digest mismatch") - return dict(value) - - -class _EvaluationLock(AbstractContextManager["_EvaluationLock"]): - def __init__(self, path: Path) -> None: - self.path = path - self.fd: int | None = None - - def __enter__(self) -> "_EvaluationLock": - self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - flags = os.O_RDWR | os.O_CREAT - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(self.path, flags, 0o600) - fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - except BlockingIOError as exc: - if "fd" in locals(): - os.close(fd) - raise CheckpointEvaluationError( - "this exact checkpoint panel is already active" - ) from exc - self.fd = fd - return self - - def __exit__(self, *_args: object) -> None: - if self.fd is not None: - fcntl.flock(self.fd, fcntl.LOCK_UN) - os.close(self.fd) - self.fd = None - - -class CheckpointEvaluationStore: - """Create-only records under one receipt-bound training stage.""" - - def __init__( - self, - *, - stage_path: Path, - wave: str, - checkpoint_name: str, - panel: str, - ) -> None: - if ( - not _IDENTIFIER.fullmatch(wave) - or not _IDENTIFIER.fullmatch(checkpoint_name) - or panel not in SUPPORTED_PANELS - ): - raise ValueError("invalid checkpoint-evaluation identity") - self.stage_path = stage_path.expanduser().resolve(strict=True) - requested = ( - self.stage_path - / "evaluations" - / wave - / checkpoint_name - / panel - ) - requested.mkdir(parents=True, exist_ok=True, mode=0o700) - resolved = requested.resolve(strict=True) - if resolved != requested or not resolved.is_relative_to(self.stage_path): - raise CheckpointEvaluationError( - "evaluation root must be a real directory inside its stage" - ) - self.root = resolved - - def acquire_lock(self) -> _EvaluationLock: - return _EvaluationLock(self.root / ".lock") - - @staticmethod - def _task_component(task_id: str) -> str: - if not _IDENTIFIER.fullmatch(task_id): - raise ValueError(f"unsafe task ID: {task_id!r}") - return task_id - - def load_manifest(self) -> dict[str, Any] | None: - return self._load( - PurePosixPath("manifest.json"), - record_type="evaluation_manifest", - key={}, - ) - - def create_or_verify_manifest( - self, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify( - PurePosixPath("manifest.json"), - record_type="evaluation_manifest", - key={}, - payload=payload, - ) - - def load_sample(self, task_id: str) -> dict[str, Any] | None: - component = self._task_component(task_id) - return self._load( - PurePosixPath("candidates", component, "sample.json"), - record_type="checkpoint_sample", - key={"task_id": task_id}, - ) - - def create_or_verify_sample( - self, - task_id: str, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - component = self._task_component(task_id) - return self._create_or_verify( - PurePosixPath("candidates", component, "sample.json"), - record_type="checkpoint_sample", - key={"task_id": task_id}, - payload=payload, - ) - - def load_evaluation(self, task_id: str) -> dict[str, Any] | None: - component = self._task_component(task_id) - return self._load( - PurePosixPath("candidates", component, "evaluation.json"), - record_type="checkpoint_evaluation", - key={"task_id": task_id}, - ) - - def create_or_verify_evaluation( - self, - task_id: str, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - component = self._task_component(task_id) - return self._create_or_verify( - PurePosixPath("candidates", component, "evaluation.json"), - record_type="checkpoint_evaluation", - key={"task_id": task_id}, - payload=payload, - ) - - def load_report(self) -> dict[str, Any] | None: - return self._load( - PurePosixPath("report.json"), - record_type="checkpoint_report", - key={}, - ) - - def create_or_verify_report( - self, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify( - PurePosixPath("report.json"), - record_type="checkpoint_report", - key={}, - payload=payload, - ) - - def _path(self, relative: PurePosixPath) -> Path: - path = self.root.joinpath(*relative.parts) - parent = path.parent.resolve(strict=False) - if parent != self.root and not parent.is_relative_to(self.root): - raise CheckpointEvaluationError("evaluation record escaped its root") - return path - - def _create_or_verify( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], - ) -> dict[str, Any]: - expected = _record_document( - record_type=record_type, - key=key, - payload=payload, - ) - path = self._path(relative) - path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - raw = _canonical_bytes(expected) + b"\n" - temporary = path.parent / f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}" - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(temporary, flags, 0o600) - try: - with os.fdopen(fd, "wb", closefd=False) as stream: - stream.write(raw) - stream.flush() - os.fsync(stream.fileno()) - finally: - os.close(fd) - try: - os.link(temporary, path) - directory_fd = os.open(path.parent, os.O_RDONLY) - try: - os.fsync(directory_fd) - finally: - os.close(directory_fd) - except FileExistsError: - observed = self._load( - relative, - record_type=record_type, - key=key, - ) - if observed is None or _canonical_bytes(observed) != _canonical_bytes( - expected - ): - raise ImmutableEvaluationRecordError( - f"{relative} already contains different data" - ) - return observed - finally: - try: - temporary.unlink() - except FileNotFoundError: - pass - return expected - - def _load( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - ) -> dict[str, Any] | None: - path = self._path(relative) - if not path.exists(): - return None - return _validate_record( - self._read(path), - record_type=record_type, - key=key, - ) - - @staticmethod - def _read(path: Path) -> dict[str, Any]: - flags = os.O_RDONLY - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(path, flags) - except OSError as exc: - if exc.errno == errno.ELOOP: - raise CheckpointEvaluationError( - f"{path} must not be a symlink" - ) from exc - raise - try: - metadata = os.fstat(fd) - if ( - not stat.S_ISREG(metadata.st_mode) - or metadata.st_size > _MAX_RECORD_BYTES - ): - raise CheckpointEvaluationError( - f"{path} is not a bounded regular file" - ) - chunks: list[bytes] = [] - remaining = _MAX_RECORD_BYTES + 1 - while remaining: - chunk = os.read(fd, min(1024 * 1024, remaining)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - if len(raw) > _MAX_RECORD_BYTES: - raise CheckpointEvaluationError( - f"{path} exceeds the record bound" - ) - finally: - os.close(fd) - try: - value = json.loads(raw) - except (UnicodeDecodeError, json.JSONDecodeError) as exc: - raise CheckpointEvaluationError( - f"{path} is not valid JSON" - ) from exc - if not isinstance(value, dict): - raise CheckpointEvaluationError(f"{path} must contain an object") - return value - - -def _depth_rows(dataset_root: Path) -> list[dict[str, Any]]: - import pyarrow.parquet as pq - - rows: list[dict[str, Any]] = [] - for path in parquet_paths( - dataset_root, - configuration="depth", - split="validation", - ): - schema = set(pq.read_schema(path).names) - missing = set(_DEPTH_COLUMNS) - schema - if missing: - raise ReferencePanelFault( - f"{path} is missing evaluation fields: {sorted(missing)}" - ) - rows.extend( - pq.read_table(path, columns=list(_DEPTH_COLUMNS)).to_pylist() - ) - return rows - - -def _depth_task(row: Mapping[str, Any]) -> EvaluationTask: - sampler = sampler_from_row(row) - return EvaluationTask( - task_id=sampler.opaque_id, - level=sampler.level.upper(), - representation_id=sampler.representation_id, - observation=observation_from_row(row), - reference=reference_from_row(row), - ) - - -def _benchmark_tasks( - *, - repo_root: Path, - protocol: Mapping[str, Any], -) -> list[EvaluationTask]: - manifest = ( - protocol_path(repo_root).parent - / str(protocol["evaluation"]["task_manifest"]) - ).resolve(strict=True) - if not manifest.is_relative_to(repo_root): - raise ReferencePanelFault("benchmark manifest escaped the repository") - document = load_task_manifest(manifest, repo_root=repo_root) - if ( - document["logical_sha256"] - != protocol["evaluation"]["task_manifest_logical_sha256"] - ): - raise ReferencePanelFault( - "benchmark manifest differs from the training protocol" - ) - tasks = load_task_set( - repo_root=repo_root, - manifest=document, - task_set=BENCHMARK_TASK_SET, - ) - if [task.task_id for task in tasks] != [f"F{index}" for index in range(1, 9)]: - raise ReferencePanelFault("progress panel is not exact ordered F1-F8") - return tasks - - -def _selection_rank(*values: str) -> str: - payload = "\0".join((_PANEL_SELECTION_SEED, *values)) - return hashlib.sha256(payload.encode("utf-8")).hexdigest() - - -def _inkling_depth_tasks(rows: Sequence[dict[str, Any]]) -> list[EvaluationTask]: - quotas = {"L0": 6, "L1": 6, "L2": 6, "L3": 6, "L4": 36} - by_level_representation: dict[str, dict[str, list[dict[str, Any]]]] = { - level: {} for level in quotas - } - for row in rows: - level = str(row["level"]).upper() - if level not in by_level_representation: - raise ReferencePanelFault(f"unexpected depth level: {level!r}") - by_level_representation[level].setdefault( - str(row["representation_id"]), - [], - ).append(row) - - selected: list[EvaluationTask] = [] - for level, count in quotas.items(): - represented = by_level_representation[level] - representation_ids = sorted( - represented, - key=lambda value: _selection_rank("representation", level, value), - ) - if len(representation_ids) < count: - raise ReferencePanelFault( - f"{level} has too few representations for Inkling promotion" - ) - for representation_id in representation_ids[:count]: - row = min( - represented[representation_id], - key=lambda value: _selection_rank( - "realization", - level, - representation_id, - str(value["id"]), - ), - ) - selected.append(_depth_task(row)) - return selected - - -def _one_realization_per_representation( - rows: Sequence[dict[str, Any]], - *, - realization_slot: int, - level: str | None = None, -) -> list[EvaluationTask]: - """Select one exact held-out realization for every requested representation.""" - - requested_level = level.upper() if level is not None else None - by_representation: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list) - for row in rows: - row_level = str(row["level"]).upper() - if requested_level is not None and row_level != requested_level: - continue - if row.get("realization_slot") == realization_slot: - by_representation[(row_level, str(row["representation_id"]))].append(row) - if not by_representation: - raise ReferencePanelFault( - f"no validation rows use realization slot {realization_slot}" - ) - - selected: list[EvaluationTask] = [] - for identity, candidates in sorted(by_representation.items()): - if len(candidates) != 1: - raise ReferencePanelFault( - f"{identity[0]}/{identity[1]} has {len(candidates)} rows " - f"for validation realization slot {realization_slot}" - ) - selected.append(_depth_task(candidates[0])) - return selected - - -def _level_for_progress_panel(panel: str) -> str | None: - return next( - (level for level, panel_id in LEVEL_PROGRESS_PANELS.items() if panel == panel_id), - None, - ) - - -def build_panel_tasks( - *, - repo_root: Path, - protocol: Mapping[str, Any], - panel: str, -) -> list[EvaluationTask]: - """Build one exact label-free evaluation panel.""" - - if panel not in SUPPORTED_PANELS: - raise ValueError(f"unsupported panel {panel!r}") - if panel == PANEL_PROGRESS: - tasks = _benchmark_tasks(repo_root=repo_root, protocol=protocol) - else: - rows = _depth_rows(repo_root / "dataset") - if len(rows) != int(protocol["dataset"]["validation_rows"]): - raise ReferencePanelFault( - "depth validation row count differs from the protocol" - ) - if panel == PANEL_DEPTH_VALIDATION: - tasks = sorted( - (_depth_task(row) for row in rows), - key=lambda task: ( - task.level, - task.representation_id, - task.task_id, - ), - ) - elif panel == PANEL_INKLING_PROMOTION: - tasks = _inkling_depth_tasks(rows) - tasks.extend(_benchmark_tasks(repo_root=repo_root, protocol=protocol)) - elif panel == PANEL_DEPTH_FINAL_SELECTION: - tasks = _one_realization_per_representation( - rows, - realization_slot=7, - ) - else: - level = _level_for_progress_panel(panel) - if level is None: - raise ValueError(f"unsupported panel {panel!r}") - tasks = _one_realization_per_representation( - rows, - realization_slot=6, - level=level, - ) - - expected = { - PANEL_PROGRESS: 8, - PANEL_DEPTH_VALIDATION: 1092, - PANEL_INKLING_PROMOTION: 68, - PANEL_DEPTH_FINAL_SELECTION: 546, - **{ - panel_id: LEVEL_PROGRESS_ROWS[level] - for level, panel_id in LEVEL_PROGRESS_PANELS.items() - }, - }[panel] - if len(tasks) != expected: - raise ReferencePanelFault( - f"{panel} selected {len(tasks)} tasks; expected {expected}" - ) - task_ids = [task.task_id for task in tasks] - if len(task_ids) != len(set(task_ids)): - raise ReferencePanelFault(f"{panel} selected duplicate task IDs") - if panel == PANEL_DEPTH_VALIDATION: - representations = {task.representation_id for task in tasks} - if len(representations) != 546: - raise ReferencePanelFault( - "depth validation no longer spans 546 representations" - ) - if panel == PANEL_DEPTH_FINAL_SELECTION: - representations = { - (task.level, task.representation_id) - for task in tasks - } - if len(representations) != 546: - raise ReferencePanelFault( - "depth final-selection no longer spans 546 representations" - ) - level = _level_for_progress_panel(panel) - if level is not None: - if ( - {task.level for task in tasks} != {level} - or len({task.representation_id for task in tasks}) - != LEVEL_PROGRESS_ROWS[level] - ): - raise ReferencePanelFault( - f"{panel} no longer contains one {level} row per representation" - ) - if panel == PANEL_INKLING_PROMOTION: - counts = Counter(task.level for task in tasks) - expected_counts = { - "BENCHMARK": 8, - "L0": 6, - "L1": 6, - "L2": 6, - "L3": 6, - "L4": 36, - } - if counts != expected_counts: - raise ReferencePanelFault( - f"Inkling promotion panel changed: {dict(counts)}" - ) - return tasks - - -def _task_manifest(tasks: Sequence[EvaluationTask], *, panel: str) -> dict[str, Any]: - entries = [ - { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "image_sha256": task.observation.image_sha256, - "target_image_sha256": task.reference.target_image_sha256, - "footprint_um": list(task.observation.footprint_um), - } - for task in tasks - ] - return { - "schema_version": "pixcell-training-evaluation-task-panel-v1", - "panel": panel, - "selection_seed": ( - _PANEL_SELECTION_SEED - if panel == PANEL_INKLING_PROMOTION - else ( - _LEVEL_PROGRESS_SELECTION - if panel in LEVEL_PROGRESS_PANELS.values() - else ( - _DEPTH_FINAL_SELECTION - if panel == PANEL_DEPTH_FINAL_SELECTION - else None - ) - ) - ), - "task_count": len(entries), - "task_ids": [item["task_id"] for item in entries], - "logical_sha256": canonical_json_sha256(entries), - } - - -def _load_checkpoint_binding( - *, - store: TrainingStore, - protocol: dict[str, Any], - stage_id: str, - wave: str, - checkpoint_name: str, - replicate_id: str, - expected_receipt_sha256: str, - source_git_sha: str, -) -> CheckpointBinding: - if not _SHA256.fullmatch(expected_receipt_sha256): - raise CheckpointReceiptError( - "wave receipt SHA-256 must be explicit and complete" - ) - stage = stage_spec(protocol, stage_id) - stage.wave(wave) - key = StageKey(STUDY_ID, stage.stage_id, replicate_id) - manifest = store.load_manifest(key) - receipt = store.load_wave_receipt(key, wave=wave) - if manifest is None or receipt is None: - raise CheckpointReceiptError( - f"{stage_id}/{wave}/{replicate_id} has no complete training receipt" - ) - if receipt.record_sha256 != expected_receipt_sha256: - raise CheckpointReceiptError( - "the external wave receipt differs from the requested digest" - ) - expected_manifest = receipt.payload.get("run_manifest_record_sha256") - if expected_manifest != manifest.record_sha256: - raise CheckpointReceiptError( - "wave receipt and training run manifest differ" - ) - expected_fields = { - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "contract_version": protocol["contract_version"], - } - for field, expected in expected_fields.items(): - if manifest.payload.get(field) != expected: - raise CheckpointReceiptError( - f"training manifest {field} differs from this evaluation" - ) - manifest_stage = manifest.payload.get("stage") - if ( - not isinstance(manifest_stage, Mapping) - or manifest_stage.get("stage_id") != stage.stage_id - or manifest_stage.get("model_key") != stage.model_key - ): - raise CheckpointReceiptError("training manifest has the wrong stage") - manifest_dataset = manifest.payload.get("dataset") - if ( - not isinstance(manifest_dataset, Mapping) - or manifest_dataset.get("logical_release_sha256") - != protocol["dataset"]["logical_release_sha256"] - ): - raise CheckpointReceiptError("training manifest has the wrong dataset") - if not _IDENTIFIER.fullmatch(checkpoint_name): - raise CheckpointReceiptError("checkpoint name is not a safe identifier") - terminal = receipt.payload.get("checkpoint") - inventory = receipt.payload.get("checkpoint_inventory") - if not isinstance(terminal, Mapping) or not isinstance(inventory, Mapping): - raise CheckpointReceiptError( - "wave receipt has no immutable checkpoint inventory" - ) - if set(inventory) != {"count", "entries", "logical_sha256"}: - raise CheckpointReceiptError("checkpoint inventory schema changed") - entries = inventory.get("entries") - count = inventory.get("count") - inventory_sha256 = str(inventory.get("logical_sha256", "")) - if ( - not isinstance(entries, list) - or not entries - or isinstance(count, bool) - or not isinstance(count, int) - or count != len(entries) - or inventory_sha256 != _canonical_sha256(entries) - or not _SHA256.fullmatch(inventory_sha256) - ): - raise CheckpointReceiptError("checkpoint inventory digest or count changed") - entry_fields = { - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - "role", - "training_progress_fraction", - } - normalized_entries: list[dict[str, Any]] = [] - names: set[str] = set() - previous_progress = -1.0 - terminal_name = ( - "final" - if stage.kind == "sft" - else f"{int(stage.wave(wave)['max_steps']):06d}" - ) - if stage.kind == "rl" and len(entries) != 1: - raise CheckpointReceiptError( - "an RL wave must expose only its immutable ceiling checkpoint" - ) - for index, value in enumerate(entries): - if not isinstance(value, Mapping) or set(value) != entry_fields: - raise CheckpointReceiptError("checkpoint inventory entry schema changed") - entry = dict(value) - name = str(entry["name"]) - role = str(entry["role"]) - progress = entry["training_progress_fraction"] - if ( - not _IDENTIFIER.fullmatch(name) - or name in names - or role not in {"periodic", "terminal"} - or isinstance(progress, bool) - or not isinstance(progress, (int, float)) - or not math.isfinite(float(progress)) - or not 0.0 < float(progress) <= 1.0 - or float(progress) <= previous_progress - or not _TINKER_PATH.fullmatch(str(entry["state_path"])) - or not _TINKER_PATH.fullmatch(str(entry["sampler_path"])) - ): - raise CheckpointReceiptError("checkpoint inventory entry is invalid") - if role == "periodic": - if ( - stage.kind != "sft" - or not re.fullmatch(r"[0-9]{6}", name) - or entry["final"] is not False - or float(progress) >= 1.0 - or index == len(entries) - 1 - ): - raise CheckpointReceiptError( - "periodic checkpoint inventory entry is invalid" - ) - elif ( - name != terminal_name - or entry["final"] is not True - or float(progress) != 1.0 - or index != len(entries) - 1 - or ( - stage.kind == "rl" - and entry["batch"] != int(stage.wave(wave)["max_steps"]) - ) - ): - raise CheckpointReceiptError( - "terminal checkpoint inventory entry is invalid" - ) - names.add(name) - previous_progress = float(progress) - normalized_entries.append(entry) - terminal_fields = { - key: normalized_entries[-1][key] - for key in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - } - if dict(terminal) != terminal_fields: - raise CheckpointReceiptError( - "terminal checkpoint alias differs from the inventory" - ) - selected = next( - (entry for entry in normalized_entries if entry["name"] == checkpoint_name), - None, - ) - if selected is None: - raise CheckpointReceiptError( - f"checkpoint {checkpoint_name!r} is not in the wave receipt" - ) - sampler_path = str(selected["sampler_path"]) - if ( - receipt.payload.get("stage_id") != stage.stage_id - or receipt.payload.get("wave") != wave - ): - raise CheckpointReceiptError("wave receipt identity differs from its path") - return CheckpointBinding( - key=key, - stage=stage, - wave=wave, - checkpoint_name=checkpoint_name, - checkpoint=selected, - checkpoint_inventory_sha256=inventory_sha256, - sampler_path=sampler_path, - run_manifest=manifest, - wave_receipt=receipt, - ) - - -def _base_model_binding( - *, - protocol: dict[str, Any], - stage_id: str, - replicate_id: str, - source_git_sha: str, -) -> BaseModelBinding: - stage = stage_spec(protocol, stage_id) - if ( - stage.stage_id != "qwen-base-rl-l0" - or stage.model_key != "qwen" - or stage.parent != "base:qwen" - ): - raise CheckpointReceiptError( - "base-model evaluation is restricted to qwen-base-rl-l0" - ) - key = StageKey(STUDY_ID, stage.stage_id, replicate_id) - model = dict(protocol["models"]["qwen"]) - checkpoint = { - "name": "base", - "model": model["model"], - "renderer": model["renderer"], - "thinking": model["thinking"], - } - inventory_sha256 = _canonical_sha256([checkpoint]) - identity = { - "schema_version": "pixcell-training-base-model-binding-v1", - "study_id": STUDY_ID, - "stage_id": stage.stage_id, - "replicate_id": replicate_id, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "model": checkpoint, - "checkpoint_inventory_sha256": inventory_sha256, - } - return BaseModelBinding( - key=key, - stage=stage, - wave="base", - checkpoint_name="base", - checkpoint=checkpoint, - checkpoint_inventory_sha256=inventory_sha256, - sampler_path=str(model["model"]), - binding_sha256=_canonical_sha256(identity), - ) - - -def _sample_binding_identity(binding: EvaluationBinding) -> dict[str, Any]: - common = { - "checkpoint_name": binding.checkpoint_name, - "checkpoint_inventory_sha256": binding.checkpoint_inventory_sha256, - "checkpoint": binding.checkpoint, - } - if isinstance(binding, CheckpointBinding): - return { - "training_wave_receipt_record_sha256": ( - binding.wave_receipt.record_sha256 - ), - **common, - } - return { - "base_model_binding_sha256": binding.binding_sha256, - **common, - } - - -def _sampler_provenance(binding: EvaluationBinding) -> dict[str, Any]: - common = { - "sampler_path": binding.sampler_path, - "checkpoint_name": binding.checkpoint_name, - "checkpoint": binding.checkpoint, - "checkpoint_inventory_sha256": binding.checkpoint_inventory_sha256, - } - if isinstance(binding, CheckpointBinding): - return { - **common, - "training_run_manifest_record_sha256": ( - binding.run_manifest.record_sha256 - ), - "training_wave_receipt_record_sha256": ( - binding.wave_receipt.record_sha256 - ), - "training_wave_receipt_relative_path": str( - binding.wave_receipt.relative_path - ), - } - return { - **common, - "binding_kind": "base-model", - "base_model_binding_sha256": binding.binding_sha256, - } - - -def _sampling_client_kwargs(binding: EvaluationBinding) -> dict[str, str]: - if isinstance(binding, BaseModelBinding): - return {"base_model": binding.sampler_path} - return {"model_path": binding.sampler_path} - - -def _model_binding( - protocol: Mapping[str, Any], - stage: StageSpec, -) -> dict[str, Any]: - model = dict(protocol["models"][stage.model_key]) - exact = { - "max_image_long_edge": MAX_IMAGE_LONG_EDGE, - "max_output_tokens": MAX_OUTPUT_TOKENS, - "context_tokens": 65_536, - } - for field, expected in exact.items(): - if model.get(field) != expected: - raise CheckpointEvaluationError( - f"{stage.model_key} {field} differs from the evaluation contract" - ) - if model.get("thinking") is not True: - raise CheckpointEvaluationError("checkpoint evaluation requires thinking on") - return model - - -def _direct_prompt_text(observation: ModelObservation) -> str: - text = build_prompt_text(observation) - leaked = [fragment for fragment in _FORBIDDEN_PROMPT_FRAGMENTS if fragment in text] - if leaked: - raise CheckpointEvaluationError( - f"direct Phase-A prompt leaked verifier fields: {leaked}" - ) - return text - - -def _preflight_tasks( - *, - tasks: Sequence[EvaluationTask], - renderer: Any, - evaluator: PixCellEvaluator, - context_tokens: int, -) -> tuple[list[PreparedTask], dict[str, Any]]: - """Check every prompt and reference before any sampling client is created.""" - - prepared: list[PreparedTask] = [] - prompt_digest = hashlib.sha256() - reference_digest = hashlib.sha256() - for task in tasks: - prompt_text = _direct_prompt_text(task.observation) - prompt_text_sha256 = hashlib.sha256( - prompt_text.encode("utf-8") - ).hexdigest() - try: - evidence = evaluator.validate_reference(task.reference) - except Exception as exc: - raise ReferencePanelFault( - f"{task.task_id} reference validation failed: {exc}" - ) from exc - if ( - evidence.get("target_image_sha256") - != task.reference.target_image_sha256 - ): - raise ReferencePanelFault( - f"{task.task_id} reference digest changed during validation" - ) - prompt = renderer.build_generation_prompt( - [_message(task, max_image=MAX_IMAGE_LONG_EDGE)] - ) - prompt_tokens = int(prompt.length) - if prompt_tokens + MAX_OUTPUT_TOKENS > context_tokens: - raise CheckpointEvaluationError( - f"{task.task_id} exceeds context: " - f"{prompt_tokens}+{MAX_OUTPUT_TOKENS}>{context_tokens}" - ) - prompt_digest.update(task.task_id.encode("utf-8")) - prompt_digest.update(str(prompt_tokens).encode("ascii")) - prompt_digest.update(bytes.fromhex(prompt_text_sha256)) - prompt_digest.update(bytes.fromhex(task.observation.image_sha256)) - reference_digest.update(task.task_id.encode("utf-8")) - reference_digest.update( - _canonical_bytes( - { - "target_image_sha256": task.reference.target_image_sha256, - "evidence": evidence, - } - ) - ) - prepared.append( - PreparedTask( - task=task, - prompt_tokens=prompt_tokens, - prompt_text_sha256=prompt_text_sha256, - reference_evidence=dict(evidence), - ) - ) - prompt_lengths = [item.prompt_tokens for item in prepared] - return prepared, { - "task_count": len(prepared), - "prompt_tokens": { - "min": min(prompt_lengths), - "max": max(prompt_lengths), - }, - "minimum_context_headroom": ( - context_tokens - MAX_OUTPUT_TOKENS - max(prompt_lengths) - ), - "prompt_set_sha256": prompt_digest.hexdigest(), - "reference_set_sha256": reference_digest.hexdigest(), - } - - -def _sample_seed( - *, - protocol_sha256: str, - binding: EvaluationBinding, - panel: str, - task_id: str, -) -> int: - digest = hashlib.sha256( - "\0".join( - ( - "pixcell-checkpoint-evaluation-seed-v1", - protocol_sha256, - binding.stage.stage_id, - binding.wave, - binding.checkpoint_name, - binding.checkpoint_inventory_sha256, - binding.key.replicate_id, - panel, - task_id, - ) - ).encode("utf-8") - ).digest() - return int.from_bytes(digest[:4], "big") & 0x7FFFFFFF - - -def _message_reasoning(message: Any) -> str: - content = message.get("content") - if not isinstance(content, list): - return "" - return "\n".join( - str(part.get("thinking", "")) - for part in content - if isinstance(part, Mapping) and part.get("type") == "thinking" - ) - - -def _decode_sample(renderer: Any, sequence: Any) -> dict[str, Any]: - from tinker_cookbook.renderers import get_text_content - - tokens = [int(token) for token in sequence.tokens] - raw_text = str(renderer.tokenizer.decode(tokens)) - try: - message, termination = renderer.parse_response(tokens) - except Exception as exc: - raise SamplingInfrastructureFault( - f"renderer could not parse a sampled response: {exc}" - ) from exc - answer_text = str(get_text_content(message)) - reasoning_text = _message_reasoning(message) - stop_reason = str(sequence.stop_reason) - cap_hit = ( - len(tokens) >= MAX_OUTPUT_TOKENS - or "length" in stop_reason.lower() - or "max_token" in stop_reason.lower() - ) - return { - "completion_tokens": len(tokens), - "stop_reason": stop_reason, - "cap_hit": cap_hit, - "raw_text": raw_text, - "raw_text_sha256": hashlib.sha256(raw_text.encode("utf-8")).hexdigest(), - "answer_text": answer_text, - "answer_text_sha256": hashlib.sha256( - answer_text.encode("utf-8") - ).hexdigest(), - "reasoning_text": reasoning_text, - "reasoning_text_sha256": hashlib.sha256( - reasoning_text.encode("utf-8") - ).hexdigest(), - "channel_parse_complete": bool(termination.is_clean), - } - - -def _sample_payload( - *, - prepared: PreparedTask, - response: dict[str, Any], - seed: int, - manifest_record_sha256: str, - binding: EvaluationBinding, -) -> dict[str, Any]: - return { - "evaluation_manifest_record_sha256": manifest_record_sha256, - **_sample_binding_identity(binding), - "task": { - "task_id": prepared.task.task_id, - "level": prepared.task.level, - "representation_id": prepared.task.representation_id, - "image_sha256": prepared.task.observation.image_sha256, - "target_image_sha256": prepared.task.reference.target_image_sha256, - }, - "request": { - "num_samples": 1, - "seed": seed, - "max_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "prompt_tokens": prepared.prompt_tokens, - "prompt_text_sha256": prepared.prompt_text_sha256, - }, - "response": response, - } - - -def _evaluation_payload( - *, - task: EvaluationTask, - result: EvaluationResult, - sample_record_sha256: str, - manifest_record_sha256: str, -) -> dict[str, Any]: - if result.attribution is Attribution.REFERENCE: - raise ReferencePanelFault( - f"{task.task_id}: {result.status.value}: {result.error}" - ) - if result.attribution is Attribution.EVALUATOR: - raise EvaluationInfrastructureFault( - f"{task.task_id}: {result.status.value}: {result.error}" - ) - if result.attribution is not Attribution.MODEL: - raise EvaluationInfrastructureFault( - f"{task.task_id}: evaluator returned an unknown attribution" - ) - iou = 0.0 if result.iou is None else float(result.iou) - if not math.isfinite(iou) or not 0.0 <= iou <= 1.0: - raise EvaluationInfrastructureFault( - f"{task.task_id}: evaluator returned invalid IoU {result.iou!r}" - ) - if result.status is EvaluationStatus.OK and result.iou is None: - raise EvaluationInfrastructureFault( - f"{task.task_id}: successful evaluation has no IoU" - ) - return { - "evaluation_manifest_record_sha256": manifest_record_sha256, - "sample_record_sha256": sample_record_sha256, - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "status": result.status.value, - "attribution": result.attribution.value, - "pure_executable": result.status is EvaluationStatus.OK, - "raw_absolute_scale_iou": iou, - "dice": float(result.dice) if result.dice is not None else None, - "render_sha256": result.metrics.get("render_sha256"), - "reference_sha256": result.metrics.get("reference_sha256"), - "violations": list(result.violations), - "error": result.error, - "latency_seconds": float(result.latency_seconds), - "diagnostics": _safe_json(result.metrics), - } - - -def _validate_sample_record( - *, - record: Mapping[str, Any], - prepared: PreparedTask, - seed: int, - manifest_record_sha256: str, - binding: EvaluationBinding, -) -> None: - payload = record["payload"] - expected = { - "evaluation_manifest_record_sha256": manifest_record_sha256, - **_sample_binding_identity(binding), - } - for field, value in expected.items(): - if payload.get(field) != value: - raise ImmutableEvaluationRecordError( - f"{prepared.task.task_id} sample belongs to another evaluation" - ) - task = payload.get("task") - request = payload.get("request") - if not isinstance(task, Mapping) or not isinstance(request, Mapping): - raise ImmutableEvaluationRecordError("sample record is incomplete") - expected_task = { - "task_id": prepared.task.task_id, - "level": prepared.task.level, - "representation_id": prepared.task.representation_id, - "image_sha256": prepared.task.observation.image_sha256, - "target_image_sha256": prepared.task.reference.target_image_sha256, - } - expected_request = { - "num_samples": 1, - "seed": seed, - "max_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "prompt_tokens": prepared.prompt_tokens, - "prompt_text_sha256": prepared.prompt_text_sha256, - } - if task != expected_task or request != expected_request: - raise ImmutableEvaluationRecordError( - f"{prepared.task.task_id} sample request changed" - ) - - -def _validate_evaluation_record( - *, - record: Mapping[str, Any], - task: EvaluationTask, - sample_record_sha256: str, - manifest_record_sha256: str, -) -> None: - payload = record["payload"] - expected = { - "evaluation_manifest_record_sha256": manifest_record_sha256, - "sample_record_sha256": sample_record_sha256, - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "attribution": Attribution.MODEL.value, - } - for field, value in expected.items(): - if payload.get(field) != value: - raise ImmutableEvaluationRecordError( - f"{task.task_id} evaluation record changed" - ) - status = str(payload.get("status", "")) - if status not in _MODEL_EVALUATION_STATUSES: - raise ImmutableEvaluationRecordError( - f"{task.task_id} has a non-model evaluation status" - ) - if payload.get("pure_executable") != (status == EvaluationStatus.OK.value): - raise ImmutableEvaluationRecordError( - f"{task.task_id} status and pure-executable flag differ" - ) - iou = payload.get("raw_absolute_scale_iou") - if ( - isinstance(iou, bool) - or not isinstance(iou, (int, float)) - or not math.isfinite(float(iou)) - or not 0.0 <= float(iou) <= 1.0 - ): - raise ImmutableEvaluationRecordError( - f"{task.task_id} has an invalid stored IoU" - ) - if status != EvaluationStatus.OK.value and float(iou) != 0.0: - raise ImmutableEvaluationRecordError( - f"{task.task_id} failed model output has nonzero IoU" - ) - - -async def _sample_missing( - *, - prepared: Sequence[PreparedTask], - renderer: Any, - sampling_client: Any, - store: CheckpointEvaluationStore, - manifest_record_sha256: str, - binding: EvaluationBinding, - protocol_sha256: str, - panel: str, -) -> None: - import tinker - - semaphore = asyncio.Semaphore(SAMPLE_CONCURRENCY) - - async def sample_one(item: PreparedTask) -> None: - seed = _sample_seed( - protocol_sha256=protocol_sha256, - binding=binding, - panel=panel, - task_id=item.task.task_id, - ) - existing = store.load_sample(item.task.task_id) - if existing is not None: - _validate_sample_record( - record=existing, - prepared=item, - seed=seed, - manifest_record_sha256=manifest_record_sha256, - binding=binding, - ) - return - messages = [_message(item.task, max_image=MAX_IMAGE_LONG_EDGE)] - prompt_text = str(messages[0]["content"][1]["text"]) - if ( - hashlib.sha256(prompt_text.encode("utf-8")).hexdigest() - != item.prompt_text_sha256 - ): - raise SamplingInfrastructureFault( - f"{item.task.task_id} prompt text changed after preflight" - ) - prompt = renderer.build_generation_prompt(messages) - if int(prompt.length) != item.prompt_tokens: - raise SamplingInfrastructureFault( - f"{item.task.task_id} prompt changed after preflight" - ) - async with semaphore: - try: - result = await sampling_client.sample_async( - prompt=prompt, - num_samples=1, - sampling_params=tinker.SamplingParams( - stop=renderer.get_stop_sequences(), - max_tokens=MAX_OUTPUT_TOKENS, - temperature=TEMPERATURE, - top_p=TOP_P, - seed=seed, - ), - ) - except Exception as exc: - raise SamplingInfrastructureFault( - f"{item.task.task_id} sampling failed: {exc}" - ) from exc - sequences = list(result.sequences) - if len(sequences) != 1: - raise SamplingInfrastructureFault( - f"{item.task.task_id} returned {len(sequences)} samples, expected 1" - ) - response = _decode_sample(renderer, sequences[0]) - store.create_or_verify_sample( - item.task.task_id, - _sample_payload( - prepared=item, - response=response, - seed=seed, - manifest_record_sha256=manifest_record_sha256, - binding=binding, - ), - ) - - await asyncio.gather(*(sample_one(item) for item in prepared)) - - -def _score_missing( - *, - prepared: Sequence[PreparedTask], - evaluator: PixCellEvaluator, - store: CheckpointEvaluationStore, - manifest_record_sha256: str, -) -> None: - pending: list[tuple[PreparedTask, dict[str, Any]]] = [] - for item in prepared: - sample = store.load_sample(item.task.task_id) - if sample is None: - raise CheckpointEvaluationError( - f"{item.task.task_id} has no resumable sample" - ) - existing = store.load_evaluation(item.task.task_id) - if existing is not None: - _validate_evaluation_record( - record=existing, - task=item.task, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - ) - continue - pending.append((item, sample)) - - for offset in range(0, len(pending), EVALUATION_BATCH_SIZE): - batch = pending[offset : offset + EVALUATION_BATCH_SIZE] - requests = [ - ( - item.task.reference, - str(sample["payload"]["response"]["answer_text"]), - ) - for item, sample in batch - ] - try: - results = evaluator.evaluate_batch(requests) - except Exception as exc: - raise EvaluationInfrastructureFault( - f"isolated evaluation batch failed: {exc}" - ) from exc - if len(results) != len(batch): - raise EvaluationInfrastructureFault( - "isolated evaluator returned a different result count" - ) - faults: list[CheckpointEvaluationError] = [] - for (item, sample), result in zip(batch, results, strict=True): - try: - payload = _evaluation_payload( - task=item.task, - result=result, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - ) - except (ReferencePanelFault, EvaluationInfrastructureFault) as exc: - faults.append(exc) - continue - store.create_or_verify_evaluation(item.task.task_id, payload) - if faults: - raise faults[0] - - -def _aggregate(values: Sequence[Mapping[str, Any]]) -> dict[str, Any]: - if not values: - raise CheckpointEvaluationError("cannot aggregate an empty panel") - ious = [float(value["raw_absolute_scale_iou"]) for value in values] - pure = [bool(value["pure_executable"]) for value in values] - return { - "tasks": len(values), - "mean_raw_absolute_scale_iou": sum(ious) / len(ious), - "pure_executable": sum(pure), - "pure_executable_rate": sum(pure) / len(pure), - "statuses": dict( - sorted(Counter(str(value["status"]) for value in values).items()) - ), - } - - -def validate_checkpoint_report( - record: Mapping[str, Any], - *, - manifest_record_sha256: str, - binding: EvaluationBinding, - panel: str, - expected_task_ids: Sequence[str], -) -> dict[str, Any]: - """Validate the immutable report identity used by promotion gates.""" - - validated = _validate_record( - record, - record_type="checkpoint_report", - key={}, - ) - payload = validated["payload"] - expected_identity = { - "schema_version": EVALUATION_SCHEMA_VERSION, - "study_id": STUDY_ID, - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "replicate_id": binding.key.replicate_id, - "panel": panel, - } - for field, expected in expected_identity.items(): - if payload.get(field) != expected: - raise CheckpointEvaluationError( - f"checkpoint report {field} differs from its request" - ) - provenance = payload.get("provenance") - summary = payload.get("summary") - records = payload.get("records") - if ( - not isinstance(provenance, Mapping) - or not isinstance(summary, Mapping) - or not isinstance(records, list) - ): - raise CheckpointEvaluationError("checkpoint report is incomplete") - sampler = provenance.get("sampler") - task_panel = provenance.get("task_panel") - if not isinstance(sampler, Mapping) or not isinstance(task_panel, Mapping): - raise CheckpointEvaluationError( - "checkpoint report has incomplete provenance" - ) - expected_provenance = { - "evaluation_manifest_record_sha256": manifest_record_sha256, - } - for field, expected in expected_provenance.items(): - if provenance.get(field) != expected: - raise CheckpointEvaluationError( - f"checkpoint report {field} differs from its manifest" - ) - expected_sampler = _sampler_provenance(binding) - if any( - sampler.get(field) != expected - for field, expected in expected_sampler.items() - ): - raise CheckpointEvaluationError( - "checkpoint report sampler provenance differs from training" - ) - if isinstance(binding, BaseModelBinding) and any( - field in sampler - for field in ( - "training_run_manifest_record_sha256", - "training_wave_receipt_record_sha256", - "training_wave_receipt_relative_path", - ) - ): - raise CheckpointEvaluationError( - "base-model report fabricates training provenance" - ) - task_ids = [str(value) for value in expected_task_ids] - if ( - task_panel.get("task_ids") != task_ids - or task_panel.get("task_count") != len(task_ids) - or summary.get("tasks") != len(task_ids) - or [item.get("task_id") for item in records] != task_ids - ): - raise CheckpointEvaluationError( - "checkpoint report task inventory differs from the sealed panel" - ) - iou = summary.get("mean_raw_absolute_scale_iou") - pure_rate = summary.get("pure_executable_rate") - representation_macro = payload.get("representation_macro_mean_iou") - if any( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or not math.isfinite(float(value)) - or not 0.0 <= float(value) <= 1.0 - for value in (iou, pure_rate, representation_macro) - ): - raise CheckpointEvaluationError( - "checkpoint report summary contains invalid metrics" - ) - return validated - - -def _report_payload( - *, - repo_root: Path, - prepared: Sequence[PreparedTask], - store: CheckpointEvaluationStore, - manifest_record: Mapping[str, Any], - binding: EvaluationBinding, - protocol: Mapping[str, Any], - source_git_sha: str, - panel: str, - dataset: Mapping[str, Any], - sandbox: Mapping[str, Any], -) -> dict[str, Any]: - records: list[dict[str, Any]] = [] - by_level: dict[str, list[dict[str, Any]]] = defaultdict(list) - by_representation: dict[str, list[dict[str, Any]]] = defaultdict(list) - for item in prepared: - sample = store.load_sample(item.task.task_id) - evaluation = store.load_evaluation(item.task.task_id) - if sample is None or evaluation is None: - raise CheckpointEvaluationError( - f"{item.task.task_id} is incomplete" - ) - _validate_evaluation_record( - record=evaluation, - task=item.task, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=str(manifest_record["record_sha256"]), - ) - payload = evaluation["payload"] - response = sample["payload"]["response"] - record = { - "task_id": item.task.task_id, - "level": item.task.level, - "representation_id": item.task.representation_id, - "status": payload["status"], - "pure_executable": payload["pure_executable"], - "raw_absolute_scale_iou": payload["raw_absolute_scale_iou"], - "completion_tokens": response["completion_tokens"], - "stop_reason": response["stop_reason"], - "cap_hit": response["cap_hit"], - "channel_parse_complete": response["channel_parse_complete"], - "answer_text_sha256": response["answer_text_sha256"], - "render_sha256": payload["render_sha256"], - "sample_record_sha256": sample["record_sha256"], - "evaluation_record_sha256": evaluation["record_sha256"], - } - records.append(record) - by_level[item.task.level].append(record) - by_representation[item.task.representation_id].append(record) - representation_summaries = { - key: _aggregate(values) - for key, values in sorted(by_representation.items()) - } - return { - "schema_version": EVALUATION_SCHEMA_VERSION, - "study_id": STUDY_ID, - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "replicate_id": binding.key.replicate_id, - "panel": panel, - "summary": _aggregate(records), - "representation_macro_mean_iou": ( - sum( - value["mean_raw_absolute_scale_iou"] - for value in representation_summaries.values() - ) - / len(representation_summaries) - ), - "levels": { - key: _aggregate(values) for key, values in sorted(by_level.items()) - }, - "representations": representation_summaries, - "records": records, - "provenance": { - "source_git_sha": source_git_sha, - "protocol": { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - }, - "dataset": { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": ( - "fixed-f1-f8" - if panel == PANEL_PROGRESS - else ( - "depth/validation+fixed-f1-f8" - if panel == PANEL_INKLING_PROMOTION - else ( - "depth/validation/slot-7" - if panel == PANEL_DEPTH_FINAL_SELECTION - else ( - f"depth/validation/{_level_for_progress_panel(panel)}/slot-6" - if _level_for_progress_panel(panel) is not None - else "depth/validation" - ) - ) - ) - ), - "logical_release_sha256": dataset["logical_release_sha256"], - "freeze_file_sha256": dataset["freeze_file_sha256"], - "parquet_shards": dataset["parquet_shards"], - }, - "task_panel": _task_manifest( - [item.task for item in prepared], - panel=panel, - ), - "sampler": { - "model": protocol["models"][binding.stage.model_key]["model"], - "renderer": protocol["models"][binding.stage.model_key]["renderer"], - "effort": protocol["models"][binding.stage.model_key].get("effort"), - **_sampler_provenance(binding), - }, - "sampling": { - "attempts_per_task": 1, - "max_output_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "max_image_long_edge": MAX_IMAGE_LONG_EDGE, - "seed_algorithm": "sha256-bound deterministic 31-bit seed", - }, - "prompt_assets": prompt_asset_hashes(), - "sandbox": dict(sandbox), - "evaluation_manifest_record_sha256": manifest_record["record_sha256"], - }, - } - - -def _sandbox_provenance(evaluator: PixCellEvaluator) -> dict[str, Any]: - boundary = evaluator.execution_boundary - if boundary is None: - raise EvaluationInfrastructureFault( - "checkpoint evaluation has no execution isolation boundary" - ) - return { - "runtime_path": str(boundary.runtime_path), - "daemon_endpoint": boundary.daemon_endpoint, - "image_ref": boundary.image_ref, - "image_id": boundary.image_id, - "workspace_root": str(boundary.workspace_root), - } - - -def _default_evaluator_factory() -> PixCellEvaluator: - return PixCellEvaluator( - max_workers=EVALUATOR_WORKERS, - evaluator_retries=1, - require_isolation=True, - ) - - -def _default_service_client_factory() -> Any: - import tinker - - return tinker.ServiceClient() - - -async def run_checkpoint_evaluation( - *, - repo_root: Path, - stage_id: str, - wave: str, - checkpoint_name: str, - replicate_id: str, - panel: str, - expected_source_sha: str, - expected_wave_receipt_sha256: str, - external_root: Path, - confirmation: str, - service_client_factory: Callable[[], Any] = _default_service_client_factory, - evaluator_factory: Callable[[], PixCellEvaluator] = _default_evaluator_factory, - base_model: bool = False, -) -> dict[str, Any]: - """Run or resume one exact checkpoint panel. - - ``service_client_factory`` is intentionally injectable so tests can prove - the entire zero-spend preflight precedes the first paid client. - """ - - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol(root, require_committed=True) - source_git_sha = validate_source_sha(root, expected_source_sha) - if confirmation != protocol["launch"]["confirmation_token"]: - raise CheckpointEvaluationError( - "explicit paid-evaluation confirmation is missing" - ) - try: - validate_stage_replicate( - protocol, - stage_id=stage_id, - replicate_id=replicate_id, - ) - except ValueError as exc: - raise CheckpointEvaluationError(str(exc)) from exc - training_store = TrainingStore( - repo_root=root, - external_root=external_root, - ) - if base_model: - if ( - wave != "base" - or checkpoint_name != "base" - or expected_wave_receipt_sha256 - ): - raise CheckpointReceiptError( - "base-model evaluation must use wave/checkpoint 'base' and no receipt" - ) - binding: EvaluationBinding = _base_model_binding( - protocol=protocol, - stage_id=stage_id, - replicate_id=replicate_id, - source_git_sha=source_git_sha, - ) - if panel != LEVEL_PROGRESS_PANELS["L0"]: - raise CheckpointEvaluationError( - "base Qwen evaluation is restricted to level-progress-l0" - ) - else: - binding = _load_checkpoint_binding( - store=training_store, - protocol=protocol, - stage_id=stage_id, - wave=wave, - checkpoint_name=checkpoint_name, - replicate_id=replicate_id, - expected_receipt_sha256=expected_wave_receipt_sha256, - source_git_sha=source_git_sha, - ) - if panel == PANEL_INKLING_PROMOTION and binding.stage.model_key != "inkling": - raise CheckpointEvaluationError( - "inkling-promotion is restricted to an Inkling checkpoint" - ) - model = _model_binding(protocol, binding.stage) - dataset = _dataset_binding(root / "dataset", protocol) - tasks = build_panel_tasks(repo_root=root, protocol=protocol, panel=panel) - renderer = _renderer( - model["model"], - model["renderer"], - effort=model.get("effort"), - ) - base_stage_lock = ( - training_store.acquire_stage_lock(binding.key) - if isinstance(binding, BaseModelBinding) - else None - ) - if base_stage_lock is not None: - base_stage_lock.__enter__() - try: - evaluator = evaluator_factory() - except Exception: - if base_stage_lock is not None: - base_stage_lock.__exit__(None, None, None) - raise - try: - prepared, preflight = _preflight_tasks( - tasks=tasks, - renderer=renderer, - evaluator=evaluator, - context_tokens=int(model["context_tokens"]), - ) - sandbox = _sandbox_provenance(evaluator) - panel_manifest = _task_manifest(tasks, panel=panel) - evaluation_stage_path = training_store.stage_path(binding.key) - if isinstance(binding, BaseModelBinding): - evaluation_stage_path.mkdir(parents=True, exist_ok=True, mode=0o700) - evaluation_store = CheckpointEvaluationStore( - stage_path=evaluation_stage_path, - wave=wave, - checkpoint_name=binding.checkpoint_name, - panel=panel, - ) - with evaluation_store.acquire_lock(): - manifest_payload = { - "study_id": STUDY_ID, - "stage_id": binding.stage.stage_id, - "wave": wave, - "checkpoint_name": binding.checkpoint_name, - "replicate_id": replicate_id, - "panel": panel, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": file_sha256(protocol_path(root)), - "contract_version": protocol["contract_version"], - "dataset": dataset, - "task_panel": panel_manifest, - "model": model, - "sampling": { - "attempts_per_task": 1, - "max_output_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "max_image_long_edge": MAX_IMAGE_LONG_EDGE, - "sample_concurrency": SAMPLE_CONCURRENCY, - "evaluation_batch_size": EVALUATION_BATCH_SIZE, - "evaluator_workers": EVALUATOR_WORKERS, - }, - "prompt_assets": prompt_asset_hashes(), - "preflight": preflight, - "sandbox": sandbox, - "sampler": _sampler_provenance(binding), - } - manifest = evaluation_store.create_or_verify_manifest( - manifest_payload - ) - existing_report = evaluation_store.load_report() - if existing_report is not None: - validate_checkpoint_report( - existing_report, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - panel=panel, - expected_task_ids=[item.task.task_id for item in prepared], - ) - return { - "status": "already_complete", - "stage_id": binding.stage.stage_id, - "wave": wave, - "checkpoint_name": binding.checkpoint_name, - "panel": panel, - "report": str( - evaluation_store.root / "report.json" - ), - "report_record_sha256": existing_report["record_sha256"], - "summary": existing_report["payload"]["summary"], - } - - missing_samples = [ - item - for item in prepared - if evaluation_store.load_sample(item.task.task_id) is None - ] - if missing_samples: - if not os.environ.get("TINKER_API_KEY"): - raise CheckpointEvaluationError("TINKER_API_KEY is not present") - try: - validate_source_sha(root, source_git_sha) - except (OSError, ValueError, subprocess.SubprocessError) as exc: - raise CheckpointEvaluationError( - "source changed after evaluation preflight; refusing " - "paid sampling access" - ) from exc - # The first paid-capable object is created only after all - # prompts, references, records, and sandbox provenance pass. - service = service_client_factory() - sampling_client = service.create_sampling_client( - **_sampling_client_kwargs(binding) - ) - await _sample_missing( - prepared=missing_samples, - renderer=renderer, - sampling_client=sampling_client, - store=evaluation_store, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - protocol_sha256=str(protocol["logical_sha256"]), - panel=panel, - ) - - # Validate every existing sample against this exact manifest before - # execution. A resumed run therefore cannot silently mix requests. - for item in prepared: - seed = _sample_seed( - protocol_sha256=str(protocol["logical_sha256"]), - binding=binding, - panel=panel, - task_id=item.task.task_id, - ) - sample = evaluation_store.load_sample(item.task.task_id) - if sample is None: - raise CheckpointEvaluationError( - f"{item.task.task_id} sampling did not persist a record" - ) - _validate_sample_record( - record=sample, - prepared=item, - seed=seed, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - ) - _score_missing( - prepared=prepared, - evaluator=evaluator, - store=evaluation_store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - report_payload = _report_payload( - repo_root=root, - prepared=prepared, - store=evaluation_store, - manifest_record=manifest, - binding=binding, - protocol=protocol, - source_git_sha=source_git_sha, - panel=panel, - dataset=dataset, - sandbox=sandbox, - ) - report = evaluation_store.create_or_verify_report(report_payload) - validate_checkpoint_report( - report, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - panel=panel, - expected_task_ids=[item.task.task_id for item in prepared], - ) - return { - "status": "complete", - "stage_id": binding.stage.stage_id, - "wave": wave, - "checkpoint_name": binding.checkpoint_name, - "panel": panel, - "report": str(evaluation_store.root / "report.json"), - "report_record_sha256": report["record_sha256"], - "summary": report_payload["summary"], - } - finally: - try: - evaluator.close() - finally: - if base_stage_lock is not None: - base_stage_lock.__exit__(None, None, None) - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, required=True) - result.add_argument("--stage", required=True) - result.add_argument("--wave", required=True) - result.add_argument("--checkpoint-name", required=True) - result.add_argument("--replicate", default="r0") - result.add_argument("--panel", choices=SUPPORTED_PANELS, required=True) - result.add_argument("--source-git-sha", required=True) - result.add_argument("--wave-receipt-record-sha256", default="") - result.add_argument("--base-model", action="store_true") - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--confirm-spend", default="") - return result - - -def main() -> None: - args = parser().parse_args() - report = asyncio.run( - run_checkpoint_evaluation( - repo_root=args.repo_root, - stage_id=args.stage, - wave=args.wave, - checkpoint_name=args.checkpoint_name, - replicate_id=args.replicate, - panel=args.panel, - expected_source_sha=args.source_git_sha, - expected_wave_receipt_sha256=args.wave_receipt_record_sha256, - external_root=args.external_root, - confirmation=args.confirm_spend, - base_model=args.base_model, - ) - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/final_benchmark.py b/rl/studies/representation_training_v1/final_benchmark.py deleted file mode 100644 index 0b98149a..00000000 --- a/rl/studies/representation_training_v1/final_benchmark.py +++ /dev/null @@ -1,1724 +0,0 @@ -"""Receipt-bound final-checkpoint F1--F8 best@4 evaluation. - -This is deliberately a new, versioned path rather than an extension of the -v1 checkpoint panels. The progress, depth, and promotion records therefore -keep their original one-attempt semantics and byte contracts. -""" - -from __future__ import annotations - -import argparse -import asyncio -import hashlib -import json -import math -import os -import uuid -from collections import Counter -from collections.abc import Callable, Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any - -from rl.common.evaluator import ( - Attribution, - EvaluationResult, - EvaluationStatus, - PixCellEvaluator, -) -from rl.common.prompt import prompt_asset_hashes -from rl.evaluation.tasks import EvaluationTask, canonical_json_sha256 -from rl.track_a.tinker_data import _message, _renderer - -from . import evaluation as checkpoint_v1 -from .dataset_binding import _dataset_binding -from .protocol import ( - STUDY_ID, - file_sha256, - load_protocol, - protocol_path, - validate_stage_replicate, - validate_source_sha, -) -from .store import TrainingStore - - -PANEL_FINAL_BENCHMARK_V2 = "final-benchmark-best4-v2" -FINAL_BENCHMARK_SCHEMA_VERSION = "pixcell-training-final-benchmark-v2" -FINAL_BENCHMARK_RECORD_SCHEMA_VERSION = "pixcell-training-final-benchmark-record-v2" -FINAL_BENCHMARK_TASK_PANEL_SCHEMA_VERSION = "pixcell-training-final-benchmark-task-panel-v2" -ATTEMPTS_PER_TASK = 4 -EXPECTED_TASK_IDS = tuple(f"F{index}" for index in range(1, 9)) -EXPECTED_ATTEMPTS = len(EXPECTED_TASK_IDS) * ATTEMPTS_PER_TASK -SAMPLE_CONCURRENCY = 8 -EVALUATION_BATCH_SIZE = 32 -EVALUATOR_WORKERS = 8 -MAX_IMAGE_LONG_EDGE = checkpoint_v1.MAX_IMAGE_LONG_EDGE -MAX_OUTPUT_TOKENS = checkpoint_v1.MAX_OUTPUT_TOKENS -TEMPERATURE = checkpoint_v1.TEMPERATURE -TOP_P = checkpoint_v1.TOP_P -_SEED_DOMAIN = "pixcell-representation-training-final-benchmark-v2" -_RESPONSE_FIELDS = { - "token_ids", - "completion_tokens", - "stop_reason", - "cap_hit", - "raw_text", - "raw_text_sha256", - "answer_text", - "answer_text_sha256", - "reasoning_text", - "reasoning_text_sha256", - "token_decode_complete", - "token_decode_error", - "channel_parse_complete", - "channel_parse_error", -} -_SAMPLE_PAYLOAD_FIELDS = { - "final_benchmark_manifest_record_sha256", - "training_wave_receipt_record_sha256", - "panel", - "checkpoint_name", - "checkpoint_inventory_sha256", - "checkpoint", - "task", - "attempt_index", - "request", - "response", -} -_EVALUATION_PAYLOAD_FIELDS = { - "final_benchmark_manifest_record_sha256", - "sample_record_sha256", - "panel", - "task_id", - "level", - "representation_id", - "attempt_index", - "seed", - "status", - "attribution", - "pure_executable", - "raw_absolute_scale_iou", - "dice", - "render_sha256", - "reference_sha256", - "violations", - "error", - "latency_seconds", - "diagnostics", -} -_REPORT_RECORD_FIELDS = { - "task_id", - "attempt_index", - "seed", - "status", - "attribution", - "pure_executable", - "raw_absolute_scale_iou", - "completion_tokens", - "stop_reason", - "cap_hit", - "channel_parse_complete", - "answer_text_sha256", - "render_sha256", - "reference_sha256", - "violations", - "error", - "sample_record_sha256", - "evaluation_record_sha256", -} -_REPORT_PAYLOAD_FIELDS = { - "schema_version", - "study_id", - "stage_id", - "wave", - "checkpoint_name", - "replicate_id", - "panel", - "summary", - "figures", - "records", - "attempt_records_sha256", - "provenance", -} - - -class FinalBenchmarkError(checkpoint_v1.CheckpointEvaluationError): - """The requested final benchmark is unsafe, incomplete, or inconsistent.""" - - -class ImmutableFinalBenchmarkRecordError(checkpoint_v1.ImmutableEvaluationRecordError): - """A v2 final-benchmark record cannot be replaced or mixed.""" - - -@dataclass(frozen=True) -class PreparedAttempt: - prepared: checkpoint_v1.PreparedTask - attempt_index: int - seed: int - - @property - def task(self) -> EvaluationTask: - return self.prepared.task - - @property - def key(self) -> tuple[str, int]: - return (self.task.task_id, self.attempt_index) - - -def _canonical_bytes(value: Any) -> bytes: - try: - return json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise FinalBenchmarkError("final-benchmark record is not finite JSON") from exc - - -def _canonical_sha256(value: Any) -> str: - return hashlib.sha256(_canonical_bytes(value)).hexdigest() - - -def _record_document( - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], -) -> dict[str, Any]: - normalized = json.loads(_canonical_bytes(payload)) - document = { - "schema_version": FINAL_BENCHMARK_RECORD_SCHEMA_VERSION, - "record_type": record_type, - "key": dict(key), - "payload": normalized, - "payload_sha256": _canonical_sha256(normalized), - } - document["record_sha256"] = _canonical_sha256(document) - return document - - -def _validate_record( - value: Mapping[str, Any], - *, - record_type: str, - key: Mapping[str, Any], -) -> dict[str, Any]: - required = { - "schema_version", - "record_type", - "key", - "payload", - "payload_sha256", - "record_sha256", - } - if set(value) != required: - raise FinalBenchmarkError("final-benchmark record schema changed") - if value["schema_version"] != FINAL_BENCHMARK_RECORD_SCHEMA_VERSION: - raise FinalBenchmarkError("final-benchmark record has a foreign schema") - if value["record_type"] != record_type or value["key"] != dict(key): - raise FinalBenchmarkError("final-benchmark record key differs from its canonical path") - payload = value["payload"] - if not isinstance(payload, dict): - raise FinalBenchmarkError("final-benchmark payload is not an object") - if value["payload_sha256"] != _canonical_sha256(payload): - raise FinalBenchmarkError("final-benchmark payload digest mismatch") - unsigned = dict(value) - observed = unsigned.pop("record_sha256") - if ( - not isinstance(observed, str) - or not checkpoint_v1._SHA256.fullmatch(observed) - or observed != _canonical_sha256(unsigned) - ): - raise FinalBenchmarkError("final-benchmark envelope digest mismatch") - return dict(value) - - -class FinalBenchmarkStore(checkpoint_v1.CheckpointEvaluationStore): - """Create-only v2 attempt records in a panel-isolated namespace.""" - - def __init__( - self, - *, - stage_path: Path, - wave: str, - checkpoint_name: str, - ) -> None: - if not checkpoint_v1._IDENTIFIER.fullmatch(wave) or not checkpoint_v1._IDENTIFIER.fullmatch( - checkpoint_name - ): - raise ValueError("invalid final-benchmark identity") - self.stage_path = stage_path.expanduser().resolve(strict=True) - requested = ( - self.stage_path / "evaluations" / wave / checkpoint_name / PANEL_FINAL_BENCHMARK_V2 - ) - requested.mkdir(parents=True, exist_ok=True, mode=0o700) - resolved = requested.resolve(strict=True) - if resolved != requested or not resolved.is_relative_to(self.stage_path): - raise FinalBenchmarkError( - "final-benchmark root must be a real directory inside its stage" - ) - self.root = resolved - - @staticmethod - def _attempt_component(attempt_index: int) -> str: - if ( - isinstance(attempt_index, bool) - or not isinstance(attempt_index, int) - or not 1 <= attempt_index <= ATTEMPTS_PER_TASK - ): - raise ValueError(f"attempt index must be in [1, {ATTEMPTS_PER_TASK}]") - return f"{attempt_index:06d}" - - @staticmethod - def _attempt_key(task_id: str, attempt_index: int) -> dict[str, Any]: - return { - "task_id": task_id, - "attempt_index": attempt_index, - } - - def _attempt_path( - self, - task_id: str, - attempt_index: int, - filename: str, - ) -> PurePosixPath: - task = self._task_component(task_id) - attempt = self._attempt_component(attempt_index) - return PurePosixPath("attempts", task, attempt, filename) - - def load_manifest(self) -> dict[str, Any] | None: - return self._load_v2( - PurePosixPath("manifest.json"), - record_type="final_benchmark_manifest", - key={}, - ) - - def create_or_verify_manifest( - self, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_v2( - PurePosixPath("manifest.json"), - record_type="final_benchmark_manifest", - key={}, - payload=payload, - ) - - def load_sample( - self, - task_id: str, - attempt_index: int, - ) -> dict[str, Any] | None: - return self._load_v2( - self._attempt_path(task_id, attempt_index, "sample.json"), - record_type="final_benchmark_sample", - key=self._attempt_key(task_id, attempt_index), - ) - - def create_or_verify_sample( - self, - task_id: str, - attempt_index: int, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_v2( - self._attempt_path(task_id, attempt_index, "sample.json"), - record_type="final_benchmark_sample", - key=self._attempt_key(task_id, attempt_index), - payload=payload, - ) - - def load_evaluation( - self, - task_id: str, - attempt_index: int, - ) -> dict[str, Any] | None: - return self._load_v2( - self._attempt_path(task_id, attempt_index, "evaluation.json"), - record_type="final_benchmark_evaluation", - key=self._attempt_key(task_id, attempt_index), - ) - - def create_or_verify_evaluation( - self, - task_id: str, - attempt_index: int, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_v2( - self._attempt_path(task_id, attempt_index, "evaluation.json"), - record_type="final_benchmark_evaluation", - key=self._attempt_key(task_id, attempt_index), - payload=payload, - ) - - def load_report(self) -> dict[str, Any] | None: - return self._load_v2( - PurePosixPath("report.json"), - record_type="final_benchmark_report", - key={}, - ) - - def create_or_verify_report( - self, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_v2( - PurePosixPath("report.json"), - record_type="final_benchmark_report", - key={}, - payload=payload, - ) - - def assert_attempt_inventory( - self, - attempts: Sequence[PreparedAttempt], - ) -> None: - """Reject foreign rows, alternate indices, and symlinks before resume.""" - - attempts_root = self.root / "attempts" - if not attempts_root.exists(): - return - expected_files = { - self._path( - self._attempt_path( - attempt.task.task_id, - attempt.attempt_index, - filename, - ) - ) - for attempt in attempts - for filename in ("sample.json", "evaluation.json") - } - expected_directories = {attempts_root} - for path in expected_files: - expected_directories.update(path.parents) - for path in attempts_root.rglob("*"): - if path.is_symlink(): - raise FinalBenchmarkError( - f"final-benchmark attempt inventory contains a symlink: {path}" - ) - if path.is_file() and path not in expected_files: - raise FinalBenchmarkError( - f"final-benchmark attempt inventory contains a foreign row: {path}" - ) - if path.is_dir() and path not in expected_directories: - raise FinalBenchmarkError( - "final-benchmark attempt inventory contains an unexpected " - f"task or attempt: {path}" - ) - - def _create_or_verify_v2( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], - ) -> dict[str, Any]: - expected = _record_document( - record_type=record_type, - key=key, - payload=payload, - ) - path = self._path(relative) - path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - raw = _canonical_bytes(expected) + b"\n" - temporary = path.parent / f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}" - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(temporary, flags, 0o600) - try: - with os.fdopen(fd, "wb", closefd=False) as stream: - stream.write(raw) - stream.flush() - os.fsync(stream.fileno()) - finally: - os.close(fd) - try: - os.link(temporary, path) - directory_fd = os.open(path.parent, os.O_RDONLY) - try: - os.fsync(directory_fd) - finally: - os.close(directory_fd) - except FileExistsError: - observed = self._load_v2( - relative, - record_type=record_type, - key=key, - ) - if observed is None or _canonical_bytes(observed) != _canonical_bytes(expected): - raise ImmutableFinalBenchmarkRecordError( - f"{relative} already contains different data" - ) - return observed - finally: - try: - temporary.unlink() - except FileNotFoundError: - pass - return expected - - def _load_v2( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - ) -> dict[str, Any] | None: - path = self._path(relative) - if not path.exists(): - return None - return _validate_record( - self._read(path), - record_type=record_type, - key=key, - ) - - -def _final_checkpoint( - binding: checkpoint_v1.CheckpointBinding, -) -> None: - checkpoint = binding.checkpoint - if ( - checkpoint.get("role") != "terminal" - or checkpoint.get("final") is not True - or checkpoint.get("training_progress_fraction") != 1.0 - ): - raise checkpoint_v1.CheckpointReceiptError( - "final benchmark requires the terminal checkpoint in its wave receipt" - ) - - -def _task_panel(tasks: Sequence[EvaluationTask]) -> dict[str, Any]: - entries = [ - { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "image_sha256": task.observation.image_sha256, - "target_image_sha256": task.reference.target_image_sha256, - "footprint_um": list(task.observation.footprint_um), - } - for task in tasks - ] - if tuple(item["task_id"] for item in entries) != EXPECTED_TASK_IDS: - raise checkpoint_v1.ReferencePanelFault("final benchmark must be exact ordered F1-F8") - return { - "schema_version": FINAL_BENCHMARK_TASK_PANEL_SCHEMA_VERSION, - "panel": PANEL_FINAL_BENCHMARK_V2, - "task_count": len(entries), - "task_ids": list(EXPECTED_TASK_IDS), - "logical_sha256": canonical_json_sha256(entries), - } - - -def _sample_seed( - *, - source_git_sha: str, - protocol_sha256: str, - dataset_sha256: str, - task_panel_sha256: str, - binding: checkpoint_v1.CheckpointBinding, - task_id: str, - attempt_index: int, -) -> int: - digest = hashlib.sha256( - "\0".join( - ( - _SEED_DOMAIN, - source_git_sha, - protocol_sha256, - dataset_sha256, - task_panel_sha256, - binding.stage.stage_id, - binding.wave, - binding.checkpoint_name, - binding.checkpoint_inventory_sha256, - binding.wave_receipt.record_sha256, - binding.key.replicate_id, - PANEL_FINAL_BENCHMARK_V2, - task_id, - str(attempt_index), - ) - ).encode("utf-8") - ).digest() - return int.from_bytes(digest[:4], "big") & 0x7FFFFFFF - - -def _attempts( - *, - prepared: Sequence[checkpoint_v1.PreparedTask], - source_git_sha: str, - protocol_sha256: str, - dataset_sha256: str, - task_panel_sha256: str, - binding: checkpoint_v1.CheckpointBinding, -) -> list[PreparedAttempt]: - attempts = [ - PreparedAttempt( - prepared=item, - attempt_index=attempt_index, - seed=_sample_seed( - source_git_sha=source_git_sha, - protocol_sha256=protocol_sha256, - dataset_sha256=dataset_sha256, - task_panel_sha256=task_panel_sha256, - binding=binding, - task_id=item.task.task_id, - attempt_index=attempt_index, - ), - ) - for item in prepared - for attempt_index in range(1, ATTEMPTS_PER_TASK + 1) - ] - if len(attempts) != EXPECTED_ATTEMPTS: - raise FinalBenchmarkError( - f"final benchmark resolved {len(attempts)} attempts, expected {EXPECTED_ATTEMPTS}" - ) - keys = [attempt.key for attempt in attempts] - seeds = [attempt.seed for attempt in attempts] - if len(set(keys)) != EXPECTED_ATTEMPTS: - raise FinalBenchmarkError("final benchmark resolved duplicate attempt keys") - if len(set(seeds)) != EXPECTED_ATTEMPTS: - raise FinalBenchmarkError("final benchmark deterministic seeds collided") - return attempts - - -def _seed_inventory(attempts: Sequence[PreparedAttempt]) -> list[dict[str, Any]]: - return [ - { - "task_id": attempt.task.task_id, - "attempt_index": attempt.attempt_index, - "seed": attempt.seed, - } - for attempt in attempts - ] - - -def _decode_response(renderer: Any, sequence: Any) -> dict[str, Any]: - from tinker_cookbook.renderers import get_text_content - - tokens = [int(token) for token in sequence.tokens] - token_decode_error: str | None = None - try: - raw_text = str(renderer.tokenizer.decode(tokens)) - except Exception as exc: - raw_text = "" - token_decode_error = str(exc).replace("\x00", "")[-1200:] - channel_parse_error: str | None = None - if token_decode_error is not None: - answer_text = "" - reasoning_text = "" - channel_parse_complete = False - else: - try: - message, termination = renderer.parse_response(tokens) - answer_text = str(get_text_content(message)) - reasoning_text = checkpoint_v1._message_reasoning(message) - channel_parse_complete = bool(termination.is_clean) - except Exception as exc: - # Malformed channel structure is model output. Keep the exact raw - # response and let the deterministic evaluator score extracted code. - answer_text = raw_text - reasoning_text = "" - channel_parse_complete = False - channel_parse_error = str(exc).replace("\x00", "")[-1200:] - stop_reason = str(sequence.stop_reason) - cap_hit = ( - len(tokens) >= MAX_OUTPUT_TOKENS - or "length" in stop_reason.lower() - or "max_token" in stop_reason.lower() - ) - return { - "token_ids": tokens, - "completion_tokens": len(tokens), - "stop_reason": stop_reason, - "cap_hit": cap_hit, - "raw_text": raw_text, - "raw_text_sha256": hashlib.sha256(raw_text.encode("utf-8")).hexdigest(), - "answer_text": answer_text, - "answer_text_sha256": hashlib.sha256(answer_text.encode("utf-8")).hexdigest(), - "reasoning_text": reasoning_text, - "reasoning_text_sha256": hashlib.sha256(reasoning_text.encode("utf-8")).hexdigest(), - "token_decode_complete": token_decode_error is None, - "token_decode_error": token_decode_error, - "channel_parse_complete": channel_parse_complete, - "channel_parse_error": channel_parse_error, - } - - -def _sample_payload( - *, - attempt: PreparedAttempt, - response: Mapping[str, Any], - manifest_record_sha256: str, - binding: checkpoint_v1.CheckpointBinding, -) -> dict[str, Any]: - prepared = attempt.prepared - return { - "final_benchmark_manifest_record_sha256": manifest_record_sha256, - "training_wave_receipt_record_sha256": (binding.wave_receipt.record_sha256), - "panel": PANEL_FINAL_BENCHMARK_V2, - "checkpoint_name": binding.checkpoint_name, - "checkpoint_inventory_sha256": binding.checkpoint_inventory_sha256, - "checkpoint": binding.checkpoint, - "task": { - "task_id": prepared.task.task_id, - "level": prepared.task.level, - "representation_id": prepared.task.representation_id, - "image_sha256": prepared.task.observation.image_sha256, - "target_image_sha256": prepared.task.reference.target_image_sha256, - }, - "attempt_index": attempt.attempt_index, - "request": { - "num_samples": 1, - "independent_request": True, - "seed": attempt.seed, - "max_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "prompt_tokens": prepared.prompt_tokens, - "prompt_text_sha256": prepared.prompt_text_sha256, - }, - "response": dict(response), - } - - -def _validate_response(response: Any, *, attempt: PreparedAttempt) -> None: - if not isinstance(response, Mapping) or set(response) != _RESPONSE_FIELDS: - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} raw response is incomplete") - tokens = response["token_ids"] - if ( - not isinstance(tokens, list) - or any( - isinstance(token, bool) or not isinstance(token, int) or token < 0 for token in tokens - ) - or response["completion_tokens"] != len(tokens) - ): - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} token inventory changed") - for field in ("raw_text", "answer_text", "reasoning_text"): - text = response[field] - if ( - not isinstance(text, str) - or response[f"{field}_sha256"] != hashlib.sha256(text.encode("utf-8")).hexdigest() - ): - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} {field} digest changed") - stop_reason = response["stop_reason"] - expected_cap = len(tokens) >= MAX_OUTPUT_TOKENS or ( - isinstance(stop_reason, str) - and ("length" in stop_reason.lower() or "max_token" in stop_reason.lower()) - ) - if ( - not isinstance(stop_reason, str) - or not isinstance(response["cap_hit"], bool) - or response["cap_hit"] is not expected_cap - or not isinstance(response["token_decode_complete"], bool) - or response["token_decode_complete"] is not (response["token_decode_error"] is None) - or ( - response["token_decode_error"] is not None - and not isinstance(response["token_decode_error"], str) - ) - or not isinstance(response["channel_parse_complete"], bool) - or ( - response["channel_parse_error"] is not None - and not isinstance(response["channel_parse_error"], str) - ) - or (response["channel_parse_complete"] and response["channel_parse_error"] is not None) - or ( - not response["token_decode_complete"] - and ( - response["channel_parse_complete"] - or any(response[field] for field in ("raw_text", "answer_text", "reasoning_text")) - ) - ) - ): - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} response diagnostics changed") - - -def _validate_sample_record( - *, - record: Mapping[str, Any], - attempt: PreparedAttempt, - manifest_record_sha256: str, - binding: checkpoint_v1.CheckpointBinding, -) -> None: - payload = record["payload"] - if set(payload) != _SAMPLE_PAYLOAD_FIELDS: - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} sample payload schema changed") - expected_top = { - "final_benchmark_manifest_record_sha256": manifest_record_sha256, - "training_wave_receipt_record_sha256": (binding.wave_receipt.record_sha256), - "panel": PANEL_FINAL_BENCHMARK_V2, - "checkpoint_name": binding.checkpoint_name, - "checkpoint_inventory_sha256": binding.checkpoint_inventory_sha256, - "checkpoint": binding.checkpoint, - "attempt_index": attempt.attempt_index, - } - for field, expected in expected_top.items(): - if payload.get(field) != expected: - raise ImmutableFinalBenchmarkRecordError( - f"{attempt.key} sample belongs to another final benchmark" - ) - expected_task = { - "task_id": attempt.task.task_id, - "level": attempt.task.level, - "representation_id": attempt.task.representation_id, - "image_sha256": attempt.task.observation.image_sha256, - "target_image_sha256": attempt.task.reference.target_image_sha256, - } - expected_request = { - "num_samples": 1, - "independent_request": True, - "seed": attempt.seed, - "max_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "prompt_tokens": attempt.prepared.prompt_tokens, - "prompt_text_sha256": attempt.prepared.prompt_text_sha256, - } - if payload.get("task") != expected_task or payload.get("request") != expected_request: - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} sample request changed") - _validate_response(payload.get("response"), attempt=attempt) - - -def _evaluation_payload( - *, - attempt: PreparedAttempt, - result: EvaluationResult, - sample_record_sha256: str, - manifest_record_sha256: str, -) -> dict[str, Any]: - payload = checkpoint_v1._evaluation_payload( - task=attempt.task, - result=result, - sample_record_sha256=sample_record_sha256, - manifest_record_sha256=manifest_record_sha256, - ) - if result.status is not EvaluationStatus.OK: - payload["raw_absolute_scale_iou"] = 0.0 - payload["dice"] = None - payload["pure_executable"] = False - try: - normalized = json.loads(_canonical_bytes(payload)) - except FinalBenchmarkError as exc: - raise checkpoint_v1.EvaluationInfrastructureFault( - f"{attempt.key}: evaluator returned non-finite diagnostics" - ) from exc - latency = normalized.get("latency_seconds") - dice = normalized.get("dice") - if ( - isinstance(latency, bool) - or not isinstance(latency, (int, float)) - or not math.isfinite(float(latency)) - or float(latency) < 0.0 - or ( - dice is not None - and ( - isinstance(dice, bool) - or not isinstance(dice, (int, float)) - or not math.isfinite(float(dice)) - or not 0.0 <= float(dice) <= 1.0 - ) - ) - ): - raise checkpoint_v1.EvaluationInfrastructureFault( - f"{attempt.key}: evaluator returned invalid finite metrics" - ) - normalized["final_benchmark_manifest_record_sha256"] = normalized.pop( - "evaluation_manifest_record_sha256" - ) - normalized["panel"] = PANEL_FINAL_BENCHMARK_V2 - normalized["attempt_index"] = attempt.attempt_index - normalized["seed"] = attempt.seed - return normalized - - -def _validate_evaluation_record( - *, - record: Mapping[str, Any], - attempt: PreparedAttempt, - sample_record_sha256: str, - manifest_record_sha256: str, -) -> None: - payload = record["payload"] - if set(payload) != _EVALUATION_PAYLOAD_FIELDS: - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} evaluation payload schema changed") - expected = { - "final_benchmark_manifest_record_sha256": manifest_record_sha256, - "sample_record_sha256": sample_record_sha256, - "panel": PANEL_FINAL_BENCHMARK_V2, - "task_id": attempt.task.task_id, - "level": attempt.task.level, - "representation_id": attempt.task.representation_id, - "attempt_index": attempt.attempt_index, - "seed": attempt.seed, - "attribution": Attribution.MODEL.value, - } - for field, value in expected.items(): - if payload.get(field) != value: - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} evaluation record changed") - iou = payload.get("raw_absolute_scale_iou") - latency = payload.get("latency_seconds") - if ( - isinstance(iou, bool) - or not isinstance(iou, (int, float)) - or not math.isfinite(float(iou)) - or not 0.0 <= float(iou) <= 1.0 - or isinstance(latency, bool) - or not isinstance(latency, (int, float)) - or not math.isfinite(float(latency)) - or float(latency) < 0.0 - ): - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} has invalid stored metrics") - status = payload.get("status") - pure = payload.get("pure_executable") - if ( - status not in {item.value for item in EvaluationStatus} - or not isinstance(pure, bool) - or pure is not (status == EvaluationStatus.OK.value) - or (not pure and float(iou) != 0.0) - ): - raise ImmutableFinalBenchmarkRecordError( - f"{attempt.key} has inconsistent executable status" - ) - dice = payload.get("dice") - violations = payload.get("violations") - error = payload.get("error") - diagnostics = payload.get("diagnostics") - if ( - ( - dice is not None - and ( - isinstance(dice, bool) - or not isinstance(dice, (int, float)) - or not math.isfinite(float(dice)) - or not 0.0 <= float(dice) <= 1.0 - ) - ) - or not isinstance(violations, list) - or any(not isinstance(value, str) for value in violations) - or (error is not None and not isinstance(error, str)) - or not isinstance(diagnostics, Mapping) - ): - raise ImmutableFinalBenchmarkRecordError( - f"{attempt.key} has malformed evaluator diagnostics" - ) - for field in ("render_sha256", "reference_sha256"): - value = payload.get(field) - if value is not None and not checkpoint_v1._SHA256.fullmatch(str(value)): - raise ImmutableFinalBenchmarkRecordError(f"{attempt.key} has an invalid {field}") - # Re-run strict finite normalization over every nested diagnostic. - try: - checkpoint_v1._safe_json(payload) - _canonical_bytes(payload) - except ( - checkpoint_v1.EvaluationInfrastructureFault, - FinalBenchmarkError, - ) as exc: - raise ImmutableFinalBenchmarkRecordError( - f"{attempt.key} has non-finite diagnostics" - ) from exc - - -async def _sample_missing( - *, - attempts: Sequence[PreparedAttempt], - renderer: Any, - sampling_client: Any, - store: FinalBenchmarkStore, - manifest_record_sha256: str, - binding: checkpoint_v1.CheckpointBinding, -) -> None: - import tinker - - semaphore = asyncio.Semaphore(SAMPLE_CONCURRENCY) - - async def sample_one(attempt: PreparedAttempt) -> None: - messages = [_message(attempt.task, max_image=MAX_IMAGE_LONG_EDGE)] - prompt_text = str(messages[0]["content"][1]["text"]) - if ( - hashlib.sha256(prompt_text.encode("utf-8")).hexdigest() - != attempt.prepared.prompt_text_sha256 - ): - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} prompt text changed after preflight" - ) - prompt = renderer.build_generation_prompt(messages) - if int(prompt.length) != attempt.prepared.prompt_tokens: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} prompt changed after preflight" - ) - async with semaphore: - try: - result = await sampling_client.sample_async( - prompt=prompt, - num_samples=1, - sampling_params=tinker.SamplingParams( - stop=renderer.get_stop_sequences(), - max_tokens=MAX_OUTPUT_TOKENS, - temperature=TEMPERATURE, - top_p=TOP_P, - seed=attempt.seed, - ), - ) - except Exception as exc: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} sampling failed: {exc}" - ) from exc - sequences = list(result.sequences) - if len(sequences) != 1: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} returned {len(sequences)} samples, expected 1" - ) - response = _decode_response(renderer, sequences[0]) - # This durable record is the boundary before any candidate execution. - store.create_or_verify_sample( - attempt.task.task_id, - attempt.attempt_index, - _sample_payload( - attempt=attempt, - response=response, - manifest_record_sha256=manifest_record_sha256, - binding=binding, - ), - ) - if not response["token_decode_complete"]: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} raw token IDs were persisted but could not be " - f"decoded: {response['token_decode_error']}" - ) - - tasks = [asyncio.create_task(sample_one(attempt)) for attempt in attempts] - try: - outcomes = await asyncio.gather(*tasks, return_exceptions=True) - except BaseException: - for task in tasks: - if not task.done(): - task.cancel() - await asyncio.gather(*tasks, return_exceptions=True) - raise - faults = [outcome for outcome in outcomes if isinstance(outcome, BaseException)] - if faults: - # All sibling requests have settled and every successful raw response - # is durable before the run releases its lock or reports the fault. - raise faults[0] - - -def _score_missing( - *, - attempts: Sequence[PreparedAttempt], - evaluator: PixCellEvaluator, - store: FinalBenchmarkStore, - manifest_record_sha256: str, -) -> None: - pending: list[tuple[PreparedAttempt, dict[str, Any]]] = [] - for attempt in attempts: - sample = store.load_sample(attempt.task.task_id, attempt.attempt_index) - if sample is None: - raise FinalBenchmarkError(f"{attempt.key} has no resumable sample") - response = sample["payload"]["response"] - if not response["token_decode_complete"]: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} has persisted raw token IDs that the renderer " - f"could not decode: {response['token_decode_error']}" - ) - existing = store.load_evaluation( - attempt.task.task_id, - attempt.attempt_index, - ) - if existing is not None: - _validate_evaluation_record( - record=existing, - attempt=attempt, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - ) - continue - pending.append((attempt, sample)) - - for offset in range(0, len(pending), EVALUATION_BATCH_SIZE): - batch = pending[offset : offset + EVALUATION_BATCH_SIZE] - requests = [ - ( - attempt.task.reference, - str(sample["payload"]["response"]["answer_text"]), - ) - for attempt, sample in batch - ] - try: - results = evaluator.evaluate_batch(requests) - except Exception as exc: - raise checkpoint_v1.EvaluationInfrastructureFault( - f"isolated final-benchmark batch failed: {exc}" - ) from exc - if len(results) != len(batch): - raise checkpoint_v1.EvaluationInfrastructureFault( - "isolated evaluator returned a different result count" - ) - faults: list[checkpoint_v1.CheckpointEvaluationError] = [] - for (attempt, sample), result in zip(batch, results, strict=True): - try: - payload = _evaluation_payload( - attempt=attempt, - result=result, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - ) - except ( - checkpoint_v1.ReferencePanelFault, - checkpoint_v1.EvaluationInfrastructureFault, - ) as exc: - faults.append(exc) - continue - store.create_or_verify_evaluation( - attempt.task.task_id, - attempt.attempt_index, - payload, - ) - if faults: - raise faults[0] - - -def _validate_attempt_state( - *, - attempts: Sequence[PreparedAttempt], - store: FinalBenchmarkStore, - manifest_record_sha256: str, - binding: checkpoint_v1.CheckpointBinding, -) -> tuple[list[PreparedAttempt], list[PreparedAttempt]]: - store.assert_attempt_inventory(attempts) - missing_samples: list[PreparedAttempt] = [] - missing_evaluations: list[PreparedAttempt] = [] - for attempt in attempts: - sample = store.load_sample(attempt.task.task_id, attempt.attempt_index) - evaluation = store.load_evaluation( - attempt.task.task_id, - attempt.attempt_index, - ) - if sample is None: - if evaluation is not None: - raise ImmutableFinalBenchmarkRecordError( - f"{attempt.key} has an evaluation without its raw sample" - ) - missing_samples.append(attempt) - missing_evaluations.append(attempt) - continue - _validate_sample_record( - record=sample, - attempt=attempt, - manifest_record_sha256=manifest_record_sha256, - binding=binding, - ) - response = sample["payload"]["response"] - if not response["token_decode_complete"] and evaluation is not None: - raise ImmutableFinalBenchmarkRecordError( - f"{attempt.key} has an evaluation for an undecodable response" - ) - if evaluation is None: - missing_evaluations.append(attempt) - continue - _validate_evaluation_record( - record=evaluation, - attempt=attempt, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - ) - return missing_samples, missing_evaluations - - -def _diagnostics(records: Sequence[Mapping[str, Any]]) -> dict[str, Any]: - if not records: - raise FinalBenchmarkError("cannot summarize an empty attempt set") - count = len(records) - completions = [int(record["completion_tokens"]) for record in records] - pure = sum(bool(record["pure_executable"]) for record in records) - cap_hits = sum(bool(record["cap_hit"]) for record in records) - parse_complete = sum(bool(record["channel_parse_complete"]) for record in records) - return { - "attempt_count": count, - "pure_executable_count": pure, - "pure_executable_rate": pure / count, - "cap_hit_count": cap_hits, - "cap_hit_rate": cap_hits / count, - "channel_parse_complete_count": parse_complete, - "channel_parse_complete_rate": parse_complete / count, - "statuses": dict(sorted(Counter(str(record["status"]) for record in records).items())), - "stop_reasons": dict( - sorted(Counter(str(record["stop_reason"]) for record in records).items()) - ), - "completion_tokens": { - "min": min(completions), - "max": max(completions), - "mean": sum(completions) / count, - "total": sum(completions), - }, - } - - -def _summaries( - records: Sequence[Mapping[str, Any]], -) -> tuple[dict[str, Any], dict[str, Any]]: - if len(records) != EXPECTED_ATTEMPTS: - raise FinalBenchmarkError( - f"final report has {len(records)} attempts, expected {EXPECTED_ATTEMPTS}" - ) - expected_keys = [ - (task_id, attempt_index) - for task_id in EXPECTED_TASK_IDS - for attempt_index in range(1, ATTEMPTS_PER_TASK + 1) - ] - observed_keys = [ - (str(record.get("task_id")), record.get("attempt_index")) for record in records - ] - if observed_keys != expected_keys or len(set(observed_keys)) != len(observed_keys): - raise FinalBenchmarkError( - "final report attempt inventory is incomplete, duplicated, or reordered" - ) - figures: dict[str, Any] = {} - best_scores: list[float] = [] - all_scores: list[float] = [] - for task_id in EXPECTED_TASK_IDS: - figure_records = [record for record in records if record["task_id"] == task_id] - scores = [ - _unit_metric( - record["raw_absolute_scale_iou"], - field=f"{task_id} raw_absolute_scale_iou", - ) - for record in figure_records - ] - all_scores.extend(scores) - best_scores.append(max(scores)) - figures[task_id] = { - "mean_at_1_raw_absolute_scale_iou": sum(scores) / len(scores), - "best_at_4_raw_absolute_scale_iou": max(scores), - **_diagnostics(figure_records), - } - summary = { - "figure_count": len(EXPECTED_TASK_IDS), - "mean_at_1_raw_absolute_scale_iou": sum(all_scores) / len(all_scores), - "best_at_4_raw_absolute_scale_iou": (sum(best_scores) / len(best_scores)), - **_diagnostics(records), - } - return summary, figures - - -def _unit_metric(value: Any, *, field: str) -> float: - if ( - isinstance(value, bool) - or not isinstance(value, (int, float)) - or not math.isfinite(float(value)) - or not 0.0 <= float(value) <= 1.0 - ): - raise FinalBenchmarkError(f"{field} must be finite and in [0, 1]") - return float(value) - - -def _report_payload( - *, - attempts: Sequence[PreparedAttempt], - store: FinalBenchmarkStore, - manifest_record: Mapping[str, Any], - binding: checkpoint_v1.CheckpointBinding, -) -> dict[str, Any]: - records: list[dict[str, Any]] = [] - manifest_sha = str(manifest_record["record_sha256"]) - for attempt in attempts: - sample = store.load_sample(attempt.task.task_id, attempt.attempt_index) - evaluation = store.load_evaluation( - attempt.task.task_id, - attempt.attempt_index, - ) - if sample is None or evaluation is None: - raise FinalBenchmarkError(f"{attempt.key} is incomplete") - _validate_sample_record( - record=sample, - attempt=attempt, - manifest_record_sha256=manifest_sha, - binding=binding, - ) - _validate_evaluation_record( - record=evaluation, - attempt=attempt, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_sha, - ) - response = sample["payload"]["response"] - measured = evaluation["payload"] - records.append( - { - "task_id": attempt.task.task_id, - "attempt_index": attempt.attempt_index, - "seed": attempt.seed, - "status": measured["status"], - "attribution": measured["attribution"], - "pure_executable": measured["pure_executable"], - "raw_absolute_scale_iou": measured["raw_absolute_scale_iou"], - "completion_tokens": response["completion_tokens"], - "stop_reason": response["stop_reason"], - "cap_hit": response["cap_hit"], - "channel_parse_complete": response["channel_parse_complete"], - "answer_text_sha256": response["answer_text_sha256"], - "render_sha256": measured["render_sha256"], - "reference_sha256": measured["reference_sha256"], - "violations": measured["violations"], - "error": measured["error"], - "sample_record_sha256": sample["record_sha256"], - "evaluation_record_sha256": evaluation["record_sha256"], - } - ) - summary, figures = _summaries(records) - return { - "schema_version": FINAL_BENCHMARK_SCHEMA_VERSION, - "study_id": STUDY_ID, - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "replicate_id": binding.key.replicate_id, - "panel": PANEL_FINAL_BENCHMARK_V2, - "summary": summary, - "figures": figures, - "records": records, - "attempt_records_sha256": canonical_json_sha256(records), - "provenance": { - "final_benchmark_manifest_record_sha256": manifest_sha, - "manifest": manifest_record["payload"], - }, - } - - -def validate_final_benchmark_report( - record: Mapping[str, Any], - *, - manifest_record: Mapping[str, Any], - binding: checkpoint_v1.CheckpointBinding, - attempts: Sequence[PreparedAttempt], -) -> dict[str, Any]: - """Validate a complete best@4 report without trusting claimed aggregates.""" - - validated = _validate_record( - record, - record_type="final_benchmark_report", - key={}, - ) - payload = validated["payload"] - if set(payload) != _REPORT_PAYLOAD_FIELDS: - raise FinalBenchmarkError("final-benchmark report payload schema changed") - expected_identity = { - "schema_version": FINAL_BENCHMARK_SCHEMA_VERSION, - "study_id": STUDY_ID, - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "replicate_id": binding.key.replicate_id, - "panel": PANEL_FINAL_BENCHMARK_V2, - } - for field, expected in expected_identity.items(): - if payload.get(field) != expected: - raise FinalBenchmarkError(f"final-benchmark report {field} differs from its request") - records = payload.get("records") - provenance = payload.get("provenance") - if not isinstance(records, list) or not isinstance(provenance, Mapping): - raise FinalBenchmarkError("final-benchmark report is incomplete") - if ( - provenance.get("final_benchmark_manifest_record_sha256") != manifest_record["record_sha256"] - or provenance.get("manifest") != manifest_record["payload"] - ): - raise FinalBenchmarkError("final-benchmark report provenance differs from its manifest") - if len(attempts) != EXPECTED_ATTEMPTS: - raise FinalBenchmarkError("validator did not receive the sealed 32 attempts") - expected_keys = [attempt.key for attempt in attempts] - observed_keys: list[tuple[str, int]] = [] - for row, attempt in zip(records, attempts, strict=False): - if not isinstance(row, Mapping) or set(row) != _REPORT_RECORD_FIELDS: - raise FinalBenchmarkError("final-benchmark attempt row is incomplete") - key = (str(row.get("task_id")), row.get("attempt_index")) - observed_keys.append(key) - if key != attempt.key or row.get("seed") != attempt.seed: - raise FinalBenchmarkError( - "final-benchmark attempt row has the wrong task, index, or seed" - ) - _unit_metric( - row.get("raw_absolute_scale_iou"), - field=f"{attempt.key} raw_absolute_scale_iou", - ) - if ( - row.get("attribution") != Attribution.MODEL.value - or row.get("status") not in {item.value for item in EvaluationStatus} - or not isinstance(row.get("pure_executable"), bool) - or not isinstance(row.get("cap_hit"), bool) - or not isinstance(row.get("channel_parse_complete"), bool) - or isinstance(row.get("completion_tokens"), bool) - or not isinstance(row.get("completion_tokens"), int) - or row["completion_tokens"] < 0 - ): - raise FinalBenchmarkError(f"{attempt.key} attempt diagnostics are invalid") - expected_cap = ( - row["completion_tokens"] >= MAX_OUTPUT_TOKENS - or "length" in str(row["stop_reason"]).lower() - or "max_token" in str(row["stop_reason"]).lower() - ) - if ( - row["pure_executable"] is not (row["status"] == EvaluationStatus.OK.value) - or (not row["pure_executable"] and float(row["raw_absolute_scale_iou"]) != 0.0) - or not isinstance(row.get("stop_reason"), str) - or row["cap_hit"] is not expected_cap - or not isinstance(row.get("violations"), list) - or any(not isinstance(value, str) for value in row["violations"]) - or (row.get("error") is not None and not isinstance(row["error"], str)) - ): - raise FinalBenchmarkError(f"{attempt.key} attempt status is internally inconsistent") - for field in ( - "answer_text_sha256", - "sample_record_sha256", - "evaluation_record_sha256", - ): - if not checkpoint_v1._SHA256.fullmatch(str(row.get(field, ""))): - raise FinalBenchmarkError(f"{attempt.key} {field} is not a SHA-256") - for field in ("render_sha256", "reference_sha256"): - value = row.get(field) - if value is not None and not checkpoint_v1._SHA256.fullmatch(str(value)): - raise FinalBenchmarkError(f"{attempt.key} {field} is not a SHA-256") - if ( - len(records) != EXPECTED_ATTEMPTS - or observed_keys != expected_keys - or len(set(observed_keys)) != EXPECTED_ATTEMPTS - ): - raise FinalBenchmarkError("final-benchmark report has incomplete or duplicate attempt rows") - summary, figures = _summaries(records) - if payload.get("summary") != summary or payload.get("figures") != figures: - raise FinalBenchmarkError("final-benchmark claimed metrics differ from its 32 attempts") - if payload.get("attempt_records_sha256") != canonical_json_sha256(records): - raise FinalBenchmarkError("final-benchmark attempt-set digest changed") - return validated - - -def _manifest_payload( - *, - repo_root: Path, - protocol: Mapping[str, Any], - source_git_sha: str, - dataset: Mapping[str, Any], - task_panel: Mapping[str, Any], - model: Mapping[str, Any], - preflight: Mapping[str, Any], - sandbox: Mapping[str, Any], - binding: checkpoint_v1.CheckpointBinding, - attempts: Sequence[PreparedAttempt], -) -> dict[str, Any]: - seeds = _seed_inventory(attempts) - return { - "schema_version": FINAL_BENCHMARK_SCHEMA_VERSION, - "study_id": STUDY_ID, - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "replicate_id": binding.key.replicate_id, - "panel": PANEL_FINAL_BENCHMARK_V2, - "source_git_sha": source_git_sha, - "protocol": { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - }, - "dataset": { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": "fixed-f1-f8", - **dict(dataset), - }, - "task_panel": dict(task_panel), - "prompt": { - "contract_version": protocol["contract_version"], - "assets": prompt_asset_hashes(), - "prompt_set_sha256": preflight["prompt_set_sha256"], - "prompt_tokens": preflight["prompt_tokens"], - "minimum_context_headroom": preflight["minimum_context_headroom"], - }, - "model": { - "model_key": binding.stage.model_key, - **dict(model), - }, - "sampler": { - "sampler_path": binding.sampler_path, - "checkpoint_name": binding.checkpoint_name, - "checkpoint": binding.checkpoint, - "checkpoint_inventory_sha256": (binding.checkpoint_inventory_sha256), - "training_run_manifest_record_sha256": (binding.run_manifest.record_sha256), - "training_wave_receipt_record_sha256": (binding.wave_receipt.record_sha256), - "training_wave_receipt_relative_path": str(binding.wave_receipt.relative_path), - }, - "sandbox": dict(sandbox), - "reference_set_sha256": preflight["reference_set_sha256"], - "sampling": { - "attempts_per_task": ATTEMPTS_PER_TASK, - "task_count": len(EXPECTED_TASK_IDS), - "total_attempts": EXPECTED_ATTEMPTS, - "independent_requests": True, - "num_samples_per_request": 1, - "max_output_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "max_image_long_edge": MAX_IMAGE_LONG_EDGE, - "sample_concurrency": SAMPLE_CONCURRENCY, - "evaluation_batch_size": EVALUATION_BATCH_SIZE, - "evaluator_workers": EVALUATOR_WORKERS, - "seed_algorithm": ("sha256-bound deterministic 31-bit seed with attempt index"), - "seed_domain": _SEED_DOMAIN, - "seeds": seeds, - "seeds_sha256": canonical_json_sha256(seeds), - }, - } - - -def _default_evaluator_factory() -> PixCellEvaluator: - return PixCellEvaluator( - max_workers=EVALUATOR_WORKERS, - evaluator_retries=1, - require_isolation=True, - ) - - -def _default_service_client_factory() -> Any: - import tinker - - return tinker.ServiceClient() - - -async def run_final_benchmark( - *, - repo_root: Path, - stage_id: str, - wave: str, - checkpoint_name: str, - replicate_id: str, - expected_source_sha: str, - expected_wave_receipt_sha256: str, - external_root: Path, - confirmation: str, - service_client_factory: Callable[[], Any] = _default_service_client_factory, - evaluator_factory: Callable[[], PixCellEvaluator] = _default_evaluator_factory, -) -> dict[str, Any]: - """Run or resume the exact 32-attempt final-checkpoint best@4 panel.""" - - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol(root, require_committed=True) - source_git_sha = validate_source_sha(root, expected_source_sha) - if confirmation != protocol["launch"]["confirmation_token"]: - raise FinalBenchmarkError("explicit paid-evaluation confirmation is missing") - try: - validate_stage_replicate( - protocol, - stage_id=stage_id, - replicate_id=replicate_id, - ) - except ValueError as exc: - raise FinalBenchmarkError(str(exc)) from exc - if protocol["evaluation"].get("final_benchmark_attempts") != ATTEMPTS_PER_TASK: - raise FinalBenchmarkError("protocol no longer seals four final-benchmark attempts") - - training_store = TrainingStore( - repo_root=root, - external_root=external_root, - ) - binding = checkpoint_v1._load_checkpoint_binding( - store=training_store, - protocol=protocol, - stage_id=stage_id, - wave=wave, - checkpoint_name=checkpoint_name, - replicate_id=replicate_id, - expected_receipt_sha256=expected_wave_receipt_sha256, - source_git_sha=source_git_sha, - ) - _final_checkpoint(binding) - model = checkpoint_v1._model_binding(protocol, binding.stage) - dataset = _dataset_binding(root / "dataset", protocol) - tasks = checkpoint_v1._benchmark_tasks(repo_root=root, protocol=protocol) - task_panel = _task_panel(tasks) - renderer = _renderer( - model["model"], - model["renderer"], - effort=model.get("effort"), - ) - evaluator = evaluator_factory() - try: - prepared, preflight = checkpoint_v1._preflight_tasks( - tasks=tasks, - renderer=renderer, - evaluator=evaluator, - context_tokens=int(model["context_tokens"]), - ) - sandbox = checkpoint_v1._sandbox_provenance(evaluator) - attempts = _attempts( - prepared=prepared, - source_git_sha=source_git_sha, - protocol_sha256=str(protocol["logical_sha256"]), - dataset_sha256=str(dataset["logical_release_sha256"]), - task_panel_sha256=str(task_panel["logical_sha256"]), - binding=binding, - ) - store = FinalBenchmarkStore( - stage_path=training_store.stage_path(binding.key), - wave=wave, - checkpoint_name=binding.checkpoint_name, - ) - with store.acquire_lock(): - manifest_payload = _manifest_payload( - repo_root=root, - protocol=protocol, - source_git_sha=source_git_sha, - dataset=dataset, - task_panel=task_panel, - model=model, - preflight=preflight, - sandbox=sandbox, - binding=binding, - attempts=attempts, - ) - manifest = store.create_or_verify_manifest(manifest_payload) - missing_samples, missing_evaluations = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - ) - existing_report = store.load_report() - if existing_report is not None: - if missing_samples or missing_evaluations: - raise FinalBenchmarkError( - "final-benchmark report exists with incomplete attempt rows" - ) - expected_payload = _report_payload( - attempts=attempts, - store=store, - manifest_record=manifest, - binding=binding, - ) - observed = store.create_or_verify_report(expected_payload) - validate_final_benchmark_report( - observed, - manifest_record=manifest, - binding=binding, - attempts=attempts, - ) - return { - "status": "already_complete", - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "panel": PANEL_FINAL_BENCHMARK_V2, - "report": str(store.root / "report.json"), - "report_record_sha256": observed["record_sha256"], - "summary": observed["payload"]["summary"], - } - - if missing_samples: - if not os.environ.get("TINKER_API_KEY"): - raise FinalBenchmarkError("TINKER_API_KEY is not present") - # Every prompt, reference, existing record, receipt, sandbox, - # seed, and manifest has passed before this paid-capable object. - service = service_client_factory() - sampling_client = service.create_sampling_client(model_path=binding.sampler_path) - await _sample_missing( - attempts=missing_samples, - renderer=renderer, - sampling_client=sampling_client, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - ) - - missing_samples, _ = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - ) - if missing_samples: - raise FinalBenchmarkError("sampling did not persist all 32 raw responses") - _score_missing( - attempts=attempts, - evaluator=evaluator, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - missing_samples, missing_evaluations = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - ) - if missing_samples or missing_evaluations: - raise FinalBenchmarkError( - "final benchmark is incomplete after deterministic evaluation" - ) - report_payload = _report_payload( - attempts=attempts, - store=store, - manifest_record=manifest, - binding=binding, - ) - report = store.create_or_verify_report(report_payload) - validate_final_benchmark_report( - report, - manifest_record=manifest, - binding=binding, - attempts=attempts, - ) - return { - "status": "complete", - "stage_id": binding.stage.stage_id, - "wave": binding.wave, - "checkpoint_name": binding.checkpoint_name, - "panel": PANEL_FINAL_BENCHMARK_V2, - "report": str(store.root / "report.json"), - "report_record_sha256": report["record_sha256"], - "summary": report_payload["summary"], - } - finally: - evaluator.close() - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, required=True) - result.add_argument("--stage", required=True) - result.add_argument("--wave", required=True) - result.add_argument("--checkpoint-name", required=True) - result.add_argument("--replicate", default="r0") - result.add_argument("--source-git-sha", required=True) - result.add_argument("--wave-receipt-record-sha256", required=True) - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--confirm-spend", default="") - return result - - -def main() -> None: - args = parser().parse_args() - report = asyncio.run( - run_final_benchmark( - repo_root=args.repo_root, - stage_id=args.stage, - wave=args.wave, - checkpoint_name=args.checkpoint_name, - replicate_id=args.replicate, - expected_source_sha=args.source_git_sha, - expected_wave_receipt_sha256=args.wave_receipt_record_sha256, - external_root=args.external_root, - confirmation=args.confirm_spend, - ) - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/launcher.py b/rl/studies/representation_training_v1/launcher.py deleted file mode 100644 index 38463b36..00000000 --- a/rl/studies/representation_training_v1/launcher.py +++ /dev/null @@ -1,2746 +0,0 @@ -"""Sealed, resumable launch controller for representation-training-v1.""" - -from __future__ import annotations - -import hashlib -import importlib -import json -import math -import os -import re -import subprocess -import uuid -from collections.abc import Awaitable, Callable, Iterator, Mapping -from contextlib import contextmanager -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from tinker_cookbook import checkpoint_utils -from tinker_cookbook.rl.train import Config as RLConfig -from tinker_cookbook.supervised.train import Config as SFTConfig - -from rl.evaluation.tracking import validate_wandb_access -from rl.track_a.tinker_data import ( - TrackARLDatasetBuilder, - TrackASupervisedDatasetBuilder, -) - -from .preflight import stage_preflight -from .promotion import ( - APPROVAL_SCHEMA_VERSION, - LEVEL_PROGRESS_PANELS, - PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION, - SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION, - PromotionError, - record_base_baseline_approval, - record_rollout_health_approval, - validate_parent_receipt_transition_approval, - validate_same_source_wave_approval, - validate_source_transition_approval, -) -from .protocol import ( - LEVELS, - STUDY_ID, - StageSpec, - file_sha256, - load_protocol, - parent_replicate_id, - stage_spec, - validate_stage_replicate, - validate_source_sha, -) -from .schedule import SCHEDULE_SEED -from .store import StageKey, TrainingRecord, TrainingStore, canonical_sha256 - - -WANDB_RUN_ID_PREFIX = "pxct-" -_TINKER_PATH = re.compile(r"^tinker://[^\s]+$") -_GIT_SHA = re.compile(r"^[0-9a-f]{40}$") -_SHA256 = re.compile(r"^[0-9a-f]{64}$") -_CHILD_SOURCE_TRANSITION_SCHEMA_VERSION = "pixcell-training-child-source-transition-v1" -_WAVE_ARTIFACT_SNAPSHOT_SCHEMA_VERSION = "pixcell-training-wave-artifact-snapshot-v1" -_COOKBOOK_FRESH_RESUME_BOOTSTRAP_FILES = frozenset( - {"code.diff", "config.json", "logs.log"} -) -_ENVIRONMENT_FIELDS = ( - "WANDB_DIR", - "WANDB_DISABLE_CODE", - "WANDB_DISABLE_GIT", - "WANDB_ENTITY", - "WANDB_JOB_TYPE", - "WANDB_MODE", - "WANDB_NAME", - "WANDB_PROJECT", - "WANDB_RESUME", - "WANDB_RUN_GROUP", - "WANDB_RUN_ID", - "WANDB_SAVE_CODE", - "WANDB_TAGS", -) - - -class TrainingLaunchError(RuntimeError): - """A paid stage is ambiguous, inconsistent, or not approved.""" - - -@dataclass(frozen=True) -class ParentBinding: - stage_id: str - wave: str - replicate_id: str - receipt: TrainingRecord - checkpoint: dict[str, Any] - source_git_sha: str - run_manifest: TrainingRecord - entry_approval: TrainingRecord | None = None - source_transition: dict[str, Any] | None = None - - -@dataclass(frozen=True) -class ResumeBinding: - """Exact optimizer state the pinned Cookbook is allowed to resume.""" - - batch: int - checkpoint: dict[str, Any] - - -def _child_source_transition_binding( - approval: TrainingRecord, -) -> dict[str, Any]: - payload = approval.payload - if payload.get("schema_version") not in { - SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION, - PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION, - }: - raise TrainingLaunchError( - "child source transition is not backed by an approved transition" - ) - transition = payload.get("source_transition") - if not isinstance(transition, Mapping): - raise TrainingLaunchError("transition approval has no source transition") - logical_sha256 = str(transition.get("logical_sha256", "")) - if not re.fullmatch(r"[0-9a-f]{64}", logical_sha256): - raise TrainingLaunchError( - "transition approval source transition has no logical digest" - ) - return { - "schema_version": _CHILD_SOURCE_TRANSITION_SCHEMA_VERSION, - "authorized_child_wave": str(payload.get("wave", "")), - "producer_source_git_sha": str(payload.get("evidence_source_git_sha", "")), - "consumer_source_git_sha": str(payload.get("source_git_sha", "")), - "approval_schema_version": str(payload.get("schema_version", "")), - "approval": { - "relative_path": str(approval.relative_path), - "payload_sha256": approval.payload_sha256, - "record_sha256": approval.record_sha256, - }, - "source_transition_logical_sha256": logical_sha256, - "source_transition": dict(transition), - } - - -def deterministic_wandb_run_id( - *, - source_git_sha: str, - protocol_logical_sha256: str, - key: StageKey, -) -> str: - digest = canonical_sha256( - { - "schema_version": "pixcell-training-wandb-identity-v1", - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol_logical_sha256, - **key.as_dict(), - } - ) - return WANDB_RUN_ID_PREFIX + digest[:27] - - -def _tracking_binding( - *, - protocol: Mapping[str, Any], - source_git_sha: str, - key: StageKey, - stage: StageSpec, - store: TrainingStore, -) -> dict[str, Any]: - model_family = "inkling" if stage.model_key == "inkling" else "qwen-open" - run_id = deterministic_wandb_run_id( - source_git_sha=source_git_sha, - protocol_logical_sha256=str(protocol["logical_sha256"]), - key=key, - ) - return { - "provider": "wandb", - "entity": protocol["tracking"]["entity"], - "project": protocol["tracking"]["project"], - "run_id": run_id, - "name": (f"{STUDY_ID}/{stage.stage_id}/{key.replicate_id}@{source_git_sha[:12]}"), - "group": f"{STUDY_ID}/{model_family}", - "job_type": stage.kind, - "tags": [ - STUDY_ID, - stage.stage_id, - stage.model_key, - stage.kind, - key.replicate_id, - *stage.hypotheses, - ], - "directory": str(store.stage_path(key) / "wandb"), - } - - -@contextmanager -def _wandb_environment( - binding: Mapping[str, Any], - *, - resume: bool, -) -> Iterator[None]: - previous = {name: os.environ.get(name) for name in _ENVIRONMENT_FIELDS} - directory = Path(str(binding["directory"])) - directory.mkdir(parents=True, exist_ok=True, mode=0o700) - os.environ.update( - { - "WANDB_DIR": str(directory), - "WANDB_DISABLE_CODE": "true", - "WANDB_DISABLE_GIT": "true", - "WANDB_ENTITY": str(binding["entity"]), - "WANDB_JOB_TYPE": str(binding["job_type"]), - "WANDB_MODE": "online", - "WANDB_NAME": str(binding["name"]), - "WANDB_PROJECT": str(binding["project"]), - "WANDB_RESUME": "must" if resume else "never", - "WANDB_RUN_GROUP": str(binding["group"]), - "WANDB_RUN_ID": str(binding["run_id"]), - "WANDB_SAVE_CODE": "false", - "WANDB_TAGS": ",".join(str(item) for item in binding["tags"]), - } - ) - try: - yield - finally: - for name, value in previous.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value - - -def _finish_wandb_run(wandb_module: Any, run: Any) -> None: - """Best-effort cleanup after rejecting a live W&B binding.""" - - finish = getattr(run, "finish", None) - if callable(finish): - try: - finish() - return - except BaseException: - pass - module_finish = getattr(wandb_module, "finish", None) - if callable(module_finish): - try: - module_finish() - except BaseException: - pass - - -@contextmanager -def _sealed_wandb_initialization( - binding: Mapping[str, Any], - *, - resume: bool, - wandb_module: Any | None = None, - on_initialized: Callable[[Mapping[str, Any]], None] | None = None, -) -> Iterator[None]: - """Force Cookbook logging onto the exact run sealed in the manifest. - - The Cookbook intentionally owns the call to ``wandb.init``. This narrow - wrapper preserves that integration while replacing identity-bearing - arguments with the immutable launch binding. The observed live run is - checked and recorded before the Cookbook can construct a Tinker client. - """ - - module = wandb_module if wandb_module is not None else importlib.import_module("wandb") - if getattr(module, "run", None) is not None: - raise TrainingLaunchError("a W&B run is already active before sealed initialization") - original_init = getattr(module, "init", None) - if not callable(original_init): - raise TrainingLaunchError("the W&B module has no callable init") - init_calls = 0 - initialized_run: Any | None = None - - def sealed_init(*args: Any, **kwargs: Any) -> Any: - nonlocal init_calls, initialized_run - init_calls += 1 - if init_calls != 1: - raise TrainingLaunchError("the training process attempted multiple W&B initializations") - if args: - raise TrainingLaunchError("the training process used positional W&B identity arguments") - for field in ("project", "name"): - supplied = kwargs.get(field) - if supplied is not None and str(supplied) != str(binding[field]): - raise TrainingLaunchError(f"the training process supplied a different W&B {field}") - cookbook_config = kwargs.get("config") - sealed = dict(kwargs) - sealed.update( - { - # Cookbook logs the same config again immediately after init. - # RL callback fields are sanitized differently across the two - # W&B calls, so initialize identity first and write hparams - # through the same canonical post-init path both times. - "config": None, - "entity": str(binding["entity"]), - "project": str(binding["project"]), - "dir": str(binding["directory"]), - "id": str(binding["run_id"]), - "name": str(binding["name"]), - "group": str(binding["group"]), - "job_type": str(binding["job_type"]), - "tags": tuple(str(item) for item in binding["tags"]), - "mode": "online", - "resume": "must" if resume else "never", - "save_code": False, - } - ) - run = original_init(**sealed) - initialized_run = run - settings = getattr(run, "settings", None) - observed = { - "run_id": str(getattr(run, "id", "")), - "entity": str(getattr(run, "entity", "")), - "project": str(getattr(run, "project", "")), - "name": str(getattr(run, "name", "")), - "group": str(getattr(run, "group", "")), - "job_type": str(getattr(run, "job_type", "")), - "tags": [str(item) for item in getattr(run, "tags", ())], - "resumed": bool(getattr(run, "resumed", False)), - "url": str(getattr(run, "url", "")), - "requested_directory": str(binding["directory"]), - "resolved_root_directory": str(getattr(settings, "root_dir", "")), - "files_directory": str(getattr(run, "dir", "")), - "requested_resume": "must" if resume else "never", - "resolved_resume": str(getattr(settings, "resume", "")), - } - expected = { - "run_id": str(binding["run_id"]), - "entity": str(binding["entity"]), - "project": str(binding["project"]), - "name": str(binding["name"]), - "group": str(binding["group"]), - "job_type": str(binding["job_type"]), - "tags": [str(item) for item in binding["tags"]], - "resumed": resume, - "resolved_resume": "must" if resume else "never", - } - differences = [field for field, value in expected.items() if observed[field] != value] - requested_directory = Path(str(binding["directory"])).resolve(strict=False) - try: - resolved_directory = Path(observed["resolved_root_directory"]).resolve(strict=True) - except (OSError, RuntimeError): - differences.append("resolved_root_directory") - else: - if resolved_directory != requested_directory: - differences.append("resolved_root_directory") - try: - files_directory = Path(observed["files_directory"]).resolve(strict=True) - except (OSError, RuntimeError): - differences.append("files_directory") - else: - if not files_directory.is_relative_to(requested_directory): - differences.append("files_directory") - if differences: - _finish_wandb_run(module, run) - raise TrainingLaunchError( - "the live W&B run differs from its sealed " + ", ".join(differences) - ) - if cookbook_config is not None: - config = getattr(module, "config", None) - update = getattr(config, "update", None) - if not callable(update): - _finish_wandb_run(module, run) - raise TrainingLaunchError("the live W&B run has no configurable hparam store") - try: - # Use Config.update for both writes. W&B sanitizes callable - # fields differently in wandb.init, while the Cookbook writes - # the same hparams again immediately after init. Continuation - # waves intentionally change their ceiling and invocation - # metadata, so only the preseed update may replace prior-wave - # values; the Cookbook's following update is then identical. - update(cookbook_config, allow_val_change=resume) - except BaseException: - _finish_wandb_run(module, run) - raise - observed["hparams_preseeded"] = True - observed["hparams_allow_val_change"] = resume - else: - observed["hparams_preseeded"] = False - observed["hparams_allow_val_change"] = False - try: - if on_initialized is not None: - on_initialized(observed) - except BaseException: - _finish_wandb_run(module, run) - raise - return run - - module.init = sealed_init - try: - yield - except BaseException: - if initialized_run is not None: - _finish_wandb_run(module, initialized_run) - raise - else: - if init_calls != 1: - if initialized_run is not None: - _finish_wandb_run(module, initialized_run) - raise TrainingLaunchError("training returned without initializing its sealed W&B run") - finally: - module.init = original_init - - -def _checkpoint_from_inventory( - receipt: TrainingRecord, - selected: Mapping[str, Any], - *, - inventory_sha256: Any, -) -> dict[str, Any]: - inventory = receipt.payload.get("checkpoint_inventory") - if not isinstance(inventory, Mapping): - raise TrainingLaunchError("parent receipt has no checkpoint inventory") - if ( - inventory.get("logical_sha256") != inventory_sha256 - or canonical_sha256(inventory.get("entries")) != inventory_sha256 - ): - raise TrainingLaunchError("parent checkpoint inventory digest differs") - entries = inventory.get("entries") - if not isinstance(entries, list) or dict(selected) not in entries: - raise TrainingLaunchError("selected parent checkpoint is not in the parent receipt") - checkpoint = dict(selected) - if not _TINKER_PATH.fullmatch( - str(checkpoint.get("state_path", "")) - ) or not _TINKER_PATH.fullmatch(str(checkpoint.get("sampler_path", ""))): - raise TrainingLaunchError("selected parent checkpoint has invalid URIs") - return checkpoint - - -def _manifest_transition_approval( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - key: StageKey, - stage: StageSpec, - source_git_sha: str, - existing_manifest: TrainingRecord, -) -> TrainingRecord: - observed = existing_manifest.payload.get("source_transition") - if not isinstance(observed, Mapping): - raise TrainingLaunchError("cross-source parent requires a transition in the child manifest") - authorized_wave = str(observed.get("authorized_child_wave", "")) - if authorized_wave not in stage.waves: - raise TrainingLaunchError("child manifest transition names an unknown wave") - approval = store.load_wave_approval(key, wave=authorized_wave) - if approval is None: - raise TrainingLaunchError("child manifest transition approval is missing") - try: - if ( - approval.payload.get("schema_version") - == PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ): - validate_parent_receipt_transition_approval( - repo_root=repo_root, - protocol=protocol, - store=store, - stage=stage, - wave_name=authorized_wave, - replicate_id=key.replicate_id, - consumer_source_git_sha=source_git_sha, - approval=approval, - ) - else: - validate_source_transition_approval( - repo_root=repo_root, - protocol=protocol, - store=store, - stage=stage, - wave_name=authorized_wave, - replicate_id=key.replicate_id, - consumer_source_git_sha=source_git_sha, - approval=approval, - ) - except PromotionError as exc: - raise TrainingLaunchError( - f"child manifest source transition failed validation: {exc}" - ) from exc - expected = _child_source_transition_binding(approval) - if dict(observed) != expected: - raise TrainingLaunchError("child manifest source transition differs from its v2 approval") - return approval - - -def _parent_binding( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - key: StageKey, - stage: StageSpec, - source_git_sha: str, - approval: TrainingRecord | None, - existing_manifest: TrainingRecord | None, -) -> ParentBinding | None: - if stage.parent.startswith("base:"): - expected_model = stage.parent.split(":", 1)[1] - if expected_model != stage.model_key: - raise TrainingLaunchError("base parent and stage model differ") - if existing_manifest is not None and "source_transition" in existing_manifest.payload: - raise TrainingLaunchError("base-parent child manifest contains a source transition") - return None - - parent_stage_id, parent_wave = stage.parent.split(":", 1) - if parent_wave == "complete-or-skip": - frozen_parent = ( - existing_manifest.payload.get("parent_receipt") - if existing_manifest is not None - else None - ) - if isinstance(frozen_parent, Mapping): - selected_stage = str(frozen_parent.get("stage_id", "")) - selected_wave = str(frozen_parent.get("wave", "")) - elif approval is not None: - selected_stage = str(approval.payload.get("parent_stage_id", "")) - selected_wave = str(approval.payload.get("parent_wave", "")) - else: - raise TrainingLaunchError(f"{stage.stage_id} requires an approved dynamic parent") - allowed = {"qwen-mixed-sft"} - target_index = int(str(stage.current_level)[1:]) - allowed.update(f"qwen-mixed-rl-l{index}" for index in range(target_index)) - if selected_stage not in allowed or selected_wave != "complete": - raise TrainingLaunchError(f"{stage.stage_id} approval selected an invalid parent") - parent_stage_id, parent_wave = selected_stage, selected_wave - - parent_replicate = parent_replicate_id( - protocol, - stage_id=stage.stage_id, - replicate_id=key.replicate_id, - ) - parent_key = StageKey(key.study_id, parent_stage_id, parent_replicate) - parent_manifest = store.load_manifest(parent_key) - receipt = store.load_wave_receipt(parent_key, wave=parent_wave) - if parent_manifest is None or receipt is None: - raise TrainingLaunchError(f"{stage.stage_id} requires {parent_stage_id}/{parent_wave}") - parent_source_git_sha = str(parent_manifest.payload.get("source_git_sha", "")) - if not _GIT_SHA.fullmatch(parent_source_git_sha): - raise TrainingLaunchError("parent run manifest has an invalid source commit") - if ( - receipt.payload.get("stage_id") != parent_stage_id - or receipt.payload.get("wave") != parent_wave - or receipt.payload.get("run_manifest_record_sha256") != parent_manifest.record_sha256 - ): - raise TrainingLaunchError("parent receipt and run manifest identities differ") - if ( - parent_manifest.payload.get("contract_version") != protocol["contract_version"] - ): - raise TrainingLaunchError("parent run manifest differs from the consumer protocol") - historical_parent_transition = ( - ( - ( - approval is not None - and approval.payload.get("schema_version") - == PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ) - or ( - existing_manifest is not None - and isinstance( - existing_manifest.payload.get("source_transition"), - Mapping, - ) - and existing_manifest.payload["source_transition"].get( - "approval_schema_version" - ) - == PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ) - ) - and stage.stage_id == "qwen-l0-rl-l1" - and parent_stage_id == "qwen-l0-sft" - and parent_wave == "complete" - ) - if ( - parent_manifest.payload.get("protocol_logical_sha256") - != protocol["logical_sha256"] - and not historical_parent_transition - ): - raise TrainingLaunchError("parent run manifest differs from the consumer protocol") - parent_stage = parent_manifest.payload.get("stage") - if ( - not isinstance(parent_stage, Mapping) - or parent_stage.get("stage_id") != parent_stage_id - or parent_stage.get("model_key") != stage.model_key - ): - raise TrainingLaunchError("parent run manifest has the wrong stage or model") - checkpoint: dict[str, Any] - frozen_parent = ( - existing_manifest.payload.get("parent_receipt") if existing_manifest is not None else None - ) - if isinstance(frozen_parent, Mapping): - if ( - frozen_parent.get("stage_id") != parent_stage_id - or frozen_parent.get("wave") != parent_wave - or frozen_parent.get("replicate_id", parent_replicate) - != parent_replicate - or frozen_parent.get("record_sha256") != receipt.record_sha256 - ): - raise TrainingLaunchError("frozen stage manifest and current parent receipt differ") - frozen_checkpoint = frozen_parent.get("checkpoint") - if not isinstance(frozen_checkpoint, Mapping): - raise TrainingLaunchError("frozen stage parent has no checkpoint") - checkpoint = _checkpoint_from_inventory( - receipt, - frozen_checkpoint, - inventory_sha256=frozen_parent.get("checkpoint_inventory_logical_sha256"), - ) - else: - if approval is None: - raise TrainingLaunchError(f"{stage.stage_id} requires a selected parent checkpoint") - selected = approval.payload.get("selected_parent_checkpoint") - if not isinstance(selected, Mapping): - raise TrainingLaunchError( - f"{stage.stage_id} approval has no selected parent checkpoint" - ) - checkpoint = _checkpoint_from_inventory( - receipt, - selected, - inventory_sha256=approval.payload.get("selected_parent_checkpoint_inventory_sha256"), - ) - inventory = receipt.payload.get("checkpoint_inventory") - if not isinstance(inventory, Mapping): - raise TrainingLaunchError("parent receipt has no checkpoint inventory") - inventory_sha256 = str(inventory.get("logical_sha256", "")) - if approval is not None: - claimed = approval.payload.get("parent_receipt_record_sha256") - if claimed is not None and claimed != receipt.record_sha256: - raise TrainingLaunchError("approval and parent receipt differ") - claimed_parent_replicate = approval.payload.get("parent_replicate_id") - if ( - claimed_parent_replicate is not None - and claimed_parent_replicate != parent_replicate - ): - raise TrainingLaunchError("approval and parent execution identity differ") - selected = approval.payload.get("selected_parent_checkpoint") - if selected is not None and selected != checkpoint: - raise TrainingLaunchError("approval and selected parent checkpoint differ") - selected_inventory = approval.payload.get("selected_parent_checkpoint_inventory_sha256") - if selected_inventory is not None and selected_inventory != inventory_sha256: - raise TrainingLaunchError("approval and parent checkpoint inventory differ") - - entry_approval: TrainingRecord | None = None - source_transition: dict[str, Any] | None = None - if parent_source_git_sha != source_git_sha: - if existing_manifest is None: - if ( - approval is None - or approval.payload.get("schema_version") - not in { - SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION, - PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION, - } - ): - raise TrainingLaunchError( - "cross-source parent requires a v2 transition approval" - ) - entry_approval = approval - else: - entry_approval = _manifest_transition_approval( - repo_root=repo_root, - protocol=protocol, - store=store, - key=key, - stage=stage, - source_git_sha=source_git_sha, - existing_manifest=existing_manifest, - ) - if ( - entry_approval.payload.get("evidence_source_git_sha") != parent_source_git_sha - or entry_approval.payload.get("parent_stage_id") != parent_stage_id - or entry_approval.payload.get("parent_wave") != parent_wave - or entry_approval.payload.get("parent_receipt_record_sha256") != receipt.record_sha256 - or entry_approval.payload.get("selected_parent_checkpoint") != checkpoint - or entry_approval.payload.get("selected_parent_checkpoint_inventory_sha256") - != inventory_sha256 - ): - raise TrainingLaunchError("v2 transition approval differs from the selected parent") - raw_transition = entry_approval.payload.get("source_transition") - if not isinstance(raw_transition, Mapping): - raise TrainingLaunchError("v2 transition approval has no source transition") - source_transition = dict(raw_transition) - elif existing_manifest is not None and "source_transition" in existing_manifest.payload: - raise TrainingLaunchError("same-source parent cannot reuse a child source transition") - return ParentBinding( - stage_id=parent_stage_id, - wave=parent_wave, - replicate_id=parent_replicate, - receipt=receipt, - checkpoint=checkpoint, - source_git_sha=parent_source_git_sha, - run_manifest=parent_manifest, - entry_approval=entry_approval, - source_transition=source_transition, - ) - - -def _required_prior_wave(stage: StageSpec, wave_name: str) -> str | None: - target = int(stage.wave(wave_name)["max_steps"]) - earlier = [ - (int(wave["max_steps"]), name) - for name, wave in stage.waves.items() - if int(wave["max_steps"]) < target - ] - return max(earlier)[1] if earlier else None - - -def _approval_record( - *, - repo_root: Path, - store: TrainingStore, - key: StageKey, - stage: StageSpec, - wave_name: str, - source_git_sha: str, - protocol: Mapping[str, Any], -) -> TrainingRecord | None: - gate = str(stage.wave(wave_name)["approval"]) - prior_wave = _required_prior_wave(stage, wave_name) - prior: TrainingRecord | None = None - if prior_wave is not None: - prior = store.load_wave_receipt(key, wave=prior_wave) - if prior is None: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} requires its {prior_wave} receipt" - ) - if gate == "initial": - return None - if gate == "base-level-held-out-baseline-receipt": - if prior_wave is not None: - raise TrainingLaunchError("base baseline gate cannot follow a training wave") - try: - return record_base_baseline_approval( - repo_root=repo_root, - store=store, - key=key, - stage=stage, - wave_name=wave_name, - source_git_sha=source_git_sha, - protocol=protocol, - ) - except (PromotionError, ValueError) as exc: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} base baseline gate failed: {exc}" - ) from exc - if gate == "rollout-health-receipt": - if prior_wave != "smoke" or prior is None: - raise TrainingLaunchError( - "rollout-health gate requires the exact smoke receipt" - ) - try: - return record_rollout_health_approval( - store=store, - key=key, - stage=stage, - wave_name=wave_name, - prior_wave=prior_wave, - prior_receipt=prior, - source_git_sha=source_git_sha, - protocol=protocol, - ) - except (PromotionError, ValueError) as exc: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} rollout-health gate failed: {exc}" - ) from exc - if gate == "parent-receipt-and-source-transition": - if prior_wave is not None: - raise TrainingLaunchError( - "historical-parent transition cannot follow a child training wave" - ) - approval = store.load_wave_approval(key, wave=wave_name) - if approval is None: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} requires an immutable {gate} approval" - ) - try: - validate_parent_receipt_transition_approval( - repo_root=repo_root, - protocol=protocol, - store=store, - stage=stage, - wave_name=wave_name, - replicate_id=key.replicate_id, - consumer_source_git_sha=source_git_sha, - approval=approval, - ) - except PromotionError as exc: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} parent transition failed: {exc}" - ) from exc - return approval - approval = store.load_wave_approval(key, wave=wave_name) - if approval is None: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} requires an immutable {gate} approval" - ) - schema_version = approval.payload.get("schema_version") - if schema_version == APPROVAL_SCHEMA_VERSION: - if any( - field in approval.payload for field in ("evidence_source_git_sha", "source_transition") - ): - raise TrainingLaunchError("same-source v1 approval contains source-transition fields") - try: - validate_same_source_wave_approval( - repo_root=repo_root, - protocol=protocol, - store=store, - stage=stage, - wave_name=wave_name, - replicate_id=key.replicate_id, - source_git_sha=source_git_sha, - approval=approval, - ) - except PromotionError as exc: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} same-source approval " - f"failed validation: {exc}" - ) from exc - elif schema_version == SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION: - try: - validate_source_transition_approval( - repo_root=repo_root, - protocol=protocol, - store=store, - stage=stage, - wave_name=wave_name, - replicate_id=key.replicate_id, - consumer_source_git_sha=source_git_sha, - approval=approval, - ) - except PromotionError as exc: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} source-transition approval failed validation: {exc}" - ) from exc - else: - raise TrainingLaunchError(f"{stage.stage_id}/{wave_name} approval has a foreign schema") - expected = { - "decision": "approve", - "approval_gate": gate, - "stage_id": stage.stage_id, - "wave": wave_name, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - } - for field, value in expected.items(): - if approval.payload.get(field) != value: - raise TrainingLaunchError(f"{stage.stage_id}/{wave_name} approval has wrong {field}") - if gate == "clean-smoke-receipt": - if prior_wave is None or prior is None: - raise TrainingLaunchError("clean-smoke gate has no prior wave receipt") - if ( - approval.payload.get("prior_wave") != prior_wave - or approval.payload.get("prior_receipt_record_sha256") != prior.record_sha256 - ): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} approval is not bound to its {prior_wave} receipt" - ) - elif gate == "promotion-receipt-and-explicit-human-approval": - if prior_wave is None or prior is None: - raise TrainingLaunchError("promotion gate has no prior wave receipt") - evidence = approval.payload.get("evaluation") - if not isinstance(evidence, Mapping): - raise TrainingLaunchError("promotion approval has no evaluation evidence") - expected_evidence = { - "stage_id": stage.stage_id, - "wave": prior_wave, - "panel": "inkling-promotion", - "training_wave_receipt_record_sha256": prior.record_sha256, - } - if any(evidence.get(field) != value for field, value in expected_evidence.items()): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} promotion is not bound " - f"to its {prior_wave} evaluation" - ) - evaluated_checkpoint = evidence.get("checkpoint") - receipt_checkpoint = prior.payload.get("checkpoint") - checkpoint_fields = ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - if ( - not isinstance(evaluated_checkpoint, Mapping) - or not isinstance(receipt_checkpoint, Mapping) - or any( - evaluated_checkpoint.get(field) != receipt_checkpoint.get(field) - for field in checkpoint_fields - ) - ): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} promotion checkpoint differs " - f"from its {prior_wave} receipt" - ) - elif gate == "current-level-held-out-promotion-receipt": - if ( - prior_wave is None - or prior is None - or stage.current_level not in LEVEL_PROGRESS_PANELS - ): - raise TrainingLaunchError("current-level promotion has no prior receipt") - evidence = approval.payload.get("evaluation") - expected_evidence = { - "stage_id": stage.stage_id, - "wave": prior_wave, - "panel": LEVEL_PROGRESS_PANELS[stage.current_level], - "training_wave_receipt_record_sha256": prior.record_sha256, - } - if not isinstance(evidence, Mapping) or any( - evidence.get(field) != value - for field, value in expected_evidence.items() - ): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} is not bound to its " - f"{stage.current_level} held-out report" - ) - checkpoint_fields = ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - evaluated_checkpoint = evidence.get("checkpoint") - receipt_checkpoint = prior.payload.get("checkpoint") - if ( - approval.payload.get("prior_wave") != prior_wave - or approval.payload.get("prior_receipt_record_sha256") - != prior.record_sha256 - or not isinstance(evaluated_checkpoint, Mapping) - or not isinstance(receipt_checkpoint, Mapping) - or any( - evaluated_checkpoint.get(field) != receipt_checkpoint.get(field) - for field in checkpoint_fields - ) - or not _SHA256.fullmatch( - str(evidence.get("report_record_sha256", "")) - ) - ): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} held-out evidence differs " - f"from its {prior_wave} checkpoint" - ) - elif gate == "prior-level-held-out-promotion-receipt": - if stage.current_level not in LEVELS[1:]: - raise TrainingLaunchError("prior-level promotion has no prior level") - parent_stage_id, parent_wave = stage.parent.split(":", 1) - parent_key = StageKey(key.study_id, parent_stage_id, key.replicate_id) - parent_receipt = store.load_wave_receipt(parent_key, wave=parent_wave) - prior_level = LEVELS[LEVELS.index(stage.current_level) - 1] - evidence = approval.payload.get("evaluation") - expected_evidence = { - "stage_id": parent_stage_id, - "wave": parent_wave, - "panel": LEVEL_PROGRESS_PANELS[prior_level], - "training_wave_receipt_record_sha256": ( - parent_receipt.record_sha256 if parent_receipt is not None else None - ), - } - if parent_receipt is None or not isinstance(evidence, Mapping) or any( - evidence.get(field) != value - for field, value in expected_evidence.items() - ): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} is not bound to its " - f"{prior_level} parent held-out report" - ) - checkpoint_fields = ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - evaluated_checkpoint = evidence.get("checkpoint") - receipt_checkpoint = parent_receipt.payload.get("checkpoint") - if ( - approval.payload.get("parent_stage_id") != parent_stage_id - or approval.payload.get("parent_wave") != parent_wave - or approval.payload.get("parent_receipt_record_sha256") - != parent_receipt.record_sha256 - or approval.payload.get("selected_parent_checkpoint") - != evaluated_checkpoint - or not isinstance(evaluated_checkpoint, Mapping) - or not isinstance(receipt_checkpoint, Mapping) - or any( - evaluated_checkpoint.get(field) != receipt_checkpoint.get(field) - for field in checkpoint_fields - ) - or not _SHA256.fullmatch( - str(evidence.get("report_record_sha256", "")) - ) - ): - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} parent held-out evidence differs " - f"from its {parent_stage_id}/{parent_wave} checkpoint" - ) - return approval - - -def _manifest_payload( - *, - protocol: Mapping[str, Any], - source_git_sha: str, - stage: StageSpec, - preflight: Mapping[str, Any], - tracking: Mapping[str, Any], - parent: ParentBinding | None, -) -> dict[str, Any]: - payload = { - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": preflight["protocol_file_sha256"], - "contract_version": protocol["contract_version"], - "prompt_assets": preflight["prompt_assets"], - "dataset": preflight["dataset"], - "runtime": preflight["runtime"], - "sandbox": preflight["sandbox"], - "stage": { - "stage_id": stage.stage_id, - "kind": stage.kind, - "model_key": stage.model_key, - "recipe_key": stage.recipe_key, - "hypotheses": list(stage.hypotheses), - "parent_policy": stage.parent, - "levels": list(stage.levels), - "current_level": stage.current_level, - "replay_levels": list(stage.replay_levels), - "waves": stage.waves, - }, - "model": preflight["model"], - "recipe": preflight["recipe"], - "schedule": preflight["schedule"], - "tracking": dict(tracking), - "parent_receipt": ( - { - "relative_path": str(parent.receipt.relative_path), - "stage_id": parent.stage_id, - "wave": parent.wave, - "replicate_id": parent.replicate_id, - "record_sha256": parent.receipt.record_sha256, - "checkpoint": parent.checkpoint, - "checkpoint_inventory_logical_sha256": parent.receipt.payload[ - "checkpoint_inventory" - ]["logical_sha256"], - } - if parent is not None - else None - ), - } - if parent is not None and parent.source_transition is not None: - if parent.entry_approval is None: - raise TrainingLaunchError("cross-source parent has no v2 entry approval") - payload["source_transition"] = _child_source_transition_binding(parent.entry_approval) - return payload - - -def _candidate_recorder( - store: TrainingStore, - key: StageKey, - *, - invocation_id: str, -) -> Callable[[Mapping[str, object]], None]: - def record(value: Mapping[str, object]) -> None: - payload = dict(value) - try: - step = int(payload["step"]) - attempt = int(payload["attempt"]) - task = payload["task"] - if not isinstance(task, Mapping): - raise TypeError("task is not an object") - task_id = str(task["opaque_id"]) - except (KeyError, TypeError, ValueError) as exc: - raise TrainingLaunchError("candidate record has no immutable key") from exc - sample_payload = dict(payload) - evaluation = sample_payload.pop("evaluation", None) - if not isinstance(evaluation, Mapping): - raise TrainingLaunchError("candidate record has no evaluation") - sample = store.load_candidate_sample( - key, - invocation_id=invocation_id, - step=step, - task_id=task_id, - attempt=attempt, - ) - if sample is None or canonical_sha256(sample_payload) != sample.payload_sha256: - raise TrainingLaunchError( - "candidate evaluation does not match its persisted paid sample" - ) - store.write_candidate( - key, - invocation_id=invocation_id, - step=step, - task_id=task_id, - attempt=attempt, - payload={ - "sample_record": { - "relative_path": str(sample.relative_path), - "payload_sha256": sample.payload_sha256, - "record_sha256": sample.record_sha256, - }, - "evaluation": dict(evaluation), - }, - ) - - return record - - -def _candidate_sample_recorder( - store: TrainingStore, - key: StageKey, - *, - invocation_id: str, -) -> Callable[[Mapping[str, object]], None]: - def record(value: Mapping[str, object]) -> None: - payload = dict(value) - try: - step = int(payload["step"]) - attempt = int(payload["attempt"]) - task = payload["task"] - if not isinstance(task, Mapping): - raise TypeError("task is not an object") - task_id = str(task["opaque_id"]) - except (KeyError, TypeError, ValueError) as exc: - raise TrainingLaunchError("candidate sample has no immutable key") from exc - store.write_candidate_sample( - key, - invocation_id=invocation_id, - step=step, - task_id=task_id, - attempt=attempt, - payload=payload, - ) - - return record - - -def _resolved_launch_binding( - *, - repo_root: Path, - protocol: Mapping[str, Any], - stage: StageSpec, - wave_name: str, - store: TrainingStore, - key: StageKey, - parent: ParentBinding | None, - resume_batch: int | None, - resume_checkpoint: Mapping[str, Any] | None = None, -) -> dict[str, Any]: - """Return the complete scientific policy consumed by the Cookbook config.""" - - model = protocol["models"][stage.model_key] - recipe = protocol["recipes"][stage.recipe_key] - max_steps = int(stage.wave(wave_name)["max_steps"]) - checkpoint = parent.checkpoint if parent is not None else None - manifest = store.load_manifest(key) - if manifest is None: - raise TrainingLaunchError("resolved launch cannot precede its run manifest") - common: dict[str, Any] = { - "schema_version": "pixcell-resolved-training-launch-v1", - "stage_id": stage.stage_id, - "wave": wave_name, - "kind": stage.kind, - "model_name": model["model"], - "renderer_name": model["renderer"], - "renderer_effort": model.get("effort"), - "thinking": bool(model["thinking"]), - "context_tokens": int(model["context_tokens"]), - "max_image_long_edge": int(model["max_image_long_edge"]), - "lora_rank": int(model["lora_rank"]), - "learning_rate": float(recipe["learning_rate"]), - "max_steps": max_steps, - "dataset_root": str((repo_root / "dataset").relative_to(repo_root)), - "dataset_configuration": protocol["dataset"]["configuration"], - "dataset_revision": protocol["dataset"]["revision"], - "dataset_logical_release_sha256": protocol["dataset"]["logical_release_sha256"], - "schedule_seed": SCHEDULE_SEED, - "tinker_log_relative_path": str( - store.tinker_log_path(key).relative_to(store.external_root) - ), - "initial_checkpoint": checkpoint, - "resume_batch": resume_batch, - "resume_checkpoint": ( - dict(resume_checkpoint) if resume_checkpoint is not None else None - ), - "wandb_run_id": deterministic_wandb_run_id( - source_git_sha=str(manifest.payload["source_git_sha"]), - protocol_logical_sha256=str(protocol["logical_sha256"]), - key=key, - ), - } - if stage.kind == "sft": - common["dataset"] = { - "split": "train", - "levels": list(stage.levels), - "batch_size": int(recipe["batch_size"]), - "max_sequence_tokens": int(recipe["max_sequence_tokens"]), - "include_validation": bool(recipe["automatic_nll_validation"]), - "loss_mass": recipe["loss_mass"], - "target": recipe["target"], - } - common["optimizer"] = { - "name": "adam", - "beta1": float(recipe["adam_beta1"]), - "beta2": float(recipe["adam_beta2"]), - "epsilon": float(recipe["adam_epsilon"]), - "lr_schedule": recipe["lr_schedule"], - "num_epochs": int(recipe["effective_passes"]), - "optimizer_reset_each_stage": bool(recipe["optimizer_reset_each_stage"]), - } - common["checkpointing"] = { - "save_every": max( - 1, - round(max_steps * float(recipe["checkpoint_fraction"])), - ), - "rolling_save_every": int(recipe["rolling_save_every_steps"]), - "periodic_ttl_seconds": recipe["checkpoint_ttl_seconds"], - } - common["automatic_evaluation"] = { - "eval_every": 0, - "automatic_nll_validation": False, - } - else: - common["dataset"] = { - "split": "train", - "current_level": stage.current_level, - "replay_levels": list(stage.replay_levels), - "groups_per_batch": int(recipe["groups_per_batch"]), - "group_size": int(recipe["group_size"]), - "replay_fraction": float(recipe["replay_fraction"]), - "require_prefix_replay": stage.model_key == "qwen", - "validation_canaries": int(recipe["automatic_validation_rows"]), - } - common["sampling"] = { - "max_output_tokens": int(model["max_output_tokens"]), - "temperature": float(recipe["temperature"]), - # The pinned Cookbook constructs SamplingParams without top_p; - # Tinker's pinned default is 1.0 and preflight verifies that fact. - "top_p": float(recipe["top_p"]), - } - common["optimizer"] = { - "loss_fn": recipe["loss_fn"], - "kl_coefficient": float(recipe["kl_coefficient"]), - "remove_constant_reward_groups": bool(recipe["remove_constant_reward_groups"]), - "reward": recipe["reward"], - } - common["checkpointing"] = { - "policy": recipe["checkpoint_policy"], - "save_every": max_steps, - "rolling_save_every": int(recipe["rolling_save_every_steps"]), - "periodic_ttl_seconds": recipe["checkpoint_ttl_seconds"], - } - common["automatic_evaluation"] = { - "eval_every": 0, - "validation_rows": 0, - } - common["rollout"] = { - "error_policy": recipe["rollout_error_policy"], - "json_export": bool(recipe["rollout_json_export"]), - "groups_to_log": 0, - } - if stage.stage_id.startswith(("qwen-base-rl-", "qwen-l0-rl-")): - gate = str(stage.wave(wave_name)["approval"]) - gate_record = ( - store.load_base_baseline_approval(key, wave=wave_name) - if gate == "base-level-held-out-baseline-receipt" - else ( - store.load_rollout_health_approval(key, wave=wave_name) - if gate == "rollout-health-receipt" - else ( - store.load_wave_approval(key, wave=wave_name) - if gate - in { - "current-level-held-out-promotion-receipt", - "parent-receipt-and-source-transition", - "prior-level-held-out-promotion-receipt", - "prior-and-current-level-held-out-transition-receipt", - } - else None - ) - ) - ) - if gate != "initial": - if gate_record is None: - raise TrainingLaunchError( - f"{stage.stage_id}/{wave_name} has no immutable gate evidence" - ) - common["wave_gate_evidence"] = { - "approval_gate": gate, - "relative_path": str(gate_record.relative_path), - "payload_sha256": gate_record.payload_sha256, - "record_sha256": gate_record.record_sha256, - } - common["logical_sha256"] = canonical_sha256(common) - return common - - -def _invocation_payload( - *, - protocol: Mapping[str, Any], - stage: StageSpec, - wave_name: str, - source_git_sha: str, - invocation_id: str, - manifest: TrainingRecord, - preflight: Mapping[str, Any], - tracking: Mapping[str, Any], - resume_batch: int | None, - resolved_launch: Mapping[str, Any], - parent: ParentBinding | None, - resume_checkpoint: Mapping[str, Any] | None = None, -) -> dict[str, Any]: - """Build the immutable invocation record shared by launch and replay.""" - - return { - "invocation_id": invocation_id, - "stage_id": stage.stage_id, - "wave": wave_name, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "run_manifest_record_sha256": manifest.record_sha256, - "preflight_sha256": preflight["preflight_sha256"], - "wandb_run_id": tracking["run_id"], - "resume_batch": resume_batch, - "resume_checkpoint": ( - dict(resume_checkpoint) if resume_checkpoint is not None else None - ), - "resolved_launch": dict(resolved_launch), - "resolved_launch_sha256": resolved_launch["logical_sha256"], - "policy": { - "model": protocol["models"][stage.model_key]["model"], - "renderer": protocol["models"][stage.model_key]["renderer"], - "effort": protocol["models"][stage.model_key].get("effort"), - "max_output_tokens": protocol["models"][stage.model_key]["max_output_tokens"], - "temperature": protocol["recipes"][stage.recipe_key].get("temperature"), - "top_p": protocol["recipes"][stage.recipe_key].get("top_p"), - "reward": protocol["recipes"][stage.recipe_key].get("reward"), - "group_size": protocol["recipes"][stage.recipe_key].get("group_size"), - "groups_per_batch": protocol["recipes"][stage.recipe_key].get("groups_per_batch"), - }, - "initial_checkpoint": parent.checkpoint if parent is not None else None, - } - - -def build_tinker_config( - *, - repo_root: Path, - protocol: Mapping[str, Any], - stage: StageSpec, - wave_name: str, - store: TrainingStore, - key: StageKey, - parent: ParentBinding | None, - invocation: TrainingRecord | None = None, -) -> SFTConfig | RLConfig: - """Resolve one sealed Cookbook config without creating a service client.""" - - model = protocol["models"][stage.model_key] - recipe = protocol["recipes"][stage.recipe_key] - max_steps = int(stage.wave(wave_name)["max_steps"]) - checkpoint = parent.checkpoint if parent is not None else None - load_checkpoint_path = str(checkpoint["state_path"]) if checkpoint is not None else None - log_path = str(store.tinker_log_path(key)) - manifest = store.load_manifest(key) - if manifest is None: - raise TrainingLaunchError("Tinker config cannot precede its run manifest") - resolved_launch = _resolved_launch_binding( - repo_root=repo_root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - store=store, - key=key, - parent=parent, - resume_batch=( - int(invocation.payload["resume_batch"]) - if invocation is not None and invocation.payload.get("resume_batch") is not None - else None - ), - resume_checkpoint=( - invocation.payload.get("resume_checkpoint") - if invocation is not None - else None - ), - ) - if invocation is not None and invocation.payload.get("resolved_launch") != resolved_launch: - raise TrainingLaunchError("immutable invocation differs from the resolved Tinker launch") - tracking = _tracking_binding( - protocol=protocol, - source_git_sha=str(manifest.payload["source_git_sha"]), - key=key, - stage=stage, - store=store, - ) - common = { - "log_path": log_path, - "model_name": model["model"], - "recipe_name": f"pixcell_{STUDY_ID}_{stage.stage_id}", - "renderer_name": model["renderer"], - "load_checkpoint_path": load_checkpoint_path, - "learning_rate": float(recipe["learning_rate"]), - "lora_rank": int(model["lora_rank"]), - "wandb_project": tracking["project"], - "wandb_name": tracking["name"], - "max_steps": max_steps, - } - if stage.kind == "sft": - save_every = max( - 1, - round(max_steps * float(recipe["checkpoint_fraction"])), - ) - return SFTConfig( - **common, - dataset_builder=TrackASupervisedDatasetBuilder( - dataset_root=str(repo_root / "dataset"), - model_name=model["model"], - renderer_name=model["renderer"], - batch_size=int(recipe["batch_size"]), - max_length=int(recipe["max_sequence_tokens"]), - max_image=int(model["max_image_long_edge"]), - levels=",".join(stage.levels), - include_validation=bool(recipe["automatic_nll_validation"]), - schedule_seed=SCHEDULE_SEED, - ), - lr_schedule=str(recipe["lr_schedule"]), - num_epochs=int(recipe["effective_passes"]), - adam_beta1=float(recipe["adam_beta1"]), - adam_beta2=float(recipe["adam_beta2"]), - adam_eps=float(recipe["adam_epsilon"]), - save_every=save_every, - eval_every=0, - rolling_save_every=int(recipe["rolling_save_every_steps"]), - ttl_seconds=recipe["checkpoint_ttl_seconds"], - ) - return RLConfig( - **common, - dataset_builder=TrackARLDatasetBuilder( - dataset_root=str(repo_root / "dataset"), - batch_size=int(recipe["groups_per_batch"]), - group_size=int(recipe["group_size"]), - model_name=model["model"], - renderer_name=model["renderer"], - model_effort=model.get("effort"), - max_image=int(model["max_image_long_edge"]), - current_level=str(stage.current_level), - replay_levels=",".join(stage.replay_levels), - validation_canaries=int(recipe["automatic_validation_rows"]), - attempt_recorder=( - _candidate_recorder( - store, - key, - invocation_id=str(invocation.payload["invocation_id"]), - ) - if invocation is not None - else None - ), - sample_recorder=( - _candidate_sample_recorder( - store, - key, - invocation_id=str(invocation.payload["invocation_id"]), - ) - if invocation is not None - else None - ), - reward_policy="raw_iou", - require_prefix_replay=stage.model_key == "qwen", - schedule_seed=SCHEDULE_SEED, - attempt_metadata=( - { - "invocation_id": invocation.payload["invocation_id"], - "run_manifest_record_sha256": invocation.payload["run_manifest_record_sha256"], - "source_git_sha": invocation.payload["source_git_sha"], - "protocol_logical_sha256": invocation.payload["protocol_logical_sha256"], - "policy": invocation.payload["policy"], - "initial_checkpoint": invocation.payload["initial_checkpoint"], - } - if invocation is not None - else {} - ), - ), - max_tokens=int(model["max_output_tokens"]), - temperature=float(recipe["temperature"]), - eval_every=0, - save_every=max_steps, - rolling_save_every=int(recipe["rolling_save_every_steps"]), - loss_fn=str(recipe["loss_fn"]), - remove_constant_reward_groups=bool(recipe["remove_constant_reward_groups"]), - rollout_error_tolerance=( - False - if recipe["rollout_error_policy"] == "fail_fast" - else recipe["rollout_error_policy"] - ), - kl_penalty_coef=float(recipe["kl_coefficient"]), - num_groups_to_log=0, - rollout_json_export=bool(recipe["rollout_json_export"]), - ttl_seconds=recipe["checkpoint_ttl_seconds"], - ) - - -def _sha256_bytes(value: bytes) -> str: - return hashlib.sha256(value).hexdigest() - - -def _wave_artifact_snapshot( - *, - log_path: Path, - wave_name: str, - recorded: Mapping[str, Any] | None = None, -) -> tuple[dict[str, Any], dict[str, bytes]]: - """Freeze or verify the append-only log prefix belonging to one wave. - - Cookbook deliberately reuses ``log_path`` across resumable waves. A - receipt therefore cannot hash the live files directly: later waves append - to them. Each receipt instead binds create-only copies under its own wave - directory. During replay, the copies must still match their receipt and - the append-only live logs must retain the exact frozen prefix. - """ - - relative_root = Path("waves") / wave_name / "artifacts" - snapshot_root = log_path.parent / relative_root - expected_names = ("checkpoints.jsonl", "metrics.jsonl", "hparams.json") - stage_root = log_path.parent.resolve(strict=True) - resolved_snapshot_root = snapshot_root.resolve(strict=False) - if ( - resolved_snapshot_root == stage_root - or not resolved_snapshot_root.is_relative_to(stage_root) - ): - raise TrainingLaunchError("completed wave artifact snapshot escapes its stage") - - if recorded is None: - source_bytes: dict[str, bytes] = {} - for name in expected_names: - source = log_path / name - if source.is_symlink(): - raise TrainingLaunchError("live training artifact cannot be a symlink") - if source.is_file(): - source_bytes[name] = source.read_bytes() - if not {"checkpoints.jsonl", "metrics.jsonl"}.issubset(source_bytes): - raise TrainingLaunchError( - "training returned without checkpoint and metric artifacts" - ) - snapshot_root.mkdir(parents=True, exist_ok=True, mode=0o700) - for name, value in source_bytes.items(): - destination = snapshot_root / name - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - descriptor = os.open(destination, flags, 0o600) - except FileExistsError: - if destination.is_symlink() or destination.read_bytes() != value: - raise TrainingLaunchError( - f"{wave_name} immutable artifact snapshot already differs" - ) - else: - try: - with os.fdopen(descriptor, "wb") as stream: - stream.write(value) - stream.flush() - os.fsync(stream.fileno()) - except BaseException: - destination.unlink(missing_ok=True) - raise - snapshot = { - "schema_version": _WAVE_ARTIFACT_SNAPSHOT_SCHEMA_VERSION, - "relative_root": relative_root.as_posix(), - "files": { - name: { - "byte_count": len(value), - "sha256": _sha256_bytes(value), - } - for name, value in sorted(source_bytes.items()) - }, - } - else: - snapshot = dict(recorded) - - if ( - set(snapshot) != {"schema_version", "relative_root", "files"} - or snapshot.get("schema_version") != _WAVE_ARTIFACT_SNAPSHOT_SCHEMA_VERSION - or snapshot.get("relative_root") != relative_root.as_posix() - or not isinstance(snapshot.get("files"), Mapping) - ): - raise TrainingLaunchError("completed wave artifact snapshot is malformed") - files = snapshot["files"] - if ( - not {"checkpoints.jsonl", "metrics.jsonl"}.issubset(files) - or not set(files).issubset(expected_names) - ): - raise TrainingLaunchError("completed wave artifact snapshot files differ") - - verified: dict[str, bytes] = {} - for name, metadata in files.items(): - if ( - not isinstance(name, str) - or not isinstance(metadata, Mapping) - or set(metadata) != {"byte_count", "sha256"} - or isinstance(metadata.get("byte_count"), bool) - or not isinstance(metadata.get("byte_count"), int) - or int(metadata["byte_count"]) < 0 - or not _SHA256.fullmatch(str(metadata.get("sha256", ""))) - ): - raise TrainingLaunchError("completed wave artifact snapshot entry is malformed") - snapshot_path = snapshot_root / name - if not snapshot_path.is_file() or snapshot_path.is_symlink(): - raise TrainingLaunchError("completed wave artifact snapshot is missing") - value = snapshot_path.read_bytes() - if ( - len(value) != int(metadata["byte_count"]) - or _sha256_bytes(value) != metadata["sha256"] - ): - raise TrainingLaunchError("completed wave artifact snapshot digest differs") - if name in {"checkpoints.jsonl", "metrics.jsonl"}: - live_path = log_path / name - if not live_path.is_file(): - if name == "checkpoints.jsonl": - raise TrainingLaunchError( - "completed wave is without a resumable checkpoint artifact" - ) - raise TrainingLaunchError("completed wave live artifact is missing") - with live_path.open("rb") as stream: - live_prefix = stream.read(len(value)) - if live_prefix != value: - raise TrainingLaunchError( - "completed wave canonical training artifacts no longer have " - "their frozen prefix" - ) - verified[name] = value - return snapshot, verified - - -def _checkpoint_records_from_bytes(value: bytes) -> list[Any]: - records: list[Any] = [] - try: - for raw_line in value.decode("utf-8").splitlines(): - if raw_line.strip(): - item = json.loads(raw_line) - if not isinstance(item, dict): - raise ValueError("checkpoint row is not an object") - records.append(checkpoint_utils.CheckpointRecord.from_dict(item)) - except (UnicodeDecodeError, ValueError, TypeError, json.JSONDecodeError) as exc: - raise TrainingLaunchError("checkpoint snapshot is not valid JSONL") from exc - return records - - -def _metrics_from_bytes(value: bytes) -> list[dict[str, Any]]: - records: list[dict[str, Any]] = [] - try: - for raw_line in value.decode("utf-8").splitlines(): - if raw_line.strip(): - item = json.loads(raw_line) - if not isinstance(item, dict): - raise ValueError("metric row is not an object") - records.append(item) - except (UnicodeDecodeError, ValueError, TypeError, json.JSONDecodeError) as exc: - raise TrainingLaunchError("metric snapshot is not valid JSONL") from exc - return records - - -def _checkpoint_receipt( - *, - stage: StageSpec, - wave_name: str, - log_path: Path, - manifest: TrainingRecord, - preflight: Mapping[str, Any], - invocation: TrainingRecord, - tracking: TrainingRecord | None, - artifact_snapshot: Mapping[str, Any] | None = None, -) -> dict[str, Any]: - snapshot, snapshot_files = _wave_artifact_snapshot( - log_path=log_path, - wave_name=wave_name, - recorded=artifact_snapshot, - ) - records = _checkpoint_records_from_bytes(snapshot_files["checkpoints.jsonl"]) - complete = [ - item for item in records if item.state_path is not None and item.sampler_path is not None - ] - if not complete: - raise TrainingLaunchError("training returned without a resumable checkpoint") - checkpoint = complete[-1] - target_steps = int(stage.wave(wave_name)["max_steps"]) - if stage.kind == "rl": - unique_name = f"{target_steps:06d}" - matching = [ - item for item in complete if item.name == unique_name and item.batch == target_steps - ] - if len(matching) != 1: - raise TrainingLaunchError( - f"RL wave has {len(matching)} unique {unique_name} checkpoints" - ) - checkpoint = matching[0] - if stage.kind == "sft": - if checkpoint.name != "final" or checkpoint.epoch != 1: - raise TrainingLaunchError("SFT terminal checkpoint is not the completed epoch") - if not _TINKER_PATH.fullmatch(str(checkpoint.state_path)) or not _TINKER_PATH.fullmatch( - str(checkpoint.sampler_path) - ): - raise TrainingLaunchError("checkpoint paths are not Tinker URIs") - recipe = preflight["recipe"] - save_every = ( - max( - 1, - round(target_steps * float(recipe["checkpoint_fraction"])), - ) - if stage.kind == "sft" - else target_steps - ) - expected_periodic_names = ( - [f"{step:06d}" for step in range(save_every, target_steps, save_every)] - if stage.kind == "sft" - else [] - ) - periodic_by_name: dict[str, Any] = {} - for item in complete: - if item.name not in expected_periodic_names: - continue - if item.name in periodic_by_name: - raise TrainingLaunchError(f"checkpoint inventory repeats {item.name}") - periodic_by_name[item.name] = item - if set(periodic_by_name) != set(expected_periodic_names): - raise TrainingLaunchError("periodic checkpoint inventory differs from the sealed cadence") - - def checkpoint_entry(item: Any, *, role: str) -> dict[str, Any]: - if not _TINKER_PATH.fullmatch(str(item.state_path)) or not _TINKER_PATH.fullmatch( - str(item.sampler_path) - ): - raise TrainingLaunchError("checkpoint inventory contains invalid URIs") - progress = 1.0 if role == "terminal" else int(item.name) / target_steps - return { - "name": str(item.name), - "batch": item.batch, - "epoch": item.epoch, - "final": role == "terminal", - "state_path": str(item.state_path), - "sampler_path": str(item.sampler_path), - "role": role, - "training_progress_fraction": progress, - } - - checkpoint_entries = [ - checkpoint_entry(periodic_by_name[name], role="periodic") - for name in expected_periodic_names - ] - terminal_entry = checkpoint_entry(checkpoint, role="terminal") - checkpoint_entries.append(terminal_entry) - local_artifacts = { - name: str(metadata["sha256"]) - for name, metadata in snapshot["files"].items() - } - candidate_root = log_path.parent / "candidates" / str(invocation.payload["invocation_id"]) - candidate_files = ( - sorted(candidate_root.rglob("evaluation.json")) if candidate_root.is_dir() else [] - ) - sample_files = sorted(candidate_root.rglob("sample.json")) if candidate_root.is_dir() else [] - candidate_inventory = [ - { - "path": str(path.relative_to(log_path.parent)), - "sha256": file_sha256(path), - } - for path in candidate_files - ] - if stage.kind == "rl": - start_batch = invocation.payload.get("resume_batch") - start = int(start_batch) if start_batch is not None else 0 - policy = invocation.payload["policy"] - expected_candidates = ( - (target_steps - start) * int(policy["groups_per_batch"]) * int(policy["group_size"]) - ) - if len(candidate_inventory) != expected_candidates: - raise TrainingLaunchError( - "candidate inventory does not cover every sampled rollout: " - f"{len(candidate_inventory)} != {expected_candidates}" - ) - if len(sample_files) != expected_candidates: - raise TrainingLaunchError( - "sample inventory does not cover every sampled rollout: " - f"{len(sample_files)} != {expected_candidates}" - ) - else: - expected_candidates = 0 - metrics = _metrics_from_bytes(snapshot_files["metrics.jsonl"]) - last_metrics = metrics[-1] if metrics else None - if last_metrics is None or last_metrics.get("step") != target_steps - 1: - raise TrainingLaunchError("metrics do not reach the exact wave ceiling") - required_metric = "optim/entropy" if stage.kind == "rl" else "train_mean_nll" - metric = last_metrics.get(required_metric) - if ( - isinstance(metric, bool) - or not isinstance(metric, (int, float)) - or not math.isfinite(float(metric)) - ): - raise TrainingLaunchError( - f"final metrics do not prove an optimizer update ({required_metric})" - ) - return { - "stage_id": stage.stage_id, - "wave": wave_name, - "max_steps": target_steps, - "run_manifest_record_sha256": manifest.record_sha256, - "invocation_record_sha256": invocation.record_sha256, - "invocation_id": invocation.payload["invocation_id"], - "invocation_tracking_record_sha256": ( - tracking.record_sha256 if tracking is not None else None - ), - "preflight_sha256": preflight["preflight_sha256"], - "checkpoint": { - key: terminal_entry[key] - for key in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - }, - "checkpoint_inventory": { - "count": len(checkpoint_entries), - "entries": checkpoint_entries, - "logical_sha256": canonical_sha256(checkpoint_entries), - }, - "artifact_snapshot": snapshot, - "local_artifact_sha256": local_artifacts, - "candidate_inventory": { - "count": len(candidate_inventory), - "expected_count": expected_candidates, - "logical_sha256": canonical_sha256(candidate_inventory), - }, - "sample_inventory": { - "count": len(sample_files), - "expected_count": expected_candidates, - "logical_sha256": canonical_sha256( - [ - { - "path": str(path.relative_to(log_path.parent)), - "sha256": file_sha256(path), - } - for path in sample_files - ] - ), - }, - "final_training_metric": { - "step": last_metrics["step"], - required_metric: float(metric), - }, - } - - -def _validate_completed_wave_receipt( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - key: StageKey, - stage: StageSpec, - wave_name: str, - manifest: TrainingRecord, - preflight: Mapping[str, Any], - parent: ParentBinding | None, - receipt: TrainingRecord, -) -> None: - """Fully validate an immutable completed-wave claim before replaying it.""" - - payload = receipt.payload - required_fields = { - "stage_id", - "wave", - "max_steps", - "run_manifest_record_sha256", - "invocation_record_sha256", - "invocation_id", - "invocation_tracking_record_sha256", - "preflight_sha256", - "checkpoint", - "checkpoint_inventory", - "artifact_snapshot", - "local_artifact_sha256", - "candidate_inventory", - "sample_inventory", - "final_training_metric", - } - if set(payload) != required_fields: - raise TrainingLaunchError("completed wave receipt fields differ from the canonical schema") - target_steps = int(stage.wave(wave_name)["max_steps"]) - if ( - payload.get("stage_id") != stage.stage_id - or payload.get("wave") != wave_name - or payload.get("max_steps") != target_steps - or payload.get("run_manifest_record_sha256") != manifest.record_sha256 - or payload.get("preflight_sha256") != preflight.get("preflight_sha256") - ): - raise TrainingLaunchError("completed wave receipt identity or preflight differs") - - invocation_id = payload.get("invocation_id") - if not isinstance(invocation_id, str): - raise TrainingLaunchError("completed wave receipt has no invocation ID") - try: - invocation = store.load_invocation(key, invocation_id=invocation_id) - except ValueError as exc: - raise TrainingLaunchError("completed wave receipt has an invalid invocation ID") from exc - if invocation is None or payload.get("invocation_record_sha256") != invocation.record_sha256: - raise TrainingLaunchError("completed wave receipt invocation is missing or differs") - resume_batch = invocation.payload.get("resume_batch") - if resume_batch is not None and ( - isinstance(resume_batch, bool) - or not isinstance(resume_batch, int) - or not 0 <= resume_batch <= target_steps - ): - raise TrainingLaunchError("completed wave invocation has an invalid resume batch") - resume_checkpoint = invocation.payload.get("resume_checkpoint") - if (resume_batch is None) != (resume_checkpoint is None): - raise TrainingLaunchError( - "completed wave invocation has an incomplete optimizer-resume binding" - ) - if resume_checkpoint is not None and not isinstance(resume_checkpoint, Mapping): - raise TrainingLaunchError( - "completed wave invocation optimizer-resume checkpoint is malformed" - ) - resolved_launch = _resolved_launch_binding( - repo_root=repo_root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - store=store, - key=key, - parent=parent, - resume_batch=resume_batch, - resume_checkpoint=resume_checkpoint, - ) - tracking_binding = manifest.payload.get("tracking") - if not isinstance(tracking_binding, Mapping): - raise TrainingLaunchError("completed wave manifest has no tracking binding") - expected_invocation = _invocation_payload( - protocol=protocol, - stage=stage, - wave_name=wave_name, - source_git_sha=str(manifest.payload.get("source_git_sha", "")), - invocation_id=invocation_id, - manifest=manifest, - preflight=preflight, - tracking=tracking_binding, - resume_batch=resume_batch, - resolved_launch=resolved_launch, - parent=parent, - resume_checkpoint=resume_checkpoint, - ) - if invocation.payload != expected_invocation: - raise TrainingLaunchError("completed wave invocation differs from the resolved launch") - - tracking_sha256 = payload.get("invocation_tracking_record_sha256") - tracking = store.load_invocation_tracking(key, invocation_id=invocation_id) - if tracking_sha256 is None: - if tracking is not None: - raise TrainingLaunchError("completed wave receipt omits existing tracking evidence") - else: - if ( - not isinstance(tracking_sha256, str) - or not _SHA256.fullmatch(tracking_sha256) - or tracking is None - or tracking.record_sha256 != tracking_sha256 - ): - raise TrainingLaunchError("completed wave tracking evidence is missing or differs") - expected_tracking = { - "invocation_record_sha256": invocation.record_sha256, - "run_manifest_record_sha256": manifest.record_sha256, - "source_git_sha": manifest.payload.get("source_git_sha"), - "protocol_logical_sha256": protocol["logical_sha256"], - "expected": dict(tracking_binding), - } - for field, expected in expected_tracking.items(): - if tracking.payload.get(field) != expected: - raise TrainingLaunchError(f"completed wave tracking evidence has wrong {field}") - observed = tracking.payload.get("observed") - if not isinstance(observed, Mapping): - raise TrainingLaunchError("completed wave tracking evidence has no live observation") - observed_expected = { - field: tracking_binding[field] - for field in ("run_id", "entity", "project", "name", "group", "job_type", "tags") - } - for field, expected in observed_expected.items(): - if observed.get(field) != expected: - raise TrainingLaunchError(f"completed wave live tracking has wrong {field}") - expected_resume = resume_batch is not None - if ( - observed.get("resumed") is not expected_resume - or observed.get("requested_resume") != ("must" if expected_resume else "never") - or observed.get("resolved_resume") != ("must" if expected_resume else "never") - or observed.get("requested_directory") != tracking_binding["directory"] - ): - raise TrainingLaunchError("completed wave live tracking resume binding differs") - - inventory = payload.get("checkpoint_inventory") - if not isinstance(inventory, Mapping) or set(inventory) != { - "count", - "entries", - "logical_sha256", - }: - raise TrainingLaunchError("completed wave checkpoint inventory is malformed") - entries = inventory.get("entries") - if ( - isinstance(inventory.get("count"), bool) - or not isinstance(inventory.get("count"), int) - or not isinstance(entries, list) - or inventory.get("count") != len(entries) - or inventory.get("logical_sha256") != canonical_sha256(entries) - ): - raise TrainingLaunchError("completed wave checkpoint inventory digest differs") - entry_fields = { - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - "role", - "training_progress_fraction", - } - names: set[str] = set() - terminal_entries: list[Mapping[str, Any]] = [] - periodic_entries: list[Mapping[str, Any]] = [] - for index, entry in enumerate(entries): - if not isinstance(entry, Mapping) or set(entry) != entry_fields: - raise TrainingLaunchError(f"completed wave checkpoint {index} is malformed") - name = entry.get("name") - role = entry.get("role") - progress = entry.get("training_progress_fraction") - if ( - not isinstance(name, str) - or name in names - or role not in {"periodic", "terminal"} - or entry.get("final") is not (role == "terminal") - or isinstance(entry.get("batch"), bool) - or ( - entry.get("batch") is not None - and (not isinstance(entry.get("batch"), int) or int(entry["batch"]) < 0) - ) - or isinstance(entry.get("epoch"), bool) - or ( - entry.get("epoch") is not None - and (not isinstance(entry.get("epoch"), int) or int(entry["epoch"]) < 0) - ) - or not _TINKER_PATH.fullmatch(str(entry.get("state_path", ""))) - or not _TINKER_PATH.fullmatch(str(entry.get("sampler_path", ""))) - or isinstance(progress, bool) - or not isinstance(progress, (int, float)) - or not math.isfinite(float(progress)) - or not 0.0 <= float(progress) <= 1.0 - ): - raise TrainingLaunchError(f"completed wave checkpoint {index} is invalid") - names.add(name) - if role == "terminal": - terminal_entries.append(entry) - if float(progress) != 1.0: - raise TrainingLaunchError("terminal checkpoint has wrong progress") - else: - periodic_entries.append(entry) - if len(terminal_entries) != 1: - raise TrainingLaunchError("completed wave must have one terminal checkpoint") - terminal = terminal_entries[0] - if stage.kind == "rl": - if ( - len(entries) != 1 - or terminal.get("name") != f"{target_steps:06d}" - or terminal.get("batch") != target_steps - ): - raise TrainingLaunchError("completed RL wave checkpoint differs from its ceiling") - else: - save_every = max( - 1, - round( - target_steps * float(protocol["recipes"][stage.recipe_key]["checkpoint_fraction"]) - ), - ) - expected_periodic = [f"{step:06d}" for step in range(save_every, target_steps, save_every)] - if ( - [str(entry["name"]) for entry in entries] != [*expected_periodic, "final"] - or terminal.get("name") != "final" - or terminal.get("epoch") != 1 - ): - raise TrainingLaunchError("completed SFT checkpoint cadence differs") - for entry in periodic_entries: - expected_progress = int(str(entry["name"])) / target_steps - if not math.isclose( - float(entry["training_progress_fraction"]), - expected_progress, - rel_tol=0.0, - abs_tol=1e-12, - ): - raise TrainingLaunchError("periodic checkpoint progress differs") - selected = payload.get("checkpoint") - checkpoint_fields = { - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - } - if not isinstance(selected, Mapping) or set(selected) != checkpoint_fields: - raise TrainingLaunchError("completed wave selected checkpoint is malformed") - if any(selected[field] != terminal[field] for field in checkpoint_fields): - raise TrainingLaunchError("completed wave selected checkpoint is not terminal") - - local_artifacts = payload.get("local_artifact_sha256") - if ( - not isinstance(local_artifacts, Mapping) - or not {"checkpoints.jsonl", "metrics.jsonl"}.issubset(local_artifacts) - or not set(local_artifacts).issubset({"checkpoints.jsonl", "metrics.jsonl", "hparams.json"}) - or any( - not isinstance(value, str) or not _SHA256.fullmatch(value) - for value in local_artifacts.values() - ) - ): - raise TrainingLaunchError("completed wave local artifact inventory is malformed") - - start = resume_batch if resume_batch is not None else 0 - recipe = protocol["recipes"][stage.recipe_key] - expected_candidates = ( - (target_steps - start) * int(recipe["groups_per_batch"]) * int(recipe["group_size"]) - if stage.kind == "rl" - else 0 - ) - candidate_root = store.tinker_log_path(key).parent / "candidates" / invocation_id - candidate_files = ( - sorted(candidate_root.rglob("evaluation.json")) if candidate_root.is_dir() else [] - ) - sample_files = sorted(candidate_root.rglob("sample.json")) if candidate_root.is_dir() else [] - for field, files in ( - ("candidate_inventory", candidate_files), - ("sample_inventory", sample_files), - ): - observed_inventory = [ - { - "path": str(path.relative_to(store.tinker_log_path(key).parent)), - "sha256": file_sha256(path), - } - for path in files - ] - recorded = payload.get(field) - if ( - not isinstance(recorded, Mapping) - or set(recorded) != {"count", "expected_count", "logical_sha256"} - or isinstance(recorded.get("count"), bool) - or not isinstance(recorded.get("count"), int) - or isinstance(recorded.get("expected_count"), bool) - or not isinstance(recorded.get("expected_count"), int) - or recorded.get("count") != len(observed_inventory) - or recorded.get("expected_count") != expected_candidates - or len(observed_inventory) != expected_candidates - or recorded.get("logical_sha256") != canonical_sha256(observed_inventory) - ): - raise TrainingLaunchError(f"completed wave {field} does not prove full coverage") - - final_metric = payload.get("final_training_metric") - required_metric = "optim/entropy" if stage.kind == "rl" else "train_mean_nll" - if ( - not isinstance(final_metric, Mapping) - or set(final_metric) != {"step", required_metric} - or final_metric.get("step") != target_steps - 1 - ): - raise TrainingLaunchError("completed wave final optimizer metric is malformed") - metric = final_metric.get(required_metric) - if ( - isinstance(metric, bool) - or not isinstance(metric, (int, float)) - or not math.isfinite(float(metric)) - ): - raise TrainingLaunchError("completed wave final optimizer metric is not finite") - - # A receipt is only a cache of facts reconstructed from the immutable - # invocation and the local training artifacts. Rebuild the canonical - # payload so a well-formed but forged hash, metric, checkpoint inventory, - # or rollout count cannot turn an incomplete run into ``already_complete``. - try: - reconstructed = _checkpoint_receipt( - stage=stage, - wave_name=wave_name, - log_path=store.tinker_log_path(key), - manifest=manifest, - preflight=preflight, - invocation=invocation, - tracking=tracking, - artifact_snapshot=payload.get("artifact_snapshot"), - ) - except TrainingLaunchError: - raise - except Exception as exc: - raise TrainingLaunchError( - "completed wave artifacts could not reconstruct the canonical receipt" - ) from exc - if payload != reconstructed: - raise TrainingLaunchError( - "completed wave receipt differs from the current canonical training artifacts" - ) - - -def _checkpoint_identity(item: Any) -> dict[str, Any]: - if hasattr(item, "to_dict"): - value = item.to_dict() - elif isinstance(item, Mapping): - value = dict(item) - else: - raise TrainingLaunchError("optimizer checkpoint has an unsupported record type") - try: - normalized = json.loads( - json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ) - ) - except (TypeError, ValueError) as exc: - raise TrainingLaunchError("optimizer checkpoint is not finite JSON") from exc - if not isinstance(normalized, dict): - raise TrainingLaunchError("optimizer checkpoint is not an object") - return normalized - - -def _matches_receipt_checkpoint(item: Any, expected: Mapping[str, Any]) -> bool: - return all( - getattr(item, field, None) == expected.get(field) - for field in ( - "name", - "batch", - "epoch", - "state_path", - "sampler_path", - ) - ) - - -def _validate_resume_state( - *, - stage: StageSpec, - wave_name: str, - log_path: Path, - prior_wave_receipt: TrainingRecord | None, -) -> ResumeBinding | None: - if not log_path.exists(): - return None - records = checkpoint_utils.load_checkpoints_file(str(log_path)) - target = int(stage.wave(wave_name)["max_steps"]) - resumable = [ - (index, item) - for index, item in enumerate(records) - if item.state_path is not None - ] - progress: list[int] = [] - for _index, item in resumable: - if isinstance(item.batch, bool) or not isinstance(item.batch, int): - raise TrainingLaunchError("resumable optimizer checkpoint has no integer batch") - progress.append(item.batch) - if progress and max(progress) > target: - raise TrainingLaunchError("local optimizer state is ahead of this wave") - - selected: Any | None = None - if prior_wave_receipt is not None: - expected = prior_wave_receipt.payload["checkpoint"] - matches = [ - (index, item) - for index, item in resumable - if _matches_receipt_checkpoint(item, expected) - ] - if len(matches) != 1: - raise TrainingLaunchError("optimizer log does not contain the prior-wave checkpoint") - expected_index, expected_item = matches[0] - expected_batch = expected.get("batch") - if ( - isinstance(expected_batch, bool) - or not isinstance(expected_batch, int) - or expected_batch < 0 - or expected_batch >= target - ): - raise TrainingLaunchError("prior-wave checkpoint cannot precede this wave") - maximum = max(progress) if progress else expected_batch - if maximum == expected_batch: - later = [ - item - for index, item in resumable - if index > expected_index - ] - if later: - # The pinned Cookbook always appends a permanent ``final`` - # checkpoint after the numbered wave-ceiling checkpoint. - # It represents the same completed batch, but it is a - # different URI. We recognize exactly that one safe shape - # and still pin the next launch to the receipt's numbered URI. - if ( - len(later) != 1 - or later[0].name != "final" - or later[0].batch != expected_batch - or later[0].state_path is None - or later[0].sampler_path is None - ): - raise TrainingLaunchError( - "optimizer log has ambiguous state after the prior-wave checkpoint" - ) - selected = expected_item - else: - selected_index, candidate = resumable[-1] - if ( - selected_index <= expected_index - or candidate.batch != maximum - or candidate.name == "final" - or candidate.batch >= target - ): - raise TrainingLaunchError( - "optimizer log does not have an exact in-wave resume checkpoint" - ) - selected = candidate - elif resumable: - maximum = max(progress) - _selected_index, candidate = resumable[-1] - if ( - candidate.batch != maximum - or candidate.name == "final" - or candidate.batch >= target - ): - raise TrainingLaunchError( - "training reached its wave ceiling without an immutable receipt" - ) - selected = candidate - - if not records and any(log_path.iterdir()): - raise TrainingLaunchError( - "training log has no resumable checkpoint; reconcile it before retrying" - ) - if selected is None: - return None - return ResumeBinding( - batch=int(selected.batch), - checkpoint=_checkpoint_identity(selected), - ) - - -@contextmanager -def _sealed_cookbook_resume( - *, - stage: StageSpec, - wave_name: str, - log_path: Path, - prior_wave_receipt: TrainingRecord | None, - expected: ResumeBinding | None, -) -> Iterator[None]: - """Make Cookbook consume exactly the optimizer state in the invocation.""" - - original = checkpoint_utils.get_last_checkpoint - expected_log = log_path.expanduser().resolve(strict=False) - fresh_at_entry = False - if expected is None: - observed_at_entry = _validate_resume_state( - stage=stage, - wave_name=wave_name, - log_path=log_path, - prior_wave_receipt=prior_wave_receipt, - ) - if observed_at_entry is not None: - raise TrainingLaunchError( - "optimizer state changed after the immutable invocation was recorded" - ) - fresh_at_entry = not log_path.exists() or not any(log_path.iterdir()) - - def sealed(log_dir: str, required_key: str = "state_path") -> Any: - requested = Path(log_dir).expanduser().resolve(strict=False) - if requested != expected_log: - return original(log_dir, required_key=required_key) - if fresh_at_entry: - if stage.kind == "sft": - if log_path.exists() and ( - not log_path.is_dir() or any(log_path.iterdir()) - ): - raise TrainingLaunchError( - "fresh SFT resume boundary contains unexpected training artifacts" - ) - return None - if not log_path.is_dir(): - raise TrainingLaunchError( - "fresh Cookbook bootstrap did not create its training log directory" - ) - entries = tuple(log_path.iterdir()) - names = {entry.name for entry in entries} - if ( - names != _COOKBOOK_FRESH_RESUME_BOOTSTRAP_FILES - or any(entry.is_symlink() or not entry.is_file() for entry in entries) - ): - raise TrainingLaunchError( - "fresh Cookbook bootstrap created unexpected training artifacts" - ) - try: - config = json.loads( - (log_path / "config.json").read_text(encoding="utf-8") - ) - except (OSError, UnicodeError, json.JSONDecodeError) as exc: - raise TrainingLaunchError( - "fresh Cookbook bootstrap config is not valid JSON" - ) from exc - if not isinstance(config, dict): - raise TrainingLaunchError( - "fresh Cookbook bootstrap config is not a JSON object" - ) - return None - observed = _validate_resume_state( - stage=stage, - wave_name=wave_name, - log_path=log_path, - prior_wave_receipt=prior_wave_receipt, - ) - if observed != expected: - raise TrainingLaunchError( - "optimizer state changed after the immutable invocation was recorded" - ) - if observed is None: - return None - records = checkpoint_utils.load_checkpoints_file(str(log_path)) - matches = [ - item - for item in records - if item.state_path is not None - and _checkpoint_identity(item) == observed.checkpoint - ] - if len(matches) != 1 or not matches[0].has(required_key): - raise TrainingLaunchError( - "sealed optimizer checkpoint is missing at the Cookbook boundary" - ) - return matches[0] - - checkpoint_utils.get_last_checkpoint = sealed - try: - yield - finally: - checkpoint_utils.get_last_checkpoint = original - - -async def run_training_stage( - *, - repo_root: Path, - stage_id: str, - wave_name: str, - replicate_id: str, - expected_source_sha: str, - external_root: Path, - confirmation: str, - sft_main: Callable[[SFTConfig], Awaitable[None]] | None = None, - rl_main: Callable[[RLConfig], Awaitable[None]] | None = None, - wandb_access_validator: Callable[..., Mapping[str, Any]] = validate_wandb_access, -) -> dict[str, Any]: - """Run or resume one exact paid wave after every local gate succeeds.""" - - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol(root, require_committed=True) - source_git_sha = validate_source_sha(root, expected_source_sha) - if confirmation != protocol["launch"]["confirmation_token"]: - raise TrainingLaunchError("explicit paid-training confirmation is missing") - stage = stage_spec(protocol, stage_id) - try: - validate_stage_replicate( - protocol, - stage_id=stage.stage_id, - replicate_id=replicate_id, - ) - except ValueError as exc: - raise TrainingLaunchError(str(exc)) from exc - if wave_name not in stage.waves: - stage.wave(wave_name) - if not os.environ.get("TINKER_API_KEY"): - raise TrainingLaunchError("TINKER_API_KEY is not present") - if not os.environ.get("WANDB_API_KEY"): - raise TrainingLaunchError("WANDB_API_KEY is not present") - - preflight = stage_preflight( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - require_sandbox=stage.kind == "rl", - ) - store = TrainingStore(repo_root=root, external_root=external_root) - key = StageKey(STUDY_ID, stage.stage_id, replicate_id) - with store.acquire_stage_lock(key): - existing_manifest = store.load_manifest(key) - if ( - existing_manifest is not None - and existing_manifest.payload.get("source_git_sha") != source_git_sha - ): - raise TrainingLaunchError("existing child manifest belongs to another source commit") - if str(stage.wave(wave_name)["approval"]) == "rollout-health-receipt": - if existing_manifest is None: - raise TrainingLaunchError( - "rollout-health continuation has no completed smoke manifest" - ) - smoke_receipt = store.load_wave_receipt(key, wave="smoke") - if smoke_receipt is None: - raise TrainingLaunchError( - "rollout-health continuation has no smoke receipt" - ) - smoke_preflight = stage_preflight( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name="smoke", - require_sandbox=True, - ) - smoke_parent = _parent_binding( - repo_root=root, - protocol=protocol, - store=store, - key=key, - stage=stage, - source_git_sha=source_git_sha, - approval=None, - existing_manifest=existing_manifest, - ) - _validate_completed_wave_receipt( - repo_root=root, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name="smoke", - manifest=existing_manifest, - preflight=smoke_preflight, - parent=smoke_parent, - receipt=smoke_receipt, - ) - approval = _approval_record( - repo_root=root, - store=store, - key=key, - stage=stage, - wave_name=wave_name, - source_git_sha=source_git_sha, - protocol=protocol, - ) - parent = _parent_binding( - repo_root=root, - protocol=protocol, - store=store, - key=key, - stage=stage, - source_git_sha=source_git_sha, - approval=approval, - existing_manifest=existing_manifest, - ) - tracking = _tracking_binding( - protocol=protocol, - source_git_sha=source_git_sha, - key=key, - stage=stage, - store=store, - ) - manifest = store.create_or_verify_manifest( - key, - _manifest_payload( - protocol=protocol, - source_git_sha=source_git_sha, - stage=stage, - preflight=preflight, - tracking=tracking, - parent=parent, - ), - ) - existing = store.load_wave_receipt(key, wave=wave_name) - if existing is not None: - _validate_completed_wave_receipt( - repo_root=root, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name=wave_name, - manifest=manifest, - preflight=preflight, - parent=parent, - receipt=existing, - ) - return { - "status": "already_complete", - "stage_id": stage.stage_id, - "wave": wave_name, - "receipt": str(existing.relative_path), - "receipt_sha256": existing.record_sha256, - } - log_path = store.tinker_log_path(key) - prior_wave_name = _required_prior_wave(stage, wave_name) - prior_wave_receipt = ( - store.load_wave_receipt(key, wave=prior_wave_name) - if prior_wave_name is not None - else None - ) - resume_binding = _validate_resume_state( - stage=stage, - wave_name=wave_name, - log_path=log_path, - prior_wave_receipt=prior_wave_receipt, - ) - resume_batch = resume_binding.batch if resume_binding is not None else None - resume_checkpoint = ( - resume_binding.checkpoint if resume_binding is not None else None - ) - resume = resume_batch is not None - # Install one continuous environment before the first W&B SDK call. - # This avoids the SDK caching an unbound session during access - # validation. Programmatic init arguments below are a second, - # independently checked identity boundary. - with _wandb_environment(tracking, resume=resume): - try: - # Preflight and transition validation can be lengthy. Bind - # the actual live boundary to the clean commit once more - # after every local record has been resolved. - validate_source_sha(root, source_git_sha) - except (OSError, ValueError, subprocess.SubprocessError) as exc: - raise TrainingLaunchError( - "source changed after preflight; refusing live W&B or Tinker access" - ) from exc - wandb_access_validator(expected_entity=tracking["entity"]) - resolved_launch = _resolved_launch_binding( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - store=store, - key=key, - parent=parent, - resume_batch=resume_batch, - resume_checkpoint=resume_checkpoint, - ) - invocation_id = f"{wave_name}-{uuid.uuid4().hex}" - invocation = store.begin_invocation( - key, - invocation_id=invocation_id, - payload=_invocation_payload( - protocol=protocol, - stage=stage, - wave_name=wave_name, - source_git_sha=source_git_sha, - invocation_id=invocation_id, - manifest=manifest, - preflight=preflight, - tracking=tracking, - resume_batch=resume_batch, - resolved_launch=resolved_launch, - parent=parent, - resume_checkpoint=resume_checkpoint, - ), - ) - config = build_tinker_config( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - store=store, - key=key, - parent=parent, - invocation=invocation, - ) - - def record_live_tracking(observed: Mapping[str, Any]) -> None: - store.write_invocation_tracking( - key, - invocation_id=invocation_id, - payload={ - "invocation_record_sha256": invocation.record_sha256, - "run_manifest_record_sha256": manifest.record_sha256, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "expected": dict(tracking), - "observed": dict(observed), - }, - ) - - with _sealed_cookbook_resume( - stage=stage, - wave_name=wave_name, - log_path=log_path, - prior_wave_receipt=prior_wave_receipt, - expected=resume_binding, - ): - if stage.kind == "sft": - if sft_main is None: - from tinker_cookbook.supervised.train import main as resolved_main - else: - resolved_main = sft_main - if sft_main is None: - with _sealed_wandb_initialization( - tracking, - resume=resume, - on_initialized=record_live_tracking, - ): - await resolved_main(config) - else: - await resolved_main(config) - else: - if stage.model_key == "inkling": - from rl.track_b.tml_shim import install_inkling_runtime - - install_inkling_runtime( - effort=float(protocol["models"]["inkling"]["effort"]) - ) - if rl_main is None: - from tinker_cookbook.rl.train import main as resolved_main - else: - resolved_main = rl_main - if rl_main is None: - with _sealed_wandb_initialization( - tracking, - resume=resume, - on_initialized=record_live_tracking, - ): - await resolved_main(config) - else: - await resolved_main(config) - live_tracking = store.load_invocation_tracking( - key, - invocation_id=invocation_id, - ) - built_in_main = sft_main is None if stage.kind == "sft" else rl_main is None - if built_in_main and live_tracking is None: - raise TrainingLaunchError("training returned without an observed W&B tracking record") - receipt = store.write_wave_receipt( - key, - wave=wave_name, - payload=_checkpoint_receipt( - stage=stage, - wave_name=wave_name, - log_path=log_path, - manifest=manifest, - preflight=preflight, - invocation=invocation, - tracking=live_tracking, - ), - ) - return { - "status": "complete", - "stage_id": stage.stage_id, - "wave": wave_name, - "manifest": str(manifest.relative_path), - "manifest_sha256": manifest.record_sha256, - "receipt": str(receipt.relative_path), - "receipt_sha256": receipt.record_sha256, - "wandb_run_id": tracking["run_id"], - "checkpoint": receipt.payload["checkpoint"], - } diff --git a/rl/studies/representation_training_v1/preflight.py b/rl/studies/representation_training_v1/preflight.py deleted file mode 100644 index c8b83947..00000000 --- a/rl/studies/representation_training_v1/preflight.py +++ /dev/null @@ -1,381 +0,0 @@ -"""Zero-spend validation for one sealed training stage.""" - -from __future__ import annotations - -import hashlib -import importlib.metadata -import json -import subprocess -from pathlib import Path -from typing import Any -from urllib.parse import unquote, urlparse - -from tinker import SamplingParams -from tinker_cookbook.renderers import Message, TextPart, TrainOnWhat - -from homi.conductor.source_policy import strict_purity_violations - -from rl.common.evaluator import Attribution, EvaluationStatus, PixCellEvaluator -from rl.common.isolation import require_execution_boundary -from rl.common.prompt import prompt_asset_hashes -from rl.common.runtime import validate_runtime_stack -from rl.track_a.tinker_data import _message, _renderer - -from .dataset_binding import _dataset_binding -from .protocol import StageSpec, file_sha256, protocol_path -from .schedule import ( - stage_replay_tasks, - stage_schedule_document, - stage_tasks, - verify_sft_step_count, -) - - -def _positive_target_text(model_input: Any, weights: Any, tokenizer: Any) -> str: - offset = 0 - tokens: list[int] = [] - for chunk in model_input.chunks: - length = int(chunk.length) - chunk_weights = weights[offset : offset + length] - chunk_tokens = getattr(chunk, "tokens", None) - if chunk_tokens is not None: - tokens.extend( - int(token) - for token, weight in zip(chunk_tokens, chunk_weights, strict=True) - if float(weight) > 0 - ) - elif int((chunk_weights > 0).sum().item()) != 0: - raise RuntimeError("an image token unexpectedly carries SFT loss") - offset += length - if offset != int(model_input.length) or not tokens: - raise RuntimeError("SFT target mask is empty or misaligned") - return str(tokenizer.decode(tokens)) - - -def _audit_sft( - dataset_root: Path, - stage: StageSpec, - model: dict[str, Any], - recipe: dict[str, Any], - *, - max_steps: int, -) -> dict[str, Any]: - verify_sft_step_count(dataset_root, stage, max_steps=max_steps) - renderer = _renderer( - model["model"], - model["renderer"], - effort=model.get("effort"), - ) - lengths: list[int] = [] - trained: list[int] = [] - for task in stage_tasks(dataset_root, stage): - assistant = Message( - role="assistant", - content=[TextPart(type="text", text=task.label)], - ) - model_input, weights = renderer.build_supervised_example( - [_message(task, max_image=int(model["max_image_long_edge"])), assistant], - train_on_what=TrainOnWhat.LAST_ASSISTANT_MESSAGE, - ) - length = int(model_input.length) - positive = int((weights > 0).sum().item()) - if length > int(recipe["max_sequence_tokens"]): - raise RuntimeError( - f"{task.sampler.opaque_id} has {length} tokens, exceeding " - f"{recipe['max_sequence_tokens']}" - ) - target = _positive_target_text(model_input, weights, renderer.tokenizer) - expected_target = task.label + "<|im_end|>" - if target != expected_target: - raise RuntimeError( - f"{task.sampler.opaque_id} does not have an exact code-only SFT target" - ) - lengths.append(length) - trained.append(positive) - return { - "rows": len(lengths), - "sequence_tokens": { - "min": min(lengths), - "max": max(lengths), - }, - "trained_tokens": { - "min": min(trained), - "max": max(trained), - }, - "target_policy": recipe["target"], - } - - -def _audit_rl_context( - dataset_root: Path, - stage: StageSpec, - model: dict[str, Any], -) -> dict[str, Any]: - sampling_defaults = SamplingParams( - max_tokens=int(model["max_output_tokens"]), - temperature=1.0, - ) - if float(sampling_defaults.top_p) != 1.0: - raise RuntimeError("pinned Tinker top_p default is no longer 1.0") - renderer = _renderer( - model["model"], - model["renderer"], - effort=model.get("effort"), - ) - lengths: list[int] = [] - tasks = stage_tasks(dataset_root, stage) + stage_replay_tasks( - dataset_root, - stage, - ) - for task in tasks: - prompt = renderer.build_generation_prompt( - [_message(task, max_image=int(model["max_image_long_edge"]))] - ) - length = int(prompt.length) - if length + int(model["max_output_tokens"]) > int(model["context_tokens"]): - raise RuntimeError( - f"{task.sampler.opaque_id} leaves no room for the sealed output cap" - ) - lengths.append(length) - return { - "rows": len(lengths), - "prompt_tokens": { - "min": min(lengths), - "max": max(lengths), - }, - "minimum_context_headroom": ( - int(model["context_tokens"]) - - int(model["max_output_tokens"]) - - max(lengths) - ), - "sampling_defaults": { - "top_p": float(sampling_defaults.top_p), - }, - } - - -def _source_and_reference_audit( - dataset_root: Path, - stage: StageSpec, - *, - execute_probe: bool, -) -> dict[str, Any]: - tasks = stage_tasks(dataset_root, stage) - if stage.kind == "rl": - tasks.extend(stage_replay_tasks(dataset_root, stage)) - unique = {task.sampler.opaque_id: task for task in tasks} - source_sha = hashlib.sha256() - reference_sha = hashlib.sha256() - with PixCellEvaluator( - max_workers=1, - evaluator_retries=1, - require_isolation=execute_probe, - ) as evaluator: - for task_id, task in sorted(unique.items()): - violations = strict_purity_violations(task.label) - if violations: - raise RuntimeError( - f"{task_id} violates the strict source policy: {violations}" - ) - evidence = evaluator.validate_reference(task.reference) - if ( - evidence["target_image_sha256"] - != task.reference.target_image_sha256 - ): - raise RuntimeError(f"{task_id} reference digest changed") - source_sha.update(task_id.encode("utf-8")) - source_sha.update(hashlib.sha256(task.label.encode("utf-8")).digest()) - reference_sha.update(task_id.encode("utf-8")) - reference_sha.update( - bytes.fromhex(task.reference.target_image_sha256) - ) - probe: dict[str, Any] | None = None - if execute_probe: - first = unique[sorted(unique)[0]] - result = evaluator.evaluate(first.reference, first.label) - if ( - result.status is not EvaluationStatus.OK - or result.attribution is not Attribution.MODEL - or result.iou is None - or result.iou < 0.99 - ): - raise RuntimeError( - "isolated ground-truth probe failed: " - f"{result.status.value}/{result.attribution.value}/" - f"{result.iou}: {result.error}" - ) - probe = { - "task_id": first.sampler.opaque_id, - "status": result.status.value, - "iou": result.iou, - } - return { - "rows": len(unique), - "source_set_sha256": source_sha.hexdigest(), - "reference_set_sha256": reference_sha.hexdigest(), - "sandbox_probe": probe, - } - - -def _cookbook_source_binding(runtime: dict[str, Any]) -> dict[str, Any]: - expected = str(runtime["tinker_cookbook_commit"]) - distribution = importlib.metadata.distribution("tinker-cookbook") - version = importlib.metadata.version("tinker-cookbook") - direct = distribution.read_text("direct_url.json") - if not direct: - if f"g{expected[:10]}" not in version: - raise RuntimeError( - "installed Tinker Cookbook does not identify the pinned commit" - ) - return { - "kind": "installed-distribution", - "version": version, - "expected_commit": expected, - } - document = json.loads(direct) - if document.get("dir_info", {}).get("editable") is not True: - if f"g{expected[:10]}" not in version: - raise RuntimeError( - "direct Tinker Cookbook install does not identify the pinned commit" - ) - return { - "kind": "direct-install", - "version": version, - "expected_commit": expected, - "direct_url_sha256": hashlib.sha256(direct.encode()).hexdigest(), - } - parsed = urlparse(str(document.get("url", ""))) - if parsed.scheme != "file": - raise RuntimeError("editable Tinker Cookbook is not a local file checkout") - checkout = Path(unquote(parsed.path)).resolve(strict=True) - head = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=checkout, - text=True, - ).strip() - status = subprocess.check_output( - ["git", "status", "--porcelain", "--untracked-files=all"], - cwd=checkout, - text=True, - ).strip() - if head != expected or status: - raise RuntimeError( - "editable Tinker Cookbook differs from the pinned clean commit" - ) - return { - "kind": "clean-editable-checkout", - "path": str(checkout), - "git_sha": head, - } - - -def _sandbox_binding() -> dict[str, Any]: - boundary = require_execution_boundary() - return { - "runtime_path": str(boundary.runtime_path), - "daemon_endpoint": boundary.daemon_endpoint, - "image_ref": boundary.image_ref, - "image_id": boundary.image_id, - "workspace_root": str(boundary.workspace_root), - } - - -def _benchmark_binding(repo_root: Path, protocol: dict[str, Any]) -> dict[str, Any]: - manifest_path = ( - protocol_path(repo_root).parent / protocol["evaluation"]["task_manifest"] - ).resolve(strict=True) - if not manifest_path.is_relative_to(repo_root.resolve()): - raise RuntimeError("benchmark manifest escaped the repository") - document = json.loads(manifest_path.read_text(encoding="utf-8")) - expected = protocol["evaluation"]["task_manifest_logical_sha256"] - if document.get("logical_sha256") != expected: - raise RuntimeError("benchmark task manifest differs from the protocol") - unsigned = {key: value for key, value in document.items() if key != "logical_sha256"} - logical = hashlib.sha256( - json.dumps( - unsigned, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - ).hexdigest() - if logical != expected: - raise RuntimeError("benchmark task manifest has an invalid logical digest") - task_set = document.get("task_sets", {}).get("f1_f8") - if not isinstance(task_set, list) or len(task_set) != 8: - raise RuntimeError("benchmark manifest does not contain exact F1-F8") - return { - "relative_path": str(manifest_path.relative_to(repo_root)), - "file_sha256": file_sha256(manifest_path), - "logical_sha256": logical, - "task_ids": [item["task_id"] for item in task_set], - } - - -def stage_preflight( - *, - repo_root: Path, - protocol: dict[str, Any], - stage: StageSpec, - wave_name: str, - require_sandbox: bool, -) -> dict[str, Any]: - """Validate all deterministic launch inputs without creating a client.""" - - dataset_root = repo_root / "dataset" - wave = stage.wave(wave_name) - model = dict(protocol["models"][stage.model_key]) - recipe = dict(protocol["recipes"][stage.recipe_key]) - schedule = stage_schedule_document( - dataset_root, - stage, - groups_per_batch=( - int(recipe["groups_per_batch"]) if stage.kind == "rl" else None - ), - ) - policy_audit = ( - _audit_sft( - dataset_root, - stage, - model, - recipe, - max_steps=int(wave["max_steps"]), - ) - if stage.kind == "sft" - else _audit_rl_context(dataset_root, stage, model) - ) - protocol_file = protocol_path(repo_root) - runtime = validate_runtime_stack(repo_root) - report = { - "ok": True, - "spend": False, - "stage_id": stage.stage_id, - "wave": wave_name, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": file_sha256(protocol_file), - "prompt_assets": prompt_asset_hashes(), - "benchmark": _benchmark_binding(repo_root, protocol), - "dataset": _dataset_binding(dataset_root, protocol), - "runtime": { - **runtime, - "cookbook_source": _cookbook_source_binding(runtime), - }, - "schedule": schedule, - "model": model, - "recipe": recipe, - "policy_audit": policy_audit, - "source_and_reference_audit": _source_and_reference_audit( - dataset_root, - stage, - execute_probe=require_sandbox, - ), - "sandbox": _sandbox_binding() if require_sandbox else None, - } - report["preflight_sha256"] = hashlib.sha256( - json.dumps( - report, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - ).hexdigest() - return report diff --git a/rl/studies/representation_training_v1/promotion.py b/rl/studies/representation_training_v1/promotion.py deleted file mode 100644 index 85be6cb9..00000000 --- a/rl/studies/representation_training_v1/promotion.py +++ /dev/null @@ -1,2773 +0,0 @@ -"""Fail-closed, human-confirmed promotion gates for the training study. - -This module never samples a model and never creates a Tinker client. It turns -one already-sealed checkpoint-evaluation report into the immutable approval -record consumed by :mod:`launcher`. -""" - -from __future__ import annotations - -import math -import re -import subprocess -from collections import Counter, defaultdict -from collections.abc import Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from rl.common.prompt import prompt_asset_hashes -from rl.evaluation.tasks import canonical_json_sha256 - -from .dataset_binding import _dataset_binding -from .protocol import ( - LEVELS, - STUDY_ID, - StageSpec, - file_sha256, - load_protocol, - parent_replicate_id, - protocol_path, - stage_spec, - validate_stage_replicate, - validate_source_sha, -) -from .store import ( - StageKey, - TrainingRecord, - TrainingStore, - canonical_sha256, -) -from .source_transition import ( - SourceTransitionError, - approved_change_scope, - build_source_transition, - source_transition_git_facts, -) - - -APPROVAL_SCHEMA_VERSION = "pixcell-training-wave-approval-v1" -SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION = "pixcell-training-wave-approval-v2" -PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION = ( - "pixcell-training-parent-receipt-transition-approval-v1" -) -ROLLOUT_HEALTH_APPROVAL_SCHEMA_VERSION = ( - "pixcell-training-rollout-health-approval-v1" -) -BASE_BASELINE_APPROVAL_SCHEMA_VERSION = ( - "pixcell-training-base-baseline-approval-v1" -) -APPROVAL_CONFIRMATION_TOKEN = "PIXCELL_APPROVE_TRAINING_V1" -MASTERED_MINIMUM_IOU = 0.80 -MASTERED_MINIMUM_PURE_EXECUTABLE_RATE = 0.95 - -PANEL_PROGRESS = "progress" -PANEL_DEPTH_VALIDATION = "depth-validation" -PANEL_INKLING_PROMOTION = "inkling-promotion" -LEVEL_PROGRESS_PANELS = { - "L0": "level-progress-l0", - "L1": "level-progress-l1", - "L2": "level-progress-l2", - "L3": "level-progress-l3", - "L4": "level-progress-l4", -} -PANEL_DEPTH_FINAL_SELECTION = "depth-final-selection" -_INKLING_PANEL_SELECTION_SEED = ( - "pixcell-representation-training-v1-inkling-promotion-20260728" -) -_LEVEL_PROGRESS_SELECTION = "validation-realization-slot-6" -_DEPTH_FINAL_SELECTION = "validation-realization-slot-7" -_MODEL_EVALUATION_STATUSES = frozenset( - { - "ok", - "syntax_error", - "source_rejected", - "runtime_timeout", - "runtime_error", - "no_gds", - "bad_gds", - } -) -EXPECTED_PANEL_ROWS = { - PANEL_PROGRESS: 8, - PANEL_DEPTH_VALIDATION: 1_092, - PANEL_INKLING_PROMOTION: 68, - "level-progress-l0": 93, - "level-progress-l1": 127, - "level-progress-l2": 98, - "level-progress-l3": 120, - "level-progress-l4": 108, - PANEL_DEPTH_FINAL_SELECTION: 546, -} - -_REPORT_SCHEMA = "pixcell-training-checkpoint-evaluation-v1" -_SHA256 = re.compile(r"^[0-9a-f]{64}$") -_TINKER_PATH = re.compile(r"^tinker://[^\s]+$") -_IMMUTABLE_IMAGE = re.compile(r"^(?:sha256:[0-9a-f]{64}|[^\s@]+@sha256:[0-9a-f]{64})$") - - -class PromotionError(RuntimeError): - """An evaluation cannot authorize the requested training transition.""" - - -@dataclass(frozen=True) -class PanelTaskIdentity: - task_id: str - level: str - representation_id: str - image_sha256: str - target_image_sha256: str - footprint_um: tuple[float, float] - - -@dataclass(frozen=True) -class EvaluationEvidence: - """A fully checked immutable checkpoint report.""" - - key: StageKey - wave: str - checkpoint_name: str - checkpoint: dict[str, Any] - checkpoint_inventory_sha256: str - panel: str - report: dict[str, Any] - report_payload: dict[str, Any] - report_record_sha256: str - report_payload_sha256: str - report_relative_path: str - evaluation_manifest_payload_sha256: str - evaluation_manifest_record_sha256: str - evaluation_manifest_relative_path: str - training_manifest: TrainingRecord - training_receipt: TrainingRecord - - -@dataclass(frozen=True) -class ApprovalPlan: - gate: str - storage_wave: str - expected_panel: str - allowed_evaluation_checkpoints: tuple[tuple[str, str], ...] - entry_baseline_panel: str | None = None - - -def _mapping(value: Any, path: str) -> Mapping[str, Any]: - if not isinstance(value, Mapping): - raise PromotionError(f"{path} must be an object") - return value - - -def _integer(value: Any, path: str, *, minimum: int = 0) -> int: - if isinstance(value, bool) or not isinstance(value, int) or value < minimum: - raise PromotionError(f"{path} must be an integer >= {minimum}") - return value - - -def _unit(value: Any, path: str) -> float: - if isinstance(value, bool): - raise PromotionError(f"{path} must be numeric") - try: - result = float(value) - except (TypeError, ValueError) as exc: - raise PromotionError(f"{path} must be numeric") from exc - if not math.isfinite(result) or not 0.0 <= result <= 1.0: - raise PromotionError(f"{path} must be finite and in [0, 1]") - return result - - -def _close(actual: Any, expected: float, path: str) -> None: - value = _unit(actual, path) - if not math.isclose(value, expected, rel_tol=0.0, abs_tol=1e-12): - raise PromotionError(f"{path}={value!r}; expected {expected!r}") - - -def _sha(value: Any, path: str) -> str: - result = str(value) - if not _SHA256.fullmatch(result): - raise PromotionError(f"{path} must be a lowercase SHA-256") - return result - - -def _finite_json(value: Any, path: str) -> None: - if value is None or isinstance(value, (str, bool, int)): - return - if isinstance(value, float): - if not math.isfinite(value): - raise PromotionError(f"{path} contains a non-finite number") - return - if isinstance(value, Mapping): - for key, item in value.items(): - _finite_json(item, f"{path}.{key}") - return - if isinstance(value, (list, tuple)): - for index, item in enumerate(value): - _finite_json(item, f"{path}[{index}]") - return - raise PromotionError(f"{path} contains unsupported metric data") - - -def _parent_receipt_transition_evidence( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - replicate_id: str, -) -> tuple[TrainingRecord, TrainingRecord, TrainingRecord, dict[str, Any], dict[str, Any]]: - """Revalidate the completed historical L0-SFT parent without an evaluation.""" - - parent_key = StageKey(STUDY_ID, "qwen-l0-sft", replicate_id) - manifest = store.load_manifest(parent_key) - receipt = store.load_wave_receipt(parent_key, wave="complete") - if manifest is None or receipt is None: - raise PromotionError("the exact qwen-l0-sft/complete parent receipt is missing") - - parent_source = str(manifest.payload.get("source_git_sha", "")) - if not re.fullmatch(r"[0-9a-f]{40}", parent_source): - raise PromotionError("parent manifest has an invalid source commit") - if ( - receipt.payload.get("stage_id") != "qwen-l0-sft" - or receipt.payload.get("wave") != "complete" - or receipt.payload.get("run_manifest_record_sha256") != manifest.record_sha256 - ): - raise PromotionError("parent receipt and manifest identities differ") - - parent_stage = _mapping(manifest.payload.get("stage"), "parent_manifest.stage") - current_parent = stage_spec(dict(protocol), "qwen-l0-sft") - expected_stage = { - "stage_id": current_parent.stage_id, - "kind": current_parent.kind, - "model_key": current_parent.model_key, - "recipe_key": current_parent.recipe_key, - "hypotheses": list(current_parent.hypotheses), - "parent_policy": current_parent.parent, - "levels": list(current_parent.levels), - "current_level": current_parent.current_level, - "replay_levels": list(current_parent.replay_levels), - "waves": current_parent.waves, - } - local_dataset = _dataset_binding(repo_root / "dataset", protocol) - stable_equivalence = { - "contract_version": protocol["contract_version"], - "prompt_assets": prompt_asset_hashes(), - "dataset": local_dataset, - "model": dict(protocol["models"]["qwen"]), - "recipe": dict(protocol["recipes"]["qwen_sft"]), - "stage": expected_stage, - } - observed_equivalence = { - "contract_version": manifest.payload.get("contract_version"), - "prompt_assets": manifest.payload.get("prompt_assets"), - "dataset": manifest.payload.get("dataset"), - "model": manifest.payload.get("model"), - "recipe": manifest.payload.get("recipe"), - "stage": dict(parent_stage), - } - if observed_equivalence != stable_equivalence: - raise PromotionError( - "historical L0-SFT parent differs from the current stable " - "contract, prompt, dataset, model, recipe, or stage" - ) - - inventory = _mapping( - receipt.payload.get("checkpoint_inventory"), - "parent_receipt.checkpoint_inventory", - ) - entries = inventory.get("entries") - logical_sha256 = _sha( - inventory.get("logical_sha256"), - "parent_receipt.checkpoint_inventory.logical_sha256", - ) - if ( - not isinstance(entries, list) - or inventory.get("count") != len(entries) - or canonical_sha256(entries) != logical_sha256 - ): - raise PromotionError("parent checkpoint inventory digest differs") - terminals = [ - item - for item in entries - if isinstance(item, Mapping) - and item.get("role") == "terminal" - and item.get("final") is True - ] - if len(terminals) != 1: - raise PromotionError("parent checkpoint inventory has no unique terminal") - checkpoint = dict(terminals[0]) - selected = _mapping(receipt.payload.get("checkpoint"), "parent_receipt.checkpoint") - selected_fields = ("name", "batch", "epoch", "final", "state_path", "sampler_path") - if ( - any(selected.get(field) != checkpoint.get(field) for field in selected_fields) - or checkpoint.get("name") != "final" - or checkpoint.get("epoch") != 1 - or not _TINKER_PATH.fullmatch(str(checkpoint.get("state_path", ""))) - or not _TINKER_PATH.fullmatch(str(checkpoint.get("sampler_path", ""))) - ): - raise PromotionError("parent selected checkpoint is not the completed SFT terminal") - - invocation_id = str(receipt.payload.get("invocation_id", "")) - try: - invocation = store.load_invocation(parent_key, invocation_id=invocation_id) - except ValueError as exc: - raise PromotionError("parent receipt has an invalid invocation ID") from exc - if ( - invocation is None - or receipt.payload.get("invocation_record_sha256") != invocation.record_sha256 - or invocation.payload.get("run_manifest_record_sha256") != manifest.record_sha256 - or invocation.payload.get("source_git_sha") != parent_source - ): - raise PromotionError("parent invocation is missing or differs") - - artifacts = _mapping( - receipt.payload.get("local_artifact_sha256"), - "parent_receipt.local_artifact_sha256", - ) - if not {"checkpoints.jsonl", "metrics.jsonl"}.issubset(artifacts): - raise PromotionError("parent receipt lacks its local checkpoint evidence") - for name, expected_sha in artifacts.items(): - _sha(expected_sha, f"parent_receipt.local_artifact_sha256.{name}") - path = store.tinker_log_path(parent_key) / str(name) - if not path.is_file() or file_sha256(path) != expected_sha: - raise PromotionError(f"parent local artifact {name} differs from its receipt") - - evidence = { - "parent_run_manifest": { - "relative_path": str(manifest.relative_path), - "payload_sha256": manifest.payload_sha256, - "record_sha256": manifest.record_sha256, - }, - "parent_wave_receipt": { - "relative_path": str(receipt.relative_path), - "payload_sha256": receipt.payload_sha256, - "record_sha256": receipt.record_sha256, - }, - "parent_invocation": { - "relative_path": str(invocation.relative_path), - "payload_sha256": invocation.payload_sha256, - "record_sha256": invocation.record_sha256, - }, - "local_artifact_sha256": dict(artifacts), - "checkpoint_inventory": dict(inventory), - "selected_checkpoint": checkpoint, - } - return manifest, receipt, invocation, checkpoint, { - "stable_equivalence": stable_equivalence, - "evidence": evidence, - } - - -def _parent_receipt_transition_payload( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - stage: StageSpec, - wave_name: str, - replicate_id: str, - consumer_source_git_sha: str, - approved_change_paths: Sequence[str] | None, -) -> dict[str, Any]: - if ( - stage.stage_id != "qwen-l0-rl-l1" - or stage.parent != "qwen-l0-sft:complete" - or wave_name != "smoke" - or str(stage.wave(wave_name)["approval"]) - != "parent-receipt-and-source-transition" - ): - raise PromotionError("parent-receipt transition is not the L0-SFT→L1-RL entry") - producer_replicate_id = parent_replicate_id( - protocol, - stage_id=stage.stage_id, - replicate_id=replicate_id, - ) - manifest, receipt, invocation, checkpoint, documents = ( - _parent_receipt_transition_evidence( - repo_root=repo_root, - protocol=protocol, - store=store, - replicate_id=producer_replicate_id, - ) - ) - producer_source_git_sha = str(manifest.payload["source_git_sha"]) - if producer_source_git_sha == consumer_source_git_sha: - raise PromotionError( - "historical-parent transition requires different producer and consumer commits" - ) - scope = { - "study_id": STUDY_ID, - "child": { - "stage_id": stage.stage_id, - "wave": wave_name, - "replicate_id": replicate_id, - "approval_gate": "parent-receipt-and-source-transition", - }, - "producer_training": { - "stage_id": "qwen-l0-sft", - "wave": "complete", - "replicate_id": producer_replicate_id, - }, - } - equivalence = { - "producer_protocol": { - "logical_sha256": manifest.payload.get("protocol_logical_sha256"), - "file_sha256": manifest.payload.get("protocol_file_sha256"), - }, - "consumer_protocol": { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - }, - **documents["stable_equivalence"], - } - try: - transition = build_source_transition( - repo_root=repo_root, - producer_source_git_sha=producer_source_git_sha, - consumer_source_git_sha=consumer_source_git_sha, - scope=scope, - approved_change_paths=approved_change_paths, - equivalence=equivalence, - evidence=documents["evidence"], - ) - except SourceTransitionError as exc: - raise PromotionError(f"parent source transition is invalid: {exc}") from exc - inventory = _mapping( - receipt.payload.get("checkpoint_inventory"), - "parent_receipt.checkpoint_inventory", - ) - return { - "schema_version": PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "parent-receipt-and-source-transition", - "stage_id": stage.stage_id, - "wave": wave_name, - "source_git_sha": consumer_source_git_sha, - "evidence_source_git_sha": producer_source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "explicit_human_confirmation": True, - "parent_stage_id": "qwen-l0-sft", - "parent_wave": "complete", - "parent_replicate_id": producer_replicate_id, - "parent_manifest_record_sha256": manifest.record_sha256, - "parent_receipt_record_sha256": receipt.record_sha256, - "parent_invocation_record_sha256": invocation.record_sha256, - "selected_parent_checkpoint": checkpoint, - "selected_parent_checkpoint_inventory_sha256": inventory["logical_sha256"], - "source_transition": transition, - } - - -def validate_parent_receipt_transition_approval( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - stage: StageSpec, - wave_name: str, - replicate_id: str, - consumer_source_git_sha: str, - approval: TrainingRecord, -) -> None: - """Recompute the receipt-only historical-parent bridge at launch time.""" - - if ( - approval.payload.get("schema_version") - != PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ): - raise PromotionError("parent-receipt transition approval has a foreign schema") - if approval.payload.get("source_git_sha") != consumer_source_git_sha: - raise PromotionError("parent-receipt transition names another consumer") - try: - validate_source_sha(repo_root, consumer_source_git_sha) - except (OSError, ValueError, subprocess.SubprocessError) as exc: - raise PromotionError( - f"parent-receipt transition consumer is not the clean HEAD: {exc}" - ) from exc - approved_paths = _approved_change_paths_from_transition( - approval.payload.get("source_transition") - ) - expected = _parent_receipt_transition_payload( - repo_root=repo_root, - protocol=protocol, - store=store, - stage=stage, - wave_name=wave_name, - replicate_id=replicate_id, - consumer_source_git_sha=consumer_source_git_sha, - approved_change_paths=approved_paths, - ) - if approval.payload != expected: - raise PromotionError( - "parent-receipt transition approval differs from recomputed evidence" - ) - - -def record_parent_receipt_transition_approval( - *, - repo_root: Path, - external_root: Path, - stage_id: str, - wave_name: str, - replicate_id: str, - expected_source_sha: str, - approved_change_paths: Sequence[str] | None, - confirmation: str, -) -> dict[str, Any]: - """Record the one explicit bridge from historical L0 SFT to current L1 RL.""" - - if confirmation != APPROVAL_CONFIRMATION_TOKEN: - raise PromotionError("explicit human approval confirmation is missing") - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol(root, require_committed=True) - source_git_sha = validate_source_sha(root, expected_source_sha) - stage = stage_spec(protocol, stage_id) - try: - validate_stage_replicate( - protocol, - stage_id=stage.stage_id, - replicate_id=replicate_id, - ) - except ValueError as exc: - raise PromotionError(str(exc)) from exc - store = TrainingStore(repo_root=root, external_root=external_root) - payload = _parent_receipt_transition_payload( - repo_root=root, - protocol=protocol, - store=store, - stage=stage, - wave_name=wave_name, - replicate_id=replicate_id, - consumer_source_git_sha=source_git_sha, - approved_change_paths=approved_change_paths, - ) - key = StageKey(STUDY_ID, stage.stage_id, replicate_id) - with store.acquire_stage_lock(key): - validate_source_sha(root, source_git_sha) - rebuilt = _parent_receipt_transition_payload( - repo_root=root, - protocol=protocol, - store=store, - stage=stage, - wave_name=wave_name, - replicate_id=replicate_id, - consumer_source_git_sha=source_git_sha, - approved_change_paths=approved_change_paths, - ) - if rebuilt != payload: - raise PromotionError("parent-receipt transition changed before recording") - approval = store.write_wave_approval( - key, - wave=wave_name, - payload=payload, - ) - return { - "status": "approved", - "stage_id": stage.stage_id, - "wave": wave_name, - "approval": str(approval.relative_path), - "approval_record_sha256": approval.record_sha256, - "parent_receipt_record_sha256": payload["parent_receipt_record_sha256"], - "source_transition_logical_sha256": payload["source_transition"][ - "logical_sha256" - ], - } - - -def record_rollout_health_approval( - *, - store: TrainingStore, - key: StageKey, - stage: StageSpec, - wave_name: str, - prior_wave: str, - prior_receipt: TrainingRecord, - source_git_sha: str, - protocol: Mapping[str, Any], -) -> TrainingRecord: - """Derive the smoke→step-5 decision from exactly one sealed G4 receipt.""" - - if ( - stage.kind != "rl" - or stage.model_key != "qwen" - or wave_name != "step-5" - or prior_wave != "smoke" - or str(stage.wave(wave_name)["approval"]) != "rollout-health-receipt" - or int(stage.wave(prior_wave)["max_steps"]) != 1 - ): - raise PromotionError("rollout-health gate is not the Qwen smoke→step-5 transition") - recipe = _mapping(protocol["recipes"].get(stage.recipe_key), "recipe") - group_size = _integer(recipe.get("group_size"), "recipe.group_size", minimum=1) - groups_per_batch = _integer( - recipe.get("groups_per_batch"), - "recipe.groups_per_batch", - minimum=1, - ) - if group_size != 4 or groups_per_batch != 8: - raise PromotionError("rollout-health gate requires the sealed 8×G4 recipe") - expected_candidates = group_size * groups_per_batch - - if ( - prior_receipt.payload.get("stage_id") != stage.stage_id - or prior_receipt.payload.get("wave") != prior_wave - or prior_receipt.payload.get("max_steps") != 1 - ): - raise PromotionError("rollout-health receipt identity differs") - invocation_id = str(prior_receipt.payload.get("invocation_id", "")) - invocation = store.load_invocation(key, invocation_id=invocation_id) - if ( - invocation is None - or prior_receipt.payload.get("invocation_record_sha256") - != invocation.record_sha256 - ): - raise PromotionError("rollout-health receipt invocation is missing or differs") - manifest = store.load_manifest(key) - if ( - manifest is None - or prior_receipt.payload.get("run_manifest_record_sha256") - != manifest.record_sha256 - or manifest.payload.get("protocol_logical_sha256") - != protocol["logical_sha256"] - ): - raise PromotionError("rollout-health receipt manifest differs") - - recorded_inventory = _mapping( - prior_receipt.payload.get("candidate_inventory"), - "receipt.candidate_inventory", - ) - recorded_samples = _mapping( - prior_receipt.payload.get("sample_inventory"), - "receipt.sample_inventory", - ) - for name, inventory in ( - ("candidate", recorded_inventory), - ("sample", recorded_samples), - ): - if ( - _integer(inventory.get("count"), f"{name}_inventory.count") - != expected_candidates - or _integer( - inventory.get("expected_count"), - f"{name}_inventory.expected_count", - ) - != expected_candidates - ): - raise PromotionError( - f"rollout-health {name} inventory does not contain exactly " - f"{expected_candidates} records" - ) - _sha(inventory.get("logical_sha256"), f"{name}_inventory.logical_sha256") - - candidate_root = store.candidate_root(key) / invocation_id - candidate_files = ( - sorted(candidate_root.rglob("evaluation.json")) - if candidate_root.is_dir() - else [] - ) - sample_files = ( - sorted(candidate_root.rglob("sample.json")) - if candidate_root.is_dir() - else [] - ) - observed_inventory = [ - { - "path": str(path.relative_to(store.stage_path(key))), - "sha256": file_sha256(path), - } - for path in candidate_files - ] - if ( - len(candidate_files) != expected_candidates - or canonical_sha256(observed_inventory) - != recorded_inventory["logical_sha256"] - ): - raise PromotionError( - "rollout-health candidate files differ from the sealed receipt" - ) - observed_sample_inventory = [ - { - "path": str(path.relative_to(store.stage_path(key))), - "sha256": file_sha256(path), - } - for path in sample_files - ] - if ( - len(sample_files) != expected_candidates - or canonical_sha256(observed_sample_inventory) - != recorded_samples["logical_sha256"] - ): - raise PromotionError( - "rollout-health sample files differ from the sealed receipt" - ) - - rewards_by_group: dict[tuple[int, str], dict[int, float]] = defaultdict(dict) - executable = 0 - rewards: list[float] = [] - record_bindings: list[dict[str, Any]] = [] - sample_record_bindings: list[dict[str, Any]] = [] - from rl.common.evaluator import EvaluationStatus - - allowed_statuses = {item.value for item in EvaluationStatus} - for path in candidate_files: - relative = path.relative_to(candidate_root) - parts = relative.parts - if ( - len(parts) != 4 - or not re.fullmatch(r"step-[0-9]{6}", parts[0]) - or not re.fullmatch(r"attempt-[0-9]{2}", parts[2]) - or parts[3] != "evaluation.json" - ): - raise PromotionError("rollout-health candidate path is not canonical") - step = int(parts[0].removeprefix("step-")) - task_id = parts[1] - attempt = int(parts[2].removeprefix("attempt-")) - if step != 0: - raise PromotionError("rollout-health smoke contains a nonzero step") - record = store.load_candidate( - key, - invocation_id=invocation_id, - step=step, - task_id=task_id, - attempt=attempt, - ) - if record is None: - raise PromotionError("rollout-health candidate record disappeared") - payload = _mapping(record.payload, "candidate.payload") - sample_binding = _mapping(payload.get("sample_record"), "candidate.sample_record") - sample = store.load_candidate_sample( - key, - invocation_id=invocation_id, - step=step, - task_id=task_id, - attempt=attempt, - ) - if ( - sample is None - or sample_binding.get("relative_path") != str(sample.relative_path) - or sample_binding.get("payload_sha256") != sample.payload_sha256 - or sample_binding.get("record_sha256") != sample.record_sha256 - ): - raise PromotionError( - "rollout-health candidate is not bound to its paid sample" - ) - evaluation = _mapping(payload.get("evaluation"), "candidate.evaluation") - _finite_json(evaluation, "candidate.evaluation") - attribution = str(evaluation.get("attribution", "")) - if attribution != "model": - raise PromotionError( - "rollout-health contains an evaluator or reference fault" - ) - reward = _unit(evaluation.get("reward"), "candidate.evaluation.reward") - status = str(evaluation.get("status", "")) - if status not in allowed_statuses: - raise PromotionError("rollout-health contains an unknown evaluator status") - if status not in _MODEL_EVALUATION_STATUSES: - raise PromotionError( - "rollout-health contains an evaluator or reference status " - "mislabeled as model-attributed" - ) - iou = evaluation.get("iou") - dice = evaluation.get("dice") - if status == "ok": - if iou is None or dice is None: - raise PromotionError( - "rollout-health successful evaluation lacks IoU or Dice" - ) - measured_iou = _unit(iou, "candidate.evaluation.iou") - _unit(dice, "candidate.evaluation.dice") - if not math.isclose( - reward, - measured_iou, - rel_tol=0.0, - abs_tol=1e-12, - ): - raise PromotionError( - "rollout-health reward differs from raw IoU" - ) - executable += 1 - else: - if reward != 0.0: - raise PromotionError( - "rollout-health model failure has nonzero reward" - ) - if iou is not None or dice is not None: - raise PromotionError( - "rollout-health model failure contains geometry metrics" - ) - rewards.append(reward) - group = (step, task_id) - if attempt in rewards_by_group[group]: - raise PromotionError("rollout-health repeats a group attempt") - rewards_by_group[group][attempt] = reward - record_bindings.append( - { - "relative_path": str(record.relative_path), - "payload_sha256": record.payload_sha256, - "record_sha256": record.record_sha256, - } - ) - sample_record_bindings.append( - { - "relative_path": str(sample.relative_path), - "payload_sha256": sample.payload_sha256, - "record_sha256": sample.record_sha256, - } - ) - - if len(rewards_by_group) != groups_per_batch or any( - set(attempts) != set(range(1, group_size + 1)) - for attempts in rewards_by_group.values() - ): - raise PromotionError("rollout-health candidates are not exactly eight G4 groups") - nonconstant = sum( - max(attempts.values()) - min(attempts.values()) > 1e-12 - for attempts in rewards_by_group.values() - ) - if executable < 1: - raise PromotionError("rollout-health smoke has no pure executable candidate") - if nonconstant < 1: - raise PromotionError("rollout-health smoke has no nonconstant G4 group") - - payload = { - "schema_version": ROLLOUT_HEALTH_APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "rollout-health-receipt", - "stage_id": stage.stage_id, - "wave": wave_name, - "source_git_sha": source_git_sha, - "evidence_source_git_sha": manifest.payload.get("source_git_sha"), - "protocol_logical_sha256": protocol["logical_sha256"], - "prior_wave": prior_wave, - "prior_receipt_record_sha256": prior_receipt.record_sha256, - "invocation_id": invocation_id, - "candidate_inventory_logical_sha256": recorded_inventory[ - "logical_sha256" - ], - "sample_inventory_logical_sha256": recorded_samples[ - "logical_sha256" - ], - "candidate_record_bindings_sha256": canonical_sha256(record_bindings), - "sample_record_bindings_sha256": canonical_sha256( - sample_record_bindings - ), - "metrics": { - "candidates": expected_candidates, - "groups": groups_per_batch, - "group_size": group_size, - "mean_raw_absolute_scale_iou_including_zeros": ( - sum(rewards) / len(rewards) - ), - "pure_executable": executable, - "pure_executable_rate": executable / expected_candidates, - "nonconstant_reward_groups": nonconstant, - "nonconstant_reward_group_fraction": ( - nonconstant / groups_per_batch - ), - "evaluator_or_reference_faults": 0, - }, - } - return store.write_rollout_health_approval( - key, - wave=wave_name, - payload=payload, - ) - - -def record_base_baseline_approval( - *, - repo_root: Path, - store: TrainingStore, - key: StageKey, - stage: StageSpec, - wave_name: str, - source_git_sha: str, - protocol: Mapping[str, Any], -) -> TrainingRecord: - """Validate and bind the pre-update base-Qwen L0 held-out report.""" - - if ( - stage.stage_id != "qwen-base-rl-l0" - or stage.parent != "base:qwen" - or stage.current_level != "L0" - or wave_name != "smoke" - or str(stage.wave(wave_name)["approval"]) - != "base-level-held-out-baseline-receipt" - ): - raise PromotionError("base baseline gate is not qwen-base-rl-l0/smoke") - - from .evaluation import ( - CheckpointEvaluationError, - CheckpointEvaluationStore, - _base_model_binding, - _sampler_provenance, - validate_checkpoint_report as validate_native_report, - ) - - panel = LEVEL_PROGRESS_PANELS["L0"] - binding = _base_model_binding( - protocol=dict(protocol), - stage_id=stage.stage_id, - replicate_id=key.replicate_id, - source_git_sha=source_git_sha, - ) - report_store = CheckpointEvaluationStore( - stage_path=store.stage_path(key), - wave="base", - checkpoint_name="base", - panel=panel, - ) - manifest = report_store.load_manifest() - report = report_store.load_report() - if manifest is None or report is None: - raise PromotionError( - "qwen-base-rl-l0/smoke requires the completed base L0 panel" - ) - expected_tasks = _expected_panel_tasks( - repo_root=repo_root, - protocol=protocol, - panel=panel, - ) - try: - validate_native_report( - report, - manifest_record_sha256=str(manifest["record_sha256"]), - binding=binding, - panel=panel, - expected_task_ids=[task.task_id for task in expected_tasks], - ) - except CheckpointEvaluationError as exc: - raise PromotionError(f"base L0 report failed its native validator: {exc}") from exc - - report_payload = _mapping(report.get("payload"), "base_report.payload") - provenance = _mapping(report_payload.get("provenance"), "base_report.provenance") - manifest_payload = _mapping(manifest.get("payload"), "base_manifest.payload") - expected_identity = { - "study_id": STUDY_ID, - "stage_id": stage.stage_id, - "wave": "base", - "checkpoint_name": "base", - "replicate_id": key.replicate_id, - "panel": panel, - } - for document_name, document in ( - ("report", report_payload), - ("manifest", manifest_payload), - ): - for field, expected in expected_identity.items(): - if document.get(field) != expected: - raise PromotionError( - f"base {document_name} {field} differs from its request" - ) - expected_protocol = { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - } - if ( - provenance.get("source_git_sha") != source_git_sha - or manifest_payload.get("source_git_sha") != source_git_sha - or dict(_mapping(provenance.get("protocol"), "base_report.protocol")) - != expected_protocol - or any( - manifest_payload.get(field) != expected - for field, expected in { - "protocol_logical_sha256": expected_protocol["logical_sha256"], - "protocol_file_sha256": expected_protocol["file_sha256"], - "contract_version": expected_protocol["contract_version"], - }.items() - ) - ): - raise PromotionError("base L0 report source or protocol differs") - panel_document = _panel_manifest(expected_tasks, panel=panel) - observed_panel = _mapping( - provenance.get("task_panel"), - "base_report.task_panel", - ) - if ( - dict(observed_panel) != panel_document - or manifest_payload.get("task_panel") != panel_document - ): - raise PromotionError("base L0 report task panel differs") - - local_dataset = _dataset_binding(repo_root / "dataset", protocol) - if manifest_payload.get("dataset") != local_dataset: - raise PromotionError("base L0 manifest dataset differs") - expected_report_dataset = { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": "depth/validation/L0/slot-6", - "logical_release_sha256": local_dataset["logical_release_sha256"], - "freeze_file_sha256": local_dataset["freeze_file_sha256"], - "parquet_shards": local_dataset["parquet_shards"], - } - if provenance.get("dataset") != expected_report_dataset: - raise PromotionError("base L0 report dataset projection differs") - - model = dict(protocol["models"]["qwen"]) - expected_sampler = { - "model": model["model"], - "renderer": model["renderer"], - "effort": model.get("effort"), - **_sampler_provenance(binding), - } - if ( - manifest_payload.get("model") != model - or manifest_payload.get("sampler") != _sampler_provenance(binding) - or provenance.get("sampler") != expected_sampler - ): - raise PromotionError("base L0 report sampler differs") - expected_report_sampling = { - "attempts_per_task": 1, - "max_output_tokens": 60_000, - "temperature": 1.0, - "top_p": 1.0, - "max_image_long_edge": 1_920, - "seed_algorithm": "sha256-bound deterministic 31-bit seed", - } - expected_manifest_sampling = { - "attempts_per_task": 1, - "max_output_tokens": 60_000, - "temperature": 1.0, - "top_p": 1.0, - "max_image_long_edge": 1_920, - "sample_concurrency": 8, - "evaluation_batch_size": 32, - "evaluator_workers": 8, - } - if ( - provenance.get("sampling") != expected_report_sampling - or manifest_payload.get("sampling") != expected_manifest_sampling - ): - raise PromotionError("base L0 report sampling policy differs") - sandbox = _mapping(provenance.get("sandbox"), "base_report.sandbox") - if ( - manifest_payload.get("prompt_assets") != prompt_asset_hashes() - or provenance.get("prompt_assets") != prompt_asset_hashes() - or dict(sandbox) != manifest_payload.get("sandbox") - or provenance.get("evaluation_manifest_record_sha256") - != manifest.get("record_sha256") - ): - raise PromotionError("base L0 report runtime provenance differs") - if ( - not _IMMUTABLE_IMAGE.fullmatch(str(sandbox.get("image_ref", ""))) - or not re.fullmatch( - r"sha256:[0-9a-f]{64}", - str(sandbox.get("image_id", "")), - ) - ): - raise PromotionError("base L0 report sandbox is not immutable") - _validate_report_metrics(report_payload, expected_tasks=expected_tasks) - - payload = { - "schema_version": BASE_BASELINE_APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "base-level-held-out-baseline-receipt", - "stage_id": stage.stage_id, - "wave": wave_name, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "evaluation": { - "wave": "base", - "checkpoint_name": "base", - "panel": panel, - "base_model_binding_sha256": binding.binding_sha256, - "manifest_relative_path": str( - report_store.root.relative_to(store.external_root) - / "manifest.json" - ), - "manifest_payload_sha256": manifest["payload_sha256"], - "manifest_record_sha256": manifest["record_sha256"], - "report_relative_path": str( - report_store.root.relative_to(store.external_root) - / "report.json" - ), - "report_payload_sha256": report["payload_sha256"], - "report_record_sha256": report["record_sha256"], - }, - } - return store.write_base_baseline_approval( - key, - wave=wave_name, - payload=payload, - ) - - -def _required_prior_wave(stage: StageSpec, wave_name: str) -> str | None: - target = int(stage.wave(wave_name)["max_steps"]) - earlier = [ - (int(wave["max_steps"]), name) - for name, wave in stage.waves.items() - if int(wave["max_steps"]) < target - ] - return max(earlier)[1] if earlier else None - - -def _mixed_allowed_checkpoints(current_level: str) -> tuple[tuple[str, str], ...]: - index = LEVELS.index(current_level) - return ( - ("qwen-mixed-sft", "complete"), - *tuple((f"qwen-mixed-rl-l{prior}", "complete") for prior in range(index)), - ) - - -def approval_plan(stage: StageSpec, wave_name: str) -> ApprovalPlan: - """Resolve the exact report and approval-record location for one wave.""" - - gate = str(stage.wave(wave_name)["approval"]) - if gate == "initial": - raise PromotionError(f"{stage.stage_id}/{wave_name} needs no approval") - - if gate == "clean-smoke-receipt": - prior = _required_prior_wave(stage, wave_name) - if prior is None: - raise PromotionError("clean-smoke gate has no preceding wave") - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=PANEL_PROGRESS, - allowed_evaluation_checkpoints=((stage.stage_id, prior),), - ) - - if gate == "rollout-health-receipt": - raise PromotionError( - "rollout-health-receipt is derived automatically from the prior " - "wave receipt; it does not accept a human checkpoint approval" - ) - - if gate == "current-level-held-out-promotion-receipt": - prior = _required_prior_wave(stage, wave_name) - if prior is None or stage.current_level not in LEVEL_PROGRESS_PANELS: - raise PromotionError("current-level gate has no prior wave or level") - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=LEVEL_PROGRESS_PANELS[stage.current_level], - allowed_evaluation_checkpoints=((stage.stage_id, prior),), - ) - - if gate == "prior-level-held-out-promotion-receipt": - parent_stage, parent_wave = stage.parent.split(":", 1) - if ( - parent_stage == "base" - or parent_wave != "complete" - or stage.current_level not in LEVELS[1:] - ): - raise PromotionError("prior-level gate has no exact curriculum parent") - prior_level = LEVELS[LEVELS.index(stage.current_level) - 1] - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=LEVEL_PROGRESS_PANELS[prior_level], - allowed_evaluation_checkpoints=((parent_stage, parent_wave),), - ) - - if gate == "prior-and-current-level-held-out-transition-receipt": - parent_stage, parent_wave = stage.parent.split(":", 1) - if ( - stage.stage_id not in { - "qwen-base-rl-l1", - "qwen-base-rl-l2", - "qwen-base-rl-l3", - "qwen-base-rl-l4", - } - or parent_wave != "complete" - or stage.current_level not in LEVELS[1:] - ): - raise PromotionError( - "dual level-transition gate has no exact pure-RL parent" - ) - prior_level = LEVELS[LEVELS.index(stage.current_level) - 1] - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=LEVEL_PROGRESS_PANELS[prior_level], - allowed_evaluation_checkpoints=((parent_stage, parent_wave),), - entry_baseline_panel=LEVEL_PROGRESS_PANELS[stage.current_level], - ) - - if gate == "promotion-receipt-and-explicit-human-approval": - prior = _required_prior_wave(stage, wave_name) - if ( - stage.stage_id != "inkling-l4-rl" - or wave_name != "coverage-extension" - or prior != "pilot" - ): - raise PromotionError("unknown advanced-policy promotion gate") - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=PANEL_INKLING_PROMOTION, - allowed_evaluation_checkpoints=((stage.stage_id, prior),), - ) - - if gate in { - "entry-evaluation-and-promotion-receipt", - "parent-evaluation-receipt", - }: - parent_stage, parent_wave = stage.parent.split(":", 1) - if parent_stage == "base" or parent_wave == "complete-or-skip": - raise PromotionError(f"{gate} requires one exact trained parent") - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=PANEL_DEPTH_VALIDATION, - allowed_evaluation_checkpoints=((parent_stage, parent_wave),), - ) - - if gate == "first-unmastered-entry-receipt": - if stage.current_level not in LEVELS or not stage.stage_id.startswith("qwen-mixed-rl-"): - raise PromotionError("first-unmastered gate is not a mixed-Qwen stage") - return ApprovalPlan( - gate=gate, - storage_wave=wave_name, - expected_panel=PANEL_DEPTH_VALIDATION, - allowed_evaluation_checkpoints=_mixed_allowed_checkpoints(stage.current_level), - ) - - raise PromotionError(f"unsupported approval gate: {gate!r}") - - -def _expected_panel_tasks( - *, - repo_root: Path, - protocol: Mapping[str, Any], - panel: str, -) -> list[PanelTaskIdentity]: - # Import lazily: loading/recording approvals must remain importable when - # the optional Tinker training stack is not installed. - from .evaluation import build_panel_tasks - - tasks = build_panel_tasks( - repo_root=repo_root, - protocol=protocol, - panel=panel, - ) - return [ - PanelTaskIdentity( - task_id=task.task_id, - level=task.level, - representation_id=task.representation_id, - image_sha256=task.observation.image_sha256, - target_image_sha256=task.reference.target_image_sha256, - footprint_um=tuple(float(value) for value in task.observation.footprint_um), - ) - for task in tasks - ] - - -def _panel_manifest( - tasks: Sequence[PanelTaskIdentity], - *, - panel: str, -) -> dict[str, Any]: - entries = [ - { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "image_sha256": task.image_sha256, - "target_image_sha256": task.target_image_sha256, - "footprint_um": list(task.footprint_um), - } - for task in tasks - ] - return { - "schema_version": "pixcell-training-evaluation-task-panel-v1", - "panel": panel, - "selection_seed": ( - _INKLING_PANEL_SELECTION_SEED - if panel == PANEL_INKLING_PROMOTION - else ( - _LEVEL_PROGRESS_SELECTION - if panel in LEVEL_PROGRESS_PANELS.values() - else ( - _DEPTH_FINAL_SELECTION - if panel == PANEL_DEPTH_FINAL_SELECTION - else None - ) - ) - ), - "task_count": len(tasks), - "task_ids": [task.task_id for task in tasks], - "logical_sha256": canonical_json_sha256(entries), - } - - -def _aggregate(records: Sequence[Mapping[str, Any]]) -> dict[str, Any]: - if not records: - raise PromotionError("evaluation aggregate is empty") - ious = [ - _unit(record.get("raw_absolute_scale_iou"), "record.raw_absolute_scale_iou") - for record in records - ] - pure: list[bool] = [] - statuses: Counter[str] = Counter() - for index, record in enumerate(records): - observed = record.get("pure_executable") - if not isinstance(observed, bool): - raise PromotionError("record.pure_executable must be boolean") - status = str(record.get("status", "")) - if status not in _MODEL_EVALUATION_STATUSES: - raise PromotionError("record contains a non-model evaluation status") - if observed != (status == "ok"): - raise PromotionError("record status and pure-executable flag differ") - if status != "ok" and float(ious[index]) != 0.0: - raise PromotionError("failed model output has nonzero IoU") - pure.append(observed) - statuses[status] += 1 - return { - "tasks": len(records), - "mean_raw_absolute_scale_iou": sum(ious) / len(ious), - "pure_executable": sum(pure), - "pure_executable_rate": sum(pure) / len(pure), - "statuses": dict(sorted(statuses.items())), - } - - -def _validate_aggregate( - observed: Any, - expected: Mapping[str, Any], - *, - path: str, -) -> None: - value = _mapping(observed, path) - for field in ("tasks", "pure_executable"): - actual = _integer(value.get(field), f"{path}.{field}") - if actual != expected[field]: - raise PromotionError(f"{path}.{field}={actual}; expected {expected[field]}") - _close( - value.get("mean_raw_absolute_scale_iou"), - float(expected["mean_raw_absolute_scale_iou"]), - f"{path}.mean_raw_absolute_scale_iou", - ) - _close( - value.get("pure_executable_rate"), - float(expected["pure_executable_rate"]), - f"{path}.pure_executable_rate", - ) - statuses = _mapping(value.get("statuses"), f"{path}.statuses") - if dict(statuses) != expected["statuses"]: - raise PromotionError(f"{path}.statuses differs from its records") - - -def _validate_report_metrics( - report: Mapping[str, Any], - *, - expected_tasks: Sequence[PanelTaskIdentity], -) -> None: - raw_records = report.get("records") - if not isinstance(raw_records, list) or len(raw_records) != len(expected_tasks): - raise PromotionError(f"report.records must contain all {len(expected_tasks)} tasks") - records: list[Mapping[str, Any]] = [] - for index, (raw, task) in enumerate(zip(raw_records, expected_tasks, strict=True)): - path = f"records[{index}]" - record = _mapping(raw, path) - expected_identity = { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - } - for field, expected in expected_identity.items(): - if record.get(field) != expected: - raise PromotionError(f"{path}.{field}={record.get(field)!r}; expected {expected!r}") - _unit( - record.get("raw_absolute_scale_iou"), - f"{path}.raw_absolute_scale_iou", - ) - tokens = _integer(record.get("completion_tokens"), f"{path}.completion_tokens") - if tokens > 60_000: - raise PromotionError(f"{path}.completion_tokens exceeds the sealed cap") - for field in ("cap_hit", "channel_parse_complete"): - if not isinstance(record.get(field), bool): - raise PromotionError(f"{path}.{field} must be boolean") - for field in ( - "answer_text_sha256", - "sample_record_sha256", - "evaluation_record_sha256", - ): - _sha(record.get(field), f"{path}.{field}") - render = record.get("render_sha256") - if render is not None: - _sha(render, f"{path}.render_sha256") - # This also enforces the status/purity relationship. - _aggregate([record]) - records.append(record) - - expected_summary = _aggregate(records) - _validate_aggregate(report.get("summary"), expected_summary, path="summary") - - by_level: dict[str, list[Mapping[str, Any]]] = defaultdict(list) - by_representation: dict[str, list[Mapping[str, Any]]] = defaultdict(list) - for record in records: - by_level[str(record["level"])].append(record) - by_representation[str(record["representation_id"])].append(record) - observed_levels = _mapping(report.get("levels"), "levels") - expected_levels = {key: _aggregate(values) for key, values in sorted(by_level.items())} - if set(observed_levels) != set(expected_levels): - raise PromotionError("report.levels differs from the panel") - for key, expected in expected_levels.items(): - _validate_aggregate(observed_levels[key], expected, path=f"levels.{key}") - - observed_representations = _mapping( - report.get("representations"), - "representations", - ) - expected_representations = { - key: _aggregate(values) for key, values in sorted(by_representation.items()) - } - if set(observed_representations) != set(expected_representations): - raise PromotionError("report.representations differs from the panel") - for key, expected in expected_representations.items(): - _validate_aggregate( - observed_representations[key], - expected, - path=f"representations.{key}", - ) - macro = sum( - value["mean_raw_absolute_scale_iou"] for value in expected_representations.values() - ) / len(expected_representations) - _close( - report.get("representation_macro_mean_iou"), - macro, - "representation_macro_mean_iou", - ) - - -def _validate_report_payload( - *, - repo_root: Path, - protocol: Mapping[str, Any], - expected_source_sha: str, - key: StageKey, - wave: str, - checkpoint_name: str, - selected_checkpoint: Mapping[str, Any], - checkpoint_inventory_sha256: str, - panel: str, - report: Mapping[str, Any], - evaluation_manifest: Mapping[str, Any], - training_manifest: TrainingRecord, - training_receipt: TrainingRecord, - expected_tasks: Sequence[PanelTaskIdentity], -) -> None: - if report.get("schema_version") != _REPORT_SCHEMA: - raise PromotionError("checkpoint report has a foreign schema") - expected_identity = { - "study_id": STUDY_ID, - "stage_id": key.stage_id, - "wave": wave, - "checkpoint_name": checkpoint_name, - "replicate_id": key.replicate_id, - "panel": panel, - } - for field, expected in expected_identity.items(): - if report.get(field) != expected: - raise PromotionError(f"report {field} differs from its path") - - expected_rows = EXPECTED_PANEL_ROWS[panel] - if len(expected_tasks) != expected_rows: - raise PromotionError( - f"{panel} local task set has {len(expected_tasks)} rows, expected {expected_rows}" - ) - panel_document = _panel_manifest(expected_tasks, panel=panel) - provenance = _mapping(report.get("provenance"), "provenance") - observed_panel = _mapping(provenance.get("task_panel"), "provenance.task_panel") - for field, expected in panel_document.items(): - if observed_panel.get(field) != expected: - raise PromotionError(f"provenance.task_panel.{field} changed") - - if provenance.get("source_git_sha") != expected_source_sha: - raise PromotionError("report source commit differs") - protocol_binding = _mapping(provenance.get("protocol"), "provenance.protocol") - expected_protocol = { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - } - for field, expected in expected_protocol.items(): - if protocol_binding.get(field) != expected: - raise PromotionError(f"report protocol {field} differs") - - manifest_payload = _mapping(evaluation_manifest.get("payload"), "eval_manifest") - for field, expected in { - **expected_identity, - "source_git_sha": expected_source_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": expected_protocol["file_sha256"], - "contract_version": protocol["contract_version"], - }.items(): - if manifest_payload.get(field) != expected: - raise PromotionError(f"evaluation manifest {field} differs") - if manifest_payload.get("task_panel") != dict(observed_panel): - raise PromotionError("report and evaluation-manifest task panels differ") - - expected_assets = prompt_asset_hashes() - if ( - provenance.get("prompt_assets") != expected_assets - or manifest_payload.get("prompt_assets") != expected_assets - ): - raise PromotionError("evaluation prompt assets differ from this source") - - dataset = _mapping(provenance.get("dataset"), "provenance.dataset") - manifest_dataset = _mapping( - manifest_payload.get("dataset"), - "evaluation_manifest.dataset", - ) - try: - expected_local_dataset = _dataset_binding(repo_root / "dataset", protocol) - except Exception as exc: - raise PromotionError("local evaluation dataset could not be verified") from exc - if dict(manifest_dataset) != expected_local_dataset: - raise PromotionError("evaluation-manifest dataset differs from the frozen release") - - expected_report_dataset = { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": { - PANEL_PROGRESS: "fixed-f1-f8", - PANEL_DEPTH_VALIDATION: "depth/validation", - PANEL_INKLING_PROMOTION: "depth/validation+fixed-f1-f8", - PANEL_DEPTH_FINAL_SELECTION: "depth/validation/slot-7", - **{ - panel_id: f"depth/validation/{level}/slot-6" - for level, panel_id in LEVEL_PROGRESS_PANELS.items() - }, - }[panel], - "logical_release_sha256": expected_local_dataset[ - "logical_release_sha256" - ], - "freeze_file_sha256": expected_local_dataset["freeze_file_sha256"], - "parquet_shards": expected_local_dataset["parquet_shards"], - } - if dict(dataset) != expected_report_dataset: - raise PromotionError( - "report dataset projection differs from the evaluation manifest" - ) - - if ( - training_manifest.payload.get("source_git_sha") != expected_source_sha - or training_manifest.payload.get("protocol_logical_sha256") != protocol["logical_sha256"] - ): - raise PromotionError("training manifest provenance differs") - manifest_training_dataset = _mapping( - training_manifest.payload.get("dataset"), - "training_manifest.dataset", - ) - if dict(manifest_training_dataset) != expected_local_dataset: - raise PromotionError("training manifest dataset differs from the frozen release") - - sampler = _mapping(provenance.get("sampler"), "provenance.sampler") - evaluation_sampler = _mapping( - manifest_payload.get("sampler"), - "evaluation_manifest.sampler", - ) - sampler_path = str(selected_checkpoint.get("sampler_path", "")) - if not _TINKER_PATH.fullmatch(sampler_path): - raise PromotionError("selected inventory entry has no sampler checkpoint") - expected_sampler = { - "sampler_path": sampler_path, - "checkpoint_name": checkpoint_name, - "checkpoint": dict(selected_checkpoint), - "checkpoint_inventory_sha256": checkpoint_inventory_sha256, - "training_run_manifest_record_sha256": training_manifest.record_sha256, - "training_wave_receipt_record_sha256": training_receipt.record_sha256, - } - for field, expected in expected_sampler.items(): - if sampler.get(field) != expected or evaluation_sampler.get(field) != expected: - raise PromotionError(f"evaluation sampler {field} differs") - if sampler.get("training_wave_receipt_relative_path") != str(training_receipt.relative_path): - raise PromotionError("evaluation sampler receipt path differs") - - evaluated_stage = stage_spec(protocol, key.stage_id) - model = protocol["models"][evaluated_stage.model_key] - for field in ("model", "renderer", "effort"): - if sampler.get(field) != model.get(field): - raise PromotionError(f"evaluation sampler {field} differs") - if manifest_payload.get("model") != model: - raise PromotionError("evaluation manifest model differs") - - sampling = _mapping(provenance.get("sampling"), "provenance.sampling") - expected_sampling = { - "attempts_per_task": 1, - "max_output_tokens": 60_000, - "temperature": 1.0, - "top_p": 1.0, - "max_image_long_edge": 1_920, - } - for field, expected in expected_sampling.items(): - if sampling.get(field) != expected: - raise PromotionError(f"evaluation sampling {field} differs") - manifest_sampling = _mapping( - manifest_payload.get("sampling"), - "evaluation_manifest.sampling", - ) - for field, expected in expected_sampling.items(): - if manifest_sampling.get(field) != expected: - raise PromotionError(f"evaluation manifest sampling {field} differs") - - sandbox = _mapping(provenance.get("sandbox"), "provenance.sandbox") - if dict(sandbox) != manifest_payload.get("sandbox"): - raise PromotionError("report and evaluation-manifest sandboxes differ") - if not _IMMUTABLE_IMAGE.fullmatch(str(sandbox.get("image_ref", ""))): - raise PromotionError("evaluation sandbox image_ref is not immutable") - if not re.fullmatch(r"sha256:[0-9a-f]{64}", str(sandbox.get("image_id", ""))): - raise PromotionError("evaluation sandbox image_id is not immutable") - if provenance.get("evaluation_manifest_record_sha256") != evaluation_manifest.get( - "record_sha256" - ): - raise PromotionError("report is not bound to its evaluation manifest") - - if ( - training_receipt.payload.get("stage_id") != key.stage_id - or training_receipt.payload.get("wave") != wave - or training_receipt.payload.get("run_manifest_record_sha256") - != training_manifest.record_sha256 - ): - raise PromotionError("training receipt identity differs") - _validate_report_metrics(report, expected_tasks=expected_tasks) - - -def validate_checkpoint_report( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - stage_id: str, - wave: str, - checkpoint_name: str, - replicate_id: str, - panel: str, - expected_source_sha: str, - expected_report_record_sha256: str, -) -> EvaluationEvidence: - """Load and fully validate one sealed report from its canonical path.""" - - if panel not in EXPECTED_PANEL_ROWS: - raise PromotionError(f"unsupported evaluation panel: {panel!r}") - _sha(expected_report_record_sha256, "expected_report_record_sha256") - stage = stage_spec(dict(protocol), stage_id) - stage.wave(wave) - key = StageKey(STUDY_ID, stage.stage_id, replicate_id) - training_manifest = store.load_manifest(key) - training_receipt = store.load_wave_receipt(key, wave=wave) - if training_manifest is None or training_receipt is None: - raise PromotionError(f"{stage_id}/{wave}/{replicate_id} has no training receipt") - - from .evaluation import ( - CheckpointEvaluationError, - CheckpointEvaluationStore, - _load_checkpoint_binding, - validate_checkpoint_report as validate_evaluation_report, - ) - - evaluation_store = CheckpointEvaluationStore( - stage_path=store.stage_path(key), - wave=wave, - checkpoint_name=checkpoint_name, - panel=panel, - ) - evaluation_manifest = evaluation_store.load_manifest() - report = evaluation_store.load_report() - if evaluation_manifest is None or report is None: - raise PromotionError(f"{stage_id}/{wave}/{panel} has no complete evaluation report") - if report.get("record_sha256") != expected_report_record_sha256: - raise PromotionError("evaluation report digest differs from the reviewed one") - expected_tasks = _expected_panel_tasks( - repo_root=repo_root, - protocol=protocol, - panel=panel, - ) - try: - binding = _load_checkpoint_binding( - store=store, - protocol=dict(protocol), - stage_id=stage_id, - wave=wave, - checkpoint_name=checkpoint_name, - replicate_id=replicate_id, - expected_receipt_sha256=training_receipt.record_sha256, - source_git_sha=expected_source_sha, - ) - validate_evaluation_report( - report, - manifest_record_sha256=str(evaluation_manifest["record_sha256"]), - binding=binding, - panel=panel, - expected_task_ids=[task.task_id for task in expected_tasks], - ) - except CheckpointEvaluationError as exc: - raise PromotionError(f"checkpoint report failed its native validator: {exc}") from exc - payload = _mapping(report.get("payload"), "report.payload") - _validate_report_payload( - repo_root=repo_root, - protocol=protocol, - expected_source_sha=expected_source_sha, - key=key, - wave=wave, - checkpoint_name=checkpoint_name, - selected_checkpoint=binding.checkpoint, - checkpoint_inventory_sha256=binding.checkpoint_inventory_sha256, - panel=panel, - report=payload, - evaluation_manifest=evaluation_manifest, - training_manifest=training_manifest, - training_receipt=training_receipt, - expected_tasks=expected_tasks, - ) - return EvaluationEvidence( - key=key, - wave=wave, - checkpoint_name=checkpoint_name, - checkpoint=dict(binding.checkpoint), - checkpoint_inventory_sha256=binding.checkpoint_inventory_sha256, - panel=panel, - report=dict(report), - report_payload=dict(payload), - report_record_sha256=str(report["record_sha256"]), - report_payload_sha256=str(report["payload_sha256"]), - report_relative_path=str( - ( - TrainingStore.stage_relative_path(key) - / "evaluations" - / wave - / checkpoint_name - / panel - / "report.json" - ) - ), - evaluation_manifest_payload_sha256=_sha( - evaluation_manifest.get("payload_sha256"), - "evaluation_manifest.payload_sha256", - ), - evaluation_manifest_record_sha256=_sha( - evaluation_manifest.get("record_sha256"), - "evaluation_manifest.record_sha256", - ), - evaluation_manifest_relative_path=str( - ( - TrainingStore.stage_relative_path(key) - / "evaluations" - / wave - / checkpoint_name - / panel - / "manifest.json" - ) - ), - training_manifest=training_manifest, - training_receipt=training_receipt, - ) - - -def first_unmastered_level( - report: Mapping[str, Any], - *, - minimum_iou: float = MASTERED_MINIMUM_IOU, - minimum_pure_executable_rate: float = (MASTERED_MINIMUM_PURE_EXECUTABLE_RATE), -) -> str | None: - """Return the first L0→L4 level missing either established threshold.""" - - minimum_iou = _unit(minimum_iou, "minimum_iou") - minimum_pure_executable_rate = _unit( - minimum_pure_executable_rate, - "minimum_pure_executable_rate", - ) - summaries = _mapping(report.get("levels"), "report.levels") - for level in LEVELS: - summary = _mapping(summaries.get(level), f"report.levels.{level}") - iou = _unit( - summary.get("mean_raw_absolute_scale_iou"), - f"report.levels.{level}.mean_raw_absolute_scale_iou", - ) - pure = _unit( - summary.get("pure_executable_rate"), - f"report.levels.{level}.pure_executable_rate", - ) - if iou < minimum_iou or pure < minimum_pure_executable_rate: - return level - return None - - -def _terminal_checkpoint(evidence: EvaluationEvidence) -> None: - terminal = _mapping( - evidence.training_receipt.payload.get("checkpoint"), - "training_receipt.checkpoint", - ) - fields = ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - selected_alias = {field: evidence.checkpoint.get(field) for field in fields} - if selected_alias != dict(terminal): - raise PromotionError( - f"{evidence.key.stage_id}/{evidence.wave} must use its terminal checkpoint" - ) - - -def _best_mixed_checkpoint( - candidates: Sequence[Mapping[str, Any]], -) -> Mapping[str, Any]: - if not candidates: - raise PromotionError("mixed-SFT checkpoint ranking is empty") - - def rank(candidate: Mapping[str, Any]) -> tuple[float, float, float]: - return ( - _unit( - candidate.get("representation_macro_mean_iou"), - "checkpoint.representation_macro_mean_iou", - ), - _unit( - candidate.get("pure_executable_rate"), - "checkpoint.pure_executable_rate", - ), - _unit( - candidate.get("training_progress_fraction"), - "checkpoint.training_progress_fraction", - ), - ) - - return max(candidates, key=rank) - - -def _mixed_checkpoint_selection( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - selected: EvaluationEvidence, - expected_source_sha: str, -) -> dict[str, Any]: - if selected.key.stage_id != "qwen-mixed-sft": - raise PromotionError("mixed checkpoint ranking received another stage") - inventory = _mapping( - selected.training_receipt.payload.get("checkpoint_inventory"), - "training_receipt.checkpoint_inventory", - ) - entries = inventory.get("entries") - if not isinstance(entries, list) or not entries: - raise PromotionError("mixed-SFT receipt has no checkpoint inventory") - inventory_sha = _sha( - inventory.get("logical_sha256"), - "training_receipt.checkpoint_inventory.logical_sha256", - ) - if ( - inventory_sha != canonical_sha256(entries) - or inventory_sha != selected.checkpoint_inventory_sha256 - ): - raise PromotionError("mixed-SFT checkpoint inventory digest differs") - - reports: list[EvaluationEvidence] = [] - for raw_entry in entries: - entry = _mapping(raw_entry, "checkpoint_inventory.entries[]") - checkpoint_name = str(entry.get("name", "")) - if checkpoint_name == selected.checkpoint_name: - evidence = selected - else: - from .evaluation import CheckpointEvaluationStore - - report_store = CheckpointEvaluationStore( - stage_path=store.stage_path(selected.key), - wave=selected.wave, - checkpoint_name=checkpoint_name, - panel=PANEL_DEPTH_VALIDATION, - ) - report = report_store.load_report() - if report is None: - raise PromotionError( - "mixed-SFT promotion requires a complete depth-validation " - f"report for checkpoint {checkpoint_name!r}" - ) - evidence = validate_checkpoint_report( - repo_root=repo_root, - protocol=protocol, - store=store, - stage_id=selected.key.stage_id, - wave=selected.wave, - checkpoint_name=checkpoint_name, - replicate_id=selected.key.replicate_id, - panel=PANEL_DEPTH_VALIDATION, - expected_source_sha=expected_source_sha, - expected_report_record_sha256=str(report["record_sha256"]), - ) - if evidence.checkpoint != dict(entry): - raise PromotionError(f"checkpoint {checkpoint_name!r} report and inventory differ") - reports.append(evidence) - - candidates = [ - { - "checkpoint_name": evidence.checkpoint_name, - "checkpoint": evidence.checkpoint, - "report_record_sha256": evidence.report_record_sha256, - "representation_macro_mean_iou": _unit( - evidence.report_payload.get("representation_macro_mean_iou"), - "report.representation_macro_mean_iou", - ), - "pure_executable_rate": _unit( - _mapping( - evidence.report_payload.get("summary"), - "report.summary", - ).get("pure_executable_rate"), - "report.summary.pure_executable_rate", - ), - "training_progress_fraction": _unit( - evidence.checkpoint.get("training_progress_fraction"), - "checkpoint.training_progress_fraction", - ), - } - for evidence in reports - ] - winner = _best_mixed_checkpoint(candidates) - if winner["checkpoint_name"] != selected.checkpoint_name: - raise PromotionError( - f"mixed-SFT checkpoint {selected.checkpoint_name!r} is not the " - f"deterministic best; expected {winner['checkpoint_name']!r}" - ) - selection = { - "schema_version": "pixcell-mixed-sft-checkpoint-selection-v1", - "selection_rule": ( - "maximize full-depth-validation representation-macro raw IoU; " - "break ties by pure-executable rate, then later training progress" - ), - "all_inventory_checkpoints_required": True, - "inventory_logical_sha256": inventory_sha, - "candidates": candidates, - "selected_checkpoint_name": selected.checkpoint_name, - } - selection["logical_sha256"] = canonical_sha256(selection) - return selection - - -def _source_transition_document( - *, - repo_root: Path, - protocol: Mapping[str, Any], - stage: StageSpec, - wave_name: str, - plan: ApprovalPlan, - evidence: EvaluationEvidence, - producer_source_git_sha: str, - consumer_source_git_sha: str, - approved_change_paths: Sequence[str] | None, -) -> dict[str, Any]: - """Bind one ancestor-produced report to one exact consumer wave.""" - - provenance = _mapping( - evidence.report_payload.get("provenance"), - "report.provenance", - ) - report_protocol = _mapping( - provenance.get("protocol"), - "report.provenance.protocol", - ) - report_dataset = _mapping( - provenance.get("dataset"), - "report.provenance.dataset", - ) - report_sampler = _mapping( - provenance.get("sampler"), - "report.provenance.sampler", - ) - report_sampling = _mapping( - provenance.get("sampling"), - "report.provenance.sampling", - ) - report_prompt_assets = _mapping( - provenance.get("prompt_assets"), - "report.provenance.prompt_assets", - ) - report_sandbox = _mapping( - provenance.get("sandbox"), - "report.provenance.sandbox", - ) - training_dataset = _mapping( - evidence.training_manifest.payload.get("dataset"), - "training_manifest.dataset", - ) - training_model = _mapping( - evidence.training_manifest.payload.get("model"), - "training_manifest.model", - ) - checkpoint_inventory = _mapping( - evidence.training_receipt.payload.get("checkpoint_inventory"), - "training_receipt.checkpoint_inventory", - ) - evaluation_manifest_record_sha256 = _sha( - provenance.get("evaluation_manifest_record_sha256"), - "report.provenance.evaluation_manifest_record_sha256", - ) - if evaluation_manifest_record_sha256 != evidence.evaluation_manifest_record_sha256: - raise PromotionError("report and evaluation manifest record digests differ") - evaluated_stage = stage_spec(dict(protocol), evidence.key.stage_id) - consumer_protocol = { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - } - consumer_prompt_assets = prompt_asset_hashes() - parent_model = dict(protocol["models"][evaluated_stage.model_key]) - if ( - provenance.get("source_git_sha") != producer_source_git_sha - or evidence.training_manifest.payload.get("source_git_sha") != producer_source_git_sha - ): - raise PromotionError("source transition evidence is not producer-source-bound") - if dict(report_protocol) != consumer_protocol: - raise PromotionError("producer evaluation protocol differs from the consumer") - training_protocol = { - "logical_sha256": evidence.training_manifest.payload.get("protocol_logical_sha256"), - "file_sha256": evidence.training_manifest.payload.get("protocol_file_sha256"), - "contract_version": evidence.training_manifest.payload.get("contract_version"), - } - if training_protocol != consumer_protocol: - raise PromotionError("producer training protocol differs from the consumer") - if ( - dict(report_prompt_assets) != consumer_prompt_assets - or evidence.training_manifest.payload.get("prompt_assets") != consumer_prompt_assets - ): - raise PromotionError("producer prompt assets differ from the consumer") - if dict(training_model) != parent_model: - raise PromotionError("producer training model differs from the consumer parent model") - renderer_contract = { - field: parent_model.get(field) for field in ("model", "renderer", "effort") - } - if any(report_sampler.get(field) != expected for field, expected in renderer_contract.items()): - raise PromotionError("producer evaluation renderer differs from the consumer") - expected_dataset = { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "logical_release_sha256": protocol["dataset"]["logical_release_sha256"], - } - if any( - report_dataset.get(field) != expected for field, expected in expected_dataset.items() - ) or ( - training_dataset.get("logical_release_sha256") != expected_dataset["logical_release_sha256"] - ): - raise PromotionError("producer dataset differs from the consumer") - scope = { - "study_id": STUDY_ID, - "child": { - "stage_id": stage.stage_id, - "wave": wave_name, - "replicate_id": evidence.key.replicate_id, - "approval_gate": plan.gate, - }, - "producer_evaluation": { - "stage_id": evidence.key.stage_id, - "wave": evidence.wave, - "checkpoint_name": evidence.checkpoint_name, - "replicate_id": evidence.key.replicate_id, - "panel": evidence.panel, - }, - } - equivalence = { - "protocol": { - "producer_training": { - **training_protocol, - }, - "producer_evaluation": dict(report_protocol), - "consumer": consumer_protocol, - }, - "prompt_assets": { - "producer_training": dict( - _mapping( - evidence.training_manifest.payload.get("prompt_assets"), - "training_manifest.prompt_assets", - ) - ), - "producer_evaluation": dict(report_prompt_assets), - "consumer": consumer_prompt_assets, - }, - "dataset": { - "producer_training": dict(training_dataset), - "producer_evaluation": dict(report_dataset), - "consumer": { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "logical_release_sha256": protocol["dataset"]["logical_release_sha256"], - }, - }, - "task_panel": dict( - _mapping( - provenance.get("task_panel"), - "report.provenance.task_panel", - ) - ), - "model": { - "producer_training": dict(training_model), - "producer_evaluation": parent_model, - "consumer_parent": parent_model, - "consumer_child": dict(protocol["models"][stage.model_key]), - }, - "renderer": { - "producer_evaluation": { - field: report_sampler.get(field) for field in ("model", "renderer", "effort") - }, - "consumer_parent": { - field: protocol["models"][evaluated_stage.model_key].get(field) - for field in ("model", "renderer", "effort") - }, - "consumer_child": { - field: protocol["models"][stage.model_key].get(field) - for field in ("model", "renderer", "effort") - }, - }, - "sampling": dict(report_sampling), - "sandbox": { - "producer_training": evidence.training_manifest.payload.get("sandbox"), - "producer_evaluation": dict(report_sandbox), - }, - } - evidence_binding = { - "training_run_manifest": { - "relative_path": str(evidence.training_manifest.relative_path), - "payload_sha256": evidence.training_manifest.payload_sha256, - "record_sha256": evidence.training_manifest.record_sha256, - }, - "training_wave_receipt": { - "relative_path": str(evidence.training_receipt.relative_path), - "payload_sha256": evidence.training_receipt.payload_sha256, - "record_sha256": evidence.training_receipt.record_sha256, - }, - "checkpoint_inventory": dict(checkpoint_inventory), - "evaluation_manifest": { - "relative_path": evidence.evaluation_manifest_relative_path, - "payload_sha256": (evidence.evaluation_manifest_payload_sha256), - "record_sha256": evaluation_manifest_record_sha256, - }, - "evaluation_report": { - "relative_path": evidence.report_relative_path, - "payload_sha256": evidence.report_payload_sha256, - "record_sha256": evidence.report_record_sha256, - }, - "checkpoint": dict(evidence.checkpoint), - "checkpoint_inventory_logical_sha256": (evidence.checkpoint_inventory_sha256), - } - try: - return build_source_transition( - repo_root=repo_root, - producer_source_git_sha=producer_source_git_sha, - consumer_source_git_sha=consumer_source_git_sha, - scope=scope, - approved_change_paths=approved_change_paths, - equivalence=equivalence, - evidence=evidence_binding, - ) - except SourceTransitionError as exc: - raise PromotionError(f"source transition is invalid: {exc}") from exc - - -def _approved_change_paths_from_transition( - value: Any, -) -> tuple[str, ...]: - transition = _mapping(value, "approval.source_transition") - scope = _mapping( - transition.get("scope"), - "approval.source_transition.scope", - ) - approved = _mapping( - scope.get("approved_changes"), - "approval.source_transition.scope.approved_changes", - ) - paths = approved.get("paths") - if not isinstance(paths, list) or any(not isinstance(path, str) for path in paths): - raise PromotionError("approval source transition has no approved path inventory") - return tuple(paths) - - -def _require_same_evaluated_checkpoint( - primary: EvaluationEvidence, - baseline: EvaluationEvidence, -) -> None: - """Require two panel reports to share one exact parent artifact chain.""" - - fields = ( - ("stage key", primary.key, baseline.key), - ("wave", primary.wave, baseline.wave), - ("checkpoint name", primary.checkpoint_name, baseline.checkpoint_name), - ("checkpoint", primary.checkpoint, baseline.checkpoint), - ( - "checkpoint inventory", - primary.checkpoint_inventory_sha256, - baseline.checkpoint_inventory_sha256, - ), - ( - "training manifest", - primary.training_manifest.record_sha256, - baseline.training_manifest.record_sha256, - ), - ( - "training receipt", - primary.training_receipt.record_sha256, - baseline.training_receipt.record_sha256, - ), - ) - for field, primary_value, baseline_value in fields: - if baseline_value != primary_value: - raise PromotionError( - "prior-level promotion and current-level entry baseline " - f"must evaluate the exact same parent checkpoint ({field} differs)" - ) - - -def _approval_payload( - *, - stage: StageSpec, - wave_name: str, - plan: ApprovalPlan, - evidence: EvaluationEvidence, - source_git_sha: str, - protocol: Mapping[str, Any], - checkpoint_selection: Mapping[str, Any] | None = None, - entry_baseline_evidence: EvaluationEvidence | None = None, - evidence_source_git_sha: str | None = None, - source_transition: Mapping[str, Any] | None = None, -) -> dict[str, Any]: - if ( - evidence.key.stage_id, - evidence.wave, - ) not in plan.allowed_evaluation_checkpoints: - raise PromotionError( - f"{evidence.key.stage_id}/{evidence.wave} cannot approve {stage.stage_id}/{wave_name}" - ) - if evidence.panel != plan.expected_panel: - raise PromotionError( - f"{stage.stage_id}/{wave_name} requires {plan.expected_panel}, not {evidence.panel}" - ) - if plan.entry_baseline_panel is None: - if entry_baseline_evidence is not None: - raise PromotionError("this approval gate does not accept an entry baseline") - else: - if entry_baseline_evidence is None: - raise PromotionError( - f"{stage.stage_id}/{wave_name} requires the " - f"{plan.entry_baseline_panel} pre-update baseline" - ) - if entry_baseline_evidence.panel != plan.entry_baseline_panel: - raise PromotionError( - f"{stage.stage_id}/{wave_name} entry baseline requires " - f"{plan.entry_baseline_panel}, not {entry_baseline_evidence.panel}" - ) - _require_same_evaluated_checkpoint( - evidence, - entry_baseline_evidence, - ) - - mastery: dict[str, Any] | None = None - if plan.gate == "first-unmastered-entry-receipt": - thresholds = _mapping( - protocol["evaluation"].get("mastery_thresholds"), - "protocol.evaluation.mastery_thresholds", - ) - minimum_iou = _unit( - thresholds.get("minimum_mean_raw_absolute_scale_iou"), - "mastery_thresholds.minimum_mean_raw_absolute_scale_iou", - ) - minimum_pure = _unit( - thresholds.get("minimum_pure_executable_rate"), - "mastery_thresholds.minimum_pure_executable_rate", - ) - selected = first_unmastered_level( - evidence.report_payload, - minimum_iou=minimum_iou, - minimum_pure_executable_rate=minimum_pure, - ) - if selected is None: - raise PromotionError( - "all L0–L4 levels are mastered; mixed curriculum RL must not start" - ) - if selected != stage.current_level: - raise PromotionError(f"first unmastered level is {selected}, not {stage.current_level}") - mastery = { - "selection": "first-unmastered-l0-to-l4", - "selected_level": selected, - "minimum_mean_raw_absolute_scale_iou": minimum_iou, - "minimum_pure_executable_rate": minimum_pure, - } - elif plan.gate == "entry-evaluation-and-promotion-receipt": - if stage.current_level not in LEVELS: - raise PromotionError("curriculum entry stage has no current level") - mastery = { - "selection": "fixed-curriculum-entry-after-reviewed-parent-evaluation", - "entry_level": stage.current_level, - "hard_mastery_threshold_applied": False, - } - - cross_source = source_transition is not None - if cross_source != (evidence_source_git_sha is not None): - raise PromotionError("source transition and evidence source must be supplied together") - if evidence_source_git_sha is not None and evidence_source_git_sha == source_git_sha: - raise PromotionError("same-source approvals must retain the v1 schema") - - payload: dict[str, Any] = { - "schema_version": ( - SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION if cross_source else APPROVAL_SCHEMA_VERSION - ), - "decision": "approve", - "approval_gate": plan.gate, - "stage_id": stage.stage_id, - "wave": wave_name, - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "explicit_human_confirmation": True, - "evaluation": { - "stage_id": evidence.key.stage_id, - "wave": evidence.wave, - "checkpoint_name": evidence.checkpoint_name, - "checkpoint": evidence.checkpoint, - "checkpoint_inventory_sha256": (evidence.checkpoint_inventory_sha256), - "replicate_id": evidence.key.replicate_id, - "panel": evidence.panel, - "report_relative_path": evidence.report_relative_path, - "report_payload_sha256": evidence.report_payload_sha256, - "report_record_sha256": evidence.report_record_sha256, - "training_wave_receipt_record_sha256": (evidence.training_receipt.record_sha256), - }, - "mastery": mastery, - "checkpoint_selection": ( - dict(checkpoint_selection) if checkpoint_selection is not None else None - ), - } - if entry_baseline_evidence is not None: - payload["entry_baseline"] = { - "stage_id": entry_baseline_evidence.key.stage_id, - "wave": entry_baseline_evidence.wave, - "checkpoint_name": entry_baseline_evidence.checkpoint_name, - "checkpoint": entry_baseline_evidence.checkpoint, - "checkpoint_inventory_sha256": ( - entry_baseline_evidence.checkpoint_inventory_sha256 - ), - "replicate_id": entry_baseline_evidence.key.replicate_id, - "panel": entry_baseline_evidence.panel, - "report_relative_path": entry_baseline_evidence.report_relative_path, - "report_payload_sha256": ( - entry_baseline_evidence.report_payload_sha256 - ), - "report_record_sha256": ( - entry_baseline_evidence.report_record_sha256 - ), - "training_wave_receipt_record_sha256": ( - entry_baseline_evidence.training_receipt.record_sha256 - ), - } - if cross_source: - payload["evidence_source_git_sha"] = evidence_source_git_sha - payload["source_transition"] = dict(source_transition) - payload["evaluation"].update( - { - "training_run_manifest_record_sha256": (evidence.training_manifest.record_sha256), - "evaluation_manifest_relative_path": (evidence.evaluation_manifest_relative_path), - "evaluation_manifest_payload_sha256": (evidence.evaluation_manifest_payload_sha256), - "evaluation_manifest_record_sha256": (evidence.evaluation_manifest_record_sha256), - } - ) - # The launcher uses these top-level fields to resolve dynamic parents and - # to bind exact-parent gates. A same-stage smoke continuation must not - # claim its smoke receipt as the stage's external parent. - if plan.gate in { - "first-unmastered-entry-receipt", - "entry-evaluation-and-promotion-receipt", - "parent-evaluation-receipt", - "prior-level-held-out-promotion-receipt", - "prior-and-current-level-held-out-transition-receipt", - }: - payload.update( - { - "parent_stage_id": evidence.key.stage_id, - "parent_wave": evidence.wave, - "parent_receipt_record_sha256": (evidence.training_receipt.record_sha256), - "selected_parent_checkpoint": evidence.checkpoint, - "selected_parent_checkpoint_inventory_sha256": ( - evidence.checkpoint_inventory_sha256 - ), - } - ) - elif plan.gate in { - "clean-smoke-receipt", - "current-level-held-out-promotion-receipt", - }: - payload.update( - { - "prior_wave": evidence.wave, - "prior_receipt_record_sha256": (evidence.training_receipt.record_sha256), - } - ) - return payload - - -def validate_source_transition_approval( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - stage: StageSpec, - wave_name: str, - replicate_id: str, - consumer_source_git_sha: str, - approval: TrainingRecord, -) -> EvaluationEvidence: - """Revalidate every fact in one cross-source approval at launch time.""" - - payload = approval.payload - if payload.get("schema_version") != SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION: - raise PromotionError("cross-source approval has a foreign schema") - if payload.get("source_git_sha") != consumer_source_git_sha: - raise PromotionError("cross-source approval names another consumer") - try: - validate_source_sha(repo_root, consumer_source_git_sha) - except (OSError, ValueError, subprocess.SubprocessError) as exc: - raise PromotionError( - f"cross-source approval consumer is not the clean HEAD: {exc}" - ) from exc - producer_source_git_sha = str(payload.get("evidence_source_git_sha", "")) - if producer_source_git_sha == consumer_source_git_sha: - raise PromotionError("same-source approval cannot use the v2 schema") - approved_change_paths = _approved_change_paths_from_transition(payload.get("source_transition")) - try: - git_facts = source_transition_git_facts( - repo_root=repo_root, - producer_source_git_sha=producer_source_git_sha, - consumer_source_git_sha=consumer_source_git_sha, - ) - approved_change_scope(git_facts, approved_change_paths) - except SourceTransitionError as exc: - raise PromotionError(f"source transition is invalid: {exc}") from exc - - plan = approval_plan(stage, wave_name) - evaluation = _mapping(payload.get("evaluation"), "approval.evaluation") - if evaluation.get("replicate_id") != replicate_id: - raise PromotionError("cross-source approval names another replicate") - evidence = validate_checkpoint_report( - repo_root=repo_root, - protocol=protocol, - store=store, - stage_id=str(evaluation.get("stage_id", "")), - wave=str(evaluation.get("wave", "")), - checkpoint_name=str(evaluation.get("checkpoint_name", "")), - replicate_id=replicate_id, - panel=str(evaluation.get("panel", "")), - expected_source_sha=producer_source_git_sha, - expected_report_record_sha256=str(evaluation.get("report_record_sha256", "")), - ) - checkpoint_selection: dict[str, Any] | None = None - if plan.gate == "first-unmastered-entry-receipt" and evidence.key.stage_id == "qwen-mixed-sft": - checkpoint_selection = _mixed_checkpoint_selection( - repo_root=repo_root, - protocol=protocol, - store=store, - selected=evidence, - expected_source_sha=producer_source_git_sha, - ) - else: - _terminal_checkpoint(evidence) - transition = _source_transition_document( - repo_root=repo_root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - plan=plan, - evidence=evidence, - producer_source_git_sha=producer_source_git_sha, - consumer_source_git_sha=consumer_source_git_sha, - approved_change_paths=approved_change_paths, - ) - expected = _approval_payload( - stage=stage, - wave_name=wave_name, - plan=plan, - evidence=evidence, - source_git_sha=consumer_source_git_sha, - protocol=protocol, - checkpoint_selection=checkpoint_selection, - evidence_source_git_sha=producer_source_git_sha, - source_transition=transition, - ) - if payload != expected: - raise PromotionError("cross-source approval differs from its recomputed evidence") - return evidence - - -def validate_same_source_wave_approval( - *, - repo_root: Path, - protocol: Mapping[str, Any], - store: TrainingStore, - stage: StageSpec, - wave_name: str, - replicate_id: str, - source_git_sha: str, - approval: TrainingRecord, -) -> EvaluationEvidence: - """Recompute a same-source approval from its sealed reports at launch. - - The approval JSON is an index, not authority. Every report, manifest, - checkpoint inventory, receipt, aggregate, and task-panel fact is loaded - from its canonical external-ledger path and validated again before the - approval may authorize a paid launch. - """ - - payload = approval.payload - if payload.get("schema_version") != APPROVAL_SCHEMA_VERSION: - raise PromotionError("same-source approval has a foreign schema") - if payload.get("source_git_sha") != source_git_sha: - raise PromotionError("same-source approval names another source") - try: - validate_source_sha(repo_root, source_git_sha) - except (OSError, ValueError, subprocess.SubprocessError) as exc: - raise PromotionError( - f"same-source approval source is not the clean HEAD: {exc}" - ) from exc - - plan = approval_plan(stage, wave_name) - evaluation = _mapping(payload.get("evaluation"), "approval.evaluation") - if evaluation.get("replicate_id") != replicate_id: - raise PromotionError("same-source approval names another replicate") - if evaluation.get("panel") != plan.expected_panel: - raise PromotionError("same-source approval names the wrong promotion panel") - evidence = validate_checkpoint_report( - repo_root=repo_root, - protocol=protocol, - store=store, - stage_id=str(evaluation.get("stage_id", "")), - wave=str(evaluation.get("wave", "")), - checkpoint_name=str(evaluation.get("checkpoint_name", "")), - replicate_id=replicate_id, - panel=plan.expected_panel, - expected_source_sha=source_git_sha, - expected_report_record_sha256=str( - evaluation.get("report_record_sha256", "") - ), - ) - - entry_baseline_evidence: EvaluationEvidence | None = None - if plan.entry_baseline_panel is not None: - entry = _mapping( - payload.get("entry_baseline"), - "approval.entry_baseline", - ) - if entry.get("replicate_id") != replicate_id: - raise PromotionError("entry baseline names another replicate") - if entry.get("panel") != plan.entry_baseline_panel: - raise PromotionError("entry baseline names the wrong current-level panel") - entry_baseline_evidence = validate_checkpoint_report( - repo_root=repo_root, - protocol=protocol, - store=store, - stage_id=str(entry.get("stage_id", "")), - wave=str(entry.get("wave", "")), - checkpoint_name=str(entry.get("checkpoint_name", "")), - replicate_id=replicate_id, - panel=plan.entry_baseline_panel, - expected_source_sha=source_git_sha, - expected_report_record_sha256=str( - entry.get("report_record_sha256", "") - ), - ) - elif "entry_baseline" in payload: - raise PromotionError("approval unexpectedly contains an entry baseline") - - checkpoint_selection: dict[str, Any] | None = None - if ( - plan.gate == "first-unmastered-entry-receipt" - and evidence.key.stage_id == "qwen-mixed-sft" - ): - checkpoint_selection = _mixed_checkpoint_selection( - repo_root=repo_root, - protocol=protocol, - store=store, - selected=evidence, - expected_source_sha=source_git_sha, - ) - else: - _terminal_checkpoint(evidence) - - expected = _approval_payload( - stage=stage, - wave_name=wave_name, - plan=plan, - evidence=evidence, - source_git_sha=source_git_sha, - protocol=protocol, - checkpoint_selection=checkpoint_selection, - entry_baseline_evidence=entry_baseline_evidence, - ) - if payload != expected: - raise PromotionError( - "same-source approval differs from its canonically recomputed evidence" - ) - return evidence - - -def record_wave_approval( - *, - repo_root: Path, - external_root: Path, - stage_id: str, - wave_name: str, - replicate_id: str, - evaluation_stage_id: str, - evaluation_wave: str, - evaluation_checkpoint_name: str, - evaluation_report_record_sha256: str, - entry_baseline_report_record_sha256: str | None = None, - expected_source_sha: str, - evidence_source_sha: str | None = None, - approved_change_paths: Sequence[str] | None = None, - confirmation: str, -) -> dict[str, Any]: - """Validate reviewed evidence and atomically record one approval.""" - - if confirmation != APPROVAL_CONFIRMATION_TOKEN: - raise PromotionError("explicit human approval confirmation is missing") - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol(root, require_committed=True) - source_git_sha = validate_source_sha(root, expected_source_sha) - evidence_git_sha = source_git_sha if evidence_source_sha is None else evidence_source_sha - if evidence_git_sha != source_git_sha: - try: - git_facts = source_transition_git_facts( - repo_root=root, - producer_source_git_sha=evidence_git_sha, - consumer_source_git_sha=source_git_sha, - ) - approved_change_scope(git_facts, approved_change_paths) - except SourceTransitionError as exc: - raise PromotionError(f"source transition is invalid: {exc}") from exc - elif approved_change_paths is not None: - raise PromotionError("same-source v1 approval cannot include an approved change scope") - stage = stage_spec(protocol, stage_id) - try: - validate_stage_replicate( - protocol, - stage_id=stage.stage_id, - replicate_id=replicate_id, - ) - except ValueError as exc: - raise PromotionError(str(exc)) from exc - plan = approval_plan(stage, wave_name) - store = TrainingStore(repo_root=root, external_root=external_root) - evidence = validate_checkpoint_report( - repo_root=root, - protocol=protocol, - store=store, - stage_id=evaluation_stage_id, - wave=evaluation_wave, - checkpoint_name=evaluation_checkpoint_name, - replicate_id=replicate_id, - panel=plan.expected_panel, - expected_source_sha=evidence_git_sha, - expected_report_record_sha256=evaluation_report_record_sha256, - ) - entry_baseline_evidence: EvaluationEvidence | None = None - if plan.entry_baseline_panel is not None: - if entry_baseline_report_record_sha256 is None: - raise PromotionError( - f"{stage.stage_id}/{wave_name} requires a reviewed " - "current-level entry-baseline report" - ) - if evidence_git_sha != source_git_sha: - raise PromotionError( - "dual level-transition approval must be produced and consumed " - "at the same source commit" - ) - entry_baseline_evidence = validate_checkpoint_report( - repo_root=root, - protocol=protocol, - store=store, - stage_id=evaluation_stage_id, - wave=evaluation_wave, - checkpoint_name=evaluation_checkpoint_name, - replicate_id=replicate_id, - panel=plan.entry_baseline_panel, - expected_source_sha=evidence_git_sha, - expected_report_record_sha256=( - entry_baseline_report_record_sha256 - ), - ) - elif entry_baseline_report_record_sha256 is not None: - raise PromotionError( - "entry-baseline report was supplied for a single-panel gate" - ) - checkpoint_selection: dict[str, Any] | None = None - if plan.gate == "first-unmastered-entry-receipt" and evidence.key.stage_id == "qwen-mixed-sft": - checkpoint_selection = _mixed_checkpoint_selection( - repo_root=root, - protocol=protocol, - store=store, - selected=evidence, - expected_source_sha=evidence_git_sha, - ) - else: - _terminal_checkpoint(evidence) - transition = ( - _source_transition_document( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - plan=plan, - evidence=evidence, - producer_source_git_sha=evidence_git_sha, - consumer_source_git_sha=source_git_sha, - approved_change_paths=approved_change_paths, - ) - if evidence_git_sha != source_git_sha - else None - ) - payload = _approval_payload( - stage=stage, - wave_name=wave_name, - plan=plan, - evidence=evidence, - source_git_sha=source_git_sha, - protocol=protocol, - checkpoint_selection=checkpoint_selection, - entry_baseline_evidence=entry_baseline_evidence, - evidence_source_git_sha=(evidence_git_sha if transition is not None else None), - source_transition=transition, - ) - key = StageKey(STUDY_ID, stage.stage_id, replicate_id) - with store.acquire_stage_lock(key): - # Close the approval-time HEAD/dirty-tree gap immediately before the - # immutable external record is created. - validate_source_sha(root, source_git_sha) - if transition is not None: - rebuilt_transition = _source_transition_document( - repo_root=root, - protocol=protocol, - stage=stage, - wave_name=wave_name, - plan=plan, - evidence=evidence, - producer_source_git_sha=evidence_git_sha, - consumer_source_git_sha=source_git_sha, - approved_change_paths=approved_change_paths, - ) - if rebuilt_transition != transition: - raise PromotionError("source transition changed before approval recording") - approval = store.write_wave_approval( - key, - wave=plan.storage_wave, - payload=payload, - ) - result = { - "status": "approved", - "stage_id": stage.stage_id, - "wave": wave_name, - "approval_storage_wave": plan.storage_wave, - "approval": str(approval.relative_path), - "approval_record_sha256": approval.record_sha256, - "evaluation_report_record_sha256": evidence.report_record_sha256, - } - if transition is not None: - result.update( - { - "approval_schema_version": payload["schema_version"], - "evidence_source_git_sha": evidence_git_sha, - "consumer_source_git_sha": source_git_sha, - "source_transition_logical_sha256": transition["logical_sha256"], - } - ) - return result diff --git a/rl/studies/representation_training_v1/protocol.json b/rl/studies/representation_training_v1/protocol.json deleted file mode 100644 index 07402f1f..00000000 --- a/rl/studies/representation_training_v1/protocol.json +++ /dev/null @@ -1,969 +0,0 @@ -{ - "contract_version": "pixcell-direct-reconstruction-v2", - "dataset": { - "configuration": "depth", - "logical_release_sha256": "676c49134d4d044c7d84426cf8eeecf09302e74ca4aa548956f14b7631a4d80b", - "repo_id": "qpaig-mit/pixcell", - "revision": "v2.0.0", - "train_rows": 3468, - "validation_rows": 1092 - }, - "evaluation": { - "final_benchmark_attempts": 4, - "geometry_signal": "raw_absolute_scale_iou", - "mastery_thresholds": { - "minimum_mean_raw_absolute_scale_iou": 0.8, - "minimum_pure_executable_rate": 0.95 - }, - "parameter_holdout": { - "configuration": "depth", - "representations": 546, - "rows": 1092, - "split": "validation" - }, - "curriculum_progress_holdout": { - "baseline_checkpoints": { - "L0": "base:qwen", - "L1": "qwen-base-rl-l0:complete", - "L2": "qwen-base-rl-l1:complete", - "L3": "qwen-base-rl-l2:complete", - "L4": "qwen-base-rl-l3:complete" - }, - "configuration": "depth", - "evaluation_waves": [ - "step-5", - "step-10", - "step-15", - "step-20", - "step-25", - "complete" - ], - "panel_prefix": "level-progress", - "realization_slot": 6, - "representations_by_level": { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108 - }, - "rows_by_level": { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108 - }, - "split": "validation" - }, - "final_selection_holdout": { - "configuration": "depth", - "panel": "depth-final-selection", - "realization_slot": 7, - "representations": 546, - "rows": 546, - "split": "validation" - }, - "progress_benchmark_attempts": 1, - "task_manifest": "../representation_curriculum_v2/task_manifest.json", - "task_manifest_logical_sha256": "05ca50526df05a3a79aa785504e36c7538106e5abdb72b64e86b20cc6f94a890" - }, - "hypotheses": { - "RT-H01": { - "name": "mixed-depth-sft", - "question": "Does one balanced mixed pass teach the representation catalogue and parameter recovery?" - }, - "RT-H02": { - "name": "sequential-sft", - "question": "Does explicit L0 to L4 answer supervision outperform one mixed pass?" - }, - "RT-H03": { - "name": "composition-by-rl", - "question": "After primitive supervision at L0, can geometry-only RL discover L1 to L4 composition?" - }, - "RT-H04": { - "name": "full-sft-then-rl", - "question": "Does curriculum RL improve the strongest mixed-SFT open checkpoint?" - }, - "RT-H05": { - "name": "inkling-advanced-rl", - "question": "Can a strong base policy improve on depth-L4 without losing retained capability?" - }, - "RT-H06": { - "name": "composition-by-pure-curriculum-rl", - "question": "Can verifier-only curriculum RL teach primitives through complete components directly from base Qwen?" - } - }, - "launch": { - "confirmation_token": "PIXCELL_TRAINING_V1", - "parent_replicate_overrides": { - "qwen-l0-rl-l1": { - "r0-retry1": "r0" - } - }, - "paid_launch_enabled": true, - "replicate_stage_scopes": { - "r0": [ - "*" - ], - "r0-retry1": [ - "qwen-l0-rl-l1", - "qwen-l0-rl-l2", - "qwen-l0-rl-l3", - "qwen-l0-rl-l4" - ] - }, - "replicates": [ - "r0", - "r0-retry1" - ] - }, - "logical_sha256": "c9bc59bd81aa0e6235aa307149e6fde02a1baf733756f31fe200ed911ffbee86", - "models": { - "inkling": { - "context_tokens": 65536, - "effort": 0.9, - "lora_rank": 32, - "max_image_long_edge": 1920, - "max_output_tokens": 60000, - "model": "thinkingmachines/Inkling", - "renderer": "tml_v0", - "thinking": true - }, - "qwen": { - "context_tokens": 65536, - "lora_rank": 32, - "max_image_long_edge": 1920, - "max_output_tokens": 60000, - "model": "Qwen/Qwen3.6-35B-A3B", - "renderer": "qwen3_5", - "thinking": true - } - }, - "recipes": { - "inkling_l4_rl": { - "evaluation_panel": { - "benchmark_rows": 8, - "l4_rows": 36, - "retention_rows": 24 - }, - "group_size": 4, - "groups_per_batch": 8, - "kl_coefficient": 0.0, - "learning_rate": 1e-05, - "loss_fn": "importance_sampling", - "automatic_validation_rows": 0, - "remove_constant_reward_groups": true, - "checkpoint_ttl_seconds": null, - "rolling_save_every_steps": 1, - "rollout_error_policy": "fail_fast", - "rollout_json_export": false, - "checkpoint_policy": "unique_wave_ceiling", - "replay_fraction": 0.0, - "reward": "raw_absolute_scale_iou", - "temperature": 1.0, - "top_p": 1.0 - }, - "qwen_rl": { - "group_size": 4, - "groups_per_batch": 8, - "kl_coefficient": 0.0, - "learning_rate": 1e-05, - "loss_fn": "importance_sampling", - "automatic_validation_rows": 0, - "remove_constant_reward_groups": true, - "checkpoint_ttl_seconds": null, - "rolling_save_every_steps": 1, - "rollout_error_policy": "fail_fast", - "rollout_json_export": false, - "checkpoint_policy": "unique_wave_ceiling", - "replay_fraction": 0.2, - "reward": "raw_absolute_scale_iou", - "stage_steps": 30, - "temperature": 1.0, - "top_p": 1.0 - }, - "qwen_sft": { - "adam_beta1": 0.9, - "adam_beta2": 0.95, - "adam_epsilon": 1e-08, - "batch_size": 64, - "effective_passes": 1, - "automatic_nll_validation": false, - "checkpoint_fraction": 0.25, - "checkpoint_ttl_seconds": null, - "learning_rate": 0.0001, - "loss_mass": "equal_by_level_then_representation", - "lr_schedule": "linear", - "max_sequence_tokens": 6144, - "optimizer_reset_each_stage": true, - "rolling_save_every_steps": 1, - "target": "program_and_end_token_only" - } - }, - "schema_version": "pixcell-representation-training-protocol-v1", - "stages": { - "inkling-l4-rl": { - "current_level": "L4", - "hypotheses": [ - "RT-H05" - ], - "kind": "rl", - "model": "inkling", - "parent": "base:inkling", - "recipe": "inkling_l4_rl", - "replay_levels": [], - "waves": { - "coverage-extension": { - "approval": "promotion-receipt-and-explicit-human-approval", - "max_steps": 14 - }, - "pilot": { - "approval": "clean-smoke-receipt", - "max_steps": 6 - }, - "smoke": { - "approval": "initial", - "max_steps": 1 - } - } - }, - "qwen-base-rl-l0": { - "current_level": "L0", - "hypotheses": [ - "RT-H06" - ], - "kind": "rl", - "model": "qwen", - "parent": "base:qwen", - "recipe": "qwen_rl", - "replay_levels": [], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "initial", - "max_steps": 1 - } - } - }, - "qwen-base-rl-l1": { - "current_level": "L1", - "hypotheses": [ - "RT-H06" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-base-rl-l0:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-and-current-level-held-out-transition-receipt", - "max_steps": 1 - } - } - }, - "qwen-base-rl-l2": { - "current_level": "L2", - "hypotheses": [ - "RT-H06" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-base-rl-l1:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-and-current-level-held-out-transition-receipt", - "max_steps": 1 - } - } - }, - "qwen-base-rl-l3": { - "current_level": "L3", - "hypotheses": [ - "RT-H06" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-base-rl-l2:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1", - "L2" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-and-current-level-held-out-transition-receipt", - "max_steps": 1 - } - } - }, - "qwen-base-rl-l4": { - "current_level": "L4", - "hypotheses": [ - "RT-H06" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-base-rl-l3:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1", - "L2", - "L3" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-and-current-level-held-out-transition-receipt", - "max_steps": 1 - } - } - }, - "qwen-l0-rl-l1": { - "current_level": "L1", - "hypotheses": [ - "RT-H03" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-l0-sft:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "parent-receipt-and-source-transition", - "max_steps": 1 - } - } - }, - "qwen-l0-rl-l2": { - "current_level": "L2", - "hypotheses": [ - "RT-H03" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-l0-rl-l1:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-level-held-out-promotion-receipt", - "max_steps": 1 - } - } - }, - "qwen-l0-rl-l3": { - "current_level": "L3", - "hypotheses": [ - "RT-H03" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-l0-rl-l2:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1", - "L2" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-level-held-out-promotion-receipt", - "max_steps": 1 - } - } - }, - "qwen-l0-rl-l4": { - "current_level": "L4", - "hypotheses": [ - "RT-H03" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-l0-rl-l3:complete", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1", - "L2", - "L3" - ], - "waves": { - "complete": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "current-level-held-out-promotion-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "rollout-health-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "prior-level-held-out-promotion-receipt", - "max_steps": 1 - } - } - }, - "qwen-l0-sft": { - "hypotheses": [ - "RT-H02", - "RT-H03" - ], - "kind": "sft", - "levels": [ - "L0" - ], - "model": "qwen", - "parent": "base:qwen", - "recipe": "qwen_sft", - "waves": { - "complete": { - "approval": "initial", - "max_steps": 11 - } - } - }, - "qwen-mixed-rl-l0": { - "current_level": "L0", - "hypotheses": [ - "RT-H04" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-mixed-sft:complete", - "recipe": "qwen_rl", - "replay_levels": [], - "waves": { - "complete": { - "approval": "clean-smoke-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "clean-smoke-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "clean-smoke-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "clean-smoke-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "clean-smoke-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "clean-smoke-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "first-unmastered-entry-receipt", - "max_steps": 1 - } - } - }, - "qwen-mixed-rl-l1": { - "current_level": "L1", - "hypotheses": [ - "RT-H04" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-mixed-rl-l0:complete-or-skip", - "recipe": "qwen_rl", - "replay_levels": [ - "L0" - ], - "waves": { - "complete": { - "approval": "clean-smoke-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "clean-smoke-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "clean-smoke-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "clean-smoke-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "clean-smoke-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "clean-smoke-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "first-unmastered-entry-receipt", - "max_steps": 1 - } - } - }, - "qwen-mixed-rl-l2": { - "current_level": "L2", - "hypotheses": [ - "RT-H04" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-mixed-rl-l1:complete-or-skip", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1" - ], - "waves": { - "complete": { - "approval": "clean-smoke-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "clean-smoke-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "clean-smoke-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "clean-smoke-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "clean-smoke-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "clean-smoke-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "first-unmastered-entry-receipt", - "max_steps": 1 - } - } - }, - "qwen-mixed-rl-l3": { - "current_level": "L3", - "hypotheses": [ - "RT-H04" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-mixed-rl-l2:complete-or-skip", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1", - "L2" - ], - "waves": { - "complete": { - "approval": "clean-smoke-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "clean-smoke-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "clean-smoke-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "clean-smoke-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "clean-smoke-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "clean-smoke-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "first-unmastered-entry-receipt", - "max_steps": 1 - } - } - }, - "qwen-mixed-rl-l4": { - "current_level": "L4", - "hypotheses": [ - "RT-H04" - ], - "kind": "rl", - "model": "qwen", - "parent": "qwen-mixed-rl-l3:complete-or-skip", - "recipe": "qwen_rl", - "replay_levels": [ - "L0", - "L1", - "L2", - "L3" - ], - "waves": { - "complete": { - "approval": "clean-smoke-receipt", - "max_steps": 30 - }, - "step-10": { - "approval": "clean-smoke-receipt", - "max_steps": 10 - }, - "step-15": { - "approval": "clean-smoke-receipt", - "max_steps": 15 - }, - "step-20": { - "approval": "clean-smoke-receipt", - "max_steps": 20 - }, - "step-25": { - "approval": "clean-smoke-receipt", - "max_steps": 25 - }, - "step-5": { - "approval": "clean-smoke-receipt", - "max_steps": 5 - }, - "smoke": { - "approval": "first-unmastered-entry-receipt", - "max_steps": 1 - } - } - }, - "qwen-mixed-sft": { - "hypotheses": [ - "RT-H01", - "RT-H04" - ], - "kind": "sft", - "levels": [ - "L0", - "L1", - "L2", - "L3", - "L4" - ], - "model": "qwen", - "parent": "base:qwen", - "recipe": "qwen_sft", - "waves": { - "complete": { - "approval": "initial", - "max_steps": 55 - } - } - }, - "qwen-sequential-sft-l1": { - "hypotheses": [ - "RT-H02" - ], - "kind": "sft", - "levels": [ - "L1" - ], - "model": "qwen", - "parent": "qwen-l0-sft:complete", - "recipe": "qwen_sft", - "waves": { - "complete": { - "approval": "parent-evaluation-receipt", - "max_steps": 14 - } - } - }, - "qwen-sequential-sft-l2": { - "hypotheses": [ - "RT-H02" - ], - "kind": "sft", - "levels": [ - "L2" - ], - "model": "qwen", - "parent": "qwen-sequential-sft-l1:complete", - "recipe": "qwen_sft", - "waves": { - "complete": { - "approval": "parent-evaluation-receipt", - "max_steps": 10 - } - } - }, - "qwen-sequential-sft-l3": { - "hypotheses": [ - "RT-H02" - ], - "kind": "sft", - "levels": [ - "L3" - ], - "model": "qwen", - "parent": "qwen-sequential-sft-l2:complete", - "recipe": "qwen_sft", - "waves": { - "complete": { - "approval": "parent-evaluation-receipt", - "max_steps": 12 - } - } - }, - "qwen-sequential-sft-l4": { - "hypotheses": [ - "RT-H02" - ], - "kind": "sft", - "levels": [ - "L4" - ], - "model": "qwen", - "parent": "qwen-sequential-sft-l3:complete", - "recipe": "qwen_sft", - "waves": { - "complete": { - "approval": "parent-evaluation-receipt", - "max_steps": 11 - } - } - } - }, - "study_id": "representation-training-v1", - "tracking": { - "entity": "aadarwal-massachusetts-institute-of-technology", - "local_record_is_authoritative": true, - "project": "pixcell-representation-training-v1", - "provider": "wandb" - } -} diff --git a/rl/studies/representation_training_v1/protocol.py b/rl/studies/representation_training_v1/protocol.py deleted file mode 100644 index 6951e366..00000000 --- a/rl/studies/representation_training_v1/protocol.py +++ /dev/null @@ -1,552 +0,0 @@ -"""Load and validate the sealed representation-training study.""" - -from __future__ import annotations - -import hashlib -import json -import math -import re -import subprocess -from collections.abc import Mapping -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from rl.common.prompt import CONTRACT_VERSION -from rl.evaluation.tasks import canonical_json_sha256 - - -SCHEMA_VERSION = "pixcell-representation-training-protocol-v1" -STUDY_ID = "representation-training-v1" -STUDY_RELATIVE_PATH = Path("rl/studies/representation_training_v1") -PROTOCOL_FILENAME = "protocol.json" -LEVELS = ("L0", "L1", "L2", "L3", "L4") -INITIAL_STAGE_IDS = ( - "qwen-mixed-sft", - "qwen-l0-sft", - "qwen-base-rl-l0", - "inkling-l4-rl", -) -BASE_QWEN_RL_STAGE_IDS = tuple(f"qwen-base-rl-l{level}" for level in range(5)) -L0_SFT_QWEN_RL_STAGE_IDS = tuple( - f"qwen-l0-rl-l{level}" for level in range(1, 5) -) -_STAGE_ID = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$") -_GIT_SHA = re.compile(r"^[0-9a-f]{40}$") -_SHA256 = re.compile(r"^[0-9a-f]{64}$") - - -@dataclass(frozen=True) -class StageSpec: - stage_id: str - kind: str - model_key: str - recipe_key: str - hypotheses: tuple[str, ...] - parent: str - levels: tuple[str, ...] - current_level: str | None - replay_levels: tuple[str, ...] - waves: dict[str, dict[str, Any]] - - def wave(self, wave_name: str) -> dict[str, Any]: - try: - return dict(self.waves[wave_name]) - except KeyError as exc: - raise ValueError( - f"{self.stage_id} has no wave {wave_name!r}; " - f"expected one of {sorted(self.waves)}" - ) from exc - - -def repository_root() -> Path: - return Path(__file__).resolve().parents[3] - - -def protocol_path(repo_root: Path) -> Path: - return repo_root.expanduser().resolve() / STUDY_RELATIVE_PATH / PROTOCOL_FILENAME - - -def file_sha256(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(1024 * 1024), b""): - digest.update(chunk) - return digest.hexdigest() - - -def _logical_document(document: dict[str, Any]) -> dict[str, Any]: - return {key: value for key, value in document.items() if key != "logical_sha256"} - - -def _require_at_head(repo_root: Path, path: Path) -> None: - relative = path.resolve(strict=True).relative_to(repo_root.resolve()).as_posix() - try: - subprocess.check_output( - ["git", "ls-files", "--error-unmatch", "--", relative], - cwd=repo_root, - stderr=subprocess.DEVNULL, - ) - committed = subprocess.check_output( - ["git", "show", f"HEAD:{relative}"], - cwd=repo_root, - stderr=subprocess.DEVNULL, - ) - except (OSError, subprocess.CalledProcessError) as exc: - raise ValueError("training protocol is not committed at HEAD") from exc - if committed != path.read_bytes(): - raise ValueError("training protocol bytes differ from HEAD") - - -def _validate_model(key: str, value: dict[str, Any]) -> None: - required = { - "model", - "renderer", - "thinking", - "lora_rank", - "context_tokens", - "max_output_tokens", - "max_image_long_edge", - } - if not required.issubset(value): - raise ValueError(f"model {key} is missing fields") - if value["max_output_tokens"] >= value["context_tokens"]: - raise ValueError(f"model {key} output exhausts its context") - if key == "qwen": - if ( - value["model"] != "Qwen/Qwen3.6-35B-A3B" - or value["renderer"] != "qwen3_5" - or value.get("effort") is not None - ): - raise ValueError("qwen must use the thinking-on Qwen3.5 contract") - elif key == "inkling": - effort = value.get("effort") - if ( - value["model"] != "thinkingmachines/Inkling" - or value["renderer"] != "tml_v0" - or isinstance(effort, bool) - or not isinstance(effort, (int, float)) - or not math.isfinite(float(effort)) - or not 0 <= float(effort) < 1 - ): - raise ValueError("inkling must use native tml_v0 with explicit effort") - else: - raise ValueError(f"unknown model key: {key}") - - -def _stage_spec( - stage_id: str, - value: dict[str, Any], - *, - protocol: dict[str, Any], -) -> StageSpec: - if not _STAGE_ID.fullmatch(stage_id): - raise ValueError(f"invalid stage ID: {stage_id!r}") - kind = str(value.get("kind")) - if kind not in {"sft", "rl"}: - raise ValueError(f"{stage_id} has unsupported kind {kind!r}") - model_key = str(value.get("model")) - recipe_key = str(value.get("recipe")) - if model_key not in protocol["models"]: - raise ValueError(f"{stage_id} names an unknown model") - if recipe_key not in protocol["recipes"]: - raise ValueError(f"{stage_id} names an unknown recipe") - hypotheses = tuple(str(item) for item in value.get("hypotheses", ())) - if not hypotheses or any(item not in protocol["hypotheses"] for item in hypotheses): - raise ValueError(f"{stage_id} has invalid hypothesis consumers") - parent = str(value.get("parent", "")) - if not parent: - raise ValueError(f"{stage_id} has no checkpoint parent") - levels = tuple(str(item).upper() for item in value.get("levels", ())) - current = value.get("current_level") - current_level = str(current).upper() if current is not None else None - replay = tuple(str(item).upper() for item in value.get("replay_levels", ())) - if kind == "sft": - if not levels or any(item not in LEVELS for item in levels): - raise ValueError(f"{stage_id} has invalid SFT levels") - if current_level is not None or replay: - raise ValueError(f"{stage_id} mixes SFT and RL selectors") - else: - if levels or current_level not in LEVELS: - raise ValueError(f"{stage_id} has invalid RL level") - expected_replay = LEVELS[: LEVELS.index(current_level)] - if model_key == "qwen" and replay != expected_replay: - raise ValueError( - f"{stage_id} replay must be the ordered prefix before {current_level}" - ) - if model_key == "inkling" and replay: - raise ValueError("Inkling L4 pilot must not replay training rows") - waves = value.get("waves") - if not isinstance(waves, dict) or not waves: - raise ValueError(f"{stage_id} has no waves") - observed_steps: list[int] = [] - for wave_name, wave in waves.items(): - if not _STAGE_ID.fullmatch(str(wave_name)): - raise ValueError(f"{stage_id} has invalid wave name") - if not isinstance(wave, dict) or not str(wave.get("approval", "")): - raise ValueError(f"{stage_id}/{wave_name} has no approval gate") - steps = wave.get("max_steps") - if isinstance(steps, bool) or not isinstance(steps, int) or steps < 1: - raise ValueError(f"{stage_id}/{wave_name} has invalid max_steps") - observed_steps.append(steps) - if len(observed_steps) != len(set(observed_steps)): - raise ValueError(f"{stage_id} waves must have distinct step ceilings") - if kind == "sft" and tuple(waves) != ("complete",): - raise ValueError("SFT stages run continuously; they do not stop after a smoke batch") - return StageSpec( - stage_id=stage_id, - kind=kind, - model_key=model_key, - recipe_key=recipe_key, - hypotheses=hypotheses, - parent=parent, - levels=levels, - current_level=current_level, - replay_levels=replay, - waves={str(key): dict(item) for key, item in waves.items()}, - ) - - -def load_protocol( - repo_root: Path | None = None, - *, - require_committed: bool = False, -) -> dict[str, Any]: - root = (repo_root or repository_root()).expanduser().resolve() - path = protocol_path(root) - document = json.loads(path.read_text(encoding="utf-8")) - if document.get("schema_version") != SCHEMA_VERSION: - raise ValueError("unsupported training protocol schema") - if document.get("study_id") != STUDY_ID: - raise ValueError("training protocol has the wrong study ID") - if document.get("contract_version") != CONTRACT_VERSION: - raise ValueError("training protocol and prompt contract differ") - observed = canonical_json_sha256(_logical_document(document)) - if document.get("logical_sha256") != observed: - raise ValueError("training protocol logical SHA-256 mismatch") - if not _SHA256.fullmatch(str(document["dataset"]["logical_release_sha256"])): - raise ValueError("training protocol has an invalid dataset digest") - if document["evaluation"].get("mastery_thresholds") != { - "minimum_mean_raw_absolute_scale_iou": 0.8, - "minimum_pure_executable_rate": 0.95, - }: - raise ValueError("training protocol has the wrong mastery thresholds") - if document["evaluation"].get("curriculum_progress_holdout") != { - "baseline_checkpoints": { - "L0": "base:qwen", - "L1": "qwen-base-rl-l0:complete", - "L2": "qwen-base-rl-l1:complete", - "L3": "qwen-base-rl-l2:complete", - "L4": "qwen-base-rl-l3:complete", - }, - "configuration": "depth", - "evaluation_waves": [ - "step-5", - "step-10", - "step-15", - "step-20", - "step-25", - "complete", - ], - "panel_prefix": "level-progress", - "realization_slot": 6, - "representations_by_level": { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108, - }, - "rows_by_level": { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108, - }, - "split": "validation", - }: - raise ValueError("training protocol has the wrong curriculum progress holdout") - if document["evaluation"].get("final_selection_holdout") != { - "configuration": "depth", - "panel": "depth-final-selection", - "realization_slot": 7, - "representations": 546, - "rows": 546, - "split": "validation", - }: - raise ValueError("training protocol has the wrong final selection holdout") - for key, value in document["models"].items(): - _validate_model(str(key), value) - stages = { - stage_id: _stage_spec(stage_id, value, protocol=document) - for stage_id, value in document["stages"].items() - } - if set(INITIAL_STAGE_IDS) - set(stages): - raise ValueError("training protocol omits an initial branch") - if set(BASE_QWEN_RL_STAGE_IDS) - set(stages): - raise ValueError("training protocol omits the pure Qwen RL curriculum") - if set(document["recipes"]) != {"qwen_sft", "qwen_rl", "inkling_l4_rl"}: - raise ValueError("training protocol has an unexpected recipe set") - for recipe_name, recipe in document["recipes"].items(): - if recipe.get("checkpoint_ttl_seconds", "missing") is not None: - raise ValueError( - f"{recipe_name} periodic checkpoints must remain available " - "for receipt-bound evaluation" - ) - for stage in stages.values(): - if ":" not in stage.parent: - raise ValueError(f"{stage.stage_id} has a malformed parent") - parent_stage, parent_wave = stage.parent.split(":", 1) - if stage.parent.startswith("base:"): - if parent_wave != stage.model_key: - raise ValueError(f"{stage.stage_id} has the wrong base parent") - else: - if parent_stage not in stages: - raise ValueError( - f"{stage.stage_id} names unknown parent {stage.parent}" - ) - parent_spec = stages[parent_stage] - if parent_spec.model_key != stage.model_key: - raise ValueError(f"{stage.stage_id} crosses model families") - if ( - parent_wave != "complete-or-skip" - and parent_wave not in parent_spec.waves - ): - raise ValueError(f"{stage.stage_id} names an unknown parent wave") - if stage.kind == "sft" and stage.recipe_key != "qwen_sft": - raise ValueError(f"{stage.stage_id} has a non-SFT recipe") - if stage.kind == "rl" and stage.recipe_key not in { - "qwen_rl", - "inkling_l4_rl", - }: - raise ValueError(f"{stage.stage_id} has a non-RL recipe") - if stage.model_key == "inkling" and stage.kind != "rl": - raise ValueError("Inkling is sealed as an RL-only policy") - if stage.kind == "rl": - reward = document["recipes"][stage.recipe_key].get("reward") - if reward != document["evaluation"]["geometry_signal"]: - raise ValueError(f"{stage.stage_id} does not optimize the recorded signal") - if stage.wave("smoke")["max_steps"] != 1: - raise ValueError(f"{stage.stage_id} smoke must be exactly one step") - if stage.model_key == "qwen": - expected_waves = { - "smoke": 1, - "step-5": 5, - "step-10": 10, - "step-15": 15, - "step-20": 20, - "step-25": 25, - "complete": 30, - } - if { - name: int(wave["max_steps"]) - for name, wave in stage.waves.items() - } != expected_waves: - raise ValueError(f"{stage.stage_id} has wrong Qwen RL waves") - if stage.stage_id in BASE_QWEN_RL_STAGE_IDS: - expected_gates = { - "smoke": ( - "initial" - if stage.current_level == "L0" - else ( - "prior-and-current-level-held-out-" - "transition-receipt" - ) - ), - "step-5": "rollout-health-receipt", - "step-10": "current-level-held-out-promotion-receipt", - "step-15": "current-level-held-out-promotion-receipt", - "step-20": "current-level-held-out-promotion-receipt", - "step-25": "current-level-held-out-promotion-receipt", - "complete": "current-level-held-out-promotion-receipt", - } - observed_gates = { - name: str(stage.wave(name)["approval"]) - for name in expected_waves - } - if observed_gates != expected_gates: - raise ValueError( - f"{stage.stage_id} has wrong pure-RL gates" - ) - elif stage.stage_id in L0_SFT_QWEN_RL_STAGE_IDS: - expected_gates = { - "smoke": ( - "parent-receipt-and-source-transition" - if stage.current_level == "L1" - else "prior-level-held-out-promotion-receipt" - ), - "step-5": "rollout-health-receipt", - "step-10": "current-level-held-out-promotion-receipt", - "step-15": "current-level-held-out-promotion-receipt", - "step-20": "current-level-held-out-promotion-receipt", - "step-25": "current-level-held-out-promotion-receipt", - "complete": "current-level-held-out-promotion-receipt", - } - observed_gates = { - name: str(stage.wave(name)["approval"]) - for name in expected_waves - } - if observed_gates != expected_gates: - raise ValueError( - f"{stage.stage_id} has wrong L0-SFT-to-RL gates" - ) - elif any( - stage.wave(name)["approval"] != "clean-smoke-receipt" - for name in expected_waves - if name != "smoke" - ): - raise ValueError(f"{stage.stage_id} continuation gates changed") - else: - if set(stage.waves) != { - "smoke", - "pilot", - "coverage-extension", - }: - raise ValueError("Inkling pilot has the wrong wave set") - if ( - stage.wave("pilot")["max_steps"] != 6 - or stage.wave("coverage-extension")["max_steps"] != 14 - ): - raise ValueError("Inkling pilot ceilings changed") - if ( - document["recipes"][stage.recipe_key].get("checkpoint_policy") - != "unique_wave_ceiling" - ): - raise ValueError( - f"{stage.stage_id} does not bind unique wave checkpoints" - ) - for level, stage_id in enumerate(BASE_QWEN_RL_STAGE_IDS): - stage = stages[stage_id] - expected_parent = ( - "base:qwen" - if level == 0 - else f"qwen-base-rl-l{level - 1}:complete" - ) - if ( - stage.kind != "rl" - or stage.model_key != "qwen" - or stage.recipe_key != "qwen_rl" - or stage.hypotheses != ("RT-H06",) - or stage.current_level != f"L{level}" - or stage.replay_levels != LEVELS[:level] - or stage.parent != expected_parent - ): - raise ValueError(f"{stage_id} changed the pure Qwen RL curriculum") - for level, stage_id in enumerate(L0_SFT_QWEN_RL_STAGE_IDS, start=1): - stage = stages[stage_id] - expected_parent = ( - "qwen-l0-sft:complete" - if level == 1 - else f"qwen-l0-rl-l{level - 1}:complete" - ) - if ( - stage.kind != "rl" - or stage.model_key != "qwen" - or stage.recipe_key != "qwen_rl" - or stage.hypotheses != ("RT-H03",) - or stage.current_level != f"L{level}" - or stage.replay_levels != LEVELS[:level] - or stage.parent != expected_parent - ): - raise ValueError(f"{stage_id} changed the L0-SFT-to-RL curriculum") - exact_edges = { - stage.stage_id: stage.parent.split(":", 1)[0] - for stage in stages.values() - if not stage.parent.startswith("base:") - } - for origin in exact_edges: - seen: set[str] = set() - cursor = origin - while cursor in exact_edges: - if cursor in seen: - raise ValueError("training stage graph contains a cycle") - seen.add(cursor) - cursor = exact_edges[cursor] - if document["launch"].get("paid_launch_enabled") is not True: - raise ValueError("training protocol is not launch-enabled") - expected_retry_stages = list(L0_SFT_QWEN_RL_STAGE_IDS) - if document["launch"].get("replicates") != ["r0", "r0-retry1"]: - raise ValueError("training protocol has the wrong launch identities") - if document["launch"].get("replicate_stage_scopes") != { - "r0": ["*"], - "r0-retry1": expected_retry_stages, - }: - raise ValueError("training protocol has the wrong launch identity scopes") - if document["launch"].get("parent_replicate_overrides") != { - "qwen-l0-rl-l1": {"r0-retry1": "r0"} - }: - raise ValueError("training protocol has the wrong parent identity override") - if require_committed: - _require_at_head(root, path) - return document - - -def stage_spec(protocol: dict[str, Any], stage_id: str) -> StageSpec: - try: - value = protocol["stages"][stage_id] - except KeyError as exc: - raise ValueError(f"unknown training stage: {stage_id!r}") from exc - return _stage_spec(stage_id, value, protocol=protocol) - - -def validate_stage_replicate( - protocol: Mapping[str, Any], - *, - stage_id: str, - replicate_id: str, -) -> None: - """Validate one declared execution identity for one exact stage.""" - - if replicate_id not in protocol["launch"]["replicates"]: - raise ValueError("replicate is not declared in the protocol") - scopes = protocol["launch"]["replicate_stage_scopes"] - allowed = scopes.get(replicate_id) - if not isinstance(allowed, list) or ( - "*" not in allowed and stage_id not in allowed - ): - raise ValueError("replicate is not declared for this training stage") - - -def parent_replicate_id( - protocol: Mapping[str, Any], - *, - stage_id: str, - replicate_id: str, -) -> str: - """Resolve a protocol-bound parent execution identity.""" - - validate_stage_replicate( - protocol, - stage_id=stage_id, - replicate_id=replicate_id, - ) - overrides = protocol["launch"]["parent_replicate_overrides"] - stage_overrides = overrides.get(stage_id, {}) - if not isinstance(stage_overrides, Mapping): - raise ValueError("parent replicate override is malformed") - result = str(stage_overrides.get(replicate_id, replicate_id)) - if result not in protocol["launch"]["replicates"]: - raise ValueError("parent replicate override names an unknown identity") - return result - - -def validate_source_sha(repo_root: Path, expected_source_sha: str) -> str: - if not _GIT_SHA.fullmatch(expected_source_sha): - raise ValueError("expected source SHA must be a 40-character Git SHA") - root = repo_root.expanduser().resolve() - head = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=root, - text=True, - ).strip() - if head != expected_source_sha: - raise ValueError(f"HEAD {head} differs from expected {expected_source_sha}") - status = subprocess.check_output( - ["git", "status", "--porcelain", "--untracked-files=all"], - cwd=root, - text=True, - ).strip() - if status: - raise ValueError("paid training requires a completely clean worktree") - return head diff --git a/rl/studies/representation_training_v1/record_approval.py b/rl/studies/representation_training_v1/record_approval.py deleted file mode 100644 index 63eaee1c..00000000 --- a/rl/studies/representation_training_v1/record_approval.py +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env python3 -"""Record one reviewed checkpoint report as an immutable wave approval.""" - -from __future__ import annotations - -import argparse -import json -import os -from pathlib import Path - -from .promotion import ( - APPROVAL_CONFIRMATION_TOKEN, - record_wave_approval, -) -from .protocol import repository_root - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--stage", required=True) - result.add_argument("--wave", required=True) - result.add_argument("--replicate", default="r0") - result.add_argument("--evaluation-stage", required=True) - result.add_argument("--evaluation-wave", required=True) - result.add_argument("--evaluation-checkpoint", required=True) - result.add_argument("--evaluation-report-record-sha256", required=True) - result.add_argument( - "--entry-baseline-report-record-sha256", - default=None, - help=( - "second report required by the pure-base L1-L4 transition gate: " - "the exact parent checkpoint evaluated on the incoming level" - ), - ) - result.add_argument("--expected-source-sha", required=True) - result.add_argument( - "--evidence-source-sha", - default=None, - help=( - "exact ancestor commit that produced the evaluation report; " - "omit for the unchanged same-source v1 approval path" - ), - ) - result.add_argument( - "--approve-changed-path", - action="append", - default=None, - dest="approved_change_paths", - help=( - "one exact producer-to-consumer changed path reviewed for a v2 " - "transition; repeat for every changed path and omit for v1" - ), - ) - result.add_argument( - "--confirm-approval", - default="", - help=f"must be exactly {APPROVAL_CONFIRMATION_TOKEN}", - ) - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_TRAINING_ROOT"]) - if os.environ.get("PIXCELL_TRAINING_ROOT") - else None - ), - ) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit("set PIXCELL_TRAINING_ROOT or pass --external-root outside Git") - result = record_wave_approval( - repo_root=repository_root(), - external_root=args.external_root, - stage_id=args.stage, - wave_name=args.wave, - replicate_id=args.replicate, - evaluation_stage_id=args.evaluation_stage, - evaluation_wave=args.evaluation_wave, - evaluation_checkpoint_name=args.evaluation_checkpoint, - evaluation_report_record_sha256=(args.evaluation_report_record_sha256), - entry_baseline_report_record_sha256=( - args.entry_baseline_report_record_sha256 - ), - expected_source_sha=args.expected_source_sha, - evidence_source_sha=args.evidence_source_sha, - approved_change_paths=args.approved_change_paths, - confirmation=args.confirm_approval, - ) - print(json.dumps(result, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/record_parent_transition.py b/rl/studies/representation_training_v1/record_parent_transition.py deleted file mode 100644 index 871af6cb..00000000 --- a/rl/studies/representation_training_v1/record_parent_transition.py +++ /dev/null @@ -1,67 +0,0 @@ -#!/usr/bin/env python3 -"""Approve the exact historical L0-SFT receipt for the current L1-RL smoke.""" - -from __future__ import annotations - -import argparse -import json -import os -from pathlib import Path - -from .promotion import ( - APPROVAL_CONFIRMATION_TOKEN, - record_parent_receipt_transition_approval, -) -from .protocol import repository_root - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--replicate", default="r0") - result.add_argument("--expected-source-sha", required=True) - result.add_argument( - "--approve-changed-path", - action="append", - required=True, - dest="approved_change_paths", - help=( - "one exact path in the historical-parent-to-current committed diff; " - "repeat once for every changed path" - ), - ) - result.add_argument( - "--confirm-approval", - default="", - help=f"must be exactly {APPROVAL_CONFIRMATION_TOKEN}", - ) - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_TRAINING_ROOT"]) - if os.environ.get("PIXCELL_TRAINING_ROOT") - else None - ), - ) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit("set PIXCELL_TRAINING_ROOT or pass --external-root outside Git") - result = record_parent_receipt_transition_approval( - repo_root=repository_root(), - external_root=args.external_root, - stage_id="qwen-l0-rl-l1", - wave_name="smoke", - replicate_id=args.replicate, - expected_source_sha=args.expected_source_sha, - approved_change_paths=args.approved_change_paths, - confirmation=args.confirm_approval, - ) - print(json.dumps(result, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/renderer_diagnostic.py b/rl/studies/representation_training_v1/renderer_diagnostic.py deleted file mode 100644 index 914b5c32..00000000 --- a/rl/studies/representation_training_v1/renderer_diagnostic.py +++ /dev/null @@ -1,2078 +0,0 @@ -"""Sealed, non-canonical diagnostic for the mixed-SFT Qwen renderer mismatch. - -This module answers one narrow postmortem question: did the mixed-SFT adapter -learn a usable direct-code policy behind the closed-thinking prefix it saw -during teacher forcing, and when did canonical open-thinking generation -degrade? - -It is intentionally not an evaluation panel. Its records live below -``diagnostics/``, use a foreign record schema, carry an explicit -``not_canonical_evidence`` marker, and cannot satisfy any promotion gate. -""" - -from __future__ import annotations - -import argparse -import asyncio -import hashlib -import json -import math -import os -import uuid -from collections import Counter -from collections.abc import Callable, Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any - -from tinker_cookbook.renderers import Message, TextPart, TrainOnWhat - -from rl.common.evaluator import ( - Attribution, - EvaluationResult, - EvaluationStatus, - PixCellEvaluator, -) -from rl.common.output import extract_code -from rl.common.prompt import prompt_asset_hashes -from rl.common.runtime import validate_runtime_stack -from rl.evaluation.tasks import EvaluationTask, canonical_json_sha256 -from rl.track_a.tinker_data import _message, _renderer - -from . import evaluation as checkpoint_v1 -from . import source_transition as source_transition_v2 -from .dataset_binding import _dataset_binding -from .preflight import ( - _cookbook_source_binding, - _positive_target_text, -) -from .protocol import ( - STUDY_ID, - file_sha256, - load_protocol, - protocol_path, - stage_spec, - validate_source_sha, -) -from .schedule import stage_tasks -from .store import TrainingStore - - -DIAGNOSTIC_ID = "qwen-sft-renderer-diagnostic-v1" -DIAGNOSTIC_SCHEMA_VERSION = "pixcell-qwen-sft-renderer-diagnostic-v1" -DIAGNOSTIC_RECORD_SCHEMA_VERSION = "pixcell-qwen-sft-renderer-diagnostic-record-v1" -TASK_PANEL_SCHEMA_VERSION = "pixcell-qwen-renderer-diagnostic-task-panel-v1" -PRODUCER_SOURCE_GIT_SHA = "bb80dbe666c7313f9a9f3bdb53765c35dfdeceab" -PRODUCER_STAGE_ID = "qwen-mixed-sft" -PRODUCER_WAVE = "complete" -PRODUCER_REPLICATE_ID = "r0" -EXPECTED_TASK_IDS = tuple(f"F{index}" for index in range(1, 9)) -MAX_IMAGE_LONG_EDGE = 1920 -MAX_OUTPUT_TOKENS = 60_000 -TEMPERATURE = 1.0 -TOP_P = 1.0 -SAMPLE_CONCURRENCY = 8 -EVALUATION_BATCH_SIZE = 32 -EVALUATOR_WORKERS = 8 -_SEED_DOMAIN = "pixcell-qwen-sft-renderer-diagnostic-seed-v1" -_PROMOTION_POLICY = "diagnostic-only-never-promotion-evidence" -_PROMPT_SOURCE_PATHS = ( - "rl/common/assets/phase_a_direct_v2.md", - "src/michaelangelo/reference/gds_factory_function_catalogue_extended.md", -) - - -@dataclass(frozen=True) -class ArmSpec: - arm_id: str - checkpoint_name: str - renderer: str - expected_sft_prefix_compatible: bool - - def as_dict(self) -> dict[str, Any]: - return { - "arm_id": self.arm_id, - "checkpoint_name": self.checkpoint_name, - "renderer": self.renderer, - "expected_sft_prefix_compatible": (self.expected_sft_prefix_compatible), - } - - -ARM_SPECS = ( - ArmSpec( - arm_id="final-no-thinking", - checkpoint_name="final", - renderer="qwen3_5_disable_thinking", - expected_sft_prefix_compatible=True, - ), - ArmSpec( - arm_id="step-14-thinking", - checkpoint_name="000014", - renderer="qwen3_5", - expected_sft_prefix_compatible=False, - ), - ArmSpec( - arm_id="step-28-thinking", - checkpoint_name="000028", - renderer="qwen3_5", - expected_sft_prefix_compatible=False, - ), - ArmSpec( - arm_id="step-42-thinking", - checkpoint_name="000042", - renderer="qwen3_5", - expected_sft_prefix_compatible=False, - ), -) -EXPECTED_CHECKPOINT_NAMES = ("000014", "000028", "000042", "final") -EXPECTED_SAMPLE_COUNT = len(ARM_SPECS) * len(EXPECTED_TASK_IDS) - - -class RendererDiagnosticError(checkpoint_v1.CheckpointEvaluationError): - """The renderer diagnostic is unsafe, inconsistent, or incomplete.""" - - -class ImmutableRendererDiagnosticError(checkpoint_v1.ImmutableEvaluationRecordError): - """A create-only diagnostic record already differs.""" - - -@dataclass(frozen=True) -class PreparedArm: - spec: ArmSpec - binding: checkpoint_v1.CheckpointBinding - renderer: Any - prepared: tuple[checkpoint_v1.PreparedTask, ...] - preflight: dict[str, Any] - - -@dataclass(frozen=True) -class PreparedAttempt: - arm: PreparedArm - prepared: checkpoint_v1.PreparedTask - seed: int - - @property - def task(self) -> EvaluationTask: - return self.prepared.task - - @property - def key(self) -> tuple[str, str]: - return (self.arm.spec.arm_id, self.task.task_id) - - -def _canonical_bytes(value: Any) -> bytes: - try: - return json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise RendererDiagnosticError("renderer diagnostic is not finite JSON") from exc - - -def _canonical_sha256(value: Any) -> str: - return hashlib.sha256(_canonical_bytes(value)).hexdigest() - - -def _record_document( - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], -) -> dict[str, Any]: - normalized = json.loads(_canonical_bytes(payload)) - document = { - "schema_version": DIAGNOSTIC_RECORD_SCHEMA_VERSION, - "record_type": record_type, - "key": dict(key), - "payload": normalized, - "payload_sha256": _canonical_sha256(normalized), - } - document["record_sha256"] = _canonical_sha256(document) - return document - - -def _validate_record( - value: Mapping[str, Any], - *, - record_type: str, - key: Mapping[str, Any], -) -> dict[str, Any]: - required = { - "schema_version", - "record_type", - "key", - "payload", - "payload_sha256", - "record_sha256", - } - if set(value) != required: - raise RendererDiagnosticError("diagnostic record schema changed") - if value["schema_version"] != DIAGNOSTIC_RECORD_SCHEMA_VERSION: - raise RendererDiagnosticError("diagnostic record has a foreign schema") - if value["record_type"] != record_type or value["key"] != dict(key): - raise RendererDiagnosticError("diagnostic record key differs from its canonical path") - payload = value["payload"] - if not isinstance(payload, dict): - raise RendererDiagnosticError("diagnostic payload is not an object") - if payload.get("not_canonical_evidence") is not True: - raise RendererDiagnosticError("diagnostic record lost its non-canonical marker") - if value["payload_sha256"] != _canonical_sha256(payload): - raise RendererDiagnosticError("diagnostic payload digest mismatch") - unsigned = dict(value) - observed = unsigned.pop("record_sha256") - if ( - not isinstance(observed, str) - or not checkpoint_v1._SHA256.fullmatch(observed) - or observed != _canonical_sha256(unsigned) - ): - raise RendererDiagnosticError("diagnostic envelope digest mismatch") - return dict(value) - - -class RendererDiagnosticStore(checkpoint_v1.CheckpointEvaluationStore): - """Create-only records in a namespace promotion code never reads.""" - - def __init__(self, *, stage_path: Path) -> None: - self.stage_path = stage_path.expanduser().resolve(strict=True) - requested = self.stage_path / "diagnostics" / DIAGNOSTIC_ID - requested.mkdir(parents=True, exist_ok=True, mode=0o700) - resolved = requested.resolve(strict=True) - if resolved != requested or not resolved.is_relative_to(self.stage_path): - raise RendererDiagnosticError( - "diagnostic root must be a real directory inside its stage" - ) - if "evaluations" in resolved.relative_to(self.stage_path).parts: - raise RendererDiagnosticError( - "diagnostic records must not enter the evaluation namespace" - ) - self.root = resolved - - @staticmethod - def _arm_component(arm_id: str) -> str: - expected = {spec.arm_id for spec in ARM_SPECS} - if arm_id not in expected: - raise ValueError(f"foreign diagnostic arm: {arm_id!r}") - return arm_id - - @classmethod - def _attempt_key(cls, arm_id: str, task_id: str) -> dict[str, str]: - cls._arm_component(arm_id) - cls._task_component(task_id) - return {"arm_id": arm_id, "task_id": task_id} - - @classmethod - def _attempt_path( - cls, - arm_id: str, - task_id: str, - filename: str, - ) -> PurePosixPath: - return PurePosixPath( - "arms", - cls._arm_component(arm_id), - "tasks", - cls._task_component(task_id), - filename, - ) - - def load_manifest(self) -> dict[str, Any] | None: - return self._load_diagnostic( - PurePosixPath("manifest.json"), - record_type="renderer_diagnostic_manifest", - key={}, - ) - - def create_or_verify_manifest( - self, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_diagnostic( - PurePosixPath("manifest.json"), - record_type="renderer_diagnostic_manifest", - key={}, - payload=payload, - ) - - def load_sample( - self, - arm_id: str, - task_id: str, - ) -> dict[str, Any] | None: - return self._load_diagnostic( - self._attempt_path(arm_id, task_id, "sample.json"), - record_type="renderer_diagnostic_sample", - key=self._attempt_key(arm_id, task_id), - ) - - def create_or_verify_sample( - self, - arm_id: str, - task_id: str, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_diagnostic( - self._attempt_path(arm_id, task_id, "sample.json"), - record_type="renderer_diagnostic_sample", - key=self._attempt_key(arm_id, task_id), - payload=payload, - ) - - def load_evaluation( - self, - arm_id: str, - task_id: str, - ) -> dict[str, Any] | None: - return self._load_diagnostic( - self._attempt_path(arm_id, task_id, "evaluation.json"), - record_type="renderer_diagnostic_evaluation", - key=self._attempt_key(arm_id, task_id), - ) - - def create_or_verify_evaluation( - self, - arm_id: str, - task_id: str, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_diagnostic( - self._attempt_path(arm_id, task_id, "evaluation.json"), - record_type="renderer_diagnostic_evaluation", - key=self._attempt_key(arm_id, task_id), - payload=payload, - ) - - def load_report(self) -> dict[str, Any] | None: - return self._load_diagnostic( - PurePosixPath("report.json"), - record_type="renderer_diagnostic_report", - key={}, - ) - - def create_or_verify_report( - self, - payload: Mapping[str, Any], - ) -> dict[str, Any]: - return self._create_or_verify_diagnostic( - PurePosixPath("report.json"), - record_type="renderer_diagnostic_report", - key={}, - payload=payload, - ) - - def assert_attempt_inventory( - self, - attempts: Sequence[PreparedAttempt], - ) -> None: - arms_root = self.root / "arms" - expected_files = { - self._path( - self._attempt_path( - attempt.arm.spec.arm_id, - attempt.task.task_id, - filename, - ) - ) - for attempt in attempts - for filename in ("sample.json", "evaluation.json") - } - expected_files.update( - { - self.root / ".lock", - self.root / "manifest.json", - self.root / "report.json", - } - ) - expected_directories = {self.root, arms_root} - for path in expected_files: - expected_directories.update(path.parents) - for path in self.root.rglob("*"): - if path.is_symlink(): - raise RendererDiagnosticError( - f"diagnostic record inventory contains a symlink: {path}" - ) - if path.is_file() and path not in expected_files: - raise RendererDiagnosticError( - f"diagnostic record inventory contains a foreign row: {path}" - ) - if path.is_dir() and path not in expected_directories: - raise RendererDiagnosticError( - f"diagnostic record inventory contains a foreign path: {path}" - ) - - def _create_or_verify_diagnostic( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], - ) -> dict[str, Any]: - expected = _record_document( - record_type=record_type, - key=key, - payload=payload, - ) - path = self._path(relative) - path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - raw = _canonical_bytes(expected) + b"\n" - temporary = path.parent / f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}" - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(temporary, flags, 0o600) - try: - with os.fdopen(fd, "wb", closefd=False) as stream: - stream.write(raw) - stream.flush() - os.fsync(stream.fileno()) - finally: - os.close(fd) - try: - os.link(temporary, path) - directory_fd = os.open(path.parent, os.O_RDONLY) - try: - os.fsync(directory_fd) - finally: - os.close(directory_fd) - except FileExistsError: - observed = self._load_diagnostic( - relative, - record_type=record_type, - key=key, - ) - if observed is None or _canonical_bytes(observed) != _canonical_bytes(expected): - raise ImmutableRendererDiagnosticError( - f"{relative} already contains different data" - ) - return observed - finally: - try: - temporary.unlink() - except FileNotFoundError: - pass - return expected - - def _load_diagnostic( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - ) -> dict[str, Any] | None: - path = self._path(relative) - if not path.exists(): - return None - return _validate_record( - self._read(path), - record_type=record_type, - key=key, - ) - - -def _not_canonical() -> dict[str, Any]: - return { - "not_canonical_evidence": True, - "promotion_eligibility": { - "eligible": False, - "policy": _PROMOTION_POLICY, - }, - } - - -def _validate_fixed_protocol(protocol: Mapping[str, Any]) -> dict[str, Any]: - stage = stage_spec(dict(protocol), PRODUCER_STAGE_ID) - model = dict(protocol["models"][stage.model_key]) - recipe = dict(protocol["recipes"][stage.recipe_key]) - if ( - stage.kind != "sft" - or stage.model_key != "qwen" - or stage.recipe_key != "qwen_sft" - or stage.wave(PRODUCER_WAVE)["max_steps"] != 55 - ): - raise RendererDiagnosticError("producer SFT stage contract changed") - expected_model = { - "model": "Qwen/Qwen3.6-35B-A3B", - "renderer": "qwen3_5", - "thinking": True, - "max_image_long_edge": MAX_IMAGE_LONG_EDGE, - "max_output_tokens": MAX_OUTPUT_TOKENS, - "context_tokens": 65_536, - "lora_rank": 32, - } - if model != expected_model: - raise RendererDiagnosticError("Qwen model contract changed") - if ( - recipe.get("target") != "program_and_end_token_only" - or recipe.get("max_sequence_tokens") != 6144 - ): - raise RendererDiagnosticError("SFT target contract changed") - return { - "stage": stage, - "model": model, - "recipe": recipe, - } - - -def _source_provenance( - *, - repo_root: Path, - evaluator_source_git_sha: str, -) -> dict[str, Any]: - facts = source_transition_v2.source_transition_git_facts( - repo_root=repo_root, - producer_source_git_sha=PRODUCER_SOURCE_GIT_SHA, - consumer_source_git_sha=evaluator_source_git_sha, - ) - paths = (str(protocol_path(repo_root).relative_to(repo_root)), *_PROMPT_SOURCE_PATHS) - bindings: dict[str, Any] = {} - for relative in paths: - producer_bytes = source_transition_v2._git( - repo_root, - "show", - f"{PRODUCER_SOURCE_GIT_SHA}:{relative}", - ) - current_path = (repo_root / relative).resolve(strict=True) - if not current_path.is_relative_to(repo_root): - raise RendererDiagnosticError("source binding escaped the repository") - current_bytes = current_path.read_bytes() - producer_sha = hashlib.sha256(producer_bytes).hexdigest() - current_sha = hashlib.sha256(current_bytes).hexdigest() - if producer_sha != current_sha: - raise RendererDiagnosticError( - f"producer and evaluator task contract differ: {relative}" - ) - bindings[relative] = { - "producer_sha256": producer_sha, - "evaluator_sha256": current_sha, - "byte_identical": True, - } - return { - "producer_source_git_sha": PRODUCER_SOURCE_GIT_SHA, - "evaluator_source_git_sha": evaluator_source_git_sha, - "git_relationship": facts, - "task_contract_files": bindings, - "task_contract_files_sha256": _canonical_sha256(bindings), - } - - -def _load_matrix_bindings( - *, - store: TrainingStore, - protocol: dict[str, Any], - expected_wave_receipt_sha256: str, -) -> tuple[checkpoint_v1.CheckpointBinding, ...]: - bindings = tuple( - checkpoint_v1._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id=PRODUCER_STAGE_ID, - wave=PRODUCER_WAVE, - checkpoint_name=spec.checkpoint_name, - replicate_id=PRODUCER_REPLICATE_ID, - expected_receipt_sha256=expected_wave_receipt_sha256, - source_git_sha=PRODUCER_SOURCE_GIT_SHA, - ) - for spec in ARM_SPECS - ) - receipts = {binding.wave_receipt.record_sha256 for binding in bindings} - inventories = {binding.checkpoint_inventory_sha256 for binding in bindings} - manifests = {binding.run_manifest.record_sha256 for binding in bindings} - if len(receipts) != 1 or len(inventories) != 1 or len(manifests) != 1: - raise RendererDiagnosticError("matrix checkpoints do not share one exact producer receipt") - inventory = bindings[0].wave_receipt.payload["checkpoint_inventory"] - entries = inventory["entries"] - if tuple(str(entry["name"]) for entry in entries) != EXPECTED_CHECKPOINT_NAMES: - raise RendererDiagnosticError( - "producer checkpoint inventory differs from the sealed matrix" - ) - expected_paths = { - spec.checkpoint_name: binding.sampler_path - for spec, binding in zip(ARM_SPECS, bindings, strict=True) - } - if len(set(expected_paths.values())) != len(ARM_SPECS): - raise RendererDiagnosticError("producer checkpoint sampler paths are not unique") - return bindings - - -def _producer_training_provenance( - *, - repo_root: Path, - protocol: Mapping[str, Any], - bindings: Sequence[checkpoint_v1.CheckpointBinding], -) -> dict[str, Any]: - first = bindings[0] - manifest = first.run_manifest - receipt = first.wave_receipt - payload = manifest.payload - expected_manifest_fields = { - "source_git_sha": PRODUCER_SOURCE_GIT_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - "prompt_assets": prompt_asset_hashes(), - "model": protocol["models"]["qwen"], - "recipe": protocol["recipes"]["qwen_sft"], - } - for field, expected in expected_manifest_fields.items(): - if payload.get(field) != expected: - raise RendererDiagnosticError(f"producer training manifest {field} changed") - stage = payload.get("stage") - dataset = payload.get("dataset") - if ( - not isinstance(stage, Mapping) - or stage.get("stage_id") != PRODUCER_STAGE_ID - or stage.get("kind") != "sft" - or stage.get("model_key") != "qwen" - or stage.get("recipe_key") != "qwen_sft" - or not isinstance(dataset, Mapping) - or dataset.get("logical_release_sha256") != protocol["dataset"]["logical_release_sha256"] - ): - raise RendererDiagnosticError("producer training manifest stage or dataset changed") - inventory = receipt.payload["checkpoint_inventory"] - return { - "stage_id": PRODUCER_STAGE_ID, - "wave": PRODUCER_WAVE, - "replicate_id": PRODUCER_REPLICATE_ID, - "run_manifest": { - "relative_path": str(manifest.relative_path), - "payload_sha256": manifest.payload_sha256, - "record_sha256": manifest.record_sha256, - }, - "wave_receipt": { - "relative_path": str(receipt.relative_path), - "payload_sha256": receipt.payload_sha256, - "record_sha256": receipt.record_sha256, - }, - "checkpoint_inventory": inventory, - "checkpoint_inventory_sha256": first.checkpoint_inventory_sha256, - "samplers": { - binding.checkpoint_name: { - "checkpoint": binding.checkpoint, - "sampler_path": binding.sampler_path, - } - for binding in bindings - }, - } - - -def _normalized_segments( - model_input: Any, - *, - stop_positions: int | None = None, -) -> list[dict[str, Any]]: - """Normalize text chunk boundaries while preserving exact image identity.""" - - total = int(model_input.length) - stop = total if stop_positions is None else stop_positions - if isinstance(stop, bool) or not isinstance(stop, int) or not 0 <= stop <= total: - raise RendererDiagnosticError("invalid renderer-prefix slice") - segments: list[dict[str, Any]] = [] - consumed = 0 - for chunk in model_input.chunks: - if consumed >= stop: - break - length = int(chunk.length) - take = min(length, stop - consumed) - tokens = getattr(chunk, "tokens", None) - if tokens is not None: - selected = [int(token) for token in list(tokens)[:take]] - if segments and segments[-1]["type"] == "tokens": - segments[-1]["values"].extend(selected) - else: - segments.append({"type": "tokens", "values": selected}) - else: - if take != length: - raise RendererDiagnosticError("renderer prefix cuts through an image chunk") - data = getattr(chunk, "data", None) - if not isinstance(data, bytes): - raise RendererDiagnosticError("renderer emitted an unhashable image chunk") - segments.append( - { - "type": "image", - "length": length, - "format": str(getattr(chunk, "format", "")), - "data_sha256": hashlib.sha256(data).hexdigest(), - } - ) - consumed += take - if consumed != stop: - raise RendererDiagnosticError("renderer prefix length is inconsistent") - return segments - - -def _append_token( - segments: Sequence[Mapping[str, Any]], - token: int, -) -> list[dict[str, Any]]: - result = json.loads(_canonical_bytes(segments)) - if result and result[-1]["type"] == "tokens": - result[-1]["values"].append(token) - else: - result.append({"type": "tokens", "values": [token]}) - return result - - -def _first_positive_offset(weights: Any, *, length: int) -> int: - found = [index for index in range(length) if float(weights[index]) > 0.0] - if not found: - raise RendererDiagnosticError("SFT supervised target has no positive loss") - first = found[0] - if any(float(weights[index]) != 0.0 for index in range(first)): - raise RendererDiagnosticError("SFT loss mask is not prefix-zero") - return first - - -def _sft_prefix_audit( - *, - dataset_root: Path, - protocol: Mapping[str, Any], - tasks: Sequence[Any] | None = None, -) -> dict[str, Any]: - """Prove the actual teacher-forced prefix against both inference renderers.""" - - stage = stage_spec(dict(protocol), PRODUCER_STAGE_ID) - model = dict(protocol["models"]["qwen"]) - source_tasks = list(tasks) if tasks is not None else stage_tasks(dataset_root, stage) - if len(source_tasks) != (len(source_tasks) if tasks is not None else 3468): - raise RendererDiagnosticError("SFT prefix audit has the wrong row count") - training_renderer = _renderer(model["model"], model["renderer"]) - inference_renderers = { - name: _renderer(model["model"], name) for name in ("qwen3_5_disable_thinking", "qwen3_5") - } - per_renderer: dict[str, dict[str, Any]] = { - name: { - "compatible_rows": 0, - "incompatible_rows": 0, - "evidence": [], - } - for name in inference_renderers - } - source_digest = hashlib.sha256() - for task in source_tasks: - messages = [_message(task, max_image=MAX_IMAGE_LONG_EDGE)] - assistant = Message( - role="assistant", - content=[TextPart(type="text", text=task.label)], - ) - supervised, weights = training_renderer.build_supervised_example( - [*messages, assistant], - train_on_what=TrainOnWhat.LAST_ASSISTANT_MESSAGE, - ) - first = _first_positive_offset(weights, length=int(supervised.length)) - target_text = _positive_target_text( - supervised, - weights, - training_renderer.tokenizer, - ) - if target_text != task.label + "<|im_end|>": - raise RendererDiagnosticError( - f"{task.sampler.opaque_id} SFT target is not exact code plus end token" - ) - before = _normalized_segments(supervised, stop_positions=first) - through_first = _normalized_segments( - supervised, - stop_positions=first + 1, - ) - first_token_segment = through_first[-1] - if first_token_segment["type"] != "tokens": - raise RendererDiagnosticError("first positive loss token is not text") - first_token = int(first_token_segment["values"][-1]) - source_digest.update(task.sampler.opaque_id.encode("utf-8")) - source_digest.update(bytes.fromhex(task.observation.image_sha256)) - source_digest.update(hashlib.sha256(task.label.encode("utf-8")).digest()) - for name, renderer in inference_renderers.items(): - generation = renderer.build_generation_prompt(messages) - generation_segments = _normalized_segments(generation) - prefix_equal = generation_segments == before - through_equal = _append_token(generation_segments, first_token) == through_first - compatible = prefix_equal and through_equal - bucket = per_renderer[name] - bucket["compatible_rows" if compatible else "incompatible_rows"] += 1 - evidence = { - "task_id": task.sampler.opaque_id, - "first_positive_loss_offset": first, - "first_positive_token": first_token, - "supervised_prefix_sha256": _canonical_sha256(before), - "supervised_through_first_positive_sha256": (_canonical_sha256(through_first)), - "generation_prefix_sha256": _canonical_sha256(generation_segments), - "generation_plus_first_positive_sha256": _canonical_sha256( - _append_token(generation_segments, first_token) - ), - "prefix_equal": prefix_equal, - "through_first_positive_equal": through_equal, - } - bucket["evidence"].append(evidence) - results: dict[str, Any] = {} - for name, bucket in per_renderer.items(): - expected = name == "qwen3_5_disable_thinking" - compatible_rows = int(bucket["compatible_rows"]) - compatible = compatible_rows == len(source_tasks) - if compatible is not expected: - claim = "compatible" if expected else "incompatible" - raise RendererDiagnosticError(f"{name} is not uniformly {claim} with the SFT prefix") - evidence = bucket.pop("evidence") - results[name] = { - **bucket, - "rows": len(source_tasks), - "compatible_with_sft_supervised_prefix": compatible, - "expected_compatible_claim": expected, - "claim_verified": True, - "comparison_scope": ( - "generation prefix plus first target token through the first positive-loss token" - ), - "diagnosis": ( - "The no-thinking generation prefix exactly matches the " - "teacher-forced closed-thinking prefix." - if expected - else ( - "The canonical renderer leaves open, while code-only " - "SFT closes before the first positive-loss code token; " - "a compatibility claim is rejected." - ) - ), - "evidence_sha256": canonical_json_sha256(evidence), - "example": evidence[0], - } - return { - "schema_version": "pixcell-qwen-sft-prefix-audit-v1", - "rows": len(source_tasks), - "training_renderer": model["renderer"], - "train_on_what": "LAST_ASSISTANT_MESSAGE", - "target_policy": "program_and_end_token_only", - "source_set_sha256": source_digest.hexdigest(), - "renderers": results, - } - - -def _task_panel(tasks: Sequence[EvaluationTask]) -> dict[str, Any]: - entries = [ - { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "image_sha256": task.observation.image_sha256, - "target_image_sha256": task.reference.target_image_sha256, - "footprint_um": list(task.observation.footprint_um), - } - for task in tasks - ] - if tuple(item["task_id"] for item in entries) != EXPECTED_TASK_IDS: - raise checkpoint_v1.ReferencePanelFault("renderer diagnostic must be exact ordered F1-F8") - return { - "schema_version": TASK_PANEL_SCHEMA_VERSION, - "diagnostic_id": DIAGNOSTIC_ID, - "task_count": len(entries), - "task_ids": list(EXPECTED_TASK_IDS), - "logical_sha256": canonical_json_sha256(entries), - } - - -def _sample_seed( - *, - evaluator_source_git_sha: str, - protocol_sha256: str, - dataset_sha256: str, - task_panel_sha256: str, - arm: ArmSpec, - binding: checkpoint_v1.CheckpointBinding, - task_id: str, -) -> int: - digest = hashlib.sha256( - "\0".join( - ( - _SEED_DOMAIN, - PRODUCER_SOURCE_GIT_SHA, - evaluator_source_git_sha, - protocol_sha256, - dataset_sha256, - task_panel_sha256, - binding.wave_receipt.record_sha256, - binding.checkpoint_inventory_sha256, - arm.arm_id, - arm.checkpoint_name, - arm.renderer, - task_id, - "1", - ) - ).encode("utf-8") - ).digest() - return int.from_bytes(digest[:4], "big") & 0x7FFFFFFF - - -def _prepare_arms( - *, - protocol: Mapping[str, Any], - model: Mapping[str, Any], - bindings: Sequence[checkpoint_v1.CheckpointBinding], - tasks: Sequence[EvaluationTask], - evaluator: PixCellEvaluator, - evaluator_source_git_sha: str, - dataset_sha256: str, - task_panel_sha256: str, -) -> tuple[tuple[PreparedArm, ...], tuple[PreparedAttempt, ...]]: - arms: list[PreparedArm] = [] - attempts: list[PreparedAttempt] = [] - for spec, binding in zip(ARM_SPECS, bindings, strict=True): - renderer = _renderer(model["model"], spec.renderer) - prepared, preflight = checkpoint_v1._preflight_tasks( - tasks=tasks, - renderer=renderer, - evaluator=evaluator, - context_tokens=int(model["context_tokens"]), - ) - arm = PreparedArm( - spec=spec, - binding=binding, - renderer=renderer, - prepared=tuple(prepared), - preflight=preflight, - ) - arms.append(arm) - attempts.extend( - PreparedAttempt( - arm=arm, - prepared=item, - seed=_sample_seed( - evaluator_source_git_sha=evaluator_source_git_sha, - protocol_sha256=str(protocol["logical_sha256"]), - dataset_sha256=dataset_sha256, - task_panel_sha256=task_panel_sha256, - arm=spec, - binding=binding, - task_id=item.task.task_id, - ), - ) - for item in prepared - ) - if len(attempts) != EXPECTED_SAMPLE_COUNT: - raise RendererDiagnosticError(f"diagnostic resolved {len(attempts)} samples, expected 32") - keys = [attempt.key for attempt in attempts] - seeds = [attempt.seed for attempt in attempts] - if len(set(keys)) != EXPECTED_SAMPLE_COUNT: - raise RendererDiagnosticError("diagnostic attempt keys are not unique") - if len(set(seeds)) != EXPECTED_SAMPLE_COUNT: - raise RendererDiagnosticError("diagnostic deterministic seeds collided") - return tuple(arms), tuple(attempts) - - -def _decode_response(renderer: Any, sequence: Any) -> dict[str, Any]: - from tinker_cookbook.renderers import get_text_content - - tokens = [int(token) for token in sequence.tokens] - token_decode_error: str | None = None - try: - raw_text = str(renderer.tokenizer.decode(tokens)) - except Exception as exc: - raw_text = "" - token_decode_error = str(exc).replace("\x00", "")[-1200:] - channel_parse_error: str | None = None - channel_termination: str | None = None - if token_decode_error is not None: - answer_text = "" - reasoning_text = "" - channel_parse_complete = False - channel_parse_status = "token_decode_error" - else: - try: - message, termination = renderer.parse_response(tokens) - answer_text = str(get_text_content(message)) - reasoning_text = checkpoint_v1._message_reasoning(message) - channel_parse_complete = bool(termination.is_clean) - channel_termination = str(termination) - channel_parse_status = "clean" if channel_parse_complete else "unclean" - except Exception as exc: - answer_text = raw_text - reasoning_text = "" - channel_parse_complete = False - channel_parse_status = "error" - channel_parse_error = str(exc).replace("\x00", "")[-1200:] - extracted_program = extract_code(answer_text) - stop_reason = str(sequence.stop_reason) - cap_hit = ( - len(tokens) >= MAX_OUTPUT_TOKENS - or "length" in stop_reason.lower() - or "max_token" in stop_reason.lower() - ) - return { - "token_ids": tokens, - "token_ids_sha256": _canonical_sha256(tokens), - "completion_tokens": len(tokens), - "stop_reason": stop_reason, - "cap_hit": cap_hit, - "raw_text": raw_text, - "raw_text_sha256": hashlib.sha256(raw_text.encode("utf-8")).hexdigest(), - "answer_text": answer_text, - "answer_text_sha256": hashlib.sha256(answer_text.encode("utf-8")).hexdigest(), - "reasoning_text": reasoning_text, - "reasoning_text_sha256": hashlib.sha256(reasoning_text.encode("utf-8")).hexdigest(), - "extracted_program": extracted_program, - "extracted_program_sha256": hashlib.sha256(extracted_program.encode("utf-8")).hexdigest(), - "token_decode_complete": token_decode_error is None, - "token_decode_error": token_decode_error, - "channel_parse_complete": channel_parse_complete, - "channel_parse_status": channel_parse_status, - "channel_parse_error": channel_parse_error, - "channel_termination": channel_termination, - } - - -_RESPONSE_FIELDS = { - "token_ids", - "token_ids_sha256", - "completion_tokens", - "stop_reason", - "cap_hit", - "raw_text", - "raw_text_sha256", - "answer_text", - "answer_text_sha256", - "reasoning_text", - "reasoning_text_sha256", - "extracted_program", - "extracted_program_sha256", - "token_decode_complete", - "token_decode_error", - "channel_parse_complete", - "channel_parse_status", - "channel_parse_error", - "channel_termination", -} - - -def _validate_response( - response: Any, - *, - attempt: PreparedAttempt, -) -> None: - if not isinstance(response, Mapping) or set(response) != _RESPONSE_FIELDS: - raise ImmutableRendererDiagnosticError(f"{attempt.key} raw response schema changed") - tokens = response["token_ids"] - if ( - not isinstance(tokens, list) - or any( - isinstance(token, bool) or not isinstance(token, int) or token < 0 for token in tokens - ) - or response["completion_tokens"] != len(tokens) - or response["token_ids_sha256"] != _canonical_sha256(tokens) - ): - raise ImmutableRendererDiagnosticError(f"{attempt.key} token inventory changed") - for field in ( - "raw_text", - "answer_text", - "reasoning_text", - "extracted_program", - ): - text = response[field] - if ( - not isinstance(text, str) - or response[f"{field}_sha256"] != hashlib.sha256(text.encode("utf-8")).hexdigest() - ): - raise ImmutableRendererDiagnosticError(f"{attempt.key} {field} digest changed") - if response["extracted_program"] != extract_code(response["answer_text"]): - raise ImmutableRendererDiagnosticError( - f"{attempt.key} extracted program differs from the answer" - ) - stop_reason = response["stop_reason"] - expected_cap = len(tokens) >= MAX_OUTPUT_TOKENS or ( - isinstance(stop_reason, str) - and ("length" in stop_reason.lower() or "max_token" in stop_reason.lower()) - ) - parse_status = response["channel_parse_status"] - if ( - not isinstance(stop_reason, str) - or not isinstance(response["cap_hit"], bool) - or response["cap_hit"] is not expected_cap - or not isinstance(response["token_decode_complete"], bool) - or response["token_decode_complete"] is not (response["token_decode_error"] is None) - or ( - response["token_decode_error"] is not None - and not isinstance(response["token_decode_error"], str) - ) - or not isinstance(response["channel_parse_complete"], bool) - or parse_status not in {"clean", "unclean", "error", "token_decode_error"} - or response["channel_parse_complete"] is not (parse_status == "clean") - or ( - response["channel_parse_error"] is not None - and not isinstance(response["channel_parse_error"], str) - ) - or ( - response["channel_termination"] is not None - and not isinstance(response["channel_termination"], str) - ) - or (parse_status in {"clean", "unclean"} and response["channel_termination"] is None) - or ( - parse_status not in {"clean", "unclean"} and response["channel_termination"] is not None - ) - or ( - not response["token_decode_complete"] - and ( - parse_status != "token_decode_error" - or any( - response[field] - for field in ( - "raw_text", - "answer_text", - "reasoning_text", - "extracted_program", - ) - ) - ) - ) - ): - raise ImmutableRendererDiagnosticError(f"{attempt.key} response diagnostics changed") - - -def _sample_payload( - *, - attempt: PreparedAttempt, - response: Mapping[str, Any], - manifest_record_sha256: str, -) -> dict[str, Any]: - binding = attempt.arm.binding - return { - **_not_canonical(), - "diagnostic_manifest_record_sha256": manifest_record_sha256, - "producer_wave_receipt_record_sha256": (binding.wave_receipt.record_sha256), - "diagnostic_id": DIAGNOSTIC_ID, - "arm": attempt.arm.spec.as_dict(), - "checkpoint_inventory_sha256": binding.checkpoint_inventory_sha256, - "checkpoint": binding.checkpoint, - "task": { - "task_id": attempt.task.task_id, - "level": attempt.task.level, - "representation_id": attempt.task.representation_id, - "image_sha256": attempt.task.observation.image_sha256, - "target_image_sha256": attempt.task.reference.target_image_sha256, - }, - "request": { - "num_samples": 1, - "attempt_index": 1, - "seed": attempt.seed, - "max_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "prompt_tokens": attempt.prepared.prompt_tokens, - "prompt_text_sha256": attempt.prepared.prompt_text_sha256, - }, - "response": dict(response), - } - - -def _validate_sample_record( - *, - record: Mapping[str, Any], - attempt: PreparedAttempt, - manifest_record_sha256: str, -) -> None: - payload = record["payload"] - binding = attempt.arm.binding - expected = { - "diagnostic_manifest_record_sha256": manifest_record_sha256, - "producer_wave_receipt_record_sha256": (binding.wave_receipt.record_sha256), - "diagnostic_id": DIAGNOSTIC_ID, - "arm": attempt.arm.spec.as_dict(), - "checkpoint_inventory_sha256": binding.checkpoint_inventory_sha256, - "checkpoint": binding.checkpoint, - } - for field, value in expected.items(): - if payload.get(field) != value: - raise ImmutableRendererDiagnosticError( - f"{attempt.key} sample belongs to another diagnostic" - ) - if payload.get("not_canonical_evidence") is not True: - raise ImmutableRendererDiagnosticError(f"{attempt.key} sample lost non-canonical marker") - expected_task = { - "task_id": attempt.task.task_id, - "level": attempt.task.level, - "representation_id": attempt.task.representation_id, - "image_sha256": attempt.task.observation.image_sha256, - "target_image_sha256": attempt.task.reference.target_image_sha256, - } - expected_request = { - "num_samples": 1, - "attempt_index": 1, - "seed": attempt.seed, - "max_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "prompt_tokens": attempt.prepared.prompt_tokens, - "prompt_text_sha256": attempt.prepared.prompt_text_sha256, - } - if payload.get("task") != expected_task or payload.get("request") != expected_request: - raise ImmutableRendererDiagnosticError(f"{attempt.key} sample request changed") - _validate_response(payload.get("response"), attempt=attempt) - - -def _evaluation_payload( - *, - attempt: PreparedAttempt, - result: EvaluationResult, - sample_record_sha256: str, - manifest_record_sha256: str, - extracted_program_sha256: str, -) -> dict[str, Any]: - payload = checkpoint_v1._evaluation_payload( - task=attempt.task, - result=result, - sample_record_sha256=sample_record_sha256, - manifest_record_sha256=manifest_record_sha256, - ) - if result.status is not EvaluationStatus.OK: - payload["raw_absolute_scale_iou"] = 0.0 - payload["dice"] = None - payload["pure_executable"] = False - try: - normalized = json.loads(_canonical_bytes(payload)) - except RendererDiagnosticError as exc: - raise checkpoint_v1.EvaluationInfrastructureFault( - f"{attempt.key}: evaluator returned non-finite diagnostics" - ) from exc - normalized["diagnostic_manifest_record_sha256"] = normalized.pop( - "evaluation_manifest_record_sha256" - ) - normalized.update( - { - **_not_canonical(), - "diagnostic_id": DIAGNOSTIC_ID, - "arm_id": attempt.arm.spec.arm_id, - "checkpoint_name": attempt.arm.spec.checkpoint_name, - "renderer": attempt.arm.spec.renderer, - "seed": attempt.seed, - "extracted_program_sha256": extracted_program_sha256, - } - ) - return normalized - - -def _validate_evaluation_record( - *, - record: Mapping[str, Any], - attempt: PreparedAttempt, - sample_record_sha256: str, - manifest_record_sha256: str, - extracted_program_sha256: str, -) -> None: - payload = record["payload"] - expected = { - "diagnostic_manifest_record_sha256": manifest_record_sha256, - "sample_record_sha256": sample_record_sha256, - "diagnostic_id": DIAGNOSTIC_ID, - "arm_id": attempt.arm.spec.arm_id, - "checkpoint_name": attempt.arm.spec.checkpoint_name, - "renderer": attempt.arm.spec.renderer, - "task_id": attempt.task.task_id, - "level": attempt.task.level, - "representation_id": attempt.task.representation_id, - "seed": attempt.seed, - "extracted_program_sha256": extracted_program_sha256, - "attribution": Attribution.MODEL.value, - "not_canonical_evidence": True, - } - for field, value in expected.items(): - if payload.get(field) != value: - raise ImmutableRendererDiagnosticError(f"{attempt.key} evaluation record changed") - iou = payload.get("raw_absolute_scale_iou") - latency = payload.get("latency_seconds") - if ( - isinstance(iou, bool) - or not isinstance(iou, (int, float)) - or not math.isfinite(float(iou)) - or not 0.0 <= float(iou) <= 1.0 - or isinstance(latency, bool) - or not isinstance(latency, (int, float)) - or not math.isfinite(float(latency)) - or float(latency) < 0.0 - ): - raise ImmutableRendererDiagnosticError(f"{attempt.key} has invalid stored metrics") - status = payload.get("status") - pure = payload.get("pure_executable") - if ( - status not in {item.value for item in EvaluationStatus} - or not isinstance(pure, bool) - or pure is not (status == EvaluationStatus.OK.value) - or (not pure and float(iou) != 0.0) - ): - raise ImmutableRendererDiagnosticError(f"{attempt.key} has inconsistent executable status") - try: - checkpoint_v1._safe_json(payload) - _canonical_bytes(payload) - except ( - checkpoint_v1.EvaluationInfrastructureFault, - RendererDiagnosticError, - ) as exc: - raise ImmutableRendererDiagnosticError(f"{attempt.key} has non-finite diagnostics") from exc - - -async def _sample_missing( - *, - attempts: Sequence[PreparedAttempt], - clients: Mapping[str, Any], - store: RendererDiagnosticStore, - manifest_record_sha256: str, -) -> None: - import tinker - - semaphore = asyncio.Semaphore(SAMPLE_CONCURRENCY) - - async def sample_one(attempt: PreparedAttempt) -> None: - messages = [_message(attempt.task, max_image=MAX_IMAGE_LONG_EDGE)] - prompt_text = str(messages[0]["content"][1]["text"]) - if ( - hashlib.sha256(prompt_text.encode("utf-8")).hexdigest() - != attempt.prepared.prompt_text_sha256 - ): - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} prompt text changed after preflight" - ) - prompt = attempt.arm.renderer.build_generation_prompt(messages) - if int(prompt.length) != attempt.prepared.prompt_tokens: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} prompt changed after preflight" - ) - async with semaphore: - try: - result = await clients[attempt.arm.spec.arm_id].sample_async( - prompt=prompt, - num_samples=1, - sampling_params=tinker.SamplingParams( - stop=attempt.arm.renderer.get_stop_sequences(), - max_tokens=MAX_OUTPUT_TOKENS, - temperature=TEMPERATURE, - top_p=TOP_P, - seed=attempt.seed, - ), - ) - except Exception as exc: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} sampling failed: {exc}" - ) from exc - sequences = list(result.sequences) - if len(sequences) != 1: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} returned {len(sequences)} samples, expected 1" - ) - response = _decode_response(attempt.arm.renderer, sequences[0]) - store.create_or_verify_sample( - attempt.arm.spec.arm_id, - attempt.task.task_id, - _sample_payload( - attempt=attempt, - response=response, - manifest_record_sha256=manifest_record_sha256, - ), - ) - if not response["token_decode_complete"]: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} raw token IDs were persisted but could not be " - f"decoded: {response['token_decode_error']}" - ) - - tasks = [asyncio.create_task(sample_one(attempt)) for attempt in attempts] - try: - outcomes = await asyncio.gather(*tasks, return_exceptions=True) - except BaseException: - for task in tasks: - if not task.done(): - task.cancel() - await asyncio.gather(*tasks, return_exceptions=True) - raise - faults = [outcome for outcome in outcomes if isinstance(outcome, BaseException)] - if faults: - raise faults[0] - - -def _score_missing( - *, - attempts: Sequence[PreparedAttempt], - evaluator: PixCellEvaluator, - store: RendererDiagnosticStore, - manifest_record_sha256: str, -) -> None: - pending: list[tuple[PreparedAttempt, dict[str, Any]]] = [] - for attempt in attempts: - sample = store.load_sample( - attempt.arm.spec.arm_id, - attempt.task.task_id, - ) - if sample is None: - raise RendererDiagnosticError(f"{attempt.key} has no resumable sample") - response = sample["payload"]["response"] - if not response["token_decode_complete"]: - raise checkpoint_v1.SamplingInfrastructureFault( - f"{attempt.key} has undecodable persisted tokens" - ) - existing = store.load_evaluation( - attempt.arm.spec.arm_id, - attempt.task.task_id, - ) - if existing is not None: - _validate_evaluation_record( - record=existing, - attempt=attempt, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - extracted_program_sha256=response["extracted_program_sha256"], - ) - continue - pending.append((attempt, sample)) - - for offset in range(0, len(pending), EVALUATION_BATCH_SIZE): - batch = pending[offset : offset + EVALUATION_BATCH_SIZE] - requests = [ - ( - attempt.task.reference, - str(sample["payload"]["response"]["extracted_program"]), - ) - for attempt, sample in batch - ] - try: - results = evaluator.evaluate_batch(requests) - except Exception as exc: - raise checkpoint_v1.EvaluationInfrastructureFault( - f"isolated renderer-diagnostic batch failed: {exc}" - ) from exc - if len(results) != len(batch): - raise checkpoint_v1.EvaluationInfrastructureFault( - "isolated evaluator returned a different result count" - ) - faults: list[checkpoint_v1.CheckpointEvaluationError] = [] - for (attempt, sample), result in zip(batch, results, strict=True): - response = sample["payload"]["response"] - try: - payload = _evaluation_payload( - attempt=attempt, - result=result, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - extracted_program_sha256=response["extracted_program_sha256"], - ) - except ( - checkpoint_v1.ReferencePanelFault, - checkpoint_v1.EvaluationInfrastructureFault, - ) as exc: - faults.append(exc) - continue - store.create_or_verify_evaluation( - attempt.arm.spec.arm_id, - attempt.task.task_id, - payload, - ) - if faults: - raise faults[0] - - -def _validate_attempt_state( - *, - attempts: Sequence[PreparedAttempt], - store: RendererDiagnosticStore, - manifest_record_sha256: str, -) -> tuple[list[PreparedAttempt], list[PreparedAttempt]]: - store.assert_attempt_inventory(attempts) - missing_samples: list[PreparedAttempt] = [] - missing_evaluations: list[PreparedAttempt] = [] - for attempt in attempts: - sample = store.load_sample( - attempt.arm.spec.arm_id, - attempt.task.task_id, - ) - evaluation = store.load_evaluation( - attempt.arm.spec.arm_id, - attempt.task.task_id, - ) - if sample is None: - if evaluation is not None: - raise ImmutableRendererDiagnosticError( - f"{attempt.key} has evaluation without raw sample" - ) - missing_samples.append(attempt) - missing_evaluations.append(attempt) - continue - _validate_sample_record( - record=sample, - attempt=attempt, - manifest_record_sha256=manifest_record_sha256, - ) - response = sample["payload"]["response"] - if not response["token_decode_complete"] and evaluation is not None: - raise ImmutableRendererDiagnosticError(f"{attempt.key} evaluates undecodable tokens") - if evaluation is None: - missing_evaluations.append(attempt) - continue - _validate_evaluation_record( - record=evaluation, - attempt=attempt, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_record_sha256, - extracted_program_sha256=response["extracted_program_sha256"], - ) - return missing_samples, missing_evaluations - - -def _arm_summary(records: Sequence[Mapping[str, Any]]) -> dict[str, Any]: - if len(records) != len(EXPECTED_TASK_IDS): - raise RendererDiagnosticError("diagnostic arm is not exact F1-F8") - ious = [float(record["raw_absolute_scale_iou"]) for record in records] - pure = [bool(record["pure_executable"]) for record in records] - return { - "tasks": len(records), - "mean_raw_absolute_scale_iou": sum(ious) / len(ious), - "pure_executable": sum(pure), - "pure_executable_rate": sum(pure) / len(pure), - "cap_hits": sum(bool(record["cap_hit"]) for record in records), - "channel_parse_complete": sum(bool(record["channel_parse_complete"]) for record in records), - "statuses": dict(sorted(Counter(str(record["status"]) for record in records).items())), - } - - -def _report_payload( - *, - attempts: Sequence[PreparedAttempt], - store: RendererDiagnosticStore, - manifest_record: Mapping[str, Any], -) -> dict[str, Any]: - records: list[dict[str, Any]] = [] - by_arm: dict[str, list[dict[str, Any]]] = {spec.arm_id: [] for spec in ARM_SPECS} - manifest_sha = str(manifest_record["record_sha256"]) - for attempt in attempts: - sample = store.load_sample( - attempt.arm.spec.arm_id, - attempt.task.task_id, - ) - evaluation = store.load_evaluation( - attempt.arm.spec.arm_id, - attempt.task.task_id, - ) - if sample is None or evaluation is None: - raise RendererDiagnosticError(f"{attempt.key} is incomplete") - response = sample["payload"]["response"] - measured = evaluation["payload"] - _validate_sample_record( - record=sample, - attempt=attempt, - manifest_record_sha256=manifest_sha, - ) - _validate_evaluation_record( - record=evaluation, - attempt=attempt, - sample_record_sha256=str(sample["record_sha256"]), - manifest_record_sha256=manifest_sha, - extracted_program_sha256=response["extracted_program_sha256"], - ) - row = { - "arm_id": attempt.arm.spec.arm_id, - "checkpoint_name": attempt.arm.spec.checkpoint_name, - "renderer": attempt.arm.spec.renderer, - "task_id": attempt.task.task_id, - "seed": attempt.seed, - "status": measured["status"], - "pure_executable": measured["pure_executable"], - "raw_absolute_scale_iou": measured["raw_absolute_scale_iou"], - "completion_tokens": response["completion_tokens"], - "stop_reason": response["stop_reason"], - "cap_hit": response["cap_hit"], - "channel_parse_status": response["channel_parse_status"], - "channel_parse_complete": response["channel_parse_complete"], - "raw_text_sha256": response["raw_text_sha256"], - "reasoning_text_sha256": response["reasoning_text_sha256"], - "answer_text_sha256": response["answer_text_sha256"], - "extracted_program_sha256": response["extracted_program_sha256"], - "sample_record_sha256": sample["record_sha256"], - "evaluation_record_sha256": evaluation["record_sha256"], - } - records.append(row) - by_arm[attempt.arm.spec.arm_id].append(row) - return { - **_not_canonical(), - "schema_version": DIAGNOSTIC_SCHEMA_VERSION, - "diagnostic_id": DIAGNOSTIC_ID, - "study_id": STUDY_ID, - "producer_stage_id": PRODUCER_STAGE_ID, - "producer_wave": PRODUCER_WAVE, - "producer_replicate_id": PRODUCER_REPLICATE_ID, - "matrix": [spec.as_dict() for spec in ARM_SPECS], - "summaries": {arm_id: _arm_summary(values) for arm_id, values in by_arm.items()}, - "records": records, - "record_set_sha256": canonical_json_sha256(records), - "provenance": { - "diagnostic_manifest_record_sha256": manifest_sha, - "manifest": manifest_record["payload"], - }, - } - - -def _manifest_payload( - *, - repo_root: Path, - protocol: Mapping[str, Any], - evaluator_source_git_sha: str, - source_provenance: Mapping[str, Any], - producer_training: Mapping[str, Any], - dataset: Mapping[str, Any], - task_panel: Mapping[str, Any], - model: Mapping[str, Any], - prefix_audit: Mapping[str, Any], - runtime: Mapping[str, Any], - sandbox: Mapping[str, Any], - arms: Sequence[PreparedArm], - attempts: Sequence[PreparedAttempt], -) -> dict[str, Any]: - arm_preflights = { - arm.spec.arm_id: { - "spec": arm.spec.as_dict(), - "checkpoint": arm.binding.checkpoint, - "sampler_path": arm.binding.sampler_path, - "prompt": arm.preflight, - "prefix_compatibility": prefix_audit["renderers"][arm.spec.renderer], - } - for arm in arms - } - seed_inventory = [ - { - "arm_id": attempt.arm.spec.arm_id, - "checkpoint_name": attempt.arm.spec.checkpoint_name, - "renderer": attempt.arm.spec.renderer, - "task_id": attempt.task.task_id, - "seed": attempt.seed, - } - for attempt in attempts - ] - return { - **_not_canonical(), - "schema_version": DIAGNOSTIC_SCHEMA_VERSION, - "diagnostic_id": DIAGNOSTIC_ID, - "study_id": STUDY_ID, - "source": dict(source_provenance), - "protocol": { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(repo_root)), - "contract_version": protocol["contract_version"], - "preserved_unchanged": True, - }, - "dataset": { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": "fixed-f1-f8", - **dict(dataset), - }, - "producer_training": dict(producer_training), - "task_panel": dict(task_panel), - "prompt_assets": prompt_asset_hashes(), - "model": { - **dict(model), - "producer_training_renderer": model["renderer"], - }, - "sft_prefix_audit": dict(prefix_audit), - "matrix": arm_preflights, - "runtime": dict(runtime), - "sandbox": dict(sandbox), - "sampling": { - "task_count": len(EXPECTED_TASK_IDS), - "arms": len(ARM_SPECS), - "total_samples": EXPECTED_SAMPLE_COUNT, - "attempts_per_arm_task": 1, - "max_output_tokens": MAX_OUTPUT_TOKENS, - "temperature": TEMPERATURE, - "top_p": TOP_P, - "max_image_long_edge": MAX_IMAGE_LONG_EDGE, - "sample_concurrency": SAMPLE_CONCURRENCY, - "evaluation_batch_size": EVALUATION_BATCH_SIZE, - "evaluator_workers": EVALUATOR_WORKERS, - "seed_domain": _SEED_DOMAIN, - "seed_algorithm": "sha256-bound deterministic 31-bit seed", - "seeds": seed_inventory, - "seeds_sha256": canonical_json_sha256(seed_inventory), - }, - "evaluator_source_git_sha": evaluator_source_git_sha, - } - - -def _runtime_binding(repo_root: Path) -> dict[str, Any]: - runtime = validate_runtime_stack(repo_root) - return { - **runtime, - "cookbook_source": _cookbook_source_binding(runtime), - } - - -def _default_evaluator_factory() -> PixCellEvaluator: - return PixCellEvaluator( - max_workers=EVALUATOR_WORKERS, - evaluator_retries=1, - require_isolation=True, - ) - - -def _default_service_client_factory() -> Any: - import tinker - - return tinker.ServiceClient() - - -async def run_renderer_diagnostic( - *, - repo_root: Path, - evaluator_source_git_sha: str, - producer_wave_receipt_record_sha256: str, - external_root: Path, - confirmation: str, - service_client_factory: Callable[[], Any] = _default_service_client_factory, - evaluator_factory: Callable[[], PixCellEvaluator] = _default_evaluator_factory, -) -> dict[str, Any]: - """Run or resume the exact, diagnostic-only 32-sample matrix.""" - - root = repo_root.expanduser().resolve(strict=True) - protocol = load_protocol(root, require_committed=True) - current_source = validate_source_sha(root, evaluator_source_git_sha) - if confirmation != protocol["launch"]["confirmation_token"]: - raise RendererDiagnosticError("explicit diagnostic spend confirmation is missing") - fixed = _validate_fixed_protocol(protocol) - training_store = TrainingStore(repo_root=root, external_root=external_root) - bindings = _load_matrix_bindings( - store=training_store, - protocol=protocol, - expected_wave_receipt_sha256=producer_wave_receipt_record_sha256, - ) - source_provenance = _source_provenance( - repo_root=root, - evaluator_source_git_sha=current_source, - ) - producer_training = _producer_training_provenance( - repo_root=root, - protocol=protocol, - bindings=bindings, - ) - dataset = _dataset_binding(root / "dataset", protocol) - runtime = _runtime_binding(root) - tasks = checkpoint_v1._benchmark_tasks(repo_root=root, protocol=protocol) - task_panel = _task_panel(tasks) - prefix_audit = _sft_prefix_audit( - dataset_root=root / "dataset", - protocol=protocol, - ) - for spec in ARM_SPECS: - observed = prefix_audit["renderers"][spec.renderer]["compatible_with_sft_supervised_prefix"] - if observed is not spec.expected_sft_prefix_compatible: - raise RendererDiagnosticError(f"{spec.arm_id} renderer compatibility claim changed") - - evaluator = evaluator_factory() - try: - sandbox = checkpoint_v1._sandbox_provenance(evaluator) - arms, attempts = _prepare_arms( - protocol=protocol, - model=fixed["model"], - bindings=bindings, - tasks=tasks, - evaluator=evaluator, - evaluator_source_git_sha=current_source, - dataset_sha256=str(dataset["logical_release_sha256"]), - task_panel_sha256=str(task_panel["logical_sha256"]), - ) - store = RendererDiagnosticStore( - stage_path=training_store.stage_path(bindings[0].key), - ) - with store.acquire_lock(): - manifest_payload = _manifest_payload( - repo_root=root, - protocol=protocol, - evaluator_source_git_sha=current_source, - source_provenance=source_provenance, - producer_training=producer_training, - dataset=dataset, - task_panel=task_panel, - model=fixed["model"], - prefix_audit=prefix_audit, - runtime=runtime, - sandbox=sandbox, - arms=arms, - attempts=attempts, - ) - manifest = store.create_or_verify_manifest(manifest_payload) - missing_samples, missing_evaluations = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - existing_report = store.load_report() - if existing_report is not None: - if missing_samples or missing_evaluations: - raise RendererDiagnosticError( - "diagnostic report exists with incomplete matrix rows" - ) - expected = _report_payload( - attempts=attempts, - store=store, - manifest_record=manifest, - ) - observed = store.create_or_verify_report(expected) - _validate_report_rows( - observed, - manifest_record=manifest, - attempts=attempts, - ) - return { - "status": "already_complete", - "diagnostic_id": DIAGNOSTIC_ID, - "not_canonical_evidence": True, - "report": str(store.root / "report.json"), - "report_record_sha256": observed["record_sha256"], - "summaries": observed["payload"]["summaries"], - } - - if missing_samples: - if not os.environ.get("TINKER_API_KEY"): - raise RendererDiagnosticError("TINKER_API_KEY is not present") - # Revalidate the exact current source, explicit spend token, - # producer receipt, existing rows, and matrix immediately - # before the first paid-capable client exists. - validate_source_sha(root, current_source) - if confirmation != protocol["launch"]["confirmation_token"]: - raise RendererDiagnosticError( - "diagnostic spend confirmation changed before client" - ) - rebound = _load_matrix_bindings( - store=training_store, - protocol=protocol, - expected_wave_receipt_sha256=(producer_wave_receipt_record_sha256), - ) - if [ - ( - binding.checkpoint, - binding.checkpoint_inventory_sha256, - binding.sampler_path, - binding.wave_receipt.record_sha256, - ) - for binding in rebound - ] != [ - ( - arm.binding.checkpoint, - arm.binding.checkpoint_inventory_sha256, - arm.binding.sampler_path, - arm.binding.wave_receipt.record_sha256, - ) - for arm in arms - ]: - raise RendererDiagnosticError("producer receipt changed before client creation") - missing_samples, _ = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - if not missing_samples: - raise RendererDiagnosticError( - "diagnostic resume state changed before client creation" - ) - service = service_client_factory() - clients = { - arm.spec.arm_id: service.create_sampling_client( - model_path=arm.binding.sampler_path - ) - for arm in arms - if any( - attempt.arm.spec.arm_id == arm.spec.arm_id for attempt in missing_samples - ) - } - await _sample_missing( - attempts=missing_samples, - clients=clients, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - - missing_samples, _ = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - if missing_samples: - raise RendererDiagnosticError( - "diagnostic sampling did not persist all raw responses" - ) - _score_missing( - attempts=attempts, - evaluator=evaluator, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - missing_samples, missing_evaluations = _validate_attempt_state( - attempts=attempts, - store=store, - manifest_record_sha256=str(manifest["record_sha256"]), - ) - if missing_samples or missing_evaluations: - raise RendererDiagnosticError( - "diagnostic is incomplete after deterministic evaluation" - ) - report_payload = _report_payload( - attempts=attempts, - store=store, - manifest_record=manifest, - ) - report = store.create_or_verify_report(report_payload) - _validate_report_rows( - report, - manifest_record=manifest, - attempts=attempts, - ) - return { - "status": "complete", - "diagnostic_id": DIAGNOSTIC_ID, - "not_canonical_evidence": True, - "report": str(store.root / "report.json"), - "report_record_sha256": report["record_sha256"], - "summaries": report_payload["summaries"], - } - finally: - evaluator.close() - - -def _validate_report_rows( - record: Mapping[str, Any], - *, - manifest_record: Mapping[str, Any], - attempts: Sequence[PreparedAttempt], -) -> dict[str, Any]: - """Validate the report without granting it a canonical report interface.""" - - validated = _validate_record( - record, - record_type="renderer_diagnostic_report", - key={}, - ) - payload = validated["payload"] - if ( - payload.get("schema_version") != DIAGNOSTIC_SCHEMA_VERSION - or payload.get("diagnostic_id") != DIAGNOSTIC_ID - or payload.get("not_canonical_evidence") is not True - or payload.get("promotion_eligibility") != {"eligible": False, "policy": _PROMOTION_POLICY} - or payload.get("matrix") != [spec.as_dict() for spec in ARM_SPECS] - ): - raise RendererDiagnosticError("diagnostic report identity changed") - provenance = payload.get("provenance") - records = payload.get("records") - if ( - not isinstance(provenance, Mapping) - or not isinstance(records, list) - or provenance.get("diagnostic_manifest_record_sha256") != manifest_record["record_sha256"] - or provenance.get("manifest") != manifest_record["payload"] - ): - raise RendererDiagnosticError("diagnostic report provenance changed") - expected_keys = [attempt.key for attempt in attempts] - observed_keys = [ - (str(row.get("arm_id")), str(row.get("task_id"))) - for row in records - if isinstance(row, Mapping) - ] - if ( - len(records) != EXPECTED_SAMPLE_COUNT - or observed_keys != expected_keys - or len(set(observed_keys)) != EXPECTED_SAMPLE_COUNT - ): - raise RendererDiagnosticError("diagnostic report has an incomplete or reordered matrix") - for row, attempt in zip(records, attempts, strict=True): - if not isinstance(row, Mapping): - raise RendererDiagnosticError("diagnostic report row is not an object") - expected = { - "arm_id": attempt.arm.spec.arm_id, - "checkpoint_name": attempt.arm.spec.checkpoint_name, - "renderer": attempt.arm.spec.renderer, - "task_id": attempt.task.task_id, - "seed": attempt.seed, - } - for field, value in expected.items(): - if row.get(field) != value: - raise RendererDiagnosticError(f"{attempt.key} report identity changed") - iou = row.get("raw_absolute_scale_iou") - completion_tokens = row.get("completion_tokens") - status = row.get("status") - pure = row.get("pure_executable") - cap_hit = row.get("cap_hit") - stop_reason = row.get("stop_reason") - parse_status = row.get("channel_parse_status") - parse_complete = row.get("channel_parse_complete") - expected_cap = ( - isinstance(completion_tokens, int) - and not isinstance(completion_tokens, bool) - and ( - completion_tokens >= MAX_OUTPUT_TOKENS - or ( - isinstance(stop_reason, str) - and ("length" in stop_reason.lower() or "max_token" in stop_reason.lower()) - ) - ) - ) - if ( - isinstance(iou, bool) - or not isinstance(iou, (int, float)) - or not math.isfinite(float(iou)) - or not 0.0 <= float(iou) <= 1.0 - or status not in {item.value for item in EvaluationStatus} - or not isinstance(pure, bool) - or pure is not (status == EvaluationStatus.OK.value) - or (not pure and float(iou) != 0.0) - or isinstance(completion_tokens, bool) - or not isinstance(completion_tokens, int) - or completion_tokens < 0 - or not isinstance(stop_reason, str) - or not isinstance(cap_hit, bool) - or cap_hit is not expected_cap - or parse_status not in {"clean", "unclean", "error", "token_decode_error"} - or not isinstance(parse_complete, bool) - or parse_complete is not (parse_status == "clean") - ): - raise RendererDiagnosticError(f"{attempt.key} report diagnostics are inconsistent") - for field in ( - "raw_text_sha256", - "reasoning_text_sha256", - "answer_text_sha256", - "extracted_program_sha256", - "sample_record_sha256", - "evaluation_record_sha256", - ): - if not checkpoint_v1._SHA256.fullmatch(str(row.get(field, ""))): - raise RendererDiagnosticError(f"{attempt.key} report {field} is not a SHA-256") - by_arm = { - spec.arm_id: [row for row in records if row["arm_id"] == spec.arm_id] for spec in ARM_SPECS - } - recomputed = {arm_id: _arm_summary(rows) for arm_id, rows in by_arm.items()} - if payload.get("summaries") != recomputed: - raise RendererDiagnosticError("diagnostic claimed metrics differ from its rows") - if payload.get("record_set_sha256") != canonical_json_sha256(records): - raise RendererDiagnosticError("diagnostic row-set digest changed") - return validated - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--repo-root", type=Path, required=True) - result.add_argument("--evaluator-source-git-sha", required=True) - result.add_argument( - "--producer-wave-receipt-record-sha256", - required=True, - ) - result.add_argument("--external-root", type=Path, required=True) - result.add_argument("--confirm-spend", default="") - return result - - -def main() -> None: - args = parser().parse_args() - result = asyncio.run( - run_renderer_diagnostic( - repo_root=args.repo_root, - evaluator_source_git_sha=args.evaluator_source_git_sha, - producer_wave_receipt_record_sha256=(args.producer_wave_receipt_record_sha256), - external_root=args.external_root, - confirmation=args.confirm_spend, - ) - ) - print(json.dumps(result, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/run_stage.py b/rl/studies/representation_training_v1/run_stage.py deleted file mode 100644 index 1bca69f3..00000000 --- a/rl/studies/representation_training_v1/run_stage.py +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env python3 -"""Run one sealed representation-training stage or continuation wave.""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from pathlib import Path - -from .launcher import run_training_stage -from .protocol import repository_root - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--stage", required=True) - result.add_argument("--wave", required=True) - result.add_argument("--replicate", default="r0") - result.add_argument("--expected-source-sha", required=True) - result.add_argument("--confirm-spend", default="") - result.add_argument( - "--external-root", - type=Path, - default=( - Path(os.environ["PIXCELL_TRAINING_ROOT"]) - if os.environ.get("PIXCELL_TRAINING_ROOT") - else None - ), - ) - return result - - -def main() -> None: - args = parser().parse_args() - if args.external_root is None: - raise SystemExit( - "set PIXCELL_TRAINING_ROOT or pass --external-root outside Git" - ) - report = asyncio.run( - run_training_stage( - repo_root=repository_root(), - stage_id=args.stage, - wave_name=args.wave, - replicate_id=args.replicate, - expected_source_sha=args.expected_source_sha, - external_root=args.external_root, - confirmation=args.confirm_spend, - ) - ) - print(json.dumps(report, indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/rl/studies/representation_training_v1/schedule.py b/rl/studies/representation_training_v1/schedule.py deleted file mode 100644 index a663fbda..00000000 --- a/rl/studies/representation_training_v1/schedule.py +++ /dev/null @@ -1,161 +0,0 @@ -"""Deterministic task selection and schedule digests for training stages.""" - -from __future__ import annotations - -import hashlib -import json -import math -from collections import Counter -from pathlib import Path -from typing import Any - -from rl.common.contracts import TaskRecord -from rl.common.dataset_io import load_tasks -from rl.track_a.curriculum import deterministic_rl_batch, stratified_physical_pass - -from .protocol import StageSpec - - -SCHEDULE_SCHEMA_VERSION = "pixcell-training-schedule-v1" -SCHEDULE_SEED = 90210 - - -def canonical_sha256(value: Any) -> str: - raw = json.dumps( - value, - allow_nan=False, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - return hashlib.sha256(raw).hexdigest() - - -def _select(tasks: list[TaskRecord], levels: tuple[str, ...]) -> list[TaskRecord]: - allowed = set(levels) - selected = [task for task in tasks if task.sampler.level.upper() in allowed] - if not selected: - raise ValueError(f"no depth/train rows match levels={levels}") - return selected - - -def stage_tasks(dataset_root: Path, stage: StageSpec) -> list[TaskRecord]: - all_train = load_tasks( - dataset_root.expanduser().resolve(), - configuration="depth", - split="train", - ) - levels = stage.levels if stage.kind == "sft" else (str(stage.current_level),) - return _select(all_train, levels) - - -def stage_replay_tasks( - dataset_root: Path, - stage: StageSpec, -) -> list[TaskRecord]: - if stage.kind != "rl" or not stage.replay_levels: - return [] - all_train = load_tasks( - dataset_root.expanduser().resolve(), - configuration="depth", - split="train", - ) - return _select(all_train, stage.replay_levels) - - -def ordered_stage_tasks(dataset_root: Path, stage: StageSpec) -> list[TaskRecord]: - return stratified_physical_pass( - stage_tasks(dataset_root, stage), - seed=SCHEDULE_SEED, - ) - - -def stage_schedule_document( - dataset_root: Path, - stage: StageSpec, - *, - groups_per_batch: int | None = None, -) -> dict[str, Any]: - tasks = ( - ordered_stage_tasks(dataset_root, stage) - if stage.kind == "sft" - else stage_tasks(dataset_root, stage) - ) - rows_by_level = Counter(task.sampler.level.upper() for task in tasks) - representations = { - task.sampler.representation_id - for task in tasks - } - slots = Counter( - str(task.sampler.realization_slot) - for task in tasks - ) - document: dict[str, Any] = { - "schema_version": SCHEDULE_SCHEMA_VERSION, - "stage_id": stage.stage_id, - "kind": stage.kind, - "seed": SCHEDULE_SEED, - "rows": len(tasks), - "rows_by_level": dict(sorted(rows_by_level.items())), - "representations": len(representations), - "realization_slots": dict(sorted(slots.items())), - "ordered_task_ids": [task.sampler.opaque_id for task in tasks], - } - if stage.kind == "sft": - document["batch_size"] = 64 - document["batches"] = math.ceil(len(tasks) / 64) - else: - if groups_per_batch is None or groups_per_batch < 1: - raise ValueError("RL schedule requires groups_per_batch") - document["groups_per_batch"] = groups_per_batch - document["coverage_steps"] = math.ceil(len(tasks) / groups_per_batch) - replay = stage_replay_tasks(dataset_root, stage) - maximum_steps = max( - int(wave["max_steps"]) for wave in stage.waves.values() - ) - training_batches: list[dict[str, Any]] = [] - for step in range(maximum_steps): - selected = deterministic_rl_batch( - tasks, - replay, - step=step, - groups_per_batch=groups_per_batch, - schedule_seed=SCHEDULE_SEED, - ) - ids = [task.sampler.opaque_id for _role, task in selected] - if len(ids) != len(set(ids)): - raise RuntimeError(f"{stage.stage_id} step {step} repeats a task") - training_batches.append( - { - "step": step, - "groups": [ - { - "role": role, - "task_id": task.sampler.opaque_id, - "level": task.sampler.level.upper(), - } - for role, task in selected - ], - } - ) - document["maximum_steps"] = maximum_steps - document["replay_pool_rows"] = len(replay) - document["replay_levels"] = list(stage.replay_levels) - document["training_batches"] = training_batches - document["logical_sha256"] = canonical_sha256(document) - return document - - -def verify_sft_step_count( - dataset_root: Path, - stage: StageSpec, - *, - max_steps: int, -) -> None: - schedule = stage_schedule_document(dataset_root, stage) - expected = int(schedule["batches"]) - if max_steps != expected: - raise ValueError( - f"{stage.stage_id} must visit one complete physical pass: " - f"max_steps={expected}, got {max_steps}" - ) diff --git a/rl/studies/representation_training_v1/source_transition.py b/rl/studies/representation_training_v1/source_transition.py deleted file mode 100644 index cb86eae5..00000000 --- a/rl/studies/representation_training_v1/source_transition.py +++ /dev/null @@ -1,433 +0,0 @@ -"""Deterministic Git facts for an explicitly reviewed source transition. - -Evaluation evidence is produced and validated at one clean commit. A later -consumer commit may use that evidence only when the producer is an exact -ancestor and an immutable approval records the complete transition. -""" - -from __future__ import annotations - -import os -import re -import subprocess -from collections.abc import Mapping, Sequence -from pathlib import Path, PurePosixPath -from typing import Any - -from .store import canonical_sha256 - - -SOURCE_TRANSITION_SCHEMA_VERSION = "pixcell-training-source-transition-v2" -_GIT_SHA = re.compile(r"^[0-9a-f]{40}$") -_GIT_MODE = re.compile(r"^[0-7]{6}$") -_GIT_STATUS = re.compile(r"^[A-Z][0-9]*$") - - -class SourceTransitionError(RuntimeError): - """A producer-to-consumer source transition is invalid or unverifiable.""" - - -def _sha(value: Any, field: str) -> str: - result = str(value) - if not _GIT_SHA.fullmatch(result): - raise SourceTransitionError(f"{field} must be a full lowercase Git SHA") - return result - - -def _git(repo_root: Path, *args: str) -> bytes: - # These are local object-database queries. Inheriting arbitrary GIT_* or - # user configuration makes even a commit-to-commit diff dependent on the - # invoking shell (for example GIT_DIFF_OPTS changes patch serialization). - # Keep only executable lookup and force Git's replace/config surfaces off. - environment = { - "GIT_ATTR_NOSYSTEM": "1", - "GIT_CONFIG_GLOBAL": os.devnull, - "GIT_CONFIG_NOSYSTEM": "1", - "GIT_NO_REPLACE_OBJECTS": "1", - "GIT_OPTIONAL_LOCKS": "0", - "LANG": "C", - "LC_ALL": "C", - "PATH": os.environ.get("PATH", os.defpath), - } - try: - result = subprocess.run( - [ - "git", - "-c", - "core.quotepath=false", - "-c", - "diff.external=", - "-c", - "diff.noprefix=false", - "-c", - "core.attributesFile=/dev/null", - *args, - ], - cwd=repo_root, - check=True, - env=environment, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - except (OSError, subprocess.CalledProcessError) as exc: - detail = "" - if isinstance(exc, subprocess.CalledProcessError): - detail = exc.stderr.decode("utf-8", errors="replace").strip() - suffix = f": {detail}" if detail else "" - raise SourceTransitionError(f"git {' '.join(args)} failed{suffix}") from exc - return result.stdout - - -def _repository_root(repo_root: Path) -> Path: - root = repo_root.expanduser().resolve(strict=True) - observed = Path(_git(root, "rev-parse", "--show-toplevel").decode("utf-8").strip()).resolve( - strict=True - ) - if observed != root: - raise SourceTransitionError( - "source transition repository root is not the Git worktree root" - ) - grafts = Path(_git(root, "rev-parse", "--git-path", "info/grafts").decode("utf-8").strip()) - if not grafts.is_absolute(): - grafts = root / grafts - if grafts.exists(): - raise SourceTransitionError("source transition refuses a repository with legacy Git grafts") - return root - - -def _relative_path(value: Any, field: str) -> str: - if not isinstance(value, str): - raise SourceTransitionError(f"{field} must be a string path") - raw = value - path = PurePosixPath(raw) - if ( - not raw - or "\0" in raw - or any(ord(character) < 32 or ord(character) == 127 for character in raw) - or path.is_absolute() - or raw != path.as_posix() - or any(part in {"", ".", ".."} for part in path.parts) - ): - raise SourceTransitionError(f"{field} must be a normalized relative POSIX path") - return raw - - -def _commit(repo_root: Path, value: Any, field: str) -> str: - requested = _sha(value, field) - resolved = ( - _git( - repo_root, - "rev-parse", - "--verify", - f"{requested}^{{commit}}", - ) - .decode("ascii") - .strip() - ) - if resolved != requested: - raise SourceTransitionError(f"{field} does not resolve exactly") - return resolved - - -def _raw_changes(repo_root: Path, producer: str, consumer: str) -> list[dict[str, str]]: - raw = _git( - repo_root, - "diff-tree", - "--raw", - "-z", - "--no-abbrev", - "--full-index", - "--no-commit-id", - "-r", - "--no-renames", - producer, - consumer, - "--", - ) - parts = raw.split(b"\0") - if parts and parts[-1] == b"": - parts.pop() - if len(parts) % 2: - raise SourceTransitionError("git returned malformed raw changed-path data") - changes: list[dict[str, str]] = [] - for index in range(0, len(parts), 2): - try: - header = parts[index].decode("ascii") - path = parts[index + 1].decode("utf-8") - except UnicodeDecodeError as exc: - raise SourceTransitionError("Git returned a raw change that is not UTF-8") from exc - fields = header.split(" ") - if len(fields) != 5 or not fields[0].startswith(":"): - raise SourceTransitionError("git returned a malformed raw change header") - old_mode = fields[0][1:] - new_mode, old_object, new_object, status = fields[1:] - if ( - not _GIT_MODE.fullmatch(old_mode) - or not _GIT_MODE.fullmatch(new_mode) - or not _GIT_SHA.fullmatch(old_object) - or not _GIT_SHA.fullmatch(new_object) - or not _GIT_STATUS.fullmatch(status) - ): - raise SourceTransitionError("git returned invalid raw change fields") - changes.append( - { - "status": status, - "old_mode": old_mode, - "new_mode": new_mode, - "old_object": old_object, - "new_object": new_object, - "path": _relative_path(path, "changed path"), - } - ) - changes.sort( - key=lambda item: ( - item["path"], - item["status"], - item["old_mode"], - item["new_mode"], - item["old_object"], - item["new_object"], - ) - ) - return changes - - -def source_transition_git_facts( - *, - repo_root: Path, - producer_source_git_sha: str, - consumer_source_git_sha: str, -) -> dict[str, Any]: - """Return the exact, recomputable Git relationship between two commits.""" - - root = _repository_root(repo_root) - producer = _commit( - root, - producer_source_git_sha, - "producer_source_git_sha", - ) - consumer = _commit( - root, - consumer_source_git_sha, - "consumer_source_git_sha", - ) - if producer == consumer: - raise SourceTransitionError( - "a source transition requires different producer and consumer commits" - ) - merge_base = _git(root, "merge-base", producer, consumer).decode("ascii").strip() - if merge_base != producer: - raise SourceTransitionError("the evidence producer is not an ancestor of the consumer") - producer_tree = ( - _git( - root, - "rev-parse", - "--verify", - f"{producer}^{{tree}}", - ) - .decode("ascii") - .strip() - ) - consumer_tree = ( - _git( - root, - "rev-parse", - "--verify", - f"{consumer}^{{tree}}", - ) - .decode("ascii") - .strip() - ) - distance_text = ( - _git( - root, - "rev-list", - "--count", - f"{producer}..{consumer}", - ) - .decode("ascii") - .strip() - ) - try: - distance = int(distance_text) - except ValueError as exc: - raise SourceTransitionError("git returned an invalid ancestor distance") from exc - if distance < 1: - raise SourceTransitionError("source transition has no consumer commits") - - raw_changes = _raw_changes(root, producer, consumer) - changed_paths = [{"status": item["status"], "path": item["path"]} for item in raw_changes] - return { - "schema_version": "pixcell-training-source-transition-git-v1", - "producer_source_git_sha": producer, - "consumer_source_git_sha": consumer, - "merge_base_git_sha": merge_base, - "producer_tree_git_sha": producer_tree, - "consumer_tree_git_sha": consumer_tree, - "ancestor_distance": distance, - "changed_path_count": len(changed_paths), - "changed_paths": changed_paths, - "raw_changes": raw_changes, - "raw_change_inventory_sha256": canonical_sha256(raw_changes), - } - - -def approved_change_scope( - git_facts: Mapping[str, Any], - approved_change_paths: Sequence[str] | None, -) -> dict[str, Any]: - """Bind explicit human-reviewed paths to the exact committed Git diff.""" - - if approved_change_paths is None: - raise SourceTransitionError( - "cross-source approval requires an explicit approved change scope" - ) - if isinstance(approved_change_paths, (str, bytes)): - raise SourceTransitionError("approved change scope must be a sequence of paths") - approved = [_relative_path(value, "approved change path") for value in approved_change_paths] - if len(set(approved)) != len(approved): - raise SourceTransitionError("approved change scope repeats a path") - approved.sort() - raw_changes = git_facts.get("raw_changes") - if not isinstance(raw_changes, list): - raise SourceTransitionError("Git facts have no raw change inventory") - observed: list[dict[str, str]] = [] - required = { - "status", - "old_mode", - "new_mode", - "old_object", - "new_object", - "path", - } - for index, raw_entry in enumerate(raw_changes): - if not isinstance(raw_entry, Mapping): - raise SourceTransitionError(f"Git raw change {index} is not an object") - if set(raw_entry) != required: - raise SourceTransitionError(f"Git raw change {index} has foreign fields") - status = str(raw_entry.get("status", "")) - old_mode = str(raw_entry.get("old_mode", "")) - new_mode = str(raw_entry.get("new_mode", "")) - old_object = str(raw_entry.get("old_object", "")) - new_object = str(raw_entry.get("new_object", "")) - if ( - not _GIT_STATUS.fullmatch(status) - or not _GIT_MODE.fullmatch(old_mode) - or not _GIT_MODE.fullmatch(new_mode) - or not _GIT_SHA.fullmatch(old_object) - or not _GIT_SHA.fullmatch(new_object) - ): - raise SourceTransitionError(f"Git raw change {index} has invalid fields") - observed.append( - { - "status": status, - "old_mode": old_mode, - "new_mode": new_mode, - "old_object": old_object, - "new_object": new_object, - "path": _relative_path( - raw_entry.get("path"), - f"Git raw change {index}", - ), - } - ) - observed.sort( - key=lambda item: ( - item["path"], - item["status"], - item["old_mode"], - item["new_mode"], - item["old_object"], - item["new_object"], - ) - ) - observed_paths = [item["path"] for item in observed] - if approved != observed_paths: - raise SourceTransitionError( - "approved change paths differ from the exact producer-to-consumer diff" - ) - changed_paths = [{"status": item["status"], "path": item["path"]} for item in observed] - if ( - git_facts.get("changed_path_count") != len(observed) - or git_facts.get("changed_paths") != changed_paths - ): - raise SourceTransitionError("Git path projection differs from its raw change inventory") - inventory_sha256 = str(git_facts.get("raw_change_inventory_sha256", "")) - if not re.fullmatch(r"[0-9a-f]{64}", inventory_sha256) or inventory_sha256 != canonical_sha256( - observed - ): - raise SourceTransitionError("Git facts have an invalid raw change inventory digest") - scope = { - "schema_version": "pixcell-training-approved-change-scope-v1", - "policy": "exact-committed-path-status-mode-and-object-inventory", - "paths": approved, - "entries": observed, - "changed_path_count": len(observed), - "raw_change_inventory_sha256": inventory_sha256, - } - scope["logical_sha256"] = canonical_sha256(scope) - return scope - - -def build_source_transition( - *, - repo_root: Path, - producer_source_git_sha: str, - consumer_source_git_sha: str, - scope: Mapping[str, Any], - approved_change_paths: Sequence[str] | None, - equivalence: Mapping[str, Any], - evidence: Mapping[str, Any], -) -> dict[str, Any]: - """Build the canonical transition document embedded in a v2 approval.""" - - git_facts = source_transition_git_facts( - repo_root=repo_root, - producer_source_git_sha=producer_source_git_sha, - consumer_source_git_sha=consumer_source_git_sha, - ) - authorization = dict(scope) - authorization["approved_changes"] = approved_change_scope( - git_facts, - approved_change_paths, - ) - document: dict[str, Any] = { - "schema_version": SOURCE_TRANSITION_SCHEMA_VERSION, - "git": git_facts, - "scope": authorization, - "equivalence": dict(equivalence), - "evidence": dict(evidence), - } - document["logical_sha256"] = canonical_sha256(document) - return document - - -def validate_source_transition( - observed: Any, - *, - repo_root: Path, - producer_source_git_sha: str, - consumer_source_git_sha: str, - scope: Mapping[str, Any], - approved_change_paths: Sequence[str] | None, - equivalence: Mapping[str, Any], - evidence: Mapping[str, Any], -) -> dict[str, Any]: - """Rebuild and compare every transition field, including its Git diff.""" - - if not isinstance(observed, Mapping): - raise SourceTransitionError("source transition must be an object") - expected = build_source_transition( - repo_root=repo_root, - producer_source_git_sha=producer_source_git_sha, - consumer_source_git_sha=consumer_source_git_sha, - scope=scope, - approved_change_paths=approved_change_paths, - equivalence=equivalence, - evidence=evidence, - ) - if dict(observed) != expected: - raise SourceTransitionError( - "source transition differs from the current repository and evidence" - ) - return expected diff --git a/rl/studies/representation_training_v1/store.py b/rl/studies/representation_training_v1/store.py deleted file mode 100644 index ff818a75..00000000 --- a/rl/studies/representation_training_v1/store.py +++ /dev/null @@ -1,733 +0,0 @@ -"""External, immutable records for paid training stages.""" - -from __future__ import annotations - -import errno -import fcntl -import hashlib -import json -import os -import re -import stat -import threading -import uuid -from collections.abc import Mapping -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Any - - -RECORD_SCHEMA_VERSION = "pixcell-training-record-v1" -MAX_RECORD_BYTES = 16 * 1024 * 1024 -_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") -_SHA256 = re.compile(r"^[0-9a-f]{64}$") - - -class TrainingStoreError(RuntimeError): - """Training record storage failed closed.""" - - -class UnsafeTrainingRootError(TrainingStoreError): - """Training output overlaps the source repository.""" - - -class TrainingRunLockedError(TrainingStoreError): - """Another process owns this exact stage and replicate.""" - - -class ImmutableTrainingRecordError(TrainingStoreError): - """An existing record differs from the proposed immutable value.""" - - -@dataclass(frozen=True, order=True) -class StageKey: - study_id: str - stage_id: str - replicate_id: str - - def __post_init__(self) -> None: - for field, value in ( - ("study_id", self.study_id), - ("stage_id", self.stage_id), - ("replicate_id", self.replicate_id), - ): - if not _IDENTIFIER.fullmatch(value): - raise ValueError(f"{field} has an invalid identifier: {value!r}") - - def as_dict(self) -> dict[str, str]: - return { - "study_id": self.study_id, - "stage_id": self.stage_id, - "replicate_id": self.replicate_id, - } - - -@dataclass(frozen=True) -class TrainingRecord: - payload: dict[str, Any] - payload_sha256: str - record_sha256: str - relative_path: PurePosixPath - - -def canonical_bytes(value: Any) -> bytes: - try: - return json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - except (TypeError, ValueError) as exc: - raise TrainingStoreError("training record is not finite JSON") from exc - - -def canonical_sha256(value: Any) -> str: - return hashlib.sha256(canonical_bytes(value)).hexdigest() - - -def _normalized_object(value: Mapping[str, Any]) -> dict[str, Any]: - if not isinstance(value, Mapping): - raise TrainingStoreError("training record payload must be an object") - result = json.loads(canonical_bytes(value)) - if not isinstance(result, dict): - raise TrainingStoreError("training record payload must be an object") - return result - - -def _document( - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], -) -> dict[str, Any]: - normalized = _normalized_object(payload) - base = { - "schema_version": RECORD_SCHEMA_VERSION, - "record_type": record_type, - "key": dict(key), - "payload": normalized, - "payload_sha256": canonical_sha256(normalized), - } - base["record_sha256"] = canonical_sha256(base) - return base - - -def _validate_document( - value: Mapping[str, Any], - *, - record_type: str, - key: Mapping[str, Any], - relative_path: PurePosixPath, -) -> TrainingRecord: - required = { - "schema_version", - "record_type", - "key", - "payload", - "payload_sha256", - "record_sha256", - } - if set(value) != required: - raise TrainingStoreError("training record fields differ from the schema") - if value["schema_version"] != RECORD_SCHEMA_VERSION: - raise TrainingStoreError("training record has a foreign schema") - if value["record_type"] != record_type or value["key"] != dict(key): - raise TrainingStoreError("training record key or type differs from its path") - payload = value["payload"] - if not isinstance(payload, dict): - raise TrainingStoreError("training record payload is not an object") - payload_sha = canonical_sha256(payload) - if payload_sha != value["payload_sha256"] or not _SHA256.fullmatch(payload_sha): - raise TrainingStoreError("training record payload digest differs") - unsigned = dict(value) - record_sha = str(unsigned.pop("record_sha256")) - if canonical_sha256(unsigned) != record_sha: - raise TrainingStoreError("training record envelope digest differs") - return TrainingRecord( - payload=dict(payload), - payload_sha256=payload_sha, - record_sha256=record_sha, - relative_path=relative_path, - ) - - -class _StageLock: - def __init__(self, store: TrainingStore, key: StageKey, path: Path) -> None: - self.store = store - self.key = key - self.path = path - self.fd: int | None = None - - def __enter__(self) -> _StageLock: - self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - flags = os.O_RDWR | os.O_CREAT - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(self.path, flags, 0o600) - fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - except BlockingIOError as exc: - if "fd" in locals(): - os.close(fd) - raise TrainingRunLockedError(f"{self.key} is already active") from exc - try: - self.store._mark_held(self.key) - except Exception: - fcntl.flock(fd, fcntl.LOCK_UN) - os.close(fd) - raise - self.fd = fd - return self - - def __exit__(self, *_: object) -> None: - if self.fd is None: - return - self.store._mark_released(self.key) - fcntl.flock(self.fd, fcntl.LOCK_UN) - os.close(self.fd) - self.fd = None - - -class TrainingStore: - """Create-only stage records and candidate evidence outside Git.""" - - def __init__(self, *, repo_root: Path, external_root: Path) -> None: - self.repo_root = repo_root.expanduser().resolve(strict=True) - requested = external_root.expanduser().resolve(strict=False) - if ( - requested == self.repo_root - or requested.is_relative_to(self.repo_root) - or self.repo_root.is_relative_to(requested) - ): - raise UnsafeTrainingRootError("training output overlaps the repository") - requested.mkdir(parents=True, exist_ok=True, mode=0o700) - self.external_root = requested.resolve(strict=True) - if ( - self.external_root == self.repo_root - or self.external_root.is_relative_to(self.repo_root) - or self.repo_root.is_relative_to(self.external_root) - ): - raise UnsafeTrainingRootError("training output resolves across the repository") - self._held: set[StageKey] = set() - self._guard = threading.Lock() - - @staticmethod - def stage_relative_path(key: StageKey) -> PurePosixPath: - return PurePosixPath( - "studies", - key.study_id, - "stages", - key.stage_id, - "runs", - key.replicate_id, - ) - - def stage_path(self, key: StageKey) -> Path: - return self._path(self.stage_relative_path(key)) - - def tinker_log_path(self, key: StageKey) -> Path: - return self.stage_path(key) / "tinker" - - def candidate_root(self, key: StageKey) -> Path: - return self.stage_path(key) / "candidates" - - def acquire_stage_lock(self, key: StageKey) -> _StageLock: - digest = canonical_sha256(key.as_dict()) - return _StageLock( - self, - key, - self.external_root / ".locks" / f"{digest}.lock", - ) - - def _mark_held(self, key: StageKey) -> None: - with self._guard: - if key in self._held: - raise TrainingRunLockedError(f"{key} is already locked here") - self._held.add(key) - - def _mark_released(self, key: StageKey) -> None: - with self._guard: - self._held.discard(key) - - def _require_lock(self, key: StageKey) -> None: - with self._guard: - if key not in self._held: - raise TrainingStoreError("a stage lock is required before writing") - - def create_or_verify_manifest( - self, - key: StageKey, - payload: Mapping[str, Any], - ) -> TrainingRecord: - self._require_lock(key) - return self._create_or_verify( - self.stage_relative_path(key) / "run_manifest.json", - record_type="run_manifest", - key=key.as_dict(), - payload=payload, - ) - - def load_manifest(self, key: StageKey) -> TrainingRecord | None: - return self._load( - self.stage_relative_path(key) / "run_manifest.json", - record_type="run_manifest", - key=key.as_dict(), - ) - - def write_wave_receipt( - self, - key: StageKey, - *, - wave: str, - payload: Mapping[str, Any], - ) -> TrainingRecord: - self._require_lock(key) - if not _IDENTIFIER.fullmatch(wave): - raise ValueError(f"invalid wave name: {wave!r}") - if self.load_manifest(key) is None: - raise TrainingStoreError("wave receipt cannot precede its run manifest") - return self._create_or_verify( - self.stage_relative_path(key) / "waves" / wave / "receipt.json", - record_type="wave_receipt", - key={**key.as_dict(), "wave": wave}, - payload=payload, - ) - - def load_wave_receipt( - self, - key: StageKey, - *, - wave: str, - ) -> TrainingRecord | None: - return self._load( - self.stage_relative_path(key) / "waves" / wave / "receipt.json", - record_type="wave_receipt", - key={**key.as_dict(), "wave": wave}, - ) - - def write_wave_approval( - self, - key: StageKey, - *, - wave: str, - payload: Mapping[str, Any], - ) -> TrainingRecord: - """Bind a non-initial paid wave to one immutable review decision.""" - - self._require_lock(key) - if not _IDENTIFIER.fullmatch(wave): - raise ValueError(f"invalid wave name: {wave!r}") - return self._create_or_verify( - self.stage_relative_path(key) / "waves" / wave / "approval.json", - record_type="wave_approval", - key={**key.as_dict(), "wave": wave}, - payload=payload, - ) - - def load_wave_approval( - self, - key: StageKey, - *, - wave: str, - ) -> TrainingRecord | None: - return self._load( - self.stage_relative_path(key) / "waves" / wave / "approval.json", - record_type="wave_approval", - key={**key.as_dict(), "wave": wave}, - ) - - def write_rollout_health_approval( - self, - key: StageKey, - *, - wave: str, - payload: Mapping[str, Any], - ) -> TrainingRecord: - """Persist one deterministic, receipt-only rollout-health decision.""" - - self._require_lock(key) - if not _IDENTIFIER.fullmatch(wave): - raise ValueError(f"invalid wave name: {wave!r}") - return self._create_or_verify( - self.stage_relative_path(key) - / "waves" - / wave - / "rollout_health_approval.json", - record_type="rollout_health_approval", - key={**key.as_dict(), "wave": wave}, - payload=payload, - ) - - def load_rollout_health_approval( - self, - key: StageKey, - *, - wave: str, - ) -> TrainingRecord | None: - return self._load( - self.stage_relative_path(key) - / "waves" - / wave - / "rollout_health_approval.json", - record_type="rollout_health_approval", - key={**key.as_dict(), "wave": wave}, - ) - - def write_base_baseline_approval( - self, - key: StageKey, - *, - wave: str, - payload: Mapping[str, Any], - ) -> TrainingRecord: - """Persist the base-policy level-panel evidence required before RL.""" - - self._require_lock(key) - if not _IDENTIFIER.fullmatch(wave): - raise ValueError(f"invalid wave name: {wave!r}") - return self._create_or_verify( - self.stage_relative_path(key) - / "waves" - / wave - / "base_baseline_approval.json", - record_type="base_baseline_approval", - key={**key.as_dict(), "wave": wave}, - payload=payload, - ) - - def load_base_baseline_approval( - self, - key: StageKey, - *, - wave: str, - ) -> TrainingRecord | None: - return self._load( - self.stage_relative_path(key) - / "waves" - / wave - / "base_baseline_approval.json", - record_type="base_baseline_approval", - key={**key.as_dict(), "wave": wave}, - ) - - def write_candidate( - self, - key: StageKey, - *, - invocation_id: str, - step: int, - task_id: str, - attempt: int, - payload: Mapping[str, Any], - ) -> TrainingRecord: - self._require_lock(key) - if ( - not _IDENTIFIER.fullmatch(invocation_id) - or step < 0 - or attempt < 1 - or not _IDENTIFIER.fullmatch(task_id) - ): - raise ValueError("invalid candidate key") - return self._create_or_verify( - self.stage_relative_path(key) - / "candidates" - / invocation_id - / f"step-{step:06d}" - / task_id - / f"attempt-{attempt:02d}" - / "evaluation.json", - record_type="candidate_evaluation", - key={ - **key.as_dict(), - "invocation_id": invocation_id, - "step": step, - "task_id": task_id, - "attempt": attempt, - }, - payload=payload, - ) - - def load_candidate( - self, - key: StageKey, - *, - invocation_id: str, - step: int, - task_id: str, - attempt: int, - ) -> TrainingRecord | None: - if ( - not _IDENTIFIER.fullmatch(invocation_id) - or step < 0 - or attempt < 1 - or not _IDENTIFIER.fullmatch(task_id) - ): - raise ValueError("invalid candidate key") - return self._load( - self.stage_relative_path(key) - / "candidates" - / invocation_id - / f"step-{step:06d}" - / task_id - / f"attempt-{attempt:02d}" - / "evaluation.json", - record_type="candidate_evaluation", - key={ - **key.as_dict(), - "invocation_id": invocation_id, - "step": step, - "task_id": task_id, - "attempt": attempt, - }, - ) - - def write_candidate_sample( - self, - key: StageKey, - *, - invocation_id: str, - step: int, - task_id: str, - attempt: int, - payload: Mapping[str, Any], - ) -> TrainingRecord: - """Persist paid model output before deterministic evaluation begins.""" - - self._require_lock(key) - if ( - not _IDENTIFIER.fullmatch(invocation_id) - or step < 0 - or attempt < 1 - or not _IDENTIFIER.fullmatch(task_id) - ): - raise ValueError("invalid candidate key") - return self._create_or_verify( - self.stage_relative_path(key) - / "candidates" - / invocation_id - / f"step-{step:06d}" - / task_id - / f"attempt-{attempt:02d}" - / "sample.json", - record_type="candidate_sample", - key={ - **key.as_dict(), - "invocation_id": invocation_id, - "step": step, - "task_id": task_id, - "attempt": attempt, - }, - payload=payload, - ) - - def load_candidate_sample( - self, - key: StageKey, - *, - invocation_id: str, - step: int, - task_id: str, - attempt: int, - ) -> TrainingRecord | None: - return self._load( - self.stage_relative_path(key) - / "candidates" - / invocation_id - / f"step-{step:06d}" - / task_id - / f"attempt-{attempt:02d}" - / "sample.json", - record_type="candidate_sample", - key={ - **key.as_dict(), - "invocation_id": invocation_id, - "step": step, - "task_id": task_id, - "attempt": attempt, - }, - ) - - def begin_invocation( - self, - key: StageKey, - *, - invocation_id: str, - payload: Mapping[str, Any], - ) -> TrainingRecord: - """Record one process entry so stochastic crash retries never collide.""" - - self._require_lock(key) - if not _IDENTIFIER.fullmatch(invocation_id): - raise ValueError(f"invalid invocation ID: {invocation_id!r}") - if self.load_manifest(key) is None: - raise TrainingStoreError("invocation cannot precede its run manifest") - return self._create_or_verify( - self.stage_relative_path(key) / "invocations" / invocation_id / "start.json", - record_type="invocation", - key={**key.as_dict(), "invocation_id": invocation_id}, - payload=payload, - ) - - def load_invocation( - self, - key: StageKey, - *, - invocation_id: str, - ) -> TrainingRecord | None: - if not _IDENTIFIER.fullmatch(invocation_id): - raise ValueError(f"invalid invocation ID: {invocation_id!r}") - return self._load( - self.stage_relative_path(key) / "invocations" / invocation_id / "start.json", - record_type="invocation", - key={**key.as_dict(), "invocation_id": invocation_id}, - ) - - def write_invocation_tracking( - self, - key: StageKey, - *, - invocation_id: str, - payload: Mapping[str, Any], - ) -> TrainingRecord: - """Bind the observed live tracking session before paid training begins.""" - - self._require_lock(key) - if not _IDENTIFIER.fullmatch(invocation_id): - raise ValueError(f"invalid invocation ID: {invocation_id!r}") - if self.load_invocation(key, invocation_id=invocation_id) is None: - raise TrainingStoreError("tracking session cannot precede its invocation") - return self._create_or_verify( - self.stage_relative_path(key) / "invocations" / invocation_id / "tracking.json", - record_type="invocation_tracking", - key={**key.as_dict(), "invocation_id": invocation_id}, - payload=payload, - ) - - def load_invocation_tracking( - self, - key: StageKey, - *, - invocation_id: str, - ) -> TrainingRecord | None: - if not _IDENTIFIER.fullmatch(invocation_id): - raise ValueError(f"invalid invocation ID: {invocation_id!r}") - return self._load( - self.stage_relative_path(key) / "invocations" / invocation_id / "tracking.json", - record_type="invocation_tracking", - key={**key.as_dict(), "invocation_id": invocation_id}, - ) - - def _path(self, relative: PurePosixPath) -> Path: - path = self.external_root.joinpath(*relative.parts) - parent = path.parent.resolve(strict=False) - if parent != self.external_root and not parent.is_relative_to(self.external_root): - raise UnsafeTrainingRootError("training record path escaped its root") - return path - - def _create_or_verify( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - payload: Mapping[str, Any], - ) -> TrainingRecord: - expected = _document(record_type=record_type, key=key, payload=payload) - path = self._path(relative) - path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) - raw = canonical_bytes(expected) + b"\n" - temporary = path.parent / f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}" - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(temporary, flags, 0o600) - try: - with os.fdopen(fd, "wb", closefd=False) as stream: - stream.write(raw) - stream.flush() - os.fsync(stream.fileno()) - finally: - os.close(fd) - try: - os.link(temporary, path) - directory_fd = os.open(path.parent, os.O_RDONLY) - try: - os.fsync(directory_fd) - finally: - os.close(directory_fd) - except FileExistsError: - observed = self._load(relative, record_type=record_type, key=key) - if observed is None or canonical_bytes(expected) != canonical_bytes( - self._read(path) - ): - raise ImmutableTrainingRecordError( - f"{relative} already contains different data" - ) - return observed - finally: - try: - temporary.unlink() - except FileNotFoundError: - pass - return _validate_document( - expected, - record_type=record_type, - key=key, - relative_path=relative, - ) - - def _load( - self, - relative: PurePosixPath, - *, - record_type: str, - key: Mapping[str, Any], - ) -> TrainingRecord | None: - path = self._path(relative) - if not path.exists(): - return None - return _validate_document( - self._read(path), - record_type=record_type, - key=key, - relative_path=relative, - ) - - @staticmethod - def _read(path: Path) -> dict[str, Any]: - flags = os.O_RDONLY - if hasattr(os, "O_NOFOLLOW"): - flags |= os.O_NOFOLLOW - try: - fd = os.open(path, flags) - except OSError as exc: - if exc.errno == errno.ELOOP: - raise TrainingStoreError(f"{path} must not be a symlink") from exc - raise - try: - metadata = os.fstat(fd) - if not stat.S_ISREG(metadata.st_mode) or metadata.st_size > MAX_RECORD_BYTES: - raise TrainingStoreError(f"{path} is not a bounded regular file") - chunks: list[bytes] = [] - remaining = MAX_RECORD_BYTES + 1 - while remaining: - chunk = os.read(fd, min(1024 * 1024, remaining)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - if len(raw) > MAX_RECORD_BYTES: - raise TrainingStoreError(f"{path} exceeds the record limit") - finally: - os.close(fd) - try: - value = json.loads(raw) - except (UnicodeDecodeError, json.JSONDecodeError) as exc: - raise TrainingStoreError(f"{path} is not valid JSON") from exc - if not isinstance(value, dict): - raise TrainingStoreError(f"{path} must contain an object") - return value diff --git a/rl/studies/representation_training_v1/tests/__init__.py b/rl/studies/representation_training_v1/tests/__init__.py deleted file mode 100644 index 9f821b1c..00000000 --- a/rl/studies/representation_training_v1/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Tests for the sealed representation training study.""" diff --git a/rl/studies/representation_training_v1/tests/test_base_rl_gates.py b/rl/studies/representation_training_v1/tests/test_base_rl_gates.py deleted file mode 100644 index 65df9b51..00000000 --- a/rl/studies/representation_training_v1/tests/test_base_rl_gates.py +++ /dev/null @@ -1,1001 +0,0 @@ -from __future__ import annotations - -import asyncio -import hashlib -import io -import json -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest -from PIL import Image - -pytest.importorskip("chz", reason="Tinker is an optional RL dependency") -pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") -pytest.importorskip( - "tinker_cookbook", - reason="Tinker Cookbook is an optional RL dependency", -) - -from tinker_cookbook.renderers import Message, ParseTermination, TextPart - -from rl.common.contracts import ModelObservation, VerifierReference -from rl.common.evaluator import ( - Attribution, - EvaluationResult, - EvaluationStatus, -) -from rl.evaluation.tasks import EvaluationTask -from rl.studies.representation_training_v1 import evaluation, launcher, promotion -from rl.studies.representation_training_v1.protocol import ( - STUDY_ID, - StageSpec, - load_protocol, - stage_spec, -) -from rl.studies.representation_training_v1.store import ( - StageKey, - TrainingRecord, - TrainingStore, - canonical_sha256, -) - - -REPO_ROOT = Path(__file__).resolve().parents[4] -SOURCE_SHA = "a" * 40 - - -def _stage(level: str = "L0") -> StageSpec: - index = int(level[1:]) - return StageSpec( - stage_id=f"qwen-base-rl-{level.lower()}", - kind="rl", - model_key="qwen", - recipe_key="qwen_rl", - hypotheses=("h-qwen-base-curriculum-rl",), - parent=("base:qwen" if index == 0 else f"qwen-base-rl-l{index - 1}:complete"), - levels=(), - current_level=level, - replay_levels=tuple(f"L{value}" for value in range(index)), - waves={ - "smoke": {"max_steps": 1, "approval": "initial"}, - "step-5": { - "max_steps": 5, - "approval": "rollout-health-receipt", - }, - "step-10": { - "max_steps": 10, - "approval": "current-level-held-out-promotion-receipt", - }, - "complete": { - "max_steps": 30, - "approval": "current-level-held-out-promotion-receipt", - }, - }, - ) - - -def _image_bytes() -> bytes: - image = Image.new("L", (8, 8), 255) - for x in range(2, 6): - for y in range(2, 6): - image.putpixel((x, y), 0) - stream = io.BytesIO() - image.save(stream, format="PNG") - return stream.getvalue() - - -def _task(task_id: str = "l0-baseline") -> EvaluationTask: - image = _image_bytes() - digest = hashlib.sha256(image).hexdigest() - return EvaluationTask( - task_id=task_id, - level="L0", - representation_id="representation-l0", - observation=ModelObservation( - image_bytes=image, - footprint_um=(2.0, 2.0), - image_sha256=digest, - ), - reference=VerifierReference( - target_image_bytes=image, - footprint_um=(2.0, 2.0), - target_image_sha256=digest, - ), - ) - - -class _Renderer: - tokenizer = SimpleNamespace(decode=lambda tokens: "decoded:" + repr(tokens)) - - @staticmethod - def build_generation_prompt(messages: list[Any]) -> Any: - assert "Phase-A Device Input" in messages[0]["content"][1]["text"] - return SimpleNamespace(length=100) - - @staticmethod - def get_stop_sequences() -> list[int]: - return [99] - - @staticmethod - def parse_response(_tokens: list[int]) -> tuple[Message, ParseTermination]: - return ( - Message( - role="assistant", - content=[TextPart(type="text", text="print('candidate')")], - ), - ParseTermination.STOP_SEQUENCE, - ) - - -class _Evaluator: - execution_boundary = SimpleNamespace( - runtime_path=Path("/usr/bin/false"), - daemon_endpoint="unix:///private/fake.sock", - image_ref="pixcell-evaluator@sha256:" + "3" * 64, - image_id="sha256:" + "4" * 64, - workspace_root=Path("/private/fake-workspace"), - ) - - @staticmethod - def validate_reference(reference: VerifierReference) -> dict[str, Any]: - return { - "target_image_sha256": reference.target_image_sha256, - "image_size_px": [8, 8], - "bbox": [2, 2, 6, 6], - "scale_px_per_um": [2.0, 2.0], - } - - @staticmethod - def evaluate_batch( - requests: list[tuple[VerifierReference, str]], - ) -> list[EvaluationResult]: - return [ - EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=0.5, - dice=2 / 3, - metrics={ - "render_sha256": "1" * 64, - "reference_sha256": "2" * 64, - }, - ) - for _ in requests - ] - - @staticmethod - def close() -> None: - return None - - -class _SamplingClient: - @staticmethod - async def sample_async(**kwargs: Any) -> Any: - assert kwargs["num_samples"] == 1 - return SimpleNamespace( - sequences=[SimpleNamespace(tokens=[1, 2, 3], stop_reason="stop")] - ) - - -class _BaseService: - def __init__(self) -> None: - self.calls: list[dict[str, str]] = [] - - def create_sampling_client(self, **kwargs: str) -> _SamplingClient: - self.calls.append(dict(kwargs)) - return _SamplingClient() - - -def test_level_panels_are_exact_fixed_validation_slots(tmp_path: Path) -> None: - protocol = load_protocol(REPO_ROOT) - validation_rows = evaluation._depth_rows(REPO_ROOT / "dataset") - assert evaluation.PANELS == ( - "progress", - "depth-validation", - "inkling-promotion", - ) - expected_counts = {"L0": 93, "L1": 127, "L2": 98, "L3": 120, "L4": 108} - for level, panel in evaluation.LEVEL_PROGRESS_PANELS.items(): - tasks = evaluation.build_panel_tasks( - repo_root=REPO_ROOT, - protocol=protocol, - panel=panel, - ) - assert len(tasks) == expected_counts[level] - assert {task.level for task in tasks} == {level} - assert len({task.representation_id for task in tasks}) == len(tasks) - assert {task.task_id for task in tasks} == { - evaluation._depth_task(row).task_id - for row in validation_rows - if str(row["level"]).upper() == level - and row["realization_slot"] == 6 - } - final = evaluation.build_panel_tasks( - repo_root=REPO_ROOT, - protocol=protocol, - panel=evaluation.PANEL_DEPTH_FINAL_SELECTION, - ) - assert len(final) == 546 - assert len({(task.level, task.representation_id) for task in final}) == 546 - assert {task.task_id for task in final} == { - evaluation._depth_task(row).task_id - for row in validation_rows - if row["realization_slot"] == 7 - } - - stage_path = tmp_path / "stage" - stage_path.mkdir() - for panel in ( - *evaluation.LEVEL_PROGRESS_PANELS.values(), - evaluation.PANEL_DEPTH_FINAL_SELECTION, - ): - store = evaluation.CheckpointEvaluationStore( - stage_path=stage_path, - wave="complete", - checkpoint_name="final", - panel=panel, - ) - assert store.root.name == panel - - -def test_base_and_checkpoint_sampler_client_arguments_are_distinct() -> None: - protocol = load_protocol(REPO_ROOT) - stage = _stage() - base = evaluation.BaseModelBinding( - key=StageKey(STUDY_ID, stage.stage_id, "r0"), - stage=stage, - wave="base", - checkpoint_name="base", - checkpoint={"name": "base"}, - checkpoint_inventory_sha256="1" * 64, - sampler_path=protocol["models"]["qwen"]["model"], - binding_sha256="2" * 64, - ) - record = TrainingRecord( - payload={}, - payload_sha256="3" * 64, - record_sha256="4" * 64, - relative_path=Path("record.json"), - ) - checkpoint = evaluation.CheckpointBinding( - key=base.key, - stage=stage, - wave="smoke", - checkpoint_name="000001", - checkpoint={"name": "000001"}, - checkpoint_inventory_sha256="5" * 64, - sampler_path="tinker://unit/sampler/000001", - run_manifest=record, - wave_receipt=record, - ) - assert evaluation._sampling_client_kwargs(base) == { - "base_model": protocol["models"]["qwen"]["model"] - } - assert evaluation._sampling_client_kwargs(checkpoint) == { - "model_path": "tinker://unit/sampler/000001" - } - - -def test_base_l0_panel_run_and_resume_has_no_fictitious_receipt( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = _stage() - service = _BaseService() - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setattr(evaluation, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - evaluation, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr(evaluation, "stage_spec", lambda *_a, **_k: stage) - monkeypatch.setattr(evaluation, "_renderer", lambda *_a, **_k: _Renderer()) - - result = asyncio.run( - evaluation.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id=stage.stage_id, - wave="base", - checkpoint_name="base", - replicate_id="r0", - panel=evaluation.LEVEL_PROGRESS_PANELS["L0"], - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256="", - external_root=tmp_path / "external", - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=lambda: service, - evaluator_factory=_Evaluator, - base_model=True, - ) - ) - assert result["status"] == "complete" - assert service.calls == [ - {"base_model": protocol["models"]["qwen"]["model"]} - ] - report = json.loads(Path(result["report"]).read_text(encoding="utf-8")) - sampler = report["payload"]["provenance"]["sampler"] - assert sampler["binding_kind"] == "base-model" - assert not any("training_" in field for field in sampler) - sample = next((tmp_path / "external").rglob("sample.json")) - sample_payload = json.loads(sample.read_text(encoding="utf-8"))["payload"] - assert "base_model_binding_sha256" in sample_payload - assert "training_wave_receipt_record_sha256" not in sample_payload - assert not list((tmp_path / "external").rglob("receipt.json")) - assert not list((tmp_path / "external").rglob("run_manifest.json")) - - gated_stage = StageSpec( - **{ - **stage.__dict__, - "waves": { - **stage.waves, - "smoke": { - "max_steps": 1, - "approval": "base-level-held-out-baseline-receipt", - }, - }, - } - ) - training_store = TrainingStore( - repo_root=REPO_ROOT, - external_root=tmp_path / "external", - ) - with training_store.acquire_stage_lock( - StageKey(STUDY_ID, stage.stage_id, "r0") - ): - direct_approval = launcher._approval_record( - repo_root=REPO_ROOT, - store=training_store, - key=StageKey(STUDY_ID, stage.stage_id, "r0"), - stage=stage, - wave_name="smoke", - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert direct_approval is None - - with training_store.acquire_stage_lock( - StageKey(STUDY_ID, stage.stage_id, "r0") - ): - base_approval = launcher._approval_record( - repo_root=REPO_ROOT, - store=training_store, - key=StageKey(STUDY_ID, stage.stage_id, "r0"), - stage=gated_stage, - wave_name="smoke", - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert base_approval is not None - assert ( - base_approval.payload["approval_gate"] - == "base-level-held-out-baseline-receipt" - ) - - monkeypatch.delenv("TINKER_API_KEY") - resumed = asyncio.run( - evaluation.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id=stage.stage_id, - wave="base", - checkpoint_name="base", - replicate_id="r0", - panel=evaluation.LEVEL_PROGRESS_PANELS["L0"], - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256="", - external_root=tmp_path / "external", - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=lambda: pytest.fail("paid client on resume"), - evaluator_factory=_Evaluator, - base_model=True, - ) - ) - assert resumed["status"] == "already_complete" - - -def test_base_panel_rechecks_source_after_preflight_before_paid_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = _stage() - source_checks = 0 - - def changing_source(_root: Path, expected: str) -> str: - nonlocal source_checks - source_checks += 1 - if source_checks == 2: - raise ValueError("worktree changed") - return expected - - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setattr(evaluation, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr(evaluation, "validate_source_sha", changing_source) - monkeypatch.setattr(evaluation, "stage_spec", lambda *_a, **_k: stage) - monkeypatch.setattr( - evaluation, - "_dataset_binding", - lambda *_a, **_k: { - "logical_release_sha256": protocol["dataset"][ - "logical_release_sha256" - ], - "freeze_file_sha256": "1" * 64, - "parquet_shards": {"depth.parquet": "2" * 64}, - }, - ) - monkeypatch.setattr(evaluation, "build_panel_tasks", lambda **_k: [_task()]) - monkeypatch.setattr(evaluation, "_renderer", lambda *_a, **_k: _Renderer()) - with pytest.raises( - evaluation.CheckpointEvaluationError, - match="source changed after evaluation preflight", - ): - asyncio.run( - evaluation.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id=stage.stage_id, - wave="base", - checkpoint_name="base", - replicate_id="r0", - panel=evaluation.LEVEL_PROGRESS_PANELS["L0"], - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256="", - external_root=tmp_path / "external", - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=lambda: pytest.fail( - "paid client created after source mutation" - ), - evaluator_factory=_Evaluator, - base_model=True, - ) - ) - assert source_checks == 2 - - -def _write_health_fixture( - tmp_path: Path, - *, - fault: bool = False, - all_constant: bool = False, - all_failed: bool = False, - unknown_status: bool = False, - misattributed_infrastructure_status: bool = False, - reward_mismatch: bool = False, - failure_nonzero_reward: bool = False, - failure_geometry: bool = False, - sample_inventory_mismatch: bool = False, -) -> tuple[ - TrainingStore, - StageKey, - StageSpec, - dict[str, Any], - TrainingRecord, -]: - protocol = load_protocol(REPO_ROOT) - stage = _stage() - key = StageKey(STUDY_ID, stage.stage_id, "r0") - store = TrainingStore(repo_root=REPO_ROOT, external_root=tmp_path / "external") - invocation_id = "smoke-unit" - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - { - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - }, - ) - invocation = store.begin_invocation( - key, - invocation_id=invocation_id, - payload={"wave": "smoke"}, - ) - for group in range(8): - task_id = f"task-{group}" - for attempt in range(1, 5): - sample = store.write_candidate_sample( - key, - invocation_id=invocation_id, - step=0, - task_id=task_id, - attempt=attempt, - payload={"task": task_id, "attempt": attempt}, - ) - status = "runtime_error" - if not all_failed and group == 0 and attempt == 2: - status = "ok" - if unknown_status and group == 0 and attempt == 1: - status = "invented_status" - if ( - misattributed_infrastructure_status - and group == 0 - and attempt == 1 - ): - status = "reference_error" - reward = ( - 0.5 - if status == "ok" and not all_constant - else 0.0 - ) - iou = reward if status == "ok" else None - dice = (2.0 * iou / (1.0 + iou)) if iou is not None else None - if reward_mismatch and status == "ok": - reward = 0.4 - if failure_nonzero_reward and group == 0 and attempt == 1: - reward = 0.25 - if failure_geometry and group == 0 and attempt == 1: - iou = 0.25 - dice = 0.4 - store.write_candidate( - key, - invocation_id=invocation_id, - step=0, - task_id=task_id, - attempt=attempt, - payload={ - "sample_record": { - "relative_path": str(sample.relative_path), - "payload_sha256": sample.payload_sha256, - "record_sha256": sample.record_sha256, - }, - "evaluation": { - "status": status, - "attribution": ( - "reference" - if fault and group == 0 and attempt == 1 - else "model" - ), - "iou": iou, - "dice": dice, - "reward": reward, - "violations": [], - "error": None, - "latency_seconds": 0.1, - "metrics": {}, - }, - }, - ) - candidate_root = store.candidate_root(key) / invocation_id - candidate_files = sorted(candidate_root.rglob("evaluation.json")) - sample_files = sorted(candidate_root.rglob("sample.json")) - candidate_inventory = [ - { - "path": str(path.relative_to(store.stage_path(key))), - "sha256": promotion.file_sha256(path), - } - for path in candidate_files - ] - sample_inventory = [ - { - "path": str(path.relative_to(store.stage_path(key))), - "sha256": promotion.file_sha256(path), - } - for path in sample_files - ] - receipt = store.write_wave_receipt( - key, - wave="smoke", - payload={ - "stage_id": stage.stage_id, - "wave": "smoke", - "max_steps": 1, - "run_manifest_record_sha256": manifest.record_sha256, - "invocation_record_sha256": invocation.record_sha256, - "invocation_id": invocation_id, - "checkpoint": { - "name": "000001", - "batch": 1, - "epoch": None, - "final": True, - "state_path": "tinker://unit/state/000001", - "sampler_path": "tinker://unit/sampler/000001", - }, - "candidate_inventory": { - "count": 32, - "expected_count": 32, - "logical_sha256": canonical_sha256(candidate_inventory), - }, - "sample_inventory": { - "count": 32, - "expected_count": 32, - "logical_sha256": ( - "f" * 64 - if sample_inventory_mismatch - else canonical_sha256(sample_inventory) - ), - }, - }, - ) - return store, key, stage, protocol, receipt - - -def test_rollout_health_gate_is_receipt_only_and_exact(tmp_path: Path) -> None: - store, key, stage, protocol, receipt = _write_health_fixture(tmp_path) - with store.acquire_stage_lock(key): - approval = promotion.record_rollout_health_approval( - store=store, - key=key, - stage=stage, - wave_name="step-5", - prior_wave="smoke", - prior_receipt=receipt, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - metrics = approval.payload["metrics"] - assert metrics["candidates"] == 32 - assert metrics["groups"] == 8 - assert metrics["group_size"] == 4 - assert metrics["pure_executable"] == 1 - assert metrics["nonconstant_reward_groups"] == 1 - assert metrics["evaluator_or_reference_faults"] == 0 - assert store.load_wave_approval(key, wave="step-5") is None - assert ( - store.load_rollout_health_approval(key, wave="step-5").record_sha256 - == approval.record_sha256 - ) - - -def test_launcher_derives_rollout_health_before_step5(tmp_path: Path) -> None: - store, key, stage, protocol, _receipt = _write_health_fixture(tmp_path) - with store.acquire_stage_lock(key): - approval = launcher._approval_record( - repo_root=REPO_ROOT, - store=store, - key=key, - stage=stage, - wave_name="step-5", - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert approval is not None - assert approval.payload["approval_gate"] == "rollout-health-receipt" - health = store.load_rollout_health_approval(key, wave="step-5") - assert health is not None - assert health.payload["approval_gate"] == "rollout-health-receipt" - assert store.load_wave_approval(key, wave="step-5") is None - resolved = launcher._resolved_launch_binding( - repo_root=REPO_ROOT, - protocol=protocol, - stage=stage, - wave_name="step-5", - store=store, - key=key, - parent=None, - resume_batch=1, - ) - assert resolved["wave_gate_evidence"] == { - "approval_gate": "rollout-health-receipt", - "relative_path": str(approval.relative_path), - "payload_sha256": approval.payload_sha256, - "record_sha256": approval.record_sha256, - } - - -def test_l0_sft_rl_resolved_launch_binds_parent_transition( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = StageSpec( - stage_id="qwen-l0-rl-l1", - kind="rl", - model_key="qwen", - recipe_key="qwen_rl", - hypotheses=("RT-H03",), - parent="qwen-l0-sft:complete", - levels=(), - current_level="L1", - replay_levels=("L0",), - waves={ - "smoke": { - "max_steps": 1, - "approval": "parent-receipt-and-source-transition", - }, - }, - ) - key = StageKey(STUDY_ID, stage.stage_id, "r0") - store = TrainingStore( - repo_root=REPO_ROOT, - external_root=tmp_path / "external", - ) - with store.acquire_stage_lock(key): - store.create_or_verify_manifest( - key, - { - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - }, - ) - approval = store.write_wave_approval( - key, - wave="smoke", - payload={ - "schema_version": ( - promotion.PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ), - "approval_gate": "parent-receipt-and-source-transition", - }, - ) - resolved = launcher._resolved_launch_binding( - repo_root=REPO_ROOT, - protocol=protocol, - stage=stage, - wave_name="smoke", - store=store, - key=key, - parent=None, - resume_batch=None, - ) - assert resolved["wave_gate_evidence"] == { - "approval_gate": "parent-receipt-and-source-transition", - "relative_path": str(approval.relative_path), - "payload_sha256": approval.payload_sha256, - "record_sha256": approval.record_sha256, - } - - -@pytest.mark.parametrize("panel", ["level-progress-l0", "progress"]) -def test_launcher_current_level_gate_binds_exact_level_panel( - tmp_path: Path, - panel: str, - monkeypatch: pytest.MonkeyPatch, -) -> None: - store, key, original, protocol, receipt = _write_health_fixture(tmp_path) - stage = StageSpec( - **{ - **original.__dict__, - "waves": { - "smoke": {"max_steps": 1, "approval": "initial"}, - "step-10": { - "max_steps": 10, - "approval": "current-level-held-out-promotion-receipt", - }, - }, - } - ) - checkpoint = receipt.payload["checkpoint"] - # The fixture below is intentionally minimal because this test exercises - # the launcher's downstream receipt/panel binding. Canonical report - # reconstruction is tested separately and must remain fail-closed. - monkeypatch.setattr( - launcher, - "validate_same_source_wave_approval", - lambda **_kwargs: None, - ) - payload = { - "schema_version": promotion.APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "current-level-held-out-promotion-receipt", - "stage_id": stage.stage_id, - "wave": "step-10", - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "prior_wave": "smoke", - "prior_receipt_record_sha256": receipt.record_sha256, - "evaluation": { - "stage_id": stage.stage_id, - "wave": "smoke", - "panel": panel, - "checkpoint": checkpoint, - "report_record_sha256": "9" * 64, - "training_wave_receipt_record_sha256": receipt.record_sha256, - }, - } - with store.acquire_stage_lock(key): - store.write_wave_approval(key, wave="step-10", payload=payload) - if panel == "level-progress-l0": - observed = launcher._approval_record( - repo_root=REPO_ROOT, - store=store, - key=key, - stage=stage, - wave_name="step-10", - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert observed is not None - else: - with pytest.raises( - launcher.TrainingLaunchError, - match="held-out report", - ): - launcher._approval_record( - repo_root=REPO_ROOT, - store=store, - key=key, - stage=stage, - wave_name="step-10", - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - - -def test_launcher_next_level_gate_uses_current_dual_panel_contract( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - store = TrainingStore(repo_root=REPO_ROOT, external_root=tmp_path / "external") - parent_stage = stage_spec(protocol, "qwen-base-rl-l0") - child_stage = stage_spec(protocol, "qwen-base-rl-l1") - assert child_stage.wave("smoke")["approval"] == ( - "prior-and-current-level-held-out-transition-receipt" - ) - parent_key = StageKey(STUDY_ID, parent_stage.stage_id, "r0") - child_key = StageKey(STUDY_ID, child_stage.stage_id, "r0") - checkpoint = { - "name": "000030", - "batch": 30, - "epoch": None, - "final": True, - "state_path": "tinker://unit/state/000030", - "sampler_path": "tinker://unit/sampler/000030", - } - inventory_entry = { - **checkpoint, - "role": "terminal", - "training_progress_fraction": 1.0, - } - with store.acquire_stage_lock(parent_key): - manifest = store.create_or_verify_manifest( - parent_key, - { - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - }, - ) - receipt = store.write_wave_receipt( - parent_key, - wave="complete", - payload={ - "stage_id": parent_stage.stage_id, - "wave": "complete", - "run_manifest_record_sha256": manifest.record_sha256, - "checkpoint": checkpoint, - "checkpoint_inventory": { - "count": 1, - "entries": [inventory_entry], - "logical_sha256": canonical_sha256([inventory_entry]), - }, - }, - ) - approval_payload = { - "schema_version": promotion.APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "prior-and-current-level-held-out-transition-receipt", - "stage_id": child_stage.stage_id, - "wave": "smoke", - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "parent_stage_id": parent_stage.stage_id, - "parent_wave": "complete", - "parent_receipt_record_sha256": receipt.record_sha256, - "selected_parent_checkpoint": checkpoint, - "selected_parent_checkpoint_inventory_sha256": canonical_sha256( - [inventory_entry] - ), - "evaluation": { - "stage_id": parent_stage.stage_id, - "wave": "complete", - "panel": "level-progress-l0", - "checkpoint": checkpoint, - "report_record_sha256": "8" * 64, - "training_wave_receipt_record_sha256": receipt.record_sha256, - }, - "entry_baseline": { - "stage_id": parent_stage.stage_id, - "wave": "complete", - "panel": "level-progress-l1", - "checkpoint": checkpoint, - "report_record_sha256": "9" * 64, - "training_wave_receipt_record_sha256": receipt.record_sha256, - }, - } - validation_calls: list[dict[str, Any]] = [] - monkeypatch.setattr( - launcher, - "validate_same_source_wave_approval", - lambda **kwargs: validation_calls.append(dict(kwargs)), - ) - with store.acquire_stage_lock(child_key): - store.write_wave_approval( - child_key, - wave="smoke", - payload=approval_payload, - ) - approval = launcher._approval_record( - repo_root=REPO_ROOT, - store=store, - key=child_key, - stage=child_stage, - wave_name="smoke", - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert approval is not None - assert approval.payload["evaluation"]["panel"] == "level-progress-l0" - assert approval.payload["entry_baseline"]["panel"] == "level-progress-l1" - assert len(validation_calls) == 1 - assert validation_calls[0]["approval"] is approval - - -@pytest.mark.parametrize( - ("kwargs", "message"), - [ - ({"fault": True}, "evaluator or reference fault"), - ({"all_constant": True}, "no nonconstant G4 group"), - ({"all_failed": True}, "no pure executable candidate"), - ({"unknown_status": True}, "unknown evaluator status"), - ( - {"misattributed_infrastructure_status": True}, - "evaluator or reference status mislabeled", - ), - ({"reward_mismatch": True}, "reward differs from raw IoU"), - ({"failure_nonzero_reward": True}, "model failure has nonzero reward"), - ({"failure_geometry": True}, "model failure contains geometry metrics"), - ({"sample_inventory_mismatch": True}, "sample files differ"), - ], -) -def test_rollout_health_gate_fails_closed( - tmp_path: Path, - kwargs: dict[str, bool], - message: str, -) -> None: - store, key, stage, protocol, receipt = _write_health_fixture( - tmp_path, - **kwargs, - ) - with store.acquire_stage_lock(key), pytest.raises( - promotion.PromotionError, - match=message, - ): - promotion.record_rollout_health_approval( - store=store, - key=key, - stage=stage, - wave_name="step-5", - prior_wave="smoke", - prior_receipt=receipt, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - - -def test_model_status_allowlists_cannot_drift() -> None: - assert promotion._MODEL_EVALUATION_STATUSES == ( - evaluation._MODEL_EVALUATION_STATUSES - ) - - -def test_level_gate_plans_never_use_f1_f8() -> None: - l0 = _stage("L0") - current = promotion.approval_plan(l0, "step-10") - assert current.expected_panel == "level-progress-l0" - assert current.allowed_evaluation_checkpoints == ((l0.stage_id, "step-5"),) - - l1 = _stage("L1") - # The first wave has the prior-level approval in the sealed protocol; - # represent it directly here without weakening StageSpec validation. - l1 = StageSpec( - **{ - **l1.__dict__, - "waves": { - **l1.waves, - "smoke": { - "max_steps": 1, - "approval": ( - "prior-and-current-level-held-out-transition-receipt" - ), - }, - }, - } - ) - prior = promotion.approval_plan(l1, "smoke") - assert prior.expected_panel == "level-progress-l0" - assert prior.allowed_evaluation_checkpoints == ( - ("qwen-base-rl-l0", "complete"), - ) - assert prior.entry_baseline_panel == "level-progress-l1" - assert current.expected_panel != promotion.PANEL_PROGRESS - assert prior.expected_panel != promotion.PANEL_PROGRESS diff --git a/rl/studies/representation_training_v1/tests/test_base_rl_protocol.py b/rl/studies/representation_training_v1/tests/test_base_rl_protocol.py deleted file mode 100644 index f06cc2a1..00000000 --- a/rl/studies/representation_training_v1/tests/test_base_rl_protocol.py +++ /dev/null @@ -1,152 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -from collections.abc import Callable -from pathlib import Path -from typing import Any - -import pytest - -from rl.studies.representation_training_v1.protocol import ( - BASE_QWEN_RL_STAGE_IDS, - L0_SFT_QWEN_RL_STAGE_IDS, - STUDY_RELATIVE_PATH, - load_protocol, - stage_spec, -) - - -REPO_ROOT = Path(__file__).resolve().parents[4] - - -def _write_mutated_protocol( - tmp_path: Path, - mutate: Callable[[dict[str, Any]], None], -) -> None: - source = ( - REPO_ROOT - / STUDY_RELATIVE_PATH - / "protocol.json" - ) - document = json.loads(source.read_text(encoding="utf-8")) - mutate(document) - logical = { - key: value for key, value in document.items() if key != "logical_sha256" - } - document["logical_sha256"] = hashlib.sha256( - json.dumps( - logical, - allow_nan=False, - ensure_ascii=True, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - ).hexdigest() - target = tmp_path / STUDY_RELATIVE_PATH / "protocol.json" - target.parent.mkdir(parents=True) - target.write_text( - json.dumps(document, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - - -def test_base_qwen_rl_contract_is_exact() -> None: - protocol = load_protocol(REPO_ROOT) - for level, stage_id in enumerate(BASE_QWEN_RL_STAGE_IDS): - stage = stage_spec(protocol, stage_id) - assert stage.current_level == f"L{level}" - assert stage.parent == ( - "base:qwen" - if level == 0 - else f"qwen-base-rl-l{level - 1}:complete" - ) - assert stage.replay_levels == tuple(f"L{prior}" for prior in range(level)) - assert stage.hypotheses == ("RT-H06",) - assert stage.wave("smoke")["approval"] == ( - "initial" - if level == 0 - else "prior-and-current-level-held-out-transition-receipt" - ) - assert stage.wave("step-5")["approval"] == "rollout-health-receipt" - for wave in ("step-10", "step-15", "step-20", "step-25", "complete"): - assert ( - stage.wave(wave)["approval"] - == "current-level-held-out-promotion-receipt" - ) - - -def test_l0_sft_to_qwen_rl_contract_uses_modern_level_gates() -> None: - protocol = load_protocol(REPO_ROOT) - for level, stage_id in enumerate(L0_SFT_QWEN_RL_STAGE_IDS, start=1): - stage = stage_spec(protocol, stage_id) - assert stage.current_level == f"L{level}" - assert stage.parent == ( - "qwen-l0-sft:complete" - if level == 1 - else f"qwen-l0-rl-l{level - 1}:complete" - ) - assert stage.replay_levels == tuple(f"L{prior}" for prior in range(level)) - assert stage.hypotheses == ("RT-H03",) - assert stage.wave("smoke")["approval"] == ( - "parent-receipt-and-source-transition" - if level == 1 - else "prior-level-held-out-promotion-receipt" - ) - assert stage.wave("step-5")["approval"] == "rollout-health-receipt" - for wave in ("step-10", "step-15", "step-20", "step-25", "complete"): - assert ( - stage.wave(wave)["approval"] - == "current-level-held-out-promotion-receipt" - ) - - -@pytest.mark.parametrize( - ("mutate", "message"), - [ - ( - lambda value: value["stages"]["qwen-base-rl-l0"]["waves"]["smoke"].update( - {"approval": "base-level-held-out-baseline-receipt"} - ), - "wrong pure-RL gates", - ), - ( - lambda value: value["stages"]["qwen-base-rl-l0"]["waves"]["step-5"].update( - {"approval": "clean-smoke-receipt"} - ), - "wrong pure-RL gates", - ), - ( - lambda value: value["stages"]["qwen-l0-rl-l1"]["waves"]["smoke"].update( - {"approval": "entry-evaluation-and-promotion-receipt"} - ), - "wrong L0-SFT-to-RL gates", - ), - ( - lambda value: value["stages"]["qwen-base-rl-l2"].update( - {"parent": "qwen-base-rl-l0:complete"} - ), - "changed the pure Qwen RL curriculum", - ), - ( - lambda value: value["evaluation"]["curriculum_progress_holdout"].update( - {"realization_slot": 7} - ), - "wrong curriculum progress holdout", - ), - ( - lambda value: value["evaluation"]["final_selection_holdout"].update( - {"realization_slot": 6} - ), - "wrong final selection holdout", - ), - ], -) -def test_base_qwen_rl_contract_mutations_fail_closed( - tmp_path: Path, - mutate: Callable[[dict[str, Any]], None], - message: str, -) -> None: - _write_mutated_protocol(tmp_path, mutate) - with pytest.raises(ValueError, match=message): - load_protocol(tmp_path) diff --git a/rl/studies/representation_training_v1/tests/test_evaluation.py b/rl/studies/representation_training_v1/tests/test_evaluation.py deleted file mode 100644 index b3a89218..00000000 --- a/rl/studies/representation_training_v1/tests/test_evaluation.py +++ /dev/null @@ -1,738 +0,0 @@ -from __future__ import annotations - -import asyncio -import hashlib -import io -from collections import Counter -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest -from PIL import Image - -pytest.importorskip("chz", reason="Tinker is an optional RL dependency") -pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") -pytest.importorskip( - "tinker_cookbook", - reason="Tinker Cookbook is an optional RL dependency", -) - -from tinker_cookbook.renderers import ( - Message, - ParseTermination, - TextPart, -) - -from rl.common.contracts import ModelObservation, VerifierReference -from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus -from rl.evaluation.tasks import EvaluationTask -from rl.studies.representation_training_v1 import evaluation as subject -from rl.studies.representation_training_v1.protocol import ( - STUDY_ID, - load_protocol, -) -from rl.studies.representation_training_v1.store import ( - StageKey, - TrainingStore, -) - - -REPO_ROOT = Path(__file__).resolve().parents[4] -SOURCE_SHA = "a" * 40 - - -def _image_bytes() -> bytes: - image = Image.new("L", (8, 8), 255) - for x in range(2, 6): - for y in range(2, 6): - image.putpixel((x, y), 0) - buffer = io.BytesIO() - image.save(buffer, format="PNG") - return buffer.getvalue() - - -def _task(task_id: str = "F1", *, level: str = "BENCHMARK") -> EvaluationTask: - image = _image_bytes() - digest = hashlib.sha256(image).hexdigest() - return EvaluationTask( - task_id=task_id, - level=level, - representation_id=f"representation-{task_id.lower()}", - observation=ModelObservation( - image_bytes=image, - footprint_um=(2.0, 2.0), - image_sha256=digest, - ), - reference=VerifierReference( - target_image_bytes=image, - footprint_um=(2.0, 2.0), - target_image_sha256=digest, - ), - ) - - -class _FakeRenderer: - def __init__(self, events: list[str]) -> None: - self.events = events - self.tokenizer = SimpleNamespace(decode=lambda tokens: "raw:" + repr(tokens)) - - def build_generation_prompt(self, messages: list[Any]) -> Any: - self.events.append("prompt") - assert len(messages) == 1 - text = messages[0]["content"][1]["text"] - assert "Phase-A Device Input" in text - assert "target_image" not in text - return SimpleNamespace(length=100) - - @staticmethod - def get_stop_sequences() -> list[int]: - return [99] - - @staticmethod - def parse_response(_tokens: list[int]) -> tuple[Message, ParseTermination]: - return ( - Message( - role="assistant", - content=[TextPart(type="text", text="print('candidate')")], - ), - ParseTermination.STOP_SEQUENCE, - ) - - -class _FakeEvaluator: - def __init__( - self, - events: list[str], - *, - result: EvaluationResult | None = None, - broken_reference: bool = False, - ) -> None: - self.events = events - self.result = result or EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=0.75, - dice=0.8, - metrics={ - "render_sha256": "1" * 64, - "reference_sha256": "2" * 64, - }, - ) - self.broken_reference = broken_reference - self.execution_boundary = SimpleNamespace( - runtime_path=Path("/usr/bin/false"), - daemon_endpoint="unix:///private/fake.sock", - image_ref="pixcell-evaluator@sha256:" + "3" * 64, - image_id="sha256:" + "4" * 64, - workspace_root=Path("/private/fake-workspace"), - ) - - def validate_reference(self, reference: VerifierReference) -> dict[str, Any]: - self.events.append("reference") - if self.broken_reference: - raise ValueError("broken reference") - return { - "target_image_sha256": reference.target_image_sha256, - "image_size_px": [8, 8], - "bbox": [2, 2, 6, 6], - "scale_px_per_um": [2.0, 2.0], - } - - def evaluate_batch( - self, - requests: list[tuple[VerifierReference, str]], - ) -> list[EvaluationResult]: - self.events.append("evaluate") - return [self.result for _ in requests] - - def close(self) -> None: - self.events.append("close") - - -class _FakeSamplingClient: - async def sample_async(self, **kwargs: Any) -> Any: - assert kwargs["num_samples"] == 1 - parameters = kwargs["sampling_params"] - assert parameters.max_tokens == subject.MAX_OUTPUT_TOKENS - assert parameters.temperature == subject.TEMPERATURE - assert parameters.top_p == subject.TOP_P - return SimpleNamespace( - sequences=[ - SimpleNamespace( - tokens=[1, 2, 3], - stop_reason="stop", - ) - ] - ) - - -class _FakeService: - def __init__(self, events: list[str]) -> None: - self.events = events - - def create_sampling_client(self, *, model_path: str) -> _FakeSamplingClient: - self.events.append("sampling-client") - assert model_path.startswith("tinker://") - return _FakeSamplingClient() - - -def _training_receipt( - tmp_path: Path, - *, - protocol: dict[str, Any], - stage_id: str = "qwen-l0-sft", - wave: str = "complete", -) -> tuple[Path, str]: - external = tmp_path / "external" - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - key = StageKey(STUDY_ID, stage_id, "r0") - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - { - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "contract_version": protocol["contract_version"], - "dataset": { - "logical_release_sha256": protocol["dataset"][ - "logical_release_sha256" - ] - }, - "stage": { - "stage_id": stage_id, - "model_key": protocol["stages"][stage_id]["model"], - }, - }, - ) - stage = protocol["stages"][stage_id] - max_steps = int(stage["waves"][wave]["max_steps"]) - entries = ( - [ - { - "name": f"{max_steps:06d}", - "batch": max_steps, - "epoch": None, - "final": True, - "state_path": f"tinker://fake/state-{max_steps:06d}", - "sampler_path": f"tinker://fake/sampler-{max_steps:06d}", - "role": "terminal", - "training_progress_fraction": 1.0, - } - ] - if stage["kind"] == "rl" - else [ - { - "name": "000003", - "batch": 3, - "epoch": None, - "final": False, - "state_path": "tinker://fake/state-000003", - "sampler_path": "tinker://fake/sampler-000003", - "role": "periodic", - "training_progress_fraction": 3 / 11, - }, - { - "name": "final", - "batch": 11, - "epoch": 1, - "final": True, - "state_path": "tinker://fake/state", - "sampler_path": "tinker://fake/sampler", - "role": "terminal", - "training_progress_fraction": 1.0, - }, - ] - ) - terminal = { - key: entries[-1][key] - for key in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - } - receipt = store.write_wave_receipt( - key, - wave=wave, - payload={ - "stage_id": stage_id, - "wave": wave, - "run_manifest_record_sha256": manifest.record_sha256, - "checkpoint": terminal, - "checkpoint_inventory": { - "count": len(entries), - "entries": entries, - "logical_sha256": subject._canonical_sha256(entries), - }, - }, - ) - return external, receipt.record_sha256 - - -def _dataset_binding(protocol: dict[str, Any]) -> dict[str, Any]: - return { - "logical_release_sha256": protocol["dataset"][ - "logical_release_sha256" - ], - "freeze_file_sha256": "5" * 64, - "train_rows": protocol["dataset"]["train_rows"], - "validation_rows": protocol["dataset"]["validation_rows"], - "parquet_shards": {"depth-v1/data/validation-00000.parquet": "6" * 64}, - "reference_artifacts_sha256": "7" * 64, - "reference_artifact_count": 1, - } - - -def test_exact_panel_inventory_and_label_free_projection() -> None: - protocol = load_protocol(REPO_ROOT) - progress = subject.build_panel_tasks( - repo_root=REPO_ROOT, - protocol=protocol, - panel=subject.PANEL_PROGRESS, - ) - depth = subject.build_panel_tasks( - repo_root=REPO_ROOT, - protocol=protocol, - panel=subject.PANEL_DEPTH_VALIDATION, - ) - promotion = subject.build_panel_tasks( - repo_root=REPO_ROOT, - protocol=protocol, - panel=subject.PANEL_INKLING_PROMOTION, - ) - - assert [task.task_id for task in progress] == [ - "F1", - "F2", - "F3", - "F4", - "F5", - "F6", - "F7", - "F8", - ] - assert len(depth) == 1092 - assert len({task.representation_id for task in depth}) == 546 - assert len(promotion) == 68 - assert Counter(task.level for task in promotion) == { - "BENCHMARK": 8, - "L0": 6, - "L1": 6, - "L2": 6, - "L3": 6, - "L4": 36, - } - assert not any(hasattr(task, "label") or hasattr(task, "code") for task in depth) - repeated = subject.build_panel_tasks( - repo_root=REPO_ROOT, - protocol=protocol, - panel=subject.PANEL_INKLING_PROMOTION, - ) - assert [task.task_id for task in repeated] == [ - task.task_id for task in promotion - ] - - -def test_store_is_create_only_and_resumable(tmp_path: Path) -> None: - stage = tmp_path / "stage" - stage.mkdir() - store = subject.CheckpointEvaluationStore( - stage_path=stage, - wave="complete", - checkpoint_name="final", - panel=subject.PANEL_PROGRESS, - ) - payload = {"source_git_sha": SOURCE_SHA} - first = store.create_or_verify_manifest(payload) - second = store.create_or_verify_manifest(payload) - - assert first == second - periodic = subject.CheckpointEvaluationStore( - stage_path=stage, - wave="complete", - checkpoint_name="000003", - panel=subject.PANEL_PROGRESS, - ) - assert periodic.root != store.root - assert periodic.create_or_verify_manifest(payload) == first - with pytest.raises(subject.ImmutableEvaluationRecordError): - store.create_or_verify_manifest({"source_git_sha": "b" * 40}) - - -def test_checkpoint_can_only_load_from_the_exact_wave_receipt( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - - binding = subject._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="000003", - replicate_id="r0", - expected_receipt_sha256=receipt_sha, - source_git_sha=SOURCE_SHA, - ) - assert binding.checkpoint_name == "000003" - assert binding.sampler_path == "tinker://fake/sampler-000003" - assert binding.checkpoint["training_progress_fraction"] == 3 / 11 - assert binding.wave_receipt.record_sha256 == receipt_sha - final_binding = subject._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="final", - replicate_id="r0", - expected_receipt_sha256=receipt_sha, - source_git_sha=SOURCE_SHA, - ) - assert subject._sample_seed( - protocol_sha256=protocol["logical_sha256"], - binding=binding, - panel=subject.PANEL_PROGRESS, - task_id="F1", - ) != subject._sample_seed( - protocol_sha256=protocol["logical_sha256"], - binding=final_binding, - panel=subject.PANEL_PROGRESS, - task_id="F1", - ) - - with pytest.raises(subject.CheckpointReceiptError): - subject._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="000003", - replicate_id="r0", - expected_receipt_sha256="f" * 64, - source_git_sha=SOURCE_SHA, - ) - with pytest.raises(subject.CheckpointReceiptError): - subject._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="user-supplied-uri-alias", - replicate_id="r0", - expected_receipt_sha256=receipt_sha, - source_git_sha=SOURCE_SHA, - ) - - -def test_rl_wave_uses_its_unique_numeric_ceiling_checkpoint( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt( - tmp_path, - protocol=protocol, - stage_id="inkling-l4-rl", - wave="smoke", - ) - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - binding = subject._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id="inkling-l4-rl", - wave="smoke", - checkpoint_name="000001", - replicate_id="r0", - expected_receipt_sha256=receipt_sha, - source_git_sha=SOURCE_SHA, - ) - assert binding.checkpoint_name == "000001" - assert binding.checkpoint["role"] == "terminal" - assert binding.checkpoint["training_progress_fraction"] == 1.0 - - with pytest.raises(subject.CheckpointReceiptError): - subject._load_checkpoint_binding( - store=store, - protocol=protocol, - stage_id="inkling-l4-rl", - wave="smoke", - checkpoint_name="final", - replicate_id="r0", - expected_receipt_sha256=receipt_sha, - source_git_sha=SOURCE_SHA, - ) - - -def test_model_fault_is_zero_and_infrastructure_fault_aborts() -> None: - task = _task() - model_failure = EvaluationResult( - status=EvaluationStatus.SYNTAX_ERROR, - attribution=Attribution.MODEL, - error="invalid syntax", - ) - payload = subject._evaluation_payload( - task=task, - result=model_failure, - sample_record_sha256="1" * 64, - manifest_record_sha256="2" * 64, - ) - assert payload["raw_absolute_scale_iou"] == 0.0 - assert payload["pure_executable"] is False - - for attribution, error_type in ( - (Attribution.REFERENCE, subject.ReferencePanelFault), - (Attribution.EVALUATOR, subject.EvaluationInfrastructureFault), - ): - with pytest.raises(error_type): - subject._evaluation_payload( - task=task, - result=EvaluationResult( - status=EvaluationStatus.WORKER_ERROR, - attribution=attribution, - error="fault", - ), - sample_record_sha256="1" * 64, - manifest_record_sha256="2" * 64, - ) - - -def test_stored_evaluation_rejects_misattributed_infrastructure_status() -> None: - task = _task() - payload = subject._evaluation_payload( - task=task, - result=EvaluationResult( - status=EvaluationStatus.SYNTAX_ERROR, - attribution=Attribution.MODEL, - error="invalid syntax", - ), - sample_record_sha256="1" * 64, - manifest_record_sha256="2" * 64, - ) - subject._validate_evaluation_record( - record={"payload": payload}, - task=task, - sample_record_sha256="1" * 64, - manifest_record_sha256="2" * 64, - ) - - payload["status"] = EvaluationStatus.REFERENCE_ERROR.value - with pytest.raises( - subject.ImmutableEvaluationRecordError, - match="non-model evaluation status", - ): - subject._validate_evaluation_record( - record={"payload": payload}, - task=task, - sample_record_sha256="1" * 64, - manifest_record_sha256="2" * 64, - ) - - -def test_all_prompt_and_reference_checks_precede_service_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - evaluator = _FakeEvaluator(events) - task = _task() - - monkeypatch.setattr(subject, "load_protocol", lambda *_args, **_kwargs: protocol) - monkeypatch.setattr(subject, "validate_source_sha", lambda *_args: SOURCE_SHA) - monkeypatch.setattr( - subject, - "_dataset_binding", - lambda *_args: _dataset_binding(protocol), - ) - monkeypatch.setattr(subject, "build_panel_tasks", lambda **_kwargs: [task]) - monkeypatch.setattr(subject, "_renderer", lambda *_args, **_kwargs: renderer) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - - def service_factory() -> _FakeService: - events.append("service") - assert "prompt" in events - assert "reference" in events - return _FakeService(events) - - result = asyncio.run( - subject.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="final", - replicate_id="r0", - panel=subject.PANEL_PROGRESS, - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256=receipt_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=service_factory, - evaluator_factory=lambda: evaluator, - ) - ) - - assert result["status"] == "complete" - assert events.index("prompt") < events.index("service") - assert events.index("reference") < events.index("service") - assert result["summary"]["mean_raw_absolute_scale_iou"] == 0.75 - assert "/complete/final/progress/report.json" in result["report"] - report = subject.CheckpointEvaluationStore( - stage_path=( - TrainingStore(repo_root=REPO_ROOT, external_root=external).stage_path( - StageKey(STUDY_ID, "qwen-l0-sft", "r0") - ) - ), - wave="complete", - checkpoint_name="final", - panel=subject.PANEL_PROGRESS, - ).load_report() - assert report is not None - assert report["payload"]["checkpoint_name"] == "final" - assert ( - report["payload"]["provenance"]["sampler"]["checkpoint"] - ["training_progress_fraction"] - == 1.0 - ) - assert set(report["payload"]["provenance"]["dataset"]) == { - "repo_id", - "revision", - "configuration", - "split", - "logical_release_sha256", - "freeze_file_sha256", - "parquet_shards", - } - - -def test_reference_failure_cannot_create_a_service_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - evaluator = _FakeEvaluator(events, broken_reference=True) - - monkeypatch.setattr(subject, "load_protocol", lambda *_args, **_kwargs: protocol) - monkeypatch.setattr(subject, "validate_source_sha", lambda *_args: SOURCE_SHA) - monkeypatch.setattr( - subject, - "_dataset_binding", - lambda *_args: _dataset_binding(protocol), - ) - monkeypatch.setattr(subject, "build_panel_tasks", lambda **_kwargs: [_task()]) - monkeypatch.setattr(subject, "_renderer", lambda *_args, **_kwargs: renderer) - - def forbidden_service() -> Any: - raise AssertionError("ServiceClient was created before reference preflight") - - with pytest.raises(subject.ReferencePanelFault): - asyncio.run( - subject.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="final", - replicate_id="r0", - panel=subject.PANEL_PROGRESS, - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256=receipt_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=forbidden_service, - evaluator_factory=lambda: evaluator, - ) - ) - - -def test_sample_survives_evaluator_fault_and_resume_does_not_resample( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - task = _task() - events: list[str] = [] - renderer = _FakeRenderer(events) - bad_evaluator = _FakeEvaluator( - events, - result=EvaluationResult( - status=EvaluationStatus.WORKER_ERROR, - attribution=Attribution.EVALUATOR, - error="worker failed", - retryable=True, - ), - ) - - monkeypatch.setattr(subject, "load_protocol", lambda *_args, **_kwargs: protocol) - monkeypatch.setattr(subject, "validate_source_sha", lambda *_args: SOURCE_SHA) - monkeypatch.setattr( - subject, - "_dataset_binding", - lambda *_args: _dataset_binding(protocol), - ) - monkeypatch.setattr(subject, "build_panel_tasks", lambda **_kwargs: [task]) - monkeypatch.setattr(subject, "_renderer", lambda *_args, **_kwargs: renderer) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - - with pytest.raises(subject.EvaluationInfrastructureFault): - asyncio.run( - subject.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="final", - replicate_id="r0", - panel=subject.PANEL_PROGRESS, - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256=receipt_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=lambda: _FakeService(events), - evaluator_factory=lambda: bad_evaluator, - ) - ) - - training_store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - record_store = subject.CheckpointEvaluationStore( - stage_path=training_store.stage_path(key), - wave="complete", - checkpoint_name="final", - panel=subject.PANEL_PROGRESS, - ) - assert record_store.load_sample(task.task_id) is not None - assert record_store.load_evaluation(task.task_id) is None - - good_evaluator = _FakeEvaluator(events) - - def forbidden_service() -> Any: - raise AssertionError("resume attempted to resample an immutable completion") - - monkeypatch.delenv("TINKER_API_KEY") - result = asyncio.run( - subject.run_checkpoint_evaluation( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="final", - replicate_id="r0", - panel=subject.PANEL_PROGRESS, - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256=receipt_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=forbidden_service, - evaluator_factory=lambda: good_evaluator, - ) - ) - assert result["status"] == "complete" - assert record_store.load_evaluation(task.task_id) is not None diff --git a/rl/studies/representation_training_v1/tests/test_final_benchmark.py b/rl/studies/representation_training_v1/tests/test_final_benchmark.py deleted file mode 100644 index b3f8896b..00000000 --- a/rl/studies/representation_training_v1/tests/test_final_benchmark.py +++ /dev/null @@ -1,711 +0,0 @@ -from __future__ import annotations - -import asyncio -import copy -import json -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -pytest.importorskip("chz", reason="Tinker is an optional RL dependency") -pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") -pytest.importorskip( - "tinker_cookbook", - reason="Tinker Cookbook is an optional RL dependency", -) - -from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus -from rl.studies.representation_training_v1 import final_benchmark as subject -from rl.studies.representation_training_v1.protocol import STUDY_ID, load_protocol -from rl.studies.representation_training_v1.store import StageKey, TrainingStore - -from .test_evaluation import ( - REPO_ROOT, - SOURCE_SHA, - _dataset_binding, - _FakeEvaluator, - _FakeRenderer, - _task, - _training_receipt, -) - - -class _SamplingClient: - def __init__( - self, - events: list[str], - *, - fail_on_call: int | None = None, - ) -> None: - self.events = events - self.fail_on_call = fail_on_call - self.seeds: list[int] = [] - - async def sample_async(self, **kwargs: Any) -> Any: - assert kwargs["num_samples"] == 1 - parameters = kwargs["sampling_params"] - seed = int(parameters.seed) - self.seeds.append(seed) - self.events.append(f"sample:{seed}") - if self.fail_on_call == len(self.seeds): - raise RuntimeError("synthetic sampling outage") - assert parameters.max_tokens == subject.MAX_OUTPUT_TOKENS - assert parameters.temperature == subject.TEMPERATURE - assert parameters.top_p == subject.TOP_P - return SimpleNamespace( - sequences=[ - SimpleNamespace( - tokens=[1, 2, seed % 1000 + 3], - stop_reason="stop", - ) - ] - ) - - -class _Service: - def __init__( - self, - events: list[str], - client: _SamplingClient, - ) -> None: - self.events = events - self.client = client - - def create_sampling_client(self, *, model_path: str) -> _SamplingClient: - self.events.append("sampling-client") - assert model_path == "tinker://fake/sampler" - return self.client - - -def _tasks() -> list[Any]: - return [_task(task_id) for task_id in subject.EXPECTED_TASK_IDS] - - -def _patch_preflight( - monkeypatch: pytest.MonkeyPatch, - *, - protocol: dict[str, Any], - tasks: list[Any], - renderer: _FakeRenderer, -) -> None: - monkeypatch.setattr(subject, "load_protocol", lambda *_args, **_kwargs: protocol) - monkeypatch.setattr( - subject, - "validate_source_sha", - lambda *_args: SOURCE_SHA, - ) - monkeypatch.setattr( - subject, - "_dataset_binding", - lambda *_args: _dataset_binding(protocol), - ) - monkeypatch.setattr( - subject.checkpoint_v1, - "_benchmark_tasks", - lambda **_kwargs: tasks, - ) - monkeypatch.setattr(subject, "_renderer", lambda *_args, **_kwargs: renderer) - - -def _run( - *, - protocol: dict[str, Any], - external: Path, - receipt_sha: str, - evaluator: _FakeEvaluator, - service_factory: Any, - checkpoint_name: str = "final", -) -> dict[str, Any]: - return asyncio.run( - subject.run_final_benchmark( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name=checkpoint_name, - replicate_id="r0", - expected_source_sha=SOURCE_SHA, - expected_wave_receipt_sha256=receipt_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - service_client_factory=service_factory, - evaluator_factory=lambda: evaluator, - ) - ) - - -def _store(external: Path) -> subject.FinalBenchmarkStore: - training = TrainingStore(repo_root=REPO_ROOT, external_root=external) - key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - return subject.FinalBenchmarkStore( - stage_path=training.stage_path(key), - wave="complete", - checkpoint_name="final", - ) - - -def _validation_context( - *, - protocol: dict[str, Any], - external: Path, - receipt_sha: str, - tasks: list[Any], - renderer: _FakeRenderer, - evaluator: _FakeEvaluator, -) -> tuple[Any, list[subject.PreparedAttempt]]: - training = TrainingStore(repo_root=REPO_ROOT, external_root=external) - binding = subject.checkpoint_v1._load_checkpoint_binding( - store=training, - protocol=protocol, - stage_id="qwen-l0-sft", - wave="complete", - checkpoint_name="final", - replicate_id="r0", - expected_receipt_sha256=receipt_sha, - source_git_sha=SOURCE_SHA, - ) - prepared, _ = subject.checkpoint_v1._preflight_tasks( - tasks=tasks, - renderer=renderer, - evaluator=evaluator, - context_tokens=65_536, - ) - panel = subject._task_panel(tasks) - attempts = subject._attempts( - prepared=prepared, - source_git_sha=SOURCE_SHA, - protocol_sha256=protocol["logical_sha256"], - dataset_sha256=protocol["dataset"]["logical_release_sha256"], - task_panel_sha256=panel["logical_sha256"], - binding=binding, - ) - return binding, attempts - - -def _completed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> dict[str, Any]: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - evaluator = _FakeEvaluator(events) - tasks = _tasks() - _patch_preflight( - monkeypatch, - protocol=protocol, - tasks=tasks, - renderer=renderer, - ) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - client = _SamplingClient(events) - - def service_factory() -> _Service: - events.append("service") - assert events.count("reference") == 8 - return _Service(events, client) - - result = _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=evaluator, - service_factory=service_factory, - ) - store = _store(external) - manifest = store.load_manifest() - report = store.load_report() - assert manifest is not None - assert report is not None - binding, attempts = _validation_context( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - tasks=tasks, - renderer=renderer, - evaluator=evaluator, - ) - return { - "protocol": protocol, - "external": external, - "receipt_sha": receipt_sha, - "events": events, - "renderer": renderer, - "evaluator": evaluator, - "tasks": tasks, - "client": client, - "result": result, - "store": store, - "manifest": manifest, - "report": report, - "binding": binding, - "attempts": attempts, - } - - -def _write_record(path: Path, record: dict[str, Any]) -> None: - path.write_text( - json.dumps( - record, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ) - + "\n", - encoding="utf-8", - ) - - -def _resigned_report( - payload: dict[str, Any], -) -> dict[str, Any]: - return subject._record_document( - record_type="final_benchmark_report", - key={}, - payload=payload, - ) - - -def test_v1_panel_and_record_namespaces_remain_unchanged() -> None: - assert subject.checkpoint_v1.PANELS == ( - subject.checkpoint_v1.PANEL_PROGRESS, - subject.checkpoint_v1.PANEL_DEPTH_VALIDATION, - subject.checkpoint_v1.PANEL_INKLING_PROMOTION, - ) - assert subject.PANEL_FINAL_BENCHMARK_V2 not in subject.checkpoint_v1.PANELS - assert ( - subject.FINAL_BENCHMARK_RECORD_SCHEMA_VERSION - != subject.checkpoint_v1.EVALUATION_RECORD_SCHEMA_VERSION - ) - - -def test_exact_best4_records_and_metrics_bind_full_provenance( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - result = completed["result"] - store = completed["store"] - manifest = completed["manifest"]["payload"] - report = completed["report"]["payload"] - client = completed["client"] - events = completed["events"] - - assert result["status"] == "complete" - assert subject.PANEL_FINAL_BENCHMARK_V2 in result["report"] - assert len(client.seeds) == 32 - assert len(set(client.seeds)) == 32 - assert events[: events.index("service")].count("reference") == 8 - assert events.index("evaluate") > max( - index for index, event in enumerate(events) if event.startswith("sample:") - ) - - assert len(report["records"]) == 32 - assert report["summary"] == { - "figure_count": 8, - "mean_at_1_raw_absolute_scale_iou": 0.75, - "best_at_4_raw_absolute_scale_iou": 0.75, - "attempt_count": 32, - "pure_executable_count": 32, - "pure_executable_rate": 1.0, - "cap_hit_count": 0, - "cap_hit_rate": 0.0, - "channel_parse_complete_count": 32, - "channel_parse_complete_rate": 1.0, - "statuses": {"ok": 32}, - "stop_reasons": {"stop": 32}, - "completion_tokens": { - "min": 3, - "max": 3, - "mean": 3.0, - "total": 96, - }, - } - assert list(report["figures"]) == list(subject.EXPECTED_TASK_IDS) - assert all( - value["attempt_count"] == 4 - and value["best_at_4_raw_absolute_scale_iou"] == 0.75 - and value["pure_executable_rate"] == 1.0 - for value in report["figures"].values() - ) - - assert manifest["source_git_sha"] == SOURCE_SHA - assert manifest["protocol"]["logical_sha256"] == completed["protocol"]["logical_sha256"] - assert ( - manifest["dataset"]["logical_release_sha256"] - == completed["protocol"]["dataset"]["logical_release_sha256"] - ) - assert manifest["task_panel"]["task_ids"] == list(subject.EXPECTED_TASK_IDS) - assert manifest["prompt"]["contract_version"] == completed["protocol"]["contract_version"] - assert manifest["model"]["model"] == "Qwen/Qwen3.6-35B-A3B" - assert manifest["model"]["renderer"] == "qwen3_5" - assert manifest["sampler"]["checkpoint"]["final"] is True - assert manifest["sampler"]["training_wave_receipt_record_sha256"] == completed["receipt_sha"] - assert manifest["sandbox"]["image_id"] == "sha256:" + "4" * 64 - assert manifest["sampling"]["attempts_per_task"] == 4 - assert len(manifest["sampling"]["seeds"]) == 32 - - for task_id in subject.EXPECTED_TASK_IDS: - for attempt_index in range(1, 5): - sample = store.load_sample(task_id, attempt_index) - evaluation = store.load_evaluation(task_id, attempt_index) - assert sample is not None - assert evaluation is not None - assert sample["payload"]["response"]["token_ids"] - assert sample["payload"]["attempt_index"] == attempt_index - assert evaluation["payload"]["attempt_index"] == attempt_index - - -def test_partial_sampling_resume_fills_only_missing_attempts( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - evaluator = _FakeEvaluator(events) - tasks = _tasks() - _patch_preflight( - monkeypatch, - protocol=protocol, - tasks=tasks, - renderer=renderer, - ) - monkeypatch.setattr(subject, "SAMPLE_CONCURRENCY", 1) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - first_client = _SamplingClient(events, fail_on_call=6) - - with pytest.raises(subject.checkpoint_v1.SamplingInfrastructureFault): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=evaluator, - service_factory=lambda: _Service(events, first_client), - ) - - store = _store(external) - samples_before = { - path: path.read_bytes() for path in store.root.glob("attempts/*/*/sample.json") - } - assert 0 < len(samples_before) < 32 - assert not list(store.root.glob("attempts/*/*/evaluation.json")) - - second_client = _SamplingClient(events) - result = _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=lambda: _Service(events, second_client), - ) - - assert result["status"] == "complete" - assert len(second_client.seeds) == 32 - len(samples_before) - assert len(list(store.root.glob("attempts/*/*/sample.json"))) == 32 - assert len(list(store.root.glob("attempts/*/*/evaluation.json"))) == 32 - assert all(path.read_bytes() == raw for path, raw in samples_before.items()) - - -def test_undecodable_raw_tokens_are_persisted_and_never_executed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - - def fail_decode(_tokens: list[int]) -> str: - raise ValueError("synthetic tokenizer drift") - - renderer.tokenizer = SimpleNamespace(decode=fail_decode) - tasks = _tasks() - _patch_preflight( - monkeypatch, - protocol=protocol, - tasks=tasks, - renderer=renderer, - ) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - client = _SamplingClient(events) - - with pytest.raises(subject.checkpoint_v1.SamplingInfrastructureFault): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=lambda: _Service(events, client), - ) - - store = _store(external) - samples = list(store.root.glob("attempts/*/*/sample.json")) - assert len(samples) == 32 - assert not list(store.root.glob("attempts/*/*/evaluation.json")) - assert "evaluate" not in events - for path in samples: - payload = json.loads(path.read_text(encoding="utf-8"))["payload"] - assert payload["response"]["token_ids"] - assert payload["response"]["token_decode_complete"] is False - assert "synthetic tokenizer drift" in payload["response"]["token_decode_error"] - - def forbidden_service() -> Any: - raise AssertionError("undecodable persisted samples were resampled") - - with pytest.raises(subject.checkpoint_v1.SamplingInfrastructureFault): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=forbidden_service, - ) - - -def test_already_complete_never_touches_tinker_or_records( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - store = completed["store"] - before = {path: path.read_bytes() for path in store.root.rglob("*.json")} - monkeypatch.delenv("TINKER_API_KEY") - - def forbidden_service() -> Any: - raise AssertionError("already-complete benchmark created a Tinker client") - - result = _run( - protocol=completed["protocol"], - external=completed["external"], - receipt_sha=completed["receipt_sha"], - evaluator=_FakeEvaluator(completed["events"]), - service_factory=forbidden_service, - ) - - assert result["status"] == "already_complete" - assert {path: path.read_bytes() for path in store.root.rglob("*.json")} == before - - -def test_tampered_raw_response_aborts_before_resume_sampling( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - sample_path = completed["store"].root / "attempts" / "F1" / "000001" / "sample.json" - sample = json.loads(sample_path.read_text(encoding="utf-8")) - sample["payload"]["response"]["raw_text"] = "tampered" - _write_record(sample_path, sample) - - def forbidden_service() -> Any: - raise AssertionError("tampered resume created a Tinker client") - - with pytest.raises(subject.FinalBenchmarkError): - _run( - protocol=completed["protocol"], - external=completed["external"], - receipt_sha=completed["receipt_sha"], - evaluator=_FakeEvaluator(completed["events"]), - service_factory=forbidden_service, - ) - - -def test_foreign_fifth_attempt_path_is_rejected_before_resume( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - source = completed["store"].root / "attempts" / "F1" / "000001" / "sample.json" - foreign = completed["store"].root / "attempts" / "F1" / "000005" / "sample.json" - foreign.parent.mkdir(parents=True) - foreign.write_bytes(source.read_bytes()) - - def forbidden_service() -> Any: - raise AssertionError("foreign attempt inventory created a Tinker client") - - with pytest.raises(subject.FinalBenchmarkError): - _run( - protocol=completed["protocol"], - external=completed["external"], - receipt_sha=completed["receipt_sha"], - evaluator=_FakeEvaluator(completed["events"]), - service_factory=forbidden_service, - ) - - -@pytest.mark.parametrize( - "mutation", - ("panel", "checkpoint", "source"), -) -def test_resigned_wrong_report_identity_is_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - mutation: str, -) -> None: - completed = _completed(tmp_path, monkeypatch) - payload = copy.deepcopy(completed["report"]["payload"]) - if mutation == "panel": - payload["panel"] = "progress" - elif mutation == "checkpoint": - payload["checkpoint_name"] = "000003" - else: - payload["provenance"]["manifest"]["source_git_sha"] = "b" * 40 - tampered = _resigned_report(payload) - - with pytest.raises(subject.FinalBenchmarkError): - subject.validate_final_benchmark_report( - tampered, - manifest_record=completed["manifest"], - binding=completed["binding"], - attempts=completed["attempts"], - ) - - -@pytest.mark.parametrize("mutation", ("incomplete", "duplicate")) -def test_resigned_incomplete_or_duplicate_attempt_rows_are_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - mutation: str, -) -> None: - completed = _completed(tmp_path, monkeypatch) - payload = copy.deepcopy(completed["report"]["payload"]) - if mutation == "incomplete": - payload["records"].pop() - else: - payload["records"][-1] = copy.deepcopy(payload["records"][0]) - payload["attempt_records_sha256"] = subject.canonical_json_sha256(payload["records"]) - tampered = _resigned_report(payload) - - with pytest.raises(subject.FinalBenchmarkError): - subject.validate_final_benchmark_report( - tampered, - manifest_record=completed["manifest"], - binding=completed["binding"], - attempts=completed["attempts"], - ) - - -def test_nonfinite_evaluator_metrics_abort_instead_of_becoming_zero() -> None: - prepared = subject.checkpoint_v1.PreparedTask( - task=_task(), - prompt_tokens=100, - prompt_text_sha256="1" * 64, - reference_evidence={}, - ) - attempt = subject.PreparedAttempt( - prepared=prepared, - attempt_index=1, - seed=123, - ) - results = ( - EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=float("nan"), - ), - EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=0.5, - dice=float("inf"), - ), - EvaluationResult( - status=EvaluationStatus.OK, - attribution=Attribution.MODEL, - iou=0.5, - metrics={"diagnostic": float("nan")}, - ), - ) - for result in results: - with pytest.raises(subject.checkpoint_v1.EvaluationInfrastructureFault): - subject._evaluation_payload( - attempt=attempt, - result=result, - sample_record_sha256="2" * 64, - manifest_record_sha256="3" * 64, - ) - - -def test_reference_and_evaluator_attribution_are_never_model_zeros() -> None: - prepared = subject.checkpoint_v1.PreparedTask( - task=_task(), - prompt_tokens=100, - prompt_text_sha256="1" * 64, - reference_evidence={}, - ) - attempt = subject.PreparedAttempt(prepared=prepared, attempt_index=1, seed=1) - for attribution, error_type in ( - (Attribution.REFERENCE, subject.checkpoint_v1.ReferencePanelFault), - (Attribution.EVALUATOR, subject.checkpoint_v1.EvaluationInfrastructureFault), - ): - with pytest.raises(error_type): - subject._evaluation_payload( - attempt=attempt, - result=EvaluationResult( - status=EvaluationStatus.WORKER_ERROR, - attribution=attribution, - error="fault", - ), - sample_record_sha256="2" * 64, - manifest_record_sha256="3" * 64, - ) - - -def test_periodic_checkpoint_cannot_be_called_final_benchmark( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - _patch_preflight( - monkeypatch, - protocol=protocol, - tasks=_tasks(), - renderer=renderer, - ) - - def forbidden_service() -> Any: - raise AssertionError("periodic checkpoint created a Tinker client") - - with pytest.raises(subject.checkpoint_v1.CheckpointReceiptError): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=forbidden_service, - checkpoint_name="000003", - ) - - -def test_reference_preflight_fault_precedes_tinker_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _training_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - renderer = _FakeRenderer(events) - _patch_preflight( - monkeypatch, - protocol=protocol, - tasks=_tasks(), - renderer=renderer, - ) - - def forbidden_service() -> Any: - raise AssertionError("reference fault created a Tinker client") - - with pytest.raises(subject.checkpoint_v1.ReferencePanelFault): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events, broken_reference=True), - service_factory=forbidden_service, - ) diff --git a/rl/studies/representation_training_v1/tests/test_launcher.py b/rl/studies/representation_training_v1/tests/test_launcher.py deleted file mode 100644 index 04de7783..00000000 --- a/rl/studies/representation_training_v1/tests/test_launcher.py +++ /dev/null @@ -1,2227 +0,0 @@ -from __future__ import annotations - -import asyncio -import copy -import json -import os -from pathlib import Path -import subprocess -from types import SimpleNamespace - -import pytest - -pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") -pytest.importorskip( - "tinker_cookbook", - reason="Tinker Cookbook is an optional RL dependency", -) - -from rl.studies.representation_training_v1 import launcher -from rl.studies.representation_training_v1.protocol import ( - STUDY_ID, - load_protocol, - stage_spec, -) -from rl.studies.representation_training_v1.store import StageKey, TrainingStore - - -REPO_ROOT = Path(__file__).resolve().parents[4] - - -def _preflight(protocol: dict, stage_id: str, wave: str) -> dict: - stage = stage_spec(protocol, stage_id) - return { - "protocol_file_sha256": "1" * 64, - "prompt_assets": {"prompt": "2" * 64}, - "dataset": {"logical_release_sha256": protocol["dataset"]["logical_release_sha256"]}, - "runtime": {"python": "3.13"}, - "sandbox": None, - "schedule": { - "stage_id": stage_id, - "logical_sha256": "3" * 64, - }, - "model": protocol["models"][stage.model_key], - "recipe": protocol["recipes"][stage.recipe_key], - "preflight_sha256": "4" * 64, - } - - -def _cross_source_parent_records( - protocol: dict, - *, - producer_source_git_sha: str, - consumer_source_git_sha: str, -) -> tuple[dict, SimpleNamespace, SimpleNamespace, SimpleNamespace]: - checkpoint = { - "name": "final", - "batch": 11, - "epoch": 1, - "final": True, - "state_path": "tinker://unit/weights/final", - "sampler_path": "tinker://unit/sampler_weights/final", - "role": "terminal", - "training_progress_fraction": 1.0, - } - inventory_sha = launcher.canonical_sha256([checkpoint]) - parent_manifest = SimpleNamespace( - payload={ - "source_git_sha": producer_source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "contract_version": protocol["contract_version"], - "stage": { - "stage_id": "qwen-l0-sft", - "model_key": "qwen", - }, - }, - payload_sha256="1" * 64, - record_sha256="2" * 64, - relative_path=( - "studies/representation-training-v1/stages/qwen-l0-sft/runs/r0/run_manifest.json" - ), - ) - receipt = SimpleNamespace( - payload={ - "stage_id": "qwen-l0-sft", - "wave": "complete", - "run_manifest_record_sha256": parent_manifest.record_sha256, - "checkpoint": { - key: checkpoint[key] - for key in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - }, - "checkpoint_inventory": { - "count": 1, - "entries": [checkpoint], - "logical_sha256": inventory_sha, - }, - }, - payload_sha256="3" * 64, - record_sha256="4" * 64, - relative_path=( - "studies/representation-training-v1/stages/" - "qwen-l0-sft/runs/r0/waves/complete/receipt.json" - ), - ) - approval_payload = { - "schema_version": (launcher.SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION), - "stage_id": "qwen-l0-rl-l1", - "wave": "smoke", - "source_git_sha": consumer_source_git_sha, - "evidence_source_git_sha": producer_source_git_sha, - "parent_stage_id": "qwen-l0-sft", - "parent_wave": "complete", - "parent_receipt_record_sha256": receipt.record_sha256, - "selected_parent_checkpoint": checkpoint, - "selected_parent_checkpoint_inventory_sha256": inventory_sha, - "source_transition": {"logical_sha256": "5" * 64}, - } - approval = SimpleNamespace( - payload=approval_payload, - payload_sha256="6" * 64, - record_sha256="7" * 64, - relative_path=( - "studies/representation-training-v1/stages/" - "qwen-l0-rl-l1/runs/r0/waves/smoke/approval.json" - ), - ) - return checkpoint, parent_manifest, receipt, approval - - -def _write_unit_sft_artifacts(config) -> None: - log = Path(config.log_path) - log.mkdir(parents=True, exist_ok=True) - checkpoints = [ - { - "name": f"{step:06d}", - "batch": step, - "epoch": 0, - "state_path": f"tinker://unit/weights/{step:06d}", - "sampler_path": f"tinker://unit/sampler_weights/{step:06d}", - } - for step in (3, 6, 9) - ] - checkpoints.append( - { - "name": "final", - "batch": 0, - "epoch": 1, - "state_path": "tinker://unit/weights/final", - "sampler_path": "tinker://unit/sampler_weights/final", - } - ) - (log / "checkpoints.jsonl").write_text( - "".join(json.dumps(value) + "\n" for value in checkpoints), - encoding="utf-8", - ) - (log / "metrics.jsonl").write_text( - json.dumps({"step": 10, "train_mean_nll": 0.5}) + "\n", - encoding="utf-8", - ) - - -class _FakeRun: - def __init__(self, module, kwargs): - self._module = module - self.id = kwargs["id"] - self.entity = kwargs["entity"] - self.project = kwargs["project"] - self.name = kwargs["name"] - self.group = kwargs["group"] - self.job_type = kwargs["job_type"] - self.tags = kwargs["tags"] - self.resumed = kwargs["resume"] == "must" - self.url = f"https://wandb.example/{self.id}" - self.settings = SimpleNamespace( - root_dir=kwargs["dir"], - resume=kwargs["resume"], - ) - self.dir = str(Path(kwargs["dir"]) / "wandb" / self.id / "files") - Path(self.dir).mkdir(parents=True, exist_ok=True) - self.finished = False - - def finish(self): - self.finished = True - self._module.run = None - - -class _FakeConfig: - def __init__(self, initial=None): - self.values = dict(initial or {}) - self.updates = [] - - def update(self, value, allow_val_change=None): - payload = dict(value) - changed = [ - key for key, item in payload.items() if key in self.values and self.values[key] != item - ] - if changed and allow_val_change is not True: - raise RuntimeError("config changed without allow_val_change: " + ", ".join(changed)) - self.values.update(payload) - self.updates.append( - { - "value": payload, - "allow_val_change": allow_val_change, - } - ) - - -class _FakeWandb: - def __init__(self, *, initial_config=None): - self.run = None - self.calls = [] - self.config = _FakeConfig(initial_config) - - def init(self, **kwargs): - self.calls.append(kwargs) - if kwargs.get("config") is not None: - self.config.update(kwargs["config"]) - self.run = _FakeRun(self, kwargs) - return self.run - - def finish(self): - if self.run is not None: - self.run.finish() - - -def test_sft_launch_records_before_main_and_binds_wandb( - tmp_path: Path, - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - source_sha = "a" * 40 - external = tmp_path / "external" - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.delenv("WANDB_RUN_ID", raising=False) - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: _preflight( - protocol, - "qwen-l0-sft", - "complete", - ), - ) - access_calls: list[str] = [] - - def validate_access(*, expected_entity): - access_calls.append(expected_entity) - assert os.environ["WANDB_ENTITY"] == expected_entity - assert os.environ["WANDB_PROJECT"] == protocol["tracking"]["project"] - assert os.environ["WANDB_RUN_ID"].startswith("pxct-") - assert os.environ["WANDB_RESUME"] == "never" - return {"username": "test", "entity": expected_entity} - - async def fake_main(config): - log = Path(config.log_path) - assert (log.parent / "run_manifest.json").is_file() - assert os.environ["WANDB_RUN_ID"].startswith("pxct-") - assert os.environ["WANDB_RESUME"] == "never" - assert os.environ["WANDB_MODE"] == "online" - assert os.environ["WANDB_JOB_TYPE"] == "sft" - _write_unit_sft_artifacts(config) - - report = asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - sft_main=fake_main, - wandb_access_validator=validate_access, - ) - ) - assert report["status"] == "complete" - assert access_calls == [protocol["tracking"]["entity"]] - assert report["checkpoint"]["state_path"] == "tinker://unit/weights/final" - assert "WANDB_RUN_ID" not in os.environ - - async def forbidden_main(_config): - raise AssertionError("completed wave must not call Tinker again") - - replay = asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - sft_main=forbidden_main, - wandb_access_validator=lambda **_kwargs: (_ for _ in ()).throw( - AssertionError("completed wave must not touch W&B") - ), - ) - ) - assert replay["status"] == "already_complete" - - -def test_source_is_revalidated_after_preflight_before_any_live_boundary( - tmp_path: Path, - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - source_sha = "9" * 40 - events: list[str] = [] - source_checks = 0 - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - - def validate_source(_root, expected): - nonlocal source_checks - source_checks += 1 - events.append(f"source-{source_checks}") - if source_checks == 1: - return expected - raise ValueError("paid training requires a completely clean worktree") - - monkeypatch.setattr(launcher, "validate_source_sha", validate_source) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: ( - events.append("preflight") or _preflight(protocol, "qwen-mixed-sft", "complete") - ), - ) - - async def forbidden_main(_config): - events.append("training") - raise AssertionError("source race reached training") - - def forbidden_wandb(**_kwargs): - events.append("wandb") - raise AssertionError("source race reached W&B") - - with pytest.raises( - launcher.TrainingLaunchError, - match="source changed after preflight", - ): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id="qwen-mixed-sft", - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=tmp_path / "external", - confirmation=protocol["launch"]["confirmation_token"], - sft_main=forbidden_main, - wandb_access_validator=forbidden_wandb, - ) - ) - assert events == ["source-1", "preflight", "source-2"] - - -def test_already_complete_rejects_minimal_receipt_before_live_boundaries( - tmp_path: Path, - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - source_sha = "8" * 40 - external = tmp_path / "external" - stage = stage_spec(protocol, "qwen-mixed-sft") - key = StageKey(STUDY_ID, stage.stage_id, "r0") - preflight = _preflight(protocol, stage.stage_id, "complete") - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - tracking = launcher._tracking_binding( - protocol=protocol, - source_git_sha=source_sha, - key=key, - stage=stage, - store=store, - ) - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - launcher._manifest_payload( - protocol=protocol, - source_git_sha=source_sha, - stage=stage, - preflight=preflight, - tracking=tracking, - parent=None, - ), - ) - store.write_wave_receipt( - key, - wave="complete", - payload={ - "stage_id": stage.stage_id, - "wave": "complete", - "run_manifest_record_sha256": manifest.record_sha256, - }, - ) - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: preflight, - ) - with pytest.raises( - launcher.TrainingLaunchError, - match="receipt fields differ", - ): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id=stage.stage_id, - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - sft_main=lambda _config: (_ for _ in ()).throw( - AssertionError("minimal receipt reached training") - ), - wandb_access_validator=lambda **_kwargs: (_ for _ in ()).throw( - AssertionError("minimal receipt reached W&B") - ), - ) - ) - - -def test_already_complete_rejects_forged_receipt_bindings( - tmp_path: Path, - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - source_sha = "7" * 40 - external = tmp_path / "external" - key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: _preflight(protocol, key.stage_id, "complete"), - ) - - async def fake_main(config): - _write_unit_sft_artifacts(config) - - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id=key.stage_id, - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - sft_main=fake_main, - wandb_access_validator=lambda **_kwargs: {"entity": "unit"}, - ) - ) - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - original = store.load_wave_receipt(key, wave="complete") - assert original is not None - receipt_path = external.joinpath(*original.relative_path.parts) - cases = [ - ( - "final optimizer metric", - lambda payload: payload["final_training_metric"].update({"step": 0}), - ), - ( - "canonical training artifacts", - lambda payload: payload["final_training_metric"].update({"train_mean_nll": 0.125}), - ), - ( - "canonical training artifacts", - lambda payload: payload["local_artifact_sha256"].update({"metrics.jsonl": "f" * 64}), - ), - ( - "candidate_inventory", - lambda payload: payload["candidate_inventory"].update({"expected_count": 1}), - ), - ( - "invocation is missing or differs", - lambda payload: payload.update({"invocation_record_sha256": "0" * 64}), - ), - ] - for message, mutate in cases: - receipt_path.unlink() - forged = copy.deepcopy(original.payload) - mutate(forged) - with store.acquire_stage_lock(key): - store.write_wave_receipt( - key, - wave="complete", - payload=forged, - ) - with pytest.raises(launcher.TrainingLaunchError, match=message): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id=key.stage_id, - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - sft_main=lambda _config: (_ for _ in ()).throw( - AssertionError("forged receipt reached training") - ), - wandb_access_validator=lambda **_kwargs: (_ for _ in ()).throw( - AssertionError("forged receipt reached W&B") - ), - ) - ) - - receipt_path.unlink() - with store.acquire_stage_lock(key): - store.write_wave_receipt( - key, - wave="complete", - payload=original.payload, - ) - metrics_path = store.tinker_log_path(key) / "metrics.jsonl" - original_metrics = metrics_path.read_bytes() - metrics_path.write_text( - json.dumps({"step": 10, "train_mean_nll": 0.125}) + "\n", - encoding="utf-8", - ) - try: - with pytest.raises( - launcher.TrainingLaunchError, - match="canonical training artifacts", - ): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id=key.stage_id, - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - sft_main=lambda _config: (_ for _ in ()).throw( - AssertionError("mutated metrics reached training") - ), - wandb_access_validator=lambda **_kwargs: (_ for _ in ()).throw( - AssertionError("mutated metrics reached W&B") - ), - ) - ) - finally: - metrics_path.write_bytes(original_metrics) - - -def test_completed_rl_receipt_requires_every_candidate_and_sample( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - source_sha = "6" * 40 - external = tmp_path / "external" - stage = stage_spec(protocol, "inkling-l4-rl") - key = StageKey(STUDY_ID, stage.stage_id, "r0") - wave = "smoke" - preflight = _preflight(protocol, stage.stage_id, wave) - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - tracking = launcher._tracking_binding( - protocol=protocol, - source_git_sha=source_sha, - key=key, - stage=stage, - store=store, - ) - invocation_id = "smoke-unit" - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - launcher._manifest_payload( - protocol=protocol, - source_git_sha=source_sha, - stage=stage, - preflight=preflight, - tracking=tracking, - parent=None, - ), - ) - resolved = launcher._resolved_launch_binding( - repo_root=REPO_ROOT, - protocol=protocol, - stage=stage, - wave_name=wave, - store=store, - key=key, - parent=None, - resume_batch=None, - ) - invocation = store.begin_invocation( - key, - invocation_id=invocation_id, - payload=launcher._invocation_payload( - protocol=protocol, - stage=stage, - wave_name=wave, - source_git_sha=source_sha, - invocation_id=invocation_id, - manifest=manifest, - preflight=preflight, - tracking=tracking, - resume_batch=None, - resolved_launch=resolved, - parent=None, - ), - ) - recipe = protocol["recipes"][stage.recipe_key] - expected = int(recipe["groups_per_batch"]) * int(recipe["group_size"]) - for index in range(expected): - task_id = f"unit-{index:02d}" - store.write_candidate_sample( - key, - invocation_id=invocation_id, - step=0, - task_id=task_id, - attempt=1, - payload={"index": index}, - ) - store.write_candidate( - key, - invocation_id=invocation_id, - step=0, - task_id=task_id, - attempt=1, - payload={"index": index}, - ) - candidate_root = store.tinker_log_path(key).parent / "candidates" / invocation_id - candidate_files = sorted(candidate_root.rglob("evaluation.json")) - log_path = store.tinker_log_path(key) - log_path.mkdir(parents=True, exist_ok=True) - (log_path / "checkpoints.jsonl").write_text( - json.dumps( - { - "name": "000001", - "batch": 1, - "epoch": None, - "state_path": "tinker://unit/weights/000001", - "sampler_path": "tinker://unit/sampler_weights/000001", - } - ) - + "\n", - encoding="utf-8", - ) - (log_path / "metrics.jsonl").write_text( - json.dumps({"step": 0, "optim/entropy": 0.5}) + "\n", - encoding="utf-8", - ) - receipt = store.write_wave_receipt( - key, - wave=wave, - payload=launcher._checkpoint_receipt( - stage=stage, - wave_name=wave, - log_path=log_path, - manifest=manifest, - preflight=preflight, - invocation=invocation, - tracking=None, - ), - ) - launcher._validate_completed_wave_receipt( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name=wave, - manifest=manifest, - preflight=preflight, - parent=None, - receipt=receipt, - ) - checkpoints_path = log_path / "checkpoints.jsonl" - original_checkpoints = checkpoints_path.read_bytes() - metrics_path = log_path / "metrics.jsonl" - original_metrics = metrics_path.read_bytes() - with checkpoints_path.open("ab") as stream: - stream.write( - ( - json.dumps( - { - "name": "final", - "batch": 1, - "state_path": "tinker://unit/weights/final", - "sampler_path": "tinker://unit/sampler_weights/final", - } - ) - + "\n" - + json.dumps( - { - "name": "000005", - "batch": 5, - "state_path": "tinker://unit/weights/000005", - "sampler_path": "tinker://unit/sampler_weights/000005", - } - ) - + "\n" - ).encode("utf-8") - ) - with metrics_path.open("ab") as stream: - stream.write( - (json.dumps({"step": 4, "optim/entropy": 0.25}) + "\n").encode("utf-8") - ) - # A later wave appends to Cookbook's shared log. The earlier receipt - # remains reconstructable from its immutable prefix snapshot. - launcher._validate_completed_wave_receipt( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name=wave, - manifest=manifest, - preflight=preflight, - parent=None, - receipt=receipt, - ) - checkpoints_path.write_bytes(original_checkpoints) - metrics_path.write_bytes(original_metrics) - metrics_path.write_text( - json.dumps({"step": 0, "optim/entropy": 0.25}) + "\n", - encoding="utf-8", - ) - with pytest.raises( - launcher.TrainingLaunchError, - match="canonical training artifacts", - ): - launcher._validate_completed_wave_receipt( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name=wave, - manifest=manifest, - preflight=preflight, - parent=None, - receipt=receipt, - ) - metrics_path.write_bytes(original_metrics) - checkpoints_path.unlink() - with pytest.raises( - launcher.TrainingLaunchError, - match="without a resumable checkpoint", - ): - launcher._validate_completed_wave_receipt( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name=wave, - manifest=manifest, - preflight=preflight, - parent=None, - receipt=receipt, - ) - checkpoints_path.write_bytes(original_checkpoints) - candidate_files[0].unlink() - with pytest.raises( - launcher.TrainingLaunchError, - match="candidate_inventory", - ): - launcher._validate_completed_wave_receipt( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - key=key, - stage=stage, - wave_name=wave, - manifest=manifest, - preflight=preflight, - parent=None, - receipt=receipt, - ) - - -def test_resume_pins_receipt_checkpoint_instead_of_later_cookbook_final( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l0") - log_path = tmp_path / "tinker" - log_path.mkdir() - numbered = { - "name": "000001", - "batch": 1, - "state_path": "tinker://unit/weights/000001", - "sampler_path": "tinker://unit/sampler_weights/000001", - } - terminal_final = { - "name": "final", - "batch": 1, - "state_path": "tinker://unit/weights/final", - "sampler_path": "tinker://unit/sampler_weights/final", - } - (log_path / "checkpoints.jsonl").write_text( - json.dumps(numbered) + "\n" + json.dumps(terminal_final) + "\n", - encoding="utf-8", - ) - prior = SimpleNamespace( - payload={ - "checkpoint": { - **numbered, - "epoch": None, - "final": True, - } - } - ) - - # Unsealed Cookbook behavior selects the later, different final URI. - assert ( - launcher.checkpoint_utils.get_last_checkpoint(str(log_path)).state_path - == terminal_final["state_path"] - ) - binding = launcher._validate_resume_state( - stage=stage, - wave_name="step-5", - log_path=log_path, - prior_wave_receipt=prior, - ) - assert binding is not None - assert binding.batch == 1 - assert binding.checkpoint["state_path"] == numbered["state_path"] - invocation = launcher._invocation_payload( - protocol=protocol, - stage=stage, - wave_name="step-5", - source_git_sha="a" * 40, - invocation_id="resume-unit", - manifest=SimpleNamespace(record_sha256="b" * 64), - preflight={"preflight_sha256": "c" * 64}, - tracking={"run_id": "unit"}, - resume_batch=binding.batch, - resume_checkpoint=binding.checkpoint, - resolved_launch={"logical_sha256": "d" * 64}, - parent=None, - ) - assert invocation["resume_batch"] == 1 - assert invocation["resume_checkpoint"] == binding.checkpoint - - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="step-5", - log_path=log_path, - prior_wave_receipt=prior, - expected=binding, - ): - selected = launcher.checkpoint_utils.get_last_checkpoint(str(log_path)) - assert selected.state_path == numbered["state_path"] - assert selected.sampler_path == numbered["sampler_path"] - - with (log_path / "checkpoints.jsonl").open("a", encoding="utf-8") as stream: - stream.write( - json.dumps( - { - "name": "000002", - "batch": 2, - "state_path": "tinker://unit/weights/000002", - "rolling": True, - } - ) - + "\n" - ) - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="step-5", - log_path=log_path, - prior_wave_receipt=prior, - expected=binding, - ): - with pytest.raises( - launcher.TrainingLaunchError, - match="changed after the immutable invocation", - ): - launcher.checkpoint_utils.get_last_checkpoint(str(log_path)) - - -def _write_fresh_cookbook_bootstrap(log_path: Path) -> None: - log_path.mkdir(parents=True, exist_ok=True) - (log_path / "config.json").write_text('{"max_steps": 1}\n', encoding="utf-8") - (log_path / "code.diff").write_text("", encoding="utf-8") - (log_path / "logs.log").write_text("bootstrap\n", encoding="utf-8") - - -def test_fresh_cookbook_bootstrap_is_allowed_only_inside_sealed_boundary( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l0") - log_path = tmp_path / "tinker" - - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="smoke", - log_path=log_path, - prior_wave_receipt=None, - expected=None, - ): - _write_fresh_cookbook_bootstrap(log_path) - assert launcher.checkpoint_utils.get_last_checkpoint(str(log_path)) is None - - with pytest.raises( - launcher.TrainingLaunchError, - match="no resumable checkpoint", - ): - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="smoke", - log_path=log_path, - prior_wave_receipt=None, - expected=None, - ): - pass - - -def test_fresh_sft_resume_boundary_precedes_logging_and_rejects_debris( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-l0-sft") - clean_log = tmp_path / "clean-sft" - - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="complete", - log_path=clean_log, - prior_wave_receipt=None, - expected=None, - ): - assert launcher.checkpoint_utils.get_last_checkpoint(str(clean_log)) is None - - dirty_log = tmp_path / "dirty-sft" - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="complete", - log_path=dirty_log, - prior_wave_receipt=None, - expected=None, - ): - dirty_log.mkdir() - (dirty_log / "logs.log").write_text("unexpected\n", encoding="utf-8") - with pytest.raises( - launcher.TrainingLaunchError, - match="SFT resume boundary contains unexpected training artifacts", - ): - launcher.checkpoint_utils.get_last_checkpoint(str(dirty_log)) - - -@pytest.mark.parametrize("unexpected_name", ["metrics.jsonl", "checkpoints.jsonl"]) -def test_fresh_cookbook_bootstrap_rejects_unexpected_training_artifacts( - tmp_path: Path, - unexpected_name: str, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l0") - log_path = tmp_path / "tinker" - - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="smoke", - log_path=log_path, - prior_wave_receipt=None, - expected=None, - ): - _write_fresh_cookbook_bootstrap(log_path) - (log_path / unexpected_name).write_text("", encoding="utf-8") - with pytest.raises( - launcher.TrainingLaunchError, - match="unexpected training artifacts", - ): - launcher.checkpoint_utils.get_last_checkpoint(str(log_path)) - - -def test_fresh_cookbook_bootstrap_rejects_invalid_config_or_non_file( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l0") - - invalid_config = tmp_path / "invalid-config" - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="smoke", - log_path=invalid_config, - prior_wave_receipt=None, - expected=None, - ): - _write_fresh_cookbook_bootstrap(invalid_config) - (invalid_config / "config.json").write_text("{", encoding="utf-8") - with pytest.raises( - launcher.TrainingLaunchError, - match="config is not valid JSON", - ): - launcher.checkpoint_utils.get_last_checkpoint(str(invalid_config)) - - non_file = tmp_path / "non-file" - with launcher._sealed_cookbook_resume( - stage=stage, - wave_name="smoke", - log_path=non_file, - prior_wave_receipt=None, - expected=None, - ): - _write_fresh_cookbook_bootstrap(non_file) - (non_file / "code.diff").unlink() - (non_file / "code.diff").mkdir() - with pytest.raises( - launcher.TrainingLaunchError, - match="unexpected training artifacts", - ): - launcher.checkpoint_utils.get_last_checkpoint(str(non_file)) - - -def test_resume_rejects_unreceipted_terminal_checkpoint( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l0") - log_path = tmp_path / "tinker" - log_path.mkdir() - (log_path / "checkpoints.jsonl").write_text( - json.dumps( - { - "name": "final", - "batch": 1, - "state_path": "tinker://unit/weights/final", - "sampler_path": "tinker://unit/sampler_weights/final", - } - ) - + "\n", - encoding="utf-8", - ) - with pytest.raises( - launcher.TrainingLaunchError, - match="wave ceiling without an immutable receipt", - ): - launcher._validate_resume_state( - stage=stage, - wave_name="smoke", - log_path=log_path, - prior_wave_receipt=None, - ) - - -def test_builtin_sft_records_observed_wandb_before_service_client( - tmp_path: Path, - monkeypatch, -) -> None: - from tinker_cookbook.supervised import train as supervised_train - from tinker_cookbook.utils import ml_log - - protocol = load_protocol(REPO_ROOT) - source_sha = "b" * 40 - external = tmp_path / "external" - fake_wandb = _FakeWandb() - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: _preflight( - protocol, - "qwen-l0-sft", - "complete", - ), - ) - monkeypatch.setattr(ml_log, "wandb", fake_wandb) - monkeypatch.setattr(ml_log, "_wandb_available", True) - real_import = launcher.importlib.import_module - monkeypatch.setattr( - launcher.importlib, - "import_module", - lambda name: fake_wandb if name == "wandb" else real_import(name), - ) - service_client_markers = [] - - def validate_access(*, expected_entity): - assert os.environ["WANDB_ENTITY"] == expected_entity - assert os.environ["WANDB_RUN_ID"].startswith("pxct-") - return {"username": "test", "entity": expected_entity} - - async def fake_builtin_main(config): - logger = ml_log.setup_logging( - log_dir=config.log_path, - wandb_project=config.wandb_project, - wandb_name=config.wandb_name, - config=config, - do_configure_logging_module=False, - ) - tracking_paths = list(external.rglob("tracking.json")) - assert len(tracking_paths) == 1 - tracking_document = json.loads(tracking_paths[0].read_text(encoding="utf-8")) - assert tracking_document["payload"]["observed"]["run_id"] == fake_wandb.calls[0]["id"] - assert service_client_markers == [] - service_client_markers.append("created") - logger.close() - _write_unit_sft_artifacts(config) - - monkeypatch.setattr(supervised_train, "main", fake_builtin_main) - report = asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave_name="complete", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - wandb_access_validator=validate_access, - ) - ) - assert report["status"] == "complete" - assert service_client_markers == ["created"] - assert fake_wandb.calls[0]["config"] is None - assert len(fake_wandb.config.updates) == 2 - assert fake_wandb.config.updates[0]["allow_val_change"] is False - assert fake_wandb.config.updates[1]["allow_val_change"] is None - assert fake_wandb.config.updates[0]["value"] == fake_wandb.config.updates[1]["value"] - assert "dataset_builder" in fake_wandb.config.values - key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - tracking_paths = list(external.rglob("tracking.json")) - tracking_document = json.loads(tracking_paths[0].read_text(encoding="utf-8")) - invocation_id = tracking_document["key"]["invocation_id"] - tracking = store.load_invocation_tracking( - key, - invocation_id=invocation_id, - ) - receipt = store.load_wave_receipt(key, wave="complete") - assert tracking is not None - assert receipt is not None - assert receipt.payload["invocation_tracking_record_sha256"] == tracking.record_sha256 - - -def test_builtin_rl_records_observed_wandb_before_service_client( - tmp_path: Path, - monkeypatch, -) -> None: - from rl.track_b import tml_shim - from tinker_cookbook.rl import train as rl_train - from tinker_cookbook.utils import ml_log - - protocol = load_protocol(REPO_ROOT) - source_sha = "c" * 40 - external = tmp_path / "external" - fake_wandb = _FakeWandb() - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: _preflight( - protocol, - "inkling-l4-rl", - "smoke", - ), - ) - monkeypatch.setattr( - tml_shim, - "install_inkling_runtime", - lambda **_kwargs: None, - ) - monkeypatch.setattr(ml_log, "wandb", fake_wandb) - monkeypatch.setattr(ml_log, "_wandb_available", True) - real_import = launcher.importlib.import_module - monkeypatch.setattr( - launcher.importlib, - "import_module", - lambda name: fake_wandb if name == "wandb" else real_import(name), - ) - service_client_markers = [] - - async def fake_builtin_main(config): - ml_log.setup_logging( - log_dir=config.log_path, - wandb_project=config.wandb_project, - wandb_name=config.wandb_name, - config=config, - do_configure_logging_module=False, - ) - tracking_paths = list(external.rglob("tracking.json")) - assert len(tracking_paths) == 1 - tracking_document = json.loads(tracking_paths[0].read_text(encoding="utf-8")) - assert tracking_document["payload"]["observed"]["run_id"] == fake_wandb.calls[0]["id"] - service_client_markers.append("created") - raise RuntimeError("stop before paid RL") - - monkeypatch.setattr(rl_train, "main", fake_builtin_main) - with pytest.raises(RuntimeError, match="stop before paid RL"): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id="inkling-l4-rl", - wave_name="smoke", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - wandb_access_validator=lambda **_kwargs: {}, - ) - ) - assert service_client_markers == ["created"] - assert fake_wandb.run is None - assert fake_wandb.calls[0]["config"] is None - assert len(fake_wandb.config.updates) == 2 - assert fake_wandb.config.updates[0]["allow_val_change"] is False - assert fake_wandb.config.updates[1]["allow_val_change"] is None - assert fake_wandb.config.updates[0]["value"] == fake_wandb.config.updates[1]["value"] - assert "dataset_builder" in fake_wandb.config.values - assert len(list(external.rglob("tracking.json"))) == 1 - - -def test_sealed_wandb_initialization_overrides_cookbook_identity( - tmp_path: Path, - monkeypatch, -) -> None: - from tinker_cookbook.utils import ml_log - - wandb_dir = tmp_path / "sealed-wandb" - cookbook_dir = tmp_path / "cookbook-log" - wandb_dir.mkdir() - binding = { - "entity": "unit-entity", - "project": "unit-project", - "run_id": "pxct-unit-run", - "name": "unit/name", - "group": "unit/group", - "job_type": "sft", - "tags": ["study", "stage", "r0"], - "directory": str(wandb_dir), - } - - fake = _FakeWandb() - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(ml_log, "wandb", fake) - monkeypatch.setattr(ml_log, "_wandb_available", True) - original_init = fake.init - observed = [] - with launcher._sealed_wandb_initialization( - binding, - resume=True, - wandb_module=fake, - on_initialized=observed.append, - ): - logger = ml_log.setup_logging( - log_dir=str(cookbook_dir), - wandb_project="unit-project", - wandb_name="unit/name", - config=None, - do_configure_logging_module=False, - ) - logger.close() - - assert fake.init == original_init - assert len(fake.calls) == 1 - call = fake.calls[0] - assert call["entity"] == binding["entity"] - assert call["project"] == binding["project"] - assert call["id"] == binding["run_id"] - assert call["name"] == binding["name"] - assert call["group"] == binding["group"] - assert call["job_type"] == binding["job_type"] - assert call["tags"] == tuple(binding["tags"]) - assert call["dir"] == binding["directory"] - assert call["mode"] == "online" - assert call["resume"] == "must" - assert call["save_code"] is False - assert call["config"] is None - assert observed == [ - { - "run_id": binding["run_id"], - "entity": binding["entity"], - "project": binding["project"], - "name": binding["name"], - "group": binding["group"], - "job_type": binding["job_type"], - "tags": binding["tags"], - "resumed": True, - "url": f"https://wandb.example/{binding['run_id']}", - "requested_directory": binding["directory"], - "resolved_root_directory": binding["directory"], - "files_directory": str( - Path(binding["directory"]) / "wandb" / binding["run_id"] / "files" - ), - "requested_resume": "must", - "resolved_resume": "must", - "hparams_preseeded": False, - "hparams_allow_val_change": False, - } - ] - - -def test_sealed_wandb_initialization_allows_resume_hparam_changes_once( - tmp_path: Path, - monkeypatch, -) -> None: - from tinker_cookbook.utils import ml_log - - wandb_dir = tmp_path / "sealed-wandb" - cookbook_dir = tmp_path / "cookbook-log" - wandb_dir.mkdir() - binding = { - "entity": "unit-entity", - "project": "unit-project", - "run_id": "pxct-unit-run", - "name": "unit/name", - "group": "unit/group", - "job_type": "rl", - "tags": ["study", "stage", "r0"], - "directory": str(wandb_dir), - } - resumed_config = { - "max_steps": 6, - "wave": "pilot", - } - - fake = _FakeWandb( - initial_config={ - "max_steps": 1, - "wave": "smoke", - } - ) - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(ml_log, "wandb", fake) - monkeypatch.setattr(ml_log, "_wandb_available", True) - observed = [] - with launcher._sealed_wandb_initialization( - binding, - resume=True, - wandb_module=fake, - on_initialized=observed.append, - ): - logger = ml_log.setup_logging( - log_dir=str(cookbook_dir), - wandb_project=binding["project"], - wandb_name=binding["name"], - config=resumed_config, - do_configure_logging_module=False, - ) - logger.close() - - assert fake.calls[0]["config"] is None - assert len(fake.config.updates) == 2 - assert fake.config.updates[0] == { - "value": resumed_config, - "allow_val_change": True, - } - assert fake.config.updates[1] == { - "value": resumed_config, - "allow_val_change": None, - } - assert fake.config.values["max_steps"] == 6 - assert fake.config.values["wave"] == "pilot" - assert observed[0]["hparams_preseeded"] is True - assert observed[0]["hparams_allow_val_change"] is True - - -def test_sealed_wandb_initialization_rejects_live_identity_mismatch( - tmp_path: Path, -) -> None: - wandb_dir = tmp_path / "sealed-wandb" - wandb_dir.mkdir() - binding = { - "entity": "unit-entity", - "project": "unit-project", - "run_id": "pxct-expected", - "name": "unit/name", - "group": "unit/group", - "job_type": "rl", - "tags": ["study"], - "directory": str(wandb_dir), - } - - class MismatchedRun: - id = "random-id" - entity = binding["entity"] - project = binding["project"] - name = binding["name"] - group = binding["group"] - job_type = binding["job_type"] - tags = tuple(binding["tags"]) - resumed = False - url = "https://wandb.example/random-id" - settings = SimpleNamespace( - root_dir=binding["directory"], - resume="never", - ) - dir = str(Path(binding["directory"]) / "wandb" / "random-id" / "files") - - def __init__(self): - self.finished = False - - def finish(self): - self.finished = True - - run = MismatchedRun() - Path(run.dir).mkdir(parents=True) - fake = SimpleNamespace(run=None, init=lambda **_kwargs: run) - with pytest.raises( - launcher.TrainingLaunchError, - match="live W&B run differs", - ): - with launcher._sealed_wandb_initialization( - binding, - resume=False, - wandb_module=fake, - ): - fake.init(project=binding["project"], name=binding["name"]) - assert run.finished is True - - -def test_sealed_wandb_cleanup_does_not_mask_training_failure( - tmp_path: Path, -) -> None: - wandb_dir = tmp_path / "sealed-wandb" - wandb_dir.mkdir() - binding = { - "entity": "unit-entity", - "project": "unit-project", - "run_id": "pxct-unit-run", - "name": "unit/name", - "group": "unit/group", - "job_type": "sft", - "tags": ["study"], - "directory": str(wandb_dir), - } - - class FailingFinishRun: - id = binding["run_id"] - entity = binding["entity"] - project = binding["project"] - name = binding["name"] - group = binding["group"] - job_type = binding["job_type"] - tags = tuple(binding["tags"]) - resumed = False - url = "https://wandb.example/pxct-unit-run" - settings = SimpleNamespace( - root_dir=binding["directory"], - resume="never", - ) - dir = str(Path(binding["directory"]) / "wandb" / id / "files") - - def finish(self): - raise RuntimeError("finish failed") - - run = FailingFinishRun() - Path(run.dir).mkdir(parents=True) - - def module_finish(): - raise RuntimeError("module finish failed") - - fake = SimpleNamespace( - run=None, - init=lambda **_kwargs: run, - finish=module_finish, - ) - original_init = fake.init - with pytest.raises(RuntimeError, match="trainer failed"): - with launcher._sealed_wandb_initialization( - binding, - resume=False, - wandb_module=fake, - ): - fake.init(project=binding["project"], name=binding["name"]) - raise RuntimeError("trainer failed") - assert fake.init == original_init - - -def test_bad_confirmation_cannot_reach_preflight_or_main( - tmp_path: Path, - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr(launcher, "load_protocol", lambda *_a, **_k: protocol) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: (_ for _ in ()).throw(AssertionError("preflight should not run")), - ) - with pytest.raises(launcher.TrainingLaunchError, match="confirmation"): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id="qwen-l0-sft", - wave_name="complete", - replicate_id="r0", - expected_source_sha="a" * 40, - external_root=tmp_path / "external", - confirmation="wrong", - sft_main=lambda _config: (_ for _ in ()).throw( - AssertionError("main should not run") - ), - ) - ) - - -def test_resolved_configs_seal_no_hidden_validation_and_raw_iou( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - store = TrainingStore( - repo_root=REPO_ROOT, - external_root=tmp_path / "external", - ) - for stage_id, wave in ( - ("qwen-mixed-sft", "complete"), - ("inkling-l4-rl", "smoke"), - ): - key = StageKey(STUDY_ID, stage_id, "r0") - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - {"source_git_sha": "a" * 40}, - ) - spec = stage_spec(protocol, stage_id) - resolved_launch = launcher._resolved_launch_binding( - repo_root=REPO_ROOT, - protocol=protocol, - stage=spec, - wave_name=wave, - store=store, - key=key, - parent=None, - resume_batch=None, - ) - invocation = store.begin_invocation( - key, - invocation_id="unit-invocation", - payload={ - "invocation_id": "unit-invocation", - "run_manifest_record_sha256": manifest.record_sha256, - "source_git_sha": "a" * 40, - "protocol_logical_sha256": protocol["logical_sha256"], - "policy": {}, - "initial_checkpoint": None, - "resume_batch": None, - "resolved_launch": resolved_launch, - }, - ) - config = launcher.build_tinker_config( - repo_root=REPO_ROOT, - protocol=protocol, - stage=spec, - wave_name=wave, - store=store, - key=key, - parent=None, - invocation=invocation, - ) - if stage_id == "qwen-mixed-sft": - assert config.max_steps == 55 - assert config.learning_rate == 1e-4 - assert config.eval_every == 0 - assert config.dataset_builder.include_validation is False - assert config.dataset_builder.schedule_seed == 90210 - else: - assert config.max_steps == 1 - assert config.max_tokens == 60_000 - assert config.eval_every == 0 - assert config.dataset_builder.validation_canaries == 0 - assert config.dataset_builder.require_prefix_replay is False - assert config.dataset_builder.reward_policy == "raw_iou" - assert config.dataset_builder.schedule_seed == 90210 - - -def test_tinker_config_rejects_invocation_policy_drift(tmp_path: Path) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "inkling-l4-rl") - key = StageKey(STUDY_ID, stage.stage_id, "r0") - store = TrainingStore( - repo_root=REPO_ROOT, - external_root=tmp_path / "external", - ) - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - {"source_git_sha": "a" * 40}, - ) - resolved = launcher._resolved_launch_binding( - repo_root=REPO_ROOT, - protocol=protocol, - stage=stage, - wave_name="smoke", - store=store, - key=key, - parent=None, - resume_batch=None, - ) - resolved["sampling"]["temperature"] = 0.5 - invocation = store.begin_invocation( - key, - invocation_id="unit-drift", - payload={ - "invocation_id": "unit-drift", - "run_manifest_record_sha256": manifest.record_sha256, - "source_git_sha": "a" * 40, - "protocol_logical_sha256": protocol["logical_sha256"], - "policy": {}, - "initial_checkpoint": None, - "resume_batch": None, - "resolved_launch": resolved, - }, - ) - with pytest.raises( - launcher.TrainingLaunchError, - match="differs from the resolved", - ): - launcher.build_tinker_config( - repo_root=REPO_ROOT, - protocol=protocol, - stage=stage, - wave_name="smoke", - store=store, - key=key, - parent=None, - invocation=invocation, - ) - - -def test_inkling_extension_rechecks_exact_pilot_evidence( - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "inkling-l4-rl") - key = StageKey(STUDY_ID, stage.stage_id, "r0") - # This test isolates the launcher's downstream binding of an already - # canonically validated approval to the exact pilot receipt. Canonical - # report reconstruction is covered independently; keep it out of this - # deliberately synthetic store. - monkeypatch.setattr( - launcher, - "validate_same_source_wave_approval", - lambda **_kwargs: None, - ) - checkpoint = { - "name": "000006", - "batch": 6, - "epoch": 0, - "final": False, - "state_path": "tinker://unit/weights/000006", - "sampler_path": "tinker://unit/sampler_weights/000006", - } - prior = SimpleNamespace( - record_sha256="b" * 64, - payload={"checkpoint": checkpoint}, - ) - approval_payload = { - "schema_version": launcher.APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "promotion-receipt-and-explicit-human-approval", - "stage_id": stage.stage_id, - "wave": "coverage-extension", - "source_git_sha": "a" * 40, - "protocol_logical_sha256": protocol["logical_sha256"], - "evaluation": { - "stage_id": stage.stage_id, - "wave": "pilot", - "panel": "inkling-promotion", - "training_wave_receipt_record_sha256": prior.record_sha256, - "checkpoint": { - **checkpoint, - "role": "wave-ceiling", - "training_progress_fraction": 6 / 14, - }, - }, - } - - class FakeStore: - def __init__(self, approval): - self.approval = approval - - def load_wave_receipt(self, _key, *, wave): - assert wave == "pilot" - return prior - - def load_wave_approval(self, _key, *, wave): - assert wave == "coverage-extension" - return self.approval - - approval = SimpleNamespace(payload=approval_payload) - assert ( - launcher._approval_record( - repo_root=REPO_ROOT, - store=FakeStore(approval), - key=key, - stage=stage, - wave_name="coverage-extension", - source_git_sha="a" * 40, - protocol=protocol, - ) - is approval - ) - - changed = dict(approval_payload) - changed["evaluation"] = { - **approval_payload["evaluation"], - "training_wave_receipt_record_sha256": "c" * 64, - } - with pytest.raises(launcher.TrainingLaunchError, match="not bound"): - launcher._approval_record( - repo_root=REPO_ROOT, - store=FakeStore(SimpleNamespace(payload=changed)), - key=key, - stage=stage, - wave_name="coverage-extension", - source_git_sha="a" * 40, - protocol=protocol, - ) - - -def test_launcher_rejects_fabricated_minimal_same_source_approval() -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "inkling-l4-rl") - key = StageKey(STUDY_ID, stage.stage_id, "r0") - source_git_sha = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=REPO_ROOT, - text=True, - ).strip() - prior = SimpleNamespace( - record_sha256="b" * 64, - payload={ - "checkpoint": { - "name": "000006", - "batch": 6, - "epoch": 0, - "final": False, - "state_path": "tinker://unit/weights/000006", - "sampler_path": "tinker://unit/sampler_weights/000006", - }, - }, - ) - fabricated = SimpleNamespace( - payload={ - "schema_version": launcher.APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "promotion-receipt-and-explicit-human-approval", - "stage_id": stage.stage_id, - "wave": "coverage-extension", - "source_git_sha": source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - }, - ) - - class FakeStore: - @staticmethod - def load_wave_receipt(_key, *, wave): - assert wave == "pilot" - return prior - - @staticmethod - def load_wave_approval(_key, *, wave): - assert wave == "coverage-extension" - return fabricated - - with pytest.raises( - launcher.TrainingLaunchError, - match="same-source approval failed validation", - ): - launcher._approval_record( - repo_root=REPO_ROOT, - store=FakeStore(), - key=key, - stage=stage, - wave_name="coverage-extension", - source_git_sha=source_git_sha, - protocol=protocol, - ) - - -def test_cross_source_parent_requires_v2_and_child_manifest_records_it() -> None: - protocol = load_protocol(REPO_ROOT) - producer = "b" * 40 - consumer = "c" * 40 - checkpoint, parent_manifest, receipt, approval = _cross_source_parent_records( - protocol, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - key = StageKey(STUDY_ID, "qwen-l0-rl-l1", "r0") - stage = stage_spec(protocol, key.stage_id) - parent_key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - - class ParentStore: - def load_manifest(self, observed_key): - assert observed_key == parent_key - return parent_manifest - - def load_wave_receipt(self, observed_key, *, wave): - assert observed_key == parent_key - assert wave == "complete" - return receipt - - v1_payload = { - key: value - for key, value in approval.payload.items() - if key not in {"evidence_source_git_sha", "source_transition"} - } - v1_payload["schema_version"] = launcher.APPROVAL_SCHEMA_VERSION - v1 = SimpleNamespace( - payload=v1_payload, - payload_sha256=approval.payload_sha256, - record_sha256=approval.record_sha256, - relative_path=approval.relative_path, - ) - with pytest.raises( - launcher.TrainingLaunchError, - match="requires a v2 transition approval", - ): - launcher._parent_binding( - repo_root=REPO_ROOT, - protocol=protocol, - store=ParentStore(), - key=key, - stage=stage, - source_git_sha=consumer, - approval=v1, - existing_manifest=None, - ) - - parent = launcher._parent_binding( - repo_root=REPO_ROOT, - protocol=protocol, - store=ParentStore(), - key=key, - stage=stage, - source_git_sha=consumer, - approval=approval, - existing_manifest=None, - ) - assert parent is not None - assert parent.source_git_sha == producer - assert parent.checkpoint == checkpoint - assert parent.entry_approval is approval - manifest = launcher._manifest_payload( - protocol=protocol, - source_git_sha=consumer, - stage=stage, - preflight=_preflight(protocol, stage.stage_id, "smoke"), - tracking={"run_id": "unit"}, - parent=parent, - ) - assert manifest["source_transition"] == (launcher._child_source_transition_binding(approval)) - assert manifest["source_transition"]["producer_source_git_sha"] == (producer) - assert manifest["source_transition"]["consumer_source_git_sha"] == (consumer) - assert manifest["source_transition"]["authorized_child_wave"] == "smoke" - assert manifest["parent_receipt"]["record_sha256"] == (receipt.record_sha256) - - -def test_historical_l0_parent_uses_receipt_transition_without_evaluation( - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - producer = "1" * 40 - consumer = "2" * 40 - checkpoint, parent_manifest, receipt, approval = _cross_source_parent_records( - protocol, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - parent_manifest.payload["protocol_logical_sha256"] = "3" * 64 - approval.payload.update( - { - "schema_version": ( - launcher.PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ), - "decision": "approve", - "approval_gate": "parent-receipt-and-source-transition", - "protocol_logical_sha256": protocol["logical_sha256"], - "explicit_human_confirmation": True, - } - ) - key = StageKey(STUDY_ID, "qwen-l0-rl-l1", "r0") - parent_key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - stage = stage_spec(protocol, key.stage_id) - validation_calls: list[dict] = [] - monkeypatch.setattr( - launcher, - "validate_parent_receipt_transition_approval", - lambda **kwargs: validation_calls.append(dict(kwargs)), - ) - - class ParentStore: - @staticmethod - def load_wave_approval(observed_key, *, wave): - assert observed_key == key - assert wave == "smoke" - return approval - - @staticmethod - def load_manifest(observed_key): - assert observed_key == parent_key - return parent_manifest - - @staticmethod - def load_wave_receipt(observed_key, *, wave): - assert observed_key == parent_key - assert wave == "complete" - return receipt - - observed_approval = launcher._approval_record( - repo_root=REPO_ROOT, - store=ParentStore(), - key=key, - stage=stage, - wave_name="smoke", - source_git_sha=consumer, - protocol=protocol, - ) - assert observed_approval is approval - assert len(validation_calls) == 1 - - parent = launcher._parent_binding( - repo_root=REPO_ROOT, - protocol=protocol, - store=ParentStore(), - key=key, - stage=stage, - source_git_sha=consumer, - approval=approval, - existing_manifest=None, - ) - assert parent is not None - assert parent.checkpoint == checkpoint - assert parent.entry_approval is approval - child_transition = launcher._child_source_transition_binding(approval) - assert child_transition["approval_schema_version"] == ( - launcher.PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ) - assert child_transition["producer_source_git_sha"] == producer - assert child_transition["consumer_source_git_sha"] == consumer - - -def test_operational_retry_consumes_historical_parent_from_r0() -> None: - protocol = load_protocol(REPO_ROOT) - producer = "1" * 40 - consumer = "2" * 40 - checkpoint, parent_manifest, receipt, approval = _cross_source_parent_records( - protocol, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - parent_manifest.payload["protocol_logical_sha256"] = "3" * 64 - approval.payload.update( - { - "schema_version": ( - launcher.PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ), - "decision": "approve", - "approval_gate": "parent-receipt-and-source-transition", - "protocol_logical_sha256": protocol["logical_sha256"], - "explicit_human_confirmation": True, - "parent_replicate_id": "r0", - } - ) - key = StageKey(STUDY_ID, "qwen-l0-rl-l1", "r0-retry1") - parent_key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - stage = stage_spec(protocol, key.stage_id) - - class ParentStore: - @staticmethod - def load_manifest(observed_key): - assert observed_key == parent_key - return parent_manifest - - @staticmethod - def load_wave_receipt(observed_key, *, wave): - assert observed_key == parent_key - assert wave == "complete" - return receipt - - parent = launcher._parent_binding( - repo_root=REPO_ROOT, - protocol=protocol, - store=ParentStore(), - key=key, - stage=stage, - source_git_sha=consumer, - approval=approval, - existing_manifest=None, - ) - assert parent is not None - assert parent.replicate_id == "r0" - assert parent.checkpoint == checkpoint - - -def test_existing_child_manifest_revalidates_exact_v2_transition( - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - producer = "d" * 40 - consumer = "e" * 40 - checkpoint, parent_manifest, receipt, approval = _cross_source_parent_records( - protocol, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - key = StageKey(STUDY_ID, "qwen-l0-rl-l1", "r0") - parent_key = StageKey(STUDY_ID, "qwen-l0-sft", "r0") - stage = stage_spec(protocol, key.stage_id) - transition_binding = launcher._child_source_transition_binding(approval) - existing_manifest = SimpleNamespace( - payload={ - "source_git_sha": consumer, - "parent_receipt": { - "stage_id": "qwen-l0-sft", - "wave": "complete", - "record_sha256": receipt.record_sha256, - "checkpoint": checkpoint, - "checkpoint_inventory_logical_sha256": ( - receipt.payload["checkpoint_inventory"]["logical_sha256"] - ), - }, - "source_transition": transition_binding, - }, - ) - validation_calls: list[dict] = [] - monkeypatch.setattr( - launcher, - "validate_source_transition_approval", - lambda **kwargs: validation_calls.append(dict(kwargs)), - ) - - class ExistingStore: - def load_manifest(self, observed_key): - assert observed_key == parent_key - return parent_manifest - - def load_wave_receipt(self, observed_key, *, wave): - assert observed_key == parent_key - assert wave == "complete" - return receipt - - def load_wave_approval(self, observed_key, *, wave): - assert observed_key == key - assert wave == "smoke" - return approval - - parent = launcher._parent_binding( - repo_root=REPO_ROOT, - protocol=protocol, - store=ExistingStore(), - key=key, - stage=stage, - source_git_sha=consumer, - approval=None, - existing_manifest=existing_manifest, - ) - assert parent is not None - assert parent.entry_approval is approval - assert len(validation_calls) == 1 - assert validation_calls[0]["replicate_id"] == "r0" - assert validation_calls[0]["wave_name"] == "smoke" - - tampered_manifest = SimpleNamespace(payload=copy.deepcopy(existing_manifest.payload)) - tampered_manifest.payload["source_transition"]["source_transition_logical_sha256"] = "0" * 64 - with pytest.raises( - launcher.TrainingLaunchError, - match="differs from its v2 approval", - ): - launcher._parent_binding( - repo_root=REPO_ROOT, - protocol=protocol, - store=ExistingStore(), - key=key, - stage=stage, - source_git_sha=consumer, - approval=None, - existing_manifest=tampered_manifest, - ) - - -def test_completed_wave_cannot_bypass_transition_checks_or_preflight( - tmp_path: Path, - monkeypatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - source_sha = "f" * 40 - external = tmp_path / "external" - key = StageKey(STUDY_ID, "qwen-l0-rl-l1", "r0") - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - {"source_git_sha": source_sha}, - ) - store.write_wave_receipt( - key, - wave="smoke", - payload={ - "stage_id": key.stage_id, - "wave": "smoke", - "run_manifest_record_sha256": manifest.record_sha256, - }, - ) - monkeypatch.setenv("TINKER_API_KEY", "test-only") - monkeypatch.setenv("WANDB_API_KEY", "test-only") - monkeypatch.setattr( - launcher, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - launcher, - "validate_source_sha", - lambda _root, expected: expected, - ) - events: list[str] = [] - monkeypatch.setattr( - launcher, - "stage_preflight", - lambda **_kwargs: events.append("preflight") or _preflight(protocol, key.stage_id, "smoke"), - ) - - def reject_transition(**_kwargs): - events.append("approval") - raise launcher.TrainingLaunchError("v2 transition required") - - monkeypatch.setattr(launcher, "_approval_record", reject_transition) - with pytest.raises( - launcher.TrainingLaunchError, - match="v2 transition required", - ): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id=key.stage_id, - wave_name="smoke", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - rl_main=lambda _config: (_ for _ in ()).throw( - AssertionError("completed bypass reached Tinker") - ), - wandb_access_validator=lambda **_kwargs: (_ for _ in ()).throw( - AssertionError("completed bypass reached W&B") - ), - ) - ) - assert events == ["preflight", "approval"] - - wrong_source_root = tmp_path / "wrong-source" - wrong_store = TrainingStore( - repo_root=REPO_ROOT, - external_root=wrong_source_root, - ) - with wrong_store.acquire_stage_lock(key): - wrong_manifest = wrong_store.create_or_verify_manifest( - key, - {"source_git_sha": "0" * 40}, - ) - wrong_store.write_wave_receipt( - key, - wave="smoke", - payload={ - "stage_id": key.stage_id, - "wave": "smoke", - "run_manifest_record_sha256": (wrong_manifest.record_sha256), - }, - ) - with pytest.raises( - launcher.TrainingLaunchError, - match="another source commit", - ): - asyncio.run( - launcher.run_training_stage( - repo_root=REPO_ROOT, - stage_id=key.stage_id, - wave_name="smoke", - replicate_id="r0", - expected_source_sha=source_sha, - external_root=wrong_source_root, - confirmation=protocol["launch"]["confirmation_token"], - ) - ) diff --git a/rl/studies/representation_training_v1/tests/test_promotion.py b/rl/studies/representation_training_v1/tests/test_promotion.py deleted file mode 100644 index e3a379f9..00000000 --- a/rl/studies/representation_training_v1/tests/test_promotion.py +++ /dev/null @@ -1,1799 +0,0 @@ -from __future__ import annotations - -import copy -import subprocess -import sys -from dataclasses import replace -from pathlib import Path, PurePosixPath - -import pytest - -from rl.common.prompt import prompt_asset_hashes -from rl.studies.representation_training_v1 import promotion -from rl.studies.representation_training_v1.protocol import ( - STUDY_ID, - file_sha256, - load_protocol, - protocol_path, - stage_spec, -) -from rl.studies.representation_training_v1.store import ( - StageKey, - TrainingRecord, - TrainingStore, - canonical_sha256, -) - - -REPO_ROOT = Path(__file__).resolve().parents[4] -SOURCE_SHA = "a" * 40 -CHECKPOINT = { - "name": "final", - "batch": 55, - "epoch": 1, - "final": True, - "state_path": "tinker://unit/weights/final", - "sampler_path": "tinker://unit/sampler_weights/final", - "role": "terminal", - "training_progress_fraction": 1.0, -} - - -def test_record_approval_import_does_not_require_tinker() -> None: - script = """ -import importlib.abc -import sys - -class BlockOptionalTrainingModules(importlib.abc.MetaPathFinder): - def find_spec(self, fullname, path=None, target=None): - if fullname.split(".", 1)[0] in {"chz", "tinker", "tinker_cookbook"}: - raise ModuleNotFoundError(f"blocked optional module: {fullname}") - return None - -sys.meta_path.insert(0, BlockOptionalTrainingModules()) -import rl.studies.representation_training_v1.record_approval -""" - subprocess.run( - [sys.executable, "-c", script], - cwd=REPO_ROOT, - check=True, - capture_output=True, - text=True, - ) - - -def test_record_approval_accepts_dual_level_transition_evidence() -> None: - from rl.studies.representation_training_v1.record_approval import parser - - args = parser().parse_args( - [ - "--stage", - "qwen-base-rl-l1", - "--wave", - "smoke", - "--evaluation-stage", - "qwen-base-rl-l0", - "--evaluation-wave", - "complete", - "--evaluation-checkpoint", - "000030", - "--evaluation-report-record-sha256", - "a" * 64, - "--entry-baseline-report-record-sha256", - "b" * 64, - "--expected-source-sha", - "c" * 40, - ] - ) - - assert args.entry_baseline_report_record_sha256 == "b" * 64 - - -CHECKPOINT_INVENTORY_SHA256 = canonical_sha256([CHECKPOINT]) - - -def test_report_aggregate_rejects_non_model_status_and_failure_iou() -> None: - base = { - "status": "runtime_error", - "pure_executable": False, - "raw_absolute_scale_iou": 0.0, - } - promotion._aggregate([base]) - - with pytest.raises( - promotion.PromotionError, - match="non-model evaluation status", - ): - promotion._aggregate([{**base, "status": "measurement_error"}]) - with pytest.raises( - promotion.PromotionError, - match="failed model output has nonzero IoU", - ): - promotion._aggregate([{**base, "raw_absolute_scale_iou": 0.1}]) - - -def _record( - payload: dict, - *, - record_sha: str, - relative: str, -) -> TrainingRecord: - return TrainingRecord( - payload=payload, - payload_sha256="e" * 64, - record_sha256=record_sha, - relative_path=PurePosixPath(relative), - ) - - -def _level_report(first_unmastered: str | None) -> dict: - levels: dict[str, dict] = {} - for level in ("L0", "L1", "L2", "L3", "L4"): - mastered = first_unmastered is None or int(level[1:]) < int(first_unmastered[1:]) - levels[level] = { - "mean_raw_absolute_scale_iou": 0.85 if mastered else 0.70, - "pure_executable_rate": 0.98 if mastered else 0.90, - } - return {"levels": levels} - - -def _evidence( - *, - stage_id: str, - wave: str, - panel: str, - first_unmastered: str | None, -) -> promotion.EvaluationEvidence: - key = StageKey(STUDY_ID, stage_id, "r0") - manifest = _record( - {}, - record_sha="b" * 64, - relative=f"studies/x/stages/{stage_id}/manifest.json", - ) - receipt = _record( - { - "checkpoint": { - key: CHECKPOINT[key] - for key in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - }, - "checkpoint_inventory": { - "count": 1, - "entries": [CHECKPOINT], - "logical_sha256": CHECKPOINT_INVENTORY_SHA256, - }, - }, - record_sha="c" * 64, - relative=f"studies/x/stages/{stage_id}/waves/{wave}/receipt.json", - ) - return promotion.EvaluationEvidence( - key=key, - wave=wave, - checkpoint_name="final", - checkpoint=CHECKPOINT, - checkpoint_inventory_sha256=CHECKPOINT_INVENTORY_SHA256, - panel=panel, - report={}, - report_payload=_level_report(first_unmastered), - report_record_sha256="d" * 64, - report_payload_sha256="e" * 64, - report_relative_path=( - f"studies/x/stages/{stage_id}/evaluations/{wave}/final/{panel}/report.json" - ), - evaluation_manifest_payload_sha256="f" * 64, - evaluation_manifest_record_sha256="a" * 64, - evaluation_manifest_relative_path=( - f"studies/x/stages/{stage_id}/evaluations/{wave}/final/{panel}/manifest.json" - ), - training_manifest=manifest, - training_receipt=receipt, - ) - - -def _git_transition_history(tmp_path: Path) -> tuple[Path, str, str]: - repo = tmp_path / "repo" - repo.mkdir() - subprocess.run( - ["git", "init", "--initial-branch=main"], - cwd=repo, - check=True, - stdout=subprocess.DEVNULL, - ) - for key, value in ( - ("user.name", "PixCell test"), - ("user.email", "pixcell-test@example.invalid"), - ("commit.gpgsign", "false"), - ("core.hooksPath", "/dev/null"), - ): - subprocess.run( - ["git", "config", key, value], - cwd=repo, - check=True, - ) - (repo / "bridge.txt").write_text("producer\n", encoding="utf-8") - subprocess.run(["git", "add", "--all"], cwd=repo, check=True) - subprocess.run( - ["git", "commit", "-m", "producer"], - cwd=repo, - check=True, - stdout=subprocess.DEVNULL, - ) - producer = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo, - text=True, - ).strip() - (repo / "bridge.txt").write_text("consumer\n", encoding="utf-8") - subprocess.run(["git", "add", "--all"], cwd=repo, check=True) - subprocess.run( - ["git", "commit", "-m", "consumer"], - cwd=repo, - check=True, - stdout=subprocess.DEVNULL, - ) - consumer = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo, - text=True, - ).strip() - return repo, producer, consumer - - -def _cross_source_evidence( - protocol: dict, - *, - producer_source_git_sha: str, -) -> promotion.EvaluationEvidence: - base = _evidence( - stage_id="qwen-l0-sft", - wave="complete", - panel=promotion.PANEL_DEPTH_VALIDATION, - first_unmastered="L1", - ) - model = dict(protocol["models"]["qwen"]) - prompt_assets = prompt_asset_hashes() - dataset = { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": "depth/validation", - "logical_release_sha256": protocol["dataset"]["logical_release_sha256"], - "freeze_file_sha256": "1" * 64, - "parquet_shards": {"depth.parquet": "2" * 64}, - } - sandbox = { - "runtime_path": "/usr/bin/docker", - "daemon_endpoint": "unix:///var/run/docker.sock", - "image_ref": "pixcell-evaluator@sha256:" + "3" * 64, - "image_id": "sha256:" + "3" * 64, - "workspace_root": "/tmp/pixcell-evaluation", - } - manifest = _record( - { - "source_git_sha": producer_source_git_sha, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": "4" * 64, - "contract_version": protocol["contract_version"], - "prompt_assets": prompt_assets, - "dataset": {key: value for key, value in dataset.items() if key != "split"}, - "model": model, - "sandbox": None, - }, - record_sha="8" * 64, - relative=( - "studies/representation-training-v1/stages/qwen-l0-sft/runs/r0/run_manifest.json" - ), - ) - receipt = replace( - base.training_receipt, - payload={ - **base.training_receipt.payload, - "stage_id": "qwen-l0-sft", - "wave": "complete", - "run_manifest_record_sha256": manifest.record_sha256, - }, - record_sha256="9" * 64, - ) - sampler = { - "model": model["model"], - "renderer": model["renderer"], - "effort": model.get("effort"), - } - report_payload = { - **base.report_payload, - "provenance": { - "source_git_sha": producer_source_git_sha, - "protocol": { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": "4" * 64, - "contract_version": protocol["contract_version"], - }, - "prompt_assets": prompt_assets, - "dataset": dataset, - "task_panel": { - "panel": promotion.PANEL_DEPTH_VALIDATION, - "task_count": 1_092, - "logical_sha256": "5" * 64, - }, - "sampler": sampler, - "sampling": { - "attempts_per_task": 1, - "max_output_tokens": 60_000, - "temperature": 1.0, - "top_p": 1.0, - "max_image_long_edge": 1_920, - }, - "sandbox": sandbox, - "evaluation_manifest_record_sha256": "a" * 64, - }, - } - return replace( - base, - training_manifest=manifest, - training_receipt=receipt, - report_payload=report_payload, - evaluation_manifest_payload_sha256="b" * 64, - evaluation_manifest_record_sha256="a" * 64, - ) - - -def _historical_parent_records( - tmp_path: Path, - protocol: dict, - *, - producer_source_git_sha: str, -) -> tuple[ - TrainingRecord, - TrainingRecord, - TrainingRecord, - dict, - dict, - object, -]: - dataset = promotion._dataset_binding(REPO_ROOT / "dataset", protocol) - parent_stage = stage_spec(protocol, "qwen-l0-sft") - stage_payload = { - "stage_id": parent_stage.stage_id, - "kind": parent_stage.kind, - "model_key": parent_stage.model_key, - "recipe_key": parent_stage.recipe_key, - "hypotheses": list(parent_stage.hypotheses), - "parent_policy": parent_stage.parent, - "levels": list(parent_stage.levels), - "current_level": parent_stage.current_level, - "replay_levels": list(parent_stage.replay_levels), - "waves": parent_stage.waves, - } - manifest = _record( - { - "source_git_sha": producer_source_git_sha, - "protocol_logical_sha256": "0" * 64, - "protocol_file_sha256": "1" * 64, - "contract_version": protocol["contract_version"], - "prompt_assets": prompt_asset_hashes(), - "dataset": dataset, - "model": protocol["models"]["qwen"], - "recipe": protocol["recipes"]["qwen_sft"], - "stage": stage_payload, - }, - record_sha="2" * 64, - relative=( - "studies/representation-training-v1/stages/" - "qwen-l0-sft/runs/r0/run_manifest.json" - ), - ) - checkpoint = { - **CHECKPOINT, - "batch": 0, - "epoch": 1, - } - inventory_sha = canonical_sha256([checkpoint]) - log_path = tmp_path / "tinker" - log_path.mkdir(parents=True) - local_artifacts = {} - for name, content in ( - ("checkpoints.jsonl", "{}\n"), - ("metrics.jsonl", "{}\n"), - ): - path = log_path / name - path.write_text(content, encoding="utf-8") - local_artifacts[name] = promotion.file_sha256(path) - invocation = _record( - { - "source_git_sha": producer_source_git_sha, - "run_manifest_record_sha256": manifest.record_sha256, - }, - record_sha="3" * 64, - relative=( - "studies/representation-training-v1/stages/qwen-l0-sft/" - "runs/r0/invocations/complete-unit/start.json" - ), - ) - receipt = _record( - { - "stage_id": "qwen-l0-sft", - "wave": "complete", - "run_manifest_record_sha256": manifest.record_sha256, - "invocation_id": "complete-unit", - "invocation_record_sha256": invocation.record_sha256, - "checkpoint": { - field: checkpoint[field] - for field in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - }, - "checkpoint_inventory": { - "count": 1, - "entries": [checkpoint], - "logical_sha256": inventory_sha, - }, - "local_artifact_sha256": local_artifacts, - }, - record_sha="4" * 64, - relative=( - "studies/representation-training-v1/stages/qwen-l0-sft/" - "runs/r0/waves/complete/receipt.json" - ), - ) - - class ParentStore: - @staticmethod - def load_manifest(_key): - return manifest - - @staticmethod - def load_wave_receipt(_key, *, wave): - assert wave == "complete" - return receipt - - @staticmethod - def load_invocation(_key, *, invocation_id): - assert invocation_id == "complete-unit" - return invocation - - @staticmethod - def tinker_log_path(_key): - return log_path - - return manifest, receipt, invocation, checkpoint, local_artifacts, ParentStore() - - -def _synthetic_depth_report( - protocol: dict, -) -> tuple[ - list[promotion.PanelTaskIdentity], - dict, - dict, - TrainingRecord, - TrainingRecord, -]: - tasks = [ - promotion.PanelTaskIdentity( - task_id=f"task-{index:04d}", - level=f"L{index % 5}", - representation_id=f"rep-{index % 546:03d}", - image_sha256=f"{index % 16:x}" * 64, - target_image_sha256=f"{(index + 1) % 16:x}" * 64, - footprint_um=(20.0 + index / 1_000, 10.0), - ) - for index in range(1_092) - ] - records = [ - { - "task_id": task.task_id, - "level": task.level, - "representation_id": task.representation_id, - "status": "ok", - "pure_executable": True, - "raw_absolute_scale_iou": 0.85, - "completion_tokens": 1_000, - "stop_reason": "stop", - "cap_hit": False, - "channel_parse_complete": True, - "answer_text_sha256": "1" * 64, - "render_sha256": "2" * 64, - "sample_record_sha256": "3" * 64, - "evaluation_record_sha256": "4" * 64, - } - for task in tasks - ] - by_level: dict[str, list[dict]] = {} - by_representation: dict[str, list[dict]] = {} - for record in records: - by_level.setdefault(record["level"], []).append(record) - by_representation.setdefault(record["representation_id"], []).append(record) - representations = { - key: promotion._aggregate(values) for key, values in sorted(by_representation.items()) - } - task_panel = { - "schema_version": "pixcell-training-evaluation-task-panel-v1", - "selection_seed": None, - **promotion._panel_manifest( - tasks, - panel=promotion.PANEL_DEPTH_VALIDATION, - ), - } - dataset_manifest = promotion._dataset_binding(REPO_ROOT / "dataset", protocol) - dataset_report = { - "repo_id": protocol["dataset"]["repo_id"], - "revision": protocol["dataset"]["revision"], - "configuration": protocol["dataset"]["configuration"], - "split": "depth/validation", - "logical_release_sha256": dataset_manifest["logical_release_sha256"], - "freeze_file_sha256": dataset_manifest["freeze_file_sha256"], - "parquet_shards": dataset_manifest["parquet_shards"], - } - sandbox = { - "runtime_path": "/usr/bin/docker", - "daemon_endpoint": "unix:///var/run/docker.sock", - "image_ref": "pixcell-evaluator@sha256:" + "7" * 64, - "image_id": "sha256:" + "7" * 64, - "workspace_root": "/tmp/workspace", - } - manifest = _record( - { - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "dataset": dataset_manifest, - }, - record_sha="8" * 64, - relative="studies/x/stages/qwen-mixed-sft/runs/r0/run_manifest.json", - ) - receipt = _record( - { - "stage_id": "qwen-mixed-sft", - "wave": "complete", - "run_manifest_record_sha256": manifest.record_sha256, - "checkpoint": { - "name": "final", - "batch": 55, - "epoch": 1, - "final": True, - "state_path": CHECKPOINT["state_path"], - "sampler_path": "tinker://unit/sampler_weights/final", - }, - "checkpoint_inventory": { - "count": 1, - "entries": [CHECKPOINT], - "logical_sha256": CHECKPOINT_INVENTORY_SHA256, - }, - }, - record_sha="9" * 64, - relative=("studies/x/stages/qwen-mixed-sft/runs/r0/waves/complete/receipt.json"), - ) - model = protocol["models"]["qwen"] - sampler = { - "model": model["model"], - "renderer": model["renderer"], - "effort": model.get("effort"), - "sampler_path": receipt.payload["checkpoint"]["sampler_path"], - "checkpoint_name": "final", - "checkpoint": CHECKPOINT, - "checkpoint_inventory_sha256": CHECKPOINT_INVENTORY_SHA256, - "training_run_manifest_record_sha256": manifest.record_sha256, - "training_wave_receipt_record_sha256": receipt.record_sha256, - "training_wave_receipt_relative_path": str(receipt.relative_path), - } - sampling = { - "attempts_per_task": 1, - "max_output_tokens": 60_000, - "temperature": 1.0, - "top_p": 1.0, - "max_image_long_edge": 1_920, - "seed_algorithm": "sha256-bound deterministic 31-bit seed", - } - evaluation_manifest = { - "record_sha256": "a" * 64, - "payload": { - "study_id": STUDY_ID, - "stage_id": "qwen-mixed-sft", - "wave": "complete", - "checkpoint_name": "final", - "replicate_id": "r0", - "panel": promotion.PANEL_DEPTH_VALIDATION, - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": file_sha256(protocol_path(REPO_ROOT)), - "contract_version": protocol["contract_version"], - "task_panel": task_panel, - "prompt_assets": prompt_asset_hashes(), - "dataset": dataset_manifest, - "sampler": { - key: sampler[key] - for key in ( - "sampler_path", - "checkpoint_name", - "checkpoint", - "checkpoint_inventory_sha256", - "training_run_manifest_record_sha256", - "training_wave_receipt_record_sha256", - ) - }, - "model": model, - "sampling": { - **sampling, - "sample_concurrency": 8, - "evaluation_batch_size": 32, - "evaluator_workers": 8, - }, - "sandbox": sandbox, - }, - } - report = { - "schema_version": "pixcell-training-checkpoint-evaluation-v1", - "study_id": STUDY_ID, - "stage_id": "qwen-mixed-sft", - "wave": "complete", - "checkpoint_name": "final", - "replicate_id": "r0", - "panel": promotion.PANEL_DEPTH_VALIDATION, - "summary": promotion._aggregate(records), - "representation_macro_mean_iou": ( - sum(value["mean_raw_absolute_scale_iou"] for value in representations.values()) - / len(representations) - ), - "levels": {key: promotion._aggregate(values) for key, values in sorted(by_level.items())}, - "representations": representations, - "records": records, - "provenance": { - "source_git_sha": SOURCE_SHA, - "protocol": { - "logical_sha256": protocol["logical_sha256"], - "file_sha256": file_sha256(protocol_path(REPO_ROOT)), - "contract_version": protocol["contract_version"], - }, - "dataset": dataset_report, - "task_panel": task_panel, - "sampler": sampler, - "sampling": sampling, - "prompt_assets": prompt_asset_hashes(), - "sandbox": sandbox, - "evaluation_manifest_record_sha256": evaluation_manifest["record_sha256"], - }, - } - return tasks, report, evaluation_manifest, manifest, receipt - - -def test_first_unmastered_is_ordered_and_rejects_nonfinite() -> None: - assert promotion.first_unmastered_level(_level_report("L2")) == "L2" - assert promotion.first_unmastered_level(_level_report(None)) is None - invalid = _level_report("L2") - invalid["levels"]["L1"]["mean_raw_absolute_scale_iou"] = float("nan") - with pytest.raises(promotion.PromotionError, match="finite"): - promotion.first_unmastered_level(invalid) - - -def test_mixed_checkpoint_ranking_uses_iou_then_purity_then_progress() -> None: - candidates = [ - { - "checkpoint_name": "000014", - "representation_macro_mean_iou": 0.80, - "pure_executable_rate": 0.99, - "training_progress_fraction": 0.25, - }, - { - "checkpoint_name": "000028", - "representation_macro_mean_iou": 0.81, - "pure_executable_rate": 0.90, - "training_progress_fraction": 0.50, - }, - { - "checkpoint_name": "000042", - "representation_macro_mean_iou": 0.81, - "pure_executable_rate": 0.95, - "training_progress_fraction": 0.75, - }, - { - "checkpoint_name": "final", - "representation_macro_mean_iou": 0.81, - "pure_executable_rate": 0.95, - "training_progress_fraction": 1.0, - }, - ] - assert promotion._best_mixed_checkpoint(candidates)["checkpoint_name"] == "final" - - -def test_mixed_selection_requires_every_report_and_rejects_nonwinner( - tmp_path: Path, - monkeypatch, -) -> None: - pytest.importorskip("chz", reason="Tinker is an optional RL dependency") - pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") - pytest.importorskip( - "tinker_cookbook", - reason="Tinker Cookbook is an optional RL dependency", - ) - from rl.studies.representation_training_v1 import evaluation - - periodic = { - "name": "000014", - "batch": 14, - "epoch": 0, - "final": False, - "state_path": "tinker://unit/weights/000014", - "sampler_path": "tinker://unit/sampler_weights/000014", - "role": "periodic", - "training_progress_fraction": 0.25, - } - entries = [periodic, CHECKPOINT] - inventory_sha = canonical_sha256(entries) - receipt = _record( - { - "checkpoint": { - key: CHECKPOINT[key] - for key in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - }, - "checkpoint_inventory": { - "count": 2, - "entries": entries, - "logical_sha256": inventory_sha, - }, - }, - record_sha="c" * 64, - relative=("studies/x/stages/qwen-mixed-sft/waves/complete/receipt.json"), - ) - - def scored( - checkpoint: dict, - *, - iou: float, - pure: float, - report_sha: str, - ) -> promotion.EvaluationEvidence: - base = _evidence( - stage_id="qwen-mixed-sft", - wave="complete", - panel=promotion.PANEL_DEPTH_VALIDATION, - first_unmastered="L2", - ) - payload = { - **base.report_payload, - "representation_macro_mean_iou": iou, - "summary": {"pure_executable_rate": pure}, - } - return replace( - base, - checkpoint_name=checkpoint["name"], - checkpoint=checkpoint, - checkpoint_inventory_sha256=inventory_sha, - report_payload=payload, - report_record_sha256=report_sha, - training_receipt=receipt, - ) - - selected = scored( - CHECKPOINT, - iou=0.80, - pure=0.99, - report_sha="d" * 64, - ) - periodic_report = scored( - periodic, - iou=0.90, - pure=0.90, - report_sha="f" * 64, - ) - - class FakeReportStore: - def __init__(self, **kwargs) -> None: - self.checkpoint_name = kwargs["checkpoint_name"] - - def load_report(self) -> dict | None: - if self.checkpoint_name == periodic["name"]: - return {"record_sha256": periodic_report.report_record_sha256} - return None - - monkeypatch.setattr( - evaluation, - "CheckpointEvaluationStore", - FakeReportStore, - ) - monkeypatch.setattr( - promotion, - "validate_checkpoint_report", - lambda **_kwargs: periodic_report, - ) - repo = tmp_path / "repo" - repo.mkdir() - store = TrainingStore( - repo_root=repo, - external_root=tmp_path / "external", - ) - with pytest.raises(promotion.PromotionError, match="deterministic best"): - promotion._mixed_checkpoint_selection( - repo_root=repo, - protocol=load_protocol(REPO_ROOT), - store=store, - selected=selected, - expected_source_sha=SOURCE_SHA, - ) - - monkeypatch.setattr( - FakeReportStore, - "load_report", - lambda _self: None, - ) - with pytest.raises(promotion.PromotionError, match="complete depth-validation"): - promotion._mixed_checkpoint_selection( - repo_root=repo, - protocol=load_protocol(REPO_ROOT), - store=store, - selected=selected, - expected_source_sha=SOURCE_SHA, - ) - - -def test_plans_bind_reports_and_clean_smoke_storage() -> None: - protocol = load_protocol(REPO_ROOT) - mixed = promotion.approval_plan( - stage_spec(protocol, "qwen-mixed-rl-l2"), - "smoke", - ) - assert mixed.expected_panel == promotion.PANEL_DEPTH_VALIDATION - assert mixed.allowed_evaluation_checkpoints == ( - ("qwen-mixed-sft", "complete"), - ("qwen-mixed-rl-l0", "complete"), - ("qwen-mixed-rl-l1", "complete"), - ) - continuation = promotion.approval_plan( - stage_spec(protocol, "qwen-mixed-rl-l2"), - "step-5", - ) - assert continuation.storage_wave == "step-5" - assert continuation.expected_panel == promotion.PANEL_PROGRESS - - -def test_pure_base_level_transition_binds_prior_and_current_slot6_panels() -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l2") - plan = promotion.approval_plan(stage, "smoke") - assert plan.gate == ( - "prior-and-current-level-held-out-transition-receipt" - ) - assert plan.expected_panel == "level-progress-l1" - assert plan.entry_baseline_panel == "level-progress-l2" - assert plan.allowed_evaluation_checkpoints == ( - ("qwen-base-rl-l1", "complete"), - ) - - prior = _evidence( - stage_id="qwen-base-rl-l1", - wave="complete", - panel="level-progress-l1", - first_unmastered=None, - ) - baseline = replace( - prior, - panel="level-progress-l2", - report_record_sha256="1" * 64, - report_payload_sha256="2" * 64, - report_relative_path=( - "studies/x/stages/qwen-base-rl-l1/evaluations/" - "complete/final/level-progress-l2/report.json" - ), - evaluation_manifest_payload_sha256="3" * 64, - evaluation_manifest_record_sha256="4" * 64, - evaluation_manifest_relative_path=( - "studies/x/stages/qwen-base-rl-l1/evaluations/" - "complete/final/level-progress-l2/manifest.json" - ), - ) - payload = promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=plan, - evidence=prior, - entry_baseline_evidence=baseline, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert payload["evaluation"]["panel"] == "level-progress-l1" - assert payload["entry_baseline"]["panel"] == "level-progress-l2" - assert ( - payload["evaluation"]["checkpoint"] - == payload["entry_baseline"]["checkpoint"] - ) - assert payload["parent_stage_id"] == "qwen-base-rl-l1" - assert payload["parent_wave"] == "complete" - - with pytest.raises(promotion.PromotionError, match="requires the .* baseline"): - promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=plan, - evidence=prior, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - with pytest.raises(promotion.PromotionError, match="exact same parent checkpoint"): - promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=plan, - evidence=prior, - entry_baseline_evidence=replace( - baseline, - checkpoint={**baseline.checkpoint, "sampler_path": "tinker://other"}, - ), - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - mismatched_artifact_chains = ( - replace(baseline, checkpoint_inventory_sha256="6" * 64), - replace( - baseline, - training_manifest=replace( - baseline.training_manifest, - record_sha256="7" * 64, - ), - ), - replace( - baseline, - training_receipt=replace( - baseline.training_receipt, - record_sha256="8" * 64, - ), - ), - ) - for mismatched in mismatched_artifact_chains: - with pytest.raises( - promotion.PromotionError, - match="exact same parent checkpoint", - ): - promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=plan, - evidence=prior, - entry_baseline_evidence=mismatched, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - - -def test_same_source_approval_is_recomputed_from_both_bound_reports( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l2") - plan = promotion.approval_plan(stage, "smoke") - prior = _evidence( - stage_id="qwen-base-rl-l1", - wave="complete", - panel=plan.expected_panel, - first_unmastered=None, - ) - baseline = replace( - prior, - panel=str(plan.entry_baseline_panel), - report_record_sha256="1" * 64, - report_payload_sha256="2" * 64, - report_relative_path=( - "studies/x/stages/qwen-base-rl-l1/evaluations/" - "complete/final/level-progress-l2/report.json" - ), - evaluation_manifest_payload_sha256="3" * 64, - evaluation_manifest_record_sha256="4" * 64, - evaluation_manifest_relative_path=( - "studies/x/stages/qwen-base-rl-l1/evaluations/" - "complete/final/level-progress-l2/manifest.json" - ), - ) - payload = promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=plan, - evidence=prior, - entry_baseline_evidence=baseline, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - approval = _record( - payload, - record_sha="5" * 64, - relative=( - "studies/representation-training-v1/stages/qwen-base-rl-l2/" - "runs/r0/waves/smoke/approval.json" - ), - ) - calls: list[dict] = [] - - def validate_report(**kwargs): - calls.append(dict(kwargs)) - if kwargs["panel"] == plan.expected_panel: - assert kwargs["expected_report_record_sha256"] == "d" * 64 - return prior - assert kwargs["panel"] == plan.entry_baseline_panel - assert kwargs["expected_report_record_sha256"] == "1" * 64 - return baseline - - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - promotion, - "validate_checkpoint_report", - validate_report, - ) - repo = tmp_path / "repo" - repo.mkdir() - store = TrainingStore(repo_root=repo, external_root=tmp_path / "external") - observed = promotion.validate_same_source_wave_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage, - wave_name="smoke", - replicate_id="r0", - source_git_sha=SOURCE_SHA, - approval=approval, - ) - assert observed is prior - assert [call["panel"] for call in calls] == [ - "level-progress-l1", - "level-progress-l2", - ] - - tampered = replace( - approval, - payload={ - **approval.payload, - "entry_baseline": { - **approval.payload["entry_baseline"], - "report_payload_sha256": "9" * 64, - }, - }, - ) - with pytest.raises(promotion.PromotionError, match="canonically recomputed"): - promotion.validate_same_source_wave_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage, - wave_name="smoke", - replicate_id="r0", - source_git_sha=SOURCE_SHA, - approval=tampered, - ) - - -def test_same_source_current_level_approval_rejects_minimal_json( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-base-rl-l0") - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda _root, expected: expected, - ) - repo = tmp_path / "repo" - repo.mkdir() - store = TrainingStore(repo_root=repo, external_root=tmp_path / "external") - approval = _record( - { - "schema_version": promotion.APPROVAL_SCHEMA_VERSION, - "decision": "approve", - "approval_gate": "current-level-held-out-promotion-receipt", - "stage_id": stage.stage_id, - "wave": "step-10", - "source_git_sha": SOURCE_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - }, - record_sha="6" * 64, - relative=( - "studies/representation-training-v1/stages/qwen-base-rl-l0/" - "runs/r0/waves/step-10/approval.json" - ), - ) - with pytest.raises(promotion.PromotionError, match="approval.evaluation"): - promotion.validate_same_source_wave_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage, - wave_name="step-10", - replicate_id="r0", - source_git_sha=SOURCE_SHA, - approval=approval, - ) - - -def test_mixed_approval_selects_first_unmastered_and_binds_parent() -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-mixed-rl-l2") - evidence = _evidence( - stage_id="qwen-mixed-sft", - wave="complete", - panel=promotion.PANEL_DEPTH_VALIDATION, - first_unmastered="L2", - ) - payload = promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=promotion.approval_plan(stage, "smoke"), - evidence=evidence, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert payload["schema_version"] == promotion.APPROVAL_SCHEMA_VERSION - assert "evidence_source_git_sha" not in payload - assert "source_transition" not in payload - assert payload["decision"] == "approve" - assert payload["approval_gate"] == "first-unmastered-entry-receipt" - assert payload["parent_stage_id"] == "qwen-mixed-sft" - assert payload["parent_wave"] == "complete" - assert payload["parent_receipt_record_sha256"] == "c" * 64 - assert payload["selected_parent_checkpoint"] == CHECKPOINT - assert payload["selected_parent_checkpoint_inventory_sha256"] == CHECKPOINT_INVENTORY_SHA256 - assert payload["mastery"]["selected_level"] == "L2" - - wrong = _evidence( - stage_id="qwen-mixed-sft", - wave="complete", - panel=promotion.PANEL_DEPTH_VALIDATION, - first_unmastered="L1", - ) - with pytest.raises(promotion.PromotionError, match="first unmastered"): - promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=promotion.approval_plan(stage, "smoke"), - evidence=wrong, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - - -def test_clean_smoke_approval_does_not_masquerade_as_parent() -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-mixed-rl-l2") - evidence = _evidence( - stage_id=stage.stage_id, - wave="smoke", - panel=promotion.PANEL_PROGRESS, - first_unmastered=None, - ) - payload = promotion._approval_payload( - stage=stage, - wave_name="step-5", - plan=promotion.approval_plan(stage, "step-5"), - evidence=evidence, - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - assert "parent_receipt_record_sha256" not in payload - assert payload["evaluation"]["training_wave_receipt_record_sha256"] == "c" * 64 - assert payload["prior_wave"] == "smoke" - assert payload["prior_receipt_record_sha256"] == "c" * 64 - - -def test_entry_approval_cannot_authorize_continuation( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-mixed-rl-l2") - entry = promotion._approval_payload( - stage=stage, - wave_name="smoke", - plan=promotion.approval_plan(stage, "smoke"), - evidence=_evidence( - stage_id="qwen-mixed-sft", - wave="complete", - panel=promotion.PANEL_DEPTH_VALIDATION, - first_unmastered="L2", - ), - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - continuation = promotion._approval_payload( - stage=stage, - wave_name="step-5", - plan=promotion.approval_plan(stage, "step-5"), - evidence=_evidence( - stage_id=stage.stage_id, - wave="smoke", - panel=promotion.PANEL_PROGRESS, - first_unmastered=None, - ), - source_git_sha=SOURCE_SHA, - protocol=protocol, - ) - repo = tmp_path / "repo" - repo.mkdir() - store = TrainingStore( - repo_root=repo, - external_root=tmp_path / "external", - ) - key = StageKey(STUDY_ID, stage.stage_id, "r0") - with store.acquire_stage_lock(key): - store.write_wave_approval(key, wave="smoke", payload=entry) - assert store.load_wave_approval(key, wave="step-5") is None - store.write_wave_approval( - key, - wave="step-5", - payload=continuation, - ) - assert ( - store.load_wave_approval(key, wave="smoke").payload["approval_gate"] - == "first-unmastered-entry-receipt" - ) - assert ( - store.load_wave_approval(key, wave="step-5").payload["approval_gate"] - == "clean-smoke-receipt" - ) - - -def test_depth_report_recomputes_all_rows_metrics_and_provenance() -> None: - protocol = load_protocol(REPO_ROOT) - tasks, report, evaluation_manifest, manifest, receipt = _synthetic_depth_report(protocol) - promotion._validate_report_payload( - repo_root=REPO_ROOT, - protocol=protocol, - expected_source_sha=SOURCE_SHA, - key=StageKey(STUDY_ID, "qwen-mixed-sft", "r0"), - wave="complete", - checkpoint_name="final", - selected_checkpoint=CHECKPOINT, - checkpoint_inventory_sha256=CHECKPOINT_INVENTORY_SHA256, - panel=promotion.PANEL_DEPTH_VALIDATION, - report=report, - evaluation_manifest=evaluation_manifest, - training_manifest=manifest, - training_receipt=receipt, - expected_tasks=tasks, - ) - report_dataset = report["provenance"]["dataset"] - manifest_dataset = evaluation_manifest["payload"]["dataset"] - manifest_only_fields = { - "train_rows", - "validation_rows", - "reference_artifacts_sha256", - "reference_artifact_count", - } - assert manifest_only_fields.isdisjoint(report_dataset) - assert manifest_only_fields.issubset(manifest_dataset) - - tampered_manifest = copy.deepcopy(evaluation_manifest) - tampered_manifest["payload"]["dataset"]["reference_artifact_count"] += 1 - with pytest.raises( - promotion.PromotionError, - match="evaluation-manifest dataset differs from the frozen release", - ): - promotion._validate_report_payload( - repo_root=REPO_ROOT, - protocol=protocol, - expected_source_sha=SOURCE_SHA, - key=StageKey(STUDY_ID, "qwen-mixed-sft", "r0"), - wave="complete", - checkpoint_name="final", - selected_checkpoint=CHECKPOINT, - checkpoint_inventory_sha256=CHECKPOINT_INVENTORY_SHA256, - panel=promotion.PANEL_DEPTH_VALIDATION, - report=report, - evaluation_manifest=tampered_manifest, - training_manifest=manifest, - training_receipt=receipt, - expected_tasks=tasks, - ) - - overfull_report = copy.deepcopy(report) - overfull_report["provenance"]["dataset"]["reference_artifact_count"] = ( - manifest_dataset["reference_artifact_count"] - ) - with pytest.raises( - promotion.PromotionError, - match="report dataset projection differs", - ): - promotion._validate_report_payload( - repo_root=REPO_ROOT, - protocol=protocol, - expected_source_sha=SOURCE_SHA, - key=StageKey(STUDY_ID, "qwen-mixed-sft", "r0"), - wave="complete", - checkpoint_name="final", - selected_checkpoint=CHECKPOINT, - checkpoint_inventory_sha256=CHECKPOINT_INVENTORY_SHA256, - panel=promotion.PANEL_DEPTH_VALIDATION, - report=overfull_report, - evaluation_manifest=evaluation_manifest, - training_manifest=manifest, - training_receipt=receipt, - expected_tasks=tasks, - ) - - nonfinite = copy.deepcopy(report) - nonfinite["summary"]["mean_raw_absolute_scale_iou"] = float("nan") - with pytest.raises(promotion.PromotionError, match="finite"): - promotion._validate_report_payload( - repo_root=REPO_ROOT, - protocol=protocol, - expected_source_sha=SOURCE_SHA, - key=StageKey(STUDY_ID, "qwen-mixed-sft", "r0"), - wave="complete", - checkpoint_name="final", - selected_checkpoint=CHECKPOINT, - checkpoint_inventory_sha256=CHECKPOINT_INVENTORY_SHA256, - panel=promotion.PANEL_DEPTH_VALIDATION, - report=nonfinite, - evaluation_manifest=evaluation_manifest, - training_manifest=manifest, - training_receipt=receipt, - expected_tasks=tasks, - ) - - with pytest.raises(promotion.PromotionError, match="expected 1092"): - promotion._validate_report_payload( - repo_root=REPO_ROOT, - protocol=protocol, - expected_source_sha=SOURCE_SHA, - key=StageKey(STUDY_ID, "qwen-mixed-sft", "r0"), - wave="complete", - checkpoint_name="final", - selected_checkpoint=CHECKPOINT, - checkpoint_inventory_sha256=CHECKPOINT_INVENTORY_SHA256, - panel=promotion.PANEL_DEPTH_VALIDATION, - report=report, - evaluation_manifest=evaluation_manifest, - training_manifest=manifest, - training_receipt=receipt, - expected_tasks=tasks[:-1], - ) - - -def test_record_approval_requires_human_confirmation_and_is_immutable( - tmp_path: Path, - monkeypatch, -) -> None: - with pytest.raises(promotion.PromotionError, match="human approval"): - promotion.record_wave_approval( - repo_root=tmp_path / "missing", - external_root=tmp_path / "external", - stage_id="qwen-mixed-rl-l2", - wave_name="smoke", - replicate_id="r0", - evaluation_stage_id="qwen-mixed-sft", - evaluation_wave="complete", - evaluation_checkpoint_name="final", - evaluation_report_record_sha256="d" * 64, - expected_source_sha=SOURCE_SHA, - confirmation="wrong", - ) - - repo = tmp_path / "repo" - repo.mkdir() - external = tmp_path / "external" - protocol = load_protocol(REPO_ROOT) - evidence = _evidence( - stage_id="qwen-l0-sft", - wave="complete", - panel=promotion.PANEL_DEPTH_VALIDATION, - first_unmastered="L1", - ) - monkeypatch.setattr( - promotion, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - promotion, - "validate_checkpoint_report", - lambda **_kwargs: evidence, - ) - kwargs = { - "repo_root": repo, - "external_root": external, - "stage_id": "qwen-sequential-sft-l1", - "wave_name": "complete", - "replicate_id": "r0", - "evaluation_stage_id": "qwen-l0-sft", - "evaluation_wave": "complete", - "evaluation_checkpoint_name": "final", - "evaluation_report_record_sha256": "d" * 64, - "expected_source_sha": SOURCE_SHA, - "confirmation": promotion.APPROVAL_CONFIRMATION_TOKEN, - } - first = promotion.record_wave_approval(**kwargs) - second = promotion.record_wave_approval(**kwargs) - assert first == second - store = TrainingStore(repo_root=repo, external_root=external) - approval = store.load_wave_approval( - StageKey(STUDY_ID, "qwen-sequential-sft-l1", "r0"), - wave="complete", - ) - assert approval is not None - assert approval.payload["decision"] == "approve" - assert approval.payload["explicit_human_confirmation"] is True - - -def test_cross_source_v2_binds_exact_evidence_scope_and_one_child( - tmp_path: Path, - monkeypatch, -) -> None: - repo, producer, consumer = _git_transition_history(tmp_path) - external = tmp_path / "external" - protocol = load_protocol(REPO_ROOT) - evidence = _cross_source_evidence( - protocol, - producer_source_git_sha=producer, - ) - monkeypatch.setattr( - promotion, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr( - promotion, - "file_sha256", - lambda _path: "4" * 64, - ) - report_calls: list[dict] = [] - - def validate_report(**kwargs): - report_calls.append(dict(kwargs)) - assert kwargs["expected_source_sha"] == producer - return evidence - - monkeypatch.setattr( - promotion, - "validate_checkpoint_report", - validate_report, - ) - result = promotion.record_wave_approval( - repo_root=repo, - external_root=external, - stage_id="qwen-sequential-sft-l1", - wave_name="complete", - replicate_id="r0", - evaluation_stage_id="qwen-l0-sft", - evaluation_wave="complete", - evaluation_checkpoint_name="final", - evaluation_report_record_sha256=evidence.report_record_sha256, - expected_source_sha=consumer, - evidence_source_sha=producer, - approved_change_paths=["bridge.txt"], - confirmation=promotion.APPROVAL_CONFIRMATION_TOKEN, - ) - assert result["approval_schema_version"] == ( - promotion.SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION - ) - assert result["evidence_source_git_sha"] == producer - assert result["consumer_source_git_sha"] == consumer - - key = StageKey(STUDY_ID, "qwen-sequential-sft-l1", "r0") - store = TrainingStore(repo_root=repo, external_root=external) - approval = store.load_wave_approval(key, wave="complete") - assert approval is not None - payload = approval.payload - assert payload["schema_version"] == (promotion.SOURCE_TRANSITION_APPROVAL_SCHEMA_VERSION) - assert payload["evaluation"]["report_record_sha256"] == (evidence.report_record_sha256) - assert payload["evaluation"]["training_run_manifest_record_sha256"] == ( - evidence.training_manifest.record_sha256 - ) - assert payload["evaluation"]["training_wave_receipt_record_sha256"] == ( - evidence.training_receipt.record_sha256 - ) - assert payload["evaluation"]["evaluation_manifest_record_sha256"] == ( - evidence.evaluation_manifest_record_sha256 - ) - transition = payload["source_transition"] - assert transition["scope"]["child"] == { - "stage_id": "qwen-sequential-sft-l1", - "wave": "complete", - "replicate_id": "r0", - "approval_gate": "parent-evaluation-receipt", - } - assert transition["scope"]["approved_changes"]["paths"] == ["bridge.txt"] - assert ( - transition["evidence"]["checkpoint_inventory"] - == (evidence.training_receipt.payload["checkpoint_inventory"]) - ) - assert transition["evidence"]["evaluation_manifest"]["record_sha256"] == ( - evidence.evaluation_manifest_record_sha256 - ) - assert transition["equivalence"]["model"]["producer_training"] == (protocol["models"]["qwen"]) - assert transition["equivalence"]["renderer"]["producer_evaluation"] == { - "model": protocol["models"]["qwen"]["model"], - "renderer": protocol["models"]["qwen"]["renderer"], - "effort": protocol["models"]["qwen"].get("effort"), - } - assert ( - promotion.validate_source_transition_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage_spec(protocol, "qwen-sequential-sft-l1"), - wave_name="complete", - replicate_id="r0", - consumer_source_git_sha=consumer, - approval=approval, - ) - == evidence - ) - with pytest.raises(promotion.PromotionError, match="another replicate"): - promotion.validate_source_transition_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage_spec(protocol, "qwen-sequential-sft-l1"), - wave_name="complete", - replicate_id="r1", - consumer_source_git_sha=consumer, - approval=approval, - ) - - tampered_payload = copy.deepcopy(payload) - tampered_payload["source_transition"]["evidence"]["evaluation_report"]["record_sha256"] = ( - "0" * 64 - ) - tampered = replace(approval, payload=tampered_payload) - with pytest.raises(promotion.PromotionError, match="recomputed evidence"): - promotion.validate_source_transition_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage_spec(protocol, "qwen-sequential-sft-l1"), - wave_name="complete", - replicate_id="r0", - consumer_source_git_sha=consumer, - approval=tampered, - ) - assert all(call["expected_source_sha"] == producer for call in report_calls) - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda *_args, **_kwargs: (_ for _ in ()).throw( - ValueError("paid training requires a completely clean worktree") - ), - ) - with pytest.raises(promotion.PromotionError, match="clean HEAD"): - promotion.validate_source_transition_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage_spec(protocol, "qwen-sequential-sft-l1"), - wave_name="complete", - replicate_id="r0", - consumer_source_git_sha=consumer, - approval=approval, - ) - - -def test_cross_source_recording_rejects_missing_or_wrong_change_scope( - tmp_path: Path, - monkeypatch, -) -> None: - repo, producer, consumer = _git_transition_history(tmp_path) - protocol = load_protocol(REPO_ROOT) - monkeypatch.setattr( - promotion, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda _root, expected: expected, - ) - common = { - "repo_root": repo, - "external_root": tmp_path / "external", - "stage_id": "qwen-sequential-sft-l1", - "wave_name": "complete", - "replicate_id": "r0", - "evaluation_stage_id": "qwen-l0-sft", - "evaluation_wave": "complete", - "evaluation_checkpoint_name": "final", - "evaluation_report_record_sha256": "d" * 64, - "expected_source_sha": consumer, - "evidence_source_sha": producer, - "confirmation": promotion.APPROVAL_CONFIRMATION_TOKEN, - } - with pytest.raises(promotion.PromotionError, match="explicit approved"): - promotion.record_wave_approval(**common) - with pytest.raises( - promotion.PromotionError, - match="approved change paths", - ): - promotion.record_wave_approval( - **common, - approved_change_paths=["wrong.txt"], - ) - with pytest.raises( - promotion.PromotionError, - match="same-source v1", - ): - promotion.record_wave_approval( - **{ - **common, - "evidence_source_sha": consumer, - }, - approved_change_paths=["bridge.txt"], - ) - - -def test_historical_parent_evidence_binds_receipt_without_evaluation( - tmp_path: Path, -) -> None: - protocol = load_protocol(REPO_ROOT) - producer = "b" * 40 - manifest, receipt, invocation, checkpoint, artifacts, store = ( - _historical_parent_records( - tmp_path, - protocol, - producer_source_git_sha=producer, - ) - ) - observed = promotion._parent_receipt_transition_evidence( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - replicate_id="r0", - ) - assert observed[:4] == (manifest, receipt, invocation, checkpoint) - document = observed[4] - assert "evaluation" not in document["evidence"] - assert document["evidence"]["local_artifact_sha256"] == artifacts - assert ( - document["stable_equivalence"]["prompt_assets"] - == prompt_asset_hashes() - ) - - manifest.payload["prompt_assets"] = { - **prompt_asset_hashes(), - "task_sha256": "f" * 64, - } - with pytest.raises(promotion.PromotionError, match="stable contract"): - promotion._parent_receipt_transition_evidence( - repo_root=REPO_ROOT, - protocol=protocol, - store=store, - replicate_id="r0", - ) - - -def test_parent_receipt_transition_records_and_revalidates_exact_bridge( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, producer, consumer = _git_transition_history(tmp_path) - protocol = load_protocol(REPO_ROOT) - manifest, receipt, invocation, checkpoint, _artifacts, _store = ( - _historical_parent_records( - tmp_path / "parent", - protocol, - producer_source_git_sha=producer, - ) - ) - stable = { - "stable_equivalence": { - "contract_version": protocol["contract_version"], - "prompt_assets": prompt_asset_hashes(), - "dataset": promotion._dataset_binding(REPO_ROOT / "dataset", protocol), - "model": protocol["models"]["qwen"], - "recipe": protocol["recipes"]["qwen_sft"], - "stage": dict(manifest.payload["stage"]), - }, - "evidence": { - "parent_run_manifest": {"record_sha256": manifest.record_sha256}, - "parent_wave_receipt": {"record_sha256": receipt.record_sha256}, - "parent_invocation": {"record_sha256": invocation.record_sha256}, - "checkpoint_inventory": receipt.payload["checkpoint_inventory"], - "selected_checkpoint": checkpoint, - }, - } - monkeypatch.setattr( - promotion, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - monkeypatch.setattr( - promotion, - "validate_source_sha", - lambda _root, expected: expected, - ) - monkeypatch.setattr(promotion, "file_sha256", lambda _path: "5" * 64) - monkeypatch.setattr( - promotion, - "_parent_receipt_transition_evidence", - lambda **_kwargs: ( - manifest, - receipt, - invocation, - checkpoint, - stable, - ), - ) - external = tmp_path / "external" - with pytest.raises(promotion.PromotionError, match="human approval"): - promotion.record_parent_receipt_transition_approval( - repo_root=repo, - external_root=external, - stage_id="qwen-l0-rl-l1", - wave_name="smoke", - replicate_id="r0", - expected_source_sha=consumer, - approved_change_paths=["bridge.txt"], - confirmation="wrong", - ) - result = promotion.record_parent_receipt_transition_approval( - repo_root=repo, - external_root=external, - stage_id="qwen-l0-rl-l1", - wave_name="smoke", - replicate_id="r0", - expected_source_sha=consumer, - approved_change_paths=["bridge.txt"], - confirmation=promotion.APPROVAL_CONFIRMATION_TOKEN, - ) - assert result["parent_receipt_record_sha256"] == receipt.record_sha256 - store = TrainingStore(repo_root=repo, external_root=external) - key = StageKey(STUDY_ID, "qwen-l0-rl-l1", "r0") - approval = store.load_wave_approval(key, wave="smoke") - assert approval is not None - assert approval.payload["schema_version"] == ( - promotion.PARENT_RECEIPT_TRANSITION_APPROVAL_SCHEMA_VERSION - ) - assert approval.payload["selected_parent_checkpoint"] == checkpoint - assert "evaluation" not in approval.payload - promotion.validate_parent_receipt_transition_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage_spec(protocol, "qwen-l0-rl-l1"), - wave_name="smoke", - replicate_id="r0", - consumer_source_git_sha=consumer, - approval=approval, - ) - - tampered = replace( - approval, - payload={ - **approval.payload, - "parent_receipt_record_sha256": "0" * 64, - }, - ) - with pytest.raises(promotion.PromotionError, match="recomputed evidence"): - promotion.validate_parent_receipt_transition_approval( - repo_root=repo, - protocol=protocol, - store=store, - stage=stage_spec(protocol, "qwen-l0-rl-l1"), - wave_name="smoke", - replicate_id="r0", - consumer_source_git_sha=consumer, - approval=tampered, - ) diff --git a/rl/studies/representation_training_v1/tests/test_protocol_schedule.py b/rl/studies/representation_training_v1/tests/test_protocol_schedule.py deleted file mode 100644 index d048bab8..00000000 --- a/rl/studies/representation_training_v1/tests/test_protocol_schedule.py +++ /dev/null @@ -1,276 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import pytest - -pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") - -import tinker - -from rl.studies.representation_training_v1.launcher import ( - deterministic_wandb_run_id, -) -from rl.studies.representation_training_v1.protocol import ( - STUDY_ID, - load_protocol, - parent_replicate_id, - stage_spec, - validate_stage_replicate, -) -from rl.studies.representation_training_v1.schedule import ( - SCHEDULE_SEED, - stage_schedule_document, - stage_tasks, -) -from rl.studies.representation_training_v1.store import StageKey -from rl.track_a.tinker_data import TrackARLDataset - - -REPO_ROOT = Path(__file__).resolve().parents[4] - - -def test_protocol_seals_the_shared_branch_graph() -> None: - protocol = load_protocol(REPO_ROOT) - assert len(protocol["stages"]) == 21 - assert protocol["recipes"]["qwen_sft"]["learning_rate"] == 1e-4 - assert stage_spec(protocol, "qwen-mixed-sft").parent == "base:qwen" - assert ( - stage_spec(protocol, "qwen-sequential-sft-l1").parent - == "qwen-l0-sft:complete" - ) - assert stage_spec(protocol, "qwen-l0-rl-l1").parent == "qwen-l0-sft:complete" - assert stage_spec(protocol, "inkling-l4-rl").parent == "base:inkling" - - -def test_protocol_seals_the_pure_qwen_rl_curriculum() -> None: - protocol = load_protocol(REPO_ROOT) - expected_gates = { - "step-5": "rollout-health-receipt", - "step-10": "current-level-held-out-promotion-receipt", - "step-15": "current-level-held-out-promotion-receipt", - "step-20": "current-level-held-out-promotion-receipt", - "step-25": "current-level-held-out-promotion-receipt", - "complete": "current-level-held-out-promotion-receipt", - } - for level in range(5): - stage = stage_spec(protocol, f"qwen-base-rl-l{level}") - assert stage.hypotheses == ("RT-H06",) - assert stage.parent == ( - "base:qwen" - if level == 0 - else f"qwen-base-rl-l{level - 1}:complete" - ) - assert stage.current_level == f"L{level}" - assert stage.replay_levels == tuple(f"L{prior}" for prior in range(level)) - assert { - name: wave["max_steps"] for name, wave in stage.waves.items() - } == { - "smoke": 1, - "step-5": 5, - "step-10": 10, - "step-15": 15, - "step-20": 20, - "step-25": 25, - "complete": 30, - } - assert stage.wave("smoke")["approval"] == ( - "initial" - if level == 0 - else "prior-and-current-level-held-out-transition-receipt" - ) - assert { - wave: stage.wave(wave)["approval"] for wave in expected_gates - } == expected_gates - - -def test_operational_retry_is_scoped_and_keeps_the_historical_r0_parent() -> None: - protocol = load_protocol(REPO_ROOT) - - validate_stage_replicate( - protocol, - stage_id="qwen-l0-rl-l1", - replicate_id="r0-retry1", - ) - assert ( - parent_replicate_id( - protocol, - stage_id="qwen-l0-rl-l1", - replicate_id="r0-retry1", - ) - == "r0" - ) - assert ( - parent_replicate_id( - protocol, - stage_id="qwen-l0-rl-l2", - replicate_id="r0-retry1", - ) - == "r0-retry1" - ) - with pytest.raises(ValueError, match="not declared for this training stage"): - validate_stage_replicate( - protocol, - stage_id="qwen-base-rl-l0", - replicate_id="r0-retry1", - ) - - -def test_protocol_seals_fixed_progress_and_final_selection_panels() -> None: - evaluation = load_protocol(REPO_ROOT)["evaluation"] - progress = evaluation["curriculum_progress_holdout"] - assert progress == { - "baseline_checkpoints": { - "L0": "base:qwen", - "L1": "qwen-base-rl-l0:complete", - "L2": "qwen-base-rl-l1:complete", - "L3": "qwen-base-rl-l2:complete", - "L4": "qwen-base-rl-l3:complete", - }, - "configuration": "depth", - "evaluation_waves": [ - "step-5", - "step-10", - "step-15", - "step-20", - "step-25", - "complete", - ], - "panel_prefix": "level-progress", - "realization_slot": 6, - "representations_by_level": { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108, - }, - "rows_by_level": { - "L0": 93, - "L1": 127, - "L2": 98, - "L3": 120, - "L4": 108, - }, - "split": "validation", - } - assert evaluation["final_selection_holdout"] == { - "configuration": "depth", - "panel": "depth-final-selection", - "realization_slot": 7, - "representations": 546, - "rows": 546, - "split": "validation", - } - - -def test_pure_qwen_rl_schedule_uses_every_level_and_full_earlier_prefix() -> None: - protocol = load_protocol(REPO_ROOT) - expected_rows = {"L0": 642, "L1": 850, "L2": 608, "L3": 720, "L4": 648} - expected_representations = {"L0": 93, "L1": 127, "L2": 99, "L3": 120, "L4": 108} - for level in range(5): - stage = stage_spec(protocol, f"qwen-base-rl-l{level}") - schedule = stage_schedule_document( - REPO_ROOT / "dataset", - stage, - groups_per_batch=8, - ) - current = f"L{level}" - assert schedule["maximum_steps"] == 30 - assert schedule["rows_by_level"] == {current: expected_rows[current]} - assert schedule["representations"] == expected_representations[current] - assert schedule["replay_levels"] == [ - f"L{prior}" for prior in range(level) - ] - assert len(schedule["training_batches"]) == 30 - assert all( - len(batch["groups"]) == 8 - for batch in schedule["training_batches"] - ) - - -def test_qwen_rl_schedule_is_the_exact_dataset_batch_schedule() -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, "qwen-l0-rl-l3") - schedule = stage_schedule_document( - REPO_ROOT / "dataset", - stage, - groups_per_batch=8, - ) - assert schedule["maximum_steps"] == 30 - assert len(schedule["training_batches"]) == 30 - assert all(len(batch["groups"]) == 8 for batch in schedule["training_batches"]) - assert all( - len({group["task_id"] for group in batch["groups"]}) == 8 - for batch in schedule["training_batches"] - ) - replay_groups = sum( - group["role"] == "replay" - for batch in schedule["training_batches"] - for group in batch["groups"] - ) - assert replay_groups == 48 - - current = stage_tasks(REPO_ROOT / "dataset", stage) - from rl.studies.representation_training_v1.schedule import stage_replay_tasks - - replay = stage_replay_tasks(REPO_ROOT / "dataset", stage) - dataset = TrackARLDataset( - current, - replay, - renderer=object(), - groups_per_batch=8, - group_size=4, - max_image=1920, - n_batches=30, - shuffle=True, - reward_policy="raw_iou", - schedule_seed=SCHEDULE_SEED, - ) - for step, expected in enumerate(schedule["training_batches"]): - observed = [builder.task.sampler.opaque_id for builder in dataset.get_batch(step)] - assert observed == [group["task_id"] for group in expected["groups"]] - - -def test_sft_stage_counts_are_one_physical_pass() -> None: - protocol = load_protocol(REPO_ROOT) - expected = { - "qwen-l0-sft": (642, 11), - "qwen-mixed-sft": (3468, 55), - "qwen-sequential-sft-l1": (850, 14), - "qwen-sequential-sft-l2": (608, 10), - "qwen-sequential-sft-l3": (720, 12), - "qwen-sequential-sft-l4": (648, 11), - } - for stage_id, (rows, batches) in expected.items(): - stage = stage_spec(protocol, stage_id) - schedule = stage_schedule_document(REPO_ROOT / "dataset", stage) - assert (schedule["rows"], schedule["batches"]) == (rows, batches) - assert stage.wave("complete")["max_steps"] == batches - - -def test_pinned_tinker_top_p_default_is_one() -> None: - params = tinker.SamplingParams(max_tokens=1, temperature=1.0) - assert params.top_p == 1 - - -def test_wandb_identity_binds_protocol_source_stage_and_replicate() -> None: - key = StageKey(STUDY_ID, "qwen-mixed-sft", "r0") - values = { - deterministic_wandb_run_id( - source_git_sha="1" * 40, - protocol_logical_sha256="2" * 64, - key=key, - ), - deterministic_wandb_run_id( - source_git_sha="3" * 40, - protocol_logical_sha256="2" * 64, - key=key, - ), - deterministic_wandb_run_id( - source_git_sha="1" * 40, - protocol_logical_sha256="4" * 64, - key=key, - ), - } - assert len(values) == 3 diff --git a/rl/studies/representation_training_v1/tests/test_renderer_diagnostic.py b/rl/studies/representation_training_v1/tests/test_renderer_diagnostic.py deleted file mode 100644 index 15f413b9..00000000 --- a/rl/studies/representation_training_v1/tests/test_renderer_diagnostic.py +++ /dev/null @@ -1,783 +0,0 @@ -from __future__ import annotations - -import asyncio -import copy -import json -from collections.abc import Mapping -from pathlib import Path -from types import SimpleNamespace -from typing import Any - -import pytest - -pytest.importorskip("chz", reason="Tinker is an optional RL dependency") -pytest.importorskip("tinker", reason="Tinker is an optional RL dependency") -pytest.importorskip( - "tinker_cookbook", - reason="Tinker Cookbook is an optional RL dependency", -) - -from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus -from rl.common.prompt import prompt_asset_hashes -from rl.studies.representation_training_v1 import renderer_diagnostic as subject -from rl.studies.representation_training_v1.protocol import ( - STUDY_ID, - file_sha256, - load_protocol, - protocol_path, - stage_spec, -) -from rl.studies.representation_training_v1.schedule import stage_tasks -from rl.studies.representation_training_v1.store import StageKey, TrainingStore - -from .test_evaluation import ( - REPO_ROOT, - _dataset_binding, - _FakeEvaluator, - _FakeRenderer, - _task, -) - - -CURRENT_SOURCE_SHA = "c" * 40 -# The diagnostic seed binds the exact study protocol and producer receipt. -# Protocol logical SHA-256: c9bc59bd81aa0e6235aa307149e6fde02a1baf733756f31fe200ed911ffbee86. -# This inventory changes only when that sealed provenance fixture changes. -EXPECTED_TEST_SEEDS = [ - 657021824, - 1434315820, - 1815886577, - 79509377, - 1329366495, - 1182061749, - 85040921, - 1903607757, - 466527302, - 296515128, - 627644229, - 346320926, - 1567918527, - 1869035973, - 1896356246, - 1699500364, - 200413036, - 173011987, - 1116287135, - 1054920427, - 1406226972, - 1053591155, - 960197130, - 100306872, - 132036433, - 1041671166, - 771608684, - 943677354, - 2037721659, - 492363738, - 807566693, - 376994082, -] - - -def _tasks() -> list[Any]: - return [_task(f"F{index}") for index in range(1, 9)] - - -def _prefix_audit() -> dict[str, Any]: - renderers = {} - for renderer, compatible in ( - ("qwen3_5_disable_thinking", True), - ("qwen3_5", False), - ): - renderers[renderer] = { - "rows": 3468, - "compatible_rows": 3468 if compatible else 0, - "incompatible_rows": 0 if compatible else 3468, - "compatible_with_sft_supervised_prefix": compatible, - "expected_compatible_claim": compatible, - "claim_verified": True, - "comparison_scope": "through first positive-loss token", - "diagnosis": renderer, - "evidence_sha256": "8" * 64, - "example": {"renderer": renderer}, - } - return { - "schema_version": "pixcell-qwen-sft-prefix-audit-v1", - "rows": 3468, - "training_renderer": "qwen3_5", - "train_on_what": "LAST_ASSISTANT_MESSAGE", - "target_policy": "program_and_end_token_only", - "source_set_sha256": "9" * 64, - "renderers": renderers, - } - - -def _mixed_receipt( - tmp_path: Path, - *, - protocol: dict[str, Any], -) -> tuple[Path, str]: - external = tmp_path / "external" - store = TrainingStore(repo_root=REPO_ROOT, external_root=external) - key = StageKey( - STUDY_ID, - subject.PRODUCER_STAGE_ID, - subject.PRODUCER_REPLICATE_ID, - ) - checkpoints = ( - ("000014", 14, False, "periodic", 14 / 55), - ("000028", 28, False, "periodic", 28 / 55), - ("000042", 42, False, "periodic", 42 / 55), - ("final", 55, True, "terminal", 1.0), - ) - entries = [ - { - "name": name, - "batch": batch, - "epoch": 1 if final else 0, - "final": final, - "state_path": f"tinker://unit/state/{name}", - "sampler_path": f"tinker://unit/sampler/{name}", - "role": role, - "training_progress_fraction": progress, - } - for name, batch, final, role, progress in checkpoints - ] - terminal = { - field: entries[-1][field] - for field in ( - "name", - "batch", - "epoch", - "final", - "state_path", - "sampler_path", - ) - } - with store.acquire_stage_lock(key): - manifest = store.create_or_verify_manifest( - key, - { - "source_git_sha": subject.PRODUCER_SOURCE_GIT_SHA, - "protocol_logical_sha256": protocol["logical_sha256"], - "protocol_file_sha256": file_sha256(protocol_path(REPO_ROOT)), - "contract_version": protocol["contract_version"], - "prompt_assets": prompt_asset_hashes(), - "dataset": { - "logical_release_sha256": protocol["dataset"]["logical_release_sha256"] - }, - "runtime": {"pinned": True}, - "sandbox": {"image_id": "sha256:" + "4" * 64}, - "stage": { - "stage_id": subject.PRODUCER_STAGE_ID, - "kind": "sft", - "model_key": "qwen", - "recipe_key": "qwen_sft", - }, - "model": protocol["models"]["qwen"], - "recipe": protocol["recipes"]["qwen_sft"], - }, - ) - receipt = store.write_wave_receipt( - key, - wave=subject.PRODUCER_WAVE, - payload={ - "stage_id": subject.PRODUCER_STAGE_ID, - "wave": subject.PRODUCER_WAVE, - "run_manifest_record_sha256": manifest.record_sha256, - "checkpoint": terminal, - "checkpoint_inventory": { - "count": len(entries), - "entries": entries, - "logical_sha256": subject._canonical_sha256(entries), - }, - }, - ) - return external, receipt.record_sha256 - - -class _Renderer(_FakeRenderer): - def __init__(self, events: list[str], name: str) -> None: - super().__init__(events) - self.name = name - - def build_generation_prompt(self, messages: list[Any]) -> Any: - self.events.append(f"prompt:{self.name}") - assert len(messages) == 1 - return SimpleNamespace(length=102 if self.name == "qwen3_5_disable_thinking" else 100) - - -class _SharedSamplingState: - def __init__( - self, - events: list[str], - *, - fail_on_call: int | None = None, - ) -> None: - self.events = events - self.fail_on_call = fail_on_call - self.calls = 0 - self.seeds: list[int] = [] - - -class _SamplingClient: - def __init__(self, state: _SharedSamplingState, path: str) -> None: - self.state = state - self.path = path - - async def sample_async(self, **kwargs: Any) -> Any: - self.state.calls += 1 - if self.state.fail_on_call == self.state.calls: - raise RuntimeError("synthetic transport fault") - parameters = kwargs["sampling_params"] - assert kwargs["num_samples"] == 1 - assert parameters.max_tokens == subject.MAX_OUTPUT_TOKENS - assert parameters.temperature == subject.TEMPERATURE - assert parameters.top_p == subject.TOP_P - self.state.seeds.append(parameters.seed) - self.state.events.append(f"sample:{self.path}:{parameters.seed}") - return SimpleNamespace(sequences=[SimpleNamespace(tokens=[1, 2, 3], stop_reason="stop")]) - - -class _Service: - def __init__(self, state: _SharedSamplingState) -> None: - self.state = state - self.paths: list[str] = [] - - def create_sampling_client(self, *, model_path: str) -> _SamplingClient: - self.paths.append(model_path) - self.state.events.append(f"client:{model_path}") - return _SamplingClient(self.state, model_path) - - -def _patch_preflight( - monkeypatch: pytest.MonkeyPatch, - *, - protocol: dict[str, Any], - events: list[str], -) -> None: - monkeypatch.setattr( - subject, - "load_protocol", - lambda *_args, **_kwargs: protocol, - ) - - def source_check(_root: Path, expected: str) -> str: - events.append(f"source:{expected}") - assert expected == CURRENT_SOURCE_SHA - return expected - - monkeypatch.setattr(subject, "validate_source_sha", source_check) - monkeypatch.setattr( - subject, - "_source_provenance", - lambda **_kwargs: { - "producer_source_git_sha": subject.PRODUCER_SOURCE_GIT_SHA, - "evaluator_source_git_sha": CURRENT_SOURCE_SHA, - "git_relationship": {"ancestor_distance": 1}, - "task_contract_files": {}, - "task_contract_files_sha256": "a" * 64, - }, - ) - monkeypatch.setattr( - subject, - "_dataset_binding", - lambda *_args: _dataset_binding(protocol), - ) - monkeypatch.setattr( - subject, - "_runtime_binding", - lambda *_args: {"stack": "pinned"}, - ) - monkeypatch.setattr( - subject.checkpoint_v1, - "_benchmark_tasks", - lambda **_kwargs: _tasks(), - ) - monkeypatch.setattr( - subject, - "_sft_prefix_audit", - lambda **_kwargs: events.append("prefix-audit") or _prefix_audit(), - ) - renderers: dict[str, _Renderer] = {} - - def renderer(_model: str, name: str, **_kwargs: Any) -> _Renderer: - return renderers.setdefault(name, _Renderer(events, name)) - - monkeypatch.setattr(subject, "_renderer", renderer) - - -def _store(external: Path) -> subject.RendererDiagnosticStore: - training = TrainingStore(repo_root=REPO_ROOT, external_root=external) - key = StageKey( - STUDY_ID, - subject.PRODUCER_STAGE_ID, - subject.PRODUCER_REPLICATE_ID, - ) - return subject.RendererDiagnosticStore(stage_path=training.stage_path(key)) - - -def _run( - *, - protocol: dict[str, Any], - external: Path, - receipt_sha: str, - evaluator: _FakeEvaluator, - service_factory: Any, -) -> dict[str, Any]: - return asyncio.run( - subject.run_renderer_diagnostic( - repo_root=REPO_ROOT, - evaluator_source_git_sha=CURRENT_SOURCE_SHA, - producer_wave_receipt_record_sha256=receipt_sha, - external_root=external, - confirmation=protocol["launch"]["confirmation_token"], - evaluator_factory=lambda: evaluator, - service_client_factory=service_factory, - ) - ) - - -def _completed( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> dict[str, Any]: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _mixed_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - _patch_preflight(monkeypatch, protocol=protocol, events=events) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - evaluator = _FakeEvaluator(events) - state = _SharedSamplingState(events) - service = _Service(state) - - def service_factory() -> _Service: - events.append("service") - assert events.count(f"source:{CURRENT_SOURCE_SHA}") == 2 - assert "prefix-audit" in events - assert events.count("reference") == 32 - return service - - result = _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=evaluator, - service_factory=service_factory, - ) - store = _store(external) - manifest = store.load_manifest() - report = store.load_report() - assert manifest is not None - assert report is not None - return { - "protocol": protocol, - "external": external, - "receipt_sha": receipt_sha, - "events": events, - "evaluator": evaluator, - "state": state, - "service": service, - "result": result, - "store": store, - "manifest": manifest, - "report": report, - } - - -def _write_record(path: Path, value: Mapping[str, Any]) -> None: - path.write_text( - json.dumps( - value, - allow_nan=False, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ) - + "\n", - encoding="utf-8", - ) - - -def test_matrix_and_cli_have_no_model_renderer_or_checkpoint_override() -> None: - assert [spec.as_dict() for spec in subject.ARM_SPECS] == [ - { - "arm_id": "final-no-thinking", - "checkpoint_name": "final", - "renderer": "qwen3_5_disable_thinking", - "expected_sft_prefix_compatible": True, - }, - { - "arm_id": "step-14-thinking", - "checkpoint_name": "000014", - "renderer": "qwen3_5", - "expected_sft_prefix_compatible": False, - }, - { - "arm_id": "step-28-thinking", - "checkpoint_name": "000028", - "renderer": "qwen3_5", - "expected_sft_prefix_compatible": False, - }, - { - "arm_id": "step-42-thinking", - "checkpoint_name": "000042", - "renderer": "qwen3_5", - "expected_sft_prefix_compatible": False, - }, - ] - assert subject.PRODUCER_SOURCE_GIT_SHA == ("bb80dbe666c7313f9a9f3bdb53765c35dfdeceab") - actions = {action.dest for action in subject.parser()._actions if action.dest} - assert not actions & { - "model", - "renderer", - "checkpoint", - "checkpoint_name", - "stage", - "wave", - "replicate", - "max_tokens", - "temperature", - "top_p", - } - - -def test_real_renderer_prefix_audit_diagnoses_exact_mismatch() -> None: - protocol = load_protocol(REPO_ROOT) - stage = stage_spec(protocol, subject.PRODUCER_STAGE_ID) - anchor = stage_tasks(REPO_ROOT / "dataset", stage)[0] - audit = subject._sft_prefix_audit( - dataset_root=REPO_ROOT / "dataset", - protocol=protocol, - tasks=[anchor], - ) - - disabled = audit["renderers"]["qwen3_5_disable_thinking"] - canonical = audit["renderers"]["qwen3_5"] - assert disabled["compatible_with_sft_supervised_prefix"] is True - assert disabled["compatible_rows"] == 1 - assert disabled["example"]["prefix_equal"] is True - assert disabled["example"]["through_first_positive_equal"] is True - assert canonical["compatible_with_sft_supervised_prefix"] is False - assert canonical["incompatible_rows"] == 1 - assert canonical["example"]["prefix_equal"] is False - assert canonical["example"]["through_first_positive_equal"] is False - assert "leaves open" in canonical["diagnosis"] - - -def test_exact_matrix_records_full_provenance_and_is_non_promotable( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - result = completed["result"] - manifest = completed["manifest"]["payload"] - report = completed["report"]["payload"] - store = completed["store"] - - assert result["status"] == "complete" - assert result["not_canonical_evidence"] is True - assert "/diagnostics/" in result["report"] - assert "/evaluations/" not in result["report"] - assert completed["state"].calls == 32 - assert len(completed["state"].seeds) == 32 - assert len(set(completed["state"].seeds)) == 32 - assert completed["service"].paths == [ - "tinker://unit/sampler/final", - "tinker://unit/sampler/000014", - "tinker://unit/sampler/000028", - "tinker://unit/sampler/000042", - ] - assert manifest["not_canonical_evidence"] is True - assert manifest["promotion_eligibility"] == { - "eligible": False, - "policy": "diagnostic-only-never-promotion-evidence", - } - assert manifest["source"]["producer_source_git_sha"] == (subject.PRODUCER_SOURCE_GIT_SHA) - assert manifest["source"]["evaluator_source_git_sha"] == CURRENT_SOURCE_SHA - assert ( - manifest["producer_training"]["wave_receipt"]["record_sha256"] == completed["receipt_sha"] - ) - assert list( - manifest["producer_training"]["checkpoint_inventory"]["entries"][index]["name"] - for index in range(4) - ) == list(subject.EXPECTED_CHECKPOINT_NAMES) - assert manifest["sampling"]["total_samples"] == 32 - assert len(manifest["sampling"]["seeds"]) == 32 - assert [row["seed"] for row in manifest["sampling"]["seeds"]] == EXPECTED_TEST_SEEDS - assert report["not_canonical_evidence"] is True - assert len(report["records"]) == 32 - assert set(report["summaries"]) == {spec.arm_id for spec in subject.ARM_SPECS} - assert all( - summary["mean_raw_absolute_scale_iou"] == 0.75 and summary["pure_executable"] == 8 - for summary in report["summaries"].values() - ) - for spec in subject.ARM_SPECS: - for task_id in subject.EXPECTED_TASK_IDS: - sample = store.load_sample(spec.arm_id, task_id) - evaluation = store.load_evaluation(spec.arm_id, task_id) - assert sample is not None - assert evaluation is not None - response = sample["payload"]["response"] - assert response["token_ids"] == [1, 2, 3] - assert response["token_ids_sha256"] == subject._canonical_sha256([1, 2, 3]) - assert response["answer_text"] == "print('candidate')" - assert response["reasoning_text"] == "" - assert response["extracted_program"] == "print('candidate')" - assert response["channel_parse_status"] == "clean" - assert evaluation["payload"]["not_canonical_evidence"] is True - assert ( - evaluation["payload"]["extracted_program_sha256"] - == response["extracted_program_sha256"] - ) - - with pytest.raises(subject.checkpoint_v1.CheckpointEvaluationError): - subject.checkpoint_v1._validate_record( - completed["report"], - record_type="checkpoint_report", - key={}, - ) - - -def test_already_complete_is_byte_stable_and_creates_no_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - store = completed["store"] - before = {path: path.read_bytes() for path in store.root.rglob("*.json")} - monkeypatch.delenv("TINKER_API_KEY") - - def forbidden_service() -> Any: - raise AssertionError("already-complete diagnostic created a client") - - result = _run( - protocol=completed["protocol"], - external=completed["external"], - receipt_sha=completed["receipt_sha"], - evaluator=_FakeEvaluator(completed["events"]), - service_factory=forbidden_service, - ) - assert result["status"] == "already_complete" - assert {path: path.read_bytes() for path in store.root.rglob("*.json")} == before - - -def test_partial_sampling_resume_only_fills_missing_rows( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _mixed_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - _patch_preflight(monkeypatch, protocol=protocol, events=events) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - monkeypatch.setattr(subject, "SAMPLE_CONCURRENCY", 1) - first_state = _SharedSamplingState(events, fail_on_call=7) - - with pytest.raises(subject.checkpoint_v1.SamplingInfrastructureFault): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=lambda: _Service(first_state), - ) - - store = _store(external) - before = {path: path.read_bytes() for path in store.root.glob("arms/*/tasks/*/sample.json")} - assert 0 < len(before) < 32 - assert not list(store.root.glob("arms/*/tasks/*/evaluation.json")) - - second_state = _SharedSamplingState(events) - result = _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=lambda: _Service(second_state), - ) - assert result["status"] == "complete" - assert second_state.calls == 32 - len(before) - assert len(list(store.root.glob("arms/*/tasks/*/sample.json"))) == 32 - assert all(path.read_bytes() == raw for path, raw in before.items()) - - -@pytest.mark.parametrize("mutation", ("raw", "program", "foreign", "symlink")) -def test_tampering_and_foreign_inventory_fail_before_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - mutation: str, -) -> None: - completed = _completed(tmp_path, monkeypatch) - store = completed["store"] - sample_path = store.root / "arms" / "final-no-thinking" / "tasks" / "F1" / "sample.json" - if mutation in {"raw", "program"}: - sample = json.loads(sample_path.read_text(encoding="utf-8")) - field = "raw_text" if mutation == "raw" else "extracted_program" - sample["payload"]["response"][field] = "tampered" - _write_record(sample_path, sample) - elif mutation == "foreign": - foreign = store.root / "arms" / "final-no-thinking" / "tasks" / "F9" / "sample.json" - foreign.parent.mkdir(parents=True) - foreign.write_bytes(sample_path.read_bytes()) - else: - link = store.root / "arms" / "final-no-thinking" / "tasks" / "F9" - link.symlink_to(sample_path.parent, target_is_directory=True) - - def forbidden_service() -> Any: - raise AssertionError("invalid resume created a client") - - with pytest.raises(subject.RendererDiagnosticError): - _run( - protocol=completed["protocol"], - external=completed["external"], - receipt_sha=completed["receipt_sha"], - evaluator=_FakeEvaluator(completed["events"]), - service_factory=forbidden_service, - ) - - -def test_wrong_receipt_and_second_source_check_precede_client( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - protocol = load_protocol(REPO_ROOT) - external, receipt_sha = _mixed_receipt(tmp_path, protocol=protocol) - events: list[str] = [] - _patch_preflight(monkeypatch, protocol=protocol, events=events) - monkeypatch.setenv("TINKER_API_KEY", "test-key") - - def forbidden_service() -> Any: - raise AssertionError("failed preflight created a client") - - with pytest.raises(subject.checkpoint_v1.CheckpointReceiptError): - _run( - protocol=protocol, - external=external, - receipt_sha="f" * 64, - evaluator=_FakeEvaluator(events), - service_factory=forbidden_service, - ) - - calls = 0 - - def changing_source(_root: Path, expected: str) -> str: - nonlocal calls - calls += 1 - if calls == 2: - raise ValueError("source changed after preflight") - return expected - - monkeypatch.setattr(subject, "validate_source_sha", changing_source) - with pytest.raises(ValueError, match="source changed"): - _run( - protocol=protocol, - external=external, - receipt_sha=receipt_sha, - evaluator=_FakeEvaluator(events), - service_factory=forbidden_service, - ) - assert calls == 2 - - -def test_model_failures_zero_and_infrastructure_failures_abort() -> None: - arm = SimpleNamespace( - spec=subject.ARM_SPECS[0], - binding=None, - ) - attempt = subject.PreparedAttempt( - arm=arm, - prepared=subject.checkpoint_v1.PreparedTask( - task=_task(), - prompt_tokens=100, - prompt_text_sha256="1" * 64, - reference_evidence={}, - ), - seed=123, - ) - model_failure = subject._evaluation_payload( - attempt=attempt, - result=EvaluationResult( - status=EvaluationStatus.SYNTAX_ERROR, - attribution=Attribution.MODEL, - error="invalid syntax", - ), - sample_record_sha256="2" * 64, - manifest_record_sha256="3" * 64, - extracted_program_sha256="4" * 64, - ) - assert model_failure["raw_absolute_scale_iou"] == 0.0 - assert model_failure["pure_executable"] is False - - for attribution, error_type in ( - (Attribution.REFERENCE, subject.checkpoint_v1.ReferencePanelFault), - ( - Attribution.EVALUATOR, - subject.checkpoint_v1.EvaluationInfrastructureFault, - ), - ): - with pytest.raises(error_type): - subject._evaluation_payload( - attempt=attempt, - result=EvaluationResult( - status=EvaluationStatus.WORKER_ERROR, - attribution=attribution, - error="fault", - ), - sample_record_sha256="2" * 64, - manifest_record_sha256="3" * 64, - extracted_program_sha256="4" * 64, - ) - - -def test_resigned_report_marker_or_metrics_tampering_is_rejected( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - completed = _completed(tmp_path, monkeypatch) - for mutation in ("marker", "summary"): - payload = copy.deepcopy(completed["report"]["payload"]) - if mutation == "marker": - payload["not_canonical_evidence"] = False - else: - payload["summaries"]["final-no-thinking"]["mean_raw_absolute_scale_iou"] = 0.99 - record = subject._record_document( - record_type="renderer_diagnostic_report", - key={}, - payload=payload, - ) - with pytest.raises(subject.RendererDiagnosticError): - subject._validate_report_rows( - record, - manifest_record=completed["manifest"], - attempts=_attempt_context(completed), - ) - - -def _attempt_context(completed: Mapping[str, Any]) -> tuple[Any, ...]: - protocol = completed["protocol"] - training = TrainingStore( - repo_root=REPO_ROOT, - external_root=completed["external"], - ) - bindings = subject._load_matrix_bindings( - store=training, - protocol=protocol, - expected_wave_receipt_sha256=completed["receipt_sha"], - ) - evaluator = _FakeEvaluator([]) - tasks = _tasks() - panel = subject._task_panel(tasks) - arms, attempts = subject._prepare_arms( - protocol=protocol, - model=protocol["models"]["qwen"], - bindings=bindings, - tasks=tasks, - evaluator=evaluator, - evaluator_source_git_sha=CURRENT_SOURCE_SHA, - dataset_sha256=protocol["dataset"]["logical_release_sha256"], - task_panel_sha256=panel["logical_sha256"], - ) - assert len(arms) == 4 - evaluator.close() - return attempts diff --git a/rl/studies/representation_training_v1/tests/test_source_transition.py b/rl/studies/representation_training_v1/tests/test_source_transition.py deleted file mode 100644 index 3aab01f1..00000000 --- a/rl/studies/representation_training_v1/tests/test_source_transition.py +++ /dev/null @@ -1,243 +0,0 @@ -from __future__ import annotations - -import copy -import subprocess -from pathlib import Path - -import pytest - -from rl.studies.representation_training_v1.protocol import ( - validate_source_sha, -) -from rl.studies.representation_training_v1.source_transition import ( - SourceTransitionError, - approved_change_scope, - build_source_transition, - source_transition_git_facts, - validate_source_transition, -) - - -def _git(repo: Path, *args: str) -> str: - return subprocess.check_output( - ["git", *args], - cwd=repo, - text=True, - ).strip() - - -def _commit(repo: Path, message: str) -> str: - subprocess.run(["git", "add", "--all"], cwd=repo, check=True) - subprocess.run( - ["git", "commit", "-m", message], - cwd=repo, - check=True, - stdout=subprocess.DEVNULL, - ) - return _git(repo, "rev-parse", "HEAD") - - -@pytest.fixture -def git_history(tmp_path: Path) -> tuple[Path, str, str]: - repo = tmp_path / "repo" - repo.mkdir() - subprocess.run( - ["git", "init", "--initial-branch=main"], - cwd=repo, - check=True, - stdout=subprocess.DEVNULL, - ) - subprocess.run( - ["git", "config", "user.name", "PixCell test"], - cwd=repo, - check=True, - ) - subprocess.run( - ["git", "config", "user.email", "pixcell-test@example.invalid"], - cwd=repo, - check=True, - ) - subprocess.run( - ["git", "config", "commit.gpgsign", "false"], - cwd=repo, - check=True, - ) - subprocess.run( - ["git", "config", "core.hooksPath", "/dev/null"], - cwd=repo, - check=True, - ) - (repo / "policy.py").write_text("VALUE = 1\n", encoding="utf-8") - producer = _commit(repo, "producer") - (repo / "policy.py").write_text("VALUE = 2\n", encoding="utf-8") - (repo / "review.txt").write_text("reviewed\n", encoding="utf-8") - consumer = _commit(repo, "consumer") - return repo, producer, consumer - - -def test_transition_binds_ancestry_trees_diff_and_exact_scope( - git_history: tuple[Path, str, str], - monkeypatch: pytest.MonkeyPatch, -) -> None: - repo, producer, consumer = git_history - facts = source_transition_git_facts( - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - assert facts["merge_base_git_sha"] == producer - assert facts["producer_source_git_sha"] == producer - assert facts["consumer_source_git_sha"] == consumer - assert facts["producer_tree_git_sha"] != facts["consumer_tree_git_sha"] - assert facts["ancestor_distance"] == 1 - assert facts["changed_path_count"] == 2 - assert facts["changed_paths"] == [ - {"status": "M", "path": "policy.py"}, - {"status": "A", "path": "review.txt"}, - ] - assert facts["raw_changes"] == [ - { - "status": "M", - "old_mode": "100644", - "new_mode": "100644", - "old_object": _git(repo, "rev-parse", f"{producer}:policy.py"), - "new_object": _git(repo, "rev-parse", f"{consumer}:policy.py"), - "path": "policy.py", - }, - { - "status": "A", - "old_mode": "000000", - "new_mode": "100644", - "old_object": "0" * 40, - "new_object": _git(repo, "rev-parse", f"{consumer}:review.txt"), - "path": "review.txt", - }, - ] - original_inventory_sha256 = facts["raw_change_inventory_sha256"] - monkeypatch.setenv("GIT_DIFF_OPTS", "--unified=0") - monkeypatch.setenv("GIT_CONFIG_COUNT", "2") - monkeypatch.setenv("GIT_CONFIG_KEY_0", "diff.algorithm") - monkeypatch.setenv("GIT_CONFIG_VALUE_0", "minimal") - monkeypatch.setenv("GIT_CONFIG_KEY_1", "diff.renames") - monkeypatch.setenv("GIT_CONFIG_VALUE_1", "true") - assert ( - source_transition_git_facts( - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - == facts - ) - assert facts["raw_change_inventory_sha256"] == original_inventory_sha256 - subprocess.run( - ["git", "replace", producer, consumer], - cwd=repo, - check=True, - ) - assert ( - source_transition_git_facts( - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - == facts - ) - - paths = ["review.txt", "policy.py"] - approved = approved_change_scope(facts, paths) - assert approved["paths"] == ["policy.py", "review.txt"] - assert approved["entries"] == facts["raw_changes"] - assert approved["raw_change_inventory_sha256"] == facts["raw_change_inventory_sha256"] - - transition = build_source_transition( - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - scope={"child": {"stage_id": "child", "wave": "smoke"}}, - approved_change_paths=paths, - equivalence={"protocol": "exact"}, - evidence={"report_record_sha256": "a" * 64}, - ) - assert ( - validate_source_transition( - transition, - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - scope={"child": {"stage_id": "child", "wave": "smoke"}}, - approved_change_paths=paths, - equivalence={"protocol": "exact"}, - evidence={"report_record_sha256": "a" * 64}, - ) - == transition - ) - - tampered = copy.deepcopy(transition) - tampered["git"]["raw_changes"][0]["new_object"] = "0" * 40 - with pytest.raises(SourceTransitionError, match="differs"): - validate_source_transition( - tampered, - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - scope={"child": {"stage_id": "child", "wave": "smoke"}}, - approved_change_paths=paths, - equivalence={"protocol": "exact"}, - evidence={"report_record_sha256": "a" * 64}, - ) - - -def test_transition_rejects_wrong_scope_and_non_ancestor( - git_history: tuple[Path, str, str], -) -> None: - repo, producer, consumer = git_history - facts = source_transition_git_facts( - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - with pytest.raises(SourceTransitionError, match="approved change paths"): - approved_change_scope(facts, ["policy.py"]) - with pytest.raises(SourceTransitionError, match="approved change paths"): - approved_change_scope( - facts, - ["policy.py", "review.txt", "unrelated.txt"], - ) - with pytest.raises(SourceTransitionError, match="explicit approved"): - approved_change_scope(facts, None) - - subprocess.run( - ["git", "switch", "--detach", producer], - cwd=repo, - check=True, - stdout=subprocess.DEVNULL, - ) - (repo / "sibling.txt").write_text("sibling\n", encoding="utf-8") - sibling = _commit(repo, "sibling") - with pytest.raises(SourceTransitionError, match="not an ancestor"): - source_transition_git_facts( - repo_root=repo, - producer_source_git_sha=sibling, - consumer_source_git_sha=consumer, - ) - grafts = repo / ".git" / "info" / "grafts" - grafts.write_text("", encoding="utf-8") - with pytest.raises(SourceTransitionError, match="grafts"): - source_transition_git_facts( - repo_root=repo, - producer_source_git_sha=producer, - consumer_source_git_sha=consumer, - ) - - -def test_source_validation_rejects_wrong_sha_and_dirty_tree( - git_history: tuple[Path, str, str], -) -> None: - repo, producer, consumer = git_history - assert validate_source_sha(repo, consumer) == consumer - with pytest.raises(ValueError, match="differs"): - validate_source_sha(repo, producer) - - (repo / "untracked.txt").write_text("dirty\n", encoding="utf-8") - with pytest.raises(ValueError, match="clean worktree"): - validate_source_sha(repo, consumer) diff --git a/rl/studies/representation_training_v1/tests/test_store.py b/rl/studies/representation_training_v1/tests/test_store.py deleted file mode 100644 index f928a20c..00000000 --- a/rl/studies/representation_training_v1/tests/test_store.py +++ /dev/null @@ -1,138 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import pytest - -from rl.studies.representation_training_v1.store import ( - ImmutableTrainingRecordError, - StageKey, - TrainingStore, - TrainingStoreError, - UnsafeTrainingRootError, -) - - -def test_store_rejects_repository_overlap(tmp_path: Path) -> None: - repo = tmp_path / "repo" - repo.mkdir() - with pytest.raises(UnsafeTrainingRootError): - TrainingStore(repo_root=repo, external_root=repo / "runs") - - -def test_records_require_lock_and_are_immutable(tmp_path: Path) -> None: - repo = tmp_path / "repo" - external = tmp_path / "external" - repo.mkdir() - store = TrainingStore(repo_root=repo, external_root=external) - key = StageKey("study", "stage", "r0") - with pytest.raises(TrainingStoreError): - store.create_or_verify_manifest(key, {"value": 1}) - with store.acquire_stage_lock(key): - first = store.create_or_verify_manifest(key, {"value": 1}) - assert store.create_or_verify_manifest(key, {"value": 1}) == first - with pytest.raises(ImmutableTrainingRecordError): - store.create_or_verify_manifest(key, {"value": 2}) - - -def test_crash_retry_invocations_never_collide_candidates(tmp_path: Path) -> None: - repo = tmp_path / "repo" - external = tmp_path / "external" - repo.mkdir() - store = TrainingStore(repo_root=repo, external_root=external) - key = StageKey("study", "stage", "r0") - with store.acquire_stage_lock(key): - store.create_or_verify_manifest(key, {"value": 1}) - for invocation, score in (("smoke-first", 0.1), ("smoke-retry", 0.2)): - store.begin_invocation( - key, - invocation_id=invocation, - payload={"invocation_id": invocation}, - ) - store.write_candidate( - key, - invocation_id=invocation, - step=0, - task_id="abc123", - attempt=1, - payload={"iou": score}, - ) - store.write_candidate_sample( - key, - invocation_id=invocation, - step=0, - task_id="abc123", - attempt=1, - payload={"tokens": [1, 2, 3]}, - ) - candidates = sorted(store.candidate_root(key).rglob("evaluation.json")) - samples = sorted(store.candidate_root(key).rglob("sample.json")) - assert len(candidates) == 2 - assert len(samples) == 2 - assert {path.parts[-5] for path in candidates} == { - "smoke-first", - "smoke-retry", - } - - -def test_invocation_tracking_is_create_only_and_requires_start( - tmp_path: Path, -) -> None: - repo = tmp_path / "repo" - external = tmp_path / "external" - repo.mkdir() - store = TrainingStore(repo_root=repo, external_root=external) - key = StageKey("study", "stage", "r0") - payload = { - "expected": {"run_id": "pxct-expected"}, - "observed": {"run_id": "pxct-expected"}, - } - with store.acquire_stage_lock(key): - store.create_or_verify_manifest(key, {"value": 1}) - with pytest.raises(TrainingStoreError, match="cannot precede"): - store.write_invocation_tracking( - key, - invocation_id="complete-unit", - payload=payload, - ) - invocation = store.begin_invocation( - key, - invocation_id="complete-unit", - payload={"invocation_id": "complete-unit"}, - ) - assert ( - store.load_invocation( - key, - invocation_id="complete-unit", - ) - == invocation - ) - tracking = store.write_invocation_tracking( - key, - invocation_id="complete-unit", - payload=payload, - ) - assert ( - store.load_invocation_tracking( - key, - invocation_id="complete-unit", - ) - == tracking - ) - assert ( - store.write_invocation_tracking( - key, - invocation_id="complete-unit", - payload=payload, - ) - == tracking - ) - with pytest.raises(ImmutableTrainingRecordError): - store.write_invocation_tracking( - key, - invocation_id="complete-unit", - payload={ - **payload, - "observed": {"run_id": "random"}, - }, - ) diff --git a/rl/track_a/README.md b/rl/track_a/README.md index 35db50f1..4833f89e 100644 --- a/rl/track_a/README.md +++ b/rl/track_a/README.md @@ -1,221 +1,60 @@ # PixCell Track A -Track A trains one self-hostable vision-language model to reconstruct a shown -photonic geometry as a primitive-only GDSFactory program. This directory is the -training boundary over the frozen PixCell Dataset. It does not contain private -paper benchmarks, historical rollouts, or model checkpoints. - -The reusable data adapters, evaluator boundary, and contract checks live here. -The current paid campaign is not launched through the older configurable -`train_sft.py`, `train_rl.py`, or `config.json` entry points; that draft remains -disabled for provenance. The sealed models, 60k thinking policy, branch graph, -raw-IoU reward, schedules, W&B identity, and guarded command for the current -campaign live in -[`representation_training_v1`](../studies/representation_training_v1/README.md). +Track A trains one self-hostable vision-language model +(`Qwen/Qwen3.6-35B-A3B`) to reconstruct a shown photonic geometry as a +primitive-only GDSFactory program. This directory is the training boundary +over the frozen PixCell Dataset. It does not contain private paper +benchmarks, historical rollouts, or model checkpoints. ## What the model receives -The `pixcell-direct-reconstruction-v2` contract is a one-shot adaptation of -the blind Phase-A reconstruction task. It has one user turn: +The `pixcell-direct-reconstruction-v3` contract is one blind user turn: 1. the row's maximum-visibility `image`; -2. its physical `footprint_um`, formatted as the historical footprint JSON; +2. its physical `footprint_um` as the historical footprint JSON, plus — + when the image's aspect is distorted more than 1.6× relative to the + footprint — an honest per-axis render-scale note computed from the + image's own ink bounding box (half the training rows are distorted + past that threshold; 78% of L4); 3. the exact PixCell GDSFactory extended primitive catalogue; -4. the direct Phase-A programming contract used by both training and - evaluation. +4. the direct Phase-A programming contract. -There is no system turn. The model does not receive the dataset's stored +There is no system turn. The model never receives the dataset's stored `instruction`, display calibration, `target_image`, row ID, class name, representation metadata, ports, structural witnesses, or expected code. -The programming contract states the required construction and serialization -rules: GDSFactory 9.20.7, material on layer `(1,0)`, catalogue-only primitive -constructors, top-level live function tunables, an `@gf.cell` device, complete -executable Python, and the exact output file `device.gds`. - -This is not the full tool-using HOMI Phase-A loop. That system lets an agent -derive calibration, inspect deterministic diffs, and revise its program. The -direct contract deliberately presents the same blind device inputs without -those tools or verifier feedback so it can be used consistently for SFT, -one-shot RL rollouts, and direct-policy evaluation. - -For SFT, the assistant label is the row's complete `code`. In other words, an -SFT example is still an image-and-code pair: the image and fixed task context -form the input, and the code is the supervised output. - -## Dataset boundary - -Track A pins `qpaig-mit/pixcell` at revision `v2.0.0`. The model-facing -configuration is `depth`; the separately keyed `references` configuration is -verifier-only. - -The `depth` table supplies the maximum-visibility `image`, physical -`footprint_um`, supervised `code`, and private sampler metadata. It does not -supply a ready-made natural-language instruction. At runtime Track A constructs -the prompt from the image and footprint plus the versioned PixCell programming -contract described above. Sampler metadata selects and balances tasks but is -never serialized into the prompt. - -The `references` table supplies the target raster and verification metadata, -joined by opaque row ID only after the model produces a program. It must never -be included in SFT messages, generation prompts, or model-visible tool output. -Local training uses the equivalent frozen Parquet files under `dataset/`; the -public repository and revision identify the portable release of the same -contract. - -## What the verifier sees - -Only after a completion exists, the evaluator receives the private -`target_image` and `footprint_um`. It: - -1. compiles and applies the canonical AST source policy before execution; -2. executes the original complete program in a fresh, networkless, - resource-limited container; -3. requires a non-empty `device.gds`; -4. renders layer `(1,0)` with Michaelangelo's absolute-scale, left-aligned and - vertically centered mapping; -5. returns full-precision raw IoU and diagnostics. - -Syntax, purity, runtime, missing-GDS, and invalid-GDS outcomes are model -failures. Broken references and evaluator faults are separate typed outcomes. -Evaluator faults are retried once and then abort the paid stage. Broken -references abort immediately. Neither can be converted into zero-quality -programs or silently dropped from a group. - -The initial reward is intentionally small: - -```text -model failure 0 -valid primitive-only program 0.05 + 0.95 * raw IoU -reference/evaluator failure no policy reward -``` - -Dice, calibrated render information, and artifact hashes are recorded as -diagnostics. They do not create reward cliffs in the first pilot. Port, count, -and structural checks can be added later as diagnostics without changing the -initial reward contract. - -## Training recipe - -The committed Qwen recipe is -[`config.json`](config.json). - -SFT visits every one of the 3,468 `depth/train` rows exactly once. Batches -interleave levels and representations. After assistant-token normalization, -the row loss is weighted so each level, then each representation within that -level, receives equal total loss mass. No row is omitted or duplicated. The -1,092 parameter-holdout rows are used only for validation. - -RL resumes the selected SFT checkpoint. Each task is sampled four times (G4). -All four programs are independently measured; the cookbook centers advantages -within that group, so programs above their prompt's group mean are reinforced -and programs below it are suppressed. This is not best-of-four branch search -and the model is not shown which answer won. - -Each paid RL stage is bounded to 30 steps, 8 prompt groups per step, and 4 -rollouts per group, with checkpoints and executable evaluation every 5 steps. -The first stage begins at the first level the selected SFT checkpoint has not -already mastered, rather than assuming L0. After promotion, every five-step -cycle uses an exact 80/20 current-level to earlier-level replay schedule. +The SFT user message is byte-identical to the RL sampling prompt — prompt +identity is what makes an SFT checkpoint resumable by RL. -## Zero-spend preflight +## Reward -Use the Python 3.13 environment from the dataset release and expose both source -roots: +Three policies live in [`reward.py`](reward.py); the campaign uses +`shaped_v3b` (see [`../README.md`](../README.md) for the formula and its +rationale). `raw_iou` and `validity_floor` remain for the record. Fault +attribution is absolute: model failures are typed zeros, reference or +evaluator faults are `None` and abort the run rather than becoming biased +zeros. -```bash -PYTHONPATH=src:. python -m rl.track_a.preflight --audit-only -PYTHONPATH=src:. pytest -q rl/common/tests rl/track_a/tests -``` - -The preflight checks every depth source and reference, catalogue/source-gate -parity, prompt leakage, frozen row counts, and representative ground-truth -self-scores. The paid-launch form also tokenizes all 3,468 SFT rows and proves -that none exceed 6,144 tokens. It does not create a Tinker client or spend -money. - -Both launchers are spend-locked: - -```bash -PYTHONPATH=src:. python -m rl.track_a.train_sft -PYTHONPATH=src:. python -m rl.track_a.train_rl -``` - -They exit before creating a client unless the matching explicit -`confirm_spend` value is supplied. RL additionally requires the selected SFT -weights checkpoint and its full executable mastery report. RL and checkpoint -evaluation also require the immutable local candidate image: - -```bash -python rl/common/sandbox/build.py -export PIXCELL_EVALUATOR_IMAGE=sha256: -``` - -At SFT steps 14, 28, and the final step 55, export the corresponding sampler -checkpoint and run two distinct measurements: - -1. Run `evaluate_checkpoint.py` at `k=1` over all of - `depth/validation`. Geometry reports reject `--rows` and `--levels`; this is - always the complete 1,092-task, 546-representation checkpoint quality - report. -2. Run the same command with `--behavioral-probe --k 4 --rows 40` and - exactly one `--levels` value set to the first curriculum level not yet - mastered. The - deterministic balanced sampler spreads this small probe across - representations in that level. - -The G4 probe records, for every prompt, hashes and counts of distinct extracted -programs and successful calibrated renders. It also reports the fraction of -groups with nonconstant rewards and the mean within-group reward standard -deviation. It declares collapse only when every probed prompt returns the same -program, or when every prompt has at most one successful render and a constant -reward. These behavioral quantities are diagnostics and a conservative -collapse veto, never checkpoint-ranking features. - -For example: +## Files -```bash -PYTHONPATH=src:. python -m rl.track_a.evaluate_checkpoint \ - --confirm-spend TRACK_A_EVAL \ - --sampler-path "$STEP14_SAMPLER" \ - --output rl/track_a/runs/step14-k1.json - -PYTHONPATH=src:. python -m rl.track_a.evaluate_checkpoint \ - --confirm-spend TRACK_A_EVAL \ - --sampler-path "$STEP14_SAMPLER" \ - --behavioral-probe --k 4 --rows 40 --levels L2 \ - --output rl/track_a/runs/step14-g4.json -``` +| File | Role | +|------|------| +| [`reward.py`](reward.py) | The three reward policies + the shaped-component breakdown | +| [`tinker_data.py`](tinker_data.py) | Cookbook adapters: supervised dataset (level-filtered, exactly-once stratified pass, hierarchical loss mass), the single-turn Env, group builder, RL dataset (staged level + exact 80/20 replay + seed-7 validation canaries) | +| [`curriculum.py`](curriculum.py) | Level algebra and the deterministic schedules | +| [`train_sft.py`](train_sft.py) / [`train_rl.py`](train_rl.py) | Lean single-stage entry points (chz CLIs) | +| [`run.py`](run.py) | The four-run campaign runner: stage sequencing, checkpoint chaining, SFT sanity gate, fixed probe, provenance | +| [`probe_eval.py`](probe_eval.py) | The fixed 80-task cross-run probe | +| [`preflight.py`](preflight.py) | The zero-spend gate (digests, whitelists, token audit, ground-truth control, sandbox self-test) | +| [`launch.py`](launch.py) | Sandbox binding + preflight subprocess wrapper | -Pair all checkpoint reports by sampler path and select with: +## Commands ```bash -PYTHONPATH=src:. python -m rl.track_a.select_sft_checkpoint \ - --geometry-reports rl/track_a/runs/step14-k1.json \ - rl/track_a/runs/step28-k1.json rl/track_a/runs/step55-k1.json \ - --behavioral-reports rl/track_a/runs/step14-g4.json \ - rl/track_a/runs/step28-g4.json rl/track_a/runs/step55-g4.json +python -m rl.track_a.run --smoke nothink --confirm-spend PIXCELL_RUNS_V2 +python -m rl.track_a.run --run A --confirm-spend PIXCELL_RUNS_V2 # or B/C/D ``` -After vetoing explicit sampling collapse, selection uses only the full `k=1` -representation-macro raw IoU, with pure-executable rate as a tie-breaker. -Validation NLL is logged but does not choose the checkpoint. The selected -checkpoint's **weights** path is then passed to `train_rl.py`; its sampler path -is only for evaluation. The guarded RL launch requires those paths to be the -`weights` and `sampler_weights` variants of the same Tinker checkpoint, binds -the weights path and the mastery-report SHA-256 into preflight, and revalidates -the report before creating a paid client. Evaluation similarly binds the exact -sampler path and a new JSON output below `rl/track_a/runs/`; it will not -overwrite an existing report. - -Every executable report carries the frozen depth release digest, source Git -commit, model, renderer, split, exact level and task-set digest, task and -attempt counts, and sampling parameters. Selection recomputes that contract -from the current committed repository and rejects partial reports, stale or -mismatched report pairs, non-finite or out-of-range metrics, and behavioral -probes other than the exact 40-prompt G4 contract. - -Track A refuses RL or checkpoint evaluation unless its per-candidate Docker -boundary is active. The launcher retains Tinker network access; each sampled -program receives no network, credentials, host filesystem, or mutable root -filesystem. The boundary is defense in depth around the canonical source -policy, not a claim that arbitrary Docker-hostile code is universally safe. +Every launch requires `TINKER_API_KEY`, `WANDB_API_KEY`, +`PIXCELL_EVALUATOR_IMAGE` (the immutable sandbox), a clean worktree, and +passing preflight. Spend is confirmed explicitly; nothing launches without +`--confirm-spend PIXCELL_RUNS_V2`. diff --git a/rl/track_a/agent_loop.py b/rl/track_a/agent_loop.py new file mode 100644 index 00000000..8dd8d6b1 --- /dev/null +++ b/rl/track_a/agent_loop.py @@ -0,0 +1,228 @@ +"""The iterative deployment loop on the F1-F8 benchmark: attempt, measure, +revise. A faithful minimal port of the private library's agent loop — +init round of k attempts, then revision rounds seeded by the current +champion plus deterministic, number-redacted feedback (closeness bucket, +material direction from chamfer asymmetry, error class). The teacher +directs attention; it never dictates values. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import re +from pathlib import Path + +import tinker + +from rl.common.baselines import cached_rect_baseline +from rl.common.contracts import TaskRecord +from rl.common.evaluator import EvaluationResult, EvaluationStatus, PixCellEvaluator +from rl.common.output import extract_code +from rl.track_a.benchmark_eval import _fixture_tasks +from rl.track_a.reward import norm_excess +from rl.track_a.tinker_data import _message, _renderer + + +def _scrub(text: str) -> str: + return re.sub(r"[0-9]+(?:\.[0-9]+)?", "#", text) + + +def _feedback(result: EvaluationResult, task: TaskRecord) -> str: + if result.status is not EvaluationStatus.OK or result.iou is None: + reason = _scrub(str(result.error or result.status.value))[:300] + return ( + "Your previous program FAILED to produce a valid device " + f"({result.status.value}): {reason}\n" + "Fix the failure and output a complete corrected program." + ) + baseline = cached_rect_baseline(task.reference) + iou_n = norm_excess(float(result.iou), baseline.iou_rect) + if iou_n >= 0.8: + closeness = "VERY CLOSE to the target" + elif iou_n >= 0.5: + closeness = "close to the target, but clearly imperfect" + elif iou_n >= 0.1: + closeness = "partially matching the target" + else: + closeness = "far from the target beyond its overall footprint" + pred_to_target = result.metrics.get("chamfer_pred_to_target_um") + target_to_pred = result.metrics.get("chamfer_target_to_pred_um") + direction = "" + if pred_to_target is not None and target_to_pred is not None: + if pred_to_target > 1.5 * target_to_pred: + direction = ( + "Your rendering has EXTRA material the target does not have — " + "simplify or remove structure." + ) + elif target_to_pred > 1.5 * pred_to_target: + direction = ( + "Your rendering is MISSING material the target has — add or " + "extend structure." + ) + else: + direction = ( + "Material amounts roughly balance; refine shapes, positions, " + "and repeated-element counts." + ) + return ( + f"Your previous program executed. Its rendering is {closeness}. " + f"{direction}\n" + "Revise the program to better match the attached target image. " + "Output only the complete corrected Python program." + ) + + +def run_agent_loop( + *, + benchmark_root: str, + model_name: str, + renderer_name: str, + model_path: str | None, + max_tokens: int, + init_attempts: int = 8, + revisions_per_round: int = 4, + rounds: int = 3, + temperature: float = 1.0, + out_path: str | None = None, + model_effort: float | None = None, +) -> dict: + tasks = _fixture_tasks(Path(benchmark_root)) + renderer = _renderer(model_name, renderer_name, effort=model_effort) + service = tinker.ServiceClient() + client = ( + service.create_sampling_client(model_path=model_path) + if model_path + else service.create_sampling_client(base_model=model_name) + ) + params = tinker.SamplingParams( + max_tokens=max_tokens, + temperature=temperature, + stop=renderer.get_stop_sequences(), + ) + from tinker_cookbook.renderers import Message, TextPart, get_text_content + + async def sample(conversation, count): + prompt = renderer.build_generation_prompt(conversation) + response = await client.sample_async( + prompt=prompt, num_samples=count, sampling_params=params + ) + out = [] + for sequence in response.sequences: + message, _t = renderer.parse_response(list(sequence.tokens)) + out.append(get_text_content(message)) + return out + + evaluator = PixCellEvaluator(max_workers=4, evaluator_retries=3) + report: dict[str, dict] = {} + try: + for task in tasks: + user = _message(task, max_image=1440) + history: list[dict] = [] + champion_code: str | None = None + champion_result: EvaluationResult | None = None + champion_iou = -1.0 + for round_index in range(rounds + 1): + if round_index == 0: + conversation = [user] + count = init_attempts + else: + assert champion_result is not None + feedback = _feedback(champion_result, task) + conversation = [ + user, + Message( + role="assistant", + content=[ + TextPart( + type="text", + text=f"```python\n{champion_code}\n```", + ) + ], + ), + Message( + role="user", + content=[TextPart(type="text", text=feedback)], + ), + ] + count = revisions_per_round + completions = asyncio.run(sample(conversation, count)) + results = evaluator.evaluate_batch( + [(task.reference, completion) for completion in completions] + ) + round_best = -1.0 + for completion, result in zip(completions, results): + iou = ( + float(result.iou) + if result.status is EvaluationStatus.OK and result.iou + else 0.0 + ) + round_best = max(round_best, iou) + if iou > champion_iou or champion_result is None: + champion_iou = max(iou, 0.0) + champion_code = extract_code(completion) + champion_result = result + history.append( + {"round": round_index, "samples": count, "round_best": round_best, + "champion_iou": champion_iou} + ) + print( + f"[{task.sampler.opaque_id}] round {round_index}: " + f"best {round_best:.3f}, champion {champion_iou:.3f}", + flush=True, + ) + report[task.sampler.opaque_id] = { + "champion_iou": champion_iou, + "champion_code": champion_code or "", + "footprint_um": list(task.observation.footprint_um), + "rounds": history, + } + finally: + evaluator.close() + champions = [entry["champion_iou"] for entry in report.values()] + summary = { + "model_path": model_path, + "init_attempts": init_attempts, + "revisions_per_round": revisions_per_round, + "rounds": rounds, + "per_target": report, + "overall_mean_champion": sum(champions) / len(champions), + } + if out_path: + path = Path(out_path) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(summary, indent=2), encoding="utf-8") + print(json.dumps({"overall_mean_champion": summary["overall_mean_champion"]}), flush=True) + return summary + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + repo = Path(__file__).resolve().parents[2] + ap.add_argument("--benchmark-root", default=str(repo / "data" / "benchmark")) + ap.add_argument("--model-name", default="Qwen/Qwen3.6-35B-A3B") + ap.add_argument("--renderer-name", default="qwen3_5_disable_thinking") + ap.add_argument("--model-path", default="") + ap.add_argument("--model-effort", type=float, default=-1.0) + ap.add_argument("--max-tokens", type=int, default=4096) + ap.add_argument("--rounds", type=int, default=3) + ap.add_argument("--out", required=True) + args = ap.parse_args() + from rl.track_a.transport import disable_pyqwest_transport + + disable_pyqwest_transport() + run_agent_loop( + benchmark_root=args.benchmark_root, + model_name=args.model_name, + renderer_name=args.renderer_name, + model_path=args.model_path or None, + max_tokens=args.max_tokens, + rounds=args.rounds, + out_path=args.out, + model_effort=args.model_effort if args.model_effort >= 0 else None, + ) + + +if __name__ == "__main__": + main() diff --git a/rl/track_a/benchmark_eval.py b/rl/track_a/benchmark_eval.py new file mode 100644 index 00000000..5b0b2647 --- /dev/null +++ b/rl/track_a/benchmark_eval.py @@ -0,0 +1,184 @@ +"""The paper's F1-F8 benchmark protocol on a checkpoint: 8 attempts per +target at T=1.0, per-target best-of-8 and mean raw IoU — the exact +evaluation behind the paper's Table III (its trained model: mean 0.254, +best-of-8 0.466). Fixtures are the frozen data/benchmark devices; the model +sees the same raster it is scored against, per the historical protocol. +""" + +from __future__ import annotations + +import argparse +import asyncio +import hashlib +import json +from pathlib import Path + +import tinker + +from rl.common.contracts import ModelObservation, TaskRecord, SamplerMetadata, VerifierReference +from rl.common.evaluator import EvaluationStatus, PixCellEvaluator +from rl.track_a.tinker_data import _message, _renderer + + +def _fixture_tasks(benchmark_root: Path) -> list[TaskRecord]: + tasks = [] + for index in range(1, 9): + directory = benchmark_root / f"final_{index}" + payload = (directory / "device_bw.png").read_bytes() + footprint = tuple( + json.loads((directory / "footprint.json").read_text())["footprint_um"] + ) + digest = hashlib.sha256(payload).hexdigest() + observation = ModelObservation( + image_bytes=payload, footprint_um=footprint, image_sha256=digest + ) + reference = VerifierReference( + target_image_bytes=payload, + footprint_um=footprint, + target_image_sha256=digest, + ) + sampler = SamplerMetadata( + opaque_id=f"final_{index}", + level="BENCH", + representation_id=f"final_{index}", + leakage_group_id=f"final_{index}", + realization_slot=None, + split_role="benchmark", + ) + tasks.append( + TaskRecord( + observation=observation, + label="", + reference=reference, + sampler=sampler, + ) + ) + return tasks + + +def run_benchmark( + *, + benchmark_root: str, + model_name: str, + renderer_name: str, + model_path: str | None, + max_tokens: int, + attempts: int = 8, + temperature: float = 1.0, + out_path: str | None = None, + model_effort: float | None = None, +) -> dict: + tasks = _fixture_tasks(Path(benchmark_root)) + renderer = _renderer(model_name, renderer_name, effort=model_effort) + service = tinker.ServiceClient() + client = ( + service.create_sampling_client(model_path=model_path) + if model_path + else service.create_sampling_client(base_model=model_name) + ) + params = tinker.SamplingParams( + max_tokens=max_tokens, + temperature=temperature, + stop=renderer.get_stop_sequences(), + ) + + async def sample_all() -> list[list[str]]: + async def one(task: TaskRecord) -> list[str]: + prompt = renderer.build_generation_prompt( + [_message(task, max_image=1440)] + ) + response = await client.sample_async( + prompt=prompt, num_samples=attempts, sampling_params=params + ) + from tinker_cookbook.renderers import get_text_content + + out = [] + for sequence in response.sequences: + message, _t = renderer.parse_response(list(sequence.tokens)) + out.append(get_text_content(message)) + return out + + return list(await asyncio.gather(*(one(t) for t in tasks))) + + completions = asyncio.run(sample_all()) + pairs = [ + (task, completion) + for task, batch in zip(tasks, completions) + for completion in batch + ] + with PixCellEvaluator( + max_workers=4, evaluator_retries=3 + ) as evaluator: + results = evaluator.evaluate_batch( + [(task.reference, completion) for task, completion in pairs] + ) + per_target: dict[str, dict] = {} + cursor = 0 + for task, batch in zip(tasks, completions): + scores = [] + executable = 0 + for _ in batch: + result = results[cursor] + cursor += 1 + if result.status is EvaluationStatus.OK and result.iou is not None: + executable += 1 + scores.append(float(result.iou)) + else: + scores.append(0.0) + per_target[task.sampler.opaque_id] = { + "executable": executable, + "attempts": len(batch), + "mean_iou": sum(scores) / len(scores), + "best_iou": max(scores), + } + targets = list(per_target.values()) + report = { + "model_name": model_name, + "model_path": model_path, + "attempts": attempts, + "temperature": temperature, + "per_target": per_target, + "overall": { + "executable": sum(t["executable"] for t in targets), + "total": sum(t["attempts"] for t in targets), + "mean_iou": sum(t["mean_iou"] for t in targets) / len(targets), + "mean_best_of_k": sum(t["best_iou"] for t in targets) / len(targets), + }, + } + if out_path: + path = Path(out_path) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(report, indent=2), encoding="utf-8") + print(json.dumps(report["overall"], indent=2), flush=True) + return report + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + repo = Path(__file__).resolve().parents[2] + ap.add_argument("--benchmark-root", default=str(repo / "data" / "benchmark")) + ap.add_argument("--model-name", default="Qwen/Qwen3.6-35B-A3B") + ap.add_argument("--renderer-name", default="qwen3_5_disable_thinking") + ap.add_argument("--model-path", default="") + ap.add_argument("--model-effort", type=float, default=-1.0) + ap.add_argument("--max-tokens", type=int, default=4096) + ap.add_argument("--attempts", type=int, default=8) + ap.add_argument("--out", required=True) + args = ap.parse_args() + from rl.track_a.transport import disable_pyqwest_transport + + disable_pyqwest_transport() + run_benchmark( + benchmark_root=args.benchmark_root, + model_name=args.model_name, + renderer_name=args.renderer_name, + model_path=args.model_path or None, + max_tokens=args.max_tokens, + attempts=args.attempts, + out_path=args.out, + model_effort=args.model_effort if args.model_effort >= 0 else None, + ) + + +if __name__ == "__main__": + main() diff --git a/rl/track_a/checkpoint_metrics.py b/rl/track_a/checkpoint_metrics.py deleted file mode 100644 index f3179368..00000000 --- a/rl/track_a/checkpoint_metrics.py +++ /dev/null @@ -1,131 +0,0 @@ -"""Track A behavioral diagnostics for executable checkpoint evaluation.""" - -from __future__ import annotations - -import hashlib -import math -import statistics -from collections import defaultdict -from collections.abc import Mapping, Sequence -from typing import Any - -from rl.common.output import extract_code - - -def program_sha256(completion: str) -> str: - """Hash normalized extracted source rather than chat or fence formatting.""" - - source = extract_code(completion).replace("\r\n", "\n").replace("\r", "\n") - normalized = "\n".join(line.rstrip() for line in source.splitlines()).strip() - return hashlib.sha256(normalized.encode("utf-8")).hexdigest() - - -def behavioral_probe_summary( - records: Sequence[Mapping[str, Any]], - *, - expected_group_size: int = 4, -) -> dict[str, Any]: - """Summarize within-prompt diversity for a small independent G4 probe. - - This is deliberately not a quality score. It detects only the strongest - collapse signatures and leaves checkpoint ordering to the full k=1 - executable-geometry report. - """ - - if expected_group_size < 2: - raise ValueError("behavioral probes require a group size of at least 2") - if not records: - raise ValueError("behavioral probe has no records") - - grouped: dict[str, list[Mapping[str, Any]]] = defaultdict(list) - for record in records: - grouped[str(record["opaque_id"])].append(record) - - groups: list[dict[str, Any]] = [] - for opaque_id, attempts in sorted(grouped.items()): - if len(attempts) != expected_group_size: - raise ValueError( - f"prompt {opaque_id!r} has {len(attempts)} attempts; " - f"expected {expected_group_size}" - ) - program_hashes = sorted( - {str(attempt["program_sha256"]) for attempt in attempts} - ) - render_hashes = sorted( - { - str(attempt["render_sha256"]) - for attempt in attempts - if attempt.get("pure_executable") and attempt.get("render_sha256") - } - ) - rewards = [float(attempt["reward"]) for attempt in attempts] - reward_std = statistics.pstdev(rewards) - reward_range = max(rewards) - min(rewards) - groups.append( - { - "opaque_id": opaque_id, - "distinct_program_count": len(program_hashes), - "distinct_program_sha256": program_hashes, - "successful_render_attempts": sum( - bool(attempt.get("pure_executable")) for attempt in attempts - ), - "distinct_successful_render_count": len(render_hashes), - "distinct_successful_render_sha256": render_hashes, - "reward_std": reward_std, - "reward_range": reward_range, - "nonconstant_reward": reward_range > 1e-12, - } - ) - - group_count = len(groups) - nonconstant_fraction = ( - sum(group["nonconstant_reward"] for group in groups) / group_count - ) - mean_reward_std = statistics.fmean(group["reward_std"] for group in groups) - mean_programs = statistics.fmean( - group["distinct_program_count"] for group in groups - ) - mean_renders = statistics.fmean( - group["distinct_successful_render_count"] for group in groups - ) - - identical_program_every_group = all( - group["distinct_program_count"] == 1 for group in groups - ) - constant_render_and_reward_every_group = all( - group["distinct_successful_render_count"] <= 1 - and not group["nonconstant_reward"] - for group in groups - ) - collapse_detected = ( - identical_program_every_group or constant_render_and_reward_every_group - ) - - if not all( - math.isfinite(value) - for value in ( - nonconstant_fraction, - mean_reward_std, - mean_programs, - mean_renders, - ) - ): - raise ValueError("behavioral probe produced a non-finite metric") - - return { - "probe_kind": "small-balanced-unmastered-g4", - "group_size": expected_group_size, - "prompt_groups": group_count, - "nonconstant_reward_group_fraction": nonconstant_fraction, - "mean_within_group_reward_std": mean_reward_std, - "mean_distinct_programs_per_prompt": mean_programs, - "mean_distinct_successful_renders_per_prompt": mean_renders, - "collapse_detected": collapse_detected, - "collapse_signals": { - "identical_program_every_group": identical_program_every_group, - "constant_render_and_reward_every_group": ( - constant_render_and_reward_every_group - ), - }, - "groups": groups, - } diff --git a/rl/track_a/checkpoint_selection.py b/rl/track_a/checkpoint_selection.py deleted file mode 100644 index 34261f56..00000000 --- a/rl/track_a/checkpoint_selection.py +++ /dev/null @@ -1,154 +0,0 @@ -"""Pair Track A checkpoint reports and apply the collapse veto.""" - -from __future__ import annotations - -from collections.abc import Mapping, Sequence -from typing import Any - -from rl.track_a.evaluation_contract import ( - CheckpointSelectionContract, - validate_evaluation_report, -) - - -def _index_by_sampler( - reports: Sequence[tuple[str, Mapping[str, Any]]], - *, - expected_role: str, - contract: CheckpointSelectionContract, -) -> dict[str, tuple[str, Mapping[str, Any]]]: - indexed: dict[str, tuple[str, Mapping[str, Any]]] = {} - for path, report in reports: - try: - validate_evaluation_report( - report, - contract=contract, - expected_role=expected_role, - ) - except ValueError as exc: - raise ValueError(f"{path}: {exc}") from exc - sampler_path = str(report.get("sampler_path", "")) - if sampler_path in indexed: - raise ValueError( - f"duplicate {expected_role} report for sampler {sampler_path!r}" - ) - indexed[sampler_path] = (path, report) - return indexed - - -def select_sft_checkpoint( - geometry_reports: Sequence[tuple[str, Mapping[str, Any]]], - behavioral_reports: Sequence[tuple[str, Mapping[str, Any]]], - *, - contract: CheckpointSelectionContract, -) -> dict[str, Any]: - """Select by full k=1 geometry after applying a paired G4 collapse veto.""" - - geometry = _index_by_sampler( - geometry_reports, - expected_role="checkpoint-geometry", - contract=contract, - ) - behavior = _index_by_sampler( - behavioral_reports, - expected_role="behavioral-collapse-probe", - contract=contract, - ) - missing = sorted(set(geometry) - set(behavior)) - extra = sorted(set(behavior) - set(geometry)) - if missing or extra: - raise ValueError( - "geometry/behavior sampler pairing mismatch: " - f"missing_behavior={missing}, extra_behavior={extra}" - ) - - sandbox_identities = { - ( - str(report["provenance"]["candidate_sandbox"]["image_ref"]), - str(report["provenance"]["candidate_sandbox"]["image_id"]), - ) - for _path, report in [*geometry.values(), *behavior.values()] - } - if len(sandbox_identities) != 1: - raise ValueError( - "checkpoint reports were measured with different candidate " - "sandbox images" - ) - sandbox_image_ref, sandbox_image_id = next(iter(sandbox_identities)) - - candidates: list[dict[str, Any]] = [] - for sampler_path, (geometry_path, geometry_report) in geometry.items(): - behavior_path, behavior_report = behavior[sampler_path] - probe = behavior_report.get("behavioral_probe") - collapsed_value = probe.get("collapse_detected") - collapsed = collapsed_value - candidates.append( - { - "sampler_path": sampler_path, - "geometry_report": geometry_path, - "behavioral_report": behavior_path, - "eligible": not collapsed, - "veto_reason": ( - "G4 behavioral probe detected sampling collapse" - if collapsed - else None - ), - "representation_macro_mean_iou": float( - geometry_report["representation_macro_mean_iou"] - ), - "pure_executable_rate": float( - geometry_report["summary"]["pure_executable_rate"] - ), - "behavioral_probe": { - "nonconstant_reward_group_fraction": probe[ - "nonconstant_reward_group_fraction" - ], - "mean_within_group_reward_std": probe[ - "mean_within_group_reward_std" - ], - "mean_distinct_programs_per_prompt": probe[ - "mean_distinct_programs_per_prompt" - ], - "mean_distinct_successful_renders_per_prompt": probe[ - "mean_distinct_successful_renders_per_prompt" - ], - "collapse_detected": collapsed, - "collapse_signals": probe.get("collapse_signals", {}), - }, - } - ) - - candidates.sort( - key=lambda item: ( - item["eligible"], - item["representation_macro_mean_iou"], - item["pure_executable_rate"], - ), - reverse=True, - ) - eligible = [candidate for candidate in candidates if candidate["eligible"]] - if not eligible: - raise ValueError("every checkpoint was vetoed by its paired G4 probe") - - return { - "selection_rule": ( - "veto only explicit G4 sampling collapse; among eligible checkpoints, " - "maximize full-validation k=1 representation-macro raw IoU; " - "pure-executable rate breaks ties" - ), - "behavioral_metrics_are_ranking_features": False, - "selection_provenance": { - "source_git_sha": contract.source_git_sha, - "depth_logical_release_sha256": ( - contract.depth_logical_release_sha256 - ), - "model": contract.model, - "renderer": contract.renderer, - "candidate_sandbox": { - "image_ref": sandbox_image_ref, - "image_id": sandbox_image_id, - }, - }, - "selected": eligible[0], - "ranking": candidates, - } diff --git a/rl/track_a/config.json b/rl/track_a/config.json deleted file mode 100644 index 31b6f814..00000000 --- a/rl/track_a/config.json +++ /dev/null @@ -1,102 +0,0 @@ -{ - "schema_version": "pixcell-track-a-config-v1", - "contract_version": "pixcell-direct-reconstruction-v2", - "model": { - "name": "Qwen/Qwen3.6-35B-A3B", - "renderer": "qwen3_5_disable_thinking", - "thinking": false, - "lora_rank": 32, - "max_image_long_edge": 1440 - }, - "sft": { - "dataset_repo_id": "qpaig-mit/pixcell", - "dataset_configuration": "depth", - "verifier_configuration": "references", - "dataset_revision": "v2.0.0", - "train_split": "train", - "validation_split": "validation", - "effective_passes": 1, - "balance_order": [ - "level", - "representation_id", - "realization" - ], - "visit_policy": "every_train_row_exactly_once", - "loss_mass": "equal_by_level_then_representation", - "batch_size": 64, - "max_sequence_tokens": 6144, - "learning_rate": 5e-05, - "learning_rate_schedule": "linear_decay", - "tinker_lr_schedule": "linear", - "optimizer": { - "name": "adam", - "beta1": 0.9, - "beta2": 0.95, - "epsilon": 1e-08 - }, - "checkpoint_fractions": [ - 0.25, - 0.5, - 0.75, - 1.0 - ], - "save_every_steps": 14, - "evaluate_every_steps": 14, - "max_steps": 55 - }, - "rl": { - "algorithm": "group-relative-policy-optimization", - "group_size": 4, - "groups_per_batch": 8, - "learning_rate": 1e-05, - "temperature": 1.0, - "top_p": 1.0, - "max_output_tokens": 4096, - "kl_coefficient": 0.0, - "loss_fn": "importance_sampling", - "initial_level": "select_first_unmastered_after_sft_executable_eval", - "earlier_level_replay_fraction": 0.2, - "curriculum": { - "levels": [ - "L0", - "L1", - "L2", - "L3", - "L4" - ], - "minimum_iou": 0.8, - "minimum_pure_executable_rate": 0.95, - "replay_policy": "all_earlier_levels_in_order" - }, - "stage_max_steps": 30, - "save_every_steps": 5, - "evaluate_every_steps": 5, - "reward": { - "policy_failure": 0.0, - "valid_program": "0.05 + 0.95 * raw_absolute_scale_iou", - "evaluator_failure": "retry_once_then_abort_stage", - "ports_counts_structure": "diagnostics_only" - } - }, - "checkpoint_evaluation": { - "configuration": "depth", - "split": "validation", - "full_rows": 1092, - "full_representations": 546, - "geometry_k": 1, - "behavioral_rows": 40, - "behavioral_k": 4, - "task_selection_seed": 7, - "sampling_concurrency": 16, - "evaluator_workers": 8, - "include_records": false - }, - "launch": { - "paid_launch_enabled": false, - "confirmation_tokens": { - "sft": "TRACK_A_SFT", - "rl": "TRACK_A_RL", - "evaluation": "TRACK_A_EVAL" - } - } -} diff --git a/rl/track_a/curriculum.py b/rl/track_a/curriculum.py index 54de43b5..da810ef0 100644 --- a/rl/track_a/curriculum.py +++ b/rl/track_a/curriculum.py @@ -10,6 +10,37 @@ from rl.common.contracts import TaskRecord +LEVELS = ("L0", "L1", "L2", "L3", "L4") + + +def normalize_level(value: str) -> str: + level = value.strip().upper() + if level not in LEVELS: + raise ValueError( + f"expected exactly one curriculum level in {LEVELS}; got {value!r}" + ) + return level + + +def normalize_replay_levels(value: str) -> tuple[str, ...]: + if not value.strip(): + return () + values = tuple(item.strip().upper() for item in value.split(",") if item.strip()) + if len(values) != len(set(values)): + raise ValueError("replay_levels must not contain duplicates") + unknown = [item for item in values if item not in LEVELS] + if unknown: + raise ValueError(f"unknown replay levels: {unknown}") + if values != tuple(sorted(values, key=LEVELS.index)): + raise ValueError("replay_levels must be in curriculum order") + return values + + +def expected_replay_levels(current_level: str) -> tuple[str, ...]: + current = normalize_level(current_level) + return LEVELS[: LEVELS.index(current)] + + def balanced_effective_pass( tasks: list[TaskRecord], *, @@ -112,8 +143,14 @@ def deterministic_rl_batch( groups_per_batch: int, schedule_seed: int, shuffle: bool = True, + replay_groups_per_step: int = 0, ) -> list[tuple[str, TaskRecord]]: - """Select one exact current/replay batch shared by launch and preflight.""" + """Select one exact current/replay batch shared by launch and preflight. + + ``replay_groups_per_step`` = 0 keeps the historical 80/20 five-step cycle; + a positive value fixes that many replay groups every step (the mixed-arm + knob: 4 of 8 = a 50% blend over all earlier levels). + """ current = list(current_tasks) replay = list(replay_tasks) @@ -121,14 +158,19 @@ def deterministic_rl_batch( raise ValueError("RL current-task pool is empty") if step < 0 or groups_per_batch < 1: raise ValueError("RL step and groups_per_batch must be nonnegative/positive") + if replay_groups_per_step < 0 or replay_groups_per_step >= groups_per_batch: + raise ValueError("replay_groups_per_step must be in [0, groups_per_batch)") count = min(groups_per_batch, len(current)) if shuffle and replay: def replay_groups(index: int) -> int: - proposed = ( - (2, 2, 2, 1, 1)[index % 5] - if groups_per_batch == 8 - else max(1, round(0.2 * groups_per_batch)) - ) + if replay_groups_per_step: + proposed = replay_groups_per_step + else: + proposed = ( + (2, 2, 2, 1, 1)[index % 5] + if groups_per_batch == 8 + else max(1, round(0.2 * groups_per_batch)) + ) return min(proposed, count, len(replay)) replay_count = replay_groups(step) diff --git a/rl/track_a/evaluate_checkpoint.py b/rl/track_a/evaluate_checkpoint.py deleted file mode 100644 index 166daee4..00000000 --- a/rl/track_a/evaluate_checkpoint.py +++ /dev/null @@ -1,334 +0,0 @@ -#!/usr/bin/env python3 -"""Spend-locked Track A evaluation for a Tinker sampler checkpoint.""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import os -from collections import Counter, defaultdict -from pathlib import Path - -import tinker - -from rl.common.dataset_io import load_tasks -from rl.common.evaluator import Attribution, EvaluationStatus, PixCellEvaluator -from rl.common.isolation import IsolationError -from rl.track_a.checkpoint_metrics import ( - behavioral_probe_summary, - program_sha256, -) -from rl.track_a.curriculum import balanced_effective_pass -from rl.track_a.evaluation_contract import ( - BEHAVIORAL_PROBE_TASKS, - FULL_GEOMETRY_REPRESENTATIONS, - FULL_GEOMETRY_TASKS, - REPORT_SCHEMA_VERSION, - build_checkpoint_selection_contract, - report_provenance, - validate_evaluation_report, -) -from rl.track_a.launch import bind_execution_boundary, run_launch_preflight -from rl.track_a.recipe import validate_evaluation_cli -from rl.track_a.reward import reward_for -from rl.track_a.tinker_data import DEFAULT_MODEL, DEFAULT_RENDERER, _message, _renderer - - -REPO_ROOT = Path(__file__).resolve().parents[2] - - -def _filter_levels(tasks, levels: str): - allowed = {item.strip().upper() for item in levels.split(",") if item.strip()} - return [ - task for task in tasks if not allowed or task.sampler.level.upper() in allowed - ] - - -async def run(args: argparse.Namespace) -> dict: - try: - resolved_launch = validate_evaluation_cli(REPO_ROOT, args) - resolved_launch = bind_execution_boundary(resolved_launch) - except (ValueError, IsolationError) as exc: - raise SystemExit(f"paid evaluation blocked: {exc}") from exc - if not os.environ.get("TINKER_API_KEY"): - raise SystemExit("TINKER_API_KEY is not present") - run_launch_preflight(REPO_ROOT, resolved_launch=resolved_launch) - dataset_root = Path(str(resolved_launch["dataset_root"])) - sampler_path = str(resolved_launch["sampler_path"]) - output_path = Path(str(resolved_launch["output"])) - levels = str(resolved_launch["levels"]) - rows = int(resolved_launch["rows"]) - k = int(resolved_launch["k"]) - temperature = float(resolved_launch["temperature"]) - max_tokens = int(resolved_launch["max_tokens"]) - max_image = int(resolved_launch["max_image"]) - top_p = float(resolved_launch["top_p"]) - seed = int(resolved_launch["seed"]) - concurrency = int(resolved_launch["concurrency"]) - evaluator_workers = int(resolved_launch["evaluator_workers"]) - include_records = bool(resolved_launch["include_records"]) - behavioral_probe = ( - resolved_launch["role"] == "behavioral-collapse-probe" - ) - contract = build_checkpoint_selection_contract( - REPO_ROOT, - dataset_root, - ) - - tasks = _filter_levels( - load_tasks( - dataset_root, - configuration="depth", - split="validation", - ), - levels, - ) - if not tasks: - raise SystemExit("no validation tasks matched --levels") - if rows: - tasks = balanced_effective_pass( - tasks, - seed=seed, - draws=rows, - ) - task_ids = [task.sampler.opaque_id for task in tasks] - if len(set(task_ids)) != len(task_ids): - raise RuntimeError("checkpoint evaluation selected duplicate tasks") - expected_tasks = ( - BEHAVIORAL_PROBE_TASKS - if behavioral_probe - else FULL_GEOMETRY_TASKS - ) - if len(tasks) != expected_tasks: - raise RuntimeError( - f"checkpoint evaluation selected {len(tasks)} tasks; " - f"expected {expected_tasks}" - ) - representation_count = len( - {task.sampler.representation_id for task in tasks} - ) - if ( - not behavioral_probe - and representation_count != FULL_GEOMETRY_REPRESENTATIONS - ): - raise RuntimeError( - "checkpoint geometry evaluation selected " - f"{representation_count} representations; expected " - f"{FULL_GEOMETRY_REPRESENTATIONS}" - ) - evaluator = PixCellEvaluator( - max_workers=evaluator_workers, - require_isolation=True, - ) - renderer = _renderer( - str(resolved_launch["model_name"]), - str(resolved_launch["renderer_name"]), - ) - decode = getattr(renderer, "decode_action_text", None) or renderer.tokenizer.decode - sampling = tinker.ServiceClient().create_sampling_client( - model_path=sampler_path - ) - semaphore = asyncio.Semaphore(concurrency) - - async def sample(task): - prompt = renderer.build_generation_prompt( - [_message(task, max_image=max_image)] - ) - stop = renderer.get_stop_sequences() - async with semaphore: - result = await sampling.sample_async( - prompt=prompt, - num_samples=k, - sampling_params=tinker.SamplingParams( - stop=stop, - max_tokens=max_tokens, - temperature=temperature, - top_p=top_p, - ), - ) - return [ - { - "completion": decode(sequence.tokens), - "completion_tokens": len(sequence.tokens), - } - for sequence in result.sequences - ] - - sampled = await asyncio.gather(*(sample(task) for task in tasks)) - requests = [] - owners = [] - token_counts = [] - program_hashes = [] - for task, completions in zip(tasks, sampled, strict=True): - for completion in completions: - requests.append((task.reference, completion["completion"])) - owners.append(task) - token_counts.append(completion["completion_tokens"]) - program_hashes.append(program_sha256(completion["completion"])) - - with evaluator: - results = evaluator.evaluate_batch(requests) - masked = [ - result for result in results if result.attribution is not Attribution.MODEL - ] - if masked: - first = masked[0] - raise RuntimeError( - f"evaluation aborted on {first.attribution.value} failure: " - f"{first.status.value}: {first.error}" - ) - - records = [] - for task, result, completion_tokens, program_hash in zip( - owners, - results, - token_counts, - program_hashes, - strict=True, - ): - records.append( - { - "opaque_id": task.sampler.opaque_id, - "level": task.sampler.level, - "representation_id": task.sampler.representation_id, - "status": result.status.value, - "pure_executable": result.status is EvaluationStatus.OK, - "iou": float(result.iou or 0.0), - "reward": float(reward_for(result) or 0.0), - "completion_tokens": completion_tokens, - "program_sha256": program_hash, - "render_sha256": result.metrics.get("render_sha256"), - } - ) - - by_representation = defaultdict(list) - by_level = defaultdict(list) - for record in records: - by_representation[record["representation_id"]].append(record) - by_level[record["level"]].append(record) - - def aggregate(values): - per_task = defaultdict(list) - for value in values: - per_task[value["opaque_id"]].append(value) - task_means = [ - sum(item["iou"] for item in attempts) / len(attempts) - for attempts in per_task.values() - ] - task_bests = [ - max(item["iou"] for item in attempts) for attempts in per_task.values() - ] - return { - "attempts": len(values), - "tasks": len(per_task), - "pure_executable_rate": ( - sum(item["pure_executable"] for item in values) / len(values) - ), - "mean_iou_at_1": sum(task_means) / len(task_means), - "mean_best_at_k_iou": sum(task_bests) / len(task_bests), - } - - representation_means = [ - sum(record["iou"] for record in values) / len(values) - for values in by_representation.values() - ] - role = ( - "behavioral-collapse-probe" - if behavioral_probe - else "checkpoint-geometry" - ) - actual_levels = sorted( - {task.sampler.level.upper() for task in tasks} - ) - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "evaluation_role": role, - "sampler_path": sampler_path, - "model": resolved_launch["model_name"], - "renderer": resolved_launch["renderer_name"], - "split": "depth/validation", - "levels": actual_levels[0] if behavioral_probe else "all", - "k": k, - "temperature": temperature, - "top_p": top_p, - "max_tokens": max_tokens, - "summary": aggregate(records), - "representation_macro_mean_iou": ( - sum(representation_means) / len(representation_means) - ), - "statuses": dict(sorted(Counter(item["status"] for item in records).items())), - "levels_summary": { - level: aggregate(values) for level, values in sorted(by_level.items()) - }, - "mean_completion_tokens": sum(token_counts) / len(token_counts), - "behavioral_probe": ( - behavioral_probe_summary(records, expected_group_size=4) - if behavioral_probe - else None - ), - "records": records if include_records else None, - } - boundary = evaluator.execution_boundary - assert boundary is not None - report["provenance"] = report_provenance( - contract=contract, - sampler_path=sampler_path, - role=role, - levels=actual_levels, - task_ids=task_ids, - representation_count=representation_count, - requested_rows=rows if behavioral_probe else None, - k=k, - attempt_count=len(records), - sandbox_image_ref=boundary.image_ref, - sandbox_image_id=boundary.image_id, - ) - validate_evaluation_report( - report, - contract=contract, - expected_role=role, - ) - output_path.parent.mkdir(parents=True, exist_ok=True) - with output_path.open("x", encoding="utf-8") as handle: - handle.write(json.dumps(report, indent=2, sort_keys=True) + "\n") - return report - - -def parser() -> argparse.ArgumentParser: - result = argparse.ArgumentParser(description=__doc__) - result.add_argument("--confirm-spend", default="") - result.add_argument("--sampler-path", required=True) - result.add_argument("--dataset-root", type=Path, default=REPO_ROOT / "dataset") - result.add_argument("--model-name", default=DEFAULT_MODEL) - result.add_argument("--renderer-name", default=DEFAULT_RENDERER) - result.add_argument("--levels", default="") - result.add_argument( - "--rows", type=int, default=0, help="0 evaluates the full split" - ) - result.add_argument("--k", type=int, default=1) - result.add_argument("--temperature", type=float, default=1.0) - result.add_argument("--max-tokens", type=int, default=4096) - result.add_argument("--concurrency", type=int, default=16) - result.add_argument("--evaluator-workers", type=int, default=8) - result.add_argument("--seed", type=int, default=7) - result.add_argument("--include-records", action="store_true") - result.add_argument( - "--behavioral-probe", - action="store_true", - help=( - "run the separate bounded G4 collapse probe; requires --k 4, " - "--rows, and explicit --levels" - ), - ) - result.add_argument("--output", type=Path, required=True) - return result - - -if __name__ == "__main__": - report = asyncio.run(run(parser().parse_args())) - print( - json.dumps( - {key: value for key, value in report.items() if key != "records"}, indent=2 - ) - ) diff --git a/rl/track_a/evaluation_contract.py b/rl/track_a/evaluation_contract.py deleted file mode 100644 index 7e0ff4fc..00000000 --- a/rl/track_a/evaluation_contract.py +++ /dev/null @@ -1,725 +0,0 @@ -"""Provenance and validation contract for Track A checkpoint reports.""" - -from __future__ import annotations - -import hashlib -import json -import math -import re -import statistics -import subprocess -from collections import Counter -from collections.abc import Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import Any - -from rl.common.dataset_io import load_tasks -from rl.common.isolation import IMMUTABLE_IMAGE -from rl.track_a.curriculum import balanced_effective_pass - - -REPORT_SCHEMA_VERSION = "pixcell-track-a-executable-eval-v2" -DATASET_CONFIGURATION = "depth" -DATASET_SPLIT = "validation" -REPORT_SPLIT = f"{DATASET_CONFIGURATION}/{DATASET_SPLIT}" -CURRICULUM_LEVELS = ("L0", "L1", "L2", "L3", "L4") -FULL_GEOMETRY_TASKS = 1092 -FULL_GEOMETRY_REPRESENTATIONS = 546 -BEHAVIORAL_PROBE_TASKS = 40 -BEHAVIORAL_PROBE_K = 4 -BEHAVIORAL_PROBE_ATTEMPTS = BEHAVIORAL_PROBE_TASKS * BEHAVIORAL_PROBE_K -TASK_SELECTION_SEED = 7 - -_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") -_GIT_SHA_RE = re.compile(r"^[0-9a-f]{40}(?:[0-9a-f]{24})?$") - - -def task_ids_sha256(opaque_ids: Sequence[str]) -> str: - """Hash an exact task multiset independently of evaluation order.""" - - payload = json.dumps( - sorted(str(value) for value in opaque_ids), - ensure_ascii=True, - separators=(",", ":"), - ).encode("utf-8") - return hashlib.sha256(payload).hexdigest() - - -def _source_git_sha(repo_root: Path) -> str: - value = subprocess.check_output( - ["git", "rev-parse", "HEAD"], - cwd=repo_root, - text=True, - ).strip() - if not _GIT_SHA_RE.fullmatch(value): - raise RuntimeError(f"invalid source Git SHA: {value!r}") - status = subprocess.check_output( - ["git", "status", "--porcelain", "--untracked-files=all"], - cwd=repo_root, - text=True, - ).strip() - if status: - raise RuntimeError( - "checkpoint report provenance requires a clean committed worktree" - ) - return value - - -def _depth_release_sha256(dataset_root: Path) -> str: - freeze_path = dataset_root / "depth-v1" / "factory" / "freeze.json" - freeze = json.loads(freeze_path.read_text(encoding="utf-8")) - value = str(freeze.get("logical_release_sha256", "")) - if not _SHA256_RE.fullmatch(value): - raise RuntimeError( - f"{freeze_path} has no valid logical_release_sha256" - ) - return value - - -@dataclass(frozen=True) -class CheckpointSelectionContract: - """Immutable facts every report used for checkpoint selection must match.""" - - source_git_sha: str - depth_logical_release_sha256: str - model: str - renderer: str - temperature: float - top_p: float - max_tokens: int - task_selection_seed: int - geometry_task_ids_sha256: str - behavioral_task_ids_sha256_by_level: Mapping[str, str] - - -def build_checkpoint_selection_contract( - repo_root: Path, - dataset_root: Path, -) -> CheckpointSelectionContract: - """Build the expected report contract from the committed recipe and data.""" - - repo_root = repo_root.resolve() - dataset_root = dataset_root.resolve() - expected_dataset_root = (repo_root / "dataset").resolve() - if dataset_root != expected_dataset_root: - raise RuntimeError( - "checkpoint report dataset_root must be the committed dataset at " - f"{expected_dataset_root}; got {dataset_root}" - ) - config = json.loads( - (repo_root / "rl" / "track_a" / "config.json").read_text( - encoding="utf-8" - ) - ) - validation = load_tasks( - dataset_root, - configuration=DATASET_CONFIGURATION, - split=DATASET_SPLIT, - ) - if len(validation) != FULL_GEOMETRY_TASKS: - raise RuntimeError( - "checkpoint selection requires the complete frozen validation " - f"split ({FULL_GEOMETRY_TASKS} tasks); observed {len(validation)}" - ) - representation_count = len( - {task.sampler.representation_id for task in validation} - ) - if representation_count != FULL_GEOMETRY_REPRESENTATIONS: - raise RuntimeError( - "checkpoint selection requires the complete frozen validation " - f"representation set ({FULL_GEOMETRY_REPRESENTATIONS}); " - f"observed {representation_count}" - ) - - observed_levels = { - task.sampler.level.upper() for task in validation - } - if observed_levels != set(CURRICULUM_LEVELS): - raise RuntimeError( - "validation curriculum levels differ from the report contract: " - f"{sorted(observed_levels)}" - ) - - behavioral_digests: dict[str, str] = {} - for level in CURRICULUM_LEVELS: - level_tasks = [ - task - for task in validation - if task.sampler.level.upper() == level - ] - selected = balanced_effective_pass( - level_tasks, - seed=TASK_SELECTION_SEED, - draws=BEHAVIORAL_PROBE_TASKS, - ) - selected_ids = [task.sampler.opaque_id for task in selected] - if ( - len(selected_ids) != BEHAVIORAL_PROBE_TASKS - or len(set(selected_ids)) != BEHAVIORAL_PROBE_TASKS - ): - raise RuntimeError( - f"{level} behavioral task selection is not exactly " - f"{BEHAVIORAL_PROBE_TASKS} unique tasks" - ) - behavioral_digests[level] = task_ids_sha256(selected_ids) - - model = config["model"] - rl = config["rl"] - return CheckpointSelectionContract( - source_git_sha=_source_git_sha(repo_root), - depth_logical_release_sha256=_depth_release_sha256(dataset_root), - model=str(model["name"]), - renderer=str(model["renderer"]), - temperature=float(rl["temperature"]), - top_p=float(rl["top_p"]), - max_tokens=int(rl["max_output_tokens"]), - task_selection_seed=TASK_SELECTION_SEED, - geometry_task_ids_sha256=task_ids_sha256( - [task.sampler.opaque_id for task in validation] - ), - behavioral_task_ids_sha256_by_level=behavioral_digests, - ) - - -def report_provenance( - *, - contract: CheckpointSelectionContract, - sampler_path: str, - role: str, - levels: Sequence[str], - task_ids: Sequence[str], - representation_count: int, - requested_rows: int | None, - k: int, - attempt_count: int, - sandbox_image_ref: str, - sandbox_image_id: str, -) -> dict[str, Any]: - """Create the complete provenance record written beside eval metrics.""" - - return { - "source_git_sha": contract.source_git_sha, - "depth_logical_release_sha256": ( - contract.depth_logical_release_sha256 - ), - "dataset_configuration": DATASET_CONFIGURATION, - "split": REPORT_SPLIT, - "model": contract.model, - "renderer": contract.renderer, - "sampler_path": sampler_path, - "evaluation_role": role, - "levels": sorted(str(level).upper() for level in levels), - "requested_rows": requested_rows, - "task_count": len(task_ids), - "attempt_count": attempt_count, - "representation_count": representation_count, - "task_ids_sha256": task_ids_sha256(task_ids), - "task_selection_seed": contract.task_selection_seed, - "candidate_sandbox": { - "contract": "per-candidate-docker-v1", - "image_ref": sandbox_image_ref, - "image_id": sandbox_image_id, - }, - "sampling": { - "k": k, - "temperature": contract.temperature, - "top_p": contract.top_p, - "max_tokens": contract.max_tokens, - }, - } - - -def _mapping(value: Any, path: str) -> Mapping[str, Any]: - if not isinstance(value, Mapping): - raise ValueError(f"{path} must be an object") - return value - - -def _integer(value: Any, path: str, *, minimum: int = 0) -> int: - if isinstance(value, bool) or not isinstance(value, int): - raise ValueError(f"{path} must be an integer") - if value < minimum: - raise ValueError(f"{path} must be >= {minimum}") - return value - - -def _finite(value: Any, path: str) -> float: - if isinstance(value, bool): - raise ValueError(f"{path} must be numeric") - try: - result = float(value) - except (TypeError, ValueError) as exc: - raise ValueError(f"{path} must be numeric") from exc - if not math.isfinite(result): - raise ValueError(f"{path} must be finite") - return result - - -def _unit_metric(value: Any, path: str) -> float: - result = _finite(value, path) - if result < 0.0 or result > 1.0: - raise ValueError(f"{path} must be in [0, 1]") - return result - - -def _bounded_metric(value: Any, path: str, *, low: float, high: float) -> float: - result = _finite(value, path) - if result < low or result > high: - raise ValueError(f"{path} must be in [{low}, {high}]") - return result - - -def _exact(value: Any, expected: Any, path: str) -> None: - if value != expected: - raise ValueError(f"{path}={value!r}; expected {expected!r}") - - -def _close(value: float, expected: float, path: str) -> None: - if not math.isclose(value, expected, rel_tol=0.0, abs_tol=1e-12): - raise ValueError(f"{path}={value!r}; expected {expected!r}") - - -def _validate_summary( - summary: Any, - *, - path: str, - tasks: int, - attempts: int, -) -> None: - values = _mapping(summary, path) - _exact(_integer(values.get("tasks"), f"{path}.tasks"), tasks, f"{path}.tasks") - _exact( - _integer(values.get("attempts"), f"{path}.attempts"), - attempts, - f"{path}.attempts", - ) - for field in ( - "pure_executable_rate", - "mean_iou_at_1", - "mean_best_at_k_iou", - ): - _unit_metric(values.get(field), f"{path}.{field}") - - -def _validate_behavioral_probe( - value: Any, - *, - expected_task_ids_sha256: str, -) -> None: - probe = _mapping(value, "behavioral_probe") - _exact( - probe.get("probe_kind"), - "small-balanced-unmastered-g4", - "behavioral_probe.probe_kind", - ) - _exact( - _integer(probe.get("group_size"), "behavioral_probe.group_size"), - BEHAVIORAL_PROBE_K, - "behavioral_probe.group_size", - ) - _exact( - _integer(probe.get("prompt_groups"), "behavioral_probe.prompt_groups"), - BEHAVIORAL_PROBE_TASKS, - "behavioral_probe.prompt_groups", - ) - nonconstant_fraction = _unit_metric( - probe.get("nonconstant_reward_group_fraction"), - "behavioral_probe.nonconstant_reward_group_fraction", - ) - mean_reward_std = _unit_metric( - probe.get("mean_within_group_reward_std"), - "behavioral_probe.mean_within_group_reward_std", - ) - mean_programs = _bounded_metric( - probe.get("mean_distinct_programs_per_prompt"), - "behavioral_probe.mean_distinct_programs_per_prompt", - low=1.0, - high=float(BEHAVIORAL_PROBE_K), - ) - mean_renders = _bounded_metric( - probe.get("mean_distinct_successful_renders_per_prompt"), - "behavioral_probe.mean_distinct_successful_renders_per_prompt", - low=0.0, - high=float(BEHAVIORAL_PROBE_K), - ) - collapsed = probe.get("collapse_detected") - if not isinstance(collapsed, bool): - raise ValueError("behavioral_probe.collapse_detected must be boolean") - - signals = _mapping( - probe.get("collapse_signals"), - "behavioral_probe.collapse_signals", - ) - for field in ( - "identical_program_every_group", - "constant_render_and_reward_every_group", - ): - if not isinstance(signals.get(field), bool): - raise ValueError( - f"behavioral_probe.collapse_signals.{field} must be boolean" - ) - - groups = probe.get("groups") - if not isinstance(groups, list): - raise ValueError("behavioral_probe.groups must be an array") - if len(groups) != BEHAVIORAL_PROBE_TASKS: - raise ValueError( - f"behavioral_probe.groups has {len(groups)} entries; " - f"expected {BEHAVIORAL_PROBE_TASKS}" - ) - opaque_ids: list[str] = [] - group_program_counts: list[int] = [] - group_render_counts: list[int] = [] - group_reward_stds: list[float] = [] - group_nonconstant: list[bool] = [] - for index, raw_group in enumerate(groups): - path = f"behavioral_probe.groups[{index}]" - group = _mapping(raw_group, path) - opaque_id = str(group.get("opaque_id", "")) - if not opaque_id: - raise ValueError(f"{path}.opaque_id must be non-empty") - opaque_ids.append(opaque_id) - distinct_programs = _integer( - group.get("distinct_program_count"), - f"{path}.distinct_program_count", - minimum=1, - ) - if distinct_programs > BEHAVIORAL_PROBE_K: - raise ValueError( - f"{path}.distinct_program_count exceeds {BEHAVIORAL_PROBE_K}" - ) - group_program_counts.append(distinct_programs) - successful = _integer( - group.get("successful_render_attempts"), - f"{path}.successful_render_attempts", - ) - if successful > BEHAVIORAL_PROBE_K: - raise ValueError( - f"{path}.successful_render_attempts exceeds " - f"{BEHAVIORAL_PROBE_K}" - ) - distinct_renders = _integer( - group.get("distinct_successful_render_count"), - f"{path}.distinct_successful_render_count", - ) - if distinct_renders > successful: - raise ValueError( - f"{path}.distinct_successful_render_count exceeds " - "successful_render_attempts" - ) - group_render_counts.append(distinct_renders) - group_reward_stds.append( - _unit_metric(group.get("reward_std"), f"{path}.reward_std") - ) - _unit_metric(group.get("reward_range"), f"{path}.reward_range") - nonconstant = group.get("nonconstant_reward") - if not isinstance(nonconstant, bool): - raise ValueError(f"{path}.nonconstant_reward must be boolean") - group_nonconstant.append(nonconstant) - - if len(set(opaque_ids)) != BEHAVIORAL_PROBE_TASKS: - raise ValueError("behavioral_probe.groups contains duplicate tasks") - observed_digest = task_ids_sha256(opaque_ids) - if observed_digest != expected_task_ids_sha256: - raise ValueError( - "behavioral_probe.groups task set differs from provenance" - ) - - expected_nonconstant_fraction = ( - sum(group_nonconstant) / BEHAVIORAL_PROBE_TASKS - ) - expected_mean_reward_std = statistics.fmean(group_reward_stds) - expected_mean_programs = statistics.fmean(group_program_counts) - expected_mean_renders = statistics.fmean(group_render_counts) - _close( - nonconstant_fraction, - expected_nonconstant_fraction, - "behavioral_probe.nonconstant_reward_group_fraction", - ) - _close( - mean_reward_std, - expected_mean_reward_std, - "behavioral_probe.mean_within_group_reward_std", - ) - _close( - mean_programs, - expected_mean_programs, - "behavioral_probe.mean_distinct_programs_per_prompt", - ) - _close( - mean_renders, - expected_mean_renders, - "behavioral_probe.mean_distinct_successful_renders_per_prompt", - ) - identical_program_every_group = all( - count == 1 for count in group_program_counts - ) - constant_render_and_reward_every_group = all( - render_count <= 1 and not nonconstant - for render_count, nonconstant in zip( - group_render_counts, - group_nonconstant, - strict=True, - ) - ) - expected_collapsed = ( - identical_program_every_group - or constant_render_and_reward_every_group - ) - _exact( - signals.get("identical_program_every_group"), - identical_program_every_group, - "behavioral_probe.collapse_signals.identical_program_every_group", - ) - _exact( - signals.get("constant_render_and_reward_every_group"), - constant_render_and_reward_every_group, - "behavioral_probe.collapse_signals." - "constant_render_and_reward_every_group", - ) - _exact( - collapsed, - expected_collapsed, - "behavioral_probe.collapse_detected", - ) - - -def validate_evaluation_report( - report: Mapping[str, Any], - *, - contract: CheckpointSelectionContract, - expected_role: str, -) -> None: - """Fail closed unless a report exactly matches the selection contract.""" - - _exact( - report.get("schema_version"), - REPORT_SCHEMA_VERSION, - "schema_version", - ) - _exact(report.get("evaluation_role"), expected_role, "evaluation_role") - provenance = _mapping(report.get("provenance"), "provenance") - - shared_expected = { - "source_git_sha": contract.source_git_sha, - "depth_logical_release_sha256": ( - contract.depth_logical_release_sha256 - ), - "dataset_configuration": DATASET_CONFIGURATION, - "split": REPORT_SPLIT, - "model": contract.model, - "renderer": contract.renderer, - "evaluation_role": expected_role, - "task_selection_seed": contract.task_selection_seed, - } - for field, expected in shared_expected.items(): - _exact(provenance.get(field), expected, f"provenance.{field}") - - sandbox = _mapping( - provenance.get("candidate_sandbox"), - "provenance.candidate_sandbox", - ) - _exact( - sandbox.get("contract"), - "per-candidate-docker-v1", - "provenance.candidate_sandbox.contract", - ) - image_ref = str(sandbox.get("image_ref", "")) - if not IMMUTABLE_IMAGE.fullmatch(image_ref): - raise ValueError( - "provenance.candidate_sandbox.image_ref must be immutable" - ) - image_id = str(sandbox.get("image_id", "")) - if not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id): - raise ValueError( - "provenance.candidate_sandbox.image_id must be a SHA-256 image ID" - ) - - sampler_path = str(report.get("sampler_path", "")) - if not sampler_path: - raise ValueError("sampler_path must be non-empty") - _exact(provenance.get("sampler_path"), sampler_path, "provenance.sampler_path") - _exact(report.get("model"), contract.model, "model") - _exact(report.get("renderer"), contract.renderer, "renderer") - _exact(report.get("split"), REPORT_SPLIT, "split") - - sampling = _mapping(provenance.get("sampling"), "provenance.sampling") - expected_sampling = { - "temperature": contract.temperature, - "top_p": contract.top_p, - "max_tokens": contract.max_tokens, - } - for field, expected in expected_sampling.items(): - observed = sampling.get(field) - if isinstance(expected, float): - observed = _finite(observed, f"provenance.sampling.{field}") - else: - observed = _integer(observed, f"provenance.sampling.{field}") - _exact(observed, expected, f"provenance.sampling.{field}") - _exact(report.get(field), expected, field) - - levels = provenance.get("levels") - if not isinstance(levels, list) or any( - not isinstance(level, str) for level in levels - ): - raise ValueError("provenance.levels must be an array of strings") - if levels != sorted(set(levels)): - raise ValueError("provenance.levels must be sorted and unique") - - task_count = _integer(provenance.get("task_count"), "provenance.task_count") - attempt_count = _integer( - provenance.get("attempt_count"), - "provenance.attempt_count", - ) - representation_count = _integer( - provenance.get("representation_count"), - "provenance.representation_count", - ) - task_digest = str(provenance.get("task_ids_sha256", "")) - if not _SHA256_RE.fullmatch(task_digest): - raise ValueError("provenance.task_ids_sha256 must be a SHA-256 digest") - k = _integer(sampling.get("k"), "provenance.sampling.k", minimum=1) - _exact(report.get("k"), k, "k") - _exact(report.get("levels"), "all" if len(levels) > 1 else levels[0], "levels") - _exact(attempt_count, task_count * k, "provenance.attempt_count") - - if expected_role == "checkpoint-geometry": - _exact(levels, list(CURRICULUM_LEVELS), "provenance.levels") - _exact( - provenance.get("requested_rows"), - None, - "provenance.requested_rows", - ) - _exact(task_count, FULL_GEOMETRY_TASKS, "provenance.task_count") - _exact(attempt_count, FULL_GEOMETRY_TASKS, "provenance.attempt_count") - _exact( - representation_count, - FULL_GEOMETRY_REPRESENTATIONS, - "provenance.representation_count", - ) - _exact(k, 1, "provenance.sampling.k") - _exact( - task_digest, - contract.geometry_task_ids_sha256, - "provenance.task_ids_sha256", - ) - if report.get("behavioral_probe") is not None: - raise ValueError( - "checkpoint-geometry report must not contain behavioral_probe" - ) - elif expected_role == "behavioral-collapse-probe": - if len(levels) != 1 or levels[0] not in CURRICULUM_LEVELS: - raise ValueError( - "behavioral report must name exactly one curriculum level" - ) - _exact( - provenance.get("requested_rows"), - BEHAVIORAL_PROBE_TASKS, - "provenance.requested_rows", - ) - _exact(task_count, BEHAVIORAL_PROBE_TASKS, "provenance.task_count") - _exact( - attempt_count, - BEHAVIORAL_PROBE_ATTEMPTS, - "provenance.attempt_count", - ) - if representation_count < 1 or representation_count > task_count: - raise ValueError( - "provenance.representation_count is outside task bounds" - ) - _exact(k, BEHAVIORAL_PROBE_K, "provenance.sampling.k") - expected_digest = contract.behavioral_task_ids_sha256_by_level.get( - levels[0] - ) - if expected_digest is None: - raise ValueError( - f"no behavioral task contract for level {levels[0]!r}" - ) - _exact( - task_digest, - expected_digest, - "provenance.task_ids_sha256", - ) - _validate_behavioral_probe( - report.get("behavioral_probe"), - expected_task_ids_sha256=expected_digest, - ) - else: - raise ValueError(f"unsupported evaluation role: {expected_role!r}") - - _validate_summary( - report.get("summary"), - path="summary", - tasks=task_count, - attempts=attempt_count, - ) - _unit_metric( - report.get("representation_macro_mean_iou"), - "representation_macro_mean_iou", - ) - completion_tokens = _finite( - report.get("mean_completion_tokens"), - "mean_completion_tokens", - ) - if completion_tokens < 0: - raise ValueError("mean_completion_tokens must be non-negative") - - statuses = _mapping(report.get("statuses"), "statuses") - status_total = sum( - _integer(value, f"statuses.{name}") for name, value in statuses.items() - ) - _exact(status_total, attempt_count, "statuses total") - - levels_summary = _mapping(report.get("levels_summary"), "levels_summary") - _exact(sorted(levels_summary), levels, "levels_summary keys") - level_tasks = 0 - level_attempts = 0 - for level, summary in levels_summary.items(): - values = _mapping(summary, f"levels_summary.{level}") - tasks = _integer(values.get("tasks"), f"levels_summary.{level}.tasks") - attempts = _integer( - values.get("attempts"), - f"levels_summary.{level}.attempts", - ) - _validate_summary( - values, - path=f"levels_summary.{level}", - tasks=tasks, - attempts=attempts, - ) - level_tasks += tasks - level_attempts += attempts - _exact(level_tasks, task_count, "levels_summary task total") - _exact(level_attempts, attempt_count, "levels_summary attempt total") - - records = report.get("records") - if records is not None: - if not isinstance(records, list): - raise ValueError("records must be null or an array") - if len(records) != attempt_count: - raise ValueError( - f"records has {len(records)} entries; expected {attempt_count}" - ) - record_ids: list[str] = [] - counts: Counter[str] = Counter() - for index, raw_record in enumerate(records): - path = f"records[{index}]" - record = _mapping(raw_record, path) - opaque_id = str(record.get("opaque_id", "")) - if not opaque_id: - raise ValueError(f"{path}.opaque_id must be non-empty") - record_ids.append(opaque_id) - counts[opaque_id] += 1 - _unit_metric(record.get("iou"), f"{path}.iou") - _unit_metric(record.get("reward"), f"{path}.reward") - if not isinstance(record.get("pure_executable"), bool): - raise ValueError(f"{path}.pure_executable must be boolean") - _integer( - record.get("completion_tokens"), - f"{path}.completion_tokens", - ) - if set(counts.values()) != {k}: - raise ValueError(f"records must contain exactly {k} attempts per task") - if task_ids_sha256(list(counts)) != task_digest: - raise ValueError("records task set differs from provenance") diff --git a/rl/track_a/harvest.py b/rl/track_a/harvest.py new file mode 100644 index 00000000..29a5aa08 --- /dev/null +++ b/rl/track_a/harvest.py @@ -0,0 +1,232 @@ +"""Winner harvesting: sample from a trained checkpoint over TRAIN tasks, +verify with the deterministic evaluator, keep gate-passing winners. + +The entropy-preserving warm-start corpus: programs the policy itself wrote +and the verifier approved (iou_n >= threshold, pure, executable). Winners +come only from the train split — validation never enters any SFT corpus. +Old-library law: winner distillation is the healthiest prior; its paper +lineage entered RL at entropy 0.32 on exactly this class of corpus, versus +0.096 for ground-truth canon. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +from collections import defaultdict +from pathlib import Path + +import tinker + +from rl.common.baselines import cached_rect_baseline +from rl.common.contracts import TaskRecord +from rl.common.dataset_io import load_tasks +from rl.common.evaluator import EvaluationStatus, PixCellEvaluator +from rl.common.output import extract_code +from rl.track_a.reward import norm_excess +from rl.track_a.tinker_data import _message, _renderer + + +async def _sample( + tasks: list[TaskRecord], + *, + model_name: str, + renderer_name: str, + model_path: str | None, + samples_per_task: int, + max_tokens: int, + temperature: float, +) -> list[list[str]]: + renderer = _renderer(model_name, renderer_name) + service = tinker.ServiceClient() + client = ( + service.create_sampling_client(model_path=model_path) + if model_path + else service.create_sampling_client(base_model=model_name) + ) + params = tinker.SamplingParams( + max_tokens=max_tokens, + temperature=temperature, + stop=renderer.get_stop_sequences(), + ) + + async def one(task: TaskRecord) -> list[str]: + prompt = renderer.build_generation_prompt([_message(task, max_image=1440)]) + response = await client.sample_async( + prompt=prompt, num_samples=samples_per_task, sampling_params=params + ) + out = [] + for sequence in response.sequences: + message, _t = renderer.parse_response(list(sequence.tokens)) + from tinker_cookbook.renderers import get_text_content + + out.append(get_text_content(message)) + return out + + results: list[list[str]] = [] + batch = 24 + for start in range(0, len(tasks), batch): + chunk = tasks[start : start + batch] + results.extend(await asyncio.gather(*(one(t) for t in chunk))) + print(f"[harvest] sampled {min(start+batch, len(tasks))}/{len(tasks)}", flush=True) + return results + + +def harvest( + *, + dataset_root: str, + levels: tuple[str, ...], + model_name: str, + renderer_name: str, + model_path: str | None, + out_path: str, + samples_per_task: int = 2, + max_tokens: int = 4096, + temperature: float = 1.0, + min_iou_n: float = 0.5, + degenerate_min_iou: float = 0.9, + per_representation_cap: int = 8, +) -> dict: + train = [ + t + for t in load_tasks(Path(dataset_root), configuration="depth", split="train") + if t.sampler.level in levels + ] + print(f"[harvest] {len(train)} train tasks across {levels}", flush=True) + completions = asyncio.run( + _sample( + train, + model_name=model_name, + renderer_name=renderer_name, + model_path=model_path, + samples_per_task=samples_per_task, + max_tokens=max_tokens, + temperature=temperature, + ) + ) + pairs = [ + (task, completion) + for task, batch in zip(train, completions) + for completion in batch + ] + with PixCellEvaluator(max_workers=4, evaluator_retries=3) as evaluator: + results = evaluator.evaluate_batch( + [(task.reference, completion) for task, completion in pairs] + ) + best: dict[str, dict] = {} + per_rep = defaultdict(int) + status_counts: dict[str, int] = defaultdict(int) + kept = examined = 0 + for (task, completion), result in zip(pairs, results): + examined += 1 + status_counts[result.status.value] += 1 + if result.status is not EvaluationStatus.OK or result.iou is None: + continue + baseline = cached_rect_baseline(task.reference) + iou_n = norm_excess(float(result.iou), baseline.iou_rect) + degenerate = baseline.iou_rect >= 0.95 + win = (iou_n >= min_iou_n) or (degenerate and float(result.iou) >= degenerate_min_iou) + if not win: + continue + key = task.sampler.opaque_id + score = iou_n if not degenerate else float(result.iou) + record = { + "opaque_id": key, + "level": task.sampler.level, + "representation_id": task.sampler.representation_id, + "iou": float(result.iou), + "iou_n": float(iou_n), + "degenerate": degenerate, + "code": extract_code(completion), + } + if key not in best or score > best[key]["iou_n"]: + best[key] = record + winners = [] + for record in sorted(best.values(), key=lambda r: -r["iou_n"]): + if per_rep[record["representation_id"]] >= per_representation_cap: + continue + per_rep[record["representation_id"]] += 1 + winners.append(record) + kept += 1 + by_level = defaultdict(int) + for w in winners: + by_level[w["level"]] += 1 + out = Path(out_path) + out.parent.mkdir(parents=True, exist_ok=True) + with out.open("w", encoding="utf-8") as sink: + for w in winners: + sink.write(json.dumps(w) + "\n") + summary = { + "examined": examined, + "status_counts": dict(status_counts), + "unique_tasks_won": len(best), + "kept_after_caps": kept, + "by_level": dict(by_level), + "out": str(out), + } + print("[harvest]", json.dumps(summary), flush=True) + return summary + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--dataset-root", default=str(Path(__file__).resolve().parents[2] / "dataset")) + ap.add_argument("--levels", default="L0,L1") + ap.add_argument("--model-name", default="Qwen/Qwen3.6-35B-A3B") + ap.add_argument("--renderer-name", default="qwen3_5_disable_thinking") + ap.add_argument("--model-path", default="") + ap.add_argument("--out", required=True) + ap.add_argument("--samples-per-task", type=int, default=2) + ap.add_argument("--temperature", type=float, default=1.0) + ap.add_argument("--min-iou-n", type=float, default=0.5) + ap.add_argument("--max-tokens", type=int, default=4096) + args = ap.parse_args() + from rl.track_a.transport import disable_pyqwest_transport + + disable_pyqwest_transport() + harvest( + dataset_root=args.dataset_root, + levels=tuple(x.strip().upper() for x in args.levels.split(",") if x.strip()), + model_name=args.model_name, + renderer_name=args.renderer_name, + model_path=args.model_path or None, + out_path=args.out, + samples_per_task=args.samples_per_task, + temperature=args.temperature, + min_iou_n=args.min_iou_n, + max_tokens=args.max_tokens, + ) + + +if __name__ == "__main__": + main() + + +def apply_winner_labels( + tasks: list[TaskRecord], + winners_path: str | Path, +) -> list[TaskRecord]: + """Project TRAIN tasks onto a harvested winner corpus: keep only tasks + that have a verified winner and replace the GT label with the policy's + own winning program. The SFT that consumes this distills the model's own + verified distribution — the entropy-preserving warm start.""" + + import dataclasses + + labels: dict[str, str] = {} + for line in Path(winners_path).read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + record = json.loads(line) + code = record.get("code", "").strip() + if code: + labels[record["opaque_id"]] = code + projected = [ + dataclasses.replace(task, label=labels[task.sampler.opaque_id]) + for task in tasks + if task.sampler.opaque_id in labels + ] + if not projected: + raise ValueError(f"no tasks match winners in {winners_path}") + return projected diff --git a/rl/track_a/launch.py b/rl/track_a/launch.py index d37fb02f..8dd2ec9c 100644 --- a/rl/track_a/launch.py +++ b/rl/track_a/launch.py @@ -2,39 +2,38 @@ from __future__ import annotations -import json import os import subprocess import sys -from collections.abc import Mapping from pathlib import Path from rl.common.isolation import require_execution_boundary -def bind_execution_boundary( - resolved_launch: Mapping[str, object], -) -> dict[str, object]: +def bind_execution_boundary() -> dict[str, object]: """Bind paid program execution to one immutable local sandbox image.""" - binding = dict(resolved_launch) - if binding.get("kind") not in {"rl", "evaluation"}: - return binding boundary = require_execution_boundary() - binding["candidate_sandbox"] = { + return { "runtime_path": str(boundary.runtime_path), "daemon_endpoint": boundary.daemon_endpoint, "image_ref": boundary.image_ref, "image_id": boundary.image_id, "workspace_root": str(boundary.workspace_root), } - return binding def run_launch_preflight( repo_root: Path, *, - resolved_launch: Mapping[str, object], + dataset_root: Path, + model_name: str, + renderer_name: str, + require_sandbox: bool, + tokenize_sft: bool = False, + sft_levels: str = "", + max_length: int = 6144, + max_image: int = 1440, ) -> None: """Run the complete local gate before any Tinker client can be created.""" @@ -44,27 +43,28 @@ def run_launch_preflight( if existing: roots.append(existing) environment["PYTHONPATH"] = os.pathsep.join(roots) - subprocess.run( - [ - sys.executable, - "-m", - "rl.track_a.preflight", - "--execute-ground-truth", - "5", - "--tokenize-sft", - "--require-clean", - "--dataset-root", - str(resolved_launch["dataset_root"]), - "--config", - str(resolved_launch["config_path"]), - "--resolved-launch-json", - json.dumps( - dict(resolved_launch), - sort_keys=True, - separators=(",", ":"), - ), - ], - cwd=repo_root, - env=environment, - check=True, - ) + command = [ + sys.executable, + "-m", + "rl.track_a.preflight", + "--execute-ground-truth", + "5", + "--require-clean", + "--dataset-root", + str(dataset_root), + "--model-name", + model_name, + "--renderer-name", + renderer_name, + "--max-length", + str(max_length), + "--max-image", + str(max_image), + ] + if require_sandbox: + command.append("--require-sandbox") + if tokenize_sft: + command.append("--tokenize-sft") + if sft_levels: + command.extend(["--sft-levels", sft_levels]) + subprocess.run(command, cwd=repo_root, env=environment, check=True) diff --git a/rl/track_a/make_gallery.py b/rl/track_a/make_gallery.py new file mode 100644 index 00000000..9a341c93 --- /dev/null +++ b/rl/track_a/make_gallery.py @@ -0,0 +1,125 @@ +"""Compose the benchmark gallery from agent-loop champions: per target, the +frozen input raster beside the champion program's render at target +calibration, with the difference map in the paper's convention (maroon +overlap, pink candidate-only, black target-only).""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +import tempfile +from pathlib import Path + +import numpy as np +from PIL import Image + +from michaelangelo.core.compare_absolute import extract_polygons, render_gds_absolute +from rl.common.evaluator import _prepare_reference +from rl.track_a.benchmark_eval import _fixture_tasks + +OVERLAP = (128, 0, 0) +CANDIDATE_ONLY = (255, 146, 153) +TARGET_ONLY = (23, 25, 28) +BACKGROUND = (255, 255, 255) + + +def _render_champion(code: str, prepared) -> Image.Image | None: + with tempfile.TemporaryDirectory() as workdir: + script = Path(workdir) / "program.py" + script.write_text(code, encoding="utf-8") + proc = subprocess.run( + [sys.executable, str(script)], + cwd=workdir, + capture_output=True, + timeout=60, + ) + gds = Path(workdir) / "device.gds" + if proc.returncode != 0 or not gds.exists(): + return None + polygons = extract_polygons(gds) + if not polygons: + return None + raster, _info = render_gds_absolute(polygons, prepared.calibration) + return raster.convert("L") + + +def _diff(render: Image.Image, target: Image.Image) -> Image.Image: + r = np.asarray(render) < 245 + t = np.asarray(target) < 245 + out = np.full((*t.shape, 3), BACKGROUND, dtype=np.uint8) + out[r & t] = OVERLAP + out[r & ~t] = CANDIDATE_ONLY + out[~r & t] = TARGET_ONLY + return Image.fromarray(out) + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + repo = Path(__file__).resolve().parents[2] + ap.add_argument("--benchmark-root", default=str(repo / "data" / "benchmark")) + ap.add_argument("--champions", required=True, help="agent-loop summary JSON") + ap.add_argument("--out-dir", required=True) + args = ap.parse_args() + summary = json.loads(Path(args.champions).read_text(encoding="utf-8")) + out = Path(args.out_dir) + out.mkdir(parents=True, exist_ok=True) + tasks = {t.sampler.opaque_id: t for t in _fixture_tasks(Path(args.benchmark_root))} + tiles = [] + manifest = {} + for name, entry in summary["per_target"].items(): + task = tasks[name] + prepared = _prepare_reference(task.reference) + target = prepared.target + render = _render_champion(entry["champion_code"], prepared) + (out / f"{name}.program.py").write_text( + entry["champion_code"], encoding="utf-8" + ) + target.save(out / f"{name}.target.png") + if render is not None: + render.save(out / f"{name}.render.png") + diff = _diff(render, target) + diff.save(out / f"{name}.diff.png") + manifest[name] = { + "champion_iou": entry["champion_iou"], + "rendered": render is not None, + } + row = [target.convert("RGB")] + if render is not None: + row.append(render.convert("RGB")) + row.append(_diff(render, target)) + height = 220 + scaled = [] + for image in row: + width = max(1, int(image.width * height / image.height)) + scaled.append(image.resize((width, height))) + strip = Image.new( + "RGB", + (sum(i.width for i in scaled) + 8 * (len(scaled) - 1), height), + BACKGROUND, + ) + x = 0 + for image in scaled: + strip.paste(image, (x, 0)) + x += image.width + 8 + tiles.append((name, entry["champion_iou"], strip)) + width = max(t[2].width for t in tiles) + 160 + total_height = sum(t[2].height + 16 for t in tiles) + sheet = Image.new("RGB", (width, total_height), BACKGROUND) + from PIL import ImageDraw + + draw = ImageDraw.Draw(sheet) + y = 0 + for name, iou, strip in tiles: + draw.text((4, y + 100), f"{name}\nIoU {iou:.3f}", fill=(23, 25, 28)) + sheet.paste(strip, (150, y)) + y += strip.height + 16 + sheet.save(out / "gallery.png") + (out / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8") + print(json.dumps(manifest, indent=2)) + print(f"gallery written to {out}/gallery.png", flush=True) + + +if __name__ == "__main__": + main() diff --git a/rl/track_a/preflight.py b/rl/track_a/preflight.py index 8f6d579c..5c726f84 100644 --- a/rl/track_a/preflight.py +++ b/rl/track_a/preflight.py @@ -32,14 +32,9 @@ catalog_text, prompt_asset_hashes, ) +from rl.common.baselines import cached_rect_baseline from rl.common.runtime import validate_runtime_stack -from rl.track_a.reward import reward_for -from rl.track_a.recipe import ( - canonical_config_path, - canonical_dataset_root, - canonical_json_sha256, - validate_resolved_binding, -) +from rl.track_a.reward import reward_for_policy def _dataset_allowed_components(dataset_root: Path) -> frozenset[str]: @@ -180,42 +175,8 @@ def _token_audit( def run(args: argparse.Namespace) -> dict: dataset_root = args.dataset_root.resolve() repo_root = dataset_root.parent - config_path = args.config.resolve() - if dataset_root != canonical_dataset_root(repo_root): + if dataset_root != (repo_root / "dataset").resolve(): raise RuntimeError("preflight dataset_root is not repo_root/dataset") - if config_path != canonical_config_path(repo_root): - raise RuntimeError("preflight config is not the committed Track A config") - config = json.loads(config_path.read_text(encoding="utf-8")) - if args.audit_only: - if args.resolved_launch_json is not None: - raise RuntimeError("audit-only preflight does not accept a launch binding") - resolved_launch = { - "kind": "audit-only", - "dataset_root": str(dataset_root), - "config_path": str(config_path), - "model_name": config["model"]["name"], - "renderer_name": config["model"]["renderer"], - "lora_rank": config["model"]["lora_rank"], - } - else: - if args.resolved_launch_json is None: - raise RuntimeError( - "paid-launch preflight requires the exact resolved launch binding" - ) - try: - resolved_launch = json.loads(args.resolved_launch_json) - except json.JSONDecodeError as exc: - raise RuntimeError("resolved launch binding is not valid JSON") from exc - if not isinstance(resolved_launch, dict): - raise RuntimeError("resolved launch binding must be a JSON object") - try: - validate_resolved_binding(repo_root, resolved_launch) - except ValueError as exc: - raise RuntimeError(f"resolved launch binding failed: {exc}") from exc - if config["contract_version"] != CONTRACT_VERSION: - raise RuntimeError("config and prompt contract versions differ") - if config["launch"]["paid_launch_enabled"] is not False: - raise RuntimeError("committed config must remain spend-locked") release = _verify_frozen_release(dataset_root) git = _git_state(repo_root, require_clean=args.require_clean) runtime = validate_runtime_stack(repo_root) @@ -275,7 +236,6 @@ def run(args: argparse.Namespace) -> dict: forbidden = ( "DISPLAY NOTE", "magnification x:y", - "px/um", "target_image", str(sample["id"]), str(sample["representation_id"]), @@ -285,7 +245,7 @@ def run(args: argparse.Namespace) -> dict: raise RuntimeError(f"policy prompt leaked hidden row state: {leaked}") sandbox: dict | None = None - if resolved_launch["kind"] in {"rl", "evaluation"}: + if args.require_sandbox: with PixCellEvaluator( max_workers=1, evaluator_retries=0, @@ -334,7 +294,11 @@ def run(args: argparse.Namespace) -> dict: "opaque_index": len(executed), "level": row["level"], "iou": result.iou, - "reward": reward_for(result), + "reward": reward_for_policy( + result, + policy="shaped_v3b", + baseline=cached_rect_baseline(reference_from_row(row)), + ), } ) @@ -343,8 +307,8 @@ def run(args: argparse.Namespace) -> dict: "spend": False, "contract_version": CONTRACT_VERSION, "prompt_assets": prompt_asset_hashes(), - "resolved_launch_sha256": canonical_json_sha256(resolved_launch), - "resolved_launch": resolved_launch, + "model_name": args.model_name, + "renderer_name": args.renderer_name, "release": release, "git": git, "dataset": { @@ -368,11 +332,24 @@ def run(args: argparse.Namespace) -> dict: }, "token_audit": ( _token_audit( - train_tasks, - model_name=str(resolved_launch["model_name"]), - renderer_name=str(resolved_launch["renderer_name"]), - max_length=int(config["sft"]["max_sequence_tokens"]), - max_image=int(config["model"]["max_image_long_edge"]), + ( + [ + task + for task in train_tasks + if task.sampler.level.upper() + in { + item.strip().upper() + for item in args.sft_levels.split(",") + if item.strip() + } + ] + if args.sft_levels.strip() + else train_tasks + ), + model_name=args.model_name, + renderer_name=args.renderer_name, + max_length=args.max_length, + max_image=args.max_image, ) if args.tokenize_sft else None @@ -385,22 +362,21 @@ def parser() -> argparse.ArgumentParser: repo_root = Path(__file__).resolve().parents[2] result = argparse.ArgumentParser(description=__doc__) result.add_argument("--dataset-root", type=Path, default=repo_root / "dataset") - result.add_argument( - "--config", - type=Path, - default=repo_root / "rl" / "track_a" / "config.json", - ) + result.add_argument("--model-name", default="Qwen/Qwen3.6-35B-A3B") + result.add_argument("--renderer-name", default="qwen3_5_disable_thinking") + result.add_argument("--max-length", type=int, default=6144) + result.add_argument("--max-image", type=int, default=1440) + result.add_argument("--sft-levels", default="") result.add_argument("--execute-ground-truth", type=int, default=5) result.add_argument("--minimum-self-iou", type=float, default=0.97) result.add_argument("--workers", type=int, default=8) result.add_argument("--tokenize-sft", action="store_true") result.add_argument("--require-clean", action="store_true") result.add_argument( - "--audit-only", + "--require-sandbox", action="store_true", - help="run the zero-spend repository audit without authorizing a launch", + help="bind and self-test the immutable Docker candidate sandbox", ) - result.add_argument("--resolved-launch-json") return result diff --git a/rl/track_a/probe_eval.py b/rl/track_a/probe_eval.py new file mode 100644 index 00000000..1e795e75 --- /dev/null +++ b/rl/track_a/probe_eval.py @@ -0,0 +1,184 @@ +"""Fixed cross-run probe evaluation. + +One deterministic probe set — the first ``per_level`` validation tasks per +level by sorted opaque id — evaluated identically at campaign start (the +base-model baseline), after every stage, and at the end. Because the set +never changes, every point on the resulting curve is comparable; this is +the discipline the old campaign lost when its eval pool grew mid-run. +""" + +from __future__ import annotations + +import asyncio +import json +from collections import defaultdict +from pathlib import Path + +import tinker + +from rl.common.baselines import cached_rect_baseline +from rl.common.contracts import TaskRecord +from rl.common.dataset_io import load_tasks +from rl.common.evaluator import EvaluationStatus, PixCellEvaluator +from rl.track_a.curriculum import LEVELS +from rl.track_a.reward import reward_for_policy +from rl.track_a.tinker_data import _message, _renderer + + +def select_probe_tasks( + dataset_root: str | Path, + *, + per_level: int = 16, +) -> list[TaskRecord]: + """Deterministic: per level, the first ``per_level`` validation tasks by + sorted opaque id. No seed, no state.""" + + validation = load_tasks(Path(dataset_root), configuration="depth", split="validation") + by_level: dict[str, list[TaskRecord]] = defaultdict(list) + for task in validation: + by_level[task.sampler.level].append(task) + selected: list[TaskRecord] = [] + for level in LEVELS: + ranked = sorted(by_level[level], key=lambda task: task.sampler.opaque_id) + if len(ranked) < per_level: + raise ValueError(f"{level} has only {len(ranked)} validation tasks") + selected.extend(ranked[:per_level]) + return selected + + +async def _sample_completions( + tasks: list[TaskRecord], + *, + model_name: str, + renderer_name: str, + model_path: str | None, + max_tokens: int, + temperature: float, +) -> list[str]: + from rl.track_a.transport import disable_pyqwest_transport + + disable_pyqwest_transport() + renderer = _renderer(model_name, renderer_name) + service = tinker.ServiceClient() + if model_path: + client = service.create_sampling_client(model_path=model_path) + else: + client = service.create_sampling_client(base_model=model_name) + params = tinker.SamplingParams( + max_tokens=max_tokens, + temperature=temperature, + stop=renderer.get_stop_sequences(), + ) + + async def one(task: TaskRecord) -> str: + prompt = renderer.build_generation_prompt( + [_message(task, max_image=1440)] + ) + response = await client.sample_async( + prompt=prompt, + num_samples=1, + sampling_params=params, + ) + tokens = list(response.sequences[0].tokens) + message, _termination = renderer.parse_response(tokens) + from tinker_cookbook.renderers import get_text_content + + return get_text_content(message) + + return list(await asyncio.gather(*(one(task) for task in tasks))) + + +def run_probe( + *, + dataset_root: str | Path, + model_name: str, + renderer_name: str, + model_path: str | None, + max_tokens: int, + out_path: str | Path | None = None, + per_level: int = 16, + temperature: float = 1.0, + max_workers: int = 8, +) -> dict: + """Sample one completion per probe task and grade it exactly as training + does. Returns (and optionally writes) the per-level summary.""" + + tasks = select_probe_tasks(dataset_root, per_level=per_level) + completions = asyncio.run( + _sample_completions( + tasks, + model_name=model_name, + renderer_name=renderer_name, + model_path=model_path, + max_tokens=max_tokens, + temperature=temperature, + ) + ) + import os + + with PixCellEvaluator( + max_workers=int(os.environ.get("PIXCELL_EVAL_WORKERS", str(max_workers))), + evaluator_retries=3, + execution_timeout_seconds=float( + os.environ.get("PIXCELL_EVAL_TIMEOUT", "20") + ), + require_isolation=( + os.environ.get("PIXCELL_REQUIRE_ISOLATION", "0") == "1" + ), + ) as evaluator: + results = evaluator.evaluate_batch( + [(task.reference, completion) for task, completion in zip(tasks, completions)] + ) + + rows = [] + for task, result in zip(tasks, results): + baseline = cached_rect_baseline(task.reference) + shaped = reward_for_policy(result, policy="shaped_v3b", baseline=baseline) + rows.append( + { + "opaque_id": task.sampler.opaque_id, + "level": task.sampler.level, + "status": result.status.value, + "executable": result.status is EvaluationStatus.OK, + "iou": float(result.iou or 0.0), + "shaped": float(shaped) if shaped is not None else None, + } + ) + + per_level_summary: dict[str, dict[str, float]] = {} + for level in LEVELS: + level_rows = [row for row in rows if row["level"] == level] + graded = [row for row in level_rows if row["shaped"] is not None] + per_level_summary[level] = { + "n": len(level_rows), + "executable_rate": ( + sum(row["executable"] for row in level_rows) / len(level_rows) + ), + "mean_iou": sum(row["iou"] for row in level_rows) / len(level_rows), + "mean_shaped": ( + sum(row["shaped"] for row in graded) / len(graded) if graded else 0.0 + ), + } + graded = [row for row in rows if row["shaped"] is not None] + report = { + "model_name": model_name, + "renderer_name": renderer_name, + "model_path": model_path, + "max_tokens": max_tokens, + "temperature": temperature, + "per_level": per_level_summary, + "overall": { + "n": len(rows), + "executable_rate": sum(row["executable"] for row in rows) / len(rows), + "mean_iou": sum(row["iou"] for row in rows) / len(rows), + "mean_shaped": ( + sum(row["shaped"] for row in graded) / len(graded) if graded else 0.0 + ), + }, + "rows": rows, + } + if out_path is not None: + path = Path(out_path) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(report, indent=2), encoding="utf-8") + return report diff --git a/rl/track_a/recipe.py b/rl/track_a/recipe.py deleted file mode 100644 index aa33f425..00000000 --- a/rl/track_a/recipe.py +++ /dev/null @@ -1,560 +0,0 @@ -"""Canonical Track A recipe and curriculum invariants. - -Paid entrypoints intentionally accept only operational inputs (confirmation, -checkpoint/report paths, and a contained log directory). Model, data, and -optimization settings are loaded from the committed recipe and cannot be -silently changed at the command line. -""" - -from __future__ import annotations - -import hashlib -import json -import math -import re -from collections.abc import Mapping -from pathlib import Path -from typing import Any - -from rl.common.isolation import require_execution_boundary - - -LEVELS = ("L0", "L1", "L2", "L3", "L4") -CONFIG_RELATIVE_PATH = Path("rl/track_a/config.json") -DATASET_RELATIVE_PATH = Path("dataset") -RUNS_RELATIVE_PATH = Path("rl/track_a/runs") -_TINKER_SAMPLER_PATH = re.compile( - r"^(tinker://[^/\s]+/)(sampler_weights)(/[^/\s]+)$" -) - - -def load_recipe(repo_root: Path) -> dict[str, Any]: - """Load the single committed Track A recipe.""" - - path = (repo_root / CONFIG_RELATIVE_PATH).resolve() - value = json.loads(path.read_text(encoding="utf-8")) - if value.get("schema_version") != "pixcell-track-a-config-v1": - raise ValueError("unsupported Track A recipe schema") - return value - - -def canonical_dataset_root(repo_root: Path) -> Path: - return (repo_root / DATASET_RELATIVE_PATH).resolve() - - -def canonical_config_path(repo_root: Path) -> Path: - return (repo_root / CONFIG_RELATIVE_PATH).resolve() - - -def _same_number(actual: Any, expected: Any) -> bool: - if isinstance(actual, bool) or isinstance(expected, bool): - return actual is expected - if isinstance(actual, (int, float)) and isinstance(expected, (int, float)): - return math.isclose(float(actual), float(expected), rel_tol=0.0, abs_tol=0.0) - return actual == expected - - -def _require_equal(name: str, actual: Any, expected: Any) -> None: - if not _same_number(actual, expected): - raise ValueError( - f"{name}={actual!r} overrides the committed Track A value {expected!r}" - ) - - -def _require_canonical_dataset(repo_root: Path, value: str | Path) -> Path: - observed = Path(value).expanduser().resolve() - expected = canonical_dataset_root(repo_root) - if observed != expected: - raise ValueError( - f"dataset_root must be the committed dataset at {expected}; got {observed}" - ) - return observed - - -def _require_contained_log_path(repo_root: Path, value: str | Path) -> Path: - observed = Path(value).expanduser().resolve() - root = (repo_root / RUNS_RELATIVE_PATH).resolve() - if observed == root or not observed.is_relative_to(root): - raise ValueError(f"log_path must be a new child of {root}; got {observed}") - return observed - - -def _require_contained_report_path(repo_root: Path, value: str | Path) -> Path: - observed = Path(value).expanduser().resolve() - root = (repo_root / RUNS_RELATIVE_PATH).resolve() - if not observed.is_relative_to(root) or observed.suffix != ".json": - raise ValueError(f"output must be a JSON child of {root}; got {observed}") - if observed.exists(): - raise ValueError(f"output must not already exist: {observed}") - return observed - - -def file_sha256(path: Path) -> str: - """Hash one launch input exactly as preflight will consume it.""" - - digest = hashlib.sha256() - with path.open("rb") as handle: - for chunk in iter(lambda: handle.read(1024 * 1024), b""): - digest.update(chunk) - return digest.hexdigest() - - -def weights_path_for_sampler(sampler_path: str) -> str: - """Return the weights checkpoint paired with a Tinker sampler checkpoint.""" - - match = _TINKER_SAMPLER_PATH.fullmatch(sampler_path) - if match is None: - raise ValueError( - "mastery report sampler_path must be a Tinker sampler_weights checkpoint" - ) - return f"{match.group(1)}weights{match.group(3)}" - - -def validate_sft_cli(repo_root: Path, cfg: Any) -> dict[str, Any]: - """Reject every paid SFT recipe override and return its resolved binding.""" - - recipe = load_recipe(repo_root) - model = recipe["model"] - sft = recipe["sft"] - launch = recipe["launch"] - _require_canonical_dataset(repo_root, cfg.dataset_root) - _require_equal("model_name", cfg.model_name, model["name"]) - _require_equal("renderer_name", cfg.renderer_name, model["renderer"]) - _require_equal("lora_rank", cfg.lora_rank, model["lora_rank"]) - _require_equal("learning_rate", cfg.learning_rate, sft["learning_rate"]) - _require_equal("lr_schedule", cfg.lr_schedule, sft["tinker_lr_schedule"]) - _require_equal("num_epochs", cfg.num_epochs, sft["effective_passes"]) - _require_equal("batch_size", cfg.batch_size, sft["batch_size"]) - _require_equal("max_length", cfg.max_length, sft["max_sequence_tokens"]) - _require_equal("max_image", cfg.max_image, model["max_image_long_edge"]) - _require_equal("save_every", cfg.save_every, sft["save_every_steps"]) - _require_equal("eval_every", cfg.eval_every, sft["evaluate_every_steps"]) - _require_equal("max_steps", cfg.max_steps, sft["max_steps"]) - _require_contained_log_path(repo_root, cfg.log_path) - _require_equal( - "confirm_spend", - cfg.confirm_spend, - launch["confirmation_tokens"]["sft"], - ) - return { - "kind": "sft", - "dataset_root": str(canonical_dataset_root(repo_root)), - "config_path": str(canonical_config_path(repo_root)), - "model_name": cfg.model_name, - "renderer_name": cfg.renderer_name, - "lora_rank": cfg.lora_rank, - "learning_rate": cfg.learning_rate, - "lr_schedule": cfg.lr_schedule, - "num_epochs": cfg.num_epochs, - "batch_size": cfg.batch_size, - "max_length": cfg.max_length, - "max_image": cfg.max_image, - "save_every": cfg.save_every, - "eval_every": cfg.eval_every, - "max_steps": cfg.max_steps, - } - - -def normalize_level(value: str) -> str: - level = value.strip().upper() - if level not in LEVELS: - raise ValueError(f"expected exactly one curriculum level in {LEVELS}; got {value!r}") - return level - - -def normalize_replay_levels(value: str) -> tuple[str, ...]: - if not value.strip(): - return () - values = tuple(item.strip().upper() for item in value.split(",") if item.strip()) - if len(values) != len(set(values)): - raise ValueError("replay_levels must not contain duplicates") - unknown = [item for item in values if item not in LEVELS] - if unknown: - raise ValueError(f"unknown replay levels: {unknown}") - if values != tuple(sorted(values, key=LEVELS.index)): - raise ValueError("replay_levels must be in curriculum order") - return values - - -def expected_replay_levels(current_level: str) -> tuple[str, ...]: - current = normalize_level(current_level) - return LEVELS[: LEVELS.index(current)] - - -def first_unmastered_level( - report: Mapping[str, Any], - *, - minimum_iou: float, - minimum_pure_executable_rate: float, -) -> str | None: - """Compute the first curriculum level that misses either mastery threshold.""" - - levels = report.get("levels_summary") - if not isinstance(levels, Mapping): - raise ValueError("mastery report has no levels_summary mapping") - for level in LEVELS: - summary = levels.get(level) - if not isinstance(summary, Mapping): - raise ValueError(f"mastery report is missing {level}") - try: - iou = float(summary["mean_iou_at_1"]) - pure = float(summary["pure_executable_rate"]) - except (KeyError, TypeError, ValueError) as exc: - raise ValueError(f"mastery report has invalid {level} metrics") from exc - if not all(math.isfinite(value) and 0.0 <= value <= 1.0 for value in (iou, pure)): - raise ValueError(f"mastery report has out-of-range {level} metrics") - if iou < minimum_iou or pure < minimum_pure_executable_rate: - return level - return None - - -def validate_curriculum_selection( - recipe: Mapping[str, Any], - report: Mapping[str, Any], - *, - current_level: str, - replay_levels: str, -) -> tuple[str, tuple[str, ...]]: - """Bind caller-supplied curriculum arguments to the computed mastery result.""" - - curriculum = recipe["rl"]["curriculum"] - computed = first_unmastered_level( - report, - minimum_iou=float(curriculum["minimum_iou"]), - minimum_pure_executable_rate=float( - curriculum["minimum_pure_executable_rate"] - ), - ) - if computed is None: - raise ValueError("all curriculum levels are mastered; no RL stage may start") - current = normalize_level(current_level) - if current != computed: - raise ValueError( - f"current_level must be the computed first unmastered level {computed}; " - f"got {current}" - ) - replay = normalize_replay_levels(replay_levels) - expected = expected_replay_levels(current) - if replay != expected: - rendered = ",".join(expected) - raise ValueError( - f"replay_levels must be exactly the earlier levels {rendered!r}; " - f"got {replay_levels!r}" - ) - return current, replay - - -def validate_rl_cli( - repo_root: Path, - cfg: Any, - *, - mastery_report: Mapping[str, Any], -) -> dict[str, Any]: - """Reject recipe overrides and validate the computed curriculum stage.""" - - recipe = load_recipe(repo_root) - model = recipe["model"] - rl = recipe["rl"] - launch = recipe["launch"] - _require_canonical_dataset(repo_root, cfg.dataset_root) - _require_equal("model_name", cfg.model_name, model["name"]) - _require_equal("renderer_name", cfg.renderer_name, model["renderer"]) - _require_equal("lora_rank", cfg.lora_rank, model["lora_rank"]) - _require_equal("group_size", cfg.group_size, rl["group_size"]) - _require_equal("groups_per_batch", cfg.groups_per_batch, rl["groups_per_batch"]) - _require_equal("learning_rate", cfg.learning_rate, rl["learning_rate"]) - _require_equal("max_tokens", cfg.max_tokens, rl["max_output_tokens"]) - _require_equal("temperature", cfg.temperature, rl["temperature"]) - if not isinstance(cfg.max_steps, int) or not 1 <= cfg.max_steps <= rl["stage_max_steps"]: - raise ValueError( - f"max_steps must be between 1 and the committed ceiling " - f"{rl['stage_max_steps']}; got {cfg.max_steps!r}" - ) - _require_equal("eval_every", cfg.eval_every, rl["evaluate_every_steps"]) - _require_equal("save_every", cfg.save_every, rl["save_every_steps"]) - _require_equal("loss_fn", cfg.loss_fn, rl["loss_fn"]) - _require_contained_log_path(repo_root, cfg.log_path) - _require_equal( - "confirm_spend", - cfg.confirm_spend, - launch["confirmation_tokens"]["rl"], - ) - if not str(cfg.load_checkpoint_path).strip(): - raise ValueError("load_checkpoint_path must name the selected SFT checkpoint") - mastery_path = Path(cfg.mastery_report).expanduser().resolve(strict=True) - if not mastery_path.is_file(): - raise ValueError("mastery_report must be a regular JSON file") - report_sampler = str(mastery_report.get("sampler_path", "")) - expected_checkpoint = weights_path_for_sampler(report_sampler) - _require_equal( - "load_checkpoint_path", - cfg.load_checkpoint_path, - expected_checkpoint, - ) - current, replay = validate_curriculum_selection( - recipe, - mastery_report, - current_level=cfg.current_level, - replay_levels=cfg.replay_levels, - ) - return { - "kind": "rl", - "dataset_root": str(canonical_dataset_root(repo_root)), - "config_path": str(canonical_config_path(repo_root)), - "model_name": cfg.model_name, - "renderer_name": cfg.renderer_name, - "lora_rank": cfg.lora_rank, - "group_size": cfg.group_size, - "groups_per_batch": cfg.groups_per_batch, - "learning_rate": cfg.learning_rate, - "max_tokens": cfg.max_tokens, - "temperature": cfg.temperature, - "max_steps": cfg.max_steps, - "eval_every": cfg.eval_every, - "save_every": cfg.save_every, - "loss_fn": cfg.loss_fn, - "current_level": current, - "replay_levels": list(replay), - "load_checkpoint_path": expected_checkpoint, - "mastery_report": str(mastery_path), - "mastery_report_sha256": file_sha256(mastery_path), - } - - -def validate_evaluation_binding( - repo_root: Path, - binding: Mapping[str, Any], -) -> dict[str, Any]: - """Recompute the exact checkpoint-evaluation contract.""" - - recipe = load_recipe(repo_root) - model = recipe["model"] - rl = recipe["rl"] - evaluation = recipe["checkpoint_evaluation"] - role = binding.get("role") - if role == "checkpoint-geometry": - expected_levels = "all" - expected_rows = 0 - expected_k = evaluation["geometry_k"] - elif role == "behavioral-collapse-probe": - expected_levels = normalize_level(str(binding.get("levels", ""))) - expected_rows = evaluation["behavioral_rows"] - expected_k = evaluation["behavioral_k"] - else: - raise ValueError(f"unknown checkpoint evaluation role: {role!r}") - sampler_path = str(binding.get("sampler_path", "")).strip() - if not sampler_path: - raise ValueError("sampler_path is required") - output = _require_contained_report_path( - repo_root, - str(binding.get("output", "")), - ) - expected = { - "kind": "evaluation", - "dataset_root": str(canonical_dataset_root(repo_root)), - "config_path": str(canonical_config_path(repo_root)), - "model_name": model["name"], - "renderer_name": model["renderer"], - "lora_rank": model["lora_rank"], - "sampler_path": sampler_path, - "role": role, - "levels": expected_levels, - "rows": expected_rows, - "k": expected_k, - "temperature": rl["temperature"], - "top_p": rl["top_p"], - "max_tokens": rl["max_output_tokens"], - "seed": evaluation["task_selection_seed"], - "max_image": model["max_image_long_edge"], - "concurrency": evaluation["sampling_concurrency"], - "evaluator_workers": evaluation["evaluator_workers"], - "include_records": evaluation["include_records"], - "output": str(output), - } - for name, expected_value in expected.items(): - _require_equal(name, binding.get(name), expected_value) - return expected - - -def validate_evaluation_cli(repo_root: Path, args: Any) -> dict[str, Any]: - """Reject checkpoint-evaluation overrides and return its launch binding.""" - - recipe = load_recipe(repo_root) - _require_equal( - "confirm_spend", - args.confirm_spend, - recipe["launch"]["confirmation_tokens"]["evaluation"], - ) - _require_canonical_dataset(repo_root, args.dataset_root) - role = ( - "behavioral-collapse-probe" - if args.behavioral_probe - else "checkpoint-geometry" - ) - levels = str(args.levels).strip().upper() if args.behavioral_probe else "all" - candidate = { - "kind": "evaluation", - "dataset_root": str(canonical_dataset_root(repo_root)), - "config_path": str(canonical_config_path(repo_root)), - "model_name": args.model_name, - "renderer_name": args.renderer_name, - "lora_rank": recipe["model"]["lora_rank"], - "sampler_path": args.sampler_path, - "role": role, - "levels": levels, - "rows": args.rows, - "k": args.k, - "temperature": args.temperature, - "top_p": recipe["rl"]["top_p"], - "max_tokens": args.max_tokens, - "seed": args.seed, - "max_image": recipe["model"]["max_image_long_edge"], - "concurrency": args.concurrency, - "evaluator_workers": args.evaluator_workers, - "include_records": args.include_records, - "output": str(Path(args.output).expanduser().resolve()), - } - return validate_evaluation_binding(repo_root, candidate) - - -def canonical_json_sha256(value: Mapping[str, Any]) -> str: - payload = json.dumps(value, sort_keys=True, separators=(",", ":")).encode() - return hashlib.sha256(payload).hexdigest() - - -def validate_resolved_binding( - repo_root: Path, - binding: Mapping[str, Any], -) -> dict[str, Any]: - """Revalidate a serialized launch binding inside the preflight process.""" - - kind = binding.get("kind") - recipe = load_recipe(repo_root) - model = recipe["model"] - _require_equal( - "dataset_root", - str(Path(str(binding.get("dataset_root", ""))).resolve()), - str(canonical_dataset_root(repo_root)), - ) - _require_equal( - "config_path", - str(Path(str(binding.get("config_path", ""))).resolve()), - str(canonical_config_path(repo_root)), - ) - _require_equal("model_name", binding.get("model_name"), model["name"]) - _require_equal("renderer_name", binding.get("renderer_name"), model["renderer"]) - _require_equal("lora_rank", binding.get("lora_rank"), model["lora_rank"]) - if kind == "sft": - sft = recipe["sft"] - expected = { - "learning_rate": sft["learning_rate"], - "lr_schedule": sft["tinker_lr_schedule"], - "num_epochs": sft["effective_passes"], - "batch_size": sft["batch_size"], - "max_length": sft["max_sequence_tokens"], - "max_image": model["max_image_long_edge"], - "save_every": sft["save_every_steps"], - "eval_every": sft["evaluate_every_steps"], - "max_steps": sft["max_steps"], - } - elif kind == "rl": - rl = recipe["rl"] - expected = { - "group_size": rl["group_size"], - "groups_per_batch": rl["groups_per_batch"], - "learning_rate": rl["learning_rate"], - "max_tokens": rl["max_output_tokens"], - "temperature": rl["temperature"], - "eval_every": rl["evaluate_every_steps"], - "save_every": rl["save_every_steps"], - "loss_fn": rl["loss_fn"], - } - checkpoint_path = str(binding.get("load_checkpoint_path", "")) - mastery_path = Path(str(binding.get("mastery_report", ""))).resolve( - strict=True - ) - if not mastery_path.is_file(): - raise ValueError("resolved RL mastery_report is not a regular file") - mastery_sha256 = file_sha256(mastery_path) - _require_equal( - "mastery_report_sha256", - binding.get("mastery_report_sha256"), - mastery_sha256, - ) - try: - mastery_report = json.loads(mastery_path.read_text(encoding="utf-8")) - except json.JSONDecodeError as exc: - raise ValueError("resolved RL mastery_report is not valid JSON") from exc - if not isinstance(mastery_report, Mapping): - raise ValueError("resolved RL mastery_report must be a JSON object") - from rl.track_a.evaluation_contract import ( - build_checkpoint_selection_contract, - validate_evaluation_report, - ) - - report_contract = build_checkpoint_selection_contract( - repo_root, - canonical_dataset_root(repo_root), - ) - validate_evaluation_report( - mastery_report, - contract=report_contract, - expected_role="checkpoint-geometry", - ) - expected_checkpoint = weights_path_for_sampler( - str(mastery_report.get("sampler_path", "")) - ) - _require_equal( - "load_checkpoint_path", - checkpoint_path, - expected_checkpoint, - ) - max_steps = binding.get("max_steps") - if ( - not isinstance(max_steps, int) - or not 1 <= max_steps <= int(rl["stage_max_steps"]) - ): - raise ValueError("resolved RL max_steps exceeds the committed ceiling") - current = normalize_level(str(binding.get("current_level", ""))) - replay_value = binding.get("replay_levels") - if not isinstance(replay_value, list): - raise ValueError("resolved RL replay_levels must be a list") - replay = normalize_replay_levels(",".join(str(item) for item in replay_value)) - if replay != expected_replay_levels(current): - raise ValueError("resolved RL replay levels are not all earlier levels") - validate_curriculum_selection( - recipe, - mastery_report, - current_level=current, - replay_levels=",".join(replay), - ) - elif kind == "evaluation": - evaluation = validate_evaluation_binding(repo_root, binding) - expected = { - name: value - for name, value in evaluation.items() - if name not in {"kind", "dataset_root", "config_path", "candidate_sandbox"} - } - else: - raise ValueError(f"unknown resolved launch kind: {kind!r}") - for name, expected_value in expected.items(): - _require_equal(name, binding.get(name), expected_value) - if kind in {"rl", "evaluation"}: - sandbox = binding.get("candidate_sandbox") - if not isinstance(sandbox, Mapping): - raise ValueError("resolved launch has no candidate_sandbox binding") - boundary = require_execution_boundary() - sandbox_expected = { - "runtime_path": str(boundary.runtime_path), - "daemon_endpoint": boundary.daemon_endpoint, - "image_ref": boundary.image_ref, - "image_id": boundary.image_id, - "workspace_root": str(boundary.workspace_root), - } - for name, expected_value in sandbox_expected.items(): - _require_equal( - f"candidate_sandbox.{name}", - sandbox.get(name), - expected_value, - ) - return dict(binding) diff --git a/rl/track_a/reward.py b/rl/track_a/reward.py index 442dab11..ff4a3640 100644 --- a/rl/track_a/reward.py +++ b/rl/track_a/reward.py @@ -1,23 +1,44 @@ -"""The deliberately small Track A reward policy.""" +"""The Track A reward policies. + +``shaped_v3b`` is the campaign policy: the private library's reward v3-b, +ported byte-for-byte in its constants. It pays only for structure beyond the +degenerate bbox-rectangle answer (``iou_n``/``dice_n`` — rectangle-baseline- +normalized excess IoU/Dice), adds a boundary-chamfer "teacher" term for +count/boundary progress IoU cannot see, keeps a 0.05 executable floor so a +valid-but-wrong program is distinguishable from a crash, and scores +rectangle-like targets raw (normalizing against a rectangle that IS the +answer would make every program look identical). + +Purity stays pre-execution: the evaluator returns ``SOURCE_REJECTED`` as a +MODEL failure, so an impure program earns 0.0 — the old ship-mode semantics +(a floor for impure programs trains the violation). + +``None`` means the rollout must be masked (or the run aborted for a broken +reference); it must never be converted to a policy reward of zero. +""" from __future__ import annotations import math +from rl.common.baselines import TaskBaseline from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus VALID_PROGRAM_FLOOR = 0.05 IOU_WEIGHT = 0.95 -REWARD_POLICIES = ("validity_floor", "raw_iou") +SHAPED_FLOOR = 0.05 +W_IOU3, W_DICE3, W_TEACHER = 0.40, 0.15, 0.40 +TAU_FRAC = 0.05 # tau = TAU_FRAC * diag_um (chamfer softness scale) +TEACHER_DENOM_FLOOR = 0.3 # floors the rect-subtraction denominator (1 - t2b_rect) +DEGENERATE_IOU_RECT = 0.95 # rect-like target: the rectangle already scores this IoU -def reward_for(result: EvaluationResult) -> float | None: - """Map measurement to reward without hiding evaluator failures. +REWARD_POLICIES = ("validity_floor", "raw_iou", "shaped_v3b") - ``None`` means the rollout must be masked (or the run aborted for a broken - reference); it must never be converted to a policy reward of zero. - """ + +def reward_for(result: EvaluationResult) -> float | None: + """Map measurement to reward without hiding evaluator failures.""" if result.status is EvaluationStatus.OK: if result.iou is None: @@ -28,27 +49,114 @@ def reward_for(result: EvaluationResult) -> float | None: return None +def norm_excess(x: float, base: float) -> float: + """Excess over the rectangle baseline, as a fraction of the remaining + headroom. 0 at (or below) the baseline, 1 at perfect.""" + + if base >= 1.0: + return 0.0 + return max(0.0, (x - base) / (1.0 - base)) + + +def _t2b(chamfer_um: float | None, tau: float) -> float: + """exp(-chamfer/tau) closeness-to-boundary score. None/inf chamfer (an + empty mask, a failed render) scores 0 — worst case, never a crash.""" + + if chamfer_um is None or chamfer_um == float("inf"): + return 0.0 + return math.exp(-float(chamfer_um) / tau) + + +def _validated_scores(result: EvaluationResult) -> tuple[float, float]: + if result.iou is None or result.dice is None: + raise ValueError("successful evaluation has no IoU/Dice") + iou = float(result.iou) + dice = float(result.dice) + for value in (iou, dice): + if not math.isfinite(value) or not 0.0 <= value <= 1.0: + raise ValueError("successful evaluation has an invalid measurement") + return iou, dice + + +def shaped_components( + result: EvaluationResult, + baseline: TaskBaseline, +) -> dict[str, float]: + """The logging breakdown of ``shaped_v3b``. Zeros for non-OK results.""" + + if result.status is not EvaluationStatus.OK: + return { + "iou_n": 0.0, + "dice_n": 0.0, + "t2b": 0.0, + "teacher_nb": 0.0, + "degenerate": float(baseline.iou_rect >= DEGENERATE_IOU_RECT), + } + iou, dice = _validated_scores(result) + diag = baseline.diag_um if baseline.diag_um > 0 else 1e-9 + tau = max(TAU_FRAC * diag, 1e-9) + chamfer = result.metrics.get("chamfer_boundary_um") + chamfer_rect = baseline.chamfer_rect_boundary_um + t2b = _t2b(chamfer, tau) + t2b_rect = _t2b(chamfer_rect if chamfer_rect is not None else diag, tau) + teacher_nb = max(0.0, t2b - t2b_rect) / max(1.0 - t2b_rect, TEACHER_DENOM_FLOOR) + return { + "iou_n": norm_excess(iou, baseline.iou_rect), + "dice_n": norm_excess(dice, baseline.dice_rect), + "t2b": t2b, + "teacher_nb": teacher_nb, + "degenerate": float(baseline.iou_rect >= DEGENERATE_IOU_RECT), + } + + +def _shaped_v3b(result: EvaluationResult, baseline: TaskBaseline) -> float: + iou, dice = _validated_scores(result) + parts = shaped_components(result, baseline) + if baseline.iou_rect >= DEGENERATE_IOU_RECT: + return ( + SHAPED_FLOOR + + W_IOU3 * iou + + W_DICE3 * dice + + W_TEACHER * parts["t2b"] + ) + return ( + SHAPED_FLOOR + + W_IOU3 * parts["iou_n"] + + W_DICE3 * parts["dice_n"] + + W_TEACHER * parts["teacher_nb"] + ) + + def reward_for_policy( result: EvaluationResult, *, policy: str, + baseline: TaskBaseline | None = None, ) -> float | None: """Apply one explicit reward policy while preserving fault attribution.""" if policy == "validity_floor": return reward_for(result) - if policy != "raw_iou": - raise ValueError(f"unknown reward policy: {policy!r}") - if result.status is EvaluationStatus.OK: - if result.iou is None: - raise ValueError("successful evaluation has no IoU") - iou = float(result.iou) - if not math.isfinite(iou) or not 0.0 <= iou <= 1.0: - raise ValueError("successful evaluation has an invalid IoU") - return iou - if result.attribution is Attribution.MODEL: - return 0.0 - return None + if policy == "raw_iou": + if result.status is EvaluationStatus.OK: + if result.iou is None: + raise ValueError("successful evaluation has no IoU") + iou = float(result.iou) + if not math.isfinite(iou) or not 0.0 <= iou <= 1.0: + raise ValueError("successful evaluation has an invalid IoU") + return iou + if result.attribution is Attribution.MODEL: + return 0.0 + return None + if policy == "shaped_v3b": + if result.status is EvaluationStatus.OK: + if baseline is None: + raise ValueError("shaped_v3b requires a per-task baseline") + return float(_shaped_v3b(result, baseline)) + if result.attribution is Attribution.MODEL: + return 0.0 + return None + raise ValueError(f"unknown reward policy: {policy!r}") def group_rewards(results: list[EvaluationResult]) -> list[float | None]: diff --git a/rl/track_a/run.py b/rl/track_a/run.py new file mode 100644 index 00000000..51479610 --- /dev/null +++ b/rl/track_a/run.py @@ -0,0 +1,701 @@ +#!/usr/bin/env python3 +"""The Track A campaign runner: one recipe, no ceremony. + + A runA-l0sft-rl no-think @ 4096 L0-SFT -> RL L0..L4 + B runB-base-rl no-think @ 4096 base -> RL L0..L4 + C runC-think-base-rl think @ 60000 base -> RL L0..L4 + D runD-think-l0sft-rl think @ 60000 L0-SFT -> RL L0..L4 (gated) + E runE-a-l1sft no-think @ 4096 fork of A after rl-l0 -> L1-SFT (gated) -> RL L1..L4 + F runF-a-mixed no-think @ 4096 fork of A after rl-l0 -> RL L1..L4 at a 50% backward mix + G runG-a-ladder no-think @ 4096 fork of A after rl-l0 -> SFT+RL ladder at every level (held until E reports) + +Each stage is one subprocess over the lean entry points (crash isolation, +one W&B run per stage, per-stage console logs); stages chain on the previous +stage's final training weights with a fresh optimizer, exactly as the old +lineage chained its runs. A stage that already finished is skipped, so +re-running the same command resumes the campaign. Forked campaigns seed a +prefix of completed stages (checkpoint ledgers + DONE) from a source run and +continue from its weights with zero recompute. + +The fixed probe (probe_eval) runs at campaign start and after every stage. +Every SFT insertion is gated by probes on the SFT'd level: fresh runs must +beat the base model (absolute rule); forked mid-lineage SFTs must not lose +executable rate and must gain IoU against the fork-origin probe (relative +rule). A failed mid-lineage gate skips the SFT and continues RL from the +pre-SFT weights — the negative result is recorded, the budget is not spent. + +Smoke mode prices an operating point before the full campaign: +``--smoke nothink`` / ``--smoke think`` runs 2 GRPO steps at 2 groups. +""" + +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +import time +from dataclasses import asdict, dataclass, field +from pathlib import Path + +from tinker_cookbook.checkpoint_utils import get_last_checkpoint + +from rl.common.baselines import cached_rect_baseline +from rl.track_a.curriculum import LEVELS +from rl.track_a.launch import run_launch_preflight +from rl.track_a.probe_eval import run_probe, select_probe_tasks + +_REPO_ROOT = Path(__file__).resolve().parents[2] +_DATASET_ROOT = _REPO_ROOT / "dataset" + +CONFIRM_SPEND = "PIXCELL_RUNS_V2" +DEFAULT_RUNS_ROOT = "~/pixcell-training/runs-v2" +FALLBACK_RUNS_ROOT = "~/pixcell-training/runs-v2" + +SFT_GATE_MIN_EXECUTABLE = 0.75 +SFT_GATE_RELATIVE_EXEC_SLACK = 0.10 + + +@dataclass(frozen=True) +class RunPlan: + key: str + name: str + renderer_name: str + max_tokens: int + sft_levels: tuple[str, ...] = () + sft_epochs: int = 2 + steps_per_level: int = 30 + replay_groups_per_step: int = 0 + fork_from: str = "" # source run name under the same runs root + fork_through: str = "" # last stage seeded from the source, e.g. "rl-l0" + + @property + def sft(self) -> bool: + return bool(self.sft_levels) + + +RUN_PLANS = { + "A": RunPlan("A", "runA-l0sft-rl", "qwen3_5_disable_thinking", 4096, ("L0",)), + "B": RunPlan("B", "runB-base-rl", "qwen3_5_disable_thinking", 4096), + "C": RunPlan("C", "runC-think-base-rl", "qwen3_5", 60000), + "D": RunPlan("D", "runD-think-l0sft-rl", "qwen3_5", 60000, ("L0",)), + "E": RunPlan( + "E", + "runE-a-l1sft", + "qwen3_5_disable_thinking", + 4096, + ("L0", "L1"), + sft_epochs=1, + fork_from="runA-l0sft-rl", + fork_through="rl-l0", + ), + "F": RunPlan( + "F", + "runF-a-mixed", + "qwen3_5_disable_thinking", + 4096, + ("L0",), + replay_groups_per_step=4, + fork_from="runA-l0sft-rl", + fork_through="rl-l0", + ), + "G": RunPlan( + "G", + "runG-a-ladder", + "qwen3_5_disable_thinking", + 4096, + ("L0", "L1", "L2", "L3", "L4"), + sft_epochs=1, + fork_from="runA-l0sft-rl", + fork_through="rl-l0", + ), +} +MODEL_NAME = "Qwen/Qwen3.6-35B-A3B" + +_STAGE_ORDER = [ + name + for level in LEVELS + for name in (f"sft-{level.lower()}", f"rl-{level.lower()}") +] + + +@dataclass(frozen=True) +class Stage: + name: str + module: str + args: dict[str, str] = field(default_factory=dict) + + @property + def sft_level(self) -> str | None: + if self.name.startswith("sft-"): + return self.name.split("-", 1)[1].upper() + return None + + +def build_stages( + plan: RunPlan, + *, + run_root: Path, + wandb_project: str, + smoke: bool = False, +) -> list[Stage]: + """The full ordered stage list for one run. Checkpoint chaining is + resolved at execution time, not here. Forked prefixes appear here too — + the DONE markers seeded by the fork make the runner skip them.""" + + steps = 2 if smoke else plan.steps_per_level + groups = "2" if smoke else "8" + canaries = "8" if smoke else "40" + stages: list[Stage] = [] + levels = ["L0"] if smoke else list(LEVELS) + for level in levels: + if not smoke and level in plan.sft_levels: + stage_name = f"sft-{level.lower()}" + stages.append( + Stage( + name=stage_name, + module="rl.track_a.train_sft", + args={ + "confirm_spend": CONFIRM_SPEND, + "model_name": MODEL_NAME, + "renderer_name": plan.renderer_name, + "levels": level, + "num_epochs": str(plan.sft_epochs), + "learning_rate": "1e-4", + "log_path": str(run_root / "stages" / stage_name), + "wandb_project": wandb_project, + "wandb_name": f"{plan.name}-{stage_name}", + "skip_preflight": "True", + }, + ) + ) + stage_name = f"rl-{level.lower()}" + rl_args = { + "confirm_spend": CONFIRM_SPEND, + "model_name": MODEL_NAME, + "renderer_name": plan.renderer_name, + "current_level": level, + "max_tokens": str(plan.max_tokens), + "max_steps": str(steps), + "groups_per_batch": groups, + "group_size": "8", + "validation_canaries": canaries, + "reward_policy": "shaped_v3b", + "log_path": str(run_root / "stages" / stage_name), + "wandb_project": wandb_project, + "wandb_name": f"{plan.name}-{stage_name}", + "skip_preflight": "True", + } + if plan.replay_groups_per_step: + rl_args["replay_groups_per_step"] = str(plan.replay_groups_per_step) + stages.append( + Stage(name=stage_name, module="rl.track_a.train_rl", args=rl_args) + ) + return stages + + +def _stage_done(stage: Stage) -> bool: + return (Path(stage.args["log_path"]) / "DONE").exists() + + +def _mark_done(stage: Stage) -> None: + (Path(stage.args["log_path"]) / "DONE").write_text("ok\n", encoding="utf-8") + + +def _last_sampler_path(log_path: str) -> str: + """Sampling-only weights (``.../sampler_weights/…``) — for probes.""" + + record = get_last_checkpoint(log_path, required_key="sampler_path") + if record is None or not record.sampler_path: + raise RuntimeError(f"no sampler checkpoint recorded under {log_path}") + return record.sampler_path + + +def _last_weights_path(log_path: str) -> str: + """Training weights (``.../weights/…``) — the only form Tinker's + load_weights accepts, so the only form stages may chain on.""" + + record = get_last_checkpoint(log_path, required_key="state_path") + if record is None or not record.state_path: + raise RuntimeError( + f"no training-weights checkpoint recorded under {log_path}" + ) + return record.state_path + + +def _environment() -> dict[str, str]: + environment = dict(os.environ) + roots = [str(_REPO_ROOT / "src"), str(_REPO_ROOT)] + if environment.get("PYTHONPATH"): + roots.append(environment["PYTHONPATH"]) + environment["PYTHONPATH"] = os.pathsep.join(roots) + return environment + + +def _execute_stage(stage: Stage, *, extra: dict[str, str]) -> None: + log_dir = Path(stage.args["log_path"]) + log_dir.mkdir(parents=True, exist_ok=True) + arguments = {**stage.args, **extra} + command = [ + sys.executable, + "-m", + stage.module, + *[f"{key}={value}" for key, value in arguments.items()], + ] + console = log_dir / "console.log" + started = time.monotonic() + with console.open("a", encoding="utf-8") as sink: + sink.write(f"# {' '.join(command)}\n") + sink.flush() + completed = subprocess.run( + command, + cwd=_REPO_ROOT, + env=_environment(), + stdout=sink, + stderr=subprocess.STDOUT, + ) + if completed.returncode != 0: + raise RuntimeError( + f"stage {stage.name} failed with exit {completed.returncode}; " + f"see {console}" + ) + print( + f"[{stage.name}] completed in {time.monotonic() - started:.0f}s", + flush=True, + ) + + +def _probe( + plan: RunPlan, + *, + run_root: Path, + tag: str, + model_path: str | None, +) -> dict: + return run_probe( + dataset_root=_DATASET_ROOT, + model_name=MODEL_NAME, + renderer_name=plan.renderer_name, + model_path=model_path, + max_tokens=plan.max_tokens, + out_path=run_root / "probes" / f"{tag}.json", + ) + + +def _sft_gate( + reference_probe: dict, + sft_probe: dict, + level: str, + *, + relative: bool, +) -> tuple[bool, str]: + """Should RL proceed from this SFT checkpoint? + + Fresh lineages (reference = raw base model) use the absolute rule: the + SFT'd level must be reliably executable and beat the base on IoU. + Forked mid-lineage insertions use the relative rule: the SFT must not + lose executable rate (beyond slack) and must gain IoU on its level — + the dose-response caution made concrete. + """ + + reference = reference_probe["per_level"][level] + candidate = sft_probe["per_level"][level] + if relative: + floor = reference["executable_rate"] - SFT_GATE_RELATIVE_EXEC_SLACK + if candidate["executable_rate"] < floor: + return False, ( + f"SFT {level} executable rate {candidate['executable_rate']:.3f} " + f"lost more than {SFT_GATE_RELATIVE_EXEC_SLACK} vs the fork origin " + f"({reference['executable_rate']:.3f})" + ) + elif candidate["executable_rate"] < SFT_GATE_MIN_EXECUTABLE: + return False, ( + f"SFT {level} executable rate {candidate['executable_rate']:.3f} < " + f"{SFT_GATE_MIN_EXECUTABLE}" + ) + if candidate["mean_iou"] <= reference["mean_iou"]: + return False, ( + f"SFT {level} mean IoU {candidate['mean_iou']:.3f} does not beat " + f"the reference {reference['mean_iou']:.3f}" + ) + return True, f"sft beats reference on {level}" + + +def _write_provenance(plan: RunPlan, *, run_root: Path, smoke: bool) -> None: + provenance = run_root / "provenance" + provenance.mkdir(parents=True, exist_ok=True) + head = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=_REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + dirty = subprocess.run( + ["git", "status", "--porcelain"], + cwd=_REPO_ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.strip() + (provenance / "plan.json").write_text( + json.dumps( + { + "plan": asdict(plan), + "model_name": MODEL_NAME, + "git_head": head, + "git_dirty": bool(dirty), + "smoke": smoke, + "confirm_spend": CONFIRM_SPEND, + }, + indent=2, + ), + encoding="utf-8", + ) + probe_tasks = select_probe_tasks(_DATASET_ROOT) + baselines = { + task.sampler.opaque_id: { + "level": task.sampler.level, + **cached_rect_baseline(task.reference).__dict__, + } + for task in probe_tasks + } + (provenance / "probe_baselines.json").write_text( + json.dumps(baselines, indent=2), + encoding="utf-8", + ) + + +def _fork_l0_stage(source_run_root: Path, run_root: Path) -> None: + """Legacy depth-variant fork: seed rl-l0 WITHOUT its DONE marker so the + stage resumes and extends (the -s60 runs). Zero recompute.""" + + source = source_run_root / "stages" / "rl-l0" + target = run_root / "stages" / "rl-l0" + if (target / "checkpoints.jsonl").exists(): + return + if not (source / "DONE").exists(): + raise SystemExit(f"fork source not complete: {source}") + target.mkdir(parents=True, exist_ok=True) + for name in ("checkpoints.jsonl", "metrics.jsonl"): + payload = (source / name).read_text(encoding="utf-8") + (target / name).write_text(payload, encoding="utf-8") + (run_root / "provenance").mkdir(parents=True, exist_ok=True) + (run_root / "provenance" / "fork.json").write_text( + json.dumps({"rl_l0_forked_from": str(source)}, indent=2), + encoding="utf-8", + ) + print(f"[fork] rl-l0 seeded from {source}", flush=True) + + +def _fork_stages_through( + source_run_root: Path, + run_root: Path, + through: str, +) -> None: + """Seed a completed prefix of stages from a source campaign: checkpoint + ledgers, metrics, and DONE markers up to and including ``through``, plus + the source's probe at the fork point as this run's gate reference. The + runner then skips the prefix and chains from its final weights.""" + + if through not in _STAGE_ORDER: + raise SystemExit(f"unknown fork stage: {through}") + if (run_root / "provenance" / "fork.json").exists(): + return + seeded = [] + for name in _STAGE_ORDER: + source = source_run_root / "stages" / name + if source.is_dir(): + if not (source / "DONE").exists(): + raise SystemExit(f"fork source stage not complete: {source}") + target = run_root / "stages" / name + target.mkdir(parents=True, exist_ok=True) + for artifact in ("checkpoints.jsonl", "metrics.jsonl"): + payload = (source / artifact).read_text(encoding="utf-8") + (target / artifact).write_text(payload, encoding="utf-8") + (target / "DONE").write_text("forked\n", encoding="utf-8") + seeded.append(name) + if name == through: + break + if not seeded or seeded[-1] != through: + raise SystemExit( + f"fork source {source_run_root} has no completed {through}" + ) + origin_probe = source_run_root / "probes" / f"{through}.json" + if origin_probe.exists(): + (run_root / "probes").mkdir(parents=True, exist_ok=True) + (run_root / "probes" / "fork-origin.json").write_text( + origin_probe.read_text(encoding="utf-8"), + encoding="utf-8", + ) + (run_root / "provenance").mkdir(parents=True, exist_ok=True) + (run_root / "provenance" / "fork.json").write_text( + json.dumps( + { + "forked_from": str(source_run_root), + "through": through, + "stages": seeded, + }, + indent=2, + ), + encoding="utf-8", + ) + print(f"[fork] seeded {seeded} from {source_run_root}", flush=True) + + +def _resolve_runs_root(value: str) -> Path: + preferred = Path(value).expanduser() + if str(preferred) == DEFAULT_RUNS_ROOT and not preferred.parent.exists(): + preferred = Path(FALLBACK_RUNS_ROOT).expanduser() + preferred.mkdir(parents=True, exist_ok=True) + return preferred + + +def run_campaign(args: argparse.Namespace) -> None: + smoke = bool(args.smoke) + if smoke: + plan_key = "B" if args.smoke == "nothink" else "C" + base_plan = RUN_PLANS[plan_key] + plan = RunPlan( + key=base_plan.key, + name=f"smoke-{args.smoke}", + renderer_name=base_plan.renderer_name, + max_tokens=base_plan.max_tokens, + ) + else: + plan = RUN_PLANS[args.run] + max_tokens = args.max_tokens or plan.max_tokens + if ( + args.steps_per_level != plan.steps_per_level + or args.fork_l0_from + or max_tokens != plan.max_tokens + ): + name = plan.name + if args.steps_per_level != plan.steps_per_level: + name = f"{name}-s{args.steps_per_level}" + if max_tokens != plan.max_tokens: + name = f"{name}-t{max_tokens}" + plan = RunPlan( + key=plan.key, + name=name, + renderer_name=plan.renderer_name, + max_tokens=max_tokens, + # The legacy rl-l0 extension fork carries its source's + # (possibly SFT-warm-started) lineage without seeding the + # SFT stage dir; re-running SFT would discard the fork. + sft_levels=() if args.fork_l0_from else plan.sft_levels, + sft_epochs=plan.sft_epochs, + steps_per_level=args.steps_per_level, + replay_groups_per_step=plan.replay_groups_per_step, + fork_from="" if args.fork_l0_from else plan.fork_from, + fork_through="" if args.fork_l0_from else plan.fork_through, + ) + runs_root = _resolve_runs_root(args.runs_root) + run_root = runs_root / plan.name + run_root.mkdir(parents=True, exist_ok=True) + if args.fork_l0_from: + _fork_l0_stage(Path(args.fork_l0_from), run_root) + fork_from = args.fork_from or plan.fork_from + fork_through = args.fork_through or plan.fork_through + forked = False + if fork_from and not args.fork_l0_from: + source_root = Path(fork_from) + if not source_root.is_absolute(): + source_root = runs_root / fork_from + _fork_stages_through(source_root, run_root, fork_through or "rl-l0") + forked = True + required = ["TINKER_API_KEY", "WANDB_API_KEY"] + if os.environ.get("PIXCELL_REQUIRE_ISOLATION", "0") == "1": + required.append("PIXCELL_EVALUATOR_IMAGE") + for variable in required: + if not os.environ.get(variable): + raise SystemExit(f"{variable} is not present") + _write_provenance(plan, run_root=run_root, smoke=smoke) + if not args.skip_preflight: + run_launch_preflight( + _REPO_ROOT, + dataset_root=_DATASET_ROOT, + model_name=MODEL_NAME, + renderer_name=plan.renderer_name, + require_sandbox=os.environ.get("PIXCELL_REQUIRE_ISOLATION", "0") == "1", + tokenize_sft=plan.sft, + sft_levels=",".join(plan.sft_levels), + ) + + stages = build_stages( + plan, + run_root=run_root, + wandb_project=args.wandb_project, + smoke=smoke, + ) + checkpoint: str | None = None + reference_probe: dict | None = None + if not smoke: + origin_path = run_root / "probes" / "fork-origin.json" + base_path = run_root / "probes" / "base.json" + if forked and origin_path.exists(): + reference_probe = json.loads(origin_path.read_text(encoding="utf-8")) + elif base_path.exists(): + reference_probe = json.loads(base_path.read_text(encoding="utf-8")) + elif not forked: + print("[probe] base model", flush=True) + reference_probe = _probe( + plan, run_root=run_root, tag="base", model_path=None + ) + + for stage in stages: + if _stage_done(stage): + print(f"[{stage.name}] already complete, skipping", flush=True) + checkpoint = _last_weights_path(stage.args["log_path"]) + continue + pre_stage_checkpoint = checkpoint + extra: dict[str, str] = {} + if checkpoint: + extra["load_checkpoint_path"] = checkpoint + _execute_stage(stage, extra=extra) + _mark_done(stage) + checkpoint = _last_weights_path(stage.args["log_path"]) + if smoke: + continue + print(f"[probe] after {stage.name}", flush=True) + probe = _probe( + plan, + run_root=run_root, + tag=stage.name, + model_path=_last_sampler_path(stage.args["log_path"]), + ) + level = stage.sft_level + if level is not None: + if reference_probe is None: + raise SystemExit( + f"no reference probe available to gate {stage.name}" + ) + passed, reason = _sft_gate( + reference_probe, + probe, + level, + relative=forked or level != "L0", + ) + gate_name = ( + "sft_gate.json" if stage.name == "sft-l0" else f"sft_gate-{stage.name}.json" + ) + (run_root / "probes" / gate_name).write_text( + json.dumps({"passed": passed, "reason": reason}, indent=2), + encoding="utf-8", + ) + if passed: + print(f"[gate] {reason}", flush=True) + elif forked or level != "L0": + print( + f"[gate] {reason} — skipping this SFT, continuing RL " + "from the pre-SFT weights", + flush=True, + ) + checkpoint = pre_stage_checkpoint + elif plan.key == "D": + print(f"[gate] {reason} — falling back to RL from base", flush=True) + checkpoint = None + else: + raise SystemExit(f"SFT sanity gate failed: {reason}") + else: + # RL boundary reached: this probe becomes the moving reference + # for any later SFT insertion (the ladder compares each rung + # against the lineage state it would improve on). + reference_probe = probe + + print(f"[{plan.name}] campaign complete", flush=True) + if smoke: + _summarize_smoke(run_root, stages) + + +def _summarize_smoke(run_root: Path, stages: list[Stage]) -> None: + summary: dict[str, object] = {} + for stage in stages: + metrics_path = Path(stage.args["log_path"]) / "metrics.jsonl" + if not metrics_path.exists(): + continue + rows = [ + json.loads(line) + for line in metrics_path.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + token_keys = sorted( + { + key + for row in rows + for key, value in row.items() + if "token" in key.lower() and isinstance(value, (int, float)) + } + ) + summary[stage.name] = { + "steps": len(rows), + "token_metrics": { + key: sum(float(row.get(key, 0.0)) for row in rows) + for key in token_keys + }, + "reward_by_step": [ + row.get("env/all/reward/total") for row in rows + ], + } + out = run_root / "smoke_summary.json" + out.write_text(json.dumps(summary, indent=2), encoding="utf-8") + print(json.dumps(summary, indent=2)) + print(f"[smoke] summary written to {out}", flush=True) + + +def parser() -> argparse.ArgumentParser: + result = argparse.ArgumentParser(description=__doc__) + result.add_argument("--run", choices=sorted(RUN_PLANS), help="which run to execute") + result.add_argument( + "--smoke", + choices=("nothink", "think"), + help="price an operating point: 2 GRPO steps at 2 groups on L0", + ) + result.add_argument("--runs-root", default=DEFAULT_RUNS_ROOT) + result.add_argument( + "--steps-per-level", + type=int, + default=30, + help="RL steps per curriculum level; non-default gets a -sN run-name suffix", + ) + result.add_argument( + "--fork-l0-from", + default="", + help="legacy depth-variant fork: seed rl-l0 WITHOUT DONE so it resumes and extends", + ) + result.add_argument( + "--fork-from", + default="", + help="run name/path whose completed stage prefix seeds this campaign (overrides the plan's fork_from)", + ) + result.add_argument( + "--fork-through", + default="", + help="last stage taken from the fork source (default: the plan's fork_through)", + ) + result.add_argument( + "--max-tokens", + type=int, + default=0, + help="optional sampling-ceiling override; non-default gets a -tN run-name suffix (never applied to an existing run)", + ) + result.add_argument("--wandb-project", default="pixcell-rl") + result.add_argument("--confirm-spend", default="") + result.add_argument("--skip-preflight", action="store_true") + return result + + +def main() -> None: + args = parser().parse_args() + if args.confirm_spend != CONFIRM_SPEND: + raise SystemExit(f"paid launch blocked: pass --confirm-spend {CONFIRM_SPEND}") + if bool(args.run) == bool(args.smoke): + raise SystemExit( + "pass exactly one of --run {A..G} or --smoke {nothink,think}" + ) + run_campaign(args) + + +if __name__ == "__main__": + main() diff --git a/rl/track_a/select_sft_checkpoint.py b/rl/track_a/select_sft_checkpoint.py deleted file mode 100644 index 24889687..00000000 --- a/rl/track_a/select_sft_checkpoint.py +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env python3 -"""Select a Track A SFT checkpoint from executable reports, never NLL.""" - -from __future__ import annotations - -import argparse -import json -from pathlib import Path - -from rl.track_a.checkpoint_selection import select_sft_checkpoint -from rl.track_a.evaluation_contract import ( - build_checkpoint_selection_contract, -) - - -REPO_ROOT = Path(__file__).resolve().parents[2] - - -def main() -> None: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--geometry-reports", - nargs="+", - required=True, - type=Path, - help="full depth/validation k=1 reports", - ) - parser.add_argument( - "--behavioral-reports", - nargs="+", - required=True, - type=Path, - help="paired small balanced unmastered-level G4 reports", - ) - args = parser.parse_args() - geometry = [ - (str(path), json.loads(path.read_text(encoding="utf-8"))) - for path in args.geometry_reports - ] - behavior = [ - (str(path), json.loads(path.read_text(encoding="utf-8"))) - for path in args.behavioral_reports - ] - try: - contract = build_checkpoint_selection_contract( - REPO_ROOT, - REPO_ROOT / "dataset", - ) - result = select_sft_checkpoint( - geometry, - behavior, - contract=contract, - ) - except (KeyError, OSError, RuntimeError, TypeError, ValueError) as exc: - parser.error(str(exc)) - print(json.dumps(result, indent=2)) - - -if __name__ == "__main__": - main() diff --git a/rl/track_a/tests/test_checkpoint_promotion.py b/rl/track_a/tests/test_checkpoint_promotion.py deleted file mode 100644 index e3711db1..00000000 --- a/rl/track_a/tests/test_checkpoint_promotion.py +++ /dev/null @@ -1,542 +0,0 @@ -from __future__ import annotations - -import hashlib - -import pytest - -from rl.track_a.checkpoint_metrics import ( - behavioral_probe_summary, - program_sha256, -) -from rl.track_a.checkpoint_selection import select_sft_checkpoint -from rl.track_a.evaluation_contract import ( - BEHAVIORAL_PROBE_ATTEMPTS, - BEHAVIORAL_PROBE_K, - BEHAVIORAL_PROBE_TASKS, - CURRICULUM_LEVELS, - FULL_GEOMETRY_REPRESENTATIONS, - FULL_GEOMETRY_TASKS, - REPORT_SCHEMA_VERSION, - CheckpointSelectionContract, - report_provenance, - task_ids_sha256, -) - - -GEOMETRY_IDS = [f"geometry-{index:04d}" for index in range(FULL_GEOMETRY_TASKS)] -BEHAVIOR_IDS = { - level: [ - f"behavior-{level.lower()}-{index:02d}" - for index in range(BEHAVIORAL_PROBE_TASKS) - ] - for level in CURRICULUM_LEVELS -} -CONTRACT = CheckpointSelectionContract( - source_git_sha="a" * 40, - depth_logical_release_sha256="b" * 64, - model="Qwen/example", - renderer="qwen_renderer", - temperature=1.0, - top_p=1.0, - max_tokens=4096, - task_selection_seed=7, - geometry_task_ids_sha256=task_ids_sha256(GEOMETRY_IDS), - behavioral_task_ids_sha256_by_level={ - level: task_ids_sha256(values) - for level, values in BEHAVIOR_IDS.items() - }, -) - - -def _attempt( - opaque_id: str, - program: str, - render: str | None, - reward: float, -) -> dict: - return { - "opaque_id": opaque_id, - "program_sha256": program, - "render_sha256": render, - "pure_executable": render is not None, - "reward": reward, - } - - -def _summary(*, tasks: int, attempts: int, iou: float, executable: float) -> dict: - return { - "tasks": tasks, - "attempts": attempts, - "pure_executable_rate": executable, - "mean_iou_at_1": iou, - "mean_best_at_k_iou": iou, - } - - -def _geometry(sampler: str, iou: float, executable: float = 1.0) -> dict: - level_tasks = { - "L0": 186, - "L1": 254, - "L2": 196, - "L3": 240, - "L4": 216, - } - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "evaluation_role": "checkpoint-geometry", - "sampler_path": sampler, - "model": CONTRACT.model, - "renderer": CONTRACT.renderer, - "split": "depth/validation", - "levels": "all", - "k": 1, - "temperature": CONTRACT.temperature, - "top_p": CONTRACT.top_p, - "max_tokens": CONTRACT.max_tokens, - "representation_macro_mean_iou": iou, - "summary": _summary( - tasks=FULL_GEOMETRY_TASKS, - attempts=FULL_GEOMETRY_TASKS, - iou=iou, - executable=executable, - ), - "statuses": {"ok": FULL_GEOMETRY_TASKS}, - "levels_summary": { - level: _summary( - tasks=tasks, - attempts=tasks, - iou=iou, - executable=executable, - ) - for level, tasks in level_tasks.items() - }, - "mean_completion_tokens": 100.0, - "behavioral_probe": None, - "records": None, - } - report["provenance"] = report_provenance( - contract=CONTRACT, - sampler_path=sampler, - role="checkpoint-geometry", - levels=CURRICULUM_LEVELS, - task_ids=GEOMETRY_IDS, - representation_count=FULL_GEOMETRY_REPRESENTATIONS, - requested_rows=None, - k=1, - attempt_count=FULL_GEOMETRY_TASKS, - sandbox_image_ref="sha256:" + "c" * 64, - sandbox_image_id="sha256:" + "c" * 64, - ) - return report - - -def _behavior(sampler: str, probe: dict, *, level: str = "L2") -> dict: - report = { - "schema_version": REPORT_SCHEMA_VERSION, - "evaluation_role": "behavioral-collapse-probe", - "sampler_path": sampler, - "model": CONTRACT.model, - "renderer": CONTRACT.renderer, - "split": "depth/validation", - "levels": level, - "k": BEHAVIORAL_PROBE_K, - "temperature": CONTRACT.temperature, - "top_p": CONTRACT.top_p, - "max_tokens": CONTRACT.max_tokens, - "representation_macro_mean_iou": 0.5, - "summary": _summary( - tasks=BEHAVIORAL_PROBE_TASKS, - attempts=BEHAVIORAL_PROBE_ATTEMPTS, - iou=0.5, - executable=0.75, - ), - "statuses": {"ok": BEHAVIORAL_PROBE_ATTEMPTS}, - "levels_summary": { - level: _summary( - tasks=BEHAVIORAL_PROBE_TASKS, - attempts=BEHAVIORAL_PROBE_ATTEMPTS, - iou=0.5, - executable=0.75, - ) - }, - "mean_completion_tokens": 100.0, - "behavioral_probe": probe, - "records": None, - } - report["provenance"] = report_provenance( - contract=CONTRACT, - sampler_path=sampler, - role="behavioral-collapse-probe", - levels=[level], - task_ids=BEHAVIOR_IDS[level], - representation_count=BEHAVIORAL_PROBE_TASKS, - requested_rows=BEHAVIORAL_PROBE_TASKS, - k=BEHAVIORAL_PROBE_K, - attempt_count=BEHAVIORAL_PROBE_ATTEMPTS, - sandbox_image_ref="sha256:" + "c" * 64, - sandbox_image_id="sha256:" + "c" * 64, - ) - return report - - -def _probe( - *, - collapsed: bool, - nonconstant: float, - program_mean: float, - level: str = "L2", -) -> dict: - groups = [] - nonconstant_groups = round(nonconstant * BEHAVIORAL_PROBE_TASKS) - extra_programs = round( - (program_mean - 1.0) * BEHAVIORAL_PROBE_TASKS - ) - program_counts = [1] * BEHAVIORAL_PROBE_TASKS - for index in range(BEHAVIORAL_PROBE_TASKS): - added = min(BEHAVIORAL_PROBE_K - 1, extra_programs) - program_counts[index] += added - extra_programs -= added - assert extra_programs == 0 - for index, opaque_id in enumerate(BEHAVIOR_IDS[level]): - is_nonconstant = index < nonconstant_groups - distinct_count = program_counts[index] - groups.append( - { - "opaque_id": opaque_id, - "distinct_program_count": distinct_count, - "successful_render_attempts": BEHAVIORAL_PROBE_K, - "distinct_successful_render_count": distinct_count, - "reward_std": 0.1 if is_nonconstant else 0.0, - "reward_range": 0.2 if is_nonconstant else 0.0, - "nonconstant_reward": is_nonconstant, - } - ) - identical_program_every_group = all( - group["distinct_program_count"] == 1 for group in groups - ) - constant_render_and_reward_every_group = all( - group["distinct_successful_render_count"] <= 1 - and not group["nonconstant_reward"] - for group in groups - ) - assert collapsed == ( - identical_program_every_group - or constant_render_and_reward_every_group - ) - return { - "probe_kind": "small-balanced-unmastered-g4", - "group_size": BEHAVIORAL_PROBE_K, - "prompt_groups": BEHAVIORAL_PROBE_TASKS, - "collapse_detected": collapsed, - "nonconstant_reward_group_fraction": nonconstant, - "mean_within_group_reward_std": nonconstant / 10, - "mean_distinct_programs_per_prompt": program_mean, - "mean_distinct_successful_renders_per_prompt": program_mean, - "collapse_signals": { - "identical_program_every_group": identical_program_every_group, - "constant_render_and_reward_every_group": ( - constant_render_and_reward_every_group - ), - }, - "groups": groups, - } - - -def test_program_hash_ignores_fences_and_trailing_whitespace(): - plain = "import gdsfactory as gf\nprint('x')" - fenced = "```python\nimport gdsfactory as gf \nprint('x')\n```\n" - assert program_sha256(plain) == program_sha256(fenced) - assert program_sha256(plain) == hashlib.sha256(plain.encode()).hexdigest() - - -def test_behavioral_probe_reports_per_prompt_diversity_and_reward_variance(): - records = [ - _attempt("a", "p1", "r1", 0.1), - _attempt("a", "p2", "r2", 0.2), - _attempt("a", "p2", "r2", 0.2), - _attempt("a", "p2", None, 0.0), - _attempt("b", "p3", "r3", 0.5), - _attempt("b", "p3", "r3", 0.5), - _attempt("b", "p3", "r3", 0.5), - _attempt("b", "p3", "r3", 0.5), - ] - summary = behavioral_probe_summary(records) - assert summary["prompt_groups"] == 2 - assert summary["nonconstant_reward_group_fraction"] == 0.5 - assert summary["mean_distinct_programs_per_prompt"] == 1.5 - assert summary["mean_distinct_successful_renders_per_prompt"] == 1.5 - assert summary["mean_within_group_reward_std"] > 0 - assert not summary["collapse_detected"] - assert summary["groups"][0]["distinct_program_sha256"] == ["p1", "p2"] - assert summary["groups"][0]["distinct_successful_render_sha256"] == [ - "r1", - "r2", - ] - - -def test_behavioral_probe_detects_every_prompt_collapsing(): - records = [ - _attempt(prompt, f"p-{prompt}", f"r-{prompt}", 0.6) - for prompt in ("a", "b") - for _ in range(4) - ] - summary = behavioral_probe_summary(records) - assert summary["collapse_detected"] - assert summary["collapse_signals"]["identical_program_every_group"] - assert summary["nonconstant_reward_group_fraction"] == 0 - - -def test_selection_vetoes_collapsed_model_but_never_ranks_on_diversity(): - geometry = [ - ("high-k1.json", _geometry("sampler-high", 0.9)), - ("middle-k1.json", _geometry("sampler-middle", 0.7)), - ("low-k1.json", _geometry("sampler-low", 0.6)), - ] - behavior = [ - ( - "high-g4.json", - _behavior( - "sampler-high", - _probe( - collapsed=True, - nonconstant=0.0, - program_mean=1.0, - ), - ), - ), - ( - "middle-g4.json", - _behavior( - "sampler-middle", - _probe( - collapsed=False, - nonconstant=0.1, - program_mean=1.2, - ), - ), - ), - ( - "low-g4.json", - _behavior( - "sampler-low", - _probe( - collapsed=False, - nonconstant=1.0, - program_mean=4.0, - ), - ), - ), - ] - selected = select_sft_checkpoint( - geometry, - behavior, - contract=CONTRACT, - ) - assert selected["selected"]["sampler_path"] == "sampler-middle" - assert selected["ranking"][-1]["sampler_path"] == "sampler-high" - assert not selected["behavioral_metrics_are_ranking_features"] - - -def test_selection_requires_one_behavior_report_per_geometry_report(): - with pytest.raises(ValueError, match="pairing mismatch"): - select_sft_checkpoint( - [("a.json", _geometry("sampler-a", 0.5))], - [], - contract=CONTRACT, - ) - - -def test_selection_fails_closed_on_missing_collapse_verdict(): - incomplete = _probe(collapsed=False, nonconstant=0.5, program_mean=2.0) - incomplete.pop("collapse_detected") - with pytest.raises(ValueError, match="collapse_detected must be boolean"): - select_sft_checkpoint( - [("a-k1.json", _geometry("sampler-a", 0.5))], - [("a-g4.json", _behavior("sampler-a", incomplete))], - contract=CONTRACT, - ) - - -@pytest.mark.parametrize( - ("field_path", "value", "message"), - [ - ( - ("representation_macro_mean_iou",), - float("nan"), - "representation_macro_mean_iou must be finite", - ), - ( - ("representation_macro_mean_iou",), - float("inf"), - "representation_macro_mean_iou must be finite", - ), - ( - ("summary", "pure_executable_rate"), - 1.01, - r"summary\.pure_executable_rate must be in \[0, 1\]", - ), - ], -) -def test_selection_rejects_nonfinite_and_out_of_range_ranking_metrics( - field_path: tuple[str, ...], - value: float, - message: str, -): - geometry = _geometry("sampler-a", 0.5) - target = geometry - for field in field_path[:-1]: - target = target[field] - target[field_path[-1]] = value - with pytest.raises(ValueError, match=message): - select_sft_checkpoint( - [("a-k1.json", geometry)], - [ - ( - "a-g4.json", - _behavior( - "sampler-a", - _probe( - collapsed=False, - nonconstant=0.5, - program_mean=2.0, - ), - ), - ) - ], - contract=CONTRACT, - ) - - -def test_selection_rejects_one_row_geometry_report_even_when_claimed_metric_is_high(): - geometry = _geometry("sampler-a", 0.99) - geometry["summary"]["tasks"] = 1 - geometry["summary"]["attempts"] = 1 - geometry["provenance"]["task_count"] = 1 - geometry["provenance"]["attempt_count"] = 1 - with pytest.raises(ValueError, match="provenance.task_count=1"): - select_sft_checkpoint( - [("one-row.json", geometry)], - [ - ( - "a-g4.json", - _behavior( - "sampler-a", - _probe( - collapsed=False, - nonconstant=0.5, - program_mean=2.0, - ), - ), - ) - ], - contract=CONTRACT, - ) - - -@pytest.mark.parametrize( - ("field", "value"), - [ - ("source_git_sha", "c" * 40), - ("depth_logical_release_sha256", "d" * 64), - ("model", "Qwen/wrong"), - ("renderer", "wrong_renderer"), - ], -) -def test_selection_rejects_report_provenance_mismatch(field: str, value: str): - geometry = _geometry("sampler-a", 0.5) - geometry["provenance"][field] = value - with pytest.raises(ValueError, match=rf"provenance\.{field}"): - select_sft_checkpoint( - [("a-k1.json", geometry)], - [ - ( - "a-g4.json", - _behavior( - "sampler-a", - _probe( - collapsed=False, - nonconstant=0.5, - program_mean=2.0, - ), - ), - ) - ], - contract=CONTRACT, - ) - - -def test_selection_rejects_behavioral_probe_outside_exact_40_by_4_contract(): - behavior = _behavior( - "sampler-a", - _probe( - collapsed=False, - nonconstant=0.5, - program_mean=2.0, - ), - ) - behavior["provenance"]["task_count"] = 39 - behavior["provenance"]["attempt_count"] = 156 - with pytest.raises(ValueError, match="provenance.task_count=39"): - select_sft_checkpoint( - [("a-k1.json", _geometry("sampler-a", 0.5))], - [("a-g4.json", behavior)], - contract=CONTRACT, - ) - - -def test_selection_rejects_sampling_mismatch_between_report_and_recipe(): - behavior = _behavior( - "sampler-a", - _probe( - collapsed=False, - nonconstant=0.5, - program_mean=2.0, - ), - ) - behavior["provenance"]["sampling"]["temperature"] = 0.7 - behavior["temperature"] = 0.7 - with pytest.raises( - ValueError, - match=r"provenance\.sampling\.temperature", - ): - select_sft_checkpoint( - [("a-k1.json", _geometry("sampler-a", 0.5))], - [("a-g4.json", behavior)], - contract=CONTRACT, - ) - - -def test_selection_rejects_behavioral_task_set_not_bound_to_declared_level(): - behavior = _behavior( - "sampler-a", - _probe( - collapsed=False, - nonconstant=0.5, - program_mean=2.0, - ), - ) - behavior["provenance"]["task_ids_sha256"] = "e" * 64 - with pytest.raises(ValueError, match="provenance.task_ids_sha256"): - select_sft_checkpoint( - [("a-k1.json", _geometry("sampler-a", 0.5))], - [("a-g4.json", behavior)], - contract=CONTRACT, - ) - - -def test_selection_recomputes_behavioral_collapse_verdict_from_groups(): - probe = _probe( - collapsed=True, - nonconstant=0.0, - program_mean=1.0, - ) - probe["collapse_detected"] = False - with pytest.raises( - ValueError, - match="behavioral_probe.collapse_detected=False; expected True", - ): - select_sft_checkpoint( - [("a-k1.json", _geometry("sampler-a", 0.5))], - [("a-g4.json", _behavior("sampler-a", probe))], - contract=CONTRACT, - ) diff --git a/rl/track_a/tests/test_curriculum.py b/rl/track_a/tests/test_curriculum.py index 7f3a61a6..c208a349 100644 --- a/rl/track_a/tests/test_curriculum.py +++ b/rl/track_a/tests/test_curriculum.py @@ -18,10 +18,6 @@ pytest.importorskip("tinker", reason="Tinker SDK env (rl/requirements-lock.txt) not installed") pytest.importorskip("chz", reason="chz env (rl/requirements-lock.txt) not installed") -from rl.track_a.recipe import ( - first_unmastered_level, - validate_curriculum_selection, -) from rl.track_a.tinker_data import TrackARLDataset @@ -107,52 +103,6 @@ def test_balanced_effective_pass_never_wraps_or_duplicates(): balanced_effective_pass(tasks, seed=3, draws=len(tasks) + 1) -def test_first_unmastered_and_replay_contract_are_computed(): - report = { - "levels_summary": { - level: { - "mean_iou_at_1": 0.9, - "pure_executable_rate": 0.99, - } - for level in ("L0", "L1", "L2", "L3", "L4") - } - } - report["levels_summary"]["L2"]["mean_iou_at_1"] = 0.7 - recipe = { - "rl": { - "curriculum": { - "minimum_iou": 0.8, - "minimum_pure_executable_rate": 0.95, - } - } - } - assert first_unmastered_level( - report, - minimum_iou=0.8, - minimum_pure_executable_rate=0.95, - ) == "L2" - assert validate_curriculum_selection( - recipe, - report, - current_level="L2", - replay_levels="L0,L1", - ) == ("L2", ("L0", "L1")) - with pytest.raises(ValueError, match="computed first unmastered"): - validate_curriculum_selection( - recipe, - report, - current_level="L3", - replay_levels="L0,L1,L2", - ) - with pytest.raises(ValueError, match="exactly the earlier levels"): - validate_curriculum_selection( - recipe, - report, - current_level="L2", - replay_levels="L1", - ) - - def test_rl_five_step_cycle_is_exactly_eighty_twenty(): current = [_task(index, "L2", f"current-{index}") for index in range(16)] replay = [ diff --git a/rl/track_a/tests/test_env_shaped_reward.py b/rl/track_a/tests/test_env_shaped_reward.py new file mode 100644 index 00000000..aeb9103e --- /dev/null +++ b/rl/track_a/tests/test_env_shaped_reward.py @@ -0,0 +1,79 @@ +"""The Env pays the shaped reward with the task's own rectangle baseline.""" + +from __future__ import annotations + +import asyncio + +import pytest + +pytest.importorskip("tinker", reason="Tinker SDK env not installed") +pytest.importorskip("chz", reason="chz env not installed") + +from rl.common.baselines import cached_rect_baseline +from rl.common.contracts import task_from_row +from rl.common.evaluator import Attribution, EvaluationResult, EvaluationStatus +from rl.track_a import tinker_data +from rl.track_a.reward import norm_excess, reward_for_policy +from rl.track_a.tinker_data import TrackAEnv + + +class _Termination: + is_clean = True + + +class _StubTokenizer: + def decode(self, tokens): + return "```python\nimport gdsfactory as gf\n```" + + +class _StubRenderer: + tokenizer = _StubTokenizer() + + def parse_response(self, tokens): + return ( + { + "role": "assistant", + "content": [{"type": "text", "text": self.tokenizer.decode(tokens)}], + }, + _Termination(), + ) + + +class _StubBatcher: + def __init__(self, result: EvaluationResult) -> None: + self._result = result + + async def submit(self, reference, completion): + return self._result + + +def test_env_step_pays_shaped_reward_with_task_baseline(core_row, monkeypatch): + task = task_from_row(core_row) + crafted = EvaluationResult( + EvaluationStatus.OK, + Attribution.MODEL, + iou=0.62, + dice=0.7, + metrics={"chamfer_boundary_um": 1.5}, + ) + monkeypatch.setattr(tinker_data, "_batcher", lambda: _StubBatcher(crafted)) + env = TrackAEnv(task, _StubRenderer(), 1440, reward_policy="shaped_v3b") + step = asyncio.run(env.step([1, 2, 3])) + baseline = cached_rect_baseline(task.reference) + expected = reward_for_policy(crafted, policy="shaped_v3b", baseline=baseline) + assert step.reward == pytest.approx(expected) + assert step.reward > 0.05 + assert step.metrics["iou_n"] == pytest.approx( + norm_excess(0.62, baseline.iou_rect) + ) + assert "teacher_nb" in step.metrics and "degenerate" in step.metrics + + +def test_env_step_zero_for_model_failure_under_shaped_policy(core_row, monkeypatch): + task = task_from_row(core_row) + crafted = EvaluationResult(EvaluationStatus.RUNTIME_ERROR, Attribution.MODEL) + monkeypatch.setattr(tinker_data, "_batcher", lambda: _StubBatcher(crafted)) + env = TrackAEnv(task, _StubRenderer(), 1440, reward_policy="shaped_v3b") + step = asyncio.run(env.step([1, 2, 3])) + assert step.reward == 0.0 + assert step.metrics["iou_n"] == 0.0 diff --git a/rl/track_a/tests/test_launch_preflight.py b/rl/track_a/tests/test_launch_preflight.py deleted file mode 100644 index ee74e0df..00000000 --- a/rl/track_a/tests/test_launch_preflight.py +++ /dev/null @@ -1,312 +0,0 @@ -from __future__ import annotations - -import importlib.util -import json -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from rl.track_a.recipe import ( - file_sha256, - load_recipe, - validate_evaluation_cli, - validate_rl_cli, - validate_resolved_binding, - validate_sft_cli, -) -import rl.track_a.recipe as recipe_module - - -REPO_ROOT = Path(__file__).resolve().parents[3] -PREFLIGHT_PATH = REPO_ROOT / "rl" / "track_a" / "preflight.py" - - -def _preflight_module(): - spec = importlib.util.spec_from_file_location( - "pixcell_track_a_preflight_test", - PREFLIGHT_PATH, - ) - assert spec is not None and spec.loader is not None - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -def test_paid_launch_clean_gate_includes_untracked_files(monkeypatch, tmp_path): - module = _preflight_module() - calls: list[list[str]] = [] - - def fake_check_output(command, **_kwargs): - calls.append(command) - if command[:2] == ["git", "rev-parse"]: - return "abc123\n" - return "?? rl/new_training_code.py\n" - - monkeypatch.setattr(module.subprocess, "check_output", fake_check_output) - - state = module._git_state(tmp_path, require_clean=False) - assert state == {"head": "abc123", "clean": False} - assert ["git", "status", "--porcelain", "--untracked-files=all"] in calls - - with pytest.raises(RuntimeError, match="including untracked files"): - module._git_state(tmp_path, require_clean=True) - - -def _canonical_sft_config() -> SimpleNamespace: - recipe = load_recipe(REPO_ROOT) - return SimpleNamespace( - confirm_spend=recipe["launch"]["confirmation_tokens"]["sft"], - dataset_root=str(REPO_ROOT / "dataset"), - model_name=recipe["model"]["name"], - renderer_name=recipe["model"]["renderer"], - lora_rank=recipe["model"]["lora_rank"], - learning_rate=recipe["sft"]["learning_rate"], - lr_schedule=recipe["sft"]["tinker_lr_schedule"], - num_epochs=recipe["sft"]["effective_passes"], - batch_size=recipe["sft"]["batch_size"], - max_length=recipe["sft"]["max_sequence_tokens"], - max_image=recipe["model"]["max_image_long_edge"], - save_every=recipe["sft"]["save_every_steps"], - eval_every=recipe["sft"]["evaluate_every_steps"], - max_steps=recipe["sft"]["max_steps"], - log_path=str(REPO_ROOT / "rl" / "track_a" / "runs" / "test-run"), - ) - - -@pytest.mark.parametrize( - ("name", "value"), - [ - ("dataset_root", "/tmp/other-dataset"), - ("model_name", "another/model"), - ("renderer_name", "another-renderer"), - ("max_length", 1024), - ("batch_size", 1), - ("max_steps", 1), - ], -) -def test_paid_sft_rejects_preflight_launch_mismatches(name, value): - cfg = _canonical_sft_config() - setattr(cfg, name, value) - with pytest.raises(ValueError, match=name): - validate_sft_cli(REPO_ROOT, cfg) - - -def test_resolved_launch_is_revalidated_in_preflight(): - binding = validate_sft_cli(REPO_ROOT, _canonical_sft_config()) - assert validate_resolved_binding(REPO_ROOT, binding) == binding - tampered = json.loads(json.dumps(binding)) - tampered["max_length"] = 512 - with pytest.raises(ValueError, match="max_length"): - validate_resolved_binding(REPO_ROOT, tampered) - - -def _mastery_report(sampler_path: str) -> dict: - mastered = { - "mean_iou_at_1": 0.9, - "pure_executable_rate": 1.0, - } - unmastered = { - "mean_iou_at_1": 0.5, - "pure_executable_rate": 1.0, - } - return { - "sampler_path": sampler_path, - "levels_summary": { - "L0": mastered, - "L1": unmastered, - "L2": unmastered, - "L3": unmastered, - "L4": unmastered, - }, - } - - -def _canonical_rl_config(mastery_path: Path) -> SimpleNamespace: - recipe = load_recipe(REPO_ROOT) - sampler = "tinker://run-id:train:0/sampler_weights/000014" - return SimpleNamespace( - confirm_spend=recipe["launch"]["confirmation_tokens"]["rl"], - dataset_root=str(REPO_ROOT / "dataset"), - model_name=recipe["model"]["name"], - renderer_name=recipe["model"]["renderer"], - lora_rank=recipe["model"]["lora_rank"], - group_size=recipe["rl"]["group_size"], - groups_per_batch=recipe["rl"]["groups_per_batch"], - learning_rate=recipe["rl"]["learning_rate"], - max_tokens=recipe["rl"]["max_output_tokens"], - temperature=recipe["rl"]["temperature"], - max_steps=recipe["rl"]["stage_max_steps"], - eval_every=recipe["rl"]["evaluate_every_steps"], - save_every=recipe["rl"]["save_every_steps"], - log_path=str(REPO_ROOT / "rl" / "track_a" / "runs" / "test-rl"), - loss_fn=recipe["rl"]["loss_fn"], - load_checkpoint_path=sampler.replace("/sampler_weights/", "/weights/"), - mastery_report=str(mastery_path), - current_level="L1", - replay_levels="L0", - ) - - -def test_rl_binding_pins_checkpoint_and_mastery_bytes(tmp_path): - sampler = "tinker://run-id:train:0/sampler_weights/000014" - mastery_path = tmp_path / "mastery.json" - mastery_path.write_text( - json.dumps(_mastery_report(sampler)), - encoding="utf-8", - ) - cfg = _canonical_rl_config(mastery_path) - - binding = validate_rl_cli( - REPO_ROOT, - cfg, - mastery_report=_mastery_report(sampler), - ) - - assert binding["load_checkpoint_path"] == ( - "tinker://run-id:train:0/weights/000014" - ) - assert binding["mastery_report"] == str(mastery_path.resolve()) - assert binding["mastery_report_sha256"] == file_sha256(mastery_path) - - cfg.load_checkpoint_path = "tinker://other:train:0/weights/000014" - with pytest.raises(ValueError, match="load_checkpoint_path"): - validate_rl_cli( - REPO_ROOT, - cfg, - mastery_report=_mastery_report(sampler), - ) - - -def test_preflight_rejects_mastery_report_changed_after_binding(tmp_path): - sampler = "tinker://run-id:train:0/sampler_weights/000014" - mastery_path = tmp_path / "mastery.json" - mastery_path.write_text( - json.dumps(_mastery_report(sampler)), - encoding="utf-8", - ) - cfg = _canonical_rl_config(mastery_path) - binding = validate_rl_cli( - REPO_ROOT, - cfg, - mastery_report=_mastery_report(sampler), - ) - binding["candidate_sandbox"] = { - "runtime_path": "/usr/bin/docker", - "daemon_endpoint": "unix:///tmp/docker.sock", - "image_ref": "sha256:" + "a" * 64, - "image_id": "sha256:" + "a" * 64, - "workspace_root": "/tmp/evaluator", - } - mastery_path.write_text( - json.dumps({**_mastery_report(sampler), "changed": True}), - encoding="utf-8", - ) - - with pytest.raises(ValueError, match="mastery_report_sha256"): - validate_resolved_binding(REPO_ROOT, binding) - - -def test_preflight_recomputes_first_unmastered_level( - monkeypatch, - tmp_path, -): - import rl.track_a.evaluation_contract as evaluation_contract - - sampler = "tinker://run-id:train:0/sampler_weights/000014" - mastery_path = tmp_path / "mastery.json" - mastery_path.write_text( - json.dumps(_mastery_report(sampler)), - encoding="utf-8", - ) - binding = validate_rl_cli( - REPO_ROOT, - _canonical_rl_config(mastery_path), - mastery_report=_mastery_report(sampler), - ) - boundary = SimpleNamespace( - runtime_path=Path("/usr/bin/docker"), - daemon_endpoint="unix:///tmp/docker.sock", - image_ref="sha256:" + "a" * 64, - image_id="sha256:" + "a" * 64, - workspace_root=Path("/tmp/evaluator"), - ) - binding["candidate_sandbox"] = { - "runtime_path": str(boundary.runtime_path), - "daemon_endpoint": boundary.daemon_endpoint, - "image_ref": boundary.image_ref, - "image_id": boundary.image_id, - "workspace_root": str(boundary.workspace_root), - } - binding["current_level"] = "L2" - binding["replay_levels"] = ["L0", "L1"] - monkeypatch.setattr( - evaluation_contract, - "build_checkpoint_selection_contract", - lambda *_args: object(), - ) - monkeypatch.setattr( - evaluation_contract, - "validate_evaluation_report", - lambda *_args, **_kwargs: None, - ) - monkeypatch.setattr( - recipe_module, - "require_execution_boundary", - lambda: boundary, - ) - - with pytest.raises(ValueError, match="first unmastered level L1"): - validate_resolved_binding(REPO_ROOT, binding) - - -def _canonical_evaluation_args(output: Path) -> SimpleNamespace: - recipe = load_recipe(REPO_ROOT) - return SimpleNamespace( - confirm_spend=recipe["launch"]["confirmation_tokens"]["evaluation"], - dataset_root=REPO_ROOT / "dataset", - model_name=recipe["model"]["name"], - renderer_name=recipe["model"]["renderer"], - sampler_path="tinker://run-id:train:0/sampler_weights/000014", - behavioral_probe=False, - levels="", - rows=0, - k=recipe["checkpoint_evaluation"]["geometry_k"], - temperature=recipe["rl"]["temperature"], - max_tokens=recipe["rl"]["max_output_tokens"], - concurrency=recipe["checkpoint_evaluation"]["sampling_concurrency"], - evaluator_workers=recipe["checkpoint_evaluation"]["evaluator_workers"], - seed=recipe["checkpoint_evaluation"]["task_selection_seed"], - include_records=recipe["checkpoint_evaluation"]["include_records"], - output=output, - ) - - -def test_evaluation_binding_pins_sampler_and_new_contained_output( - monkeypatch, - tmp_path, -): - monkeypatch.setattr(recipe_module, "RUNS_RELATIVE_PATH", tmp_path) - output = tmp_path / "checkpoint.json" - args = _canonical_evaluation_args(output) - - binding = validate_evaluation_cli(REPO_ROOT, args) - - assert binding["sampler_path"] == args.sampler_path - assert binding["output"] == str(output.resolve()) - - output.write_text("do not overwrite", encoding="utf-8") - with pytest.raises(ValueError, match="must not already exist"): - validate_evaluation_cli(REPO_ROOT, args) - - -def test_evaluation_output_must_stay_in_report_directory( - monkeypatch, - tmp_path, -): - report_root = tmp_path / "reports" - monkeypatch.setattr(recipe_module, "RUNS_RELATIVE_PATH", report_root) - args = _canonical_evaluation_args(tmp_path / "outside.json") - - with pytest.raises(ValueError, match="JSON child"): - validate_evaluation_cli(REPO_ROOT, args) diff --git a/rl/track_a/tests/test_output_reward.py b/rl/track_a/tests/test_output_reward.py index 7610234d..84a4bc2f 100644 --- a/rl/track_a/tests/test_output_reward.py +++ b/rl/track_a/tests/test_output_reward.py @@ -149,3 +149,106 @@ async def submit(_reference, _completion): assert sample_records[0]["sampling"]["raw_token_ids"] == [1, 2] assert sample_records[0]["program"] == "print('sample')" assert evaluated_records == [] + + +def _ok(iou: float, dice: float, chamfer: float | None) -> EvaluationResult: + return EvaluationResult( + EvaluationStatus.OK, + Attribution.MODEL, + iou=iou, + dice=dice, + metrics={"chamfer_boundary_um": chamfer}, + ) + + +def _baseline( + iou_rect: float = 0.5, + dice_rect: float = 0.6, + chamfer_rect: float | None = 4.0, + diag_um: float = 20.0, +): + from rl.common.baselines import TaskBaseline + + return TaskBaseline( + iou_rect=iou_rect, + dice_rect=dice_rect, + chamfer_rect_boundary_um=chamfer_rect, + diag_um=diag_um, + ) + + +def test_shaped_normalizes_iou_against_rectangle(): + # dice at its rect value and chamfer at the rect chamfer: only iou_n pays. + baseline = _baseline(iou_rect=0.5, dice_rect=0.6, chamfer_rect=4.0) + result = _ok(iou=0.75, dice=0.6, chamfer=4.0) + reward = reward_for_policy(result, policy="shaped_v3b", baseline=baseline) + assert reward == pytest.approx(0.05 + 0.40 * ((0.75 - 0.5) / 0.5)) + + +def test_shaped_rect_equivalent_program_earns_exactly_the_floor(): + baseline = _baseline(iou_rect=0.5, dice_rect=0.6, chamfer_rect=4.0) + result = _ok(iou=0.5, dice=0.6, chamfer=4.0) + assert reward_for_policy( + result, policy="shaped_v3b", baseline=baseline + ) == pytest.approx(0.05) + + +def test_shaped_degenerate_target_scores_raw(): + import math as _math + + baseline = _baseline(iou_rect=0.96, dice_rect=0.97, chamfer_rect=0.1, diag_um=20.0) + result = _ok(iou=0.9, dice=0.92, chamfer=0.5) + tau = 0.05 * 20.0 + expected = 0.05 + 0.40 * 0.9 + 0.15 * 0.92 + 0.40 * _math.exp(-0.5 / tau) + assert reward_for_policy( + result, policy="shaped_v3b", baseline=baseline + ) == pytest.approx(expected) + + +def test_shaped_teacher_denominator_floor(): + import math as _math + + # tau = 1.0 (diag 20); t2b = 0.9, t2b_rect = 0.8 -> 0.1 / max(0.2, 0.3) + baseline = _baseline( + iou_rect=0.5, + dice_rect=0.6, + chamfer_rect=-_math.log(0.8), + diag_um=20.0, + ) + result = _ok(iou=0.5, dice=0.6, chamfer=-_math.log(0.9)) + assert reward_for_policy( + result, policy="shaped_v3b", baseline=baseline + ) == pytest.approx(0.05 + 0.40 * (0.1 / 0.3)) + + +def test_shaped_model_failures_are_zero_and_faults_are_masked(): + baseline = _baseline() + crash = EvaluationResult(EvaluationStatus.RUNTIME_ERROR, Attribution.MODEL) + rejected = EvaluationResult(EvaluationStatus.SOURCE_REJECTED, Attribution.MODEL) + fault = EvaluationResult( + EvaluationStatus.MEASUREMENT_ERROR, Attribution.EVALUATOR, retryable=True + ) + assert reward_for_policy(crash, policy="shaped_v3b", baseline=baseline) == 0.0 + assert reward_for_policy(rejected, policy="shaped_v3b", baseline=baseline) == 0.0 + assert reward_for_policy(fault, policy="shaped_v3b", baseline=baseline) is None + + +def test_shaped_missing_chamfer_scores_teacher_zero(): + baseline = _baseline(iou_rect=0.5, dice_rect=0.6, chamfer_rect=4.0) + result = _ok(iou=0.5, dice=0.6, chamfer=None) + assert reward_for_policy( + result, policy="shaped_v3b", baseline=baseline + ) == pytest.approx(0.05) + + +def test_shaped_requires_a_baseline(): + with pytest.raises(ValueError): + reward_for_policy(_ok(0.5, 0.5, 1.0), policy="shaped_v3b") + + +def test_shaped_perfect_program_reaches_one(): + baseline = _baseline(iou_rect=0.3, dice_rect=0.4, chamfer_rect=50.0, diag_um=20.0) + result = _ok(iou=1.0, dice=1.0, chamfer=0.0) + assert reward_for_policy( + result, policy="shaped_v3b", baseline=baseline + ) == pytest.approx(1.0) diff --git a/rl/track_a/tests/test_prompt_contract.py b/rl/track_a/tests/test_prompt_contract.py index f0330933..e87b04c9 100644 --- a/rl/track_a/tests/test_prompt_contract.py +++ b/rl/track_a/tests/test_prompt_contract.py @@ -1,7 +1,6 @@ from __future__ import annotations import importlib.util -import json import re from pathlib import Path @@ -39,7 +38,6 @@ def test_prompt_is_one_blind_image_first_turn(core_row): forbidden = ( "DISPLAY NOTE", "magnification x:y", - "px/um", "target_image", "target raster", "candidate raster", @@ -143,19 +141,5 @@ def test_catalog_dataset_and_runtime_whitelists_are_identical(): assert "bezier" in ALLOWED_COMPONENTS -def test_committed_config_binds_contract_and_is_spend_locked(): - path = REPO_ROOT / "rl" / "track_a" / "config.json" - config = json.loads(path.read_text()) - assert config["contract_version"] == CONTRACT_VERSION - assert CONTRACT_VERSION == "pixcell-direct-reconstruction-v2" - assert config["model"]["name"] == "Qwen/Qwen3.6-35B-A3B" - assert config["model"]["renderer"] == "qwen3_5_disable_thinking" - assert config["model"]["lora_rank"] == 32 - assert config["sft"]["dataset_repo_id"] == "qpaig-mit/pixcell" - assert config["sft"]["dataset_configuration"] == "depth" - assert config["sft"]["verifier_configuration"] == "references" - assert config["sft"]["dataset_revision"] == "v2.0.0" - assert config["sft"]["max_sequence_tokens"] == 6144 - assert config["rl"]["group_size"] == 4 - assert config["rl"]["groups_per_batch"] == 8 - assert config["launch"]["paid_launch_enabled"] is False +def test_contract_version_is_pinned(): + assert CONTRACT_VERSION == "pixcell-direct-reconstruction-v3" diff --git a/rl/track_a/tests/test_run_plan.py b/rl/track_a/tests/test_run_plan.py new file mode 100644 index 00000000..62ca986b --- /dev/null +++ b/rl/track_a/tests/test_run_plan.py @@ -0,0 +1,200 @@ +"""Campaign plan construction and probe determinism (no network).""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +pytest.importorskip("tinker", reason="Tinker SDK env not installed") +pytest.importorskip("chz", reason="chz env not installed") + +from rl.track_a.probe_eval import select_probe_tasks +from rl.track_a.run import RUN_PLANS, _sft_gate, build_stages + + +REPO_ROOT = Path(__file__).resolve().parents[3] + + +def test_run_a_plan_is_sft_then_five_rl_stages(tmp_path): + stages = build_stages( + RUN_PLANS["A"], run_root=tmp_path, wandb_project="pixcell-rl" + ) + assert [stage.name for stage in stages] == [ + "sft-l0", + "rl-l0", + "rl-l1", + "rl-l2", + "rl-l3", + "rl-l4", + ] + sft = stages[0] + assert sft.args["levels"] == "L0" + assert sft.args["num_epochs"] == "2" + assert sft.args["renderer_name"] == "qwen3_5_disable_thinking" + last = stages[-1] + assert last.args["current_level"] == "L4" + assert last.args["max_steps"] == "30" + assert last.args["group_size"] == "8" + assert last.args["groups_per_batch"] == "8" + assert last.args["reward_policy"] == "shaped_v3b" + assert last.args["wandb_name"] == "runA-l0sft-rl-rl-l4" + + +def test_think_pair_uses_thinking_renderer_and_budget(tmp_path): + for key in ("C", "D"): + stages = build_stages( + RUN_PLANS[key], run_root=tmp_path, wandb_project="pixcell-rl" + ) + rl_stage = [stage for stage in stages if stage.name == "rl-l0"][0] + assert rl_stage.args["renderer_name"] == "qwen3_5" + assert rl_stage.args["max_tokens"] == "60000" + assert RUN_PLANS["C"].sft is False + assert RUN_PLANS["D"].sft is True + + +def test_smoke_plan_is_two_steps_two_groups(tmp_path): + stages = build_stages( + RUN_PLANS["B"], run_root=tmp_path, wandb_project="pixcell-rl", smoke=True + ) + assert [stage.name for stage in stages] == ["rl-l0"] + assert stages[0].args["max_steps"] == "2" + assert stages[0].args["groups_per_batch"] == "2" + assert stages[0].args["validation_canaries"] == "8" + + +def test_probe_selection_is_deterministic_and_stratified(): + first = select_probe_tasks(REPO_ROOT / "dataset", per_level=4) + second = select_probe_tasks(REPO_ROOT / "dataset", per_level=4) + assert [task.sampler.opaque_id for task in first] == [ + task.sampler.opaque_id for task in second + ] + assert len(first) == 20 + by_level: dict[str, int] = {} + for task in first: + by_level[task.sampler.level] = by_level.get(task.sampler.level, 0) + 1 + assert by_level == {"L0": 4, "L1": 4, "L2": 4, "L3": 4, "L4": 4} + + +def test_sft_gate_compares_l0_against_base(): + base = {"per_level": {"L0": {"executable_rate": 0.9, "mean_iou": 0.30}}} + good = {"per_level": {"L0": {"executable_rate": 0.9, "mean_iou": 0.45}}} + weak_exec = {"per_level": {"L0": {"executable_rate": 0.5, "mean_iou": 0.60}}} + no_gain = {"per_level": {"L0": {"executable_rate": 0.9, "mean_iou": 0.25}}} + assert _sft_gate(base, good, "L0", relative=False)[0] is True + assert _sft_gate(base, weak_exec, "L0", relative=False)[0] is False + assert _sft_gate(base, no_gain, "L0", relative=False)[0] is False + + +def test_chaining_uses_training_weights_and_probes_use_sampler(tmp_path): + from rl.track_a.run import _last_sampler_path, _last_weights_path + + record = ( + '{"name": "final", "batch": 2, ' + '"state_path": "tinker://m:train:0/weights/final", ' + '"sampler_path": "tinker://m:train:0/sampler_weights/final"}\n' + ) + (tmp_path / "checkpoints.jsonl").write_text(record, encoding="utf-8") + assert _last_weights_path(str(tmp_path)) == "tinker://m:train:0/weights/final" + assert ( + _last_sampler_path(str(tmp_path)) + == "tinker://m:train:0/sampler_weights/final" + ) + + +def test_fork_seeds_l0_and_drops_sft(tmp_path): + import json as _json + + from rl.track_a.run import _fork_l0_stage + + source = tmp_path / "src-run" + (source / "stages" / "rl-l0").mkdir(parents=True) + (source / "stages" / "rl-l0" / "DONE").write_text("ok\n") + (source / "stages" / "rl-l0" / "checkpoints.jsonl").write_text( + '{"name": "000030", "batch": 30, "state_path": "tinker://m/weights/000030", ' + '"sampler_path": "tinker://m/sampler_weights/000030"}\n' + ) + (source / "stages" / "rl-l0" / "metrics.jsonl").write_text('{"progress/batch": 29}\n') + target = tmp_path / "dst-run" + _fork_l0_stage(source, target) + assert (target / "stages" / "rl-l0" / "checkpoints.jsonl").exists() + assert not (target / "stages" / "rl-l0" / "DONE").exists() + fork = _json.loads((target / "provenance" / "fork.json").read_text()) + assert "rl-l0" in fork["rl_l0_forked_from"] + + +def test_run_e_interleaves_l1_sft_after_forked_l0(tmp_path): + stages = build_stages( + RUN_PLANS["E"], run_root=tmp_path, wandb_project="pixcell-rl" + ) + assert [stage.name for stage in stages] == [ + "sft-l0", + "rl-l0", + "sft-l1", + "rl-l1", + "rl-l2", + "rl-l3", + "rl-l4", + ] + sft_l1 = stages[2] + assert sft_l1.args["levels"] == "L1" + assert sft_l1.args["num_epochs"] == "1" + assert sft_l1.sft_level == "L1" + assert RUN_PLANS["E"].fork_from == "runA-l0sft-rl" + + +def test_run_f_uses_fifty_percent_backward_mix(tmp_path): + stages = build_stages( + RUN_PLANS["F"], run_root=tmp_path, wandb_project="pixcell-rl" + ) + rl_l1 = [stage for stage in stages if stage.name == "rl-l1"][0] + assert rl_l1.args["replay_groups_per_step"] == "4" + assert not any(stage.name == "sft-l1" for stage in stages) + + +def test_run_g_is_the_full_ladder(tmp_path): + stages = build_stages( + RUN_PLANS["G"], run_root=tmp_path, wandb_project="pixcell-rl" + ) + names = [stage.name for stage in stages] + for level in ("l1", "l2", "l3", "l4"): + assert f"sft-{level}" in names and names.index(f"sft-{level}") == names.index(f"rl-{level}") - 1 + + +def test_fork_stages_through_seeds_prefix_and_reference(tmp_path): + import json as _json + + from rl.track_a.run import _fork_stages_through + + source = tmp_path / "src-run" + for name in ("sft-l0", "rl-l0"): + d = source / "stages" / name + d.mkdir(parents=True) + (d / "DONE").write_text("ok\n") + (d / "checkpoints.jsonl").write_text( + '{"name": "final", "batch": 30, "state_path": "tinker://m/weights/final", ' + '"sampler_path": "tinker://m/sampler_weights/final"}\n' + ) + (d / "metrics.jsonl").write_text("{}\n") + (source / "probes").mkdir() + (source / "probes" / "rl-l0.json").write_text('{"per_level": {"L1": {"executable_rate": 0.6, "mean_iou": 0.1}}}') + target = tmp_path / "dst-run" + _fork_stages_through(source, target, "rl-l0") + assert (target / "stages" / "sft-l0" / "DONE").exists() + assert (target / "stages" / "rl-l0" / "DONE").exists() + origin = _json.loads((target / "probes" / "fork-origin.json").read_text()) + assert origin["per_level"]["L1"]["executable_rate"] == 0.6 + fork = _json.loads((target / "provenance" / "fork.json").read_text()) + assert fork["stages"] == ["sft-l0", "rl-l0"] + + +def test_relative_gate_allows_small_exec_loss_but_demands_iou_gain(): + from rl.track_a.run import _sft_gate + + reference = {"per_level": {"L1": {"executable_rate": 0.60, "mean_iou": 0.10}}} + better = {"per_level": {"L1": {"executable_rate": 0.55, "mean_iou": 0.16}}} + lost_exec = {"per_level": {"L1": {"executable_rate": 0.40, "mean_iou": 0.30}}} + no_gain = {"per_level": {"L1": {"executable_rate": 0.70, "mean_iou": 0.08}}} + assert _sft_gate(reference, better, "L1", relative=True)[0] is True + assert _sft_gate(reference, lost_exec, "L1", relative=True)[0] is False + assert _sft_gate(reference, no_gain, "L1", relative=True)[0] is False diff --git a/rl/track_a/tinker_data.py b/rl/track_a/tinker_data.py index 00634a36..46ff35d8 100644 --- a/rl/track_a/tinker_data.py +++ b/rl/track_a/tinker_data.py @@ -38,6 +38,7 @@ ) from tinker_cookbook.tokenizer_utils import get_tokenizer +from rl.common.baselines import cached_rect_baseline from rl.common.batcher import AsyncEvaluationBatcher from rl.common.contracts import TaskRecord from rl.common.dataset_io import load_tasks @@ -46,16 +47,18 @@ from rl.track_a.curriculum import ( balanced_effective_pass, deterministic_rl_batch, + expected_replay_levels, hierarchical_sft_weights, + normalize_level, + normalize_replay_levels, stratified_physical_pass, ) from rl.common.preprocess import model_image from rl.common.prompt import build_prompt_text -from rl.track_a.reward import REWARD_POLICIES, reward_for_policy -from rl.track_a.recipe import ( - expected_replay_levels, - normalize_level, - normalize_replay_levels, +from rl.track_a.reward import ( + REWARD_POLICIES, + reward_for_policy, + shaped_components, ) @@ -93,6 +96,9 @@ def _renderer( effort: float | None = None, ): if model_name == "thinkingmachines/Inkling": + # Re-scoped in for the bounded L4 RL-only arm (user direction, + # 2026-07-29): RL under the shaped reward was designed but never + # actually run on Inkling in either prior campaign. if renderer_name != "tml_v0" or effort is None: raise ValueError( "Inkling requires renderer_name='tml_v0' and explicit effort" @@ -108,7 +114,7 @@ def _renderer( model_name=model_name, ) if effort is not None: - raise ValueError("effort is supported only by Inkling") + raise ValueError("effort is not a Qwen sampling knob") tokenizer = get_tokenizer(model_name) processor = get_image_processor(model_name) return get_renderer( @@ -122,10 +128,27 @@ def _renderer( def _batcher() -> AsyncEvaluationBatcher: global _EVALUATION_BATCHER if _EVALUATION_BATCHER is None: + import os + evaluator = PixCellEvaluator( - max_workers=8, - evaluator_retries=1, - require_isolation=True, + # Several campaigns share one Docker daemon; transient container + # faults under contention must be retried, not abort a paid + # stage, and the wall timeout must leave headroom so contention + # slows a measurement instead of falsifying it as a model + # failure. Persistent evaluator faults still abort after the + # retries are exhausted — no biased zeros either way. + max_workers=int(os.environ.get("PIXCELL_EVAL_WORKERS", "8")), + evaluator_retries=3, + execution_timeout_seconds=float( + os.environ.get("PIXCELL_EVAL_TIMEOUT", "20") + ), + # The load-bearing safety layer is the pre-execution AST source + # gate plus subprocess rlimits — the tier the private library's + # entire recorded campaign ran on. The Docker tier remains + # available (PIXCELL_REQUIRE_ISOLATION=1) for record evals. + require_isolation=( + os.environ.get("PIXCELL_REQUIRE_ISOLATION", "0") == "1" + ), ) _EVALUATION_BATCHER = AsyncEvaluationBatcher( evaluator, @@ -216,6 +239,7 @@ class TrackASupervisedDatasetBuilder(SupervisedDatasetBuilder): levels: str = "" include_validation: bool = True schedule_seed: int = 0 + winner_labels_path: str = "" def __call__(self): renderer = _renderer(self.model_name, self.renderer_name) @@ -240,6 +264,10 @@ def __call__(self): if self.include_validation else [] ) + if self.winner_labels_path: + from rl.track_a.harvest import apply_winner_labels + + train = apply_winner_labels(train, self.winner_labels_path) return ( TrackASupervisedDataset( train, @@ -344,8 +372,17 @@ async def step(self, action, *, extra=None): } if self.sample_recorder is not None: self.sample_recorder(sample_record) + baseline = ( + cached_rect_baseline(self.task.reference) + if self.reward_policy == "shaped_v3b" + else None + ) result = await _batcher().submit(self.task.reference, completion) - reward = reward_for_policy(result, policy=self.reward_policy) + reward = reward_for_policy( + result, + policy=self.reward_policy, + baseline=baseline, + ) if reward is None: if result.attribution is Attribution.REFERENCE: raise BrokenReferenceError(result.error or result.status.value) @@ -367,24 +404,32 @@ async def step(self, action, *, extra=None): }, } ) + step_metrics = { + "iou": float(result.iou or 0.0), + "dice": float(result.dice or 0.0), + "program_valid": float(result.status is EvaluationStatus.OK), + "source_rejected": float( + result.status is EvaluationStatus.SOURCE_REJECTED + ), + "latency_seconds": float(result.latency_seconds), + "channel_parse_complete": float(termination.is_clean), + "length_capped": float( + getattr(extra, "stop_reason", None) == "length" + ), + } + if baseline is not None: + parts = shaped_components(result, baseline) + step_metrics.update( + iou_n=parts["iou_n"], + teacher_nb=parts["teacher_nb"], + degenerate=parts["degenerate"], + ) return StepResult( reward=reward, episode_done=True, next_observation=tinker.ModelInput.from_ints([]), next_stop_condition=[], - metrics={ - "iou": float(result.iou or 0.0), - "dice": float(result.dice or 0.0), - "program_valid": float(result.status is EvaluationStatus.OK), - "source_rejected": float( - result.status is EvaluationStatus.SOURCE_REJECTED - ), - "latency_seconds": float(result.latency_seconds), - "channel_parse_complete": float(termination.is_clean), - "length_capped": float( - getattr(extra, "stop_reason", None) == "length" - ), - }, + metrics=step_metrics, ) @@ -452,7 +497,9 @@ def __init__( reward_policy: str = "validity_floor", schedule_seed: int = 10_000, attempt_metadata: Mapping[str, object] | None = None, + replay_groups_per_step: int = 0, ) -> None: + self.replay_groups_per_step = replay_groups_per_step self.current_tasks = list(current_tasks) self.replay_tasks = list(replay_tasks) self.renderer = renderer @@ -480,6 +527,7 @@ def get_batch(self, index: int) -> Sequence[EnvGroupBuilder]: groups_per_batch=self.groups_per_batch, schedule_seed=self.schedule_seed, shuffle=self.shuffle, + replay_groups_per_step=self.replay_groups_per_step, ) return [ PixCellGroupBuilder( @@ -513,7 +561,7 @@ def _level_filter(tasks: list[TaskRecord], levels: str) -> list[TaskRecord]: class TrackARLDatasetBuilder(RLDatasetBuilder): dataset_root: str batch_size: int = 8 - group_size: int = 4 + group_size: int = 8 model_name: str = DEFAULT_MODEL renderer_name: str = DEFAULT_RENDERER model_effort: float | None = None @@ -523,10 +571,11 @@ class TrackARLDatasetBuilder(RLDatasetBuilder): validation_canaries: int = 40 attempt_recorder: Callable[[Mapping[str, object]], None] | None = None sample_recorder: Callable[[Mapping[str, object]], None] | None = None - reward_policy: str = "validity_floor" + reward_policy: str = "shaped_v3b" require_prefix_replay: bool = True schedule_seed: int = 10_000 attempt_metadata: Mapping[str, object] | None = None + replay_groups_per_step: int = 0 async def __call__(self): renderer = _renderer( @@ -591,6 +640,7 @@ async def __call__(self): reward_policy=self.reward_policy, schedule_seed=self.schedule_seed, attempt_metadata=self.attempt_metadata, + replay_groups_per_step=self.replay_groups_per_step, ), ( TrackARLDataset( diff --git a/rl/track_a/train_rl.py b/rl/track_a/train_rl.py index 32b08714..c043d49b 100644 --- a/rl/track_a/train_rl.py +++ b/rl/track_a/train_rl.py @@ -1,10 +1,15 @@ #!/usr/bin/env python3 -"""Spend-locked Track A G4 entrypoint.""" +"""Track A GRPO entrypoint — one curriculum stage. + +Single-turn blind reconstruction with the shaped v3-b reward. One invocation +trains one level for ``max_steps`` steps with the exact ordered replay prefix +(80/20 five-step cycle); the campaign runner chains stages by passing the +previous stage's final weights as ``load_checkpoint_path``. +""" from __future__ import annotations import asyncio -import json import os from pathlib import Path @@ -13,13 +18,8 @@ from tinker_cookbook import cli_utils from tinker_cookbook.rl.train import Config, main -from rl.common.isolation import IsolationError -from rl.track_a.evaluation_contract import ( - build_checkpoint_selection_contract, - validate_evaluation_report, -) -from rl.track_a.launch import bind_execution_boundary, run_launch_preflight -from rl.track_a.recipe import validate_rl_cli +from rl.track_a.curriculum import expected_replay_levels, normalize_level +from rl.track_a.launch import run_launch_preflight from rl.track_a.tinker_data import ( DEFAULT_MODEL, DEFAULT_RENDERER, @@ -31,6 +31,8 @@ _REPO_ROOT = Path(__file__).resolve().parents[2] _DATASET_ROOT = _REPO_ROOT / "dataset" +CONFIRM_SPEND = "PIXCELL_RUNS_V2" + @chz.chz class CLIConfig: @@ -38,12 +40,11 @@ class CLIConfig: dataset_root: str = str(_DATASET_ROOT) model_name: str = DEFAULT_MODEL renderer_name: str = DEFAULT_RENDERER + model_effort: float = -1.0 load_checkpoint_path: str = "" - mastery_report: str = "" - current_level: str = "" - replay_levels: str = "" + current_level: str = "L0" lora_rank: int = 32 - group_size: int = 4 + group_size: int = 8 groups_per_batch: int = 8 learning_rate: float = 1e-5 max_tokens: int = 4096 @@ -51,44 +52,36 @@ class CLIConfig: max_steps: int = 30 eval_every: int = 5 save_every: int = 5 + validation_canaries: int = 40 + reward_policy: str = "shaped_v3b" + replay_groups_per_step: int = 0 log_path: str = str(_TRACK_ROOT / "runs" / "track-a-rl") loss_fn: LossFnType = "importance_sampling" + wandb_project: str = "" + wandb_name: str = "" + skip_preflight: bool = False async def cli_main(cfg: CLIConfig) -> None: - if not cfg.mastery_report.strip(): - raise SystemExit("paid launch blocked: mastery_report is required") - try: - mastery_path = Path(cfg.mastery_report).expanduser().resolve(strict=True) - mastery_report = json.loads(mastery_path.read_text(encoding="utf-8")) - if not isinstance(mastery_report, dict): - raise ValueError("mastery report must be a JSON object") - report_contract = build_checkpoint_selection_contract( - _REPO_ROOT, - Path(cfg.dataset_root), - ) - validate_evaluation_report( - mastery_report, - contract=report_contract, - expected_role="checkpoint-geometry", - ) - resolved_launch = validate_rl_cli( - _REPO_ROOT, - cfg, - mastery_report=mastery_report, + from rl.track_a.transport import disable_pyqwest_transport + + disable_pyqwest_transport() + if cfg.confirm_spend != CONFIRM_SPEND: + raise SystemExit( + f"paid launch blocked: pass confirm_spend={CONFIRM_SPEND}" ) - resolved_launch = bind_execution_boundary(resolved_launch) - except ( - OSError, - json.JSONDecodeError, - RuntimeError, - ValueError, - IsolationError, - ) as exc: - raise SystemExit(f"paid launch blocked: {exc}") from exc if not os.environ.get("TINKER_API_KEY"): raise SystemExit("TINKER_API_KEY is not present") - run_launch_preflight(_REPO_ROOT, resolved_launch=resolved_launch) + current = normalize_level(cfg.current_level) + replay_levels = expected_replay_levels(current) + if not cfg.skip_preflight: + run_launch_preflight( + _REPO_ROOT, + dataset_root=Path(cfg.dataset_root), + model_name=cfg.model_name, + renderer_name=cfg.renderer_name, + require_sandbox=True, + ) config = Config( learning_rate=cfg.learning_rate, dataset_builder=TrackARLDatasetBuilder( @@ -97,15 +90,19 @@ async def cli_main(cfg: CLIConfig) -> None: group_size=cfg.group_size, model_name=cfg.model_name, renderer_name=cfg.renderer_name, + model_effort=cfg.model_effort if cfg.model_effort >= 0 else None, max_image=1440, - current_level=str(resolved_launch["current_level"]), - replay_levels=",".join(resolved_launch["replay_levels"]), + current_level=current, + replay_levels=",".join(replay_levels), + validation_canaries=cfg.validation_canaries, + reward_policy=cfg.reward_policy, + replay_groups_per_step=cfg.replay_groups_per_step, ), model_name=cfg.model_name, recipe_name="pixcell_track_a_rl", renderer_name=cfg.renderer_name, lora_rank=cfg.lora_rank, - load_checkpoint_path=str(resolved_launch["load_checkpoint_path"]), + load_checkpoint_path=cfg.load_checkpoint_path or None, max_tokens=cfg.max_tokens, temperature=cfg.temperature, log_path=cfg.log_path, @@ -119,12 +116,12 @@ async def cli_main(cfg: CLIConfig) -> None: # and must abort rather than silently dropping a biased group. rollout_error_tolerance=False, kl_penalty_coef=0.0, - num_groups_to_log=0, - ) - cli_utils.check_log_dir( - cfg.log_path, - behavior_if_exists="raise", + num_groups_to_log=2, + rollout_json_export=True, + wandb_project=cfg.wandb_project or None, + wandb_name=cfg.wandb_name or None, ) + cli_utils.check_log_dir(cfg.log_path, behavior_if_exists="resume") await main(config) diff --git a/rl/track_a/train_sft.py b/rl/track_a/train_sft.py index 5d982d84..0451abd8 100644 --- a/rl/track_a/train_sft.py +++ b/rl/track_a/train_sft.py @@ -1,5 +1,11 @@ #!/usr/bin/env python3 -"""Spend-locked Track A SFT entrypoint.""" +"""Track A SFT entrypoint. + +One stage of supervised fine-tuning over the frozen depth-v1 rows, with the +same blind prompt the RL env samples with (prompt identity is what makes the +checkpoint resumable by ``train_rl``). Level-filtered: the campaign's SFT +arms train on L0 only. +""" from __future__ import annotations @@ -12,7 +18,6 @@ from tinker_cookbook.supervised.train import Config, main from rl.track_a.launch import run_launch_preflight -from rl.track_a.recipe import validate_sft_cli from rl.track_a.tinker_data import ( DEFAULT_MODEL, DEFAULT_RENDERER, @@ -24,6 +29,8 @@ _REPO_ROOT = Path(__file__).resolve().parents[2] _DATASET_ROOT = _REPO_ROOT / "dataset" +CONFIRM_SPEND = "PIXCELL_RUNS_V2" + @chz.chz class CLIConfig: @@ -31,27 +38,46 @@ class CLIConfig: dataset_root: str = str(_DATASET_ROOT) model_name: str = DEFAULT_MODEL renderer_name: str = DEFAULT_RENDERER + load_checkpoint_path: str = "" + levels: str = "L0" + winner_labels_path: str = "" lora_rank: int = 32 - learning_rate: float = 5e-5 + learning_rate: float = 1e-4 lr_schedule: str = "linear" - num_epochs: int = 1 + num_epochs: int = 2 batch_size: int = 64 max_length: int = 6144 max_image: int = 1440 - save_every: int = 14 - eval_every: int = 14 - max_steps: int = 55 + save_every: int = 5 + eval_every: int = 5 log_path: str = str(_TRACK_ROOT / "runs" / "track-a-sft") + wandb_project: str = "" + wandb_name: str = "" + skip_preflight: bool = False async def cli_main(cfg: CLIConfig) -> None: - try: - resolved_launch = validate_sft_cli(_REPO_ROOT, cfg) - except ValueError as exc: - raise SystemExit(f"paid launch blocked: {exc}") from exc + from rl.track_a.transport import disable_pyqwest_transport + + disable_pyqwest_transport() + if cfg.confirm_spend != CONFIRM_SPEND: + raise SystemExit( + f"paid launch blocked: pass confirm_spend={CONFIRM_SPEND}" + ) if not os.environ.get("TINKER_API_KEY"): raise SystemExit("TINKER_API_KEY is not present") - run_launch_preflight(_REPO_ROOT, resolved_launch=resolved_launch) + if not cfg.skip_preflight: + run_launch_preflight( + _REPO_ROOT, + dataset_root=Path(cfg.dataset_root), + model_name=cfg.model_name, + renderer_name=cfg.renderer_name, + require_sandbox=False, + tokenize_sft=True, + sft_levels=cfg.levels, + max_length=cfg.max_length, + max_image=cfg.max_image, + ) config = Config( log_path=cfg.log_path, model_name=cfg.model_name, @@ -64,9 +90,12 @@ async def cli_main(cfg: CLIConfig) -> None: batch_size=cfg.batch_size, max_length=cfg.max_length, max_image=cfg.max_image, + levels=cfg.levels, + winner_labels_path=cfg.winner_labels_path, ), learning_rate=cfg.learning_rate, lr_schedule=cfg.lr_schedule, + load_checkpoint_path=cfg.load_checkpoint_path or None, num_epochs=cfg.num_epochs, lora_rank=cfg.lora_rank, adam_beta1=0.9, @@ -74,12 +103,10 @@ async def cli_main(cfg: CLIConfig) -> None: adam_eps=1e-8, save_every=cfg.save_every, eval_every=cfg.eval_every, - max_steps=cfg.max_steps, - ) - cli_utils.check_log_dir( - cfg.log_path, - behavior_if_exists="raise", + wandb_project=cfg.wandb_project or None, + wandb_name=cfg.wandb_name or None, ) + cli_utils.check_log_dir(cfg.log_path, behavior_if_exists="resume") await main(config) diff --git a/rl/track_a/transport.py b/rl/track_a/transport.py new file mode 100644 index 00000000..2c53b0dd --- /dev/null +++ b/rl/track_a/transport.py @@ -0,0 +1,25 @@ +"""Local kill-switch for the Tinker SDK's pyqwest transport. + +The SDK (0.22.7) defaults to a reqwest/rustls HTTP backend (``pyqwest``) +whose bundled trust store rejects the certificate chain this host currently +receives for ``tinker.thinkingmachines.dev`` (``invalid peer certificate: +UnknownIssuer`` against the Google Trust Services WE1 chain), while httpx's +certifi-based verification accepts it. The SDK's own kill-switch for this +transport is server-side only (``ClientConfigResponse.use_pyqwest_transport``), +so a client whose auth bootstrap dies on pyqwest can never be told to stop +using it. Verified empirically: the same auth POST succeeds through httpx +and curl in ~0.2s and fails through pyqwest with UnknownIssuer. + +``disable_pyqwest_transport()`` swaps the transport factory for one that +returns ``None`` — httpx then builds its default (working) transport. Call +it before any ``tinker.ServiceClient`` is constructed. Remove when the SDK +grows a client-side override or pyqwest validates this chain. +""" + +from __future__ import annotations + + +def disable_pyqwest_transport() -> None: + import tinker._base_client as base_client + + base_client._default_pyqwest_transport = lambda: None diff --git a/rl/track_b/README.md b/rl/track_b/README.md deleted file mode 100644 index 9a4a3513..00000000 --- a/rl/track_b/README.md +++ /dev/null @@ -1,16 +0,0 @@ -# Track B - -Track B is the advanced-policy path. Its first frozen use is the bounded -Inkling L4 branch in -[`representation_training_v1`](../studies/representation_training_v1/README.md). -The study owns the stage, reward, schedule, checkpoint, evaluation, and spend -contract; Track B owns only the Inkling-specific runtime bridge. - -`tml_shim.py` is the runtime bridge needed by that future launcher. It registers -native TMLv0 support only for `thinkingmachines/Inkling`, requires an explicit -programmatic thinking-effort value, keeps reasoning separate from final text, -and refuses every supervised rendering entrypoint. Importing it does not -register anything or create a Tinker client. - -The shared study launcher imports this bridge only for an Inkling RL stage. -Inkling remains RL-only and never receives an SFT answer serialization. diff --git a/rl/track_b/__init__.py b/rl/track_b/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/rl/track_b/tests/__init__.py b/rl/track_b/tests/__init__.py index 7d42b3e9..e69de29b 100644 --- a/rl/track_b/tests/__init__.py +++ b/rl/track_b/tests/__init__.py @@ -1 +0,0 @@ -"""Track B runtime tests.""" diff --git a/rl/track_c/README.md b/rl/track_c/README.md deleted file mode 100644 index d0e82874..00000000 --- a/rl/track_c/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Track C - -Track C is reserved for the third PixCell training protocol. Its implementation -will be added only after its model-visible inputs, optimization objective, -evaluation boundary, checkpoint policy, and spend controls are explicitly -approved. - -Until then, Track C has no launcher and imports no Track A policy.