diff --git a/arxiv/figures/fig_rl_gallery.pdf b/arxiv/figures/fig_rl_gallery.pdf deleted file mode 100644 index a809447a..00000000 Binary files a/arxiv/figures/fig_rl_gallery.pdf and /dev/null differ diff --git a/arxiv/figures/fig_rl_training.pdf b/arxiv/figures/fig_rl_training.pdf index 1aaa7e21..2555c0d4 100644 Binary files a/arxiv/figures/fig_rl_training.pdf and b/arxiv/figures/fig_rl_training.pdf differ diff --git a/arxiv/main.tex b/arxiv/main.tex index c758cb9b..6fca563d 100644 --- a/arxiv/main.tex +++ b/arxiv/main.tex @@ -68,14 +68,13 @@ SOI full-wave simulation, producing symmetric propagation and balanced outputs. Separate full-wave audits expose the quantitative limits of inexpensive analytic models. Finally, the same executable verifier supplies a -training reward and dataset through which a Qwen3.6-35B-A3B model trained with -low-rank adaptation (LoRA) and reinforcement learning (RL) reaches mean -single-attempt and best-of-eight IoU values of $x$ and $y$ on eight -training-excluded paper figures, while a raw-base group relative policy -optimization (GRPO) lineage reaches $x$ and $y$ without supervised -demonstrations. These results therefore establish a controlled framework for -measuring, retargeting, and improving visual-to-parametric photonic component -design. +training reward and dataset used to train a Qwen3.6-35B-A3B model from its raw +base checkpoint with low-rank adaptation (LoRA) and group relative policy +optimization (GRPO), without supervised demonstrations. On eight +training-excluded paper figures, its mean champion IoU rises from $0.422$ +after eight initial attempts to $0.491$ after three verifier-guided revision +rounds. These results therefore establish a controlled framework for measuring, +retargeting, and improving visual-to-parametric photonic component design. \end{abstract} \maketitle @@ -785,136 +784,108 @@ \subsection{Dataset construction and release} model prompts. All images and programs in the release are synthetic but pass structural checks, calibrated geometric comparison, and footprint agreement. -\subsection{Training protocol} +\subsection{Training with verifier-derived signals} -The deterministic verifier takes a median $0.18$--$0.19$ seconds per -program, making executable geometry inexpensive enough to use as a training -reward. We adapt Qwen3.6-35B-A3B along two lineages. The primary lineage -uses supervised fine-tuning on scorer-selected policy outputs followed by -95 GRPO steps. A second lineage starts from the raw base model and completes -365 GRPO steps with the same reward, without supervised demonstrations or -critic-generated feedback. Both are evaluated on a seed-disjoint synthetic -pool and eight training-excluded paper figures. - -\begin{figure*}[t] +\begin{figure*}[!t] \centering -\includegraphics[width=\linewidth]{fig_rl_gallery.pdf} -\caption{\textbf{Best-of-eight programs from the isolated second draw.} -Each target is paired with its highest-IoU program from Table~\ref{tab:rl}. -Maroon denotes overlap, pink candidate-only material, and black target-only -material. The display overlays are thresholded re-renders; reported scores -come from the archived evaluator outputs. Large-scale topology is often -present, while widths, gaps, and repeated-feature counts remain imprecise.} -\label{fig:rlgallery} +\includegraphics[width=\linewidth]{fig_rl_training.pdf} +\caption{\textbf{Run B training across the PixCell representation +curriculum.} (a) Batch IoU and shaped reward across 150 GRPO updates. Points +show individual updates and lines show five-step trailing means reset at each +level. The task distribution changes at every boundary, so discontinuities +also reflect the active curriculum level. (b) Mean IoU on the fixed 80-task +probe at the base checkpoint and after every stage. Each cell contains one +attempt on each of 16 held-out parameter settings at the indicated level. The +All row reports the mean across all 80 tasks. The archived post-L3 probe, +marked with an asterisk, was measured during a harvest and evaluation +contention window and is retained as recorded.} +\label{fig:rltraining} \end{figure*} -Each episode is one blind reconstruction without test-time iteration. The -policy receives a silhouette, physical footprint, primitive catalog, and -source contract, then emits one program. The program is executed in an -isolated worker, rasterized at task calibration, and scored by the -deterministic evaluator. The policy receives no tools, difference image, or -revision turn. - -The policy is Qwen3.6-35B-A3B, a vision-capable mixture-of-experts model with -35B total and 3B active parameters~\cite{qwen36}. We apply a rank-32 -LoRA~\cite{lora} and GRPO~\cite{grpo} through the Tinker API~\cite{tinker}, -using learning rate $10^{-5}$, temperature $1.0$, eight task groups per step, -and eight rollouts per group. Advantages are computed within each group, and -constant-reward groups are omitted from the update. All reported benchmark -measurements sample the adapted policy directly, without the coding-agent -loop of Sec.~\ref{sec:agents}. - -\subsection{Reward and controls}\label{sec:rlshortcuts} - -Let $J$ and $D$ denote IoU and Dice, and let the subscript $\mathrm{rect}$ -denote the score obtained by filling the target's bounding box. Let $c$ be -the symmetric mean boundary distance in raster space, converted with the -mean axis calibration, and let $d$ be the footprint diagonal. Define -$C=\exp[-c/(0.05d)]$, -$N(x;x_0)=\max\{0,(x-x_0)/(1-x_0)\}$ for $x_0<1$ (and zero at $x_0=1$), and -$B=\max\{0,C-C_{\mathrm{rect}}\}/\max\{1-C_{\mathrm{rect}},0.3\}$. For -$J_{\mathrm{rect}}<0.95$, the reported runs use -\begin{equation} -R = v + w\left[0.40N(J;J_{\mathrm{rect}}) - +0.15N(D;D_{\mathrm{rect}})+0.40B\right], +The same deterministic evaluator used to verify reconstructed programs also +provides a reward for training. Run B begins from the raw +Qwen3.6-35B-A3B base checkpoint~\cite{qwen36}, applies a rank-32 +low-rank adaptation (LoRA)~\cite{lora}, and completes 150 group relative +policy optimization (GRPO) steps~\cite{grpo} without supervised demonstrations +or critic-generated feedback. Each episode presents one +maximum-visibility image, its physical footprint, the permitted DSL catalog, +and the source contract. The model returns one program in a non-thinking +4{,}096-token response. The evaluator applies the source gate, executes +accepted programs, renders their geometry at the target calibration, and +compares it with the reference. Syntax failures, source violations, execution +failures, and missing GDS outputs receive zero reward. Reference or evaluator +failures abort the run rather than entering the record as model errors. + +For a source-compliant executable program, let $J$ and $D$ denote IoU and +Dice, $c$ the symmetric boundary chamfer distance in micrometers, and $d$ the +footprint diagonal. The subscript $\mathrm{rect}$ denotes the result from a +solid rectangle filling the target's calibrated foreground bounding box. +Define +\begin{align*} +N(x;x_0) &= \max\left\{0,\frac{x-x_0}{1-x_0}\right\},\\ +T(c) &= \exp\left[-\frac{c}{0.05d}\right],\\ +B(c) &= \frac{\max\{0,T(c)-T(c_{\mathrm{rect}})\}} +{\max\{1-T(c_{\mathrm{rect}}),0.3\}}. +\end{align*} +\begin{samepage} +For $x_0=1$, $N(x;x_0)$ is defined as zero. When +$J_{\mathrm{rect}}<0.95$, the training reward is +\begin{align} +R ={}& 0.05 + 0.40N(J;J_{\mathrm{rect}}) \nonumber\\ +&+ 0.15N(D;D_{\mathrm{rect}}) + 0.40B(c). \label{eq:rlreward} -\end{equation} -where $v=0.05$ for an executable GDS and zero otherwise, and the whitelist -indicator $w=1$ only when the program passes the primitive whitelist. For -rectangle-like targets with $J_{\mathrm{rect}}\ge0.95$, the implementation -instead uses $R=v+w(0.40J+0.15D+0.40C)$. This rectangle baseline prevents -area-filling shortcuts from receiving normalized geometric credit. - -The training-time catalog check sets $w=0$ for compound device cells, raw -polygons, and alias imports. Such programs lose all geometric reward but -retain the $0.05$ executable floor. Evaluation always executes model output -and applies the current source audit separately. - -A footprint-only control bounds what the parameter probes establish. Across -12 rings, footprint diameter and radius have Spearman $\rho=1$; a direct -footprint formula gives $1.41\%$ mean radius error, compared with $1.55\%$ -for the model. The probes therefore measure recovery from the combined image -and footprint input, not pixels in isolation. - -\subsection{Training lineage} - -The primary lineage begins with two epochs of supervised fine-tuning on 677 -scorer-selected image-program pairs, split into 634 training and 43 -validation examples. Earlier Qwen policies generated every program, and the -builder retained outputs whose logged normalized IoU exceeded $0.5$. Code -matching traces 337 pairs to single-turn generation, 257 to two-turn runs -with deterministic verifier feedback, and 83 to critic-configured runs. The -critic input included the target, attempted render, diagnostics, and -ground-truth program; retained logs do not distinguish critic responses from -deterministic fallbacks. External-teacher information may therefore enter -this subset. - -The builder did not independently re-execute or purity-gate the 677 selected -pairs, and one uses a raw polygon. We consequently describe the data as -scorer-selected policy outputs. Starting from the supervised checkpoint, the -lineage completes 30 GRPO steps on L0--L3, 35 with a 13-figure L4 tier, and -30 after L4 expands to 18 figures. The curriculum favors task groups with -recent reward variance. The raw-base comparison instead completes 365 GRPO -steps across nine consecutive runs using the same reward and no supervised -demonstrations or critic-generated feedback. - -The 95-step history predates per-evaluation clearing of GDSFactory's cell -cache. Isolated re-scoring leaves 12 of 12 sampled training winners and all -64 programs from the second benchmark draw unchanged. We use the history and -first benchmark draw as diagnostics and the isolated second draw as the -primary result. - -\subsection{Evaluation results} - -At the last recorded synthetic checkpoint, step 85, the primary lineage -reaches L0--L3 IoU values of $0.891$, $0.776$, $0.294$, and $0.332$ -on the fixed seed-disjoint evaluation pool. Primitive and two-element tasks -are substantially stronger than path and multi-part composition tasks. - -On the training-excluded development benchmark, the isolated second draw -produces 48 executable programs from 64 attempts, mean IoU $0.2540$, and -mean per-target best-of-eight IoU $0.4662$ (Table~\ref{tab:rl}). Every target -exceeds IoU $0.05$ at least once. The first draw produces 47 executable -programs, mean IoU $0.2395$, and best-of-eight $0.4433$. - -The 365-step raw-base GRPO lineage reaches mean IoU $0.1864$ and -best-of-eight $0.3804$, with every target above $0.05$ at least once. -Deterministic executable rewards can therefore produce measurable -cross-figure transfer without supervised demonstrations. The higher primary -lineage uses scorer-selected data and a different optimization budget, so -the score difference is not an ablation of supervised fine-tuning. +\end{align} +\end{samepage} +For rectangle-like targets with $J_{\mathrm{rect}}\geq0.95$, the reward is +\[ +R = 0.05 + 0.40J + 0.15D + 0.40T(c). +\] +The rectangle baseline removes geometric credit for filling the target +bounding box. The boundary term provides a signal for count and boundary +changes that may not yet improve overlap. + +Training uses the Tinker API~\cite{tinker} with learning rate $10^{-5}$, +temperature $1.0$, eight task groups per step, and eight rollouts per group. +Within-group advantages are optimized with importance sampling and zero KL +penalty. Constant-reward groups are omitted. Run B spends 30 steps on each +level from L0 through L4 and initializes every stage from the preceding +checkpoint. After L0, each five-step cycle draws exactly 80\% of its task +groups from the active level and 20\% from earlier levels. The stage sequence +therefore contains 150 updates over 9{,}600 sampled programs. + +The fixed probe uses the same 80 examples at the base checkpoint and after +each stage, with 16 rows from the validation split at every curriculum level +(Fig.~\ref{fig:rltraining}(b)). Between the base and final checkpoints, its +overall executable rate rises from $0.20$ to $0.55$, mean IoU from $0.079$ to +$0.179$, and mean shaped reward from $0.047$ to $0.124$. The final checkpoint +improves the base mean on L0 through L3, although none of its 16 L4 responses +executes on this probe draw. These rows hold out parameter settings of known +representations, so the probe measures recovery across the curriculum rather +than transfer to unseen representations. + +We evaluate the final policy separately on the training-excluded F1--F8 +targets used throughout the paper. The non-thinking base model produces no +executable program in 64 attempts. A direct draw from the trained policy +produces 39 executable programs in 64 attempts, with mean IoU $0.228$ and +mean best-of-eight IoU $0.467$ (Table~\ref{tab:rl}). The deployment loop uses +an independent draw of eight initial attempts per target, retains the +strongest candidate, and gives it three rounds of four revisions. Feedback +reports an error class, a closeness category, and whether material is missing +or excessive without exposing numerical measurements. Mean champion IoU +rises from $0.422$ after the initial attempts to $0.452$, $0.476$, and $0.491$ +after the three revision rounds. This is one trained lineage evaluated +through repeated samples and revisions rather than an estimate of variation +across independent training runs. \input{table_rl.tex} -Sixty-three of 64 programs in the primary draw pass the current source gate. -The remaining program calls a prohibited compound component, fails execution, -and scores zero. - -Figure~\ref{fig:rlgallery} shows that the best programs often recover -large-scale topology while widths, gaps, and repeated-feature counts remain -imprecise. The result establishes trainability from executable geometry, -with substantial room between the adapted policy and the coding agents of -Sec.~\ref{sec:results}. +The released LoRA adapter is available as +\href{https://huggingface.co/qpaig-mit/pixcell}{\texttt{qpaig-mit/pixcell}}. +Its validated benchmark and revision records are released with the code. +Together, the fixed synthetic probe and the training-excluded paper targets +show that executable geometric rewards improve program generation across the +representation curriculum while leaving substantial room between the trained +policy and the coding agents of Sec.~\ref{sec:results}. \section{Research contracts}\label{sec:conclusion} @@ -965,15 +936,16 @@ \section{Research contracts}\label{sec:conclusion} retargeting, and full-wave models test optical behavior and fabrication variables that the two-dimensional layout does not contain. -The same executable verifier supplies a training reward. The primary -Qwen3.6-35B-A3B lineage reaches mean IoU $0.254$ and best-of-eight $0.466$ -on the training-excluded development benchmark. A raw-base lineage trained -for 365 GRPO steps without supervised demonstrations or critic feedback -reaches $0.186$ and $0.380$. This establishes that deterministic geometric -rewards can produce cross-figure transfer in an open-weight model. The -different optimization budgets and scorer-selected data in the primary -lineage leave the contribution of supervised fine-tuning for a controlled -ablation. +The same executable verifier supplies a training reward. Run B adapts +Qwen3.6-35B-A3B from its raw base checkpoint through 150 GRPO steps without +supervised demonstrations or critic feedback. A direct draw on the +training-excluded paper figures reaches mean IoU $0.228$ and mean +best-of-eight IoU $0.467$. In the attempt-measure-revise loop, mean champion +IoU rises from $0.422$ after the initial attempts to $0.491$ after three +revision rounds. This establishes that deterministic geometric rewards can +produce cross-figure transfer in an open-weight model. The record contains +one training lineage and does not estimate variation across independent +training runs. The current evidence covers eight components, one retained coding-agent run per matrix cell, simplified stack models, and selected diff --git a/arxiv/scripts/FIGURES.md b/arxiv/scripts/FIGURES.md index eba20286..e7cacc70 100644 --- a/arxiv/scripts/FIGURES.md +++ b/arxiv/scripts/FIGURES.md @@ -24,8 +24,7 @@ where the generator or its data is local-only. | fig5_before_after.pdf | `make_fig5_before_after.py` | `fig_before_after_assets/manifest.json`: hash-locked candidate rasters and exact archived IoUs; target masks are shared with `fig_target_assets/` | | fig_cross_matrix.pdf + `../table_retargeting.tex` | `make_fig_pdk_swap.py` | `fig_retargeting_assets/manifest.json`: hash-locked full-precision MZI cross-stack matrices and the 60 fixed-representation verdicts, reduced from the tracked `QPG-MIT/PixCell-running` campaign record at the source commit recorded in the manifest | | fig_sweep_feasibility.pdf — REMOVED from the paper 2026-07-09 (near-uniform null heatmap; the numbers live in the prose) | `data/agents/catalog_swap/cat5/output/sweep_pilot/postprocess.py` | that dir's sweep matrices | -| fig_rl_training.pdf | `make_fig_rl_training.py` | `fig_rl_assets/manifest.json` hash-locks compact telemetry derived from the tracked trunk / bridge-1 / bridge-2 ledgers and the first $k=8$ draw; `prepare_rl_figure_assets.py` records the source commit and hashes | -| fig_rl_gallery.pdf | `make_fig_rl_gallery.py` | `fig_rl_assets/manifest.json` hash-locks the fully re-scored draw descriptor plus each target and re-executed winning raster; the same descriptor generates Table III | +| fig_rl_training.pdf | `make_fig_rl_training.py` | `fig_rl_assets/manifest.json` hash-locks all 150 Run B training updates, the six fixed 80-task probes, the base and final F1--F8 draws, and the iterative deployment record. `prepare_rl_figure_assets.py` validates the summaries and records every source hash | | fig_dataset_curriculum.pdf | `make_fig_dataset_curriculum.py` | `fig_dataset_curriculum_assets/manifest.json` selects exact model-view images from the frozen L0--L4 core Parquet shards; the generator validates every source shard, row, image, code, and release digest before rendering | | fig_sidewall_stacks.pdf | `make_fig_sidewall_stacks.py` | four-panel reference, agent reconstruction, full-wave, and fabrication-sensitivity figure; `fig_sidewall_assets/manifest.json` hash-locks the published field, target/reconstruction, forward-flux plane, and fixed-geometry coupler/control sweep matrices, and the generator reconciles IoU, monitor powers, crossings, and symmetry-control bounds before plotting | | fig_trident_sidewall.pdf | `make_fig_trident_sidewall.py` (generator reconstructed 2026-07-09; curves-only single-column since the same day — the field-map grid was redundant with fig_sidewall_stacks) | `data/agents/paper_swap/final1/output/sidewall_sweep/` (`trident_matrix.json`) | diff --git a/arxiv/scripts/fig_rl_assets/F1_render.png b/arxiv/scripts/fig_rl_assets/F1_render.png deleted file mode 100644 index 5d829cff..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F1_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F1_target.png b/arxiv/scripts/fig_rl_assets/F1_target.png deleted file mode 100644 index 5a77cdc5..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F1_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F2_render.png b/arxiv/scripts/fig_rl_assets/F2_render.png deleted file mode 100644 index 66987197..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F2_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F2_target.png b/arxiv/scripts/fig_rl_assets/F2_target.png deleted file mode 100644 index dbddc8d9..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F2_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F3_render.png b/arxiv/scripts/fig_rl_assets/F3_render.png deleted file mode 100644 index d3b5ad18..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F3_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F3_target.png b/arxiv/scripts/fig_rl_assets/F3_target.png deleted file mode 100644 index 1c297f89..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F3_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F4_render.png b/arxiv/scripts/fig_rl_assets/F4_render.png deleted file mode 100644 index 0ad7ed26..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F4_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F4_target.png b/arxiv/scripts/fig_rl_assets/F4_target.png deleted file mode 100644 index 8df89710..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F4_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F5_render.png b/arxiv/scripts/fig_rl_assets/F5_render.png deleted file mode 100644 index d8793480..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F5_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F5_target.png b/arxiv/scripts/fig_rl_assets/F5_target.png deleted file mode 100644 index 26590015..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F5_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F6_render.png b/arxiv/scripts/fig_rl_assets/F6_render.png deleted file mode 100644 index b2c5c3cc..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F6_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F6_target.png b/arxiv/scripts/fig_rl_assets/F6_target.png deleted file mode 100644 index e76ca1ed..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F6_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F7_render.png b/arxiv/scripts/fig_rl_assets/F7_render.png deleted file mode 100644 index a5d4f905..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F7_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F7_target.png b/arxiv/scripts/fig_rl_assets/F7_target.png deleted file mode 100644 index 0668b63c..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F7_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F8_render.png b/arxiv/scripts/fig_rl_assets/F8_render.png deleted file mode 100644 index 5dfb8866..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F8_render.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/F8_target.png b/arxiv/scripts/fig_rl_assets/F8_target.png deleted file mode 100644 index d2f594b9..00000000 Binary files a/arxiv/scripts/fig_rl_assets/F8_target.png and /dev/null differ diff --git a/arxiv/scripts/fig_rl_assets/gallery.json b/arxiv/scripts/fig_rl_assets/gallery.json deleted file mode 100644 index 6d7fdd80..00000000 --- a/arxiv/scripts/fig_rl_assets/gallery.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "devices": [ - { - "attempts": 8, - "best_iou": 0.44, - "device": "F1", - "eval_at_1": 0.248, - "executable": 7, - "render": "F1_render.png", - "target": "F1_target.png" - }, - { - "attempts": 8, - "best_iou": 0.5987, - "device": "F2", - "eval_at_1": 0.39611250000000003, - "executable": 7, - "render": "F2_render.png", - "target": "F2_target.png" - }, - { - "attempts": 8, - "best_iou": 0.4509, - "device": "F3", - "eval_at_1": 0.19268749999999998, - "executable": 8, - "render": "F3_render.png", - "target": "F3_target.png" - }, - { - "attempts": 8, - "best_iou": 0.6373, - "device": "F4", - "eval_at_1": 0.3470875, - "executable": 5, - "render": "F4_render.png", - "target": "F4_target.png" - }, - { - "attempts": 8, - "best_iou": 0.2461, - "device": "F5", - "eval_at_1": 0.0307625, - "executable": 1, - "render": "F5_render.png", - "target": "F5_target.png" - }, - { - "attempts": 8, - "best_iou": 0.5573, - "device": "F6", - "eval_at_1": 0.2152875, - "executable": 5, - "render": "F6_render.png", - "target": "F6_target.png" - }, - { - "attempts": 8, - "best_iou": 0.3032, - "device": "F7", - "eval_at_1": 0.208775, - "executable": 7, - "render": "F7_render.png", - "target": "F7_target.png" - }, - { - "attempts": 8, - "best_iou": 0.4969, - "device": "F8", - "eval_at_1": 0.392825, - "executable": 8, - "render": "F8_render.png", - "target": "F8_target.png" - } - ] -} diff --git a/arxiv/scripts/fig_rl_assets/manifest.json b/arxiv/scripts/fig_rl_assets/manifest.json index b0fa2f32..b4a5a097 100644 --- a/arxiv/scripts/fig_rl_assets/manifest.json +++ b/arxiv/scripts/fig_rl_assets/manifest.json @@ -1,150 +1,72 @@ { "assets": { - "F1_render.png": { - "sha256": "82ff1b29168abc59c75c575ff42e2edf2f70e1f69524072c80ed3c8c57c90a4d" - }, - "F1_target.png": { - "sha256": "4918de2860e5c0cb3c2fe6cc008239487e6c393b203f5c6abb9db5e8d17e388a" - }, - "F2_render.png": { - "sha256": "863f3e12c2d0c14d365faae55a1ddec87621bd27ce7f2764e8fabb506f870836" - }, - "F2_target.png": { - "sha256": "4f5a52bd860755672084df88668f71b19c1d57011428d77816394ae1d8b9a511" - }, - "F3_render.png": { - "sha256": "8236327f121906a4e0dcad14626436f83f489ce1de15d5e07d80a6ff163eba5d" - }, - "F3_target.png": { - "sha256": "e5f04cdeb580448b50a226d86b25a879d1cd7d94af6fe6b259a1177b19ed6317" - }, - "F4_render.png": { - "sha256": "bdbf4f495bfc06499c558521c5a80c705a61edfe619da2a4c4561b4149c0e6de" - }, - "F4_target.png": { - "sha256": "83599c61a94827616f8d697433e043e86c8fa1c2c109b81225a3388190130bb8" - }, - "F5_render.png": { - "sha256": "ab86cbfa3d36906baec4160a191ea43828540dc5e1e8f0685c5621113cd99dd5" - }, - "F5_target.png": { - "sha256": "7a34be2d69d45d993e02aa30991b0ab47c9ef971d9cfdc46badb4f04cf42df3a" - }, - "F6_render.png": { - "sha256": "244df2475026b30da6f155801c57f21eebd334c9c9f407fe82c7bd63341a0597" - }, - "F6_target.png": { - "sha256": "79d3d58e71bbaa86fec2c07f919224abc2dac4980c6d19937b0e715fa2a85c6f" - }, - "F7_render.png": { - "sha256": "3d6c58fdfeb5f10aef2bb1a19e18565bbabe4e7ea629b3469697f47350148669" - }, - "F7_target.png": { - "sha256": "1cfed9fe3c569030162929f5e8165fbf099b23bbc46d41921d7469d3430a9d13" - }, - "F8_render.png": { - "sha256": "43947290a3b69b6c1be5edc87d56772f7efb7ce1a41cef19cb839a9653d04308" - }, - "F8_target.png": { - "sha256": "28ee227266f825ecb2d9d310d4e0f0100e3f55b85d77ae71264eea8f7308ba84" - }, - "gallery.json": { - "sha256": "07b88ca74a3edbf4f91c702a5772da2c73c9a27ce1020ba0ae3dee513c77ac0c" - }, "training_telemetry.json": { - "sha256": "20510317a46d25236b723bbb73056157765d6d4ba5039f0ddc85adc56f1a978a" + "sha256": "27a2293411186cb511988a1e1827099cee67ee246a6995b7be5dbe91fb859f39" } }, - "runtime": { - "gdsfactory": "9.20.7", - "python": "3.13" - }, - "source_commit": "b2c07631d2225ddd82247c413b0f45cac4bb2978", - "source_files": [ - { - "path": "rl/reports/runs_ledger/pixcell-Qwen-Qwen3.6-35B-A3B-8g-8b-lr1e-05-20260708-0953/metrics.jsonl", - "sha256": "8e3201ecbb3d250872829ce5c2acb6b33a2d9999a4dd470edb910802b8007133" - }, - { - "path": "rl/reports/runs_ledger/pixcell-Qwen-Qwen3.6-35B-A3B-8g-8b-lr1e-05-20260708-1043/metrics.jsonl", - "sha256": "04a3728e0ff7ffd04f838fff18015b7bef7676f26d8c782182a282e3184ff34b" - }, - { - "path": "rl/reports/runs_ledger/pixcell-Qwen-Qwen3.6-35B-A3B-8g-8b-lr1e-05-20260708-1209/metrics.jsonl", - "sha256": "b8c0a4c7af4e0d975e65778264fc5700f24f645ce68d062e8275ed897fd3cc0a" - }, - { - "path": "rl/reports/benchmark_eval_bridge2.json", - "sha256": "7574084c44cc37a121d584f45550c4060b42b6e6f67dabdece5e553bb59ce81f" - }, + "record": "runs-v2/runB-base-rl", + "repository_sources": [ { - "path": "rl/reports/benchmark_attempts_bridge2.jsonl", - "sha256": "1f9f583ba1fe298b5cba45f254bbd9b9c366524031996bb476cda8159892f4a6" + "path": "data/training/qwen-rl-v2/bench/base-nothink.json", + "sha256": "ceb61c226f1d579831e0f878774b06ec20de602ddf2400e3fb7d0f2e46ba4eb3" }, { - "path": "data/benchmark/final_1/device_bw.png", - "sha256": "0fbd3bb59c59f81b606be1b5d8d948b941b73ca240d4bb31ba9c9d72e6808d3c" + "path": "data/training/qwen-rl-v2/bench/b-final.json", + "sha256": "55d9e938a4f0e97a35e520d7ea5aa9a621af9ad98555ba0a2b5a75ae2e10f8ba" }, { - "path": "data/benchmark/final_1/calibration.json", - "sha256": "00baf5808d50db5a3b0422c62c97deca19c18ecbd0737c1a4e79919937b5d55a" - }, - { - "path": "data/benchmark/final_2/device_bw.png", - "sha256": "f9aac3ccc82c4528647f741d619e246e003e42aab13a0b5687d6a9811068576b" - }, - { - "path": "data/benchmark/final_2/calibration.json", - "sha256": "1125b8622744536dfe7695c5ec8a9faa23608df889b581640610be15bbbfe341" - }, + "path": "data/training/qwen-rl-v2/agentloop/b-final.json", + "sha256": "1e1e764cdf0640408f58759c55d6adc952bf18cdee84501cfde3a1349e63a0ed" + } + ], + "run_sources": [ { - "path": "data/benchmark/final_3/device_bw.png", - "sha256": "4962daf2f6b430a3124ba83f0702742b019c24121272f111363c991d847e3c3e" + "path": "provenance/plan.json", + "sha256": "8d59fa85d95c0e81d78b7e4c839961dffbf872d2426548170de111b392dcc349" }, { - "path": "data/benchmark/final_3/calibration.json", - "sha256": "7ba84ce4db812f440a49f75e8dd21537325e097e7e9fb7b8ba18cf1889bb60fb" + "path": "stages/rl-l0/metrics.jsonl", + "sha256": "f6662beb7376d014d62619b3ba8f44058d193d54733560da27919a76a27c1209" }, { - "path": "data/benchmark/final_4/device_bw.png", - "sha256": "a57c3d7acbffa63f691d580bba31ada077ad552c94f360271160912b7e30d701" + "path": "stages/rl-l1/metrics.jsonl", + "sha256": "7b252c126487c1e86254fde10a882af1635142676d5934484614a20df353ec9c" }, { - "path": "data/benchmark/final_4/calibration.json", - "sha256": "b2121f5fe9ad59bf082f9b85be4bce4d485fb617a38fbbfeddc32f1262919803" + "path": "stages/rl-l2/metrics.jsonl", + "sha256": "9540202fddb61599cc9bbbb40c04011a625fd4ad43e6bc29125d3bab0e36916b" }, { - "path": "data/benchmark/final_5/device_bw.png", - "sha256": "6e850e58b0768aa18e828b7b0a4d1b38db4460cef7f3fef0782a58909bf457a3" + "path": "stages/rl-l3/metrics.jsonl", + "sha256": "4ba207f490d451206b744c0d7de87899f620168010808c8404566dc83c503241" }, { - "path": "data/benchmark/final_5/calibration.json", - "sha256": "eb6d2951abc0906e205cd85809c4544cb4488e115995affeb3da4feb1ad93451" + "path": "stages/rl-l4/metrics.jsonl", + "sha256": "3df127ed090fe5b491de56b776715f5f4959e06b1d637041ede7882f641ac836" }, { - "path": "data/benchmark/final_6/device_bw.png", - "sha256": "bacae5d13895fd2462fb77fda14d06c56080dd408f362323bc5a7ef2618596a5" + "path": "probes/base.json", + "sha256": "3f733388ccc8edc7b3c2427237aa9a7d62c6219532384fcc140314ea863f873f" }, { - "path": "data/benchmark/final_6/calibration.json", - "sha256": "8590266c1c1ff7d4b2a32417fb4b17146dbfe62edf9843bc1937c3a52a86e6d6" + "path": "probes/rl-l0.json", + "sha256": "5eaede4b66fec88d03ebe101c9de286d2850885176d6c8cc0eb06b04496738cf" }, { - "path": "data/benchmark/final_7/device_bw.png", - "sha256": "91847bf5797234e985065e7828fbbc306c92c26e281c685175e0efaa4749ce18" + "path": "probes/rl-l1.json", + "sha256": "b6053beffeeead77dc5f54e6d388f9240577e1664bdafc668806a0d4ff116acc" }, { - "path": "data/benchmark/final_7/calibration.json", - "sha256": "2abed81050aad4a5e6fdbe53c15990ef67757d95a75e219b897a771844a8ad51" + "path": "probes/rl-l2.json", + "sha256": "e51155df74d0b28029bddc4b690e1d98dd227885e6d2dd1b275a1b5a227ca8af" }, { - "path": "data/benchmark/final_8/device_bw.png", - "sha256": "6654508aff5e3fb52f0cecb66a0caa055049e82f6c6ad296386c9e44c341758b" + "path": "probes/rl-l3.json", + "sha256": "ff01225caa2aaf372a949c00c7dc113e0ba340708221970263664fa3494a5d05" }, { - "path": "data/benchmark/final_8/calibration.json", - "sha256": "7729ff61ac021ce05c7ce4a7e75f4ba3791ad31a11355ec1126432c3c2d40a0e" + "path": "probes/rl-l4.json", + "sha256": "722f626e8a12a2f0784335c3954f9afac2b5a16a7069d15cc6937e348a7fdc77" } - ], - "source_repository": "QPG-MIT/PixCell-running" + ] } diff --git a/arxiv/scripts/fig_rl_assets/training_telemetry.json b/arxiv/scripts/fig_rl_assets/training_telemetry.json index 24ddf1e9..f97b4ea3 100644 --- a/arxiv/scripts/fig_rl_assets/training_telemetry.json +++ b/arxiv/scripts/fig_rl_assets/training_telemetry.json @@ -1,308 +1,1378 @@ { - "evaluations": [ - { - "L0": 0.7703500000000001, - "L1": 0.5876625, - "L2": 0.14710625, - "L3": 0.15850625000000002, - "benchmark": 0.20155, + "levels": [ + "L0", + "L1", + "L2", + "L3", + "L4" + ], + "outcomes": { + "base_single_pass": { + "executable": 0, + "mean_best_of_k": 0.0, + "mean_iou": 0.0, + "total": 64 + }, + "iterative": { + "final_mean_champion_iou": 0.4910063784832184, + "initial_attempts": 8, + "revision_rounds": 3, + "revisions_per_round": 4, + "round_means": [ + { + "mean_champion_iou": 0.4217301857773721, + "round": 0 + }, + { + "mean_champion_iou": 0.452065737272719, + "round": 1 + }, + { + "mean_champion_iou": 0.4760232878786076, + "round": 2 + }, + { + "mean_champion_iou": 0.4910063784832184, + "round": 3 + } + ] + }, + "model_path": "tinker://d8269c70-0dd9-5776-9cae-53e8b24e647a:train:0/sampler_weights/final", + "single_pass": { + "executable": 39, + "mean_best_of_k": 0.4665450605692228, + "mean_iou": 0.22766049014169865, + "total": 64 + } + }, + "probes": [ + { + "checkpoint": "base", + "condition": "nominal", + "model_path": null, + "overall": { + "executable_rate": 0.2, + "mean_iou": 0.07904707003274214, + "mean_shaped": 0.046644250492282706, + "n": 80 + }, + "per_level": { + "L0": { + "executable_rate": 0.4375, + "mean_iou": 0.2345614471061806, + "mean_shaped": 0.10703802783683167, + "n": 16 + }, + "L1": { + "executable_rate": 0.3125, + "mean_iou": 0.10490326159525466, + "mean_shaped": 0.09361126739084794, + "n": 16 + }, + "L2": { + "executable_rate": 0.125, + "mean_iou": 0.015281071611656928, + "mean_shaped": 0.007093527418856394, + "n": 16 + }, + "L3": { + "executable_rate": 0.125, + "mean_iou": 0.04048956985061852, + "mean_shaped": 0.029253040898911856, + "n": 16 + }, + "L4": { + "executable_rate": 0.0, + "mean_iou": 0.0, + "mean_shaped": 0.0, + "n": 16 + } + }, "step": 0 }, { - "L0": 0.7072375, - "L1": 0.629925, - "L2": 0.24835625, - "L3": 0.15638125, - "benchmark": 0.1248625, + "checkpoint": "rl-l0", + "condition": "nominal", + "model_path": "tinker://e1cca90a-d680-5ae2-bde2-dd5eb054d85c:train:0/sampler_weights/final", + "overall": { + "executable_rate": 0.25, + "mean_iou": 0.09219087351198171, + "mean_shaped": 0.0515433630099905, + "n": 80 + }, + "per_level": { + "L0": { + "executable_rate": 0.5, + "mean_iou": 0.21467258003805417, + "mean_shaped": 0.0928922510544109, + "n": 16 + }, + "L1": { + "executable_rate": 0.25, + "mean_iou": 0.11640243233315618, + "mean_shaped": 0.06206315456563835, + "n": 16 + }, + "L2": { + "executable_rate": 0.1875, + "mean_iou": 0.06578145626238938, + "mean_shaped": 0.017856376103368113, + "n": 16 + }, + "L3": { + "executable_rate": 0.25, + "mean_iou": 0.06273952453420237, + "mean_shaped": 0.08178003332653519, + "n": 16 + }, + "L4": { + "executable_rate": 0.0625, + "mean_iou": 0.0013583743921064908, + "mean_shaped": 0.003125, + "n": 16 + } + }, + "step": 30 + }, + { + "checkpoint": "rl-l1", + "condition": "nominal", + "model_path": "tinker://7998d09d-bcfe-5270-8c8d-c147b185d776:train:0/sampler_weights/final", + "overall": { + "executable_rate": 0.25, + "mean_iou": 0.10943516052295714, + "mean_shaped": 0.09084375528701913, + "n": 80 + }, + "per_level": { + "L0": { + "executable_rate": 0.375, + "mean_iou": 0.22001774464755725, + "mean_shaped": 0.1783831339987133, + "n": 16 + }, + "L1": { + "executable_rate": 0.4375, + "mean_iou": 0.21003591853144513, + "mean_shaped": 0.1566251195469763, + "n": 16 + }, + "L2": { + "executable_rate": 0.25, + "mean_iou": 0.07030674353037351, + "mean_shaped": 0.07489885117380975, + "n": 16 + }, + "L3": { + "executable_rate": 0.125, + "mean_iou": 0.029779714779511832, + "mean_shaped": 0.03080113111372602, + "n": 16 + }, + "L4": { + "executable_rate": 0.0625, + "mean_iou": 0.01703568112589793, + "mean_shaped": 0.013510540601870229, + "n": 16 + } + }, + "step": 60 + }, + { + "checkpoint": "rl-l2", + "condition": "nominal", + "model_path": "tinker://e48c09ef-7c97-527e-b68d-eab0f4c84319:train:0/sampler_weights/final", + "overall": { + "executable_rate": 0.5375, + "mean_iou": 0.1660522203826053, + "mean_shaped": 0.10477616836942831, + "n": 80 + }, + "per_level": { + "L0": { + "executable_rate": 0.75, + "mean_iou": 0.30143518468735725, + "mean_shaped": 0.18431443859689192, + "n": 16 + }, + "L1": { + "executable_rate": 0.5, + "mean_iou": 0.2029793945053059, + "mean_shaped": 0.12646136312696143, + "n": 16 + }, + "L2": { + "executable_rate": 0.625, + "mean_iou": 0.17762753905081682, + "mean_shaped": 0.09989995977062553, + "n": 16 + }, + "L3": { + "executable_rate": 0.375, + "mean_iou": 0.0633247874466701, + "mean_shaped": 0.0552027885693823, + "n": 16 + }, + "L4": { + "executable_rate": 0.4375, + "mean_iou": 0.08489419622287654, + "mean_shaped": 0.058002291783280376, + "n": 16 + } + }, + "step": 90 + }, + { + "checkpoint": "rl-l3", + "condition": "evaluation_contention", + "model_path": "tinker://792c2abe-2859-56c6-ad7f-c887dc48a830:train:0/sampler_weights/final", + "overall": { + "executable_rate": 0.3, + "mean_iou": 0.09129397409928762, + "mean_shaped": 0.06317697819563717, + "n": 80 + }, + "per_level": { + "L0": { + "executable_rate": 0.0625, + "mean_iou": 0.024677194895613194, + "mean_shaped": 0.019080595438020063, + "n": 16 + }, + "L1": { + "executable_rate": 0.0, + "mean_iou": 0.0, + "mean_shaped": 0.0, + "n": 16 + }, + "L2": { + "executable_rate": 0.5, + "mean_iou": 0.15756835970912858, + "mean_shaped": 0.10292899295898592, + "n": 16 + }, + "L3": { + "executable_rate": 0.5625, + "mean_iou": 0.15020488316463257, + "mean_shaped": 0.14038474717113336, + "n": 16 + }, + "L4": { + "executable_rate": 0.375, + "mean_iou": 0.12401943272706374, + "mean_shaped": 0.05349055541004653, + "n": 16 + } + }, + "step": 120 + }, + { + "checkpoint": "rl-l4", + "condition": "nominal", + "model_path": "tinker://d8269c70-0dd9-5776-9cae-53e8b24e647a:train:0/sampler_weights/final", + "overall": { + "executable_rate": 0.55, + "mean_iou": 0.1791512512901597, + "mean_shaped": 0.12384156033510837, + "n": 80 + }, + "per_level": { + "L0": { + "executable_rate": 0.875, + "mean_iou": 0.28116647701088754, + "mean_shaped": 0.19433378691597814, + "n": 16 + }, + "L1": { + "executable_rate": 0.625, + "mean_iou": 0.33181424479073607, + "mean_shaped": 0.20163018581140651, + "n": 16 + }, + "L2": { + "executable_rate": 0.75, + "mean_iou": 0.18115085194673558, + "mean_shaped": 0.14571577262639687, + "n": 16 + }, + "L3": { + "executable_rate": 0.5, + "mean_iou": 0.10162468270243936, + "mean_shaped": 0.07752805632176044, + "n": 16 + }, + "L4": { + "executable_rate": 0.0, + "mean_iou": 0.0, + "mean_shaped": 0.0, + "n": 16 + } + }, + "step": 150 + } + ], + "run": { + "key": "B", + "lora_rank": 32, + "max_tokens": 4096, + "model_name": "Qwen/Qwen3.6-35B-A3B", + "name": "runB-base-rl", + "renderer_name": "qwen3_5_disable_thinking", + "source_git_dirty": true, + "source_git_head": "8aa0c3bfe598b9265b9f26fdd37129947537dafb", + "steps_per_level": 30, + "total_steps": 150 + }, + "training": [ + { + "executable_rate": 0.546875, + "level": "L0", + "mean_iou": 0.33678130675815166, + "mean_shaped": 0.26115748407505446, + "step": 1 + }, + { + "executable_rate": 0.25, + "level": "L0", + "mean_iou": 0.11381286563883672, + "mean_shaped": 0.07063513248704495, + "step": 2 + }, + { + "executable_rate": 0.35714285714285715, + "level": "L0", + "mean_iou": 0.15076481368962297, + "mean_shaped": 0.10729309360722396, + "step": 3 + }, + { + "executable_rate": 0.35714285714285715, + "level": "L0", + "mean_iou": 0.12329741193705616, + "mean_shaped": 0.08984813369919223, + "step": 4 + }, + { + "executable_rate": 0.453125, + "level": "L0", + "mean_iou": 0.17707774456215897, + "mean_shaped": 0.08732212800413156, + "step": 5 + }, + { + "executable_rate": 0.40625, + "level": "L0", + "mean_iou": 0.1764090545333118, + "mean_shaped": 0.1197659187904062, + "step": 6 + }, + { + "executable_rate": 0.4166666666666667, + "level": "L0", + "mean_iou": 0.1418492123723888, + "mean_shaped": 0.08272372501470165, + "step": 7 + }, + { + "executable_rate": 0.5, + "level": "L0", + "mean_iou": 0.11909441133971013, + "mean_shaped": 0.07144087519180789, + "step": 8 + }, + { + "executable_rate": 0.359375, + "level": "L0", + "mean_iou": 0.23186477517605933, + "mean_shaped": 0.15361201904702, + "step": 9 + }, + { + "executable_rate": 0.46875, + "level": "L0", + "mean_iou": 0.26718931971767834, + "mean_shaped": 0.20993022705916195, "step": 10 }, { - "L0": 0.7719624999999999, - "L1": 0.6725375, - "L2": 0.20782499999999998, - "L3": 0.29064375, - "benchmark": 0.243525, - "step": 20 + "executable_rate": 0.4107142857142857, + "level": "L0", + "mean_iou": 0.18596623120092426, + "mean_shaped": 0.1473253722914203, + "step": 11 }, { - "L0": 0.7707124999999999, - "L1": 0.6783250000000001, - "L2": 0.2809625, - "L3": 0.16118125, - "benchmark": 0.1829875, - "step": 30 + "executable_rate": 0.40625, + "level": "L0", + "mean_iou": 0.2259471503127956, + "mean_shaped": 0.21455018193935566, + "step": 12 }, { - "L0": 0.78285, - "L1": 0.7631875, - "L2": 0.21299375, - "L3": 0.22898749999999998, - "benchmark": 0.128625, - "step": 40 + "executable_rate": 0.4375, + "level": "L0", + "mean_iou": 0.24369532109307582, + "mean_shaped": 0.17756904225953565, + "step": 13 }, { - "L0": 0.85183125, - "L1": 0.83229375, - "L2": 0.3305125, - "L3": 0.24170625, - "benchmark": 0.07339999999999999, - "step": 50 + "executable_rate": 0.25, + "level": "L0", + "mean_iou": 0.11944094519271073, + "mean_shaped": 0.0888189912980955, + "step": 14 }, { - "L0": 0.89419375, - "L1": 0.7578374999999999, - "L2": 0.3806125, - "L3": 0.28714375000000003, - "benchmark": 0.156775, - "step": 60 + "executable_rate": 0.390625, + "level": "L0", + "mean_iou": 0.17882866156221638, + "mean_shaped": 0.07888335703746699, + "step": 15 }, { - "L0": 0.82815, - "L1": 0.7131625, - "L2": 0.32218125, - "L3": 0.30391875, - "benchmark": 0.2396875, - "step": 65 + "executable_rate": 0.328125, + "level": "L0", + "mean_iou": 0.11863354317528807, + "mean_shaped": 0.05167532250449558, + "step": 16 }, { - "L0": 0.821975, - "L1": 0.70065625, - "L2": 0.29264375, - "L3": 0.27845, - "benchmark": 0.162825, - "step": 75 + "executable_rate": 0.46875, + "level": "L0", + "mean_iou": 0.16447948808788285, + "mean_shaped": 0.09769320299144274, + "step": 17 }, { - "L0": 0.8909, - "L1": 0.7760625, - "L2": 0.2939375, - "L3": 0.33208125, - "benchmark": 0.3107625, - "step": 85 - } - ], - "final_draw": { - "best_at_8": 0.4433, - "eval_at_1": 0.2395 - }, - "stages": [ + "executable_rate": 0.46875, + "level": "L0", + "mean_iou": 0.25440696787012634, + "mean_shaped": 0.1507619726138262, + "step": 18 + }, { - "label": "L0--L3", - "start_step": 0 + "executable_rate": 0.46875, + "level": "L0", + "mean_iou": 0.17268656957618017, + "mean_shaped": 0.11473589865037091, + "step": 19 }, { - "label": "13-figure L4", - "start_step": 30 + "executable_rate": 0.65, + "level": "L0", + "mean_iou": 0.3606505963200731, + "mean_shaped": 0.2275416991883148, + "step": 20 }, { - "label": "18-figure L4", - "start_step": 65 - } - ], - "training": [ + "executable_rate": 0.453125, + "level": "L0", + "mean_iou": 0.17554096903787036, + "mean_shaped": 0.15699293264412212, + "step": 21 + }, + { + "executable_rate": 0.4375, + "level": "L0", + "mean_iou": 0.28420466761979846, + "mean_shaped": 0.23304373317763938, + "step": 22 + }, + { + "executable_rate": 0.484375, + "level": "L0", + "mean_iou": 0.23826876936837468, + "mean_shaped": 0.22325772333052407, + "step": 23 + }, { - "iou": 0.0279375, + "executable_rate": 0.5625, + "level": "L0", + "mean_iou": 0.38020040527259524, + "mean_shaped": 0.34488321492486784, + "step": 24 + }, + { + "executable_rate": 0.625, + "level": "L0", + "mean_iou": 0.49040550674954614, + "mean_shaped": 0.3816928426773368, + "step": 25 + }, + { + "executable_rate": 0.421875, + "level": "L0", + "mean_iou": 0.18828577042234018, + "mean_shaped": 0.11049661834039447, + "step": 26 + }, + { + "executable_rate": 0.390625, + "level": "L0", + "mean_iou": 0.17972463690690005, + "mean_shaped": 0.14739681131319993, + "step": 27 + }, + { + "executable_rate": 0.40625, + "level": "L0", + "mean_iou": 0.1750877951739523, + "mean_shaped": 0.11032424822772774, + "step": 28 + }, + { + "executable_rate": 0.7083333333333334, + "level": "L0", + "mean_iou": 0.3611097252616254, + "mean_shaped": 0.20484467763423464, + "step": 29 + }, + { + "executable_rate": 0.46875, + "level": "L0", + "mean_iou": 0.23102586646232237, + "mean_shaped": 0.12676140968721436, "step": 30 }, { - "iou": 0.009225, + "executable_rate": 0.3958333333333333, + "level": "L1", + "mean_iou": 0.12259945359010271, + "mean_shaped": 0.09327685588601324, "step": 31 }, { - "iou": 0.038816666666666666, + "executable_rate": 0.48214285714285715, + "level": "L1", + "mean_iou": 0.15013644313129518, + "mean_shaped": 0.11767238466920862, "step": 32 }, { - "iou": 0.26475, + "executable_rate": 0.42857142857142855, + "level": "L1", + "mean_iou": 0.1501079835935458, + "mean_shaped": 0.11036376289457497, "step": 33 }, { - "iou": 0.14175, + "executable_rate": 0.390625, + "level": "L1", + "mean_iou": 0.174359367624253, + "mean_shaped": 0.0993008595522685, "step": 34 }, { - "iou": 0.045475, + "executable_rate": 0.5178571428571429, + "level": "L1", + "mean_iou": 0.2279992102814714, + "mean_shaped": 0.16090644466191825, "step": 35 }, { - "iou": 0.2330125, + "executable_rate": 0.55, + "level": "L1", + "mean_iou": 0.22754246039670117, + "mean_shaped": 0.12111443880415282, "step": 36 }, { - "iou": 0.0254625, + "executable_rate": 0.5, + "level": "L1", + "mean_iou": 0.22678340621042356, + "mean_shaped": 0.10436239216183636, "step": 37 }, { - "iou": 0.30475, + "executable_rate": 0.46875, + "level": "L1", + "mean_iou": 0.16920938715621464, + "mean_shaped": 0.10144096910732586, "step": 38 }, { - "iou": 0.1587625, + "executable_rate": 0.375, + "level": "L1", + "mean_iou": 0.1990338953606582, + "mean_shaped": 0.12042709913413809, "step": 39 }, { - "iou": 0.1749125, + "executable_rate": 0.5178571428571429, + "level": "L1", + "mean_iou": 0.2157263753733058, + "mean_shaped": 0.16420687862221123, "step": 40 }, { - "iou": 0.020100000000000003, + "executable_rate": 0.4583333333333333, + "level": "L1", + "mean_iou": 0.14729958757023157, + "mean_shaped": 0.11544642684308708, "step": 41 }, { - "iou": 0.1332875, + "executable_rate": 0.375, + "level": "L1", + "mean_iou": 0.0984728290073601, + "mean_shaped": 0.04157849672588122, "step": 42 }, { - "iou": 0.01091875, + "executable_rate": 0.5208333333333334, + "level": "L1", + "mean_iou": 0.26777276946119805, + "mean_shaped": 0.20247662509845268, + "step": 43 + }, + { + "executable_rate": 0.28125, + "level": "L1", + "mean_iou": 0.12126666843258019, + "mean_shaped": 0.048985785247425855, "step": 44 }, { - "iou": 0.0060374999999999995, + "executable_rate": 0.1875, + "level": "L1", + "mean_iou": 0.11093883649137212, + "mean_shaped": 0.045579193556810144, + "step": 45 + }, + { + "executable_rate": 0.546875, + "level": "L1", + "mean_iou": 0.28583388115173636, + "mean_shaped": 0.20425240004200712, "step": 46 }, { - "iou": 0.35516875000000003, + "executable_rate": 0.6071428571428571, + "level": "L1", + "mean_iou": 0.25517033068882095, + "mean_shaped": 0.1561672112978929, "step": 47 }, { - "iou": 0.013250000000000001, + "executable_rate": 0.578125, + "level": "L1", + "mean_iou": 0.13764236829007365, + "mean_shaped": 0.09693334495398437, "step": 48 }, { - "iou": 0.9102875, + "executable_rate": 0.4375, + "level": "L1", + "mean_iou": 0.21171962669085714, + "mean_shaped": 0.15077899738252357, + "step": 49 + }, + { + "executable_rate": 0.5833333333333334, + "level": "L1", + "mean_iou": 0.32477510570228335, + "mean_shaped": 0.2533363585925122, "step": 50 }, { - "iou": 0.12417500000000001, + "executable_rate": 0.5357142857142857, + "level": "L1", + "mean_iou": 0.21163033866888417, + "mean_shaped": 0.18083027234355445, "step": 51 }, { - "iou": 0.19929999999999998, + "executable_rate": 0.5, + "level": "L1", + "mean_iou": 0.2040000831029387, + "mean_shaped": 0.14993746130560673, "step": 52 }, { - "iou": 0.37390625, + "executable_rate": 0.390625, + "level": "L1", + "mean_iou": 0.0878881331668308, + "mean_shaped": 0.05065783856665526, + "step": 53 + }, + { + "executable_rate": 0.4107142857142857, + "level": "L1", + "mean_iou": 0.20657844843622644, + "mean_shaped": 0.1154938214369271, "step": 54 }, { - "iou": 0.1603875, + "executable_rate": 0.625, + "level": "L1", + "mean_iou": 0.23806928650155942, + "mean_shaped": 0.22191276958141418, "step": 55 }, { - "iou": 0.045275, + "executable_rate": 0.484375, + "level": "L1", + "mean_iou": 0.28507998523470507, + "mean_shaped": 0.13821186730580967, + "step": 56 + }, + { + "executable_rate": 0.5, + "level": "L1", + "mean_iou": 0.21832006956568417, + "mean_shaped": 0.14111358923444847, "step": 57 }, { - "iou": 0.32123749999999995, + "executable_rate": 0.6785714285714286, + "level": "L1", + "mean_iou": 0.28407880200459523, + "mean_shaped": 0.22388775543886114, + "step": 58 + }, + { + "executable_rate": 0.390625, + "level": "L1", + "mean_iou": 0.21749949422620982, + "mean_shaped": 0.13945438105487878, + "step": 59 + }, + { + "executable_rate": 0.4642857142857143, + "level": "L1", + "mean_iou": 0.19885316541950768, + "mean_shaped": 0.14489006789215805, + "step": 60 + }, + { + "executable_rate": 0.4166666666666667, + "level": "L2", + "mean_iou": 0.1358005443805603, + "mean_shaped": 0.1306216267157694, "step": 61 }, { - "iou": 0.12778749999999997, + "executable_rate": 0.4375, + "level": "L2", + "mean_iou": 0.12201304722967178, + "mean_shaped": 0.06712234671865346, "step": 62 }, { - "iou": 0.41622499999999996, + "executable_rate": 0.3125, + "level": "L2", + "mean_iou": 0.0726659718744514, + "mean_shaped": 0.05922301882816261, "step": 63 }, { - "iou": 0.57231875, + "executable_rate": 0.5357142857142857, + "level": "L2", + "mean_iou": 0.07921734238776919, + "mean_shaped": 0.06037822631387219, "step": 64 }, { - "iou": 0.0090125, + "executable_rate": 0.515625, + "level": "L2", + "mean_iou": 0.21485621146850956, + "mean_shaped": 0.08340962527027025, "step": 65 }, { - "iou": 0.4592125, + "executable_rate": 0.5178571428571429, + "level": "L2", + "mean_iou": 0.245274079869366, + "mean_shaped": 0.1828016347886626, "step": 66 }, { - "iou": 0.017275, + "executable_rate": 0.453125, + "level": "L2", + "mean_iou": 0.17174570975383993, + "mean_shaped": 0.07597387383842631, "step": 67 }, { - "iou": 0.20375000000000001, + "executable_rate": 0.5357142857142857, + "level": "L2", + "mean_iou": 0.285085566886294, + "mean_shaped": 0.11780073402806067, "step": 68 }, { - "iou": 0.103425, + "executable_rate": 0.5892857142857143, + "level": "L2", + "mean_iou": 0.24782922421455453, + "mean_shaped": 0.1309190944014929, "step": 69 }, { - "iou": 0.30513125, + "executable_rate": 0.3333333333333333, + "level": "L2", + "mean_iou": 0.05979543989637781, + "mean_shaped": 0.04382890954094589, "step": 70 }, { - "iou": 0.2215375, + "executable_rate": 0.5178571428571429, + "level": "L2", + "mean_iou": 0.22225006813935577, + "mean_shaped": 0.10567495528278473, "step": 71 }, { - "iou": 0.2121666666666667, + "executable_rate": 0.3125, + "level": "L2", + "mean_iou": 0.13365906888160528, + "mean_shaped": 0.026816988398807542, "step": 72 }, { - "iou": 0.3163375, + "executable_rate": 0.5, + "level": "L2", + "mean_iou": 0.18538782725999106, + "mean_shaped": 0.07110654327500074, "step": 73 }, { - "iou": 0.18741249999999998, + "executable_rate": 0.3392857142857143, + "level": "L2", + "mean_iou": 0.15883682381987288, + "mean_shaped": 0.06574909290798132, "step": 74 }, { - "iou": 0.32771875, + "executable_rate": 0.4107142857142857, + "level": "L2", + "mean_iou": 0.13665430396668335, + "mean_shaped": 0.08716309808209587, "step": 75 }, { - "iou": 0.49505, + "executable_rate": 0.375, + "level": "L2", + "mean_iou": 0.1881517260400703, + "mean_shaped": 0.12556009042641855, "step": 76 }, { - "iou": 0.42708749999999995, + "executable_rate": 0.4791666666666667, + "level": "L2", + "mean_iou": 0.1500942161340392, + "mean_shaped": 0.11183846603229186, "step": 77 }, { - "iou": 0.23195624999999997, + "executable_rate": 0.325, + "level": "L2", + "mean_iou": 0.06475898515808576, + "mean_shaped": 0.06884472753120774, + "step": 78 + }, + { + "executable_rate": 0.625, + "level": "L2", + "mean_iou": 0.22605649296978195, + "mean_shaped": 0.18082338328377046, "step": 79 }, { - "iou": 0.147475, + "executable_rate": 0.375, + "level": "L2", + "mean_iou": 0.18605692005021304, + "mean_shaped": 0.11601973033419437, + "step": 80 + }, + { + "executable_rate": 0.5535714285714286, + "level": "L2", + "mean_iou": 0.2712988640234832, + "mean_shaped": 0.13723673767417907, "step": 81 }, { - "iou": 0.37101249999999997, + "executable_rate": 0.625, + "level": "L2", + "mean_iou": 0.23596720737371887, + "mean_shaped": 0.1643206364218349, "step": 82 }, { - "iou": 0.006925, + "executable_rate": 0.6458333333333334, + "level": "L2", + "mean_iou": 0.24735336236327612, + "mean_shaped": 0.14130995948944539, "step": 83 }, { - "iou": 0.6218375, + "executable_rate": 0.4375, + "level": "L2", + "mean_iou": 0.14581282016139716, + "mean_shaped": 0.08407067579256408, "step": 84 }, { - "iou": 0.05128125, + "executable_rate": 0.375, + "level": "L2", + "mean_iou": 0.2138865967002322, + "mean_shaped": 0.11102901227809184, "step": 85 }, { - "iou": 0.494475, + "executable_rate": 0.4791666666666667, + "level": "L2", + "mean_iou": 0.16336662441302802, + "mean_shaped": 0.03537301416724228, "step": 86 }, { - "iou": 0.0601625, + "executable_rate": 0.5357142857142857, + "level": "L2", + "mean_iou": 0.189616169874391, + "mean_shaped": 0.1631076427624298, "step": 87 }, { - "iou": 0.26808125, + "executable_rate": 0.5208333333333334, + "level": "L2", + "mean_iou": 0.20241978126109825, + "mean_shaped": 0.07363275083731176, + "step": 88 + }, + { + "executable_rate": 0.5416666666666666, + "level": "L2", + "mean_iou": 0.15716203096584788, + "mean_shaped": 0.11120210188155617, "step": 89 }, { - "iou": 0.11478749999999999, + "executable_rate": 0.625, + "level": "L2", + "mean_iou": 0.2739612001479338, + "mean_shaped": 0.12681594832098814, "step": 90 }, { - "iou": 0.3334625, + "executable_rate": 0.484375, + "level": "L3", + "mean_iou": 0.10966662155617003, + "mean_shaped": 0.08947947369259568, + "step": 91 + }, + { + "executable_rate": 0.5625, + "level": "L3", + "mean_iou": 0.14873943736952797, + "mean_shaped": 0.10995417882418367, "step": 92 + }, + { + "executable_rate": 0.42857142857142855, + "level": "L3", + "mean_iou": 0.15458366534672782, + "mean_shaped": 0.110161751700156, + "step": 93 + }, + { + "executable_rate": 0.5714285714285714, + "level": "L3", + "mean_iou": 0.19011827579898583, + "mean_shaped": 0.1573169107832255, + "step": 94 + }, + { + "executable_rate": 0.35714285714285715, + "level": "L3", + "mean_iou": 0.09862165290316909, + "mean_shaped": 0.072656079452475, + "step": 95 + }, + { + "executable_rate": 0.609375, + "level": "L3", + "mean_iou": 0.18635965957521322, + "mean_shaped": 0.1220780018618413, + "step": 96 + }, + { + "executable_rate": 0.5535714285714286, + "level": "L3", + "mean_iou": 0.12131128883639399, + "mean_shaped": 0.09732553961781683, + "step": 97 + }, + { + "executable_rate": 0.5714285714285714, + "level": "L3", + "mean_iou": 0.1601372840963407, + "mean_shaped": 0.11907203295621248, + "step": 98 + }, + { + "executable_rate": 0.4375, + "level": "L3", + "mean_iou": 0.0846974900740326, + "mean_shaped": 0.09106331064525838, + "step": 99 + }, + { + "executable_rate": 0.5357142857142857, + "level": "L3", + "mean_iou": 0.17049261187914203, + "mean_shaped": 0.09663448550807836, + "step": 100 + }, + { + "executable_rate": 0.484375, + "level": "L3", + "mean_iou": 0.1857302316158502, + "mean_shaped": 0.12140503699524893, + "step": 101 + }, + { + "executable_rate": 0.6458333333333334, + "level": "L3", + "mean_iou": 0.3350618015179445, + "mean_shaped": 0.19983568230875792, + "step": 102 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.044103805460284716, + "mean_shaped": 0.015249367384219477, + "step": 103 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.010957464842968105, + "mean_shaped": 0.01128560541672912, + "step": 104 + }, + { + "executable_rate": 0.1875, + "level": "L3", + "mean_iou": 0.08149852555791995, + "mean_shaped": 0.07028593761131322, + "step": 105 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.056673212125828196, + "mean_shaped": 0.048692960132184895, + "step": 106 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.04589736554040833, + "mean_shaped": 0.02212645328240444, + "step": 107 + }, + { + "executable_rate": 0.0, + "level": "L3", + "mean_iou": 0.0, + "mean_shaped": 0.0, + "step": 108 + }, + { + "executable_rate": 0.1875, + "level": "L3", + "mean_iou": 0.09729535497085369, + "mean_shaped": 0.05554982198934037, + "step": 109 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.06529982018606834, + "mean_shaped": 0.02590474943729338, + "step": 110 + }, + { + "executable_rate": 0.20833333333333334, + "level": "L3", + "mean_iou": 0.03997616469359907, + "mean_shaped": 0.03308624471595972, + "step": 111 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.03569930407188991, + "mean_shaped": 0.0162321729872535, + "step": 112 + }, + { + "executable_rate": 0.15, + "level": "L3", + "mean_iou": 0.05478144518871555, + "mean_shaped": 0.019130978800753585, + "step": 113 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.03742828710221215, + "mean_shaped": 0.0365604495476757, + "step": 114 + }, + { + "executable_rate": 0.3333333333333333, + "level": "L3", + "mean_iou": 0.12943284645607855, + "mean_shaped": 0.04475858982379304, + "step": 115 + }, + { + "executable_rate": 0.6428571428571429, + "level": "L3", + "mean_iou": 0.1916301215884453, + "mean_shaped": 0.1508251974215406, + "step": 116 + }, + { + "executable_rate": 0.125, + "level": "L3", + "mean_iou": 0.04086146388351563, + "mean_shaped": 0.038821578116927845, + "step": 117 + }, + { + "executable_rate": 0.22916666666666666, + "level": "L3", + "mean_iou": 0.05108917689340453, + "mean_shaped": 0.058168160575508505, + "step": 118 + }, + { + "executable_rate": 0.375, + "level": "L3", + "mean_iou": 0.13423266802110498, + "mean_shaped": 0.10793924961816186, + "step": 119 + }, + { + "executable_rate": 0.5357142857142857, + "level": "L3", + "mean_iou": 0.12784733789612593, + "mean_shaped": 0.11601323816319979, + "step": 120 + }, + { + "executable_rate": 0.6428571428571429, + "level": "L4", + "mean_iou": 0.14460845313684487, + "mean_shaped": 0.12981691160063716, + "step": 121 + }, + { + "executable_rate": 0.23214285714285715, + "level": "L4", + "mean_iou": 0.05230878843610346, + "mean_shaped": 0.03596100701488209, + "step": 122 + }, + { + "executable_rate": 0.6071428571428571, + "level": "L4", + "mean_iou": 0.2637313526606303, + "mean_shaped": 0.1681694300700784, + "step": 123 + }, + { + "executable_rate": 0.3, + "level": "L4", + "mean_iou": 0.08496266799470426, + "mean_shaped": 0.08115753044560654, + "step": 124 + }, + { + "executable_rate": 0.3, + "level": "L4", + "mean_iou": 0.1101878564996011, + "mean_shaped": 0.08765799224614927, + "step": 125 + }, + { + "executable_rate": 0.640625, + "level": "L4", + "mean_iou": 0.20496548822482868, + "mean_shaped": 0.21427423723463385, + "step": 126 + }, + { + "executable_rate": 0.515625, + "level": "L4", + "mean_iou": 0.09478946755657061, + "mean_shaped": 0.08409505811946866, + "step": 127 + }, + { + "executable_rate": 0.546875, + "level": "L4", + "mean_iou": 0.17233366811979484, + "mean_shaped": 0.11025562148628404, + "step": 128 + }, + { + "executable_rate": 0.125, + "level": "L4", + "mean_iou": 0.0693657861857043, + "mean_shaped": 0.05432378660400926, + "step": 129 + }, + { + "executable_rate": 0.15625, + "level": "L4", + "mean_iou": 0.02724960532860788, + "mean_shaped": 0.02185102063302294, + "step": 130 + }, + { + "executable_rate": 0.6875, + "level": "L4", + "mean_iou": 0.2370264107469821, + "mean_shaped": 0.19031707465064407, + "step": 131 + }, + { + "executable_rate": 0.35, + "level": "L4", + "mean_iou": 0.06948673554279597, + "mean_shaped": 0.06323021985595623, + "step": 132 + }, + { + "executable_rate": 0.40625, + "level": "L4", + "mean_iou": 0.10876934971837518, + "mean_shaped": 0.07020065579632365, + "step": 133 + }, + { + "executable_rate": 0.5625, + "level": "L4", + "mean_iou": 0.15725599088894562, + "mean_shaped": 0.10048457433827951, + "step": 134 + }, + { + "executable_rate": 0.5178571428571429, + "level": "L4", + "mean_iou": 0.1607301005782877, + "mean_shaped": 0.12204099185869445, + "step": 135 + }, + { + "executable_rate": 0.3958333333333333, + "level": "L4", + "mean_iou": 0.14056997006444527, + "mean_shaped": 0.079593419808162, + "step": 136 + }, + { + "executable_rate": 0.65625, + "level": "L4", + "mean_iou": 0.2172367689218557, + "mean_shaped": 0.1235508335796437, + "step": 137 + }, + { + "executable_rate": 0.125, + "level": "L4", + "mean_iou": 0.020066909522807517, + "mean_shaped": 0.014526604616449145, + "step": 138 + }, + { + "executable_rate": 0.16666666666666666, + "level": "L4", + "mean_iou": 0.011234373736562086, + "mean_shaped": 0.021546993152354116, + "step": 139 + }, + { + "executable_rate": 0.640625, + "level": "L4", + "mean_iou": 0.24547377066904275, + "mean_shaped": 0.10842525017409678, + "step": 140 + }, + { + "executable_rate": 0.21428571428571427, + "level": "L4", + "mean_iou": 0.1263177792592433, + "mean_shaped": 0.10389195059102267, + "step": 141 + }, + { + "executable_rate": 0.8392857142857143, + "level": "L4", + "mean_iou": 0.3005553564857034, + "mean_shaped": 0.18833058680275988, + "step": 142 + }, + { + "executable_rate": 0.3125, + "level": "L4", + "mean_iou": 0.0946192933450999, + "mean_shaped": 0.08246897064385128, + "step": 143 + }, + { + "executable_rate": 0.25, + "level": "L4", + "mean_iou": 0.03222579646021468, + "mean_shaped": 0.03735275872636308, + "step": 144 + }, + { + "executable_rate": 0.53125, + "level": "L4", + "mean_iou": 0.2215950406015169, + "mean_shaped": 0.12142611221985722, + "step": 145 + }, + { + "executable_rate": 0.375, + "level": "L4", + "mean_iou": 0.13719863588989017, + "mean_shaped": 0.04032805153778125, + "step": 146 + }, + { + "executable_rate": 0.578125, + "level": "L4", + "mean_iou": 0.12707843732784538, + "mean_shaped": 0.1492741968281025, + "step": 147 + }, + { + "executable_rate": 0.65625, + "level": "L4", + "mean_iou": 0.1706531705168382, + "mean_shaped": 0.14013395898729247, + "step": 148 + }, + { + "executable_rate": 0.71875, + "level": "L4", + "mean_iou": 0.15734496910751639, + "mean_shaped": 0.1501670147430885, + "step": 149 + }, + { + "executable_rate": 0.5625, + "level": "L4", + "mean_iou": 0.1142122207146939, + "mean_shaped": 0.0838754162530078, + "step": 150 } ] } diff --git a/arxiv/scripts/make_fig_rl_gallery.py b/arxiv/scripts/make_fig_rl_gallery.py deleted file mode 100644 index 662e8161..00000000 --- a/arxiv/scripts/make_fig_rl_gallery.py +++ /dev/null @@ -1,297 +0,0 @@ -"""Render Figure 13 and Table III from hash-locked benchmark evidence. - -Each card pairs one frozen target with the highest-IoU executable program from -the fully re-scored best-of-eight draw. Difference maps use Figure 8's -publication convention: maroon for overlap, pink for candidate-only material, -and black for target-only material. - -Run from the repository root: - python arxiv/scripts/make_fig_rl_gallery.py -""" - -from __future__ import annotations - -import hashlib -import json -from pathlib import Path - -import matplotlib - -matplotlib.use("Agg") -import matplotlib.pyplot as plt -import numpy as np -from matplotlib.colors import to_rgb -from matplotlib.lines import Line2D -from matplotlib.patches import Rectangle -from PIL import Image - - -REPO = Path(__file__).resolve().parents[2] -ASSET_DIR = Path(__file__).with_name("fig_rl_assets") -MANIFEST = ASSET_DIR / "manifest.json" -FIGURE_DIR = REPO / "arxiv" / "figures" -TABLE_PATH = REPO / "arxiv" / "table_rl.tex" - -INK = "#17191c" -SOFT_INK = "#555a60" -RULE = "#17191c" -OVERLAP = "#800000" -CANDIDATE_ONLY = "#ff9299" -TARGET_ONLY = INK -ASPECT = 0.36 - -plt.rcParams.update( - { - "font.family": "serif", - "font.serif": [ - "Iowan Old Style", - "STIX Two Text", - "STIXGeneral", - "Times New Roman", - ], - "mathtext.fontset": "stix", - "axes.unicode_minus": False, - "pdf.fonttype": 42, - "ps.fonttype": 42, - "savefig.facecolor": "white", - "savefig.bbox": None, - } -) - - -def _sha256(path: Path) -> str: - return hashlib.sha256(path.read_bytes()).hexdigest() - - -def _verified_path(manifest: dict, name: str) -> Path: - path = ASSET_DIR / name - expected = manifest["assets"][name]["sha256"] - actual = _sha256(path) - if actual != expected: - raise ValueError( - f"Hash mismatch for {path}: expected {expected}, got {actual}." - ) - return path - - -def _load_data() -> list[dict]: - manifest = json.loads(MANIFEST.read_text()) - descriptor_path = _verified_path(manifest, "gallery.json") - data = json.loads(descriptor_path.read_text())["devices"] - if [row["device"] for row in data] != [f"F{index}" for index in range(1, 9)]: - raise ValueError("Figure 13 requires ordered targets F1--F8.") - for row in data: - row["target_image"] = Image.open( - _verified_path(manifest, row["target"]) - ).convert("L") - row["render_image"] = Image.open( - _verified_path(manifest, row["render"]) - ).convert("L") - if row["target_image"].size != row["render_image"].size: - raise ValueError(f"Target/render dimensions differ for {row['device']}.") - return data - - -def _mask(image: Image.Image) -> np.ndarray: - return np.asarray(image) < 128 - - -def _rgb(color: str) -> np.ndarray: - return np.rint(255 * np.asarray(to_rgb(color))).astype(np.uint8) - - -def _target_image(image: Image.Image) -> np.ndarray: - mask = _mask(image) - output = np.full((*mask.shape, 3), 255, dtype=np.uint8) - output[mask] = _rgb(INK) - return output - - -def _difference_image(target: Image.Image, render: Image.Image) -> np.ndarray: - target_mask = _mask(target) - render_mask = _mask(render) - output = np.full((*target_mask.shape, 3), 255, dtype=np.uint8) - output[target_mask & render_mask] = _rgb(OVERLAP) - output[render_mask & ~target_mask] = _rgb(CANDIDATE_ONLY) - output[target_mask & ~render_mask] = _rgb(TARGET_ONLY) - return output - - -def _pad_to_aspect(image: np.ndarray, aspect: float = ASPECT) -> np.ndarray: - height, width = image.shape[:2] - if height / width < aspect: - target_height, target_width = int(round(width * aspect)), width - else: - target_height, target_width = height, int(round(height / aspect)) - output = np.full((target_height, target_width, 3), 255, dtype=np.uint8) - y0 = (target_height - height) // 2 - x0 = (target_width - width) // 2 - output[y0 : y0 + height, x0 : x0 + width] = image - return output - - -def _image_axes( - figure: plt.Figure, - bounds: tuple[float, float, float, float], - image: np.ndarray, -) -> None: - axes = figure.add_axes(bounds) - axes.imshow(_pad_to_aspect(image), interpolation="antialiased") - axes.axis("off") - - -def _draw_card( - figure: plt.Figure, - *, - bounds: tuple[float, float, float, float], - row: dict, -) -> None: - x, y, width, height = bounds - figure.add_artist( - Rectangle( - (x, y), - width, - height, - transform=figure.transFigure, - facecolor="none", - edgecolor=INK, - linewidth=0.68, - zorder=0, - ) - ) - figure.text( - x + 0.012, - y + height - 0.028, - row["device"], - ha="left", - va="center", - fontsize=9.2, - color=INK, - ) - figure.text( - x + width - 0.012, - y + height - 0.028, - f"IoU {row['best_iou']:.3f} · " - f"{row['executable']}/{row['attempts']} executable", - ha="right", - va="center", - fontsize=6.35, - color=INK, - ) - figure.add_artist( - Line2D( - [x + 0.010, x + width - 0.010], - [y + height - 0.055, y + height - 0.055], - transform=figure.transFigure, - color=RULE, - linewidth=0.52, - ) - ) - figure.text( - x + 0.012, - y + height - 0.077, - "Target", - ha="left", - va="center", - fontsize=6.6, - color=SOFT_INK, - ) - figure.text( - x + 0.012, - y + 0.218, - "Best of 8", - ha="left", - va="center", - fontsize=6.6, - color=SOFT_INK, - ) - _image_axes( - figure, - (x + 0.012, y + 0.245, width - 0.024, 0.120), - _target_image(row["target_image"]), - ) - _image_axes( - figure, - (x + 0.012, y + 0.060, width - 0.024, 0.135), - _difference_image(row["target_image"], row["render_image"]), - ) - - -def _render_figure(data: list[dict]) -> None: - figure = plt.figure(figsize=(7.40, 4.02), facecolor="white") - card_width = 0.230 - x_positions = (0.020, 0.263, 0.506, 0.749) - for index, row in enumerate(data): - block, column = divmod(index, 4) - y = 0.520 if block == 0 else 0.035 - _draw_card( - figure, - bounds=(x_positions[column], y, card_width, 0.445), - row=row, - ) - - FIGURE_DIR.mkdir(parents=True, exist_ok=True) - pdf_path = FIGURE_DIR / "fig_rl_gallery.pdf" - png_path = FIGURE_DIR / "fig_rl_gallery.png" - figure.savefig(pdf_path, dpi=600, facecolor="white", bbox_inches=None) - figure.savefig(png_path, dpi=300, facecolor="white", bbox_inches=None) - plt.close(figure) - print(f"wrote {pdf_path}") - print(f"wrote {png_path}") - - -def _render_table(data: list[dict]) -> None: - executable_total = sum(row["executable"] for row in data) - attempt_total = sum(row["attempts"] for row in data) - eval_mean = float(np.mean([row["eval_at_1"] for row in data])) - best_mean = float(np.mean([row["best_iou"] for row in data])) - rows = [ - ( - f"{row['device']} & " - f"{row['executable']}/{row['attempts']} & " - f"{row['eval_at_1']:.3f} & " - f"{row['best_iou']:.3f} \\\\" - ) - for row in data - ] - table = "\n".join( - [ - r"\begin{table}[t]", - r"\caption{\textbf{Fully re-scored final-checkpoint evaluation.} " - r"Eight attempts per target at temperature $1.0$. Executable counts " - r"attempts that produced a scored GDS; eval@1 is mean IoU and best@8 " - r"is the maximum over the eight attempts.}", - r"\label{tab:rl}", - r"\begingroup", - r"\footnotesize", - r"\setlength{\tabcolsep}{7.0pt}", - r"\renewcommand{\arraystretch}{1.06}", - r"\begin{ruledtabular}", - r"\begin{tabular}{@{}lccc@{}}", - r"Device & Executable & eval@1 & best@8 \\", - r"\colrule", - *rows, - r"\colrule", - ( - f"All / mean & {executable_total}/{attempt_total} & " - f"{eval_mean:.3f} & {best_mean:.3f} \\\\" - ), - r"\end{tabular}", - r"\end{ruledtabular}", - r"\endgroup", - r"\end{table}", - "", - ] - ) - TABLE_PATH.write_text(table) - print(f"wrote {TABLE_PATH}") - - -def main() -> None: - data = _load_data() - _render_figure(data) - _render_table(data) - - -if __name__ == "__main__": - main() diff --git a/arxiv/scripts/make_fig_rl_training.py b/arxiv/scripts/make_fig_rl_training.py index 8001f7c0..aaaede00 100644 --- a/arxiv/scripts/make_fig_rl_training.py +++ b/arxiv/scripts/make_fig_rl_training.py @@ -1,30 +1,33 @@ -"""Render Figure 12 from the frozen RL telemetry record. +"""Render the Run B curriculum-training figure from its frozen record. -Panel A shows scheduled synthetic evaluations for tiers L0--L3. Panel B -compares the recorded L4 training trace with the training-excluded benchmark -and the final best-of-eight draw. The compact source record is hash-locked in -``fig_rl_assets/manifest.json``. +Panel A follows the 150 GRPO updates through L0--L4. Panel B evaluates the +same six checkpoints on one fixed 80-task probe, with 16 held-out parameter +settings at each representation level. Run from the repository root: - python arxiv/scripts/make_fig_rl_training.py + uv run python arxiv/scripts/make_fig_rl_training.py """ from __future__ import annotations import hashlib import json +import math from pathlib import Path import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt +import numpy as np +from matplotlib.colors import LinearSegmentedColormap from matplotlib.lines import Line2D REPO = Path(__file__).resolve().parents[2] ASSET_DIR = Path(__file__).with_name("fig_rl_assets") MANIFEST = ASSET_DIR / "manifest.json" +TELEMETRY = ASSET_DIR / "training_telemetry.json" FIGURE_DIR = REPO / "arxiv" / "figures" INK = "#17191c" @@ -32,14 +35,10 @@ RULE = "#17191c" GRID = "#e2e4e6" CRIMSON = "#a31f34" -VERIFIED = "#267c5a" DIFF_BLUE = "#315e9f" -TIER_STYLE = { - "L0": (DIFF_BLUE, "o", "-"), - "L1": (CRIMSON, "s", "-"), - "L2": (VERIFIED, "^", "-"), - "L3": (INK, "D", (0, (4.0, 2.0))), -} +STAGE_FILL = ("#f4f6f8", "#ffffff") +LEVELS = tuple(f"L{index}" for index in range(5)) +CHECKPOINT_LABELS = ("Base", "+L0", "+L1", "+L2", "+L3", "+L4") plt.rcParams.update( { @@ -66,18 +65,25 @@ def _sha256(path: Path) -> str: def _load_data() -> dict: manifest = json.loads(MANIFEST.read_text()) - path = ASSET_DIR / "training_telemetry.json" - expected = manifest["assets"][path.name]["sha256"] - actual = _sha256(path) + expected = manifest["assets"][TELEMETRY.name]["sha256"] + actual = _sha256(TELEMETRY) if actual != expected: raise ValueError( - f"Hash mismatch for {path}: expected {expected}, got {actual}." + f"Hash mismatch for {TELEMETRY}: expected {expected}, got {actual}." ) - data = json.loads(path.read_text()) - if len(data["evaluations"]) != 10: - raise ValueError("Figure 12 requires ten scheduled evaluation checkpoints.") - if [stage["start_step"] for stage in data["stages"]] != [0, 30, 65]: - raise ValueError("Figure 12 stage boundaries do not match the paper record.") + data = json.loads(TELEMETRY.read_text()) + if data["levels"] != list(LEVELS): + raise ValueError("Run B telemetry does not contain the ordered L0--L4 levels.") + training = data["training"] + if len(training) != 150: + raise ValueError("Run B figure requires exactly 150 training steps.") + if [record["step"] for record in training] != list(range(1, 151)): + raise ValueError("Run B training steps must be contiguous.") + if len(data["probes"]) != 6: + raise ValueError("Run B figure requires base plus five post-stage probes.") + for probe in data["probes"]: + if any(probe["per_level"][level]["n"] != 16 for level in LEVELS): + raise ValueError("Every fixed-probe cell must contain 16 tasks.") return data @@ -99,7 +105,7 @@ def _panel_heading( color=INK, ) figure.text( - x + 0.035, + x + 0.034, 0.965, title, ha="left", @@ -118,7 +124,15 @@ def _panel_heading( ) -def _style_axis(axes: plt.Axes) -> None: +def _rolling(values: list[float], window: int = 5) -> list[float]: + return [ + sum(values[max(0, index - window + 1) : index + 1]) + / len(values[max(0, index - window + 1) : index + 1]) + for index in range(len(values)) + ] + + +def _style_training_axis(axes: plt.Axes) -> None: axes.spines["top"].set_visible(False) axes.spines["right"].set_visible(False) axes.spines["left"].set_linewidth(0.68) @@ -134,211 +148,228 @@ def _style_axis(axes: plt.Axes) -> None: ) axes.grid(axis="y", color=GRID, linewidth=0.52, zorder=0) axes.set_axisbelow(True) - axes.set_xlim(-2, 100) - axes.set_ylim(0, 1.02) - axes.set_xticks((0, 20, 40, 60, 80, 100)) - axes.set_yticks((0, 0.25, 0.5, 0.75, 1.0)) + axes.set_xlim(0.5, 150.5) + axes.set_ylim(0.0, 0.52) + axes.set_xticks((1, 30, 60, 90, 120, 150)) + axes.set_yticks(np.arange(0.0, 0.51, 0.1)) axes.set_xlabel("Cumulative GRPO step", fontsize=8.0, labelpad=3.5) - axes.set_ylabel("IoU", fontsize=8.0, labelpad=3.5) - - -def _draw_stages(axes: plt.Axes, stages: list[dict]) -> None: - for boundary in (30, 65): - axes.axvline( - boundary, - color="#aeb3b7", - linewidth=0.65, - linestyle=(0, (2.0, 2.5)), - zorder=1, + axes.set_ylabel("Mean score", fontsize=8.0, labelpad=3.5) + + +def _draw_training(axes: plt.Axes, data: dict) -> None: + records = data["training"] + steps = np.asarray([record["step"] for record in records]) + iou = np.asarray([record["mean_iou"] for record in records]) + shaped = np.asarray([record["mean_shaped"] for record in records]) + + for level_index, level in enumerate(LEVELS): + start = level_index * 30 + 0.5 + end = (level_index + 1) * 30 + 0.5 + axes.axvspan( + start, + end, + color=STAGE_FILL[level_index % 2], + linewidth=0, + zorder=-2, ) - boundaries = [stage["start_step"] for stage in stages] + [100] - for stage, end in zip(stages, boundaries[1:]): axes.text( - 0.5 * (stage["start_step"] + end), - 0.975, - stage["label"], + 0.5 * (start + end), + 0.985, + level, transform=axes.get_xaxis_transform(), ha="center", va="top", - fontsize=6.25, + fontsize=6.8, color=SOFT_INK, ) + if level_index: + axes.axvline( + start, + color="#aeb3b7", + linewidth=0.65, + linestyle=(0, (2.0, 2.5)), + zorder=1, + ) + + axes.scatter( + steps, + iou, + s=5.0, + color=DIFF_BLUE, + alpha=0.16, + linewidths=0, + zorder=2, + ) + axes.scatter( + steps, + shaped, + s=5.0, + color=CRIMSON, + alpha=0.14, + linewidths=0, + zorder=2, + ) - -def _rolling_mean(values: list[float], window: int = 5) -> list[float]: - return [ - sum(values[max(0, index - window + 1) : index + 1]) - / len(values[max(0, index - window + 1) : index + 1]) - for index in range(len(values)) - ] - - -def _draw_synthetic(axes: plt.Axes, data: dict) -> None: - evaluations = data["evaluations"] - steps = [record["step"] for record in evaluations] - for tier, (color, marker, line_style) in TIER_STYLE.items(): - axes.plot( - steps, - [record[tier] for record in evaluations], - color=color, - marker=marker, - linestyle=line_style, - linewidth=1.15, - markersize=3.2, - markeredgewidth=0.5, - label=tier, + iou_handle = None + shaped_handle = None + for level_index in range(5): + start = level_index * 30 + end = start + 30 + segment_steps = steps[start:end] + current_iou = axes.plot( + segment_steps, + _rolling(iou[start:end].tolist()), + color=DIFF_BLUE, + linewidth=1.35, zorder=3, - ) - _style_axis(axes) - _draw_stages(axes, data["stages"]) + )[0] + current_shaped = axes.plot( + segment_steps, + _rolling(shaped[start:end].tolist()), + color=CRIMSON, + linewidth=1.25, + zorder=3, + )[0] + if iou_handle is None: + iou_handle = current_iou + if shaped_handle is None: + shaped_handle = current_shaped + + _style_training_axis(axes) axes.legend( + handles=(iou_handle, shaped_handle), + labels=("Batch IoU · 5-step mean", "Shaped reward · 5-step mean"), loc="lower center", bbox_to_anchor=(0.5, 1.015), - ncol=4, + ncol=2, frameon=False, fontsize=6.7, - handlelength=1.55, - handletextpad=0.35, + handlelength=1.65, + handletextpad=0.38, columnspacing=0.9, borderaxespad=0.0, ) -def _draw_transfer(axes: plt.Axes, data: dict) -> None: - training = data["training"] - train_steps = [record["step"] for record in training] - train_values = [record["iou"] for record in training] - axes.plot( - train_steps, - train_values, - color=CRIMSON, - linewidth=0.55, - alpha=0.20, - zorder=2, - ) - training_line = axes.plot( - train_steps, - _rolling_mean(train_values), - color=CRIMSON, - linewidth=1.35, - label="L4 training · 5-step mean", - zorder=3, - )[0] - - evaluations = [ - record - for record in data["evaluations"] - if record["benchmark"] is not None +def _draw_probe(axes: plt.Axes, data: dict, figure: plt.Figure) -> None: + row_labels = (*LEVELS, "All") + checkpoint_labels = [ + f"{label}*" if probe["condition"] == "evaluation_contention" else label + for label, probe in zip(CHECKPOINT_LABELS, data["probes"]) ] - benchmark_line = axes.plot( - [record["step"] for record in evaluations], - [record["benchmark"] for record in evaluations], - color=DIFF_BLUE, - marker="o", - markerfacecolor="white", - markeredgecolor=DIFF_BLUE, - markeredgewidth=0.8, - linewidth=0.95, - markersize=3.3, - label="Training-excluded benchmark", - zorder=4, - )[0] - - final = data["final_draw"] - axes.plot( - [95], - [final["eval_at_1"]], - marker="o", - color=VERIFIED, - markersize=4.3, - zorder=5, + matrix = np.asarray( + [ + [ + ( + probe["overall"]["mean_iou"] + if level == "All" + else probe["per_level"][level]["mean_iou"] + ) + for probe in data["probes"] + ] + for level in row_labels + ], + dtype=float, ) - axes.plot( - [95], - [final["best_at_8"]], - marker="D", - markerfacecolor="white", - markeredgecolor=VERIFIED, - markeredgewidth=1.0, - markersize=4.4, - zorder=5, + if not np.isfinite(matrix).all() or matrix.min() < 0.0 or matrix.max() > 1.0: + raise ValueError("Fixed-probe IoU matrix is invalid.") + + upper = max(0.35, math.ceil(float(matrix.max()) * 20.0) / 20.0) + cmap = LinearSegmentedColormap.from_list( + "pixcell_probe", + ("#f7f9fc", "#dbe4f1", "#91a9c9", DIFF_BLUE), ) - axes.text( - 92.5, - final["best_at_8"] + 0.035, - f"best@8 {final['best_at_8']:.3f}", - ha="right", - va="bottom", - fontsize=6.5, - color=INK, + image = axes.imshow( + matrix, + cmap=cmap, + vmin=0.0, + vmax=upper, + interpolation="nearest", + aspect="auto", ) - axes.text( - 92.5, - final["eval_at_1"] - 0.035, - f"eval@1 {final['eval_at_1']:.3f}", - ha="right", - va="top", - fontsize=6.5, - color=INK, + axes.set_xticks(range(6), labels=checkpoint_labels) + axes.set_yticks(range(6), labels=row_labels) + axes.tick_params( + axis="both", + which="major", + length=0, + labelsize=7.0, + pad=3.0, + colors=INK, ) - - _style_axis(axes) - _draw_stages(axes, data["stages"]) - endpoint_handle = Line2D( - [], - [], - color=VERIFIED, - marker="D", - markerfacecolor="white", - linewidth=0, - markersize=3.8, - label=r"Final $k{=}8$", + axes.set_xlabel("Checkpoint", fontsize=8.0, labelpad=4.0) + axes.set_ylabel("Probe level", fontsize=8.0, labelpad=4.0) + axes.set_xticks(np.arange(-0.5, 6.0, 1.0), minor=True) + axes.set_yticks(np.arange(-0.5, 6.0, 1.0), minor=True) + axes.grid(which="minor", color="white", linewidth=1.1) + axes.tick_params(which="minor", bottom=False, left=False) + axes.axhline(4.5, color=INK, linewidth=0.72) + for spine in axes.spines.values(): + spine.set_visible(False) + + for row in range(matrix.shape[0]): + for column in range(matrix.shape[1]): + value = matrix[row, column] + axes.text( + column, + row, + f"{value:.2f}", + ha="center", + va="center", + fontsize=6.8, + color="white" if value >= 0.62 * upper else INK, + ) + + bounds = axes.get_position() + color_axes = figure.add_axes( + ( + bounds.x1 + 0.012, + bounds.y0 + 0.014, + 0.010, + bounds.height - 0.028, + ) ) - axes.legend( - handles=(training_line, benchmark_line, endpoint_handle), - loc="lower center", - bbox_to_anchor=(0.5, 1.015), - ncol=3, - frameon=False, - fontsize=6.35, - handlelength=1.45, - handletextpad=0.35, - columnspacing=0.72, - borderaxespad=0.0, + colorbar = figure.colorbar(image, cax=color_axes) + colorbar.outline.set_linewidth(0.55) + colorbar.set_ticks((0.0, upper)) + colorbar.ax.tick_params( + labelsize=6.4, + width=0.5, + length=2.0, + pad=2.0, + colors=INK, ) + colorbar.set_label("Mean IoU", fontsize=7.1, labelpad=2.5) def main() -> None: data = _load_data() figure = plt.figure(figsize=(7.40, 2.88), facecolor="white") - synthetic_axes = figure.add_axes((0.070, 0.170, 0.405, 0.650)) - transfer_axes = figure.add_axes((0.565, 0.170, 0.405, 0.650)) + training_axes = figure.add_axes((0.070, 0.175, 0.500, 0.640)) + probe_axes = figure.add_axes((0.630, 0.175, 0.292, 0.640)) _panel_heading( figure, x=0.020, letter="A", - title="Synthetic evaluation by tier", - rule_end=0.490, + title="Run B curriculum optimization", + rule_end=0.590, ) _panel_heading( figure, - x=0.510, + x=0.605, letter="B", - title="Real-figure transfer", + title="Fixed cross-level probe", rule_end=0.980, ) - _draw_synthetic(synthetic_axes, data) - _draw_transfer(transfer_axes, data) + _draw_training(training_axes, data) + _draw_probe(probe_axes, data, figure) FIGURE_DIR.mkdir(parents=True, exist_ok=True) - pdf_path = FIGURE_DIR / "fig_rl_training.pdf" - png_path = FIGURE_DIR / "fig_rl_training.png" - figure.savefig(pdf_path, dpi=600, facecolor="white", bbox_inches=None) - figure.savefig(png_path, dpi=300, facecolor="white", bbox_inches=None) + output = FIGURE_DIR / "fig_rl_training.pdf" + figure.savefig(output, dpi=600, facecolor="white", bbox_inches=None) plt.close(figure) - print(f"wrote {pdf_path}") - print(f"wrote {png_path}") + print(f"wrote {output}") if __name__ == "__main__": diff --git a/arxiv/scripts/prepare_rl_figure_assets.py b/arxiv/scripts/prepare_rl_figure_assets.py index ea77e044..edda1f5a 100644 --- a/arxiv/scripts/prepare_rl_figure_assets.py +++ b/arxiv/scripts/prepare_rl_figure_assets.py @@ -1,50 +1,39 @@ #!/usr/bin/env python3 -"""Freeze the compact evidence used by Figures 12 and 13. +"""Freeze the compact Run B evidence used by the training figure. -The complete RL ledger and executable benchmark archive live in the companion -PixCell working repository. This script reduces those records to the numeric -series and raster pairs required by the paper, records the source commit and -input hashes, and writes a hash manifest beside the derived assets. +The complete paid-run archive remains outside the repository. This script +reduces its five 30-step metric ledgers and six fixed-probe evaluations to the +numeric record needed by the paper. It also binds the derived record to every +source file by SHA-256. The F1--F8 benchmark and iterative deployment records +are already tracked under ``data/training/qwen-rl-v2`` and are incorporated +directly. -Run from the paper repository root: +Run from the repository root: - python arxiv/scripts/prepare_rl_figure_assets.py \ - --source-repo ../PixCell-running + uv run python arxiv/scripts/prepare_rl_figure_assets.py \ + --run-root /path/to/runs-v2/runB-base-rl """ from __future__ import annotations import argparse import hashlib -import importlib.metadata import json -import os -import subprocess -import sys +import math from pathlib import Path - -import numpy as np -from PIL import Image +from typing import Any REPO = Path(__file__).resolve().parents[2] ASSET_DIR = Path(__file__).with_name("fig_rl_assets") -LINEAGE = ( - ( - "L0--L3", - "pixcell-Qwen-Qwen3.6-35B-A3B-8g-8b-lr1e-05-20260708-0953", - 0, - ), - ( - "13-figure L4", - "pixcell-Qwen-Qwen3.6-35B-A3B-8g-8b-lr1e-05-20260708-1043", - 30, - ), - ( - "18-figure L4", - "pixcell-Qwen-Qwen3.6-35B-A3B-8g-8b-lr1e-05-20260708-1209", - 65, - ), +TELEMETRY_PATH = ASSET_DIR / "training_telemetry.json" +MANIFEST_PATH = ASSET_DIR / "manifest.json" +EVIDENCE_DIR = REPO / "data" / "training" / "qwen-rl-v2" +LEVELS = tuple(f"L{index}" for index in range(5)) +CHECKPOINTS = ("base", *(f"rl-l{index}" for index in range(5))) +STEPS_PER_LEVEL = 30 +CONTENTION_L3_SHA256 = ( + "ff01225caa2aaf372a949c00c7dc113e0ba340708221970263664fa3494a5d05" ) @@ -56,247 +45,322 @@ def _write_json(path: Path, value: object) -> None: path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") -def _git_commit(source_repo: Path) -> str: - return subprocess.check_output( - ["git", "-C", str(source_repo), "rev-parse", "HEAD"], - text=True, - ).strip() - - -def _load_training(source_repo: Path) -> tuple[dict, list[Path]]: - evaluations = [] - training = [] - source_paths = [] - for stage, run_directory, offset in LINEAGE: - metrics_path = ( - source_repo - / "rl" - / "reports" - / "runs_ledger" - / run_directory - / "metrics.jsonl" - ) - source_paths.append(metrics_path) - for line in metrics_path.read_text().splitlines(): - record = json.loads(line) - step = record.get("step") - if step is None: - continue - absolute_step = offset + int(step) - if record.get("env/L4/iou") is not None: - training.append( - { - "step": absolute_step, - "iou": float(record["env/L4/iou"]), - } - ) - if "test/env/all/iou" in record: - benchmark_values = [ - record.get(f"test/env/final_{index}/iou") - for index in range(1, 9) - ] - benchmark_values = [ - float(value) for value in benchmark_values if value is not None - ] - evaluations.append( - { - "step": absolute_step, - "benchmark": ( - float(np.mean(benchmark_values)) - if benchmark_values - else None - ), - **{ - f"L{index}": ( - float(record[f"test/env/L{index}/iou"]) - if record.get(f"test/env/L{index}/iou") is not None - else None - ) - for index in range(4) - }, - } - ) +def _read_json(path: Path) -> Any: + return json.loads(path.read_text()) - # The first record at a stage seam repeats the preceding checkpoint. - deduplicated = {} - for record in sorted(evaluations, key=lambda item: item["step"]): - deduplicated.setdefault(record["step"], record) - first_draw_path = ( - source_repo / "rl" / "reports" / "benchmark_eval_bridge2.json" - ) - source_paths.append(first_draw_path) - first_draw = json.loads(first_draw_path.read_text()) +def _finite_unit(value: Any, *, field: str) -> float: + result = float(value) + if not math.isfinite(result) or not 0.0 <= result <= 1.0: + raise ValueError(f"{field} must be finite and in [0, 1], got {value!r}.") + return result - return ( - { - "stages": [ - {"label": label, "start_step": offset} - for label, _run_directory, offset in LINEAGE - ], - "evaluations": list(deduplicated.values()), - "training": sorted(training, key=lambda item: item["step"]), - "final_draw": { - "eval_at_1": float(first_draw["bench_mean_of_means"]), - "best_at_8": float(first_draw["bench_mean_of_bests"]), - }, - }, - source_paths, - ) +def _load_training(run_root: Path) -> tuple[list[dict[str, Any]], list[Path]]: + records: list[dict[str, Any]] = [] + sources: list[Path] = [] + for level_index, level in enumerate(LEVELS): + path = run_root / "stages" / f"rl-l{level_index}" / "metrics.jsonl" + sources.append(path) + rows = [ + json.loads(line) + for line in path.read_text().splitlines() + if line.strip() + ] + if len(rows) != STEPS_PER_LEVEL: + raise ValueError( + f"{path} has {len(rows)} metric rows, expected {STEPS_PER_LEVEL}." + ) + for local_step, row in enumerate(rows, start=1): + records.append( + { + "step": level_index * STEPS_PER_LEVEL + local_step, + "level": level, + "mean_iou": _finite_unit( + row["env/all/iou"], + field=f"{level} step {local_step} mean_iou", + ), + "mean_shaped": _finite_unit( + row["env/all/reward/total"], + field=f"{level} step {local_step} mean_shaped", + ), + "executable_rate": _finite_unit( + row["env/all/program_valid"], + field=f"{level} step {local_step} executable_rate", + ), + } + ) + if [record["step"] for record in records] != list(range(1, 151)): + raise ValueError("Run B training steps are not the expected contiguous 1--150.") + return records, sources + + +def _summary_from_rows(rows: list[dict[str, Any]]) -> dict[str, Any]: + graded = [row for row in rows if row["shaped"] is not None] + if not graded: + raise ValueError("Probe contains no graded rows.") + return { + "n": len(rows), + "executable_rate": sum(bool(row["executable"]) for row in rows) / len(rows), + "mean_iou": sum(float(row["iou"]) for row in rows) / len(rows), + "mean_shaped": sum(float(row["shaped"]) for row in graded) / len(graded), + } -def _mask(image: Image.Image) -> np.ndarray: - return np.asarray(image.convert("L")) < 128 - - -def _load_gallery(source_repo: Path) -> tuple[dict, list[Path]]: - reports = source_repo / "rl" / "reports" - attempts_path = reports / "benchmark_attempts_bridge2.jsonl" - attempts = [ - json.loads(line) - for line in attempts_path.read_text().splitlines() - if line.strip() - ] - best = {} - grouped = {} - for attempt in attempts: - figure = attempt["figure"] - grouped.setdefault(figure, []).append(attempt) - if figure not in best or attempt["iou"] > best[figure]["iou"]: - best[figure] = attempt - - old_cwd = Path.cwd() - sys.path.insert(0, str(reports)) - sys.path.insert(0, str(source_repo / "src")) - os.chdir(source_repo) - try: - from make_gallery import render_code - - rows = [] - source_paths = [attempts_path] - for figure in sorted(best): - attempt = best[figure] - number = int(figure.removeprefix("final_")) - sample_directory = source_repo / attempt["sample_dir"] - target_source = sample_directory / "device_bw.png" - calibration_source = sample_directory / "calibration.json" - source_paths.extend((target_source, calibration_source)) - - target = Image.open(target_source).convert("L") - render = render_code(attempt["code"], attempt["sample_dir"]) - if render is None: - raise RuntimeError(f"Best archived attempt for {figure} did not render.") - render = render.convert("L") - if target.size != render.size: - raise ValueError( - f"Target/render dimensions differ for {figure}: " - f"{target.size} versus {render.size}." - ) - target_mask = _mask(target) - render_mask = _mask(render) - union = np.logical_or(target_mask, render_mask).sum() - measured_iou = float( - np.logical_and(target_mask, render_mask).sum() / union +def _assert_close(actual: Any, expected: Any, *, field: str) -> None: + if isinstance(expected, int): + if int(actual) != expected: + raise ValueError(f"{field}: {actual!r} != {expected!r}.") + return + if not math.isclose(float(actual), float(expected), rel_tol=0.0, abs_tol=1e-12): + raise ValueError(f"{field}: {actual!r} != {expected!r}.") + + +def _load_probes( + run_root: Path, + *, + model_name: str, + renderer_name: str, + max_tokens: int, +) -> tuple[list[dict[str, Any]], list[Path]]: + probes: list[dict[str, Any]] = [] + sources: list[Path] = [] + fixed_tasks: list[tuple[str, str]] | None = None + for checkpoint_index, checkpoint in enumerate(CHECKPOINTS): + path = run_root / "probes" / f"{checkpoint}.json" + sources.append(path) + report = _read_json(path) + rows = report["rows"] + if len(rows) != 80: + raise ValueError(f"{path} has {len(rows)} rows, expected 80.") + if ( + report["model_name"] != model_name + or report["renderer_name"] != renderer_name + or int(report["max_tokens"]) != max_tokens + or not math.isclose( + float(report["temperature"]), + 1.0, + rel_tol=0.0, + abs_tol=1e-12, ) - # The archived IoU is stored at two to four decimals, while this - # qualitative gallery threshold-binarizes antialiased rasters at - # mid-gray. Guard against a mismatched winner without pretending - # that the display composite is the scoring raster itself. - if not np.isclose(measured_iou, float(attempt["iou"]), atol=5.0e-2): + ): + raise ValueError(f"{path} does not use the fixed probe operating point.") + task_keys = [(row["opaque_id"], row["level"]) for row in rows] + if len(set(task_keys)) != 80: + raise ValueError(f"{path} contains duplicate fixed-probe tasks.") + if fixed_tasks is None: + fixed_tasks = task_keys + elif task_keys != fixed_tasks: + raise ValueError(f"{path} does not contain the same ordered 80-task probe.") + per_level: dict[str, dict[str, Any]] = {} + for level in LEVELS: + level_rows = [row for row in rows if row["level"] == level] + if len(level_rows) != 16: raise ValueError( - f"Re-rendered IoU {measured_iou:.6f} does not reconcile " - f"with archived IoU {attempt['iou']:.6f} for {figure}." + f"{path} has {len(level_rows)} {level} rows, expected 16." ) - - target_name = f"F{number}_target.png" - render_name = f"F{number}_render.png" - target_binary = Image.fromarray( - np.where(target_mask, 0, 255).astype(np.uint8), - mode="L", - ) - render_binary = Image.fromarray( - np.where(render_mask, 0, 255).astype(np.uint8), - mode="L", + computed = _summary_from_rows(level_rows) + stored = report["per_level"][level] + for field, expected in computed.items(): + _assert_close( + stored[field], + expected, + field=f"{checkpoint}.{level}.{field}", + ) + per_level[level] = { + "n": 16, + "executable_rate": _finite_unit( + stored["executable_rate"], + field=f"{checkpoint}.{level}.executable_rate", + ), + "mean_iou": _finite_unit( + stored["mean_iou"], + field=f"{checkpoint}.{level}.mean_iou", + ), + "mean_shaped": _finite_unit( + stored["mean_shaped"], + field=f"{checkpoint}.{level}.mean_shaped", + ), + } + computed_overall = _summary_from_rows(rows) + for field, expected in computed_overall.items(): + _assert_close( + report["overall"][field], + expected, + field=f"{checkpoint}.overall.{field}", ) - target_binary.save(ASSET_DIR / target_name, optimize=True) - render_binary.save(ASSET_DIR / render_name, optimize=True) - rows.append( - { - "device": f"F{number}", - "best_iou": float(attempt["iou"]), - "eval_at_1": float( - np.mean( - [ - float(candidate["iou"]) - for candidate in grouped[figure] - ] - ) + probes.append( + { + "checkpoint": checkpoint, + "step": checkpoint_index * STEPS_PER_LEVEL, + "condition": ( + "evaluation_contention" + if checkpoint == "rl-l3" + and _sha256(path) == CONTENTION_L3_SHA256 + else "nominal" + ), + "model_path": report["model_path"], + "overall": { + "n": 80, + "executable_rate": _finite_unit( + report["overall"]["executable_rate"], + field=f"{checkpoint}.overall.executable_rate", ), - "executable": sum( - bool(candidate.get("gds_ok")) - for candidate in grouped[figure] + "mean_iou": _finite_unit( + report["overall"]["mean_iou"], + field=f"{checkpoint}.overall.mean_iou", ), - "attempts": len(grouped[figure]), - "target": target_name, - "render": render_name, - } - ) - finally: - os.chdir(old_cwd) + "mean_shaped": _finite_unit( + report["overall"]["mean_shaped"], + field=f"{checkpoint}.overall.mean_shaped", + ), + }, + "per_level": per_level, + } + ) + return probes, sources + + +def _load_outcomes() -> tuple[dict[str, Any], list[Path]]: + base_path = EVIDENCE_DIR / "bench" / "base-nothink.json" + benchmark_path = EVIDENCE_DIR / "bench" / "b-final.json" + loop_path = EVIDENCE_DIR / "agentloop" / "b-final.json" + base = _read_json(base_path) + benchmark = _read_json(benchmark_path) + loop = _read_json(loop_path) + if benchmark["model_path"] != loop["model_path"]: + raise ValueError("Run B benchmark and iterative loop use different checkpoints.") + if base["model_path"] is not None: + raise ValueError("The base benchmark unexpectedly names an adapted checkpoint.") + if ( + len(base["per_target"]) != 8 + or int(base["overall"]["total"]) != 64 + or int(base["overall"]["executable"]) != 0 + ): + raise ValueError("The base F1--F8 benchmark is not the expected 0/64 record.") + if ( + len(benchmark["per_target"]) != 8 + or int(benchmark["overall"]["total"]) != 64 + or len(loop["per_target"]) != 8 + ): + raise ValueError("The Run B evaluation records do not cover eight targets.") + + per_target = loop["per_target"] + round_count = int(loop["rounds"]) + round_means = [] + for round_index in range(round_count + 1): + values = [ + float(target["rounds"][round_index]["champion_iou"]) + for target in per_target.values() + ] + round_means.append( + { + "round": round_index, + "mean_champion_iou": sum(values) / len(values), + } + ) + _assert_close( + round_means[-1]["mean_champion_iou"], + loop["overall_mean_champion"], + field="agent_loop.overall_mean_champion", + ) + return ( + { + "model_path": benchmark["model_path"], + "base_single_pass": base["overall"], + "single_pass": benchmark["overall"], + "iterative": { + "initial_attempts": int(loop["init_attempts"]), + "revision_rounds": round_count, + "revisions_per_round": int(loop["revisions_per_round"]), + "round_means": round_means, + "final_mean_champion_iou": float(loop["overall_mean_champion"]), + }, + }, + [base_path, benchmark_path, loop_path], + ) - return {"devices": rows}, source_paths + +def _source_entry(path: Path, root: Path) -> dict[str, str]: + return { + "path": str(path.relative_to(root)), + "sha256": _sha256(path), + } def main() -> None: parser = argparse.ArgumentParser() parser.add_argument( - "--source-repo", + "--run-root", type=Path, - default=REPO.parent / "PixCell-running", + required=True, + help="Path to the external runs-v2/runB-base-rl evidence directory.", ) arguments = parser.parse_args() - source_repo = arguments.source_repo.resolve() - if not (source_repo / "rl" / "reports").is_dir(): - raise FileNotFoundError( - f"RL evidence directory not found under {source_repo}." - ) - + run_root = arguments.run_root.resolve() + plan_path = run_root / "provenance" / "plan.json" + plan = _read_json(plan_path) + run_plan = plan["plan"] + if ( + run_plan["key"] != "B" + or run_plan["name"] != "runB-base-rl" + or run_plan["steps_per_level"] != STEPS_PER_LEVEL + or run_plan["sft"] + ): + raise ValueError("The supplied run root is not the completed raw-base Run B.") + + training, training_sources = _load_training(run_root) + probes, probe_sources = _load_probes( + run_root, + model_name=plan["model_name"], + renderer_name=run_plan["renderer_name"], + max_tokens=int(run_plan["max_tokens"]), + ) + outcomes, outcome_sources = _load_outcomes() + if probes[-1]["model_path"] != outcomes["model_path"]: + raise ValueError("Final probe and released-model outcomes use different checkpoints.") + + telemetry = { + "run": { + "key": "B", + "name": "runB-base-rl", + "model_name": plan["model_name"], + "renderer_name": run_plan["renderer_name"], + "max_tokens": int(run_plan["max_tokens"]), + "lora_rank": 32, + "steps_per_level": STEPS_PER_LEVEL, + "total_steps": 150, + "source_git_head": plan["git_head"], + "source_git_dirty": bool(plan["git_dirty"]), + }, + "levels": list(LEVELS), + "training": training, + "probes": probes, + "outcomes": outcomes, + } ASSET_DIR.mkdir(parents=True, exist_ok=True) - training, training_sources = _load_training(source_repo) - gallery, gallery_sources = _load_gallery(source_repo) - training_path = ASSET_DIR / "training_telemetry.json" - gallery_path = ASSET_DIR / "gallery.json" - _write_json(training_path, training) - _write_json(gallery_path, gallery) - - derived_paths = [training_path, gallery_path] - for row in gallery["devices"]: - derived_paths.extend( - (ASSET_DIR / row["target"], ASSET_DIR / row["render"]) - ) - source_paths = list(dict.fromkeys(training_sources + gallery_sources)) + _write_json(TELEMETRY_PATH, telemetry) + + run_sources = [plan_path, *training_sources, *probe_sources] manifest = { - "source_repository": "QPG-MIT/PixCell-running", - "source_commit": _git_commit(source_repo), - "runtime": { - "python": f"{sys.version_info.major}.{sys.version_info.minor}", - "gdsfactory": importlib.metadata.version("gdsfactory"), - }, - "source_files": [ - { - "path": str(path.relative_to(source_repo)), - "sha256": _sha256(path), - } - for path in source_paths + "record": "runs-v2/runB-base-rl", + "run_sources": [ + _source_entry(path, run_root) + for path in run_sources + ], + "repository_sources": [ + _source_entry(path, REPO) + for path in outcome_sources ], "assets": { - str(path.relative_to(ASSET_DIR)): {"sha256": _sha256(path)} - for path in derived_paths + TELEMETRY_PATH.name: { + "sha256": _sha256(TELEMETRY_PATH), + } }, } - _write_json(ASSET_DIR / "manifest.json", manifest) - print(f"wrote {ASSET_DIR / 'manifest.json'}") + _write_json(MANIFEST_PATH, manifest) + print(f"wrote {TELEMETRY_PATH}") + print(f"wrote {MANIFEST_PATH}") if __name__ == "__main__": diff --git a/arxiv/table_rl.tex b/arxiv/table_rl.tex index 06ee2274..95b1c781 100644 --- a/arxiv/table_rl.tex +++ b/arxiv/table_rl.tex @@ -1,27 +1,29 @@ \begin{table}[t] -\caption{\textbf{Isolated second-draw evaluation of the primary training -lineage.} Eight attempts per target at temperature $1.0$. Executable counts -programs that produce a scored GDS; mean IoU averages the eight attempts and -best of 8 retains the highest.} +\caption{\textbf{Run B evaluation on the training-excluded F1--F8 targets.} +The single-pass draw contains eight independent attempts per target at +temperature $1.0$. Mean IoU includes failed programs as zero and best of 8 +retains the highest. The iterative champion comes from a separate +eight-attempt draw followed by three rounds of four revisions under +number-free verifier feedback.} \label{tab:rl} \begingroup -\footnotesize -\setlength{\tabcolsep}{7.0pt} +\scriptsize +\setlength{\tabcolsep}{3.2pt} \renewcommand{\arraystretch}{1.06} \begin{ruledtabular} -\begin{tabular}{@{}lccc@{}} -Device & Executable & Mean IoU & Best of 8 \\ +\begin{tabular}{@{}lcccc@{}} +Device & Executable & Mean IoU & Best of 8 & Iterative \\ \colrule -F1 & 7/8 & 0.248 & 0.440 \\ -F2 & 7/8 & 0.396 & 0.599 \\ -F3 & 8/8 & 0.193 & 0.451 \\ -F4 & 5/8 & 0.347 & 0.637 \\ -F5 & 1/8 & 0.031 & 0.246 \\ -F6 & 5/8 & 0.215 & 0.557 \\ -F7 & 7/8 & 0.209 & 0.303 \\ -F8 & 8/8 & 0.393 & 0.497 \\ +F1 & 7/8 & 0.247 & 0.436 & 0.518 \\ +F2 & 8/8 & 0.429 & 0.523 & 0.507 \\ +F3 & 4/8 & 0.082 & 0.214 & 0.336 \\ +F4 & 5/8 & 0.314 & 0.663 & 0.651 \\ +F5 & 3/8 & 0.063 & 0.244 & 0.293 \\ +F6 & 3/8 & 0.202 & 0.643 & 0.716 \\ +F7 & 2/8 & 0.110 & 0.483 & 0.489 \\ +F8 & 7/8 & 0.375 & 0.526 & 0.418 \\ \colrule -All / mean & 48/64 & 0.254 & 0.466 \\ +All / mean & 39/64 & 0.228 & 0.467 & 0.491 \\ \end{tabular} \end{ruledtabular} \endgroup diff --git a/assets/pixcell-paper.pdf b/assets/pixcell-paper.pdf index 79746da7..25a46d62 100644 Binary files a/assets/pixcell-paper.pdf and b/assets/pixcell-paper.pdf differ