Repository navigation
RL Studio: rename Developer Mode, add fine-tuning and a feedback loop - #299
Conversation
…#294) Developer Mode becomes RL Studio. The /api/dev/* prefix is kept. Beyond pretraining, RL Studio now records a work log of answers and the feedback a person gives them, builds a dataset from that log, and runs a fine-tune round (LoRA, SFT or DPO through mlx-lm) on it. The feedback signal is the person's or code's, never the model grading itself (ADR 0063): - Thumbs up/down and an "edit as the better answer" control under each answer. A click records a work-log entry and the feedback on it; no model is asked which answer was better. - A new rl_work_log table in the session store holds the prompt, the verbatim answer, tool calls with code-derived status, and the feedback. It is never read as evidence: not by recall::index_one, the memory block, titling or branching. A test pins that the recorder writes no messages rows, and it clears with the conversations (state.rs). - zorp_agent::rl_dataset::build turns the log into an SFT set and a preference set in code. A thumbs-down with no edit produces nothing, because it says the answer was wrong, not what the right one was. Fine-tune plumbing: FineTuneJobConfig, the supervisor's start_finetune sharing one monitored-launch path with pretraining, and mlx_finetune.py, which reports the dataset and example count in its init event and refuses a run with no data. New routes /api/dev/worklog*, /api/dev/dataset/preview and /api/dev/finetune/start behind the train feature; a Fine-tune tab and a Feedback tab in RL Studio; per-answer feedback controls gated on the new rl_studio capability. No tool starts a run, trains, or serves a checkpoint: no_tool_trains_a_model_or_serves_a_checkpoint still passes. Where this diverges from the issue: fine-tune execution needs mlx-lm on Apple Silicon, which CI and a Linux box cannot run, so the Rust plumbing is unit-tested at the manifest/route level and the training itself is verified on a Metal machine. The Mac app's RL Studio routes are a follow-up; the rename reaches its menu and window title. ADR 0143 and a spec in docs/superpowers/specs/ added; CLAUDE.md / AGENTS.md zorp-train bullet updated. cargo test -p zorp-train, cargo test -p zorp-web --features train, cargo test -p zorp-agent, and the web/ check, test and build all pass. Closes #294 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
Adding idx_rl_work_log_session took the per-session index count from 3 to 4, which open_creates_session_indexes_even_on_old_databases pins. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
Spawning a just-written executable can briefly fail with Text file busy (errno 26) when a sibling thread's fork holds a write handle on it, which flaked test_supervisor_spawn_lifecycle_and_events on Linux CI. Retry the spawn a couple of times on that one errno; anything else still fails at once. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
|
CI status on the current head ( The one non-green check is I can't re-run the job myself (the GitHub App lacks Generated by Claude Code Generated by Claude Code |
set_work_log_feedback wrote both feedback and edited_answer unconditionally, but the surfaces send one signal at a time (a thumb, or a "better answer" edit). A thumbs-down after an edit, or an edit after a thumb, arrived with the other field None and NULLed it out, destroying a preference pair the feature exists to collect. Update each column with COALESCE so a None argument leaves the stored value, and add a test for both orders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
# Conflicts: # docs/DECISIONS.md # web/src/api.ts # web/src/main.ts # zorp-train/src/supervisor.rs
Closes #294
What this does
Developer Mode becomes RL Studio (the
/api/dev/*prefix is kept). Beyond pretraining, RL Studio now records a work log of answers and the feedback a person gives them, builds a dataset from that log, and runs a fine-tune round (LoRA / SFT / DPO throughmlx-lm) on it.The feedback-signal rule (ADR 0063) — the heart of this
The signal is the person's or code's, never the model grading itself:
rl_work_logtable in the session store holds the prompt, the verbatim answer, tool calls with code-derived status, and the feedback. It is never read as evidence — not byrecall::index_one, the memory block, titling or branching. A store test pins that the recorder writes nomessagesrows, and it clears with the conversations (state.rs).zorp_agent::rl_dataset::buildturns the log into an SFT set and a preference set in code. A thumbs-down with no edit produces nothing, because it says the answer was wrong, not what the right one was — inventing one would be the model-authored signal the rule forbids.Fine-tune plumbing
FineTuneJobConfig; the supervisor'sstart_finetune, sharing one monitored-launch path with pretraining;mlx_finetune.py, which reports the dataset and example count in itsinitevent (the same "what did this train on" metric) and refuses a run with no data./api/dev/worklog*,/api/dev/dataset/preview,/api/dev/finetune/startbehind thetrainfeature. A Fine-tune tab and a Feedback tab in RL Studio. Per-answer feedback controls gated on a newrl_studiocapability.Rules kept
no_tool_trains_a_model_or_serves_a_checkpointstill passes.Where the code and the issue differ (per the instruction to say so)
mlx-lmon Apple Silicon, which CI and a Linux box cannot run. So the Rust plumbing (manifest, supervisor command, dataset builder, routes) is unit-tested at the manifest/route level, and the training itself is verified on a Metal machine.mlx_finetune.pyis written against mlx-lm's tuner API and reports a clear error where mlx-lm or Metal is absent, rather than pretending.Tests
cargo test -p zorp-agent(work-log store round-trip + the keeps-it-out-of-messagesassertion,rl_datasetbuilder),cargo test -p zorp-train(fine-tune config),cargo test -p zorp-web --features train(worklog → feedback → dataset preview, and the empty-fine-tune refusal) all pass.web/:npm run check,npm test(incl. newrl-feedbackcases and the renamed developer-mode test),npm run build:sitepass.cargo fmt --all --checkandcargo clippyclean for the touched crates.CLAUDE.md/AGENTS.mdzorp-trainbullet updated.Generated by Claude Code