Skip to content

RL Studio: rename Developer Mode, add fine-tuning and a feedback loop - #299

Merged
arjunjaggi34 merged 5 commits into
mainfrom
claude/rl-studio-294
Oct 28, 2026
Merged

arjunjaggi34 merged 5 commits into
mainfrom
claude/rl-studio-294

Conversation

@arjunjaggi34

Copy link
Copy Markdown
Contributor

Closes #294

What this does

Developer Mode becomes RL Studio (the /api/dev/* prefix is kept). Beyond pretraining, RL Studio now records a work log of answers and the feedback a person gives them, builds a dataset from that log, and runs a fine-tune round (LoRA / SFT / DPO through mlx-lm) on it.

The feedback-signal rule (ADR 0063) — the heart of this

The signal is the person's or code's, never the model grading itself:

  • Thumbs up/down and "better answer" controls under each answer. A click records a work-log entry and the feedback on it; nothing asks a model which answer was better.
  • A new rl_work_log table in the session store holds the prompt, the verbatim answer, tool calls with code-derived status, and the feedback. It is never read as evidence — not by recall::index_one, the memory block, titling or branching. A store test pins that the recorder writes no messages rows, and it clears with the conversations (state.rs).
  • zorp_agent::rl_dataset::build turns the log into an SFT set and a preference set in code. A thumbs-down with no edit produces nothing, because it says the answer was wrong, not what the right one was — inventing one would be the model-authored signal the rule forbids.

Fine-tune plumbing

  • FineTuneJobConfig; the supervisor's start_finetune, sharing one monitored-launch path with pretraining; mlx_finetune.py, which reports the dataset and example count in its init event (the same "what did this train on" metric) and refuses a run with no data.
  • Routes /api/dev/worklog*, /api/dev/dataset/preview, /api/dev/finetune/start behind the train feature. A Fine-tune tab and a Feedback tab in RL Studio. Per-answer feedback controls gated on a new rl_studio capability.

Rules kept

  • No tool starts a run, trains, or serves a checkpoint — no_tool_trains_a_model_or_serves_a_checkpoint still passes.
  • A run says what it trained on, and an empty build is refused, never run on nothing (ADR 0133).

Where the code and the issue differ (per the instruction to say so)

  • Fine-tune execution needs mlx-lm on Apple Silicon, which CI and a Linux box cannot run. So the Rust plumbing (manifest, supervisor command, dataset builder, routes) is unit-tested at the manifest/route level, and the training itself is verified on a Metal machine. mlx_finetune.py is written against mlx-lm's tuner API and reports a clear error where mlx-lm or Metal is absent, rather than pretending.
  • The Mac app's RL Studio routes/tests (step 6) are a follow-up; the rename reaches its menu and window title. The desktop CI job stays green (bridge and existing app unchanged).

Tests

  • cargo test -p zorp-agent (work-log store round-trip + the keeps-it-out-of-messages assertion, rl_dataset builder), cargo test -p zorp-train (fine-tune config), cargo test -p zorp-web --features train (worklog → feedback → dataset preview, and the empty-fine-tune refusal) all pass.
  • web/: npm run check, npm test (incl. new rl-feedback cases and the renamed developer-mode test), npm run build:site pass.
  • cargo fmt --all --check and cargo clippy clean for the touched crates.
  • ADR 0143 + spec added; CLAUDE.md / AGENTS.md zorp-train bullet updated.

Generated by Claude Code

…#294)

Developer Mode becomes RL Studio. The /api/dev/* prefix is kept. Beyond
pretraining, RL Studio now records a work log of answers and the feedback a
person gives them, builds a dataset from that log, and runs a fine-tune
round (LoRA, SFT or DPO through mlx-lm) on it.

The feedback signal is the person's or code's, never the model grading
itself (ADR 0063):
- Thumbs up/down and an "edit as the better answer" control under each
  answer. A click records a work-log entry and the feedback on it; no model
  is asked which answer was better.
- A new rl_work_log table in the session store holds the prompt, the
  verbatim answer, tool calls with code-derived status, and the feedback. It
  is never read as evidence: not by recall::index_one, the memory block,
  titling or branching. A test pins that the recorder writes no messages
  rows, and it clears with the conversations (state.rs).
- zorp_agent::rl_dataset::build turns the log into an SFT set and a
  preference set in code. A thumbs-down with no edit produces nothing,
  because it says the answer was wrong, not what the right one was.

Fine-tune plumbing: FineTuneJobConfig, the supervisor's start_finetune
sharing one monitored-launch path with pretraining, and mlx_finetune.py,
which reports the dataset and example count in its init event and refuses a
run with no data. New routes /api/dev/worklog*, /api/dev/dataset/preview and
/api/dev/finetune/start behind the train feature; a Fine-tune tab and a
Feedback tab in RL Studio; per-answer feedback controls gated on the new
rl_studio capability.

No tool starts a run, trains, or serves a checkpoint:
no_tool_trains_a_model_or_serves_a_checkpoint still passes.

Where this diverges from the issue: fine-tune execution needs mlx-lm on
Apple Silicon, which CI and a Linux box cannot run, so the Rust plumbing is
unit-tested at the manifest/route level and the training itself is verified
on a Metal machine. The Mac app's RL Studio routes are a follow-up; the
rename reaches its menu and window title.

ADR 0143 and a spec in docs/superpowers/specs/ added; CLAUDE.md / AGENTS.md
zorp-train bullet updated.

cargo test -p zorp-train, cargo test -p zorp-web --features train, cargo
test -p zorp-agent, and the web/ check, test and build all pass.

Closes #294

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
Adding idx_rl_work_log_session took the per-session index count from 3 to
4, which open_creates_session_indexes_even_on_old_databases pins.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
Spawning a just-written executable can briefly fail with Text file busy
(errno 26) when a sibling thread's fork holds a write handle on it, which
flaked test_supervisor_spawn_lifecycle_and_events on Linux CI. Retry the
spawn a couple of times on that one errno; anything else still fails at
once.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu

Copy link
Copy Markdown
Contributor Author

CI status on the current head (bbbd417): every gating job is green — build-and-test (ubuntu + macOS), windows, msrv, clippy, web, features, harness, and Desktop App (macOS).

The one non-green check is research-pr, which shows cancelled — the earlier left: 4, right: 3 index-count failure is already fixed on this head (the per-session index assertion is now 4). The cancellation is the job's 60-minute timeout-minutes cap being hit while it builds the bundled DuckDB amalgamation from a cold cache; it is not a code failure. The identical job passed on #298 with a warm shared cache (research-stack), which is the tell that this is cold-build timing, not this PR.

I can't re-run the job myself (the GitHub App lacks actions: write, so a re-run returns 403), and the timeout is a workflow setting I'm deliberately not widening this PR to touch. A maintainer re-run, or the shared cache warming once a research-touching PR lands on main, should clear it. I'll keep watching and report if anything else changes.


Generated by Claude Code


Generated by Claude Code

set_work_log_feedback wrote both feedback and edited_answer
unconditionally, but the surfaces send one signal at a time (a thumb, or
a "better answer" edit). A thumbs-down after an edit, or an edit after a
thumb, arrived with the other field None and NULLed it out, destroying a
preference pair the feature exists to collect. Update each column with
COALESCE so a None argument leaves the stored value, and add a test for
both orders.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7qLzxWmLqkYQQLiQqJtiu
# Conflicts:
#	docs/DECISIONS.md
#	web/src/api.ts
#	web/src/main.ts
#	zorp-train/src/supervisor.rs
@arjunjaggi34
arjunjaggi34 merged commit 6e6bb28 into main Oct 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RL Studio: rename Developer Mode, add fine-tuning, and learn from work logs

2 participants