feat: persistent session state and experiment attempt logging for session execution agent (experiment runner agent))

[bimu233]

Summary

Adds durable inter-phase memory to the session executor so that interrupted sessions (rate limit, token exhaustion, 5-hour Claude Code session limit) can resume without repeating completed work or re-encountering already-fixed bugs.

Problem

The session executor runs all 6 phases in a single continuous session. When that session is interrupted mid-run, all in-session context is lost. On the next run, the agent restarts from Phase 0, repeats phases it already completed, and re-discovers errors it already fixed — wasting tokens and time. This is common when users share their Claude Code session with other work.

Changes

templates/base/researcher.txt

• Added RESUME CHECK block at the top: agent reads STATE.md immediately on startup, checks the phase status table, and skips all DONE phases. Writes a session-start Worklog entry before any other action so the session is recorded even if interrupted again.
• Added mandatory CHECKPOINT writes at each phase boundary: phase status, worklog entry, and section-specific content (files, results, design decisions).
• Added per-attempt experiment logging protocol (STEP A/B/C): agent writes Status: RUNNING to STATE.md before each experiment run, then updates to FAILED (with exact error and fix) or SUCCESS (with metrics and output files) immediately after. A hard constraint blocks code changes until the current attempt entry is resolved — preventing the agent from batching all error writes to the end of the session.

templates/base/deliverables/state_template.md

• Added STATE.md template with structured sections: # Current State (phase status table), # Worklog, # Research Specification, # Files and Resources, # Workflow, # Experiment Design, # Learnings, # Experiment Results, # Experiment Attempts.
• # Experiment Attempts is the append-only per-run log. Each entry records the command, error, fix applied, and output files — giving the agent a complete anti-repetition record on resume.

Why researcher.txt

researcher.txt is always injected at {{ prompt }} by Jinja2 regardless of domain override. Placing STATE.md write instructions here means all domains get consistent phase logging without needing to duplicate instructions in every domains/X/session_instructions.txt.

Verified

Tested on L2 regularization experiment. STATE.md was updated in-time at each phase transition and after each failed attempt — including exact error messages, fixes applied, and metric values. Confirmed successful resume from correct phase without repeating prior work.

[jingxuangu]

Tested this end-to-end (interrupt + resume on the ml_regularization_test idea). The resume logic itself works — but as-is the PR is a no-op against the current Docker image, and actively misleads the agent. Requesting changes on one item.

🔴 Blocker: runner.py change doesn't run, so STATE.md is never created
docker/run.sh (cmd_run) bind-mounts templates/ from the host but not src/ — runner.py runs from the baked-in chicagohai/neurico:latest image, which predates this PR. Net effect on a normal ./neurico run:

✅ researcher.txt changes go live (templates are mounted) → the agent is told "STATE.md is present in your workspace (created by the pipeline)"
❌ the runner.py STATE.md-creation code never executes (stale image) → STATE.md does not exist
So the agent gets a prompt that asserts a file exists when it doesn't. I confirmed this: a stock run produced no STATE.md and never printed Initialized STATE.md. Only after I mounted src/ into the container did STATE.md get created and the feature work.

Please make sure this PR is accompanied by a Docker image rebuild (and note it in the description), otherwise it ships as "prompt promises a file that's never there."

Also, the creation code silently no-ops if the template is missing:

if state_template.exists():
state_md.write_text(...)
Please add an else that prints a clear warning, so a missing/mis-pathed template is visible instead of producing the same "STATE.md is present but isn't" contradiction.

🟡 Minor: phase-status table can lag the prose sections
During Phase 0, the agent had already filled in Phase-1/2-level content (full Research Spec, Experiment Design, a built .venv) while the Phase Status table still read all pending — and that table is exactly what the RESUME CHECK parses. It self-corrected once Phase 0 completed and the interrupt/resume was clean, so not a blocker — just flagging that the resume signal depends on the agent keeping that table in lockstep, which it doesn't always do mid-phase.

✅ What works
After the image issue was worked around, the full cycle behaves exactly as intended. I ran to Phase 0/1/2 DONE, Phase 3 in progress, killed the container, and re-ran the same idea. On resume the agent read STATE.md, recognized "a prior session got through Phase 2 and started Phase 3," skipped motivation/planning/setup (reusing planning.md and the existing .venv), and resumed at Phase 3. Nice — the core mechanism is solid.

[bimu233]

Thanks for catching that. The issue only happens when using ./neurico option, there's no bug when using uv run and docker run. I will fix it for ./neurico option.