🛡️ ward

An event-driven guardian for your long-running commands

Prefix any command with ward. It watches, it reports when done, and it recovers when things break — without burning tokens while your job is healthy.

---

ward python train.py --epochs 100

That's the whole workflow. Put ward in front of the thing you'd otherwise sit and babysit — a training run, a data pipeline, a migration — and get on with your day.

When it's done, you get a message. When it breaks at 3am, ward is already diagnosing it.

---

✨ What you get

📣 It always reports A job that quietly finishes — or silently dies without a word — never slips past you again. ward hooks the process-exit event and emails you a summary plus the tail of the output.

♻️ It recovers On a crash or hang, a Codex-powered agent reads the error and — if you allow it — fixes the root cause and resumes the run, preferring a checkpoint restart over starting from zero.

🪶 It's token-frugal No model is called while your command is running. The AI wakes up only on an event, and only ever sees a bounded tail of the log. Idle runs cost you nothing.

---

🔍 Why it barely costs any tokens

ward is not an assistant tailing your logs in a loop. While your command runs, it's just a thin pseudo-terminal wrapper — live output preserved, exit code passed straight through, zero model calls. The OpenAI Codex CLI kernel is invoked only on an event:

flowchart LR
    A["ward &lt;command&gt;"] --> B{{"run in a pty<br/>(0 tokens)"}}
    B -->|exit 0| C["📣 report &amp; email"]
    B -->|exit ≠ 0| D["🤖 Codex: diagnose"]
    B -->|"no output N s (opt-in)"| D
    D -->|retry / fix / resume| B
    D -->|give up| E["❌ report failure"]

Loading

Event	What ward does	Model calls
Command exits 0	Send a completion report	0 (or 1 opt-in summary)
Command exits non-zero	Diagnose → fix → resume	1 per recovery attempt
No output for N seconds (opt-in)	Treat as a hang, stop it, recover	1 per attempt

The "hook" is the OS process-exit event (plus an optional local inactivity timer) — not a chat loop.

---

🚀 Install

You need exactly two things:

1. an OpenAI account (to sign the Codex CLI in), and
2. an email address for reports.

git clone https://github.com/xubinrui/trainer.git
cd trainer
./install.sh

install.sh installs the ward command (via pipx if available, else pip --user) and launches the interactive setup, which will:

1. 🔑 check for the Codex CLI and offer to run codex login (your OpenAI auth);
2. ✉️ ask for your email + optional SMTP details (Gmail is auto-detected);
3. ⚠️ ask what the recovery agent may do — behind a risk disclaimer.

Prerequisites: Python ≥ 3.11 and the OpenAI Codex CLI (npm install -g @openai/codex or brew install codex). ward calls codex as a subprocess — it never stores your OpenAI credentials; those stay with Codex.

Re-run ward install anytime to change settings, and ward doctor to check them.

---

⚠️ Permissions & risk — read this

Recovery works by letting the Codex agent act on your machine without asking for per-action approval (there's no human awake for an overnight job). You choose how much power it gets, and the elevated levels require you to type I ACCEPT during setup:

Level	Codex may…	Risk
🟢 read-only (default)	read files & reason; retry / resume your command	Low — cannot change your files
🟡 workspace-write	edit files in the working dir + run commands there	Medium — it can change your code
🔴 danger-full-access	read/write anywhere, run any command, network	High — can delete data or leak secrets

You are solely responsible for what the agent does under the level you grant. ward ships with no warranty (see LICENSE). Start with read-only; only raise it for directories you trust the agent to modify.

---

🎛️ Commands

ward <command> [args...]      Supervise a command (bare form).
ward run -- <command> ...     Explicit form (use if the command name collides
                              with a ward subcommand).
ward install                  Interactive setup.
ward doctor                   Diagnose the current configuration.
ward config [--show]          Print resolved config (password redacted).
ward version

ward python train.py --epochs 100     # the common case
ward -- bash long_job.sh              # explicit form

Per-run logs are kept under ~/.local/state/ward/logs/.

---

⚙️ Configuration

Stored at ~/.config/ward/config.toml (chmod 600 — it may hold an SMTP app-password). View the resolved config with ward config.

[notify]
email = "you@example.com"
smtp_host = "smtp.gmail.com"
smtp_port = 587
smtp_user = "you@example.com"
smtp_pass = "app-password"     # a Gmail App Password, NOT your login password
smtp_tls = true
desktop = true

[agent]
enabled = true
codex_bin = "codex"
sandbox = "read-only"          # read-only | workspace-write | danger-full-access
model = ""                     # blank = Codex default
report_ai_summary = false      # add an AI paragraph to SUCCESS reports too
max_context_bytes = 16384      # output tail handed to the agent

[recovery]
enabled = true
max_retries = 2                # fix-and-resume attempts before giving up
retry_backoff = 20             # seconds between attempts
hang_timeout = 0               # 0 = off; else seconds of silence = "hung"

📧 Email delivery notes

• With Gmail, use an App Password (Google Account → Security → App Passwords), not your normal password, and host smtp.gmail.com:587.
• If you leave smtp_host blank, ward falls back to a local sendmail if one exists; otherwise it just prints the report and shows a desktop notification.

---

🧠 How recovery decides what to do

On failure, ward hands the agent the command, the working directory and the output tail, and asks for a single structured decision:

WARD_DECISION: {
  "action": "resume",
  "command": "python train.py --resume last.ckpt",
  "reason":  "training crashed on an OOM; resuming from the last checkpoint",
  "summary": "Lowered batch size and resumed."
}

Action	Meaning
retry	transient failure — re-run unchanged
fix_and_retry	the agent edited files to fix the cause — re-run
resume	re-run using a new command (e.g. add --resume-from-checkpoint)
give_up	not safely fixable — ward reports and stops

ward loops until success or max_retries, then notifies you with either a recovered ♻️ or failed ❌ report.

---

🧪 Development

python -m pytest tests/ -v        # run the test suite
python -m ward_agent doctor       # check your setup

Releasing to PyPI / Homebrew / GitHub / standalone binaries is documented in RELEASING.md.

---

MIT licensed. No warranty — read the permissions section before granting write access.

_{Built to babysit the jobs you shouldn't have to.}