diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c29d7678..600b8eb2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -43,6 +43,14 @@ jobs: runs-on: ubuntu-24.04 container: image: mcr.microsoft.com/devcontainers/rust:2.0.13-1-bookworm + # --init: a job container's PID 1 is otherwise `tail -f /dev/null`, which + # never reaps orphans, so a killed agent's orphaned children stay + # zombies for the whole job. scripts/check-in-container.sh's PID 1 is + # bash, which reaps, and so does any real host. Without this the suite + # runs under a PID 1 neither of them has (PR #225: 18 failures, all + # zombies). The tests' liveness probes skip zombies too; this keeps CI + # matching the local gate rather than relying on that alone. + options: --init timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -96,6 +104,8 @@ jobs: runs-on: ubuntu-24.04 container: image: mcr.microsoft.com/devcontainers/rust:2.0.13-1-bookworm + # --init: see the Test job. + options: --init timeout-minutes: 45 # REQUIRED, not stylistic. A `run:` step in a CONTAINER job defaults to # `sh -e {0}`, not `bash -e {0}` as it would on a bare runner — and this diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 9a45755c..86f4e01d 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -220,12 +220,27 @@ files per gated stage, all written atomically (temp file + rename): (`Gates::ack()`) once the response is consumed, so a poller knows to clean up. -`run_gate()` (`crates/devflow-cli/src/main.rs`) writes the gate, fires the -notify hook (`fire_gate_notify()` — runs `$DEVFLOW_GATE_NOTIFY_CMD` via -`sh -c` with gate metadata as environment variables, never interpolated +`run_gate()` (`crates/devflow-cli/src/pipeline_gate.rs`) writes the gate, +fires the notify hook (`fire_gate_notify()` — runs `$DEVFLOW_GATE_NOTIFY_CMD` +via `sh -c` with gate metadata as environment variables, never interpolated into the command string; a silent no-op when the variable is unset), then -blocks in `Gates::poll_response()` with exponential backoff (1s → 2s → … -capped at 60s) until a response appears or `DEVFLOW_GATE_TIMEOUT_SECS` +waits in `wait_for_gate_response()` (`crates/devflow-cli/src/pipeline_launch.rs`), +which polls the response file one of two ways: + +- **Inside a supervisor's advance** (its stop token is engaged, 50-13): the + run's owner reads `gate_wait` in a `ping` reply for the length of the wait, + and `Gates::poll_response_cancellable()` reads the response file about once + a second (every sleep is capped at 1s) and checks the stop token before each + sleep. A stop — a `shutdown` request on the supervisor's socket, or + SIGTERM/SIGINT to the supervisor — ends the wait within about a second: the + gate's files are removed, `stopped` is recorded in the state, which is kept + for `devflow resume` (not an abort), and the advance unwinds without acting + on the gate. A response already on disk wins over a stop. +- **Anywhere else** (no stop token: the hidden `devflow advance` run from a + shell, or a gate a foreground command fires): `Gates::poll_response()` with + exponential backoff (1s → 2s → … capped at 60s), which nothing cancels. + +Either wait ends when a response appears or `DEVFLOW_GATE_TIMEOUT_SECS` (default 259200 = 3 days, held equal to `DEVFLOW_GATE_MAX_UNATTENDED_AGE_SECS` so `devflow gate sweep` cannot reap a gate this poll is still watching) elapses. `GateAction::from_response()` turns an @@ -237,24 +252,47 @@ skipping unparsable entries rather than failing. ## Monitor daemon `crates/devflow-core/src/monitor.rs` is the core automation primitive — no -cron, no scheduler, no agent cooperation. `spawn_monitor()` spawns a -**detached child process that owns the agent**: - -1. Launches the agent (`program` + `args`), redirecting stdout/stderr to - separate phase capture files (stderr is kept out of the stdout capture so - it can never corrupt the `DEVFLOW_RESULT` parse) and recording the agent - PID. -2. Waits for the agent to exit and records its exit code to - `.devflow/phase-NN-exit`. -3. Runs `devflow advance --phase N` to advance the stage machine. - The phase is threaded in at spawn time, so `advance`'s identity never - depends on a shared state singleton — under `devflow parallel`, each - phase's monitor advances exactly its own stage. - -Because the monitor outlives the `devflow start`/`devflow advance` -invocation that spawned it, agent stdout keeps flowing into the capture -file and the exit code is still reaped after the CLI process returns. -`spawn_monitor()` is the single way an agent process is spawned (23d). +cron, no scheduler, no agent cooperation. `spawn_monitor()` re-execs the CLI +as its hidden `__monitor` subcommand, **a detached supervisor process in its +own process group that owns the agent**, for both launch shapes: + +- **capture** (`MonitorLaunch::Legacy`): the agent's stdin is `/dev/null` and + its stdout and stderr go straight to the phase capture files (stderr is kept + out of the stdout capture so it can never corrupt the `DEVFLOW_RESULT` + parse). Non-Claude adapters other than Antigravity, the legacy Claude launch + opt-out and a checkpoint resume use it. +- **stream** (`MonitorLaunch::PipeOwning`): the supervisor owns the agent's + stdin and stdout pipes, delivers the prompt as the first turn and applies the + idle timeout. Claude and Antigravity `stream-json` launches use it. + +The supervisor (`run_supervised_stage()`): + +1. Spawns the agent (`program` + `args`) in the agent's own process group and + records its PID. +2. Publishes a per-launch Unix socket under `registry::cache_dir()/sock/`, + with a JSON sidecar beside it naming the launch (project root, phase, + agent process group, start time, boot id). The socket path is recorded in + the phase's `state.json` as `state.supervisor`; liveness (ALIVE / STALE / + GONE) is answered by probing that path, never from a pid. +3. Answers `ping` and `shutdown` on the socket, and treats SIGTERM and SIGINT + sent to it as a `shutdown`: the agent's group is ended and no advance runs. +4. Waits for the agent to exit and records its exit code to + `.devflow/phase-NN-exit`. A stream supervisor stops reading when the agent + exits, not when its stdout closes: when a process the agent started still + holds that stdout, it waits at most 3 s for its pipe threads, closes the + capture file, records a `stream_wait_bounded` event, and signals nothing. +5. On the agent's natural exit, runs the advance in-process. The phase is + threaded in at spawn time, so the advance's identity never depends on a + shared state singleton — under `devflow parallel`, each phase's supervisor + advances exactly its own stage. When the advance launches the next stage, + that stage's supervisor publishes its socket before this one unlinks its + own, so the hand-off leaves no gap with neither socket published. + +`spawn_monitor()` returns once the new supervisor answers a ping with this +launch's id. Because the supervisor outlives the CLI process that launched +it, agent stdout keeps flowing into the capture file and the exit code is +still reaped after that process returns. `spawn_monitor()` is the single way +an agent process is spawned (23d). ## Worktree model diff --git a/Cargo.lock b/Cargo.lock index f707ba5c..93d6cf37 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -143,6 +143,7 @@ dependencies = [ "clap", "devflow-core", "insta", + "libc", "serde_json", "tempfile", "thiserror", diff --git a/OPERATIONS.md b/OPERATIONS.md index 38e074ac..de3857a6 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -30,7 +30,7 @@ Define → Plan → Code → Validate → Ship |---|---| | `devflow start --phase N --agent claude\|codex\|opencode --mode auto\|supervise [--force] [--no-worktree] [--dry-run] [--until define\|plan\|code\|validate]` | Begin a phase; a detached monitor owns the agent and auto-advances. `--until ` runs through `` then halts cleanly (no orphaned monitor, no `doctor` false positive) instead of continuing to Ship; `--until ship` is rejected (a no-op — Ship is already terminal). Refuses before scaffolding anything if phase N is not reachable from `develop` (see Preflight refusals below) | | `devflow resume --phase N` | Resume a rate-limited or infra-paused phase from its saved stage (loads `state-NN.json`, does not recreate the branch/worktree or reset to Define) — the command a primary-loop rate-limit auto-resume cron invokes. Also clears a `--until`-stopped phase's stop marker, so the phase advances past its old stop point instead of re-stopping immediately | -| `devflow status` | All active phases: stage, gate state, agent liveness, elapsed, last action | +| `devflow status` | All active phases: stage, gate state, supervisor liveness (ALIVE / STALE / GONE, with the repair; see [Run liveness](#run-liveness)), elapsed, last action | | `devflow logs [-f] [--phase N] [--stderr]` | Print/follow a phase's captured agent output | | `devflow history [N]` | Show chronological events with retained capture and review evidence for a phase | | `devflow gate list` | Gates awaiting a response | @@ -40,23 +40,126 @@ Define → Plan → Code → Validate → Ship | `devflow parallel --phases 7,8 [--agents claude,codex] [--mode M] [--force]` | Run phases concurrently, each in its own worktree + monitor | | `devflow list` | Feature branches with divergence from develop | | `devflow reference [--branch B] [--refresh]` | Static snapshot worktree at `.worktrees/reference/` | -| `devflow stop --phase N [PROJECT] [--root PATH]` | End a running phase cleanly (23c). Answers its open gate with a rejection if one is open — the target unwinds through its own abort path, no signal sent — otherwise signals the process recorded in `.devflow/lock-{phase:02}` (never `state.monitor_pid`, which the generated monitor script's trap only ever captures for the agent, not the trailing `advance`) after confirming it is alive and its `/proc//cmdline` identifies it as belonging to DevFlow. Idempotent; marks `state.stopped`/`state.stop_reason` so `cleanup`'s existing fail-closed refusal (unweakened by this command) recognizes the phase as no longer live — `stop` then `cleanup --force` compose in that order | +| `devflow stop --phase N [PROJECT] [--root PATH] [--launch ID]` | End a running phase cleanly (23c). In this order: **(1)** an open gate with a confirmed live waiter is answered with a rejection — the waiter unwinds through its own abort path, no signal sent. **(2)** Otherwise the run's supervisor socket is probed: the one `.devflow/state-NN.json` records, or, when that file records none (a `start` still waiting for its supervisor to publish), is missing or does not load, the one live socket found through its sidecar for this project and phase. **ALIVE** → a `shutdown` request: the supervisor ends the agent's process group (exit code 143) and runs no advance. **STALE** → the guarded backstop: it signals (SIGTERM, then SIGKILL) only an agent group whose leader is still present with the recorded start time and still leads the group; a socket from a previous boot, an empty group or a recycled leader pid has its socket and sidecar removed with nothing signalled; a group whose leader is gone is refused and its members listed, with `ps -o pid,pgid,sid,lstart,args -p "$(pgrep -d, -g )"` to inspect them (residual limit: nothing proves those members are the agent's, so `stop` never signals them — end them yourself). A probe that cannot answer is refused, nothing signalled. **(3)** Only when no live socket remains (**GONE**, or nothing recorded) is the process holding `.devflow/lock-{phase:02}` signalled, after confirming it is alive and its pid and start time match the identity the lock recorded (a recycled pid, or a lock that records no start time, is refused). No pid read from the state file is ever signalled. Idempotent; records `stopped`/`stop_reason`, except after arm (1) while the gate's waiter still holds the phase lock: that waiter ends the phase through its abort path, which clears the state instead. Either way `cleanup`'s fail-closed refusal recognizes the phase as no longer live — `stop` then `cleanup --force` compose in that order. When the state file does not load, `stop` still ends the run, says `stopped` was not recorded and leaves the file untouched; several live sockets with no usable state are refused, with one `devflow stop --phase N --launch ` line each. `--launch ` tears down only the socket with that launch id — the form `devflow doctor` prints for an orphan, usable from any directory — and answers no gate, signals no lock holder and writes no state | | `devflow cleanup [--force]` | Remove phase worktrees + their feature branches | -| `devflow recover [--clean] [--phase N]` | Inspect state; `--clean` sweeps stale phases only; `--clean --phase N` clears one phase unconditionally | +| `devflow recover [--clean] [--phase N]` | Inspect state; `--clean` sweeps stale phases only; `--clean --phase N` clears one phase, refusing while its supervisor may be live (ALIVE, a STALE socket whose agent group may live, or a failed probe) | | `devflow test` | cargo test + clippy + fmt --check | -| `devflow doctor [--json]` | Environment audit (agents installed, versions, RUST_LOG) plus per-phase reconciliation; `--json` emits one object `{"environment": [...], "reconciliation": [...]}` | +| `devflow doctor [--json]` | Environment audit (agents installed, versions, RUST_LOG) plus per-phase reconciliation, including each phase's supervisor liveness and a read-only sweep for orphan supervisor sockets (see [Run liveness](#run-liveness)); `--json` emits one object with the keys `environment`, `reconciliation`, `planning_doc_staleness`, `stray_processes`, `orphan_supervisors` and `orphan_supervisor_sweep_error` | | `devflow release --check` | Read-only release-cut preflight: workspace self-pin, develop/main divergence (no `git fetch` — reads already-fetched refs), crates.io publish order, and changelog-version agreement (999.96). `--check` is required; a bare `devflow release` is rejected toward the deferred release-cut executor (merge/tag/sync/publish, DEN-50) | | `devflow ship --phase N [--force]` | **Dead-monitor recovery.** `devflow gate approve` only *writes* the Ship gate's response file — a live monitor polling for it is what actually advances the workflow. If that monitor died before consuming the response (e.g. the machine restarted mid-pipeline), the approval sits unconsumed forever. `devflow ship` reads the already-written Ship response directly and drives the phase through the same terminal path (`finish_workflow`) the live monitor would have — requires `state.stage == Stage::Ship` and an existing request+response pair with no prior ack. `--force` is accepted for explicit operator intent but never skips the stage check, the per-phase lock, the gate-existence check, or the ack check — it can never be used to skip Validate or jump ahead of a healthy pipeline. If the Ship response routes to a rejection that loops back to Code, `devflow ship` launches a **new, detached monitor agent** to drive the retry — the command prints this explicitly so it is never a silently long-running process. | | `devflow evidence --phase N [--json] [--require-shipped] [--root PATH]` | **Read-only structural oracle (23-06):** reports DevFlow's own append-only record of whether a phase actually shipped, instead of trusting an agent-authored attestation document. `shipped` is strictly true only after the terminal-only `workflow_shipped` event has been emitted (the last step inside `finish_workflow_with_gate_timeout`, once the entire post-Ship hook batch has succeeded) — a phase halted by `--until ` always reports `shipped: false`, even though it emits the older, ambiguous `workflow_finished` event too (surfaced separately as `workflow_finished_seen`/`finished_reason` for corroboration, never consulted by `shipped`). `--require-shipped` exits non-zero unless `shipped` is true, so it is declarable as a `verify::external_verify_commands` Layer 0 probe (opt-in per phase; a phase's PLAN must declare it and the operator must approve it via `DEVFLOW_TRUST_EXTERNAL_VERIFY`) — a failed declared probe outranks every agent-controlled signal. | -(`devflow advance` is internal — invoked by monitors with `--phase N`.) +(`devflow advance` is internal and takes `--phase N`. A supervisor does not +run it as a command; it advances in-process. `devflow status` names it as a +repair only for a gate-pending phase whose supervisor is GONE.) (`devflow monitor` is internal and registered as the hidden `__monitor` -subcommand — it is the pipe-owning monitor's own process body, re-exec'd -detached by `spawn_monitor` for a Claude `stream-json` launch. It supervises -the agent's stdin/stdout, then calls the same `advance` the shell monitor's -tail invokes. Never run it by hand: it expects to own a child's pipes and -writes the phase's capture, exit-code and agent-pid files as it goes.) +subcommand — it is the stage supervisor's own process body. Every stage +launch re-execs it detached, in its own process group, in one of two launch +shapes: **capture** (agent stdin `/dev/null`, stdout and stderr to the phase +capture files) for adapters other than Claude and Antigravity (such as `codex` +and `opencode`), the +`--legacy-claude-launch` opt-out and a checkpoint resume; or **stream** (the +supervisor owns the agent's stdin and stdout pipes and applies the idle +timeout) for a Claude or Antigravity `stream-json` launch. Both shapes run the +same Rust supervisor: it starts the agent in the agent's own process group, +records its pid and exit code, answers on a per-launch socket (see [Run +liveness](#run-liveness)), and when the agent exits runs the advance +in-process; the next stage's supervisor publishes its socket before this one +removes its own. Never run it by hand: it expects to own the agent and writes +the phase's capture, exit-code and agent-pid files as it goes.) + +## Run liveness + +Whether a phase's run is alive is answered from its supervisor socket, never +from a pid. Each launch records its socket's path in `.devflow/state-NN.json` +(the `supervisor` record); `devflow status`, `doctor`, `stop`, `cleanup`, +`recover --clean`, `start` and `resume` probe that recorded path (a connect +and one `ping`) and get one of three answers: + +| State | Meaning | Repair | +|---|---|---| +| **ALIVE** | The socket answered the ping. The reply names the current owner (agent running, advancing, gate wait, …) and the launch shape. A moment between stages is ALIVE, because the supervisor advances in-process | None needed. To end the run: `devflow stop --phase N` | +| **STALE** | The socket file is still there and owned by you, but nothing accepts the connect: the supervisor died without cleaning up, and its agent group may still be running | `devflow stop --phase N` (the backstop ends a validated agent group and removes the socket and sidecar), then `devflow resume --phase N` | +| **GONE** | No file at the recorded path: the supervisor has exited. Normal when the phase is stopped; when it is not, nothing will advance the phase | `devflow resume --phase N` (when a gate is pending, answer it and run `devflow advance --phase N`) | + +How each command shows them: + +- **`devflow status`** prints `supervisor: — ` for each + phase, and under it the repair: `devflow stop --phase N` then + `devflow resume --phase N` for STALE; `devflow resume --phase N` (plus + `devflow advance --phase N` when a gate is pending) for GONE while the phase + is not stopped. A phase with no supervisor recorded reads `not launched`. +- **`devflow doctor`** reports STALE, GONE-while-not-stopped and a failed probe + as problems, naming the phase and launch id but no path. Its repairs: + `devflow stop --phase N && devflow resume --phase N` for STALE, + `devflow resume --phase N` for GONE, `devflow status` for a failed probe. + ALIVE, a stopped GONE and an unlaunched phase are silent. `doctor --json` + gives each phase a `supervisor` object (`liveness`, `owner`, `launch`, + `launch_id`). +- **"probe failed".** A probe that cannot decide (the path is not a socket, + belongs to another user, cannot be stat'ed or connected to, or the peer + gives no valid ping reply) reads `UNKNOWN — the probe failed: …`. It is + never guessed to be ALIVE or GONE: `status` names the socket to inspect, + `doctor`'s repair is `devflow status`, `stop` refuses and signals nothing, + and `cleanup` keeps the worktree unless `--force`. +- **`devflow cleanup`** refuses an ALIVE run even with `--force` — end it + first with `devflow stop --phase N`, the line its refusal names, or wait + for it. It keeps a STALE run's worktree, or one whose probe failed, unless + `--force`. +- **`devflow recover --clean --phase N`** refuses, deleting nothing, while + the phase's supervisor may still run, by the same rule as `start`: an ALIVE + socket, a STALE one unless the backstop's checks prove its agent group + gone, or a probe that fails, searching the socket directory when the state + file is missing or does not load (and refusing when that directory exists + but cannot be listed). It does not read the agent pid file. The refusal + names a `devflow stop` line for each such socket. +- **`devflow start` / `devflow resume`** refuse while a run is live: an ALIVE + socket, a STALE one unless the backstop's checks prove its agent group gone, + or a probe that fails. When the state file is missing or does not load, + they search the socket directory for this project and phase instead, and + refuse when that directory exists but cannot be listed. The refusal names a + `devflow stop` line for each live socket. + +**Orphan sockets.** `devflow doctor` also lists, read-only, under +`supervisor sockets no state claims:`, each socket of this project whose state +is missing, does not load, records no supervisor or names another launch; +each socket of this project with a missing or unparsable sidecar; and, +machine-wide, each socket whose project root no longer exists. Each is +repaired by `devflow stop --phase N --launch `. `doctor` changes nothing; +`doctor --json` reports them as `orphan_supervisors`, and a socket directory +it cannot list as `orphan_supervisor_sweep_error` rather than as no orphans. +A launch caught between publishing its socket and saving its record can be +listed this way for a moment. + +**Where the sockets live.** Under `registry::cache_dir()/sock`: the cache +directory is `DEVFLOW_CACHE_DIR` when set, then `$XDG_CACHE_HOME/devflow`, +then `~/.cache/devflow`. A launch with none of `DEVFLOW_CACHE_DIR`, +`XDG_CACHE_HOME` and `HOME` set fails (`cannot resolve DevFlow's cache +directory for this launch`) instead of running without a socket. Each socket +is named `--.sock`, with a `.json` sidecar of +the same stem recording the launch (project root, phase, agent process group, +start time, boot id). The `sock` directory is mode 0700 and refused if it is a +symlink or owned by another user; sockets and sidecars are 0600. A socket path +longer than Linux's 107-byte `sun_path` limit is refused by name (`socket path +is N bytes, longer than the 107-byte sun_path limit`) before anything is +bound; point `DEVFLOW_CACHE_DIR` at a shorter directory if the cache path is +deep. + +**Signals.** `SIGTERM` or `SIGINT` sent to a supervisor process does what +`devflow stop`'s ALIVE arm does: a running agent's group is ended, no next +stage is launched, and `stopped` is recorded. A supervisor parked at a gate +clears the gate and records `stopped` rather than aborting the phase, so the +phase stays resumable. + +**Platform.** Socket supervision is unverified on macOS: the 103-byte macOS +`sun_path` limit is enforced by a constant only, and no macOS host or CI has +run it. Off Linux the STALE backstop refuses before reading anything, so a +STALE socket there is never torn down by `devflow stop`. + +**Upgrading from 2.x.** Install a 3.x binary only while no 2.x run is live; +3.x makes no promise about a 2.x state.json, which may read as GONE or fail to +load. ## Preflight refusals @@ -83,9 +186,10 @@ devflow gate reject 15 --note "abort: wrong direction" # ends the ph `--stage` is only needed when one phase somehow has several open gates. Under the hood this writes `.devflow/gates/NN-.response.json` -atomically; the blocked monitor polls it (exponential backoff, so pickup can -take up to ~60s), acks, and moves on. The CLI refuses to overwrite an -unconsumed response. +atomically; the supervisor parked at the gate checks for it about once a +second (a gate waited on outside a supervisor, such as `devflow ship`'s +foreground wait, backs off up to ~60s), acks, and moves on. The CLI refuses to +overwrite an unconsumed response. ## Notify hook (never miss a gate) @@ -134,7 +238,7 @@ never receives the variable. | `DEVFLOW_YES_SHIP` | false | Standing Ship gate pre-authorization (D-12, `28-CONTEXT.md`); overrides `devflow.toml`'s `yes_ship` key, which itself is OR-combined with `--yes-ship` at `devflow start` | | `DEVFLOW_ANTIGRAVITY_IDLE_TIMEOUT_SECS` | 120 | The per-agent idle timeout for the pipe-owning Antigravity monitor (phase 41, D-08): how long the monitor waits with NO output on the child's `stream-json` stream before recording an `idle_timeout` verdict and terminating it. Same 120s floor and clamp semantics as `DEVFLOW_CLAUDE_IDLE_TIMEOUT_SECS`. The default is a DECIDED starting point, not a silent inheritance of the Claude-measured floor — it is to be revisited after the first real Antigravity cadence measurement. Claude's variable is never read for Antigravity and vice versa | | `DEVFLOW_CLAUDE_IDLE_TIMEOUT_SECS` | 120 | How long the pipe-owning Claude monitor waits with NO output on the child's stream before recording an `idle_timeout` verdict and terminating it. Clamped to a 120s floor: a lower value is raised and the clamp is printed loudly at `devflow start`. The floor is 4x the CLI's measured `tool_progress` keepalive interval (a fixed 30.00s, measured 2026-08-03 over five workload-controlled trials on CLI 2.1.220), so it survives three consecutive missed keepalives; the earlier 30s floor sat exactly ON the keepalive interval and would have killed healthy stages running any tool call longer than ~30s, including `cargo test --workspace`. Every stream line resets the window, so a healthy long-running stage is never bounded by wall clock | -| `DEVFLOW_CLAUDE_LEGACY_LAUNCH` | false | D-11's escape hatch (31-04): force the pre-31 single-document Claude launch — prompt positionally in argv, `--output-format json`, the `sh` monitor — instead of the Phase 31 `stream-json` transport. OR-combined with `--legacy-claude-launch` at `devflow start` / `devflow resume` and persisted to `state-NN.json`, so a detached monitor honours it; a run authorized by this variable alone prints a notice naming it as the source. **Parsed as a bool: `=false` does NOT enable it, and an unparseable value warns and leaves the legacy path OFF.** Using it is announced on three channels (stdout, `phase-NN-monitor.log`, and a `claude_legacy_launch_forced` event in `events.jsonl`) because an escape hatch used routinely erodes what it protects. **What it gives up: the legacy path cannot deliver background-task notifications, so a multi-plan wave may orphan delegated work — that is 999.64, the defect Phase 31 exists to fix.** Once set for a run it is never cleared by `devflow resume`; edit `state-NN.json` to turn it off | +| `DEVFLOW_CLAUDE_LEGACY_LAUNCH` | false | D-11's escape hatch (31-04): force the pre-31 single-document Claude launch — prompt positionally in argv, `--output-format json`, the capture launch shape — instead of the Phase 31 `stream-json` transport. OR-combined with `--legacy-claude-launch` at `devflow start` / `devflow resume` and persisted to `state-NN.json`, so a detached monitor honours it; a run authorized by this variable alone prints a notice naming it as the source. **Parsed as a bool: `=false` does NOT enable it, and an unparseable value warns and leaves the legacy path OFF.** Using it is announced on three channels (stdout, `phase-NN-monitor.log`, and a `claude_legacy_launch_forced` event in `events.jsonl`) because an escape hatch used routinely erodes what it protects. **What it gives up: the legacy path cannot deliver background-task notifications, so a multi-plan wave may orphan delegated work — that is 999.64, the defect Phase 31 exists to fix.** Once set for a run it is never cleared by `devflow resume`; edit `state-NN.json` to turn it off | | `RUST_LOG` | `info` | Log verbosity (stderr) | | `DEVFLOW_LOG_FORMAT` | plain | `json` for machine-readable log lines | | `DEVFLOW_E2E_CHILD_TIMEOUT_SECS` | 90 | Test-only: bounds `gate_sweep_e2e.rs`'s patience with a spawned `devflow advance` child so CI cannot hang indefinitely; not read by any production code path | @@ -147,8 +251,7 @@ never receives the variable. | `lock-NN` | Per-phase advance lock (held across gate waits) | | `lock-project` | Coarse checkout lock (held seconds, around shared-git mutations) | | `phase-NN-stdout` / `phase-NN-stderr.log` | Agent capture files (what `logs` tails) | -| `phase-NN-exit` / `phase-NN-agent-pid` | Exit code + PID the monitor records | -| `phase-NN-monitor-stop` | Stop marker the `sh` monitor's TERM/INT trap writes before killing the agent, so an agent child that lost the TERM still stops before it starts; removed before each launch | +| `phase-NN-exit` / `phase-NN-agent-pid` | Exit code + PID the supervisor records | | `gates/NN-.json` (+ `.response.json`, `.ack.json`) | Gate request / answer / receipt | | `events.jsonl` | Append-only event log (schema v1, one JSON object per line, phase id on every line) — tail it from any tool | | `cron-instructions-NN.json` | Rate-limit resume record naming `devflow resume --phase N` for a paused run | diff --git a/crates/devflow-cli/Cargo.toml b/crates/devflow-cli/Cargo.toml index 2c60a8be..a7ea4bbe 100644 --- a/crates/devflow-cli/Cargo.toml +++ b/crates/devflow-cli/Cargo.toml @@ -23,6 +23,8 @@ thiserror.workspace = true [dev-dependencies] tempfile = "3" +# Process-group signals in the test reapers' identity-confirmed fallback (50-11). +libc = "0.2" # `test_support::git_command` for this crate's test targets (999.37). devflow-core = { workspace = true, features = ["test-support"] } # `insta::assert_snapshot!` for the prompt-text drift guard (47-CONTEXT.md D-14/D-15). diff --git a/crates/devflow-cli/src/commands.rs b/crates/devflow-cli/src/commands.rs index 56ace307..0fa6a351 100644 --- a/crates/devflow-cli/src/commands.rs +++ b/crates/devflow-cli/src/commands.rs @@ -36,7 +36,8 @@ use devflow_core::recover; use devflow_core::registry; use devflow_core::ship_evidence; use devflow_core::stage::Stage; -use devflow_core::state::{AgentKind, State}; +use devflow_core::state::{AgentKind, State, SupervisorRecord}; +use devflow_core::supervisor; use devflow_core::version; use devflow_core::workflow; use devflow_core::worktree; @@ -871,10 +872,12 @@ fn remove_worktree_with_retry( /// associated feature branches, then prune and clean up merged branches. /// /// Hard-refuses (D-06, no override flag) removal of any worktree whose owning -/// phase has a live agent (any monitor state, including Unknown/Stuck) or an -/// active monitor (Healthy/BetweenStages) — closing the race where a real -/// `cleanup --force` run could delete a worktree a live agent/monitor is +/// phase has a live agent (whatever its supervisor's liveness) or a +/// supervisor that answers ALIVE — closing the race where a real +/// `cleanup --force` run could delete a worktree a live agent/supervisor is /// still writing into (review: Codex HIGH, fail-closed on a live agent). +/// A STALE supervisor (or a failed probe) keeps the worktree unless +/// `--force` (50-12); GONE and NotLaunched follow the rules above. pub(crate) fn cleanup(project_root: &Path, force: bool) -> Result<(), CliError> { // Project-resolved (45-01): `cleanup_merged` computes "merged" relative // to the trunk and treats it as protected — both must be the configured @@ -903,19 +906,18 @@ pub(crate) fn cleanup(project_root: &Path, force: bool) -> Result<(), CliError> let agent_alive = phase .and_then(|p| agent_pid_from_file(project_root, p)) .is_some_and(agent::agent_running); - let monitor_pid = matched_state.and_then(|s| s.monitor_pid); - let monitor_alive = monitor_pid.is_some_and(agent::agent_running); - let phase_liveness = liveness(monitor_pid, monitor_alive, agent_alive); - - // A phase halted via `devflow start --until ` (20c) clears - // `monitor_pid` and its agent has already exited by design — that - // reads as `Liveness::Unknown` with `agent_alive == false`, which - // would otherwise sail straight through the live-agent refusal + // 50-12: the supervisor's liveness comes from its recorded socket, + // never a pid. + let phase_liveness = matched_state.map_or(PhaseLiveness::NotLaunched, probe_phase); + + // A phase halted via `devflow start --until ` (20c) has no + // running supervisor and its agent has already exited by design — + // which would otherwise sail straight through the live-agent refusal // below. Treat it the same way `doctor`'s `check_dead_agent`/ - // `check_dead_monitor` were taught about `facts.stopped` in this - // same phase: an intentionally-parked worktree is never implicitly - // safe to remove — require `--force`, mirroring the `reference` - // worktree's own precedent above. + // `check_supervisor` were taught about `facts.stopped`: an + // intentionally-parked worktree is never implicitly safe to remove — + // require `--force`, mirroring the `reference` worktree's own + // precedent above. let stopped = matched_state.is_some_and(|s| s.stopped); if stopped && !force { let phase_label = phase @@ -928,21 +930,41 @@ pub(crate) fn cleanup(project_root: &Path, force: bool) -> Result<(), CliError> continue; } - // Fail-closed on a live agent: refuse whenever the agent is alive - // (regardless of monitor liveness — Unknown/Stuck included) OR the - // monitor is actively running the stage (Healthy/BetweenStages). - // Only Stuck/Unknown WITHOUT a live agent proceeds. - if agent_alive || matches!(phase_liveness, Liveness::Healthy | Liveness::BetweenStages) { + // Fail-closed on a live run: refuse whenever the agent is alive + // (whatever the supervisor's liveness) OR the supervisor answers + // ALIVE, `--force` or not. The verb that ends a live run is `stop` + // (50-14); `resume` refuses while one is live. + if agent_alive || matches!(phase_liveness, PhaseLiveness::Alive { .. }) { let phase_label = phase .map(|p| p.to_string()) .unwrap_or_else(|| "?".to_string()); return Err(CliError::Message(format!( - "refusing to remove worktree {} for phase {phase_label} ({}) — run `devflow resume --phase {phase_label}` or wait for it to finish", + "refusing to remove worktree {} for phase {phase_label} ({}) — run `devflow stop --phase {phase_label}` or wait for it to finish", wt.path.display(), phase_liveness.describe(), ))); } + // 50-12: a STALE supervisor may have left its agent group running, + // and a failed probe proves nothing either way: keep the worktree + // unless `--force`, and name the verb that ends the orphaned run. + if !force + && matches!( + phase_liveness, + PhaseLiveness::Stale | PhaseLiveness::ProbeFailed(_) + ) + { + let phase_label = phase + .map(|p| p.to_string()) + .unwrap_or_else(|| "?".to_string()); + println!( + "keeping worktree {} for phase {phase_label} — supervisor {}; run `devflow stop --phase {phase_label}` first, or pass --force to discard it", + wt.path.display(), + phase_liveness.describe(), + ); + continue; + } + match remove_worktree_with_retry(project_root, &wt.path, force) { Ok(()) => { print!("removed worktree {}", wt.path.display()); @@ -981,68 +1003,164 @@ pub(crate) fn cleanup(project_root: &Path, force: bool) -> Result<(), CliError> Ok(()) } -/// A phase's monitor/agent liveness, distinguishing a dead monitor (nothing -/// will call `devflow advance` when the agent exits) from a normal -/// between-stages moment (18b — "who watches the watcher"). -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum Liveness { - /// Monitor and agent are both alive — the stage is actively running. - Healthy, - /// Monitor is alive, agent has exited — normal between-stages moment; - /// the monitor will advance the phase shortly. - BetweenStages, - /// The recorded monitor is dead. Whether or not the agent is also dead, - /// nothing will call `devflow advance` for this phase — it needs a - /// manual `devflow resume`. - Stuck, - /// No monitor PID has been recorded for this state — either none has - /// been spawned yet, or the state was written by a binary predating - /// this field. Never reported as a problem. - Unknown, -} - -impl Liveness { - pub(crate) fn describe(self) -> &'static str { +/// A phase's supervisor liveness as `status`, `cleanup` and `doctor` report +/// it (50-12): answered from the socket recorded in `state.supervisor` +/// through [`supervisor::probe`], never from a pid. Three parts, like the +/// 18b presentation it replaces: this enum, [`PhaseLiveness::describe`], and +/// the pure mapping [`phase_liveness`], which `doctor`'s `check_supervisor` +/// reuses so the two can never drift. +/// +/// A supervisor runs its own advance in-process, so a moment between stages +/// is ALIVE (its owner reads `advancing`), not a separate state. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum PhaseLiveness { + /// The recorded socket answered a ping. + Alive { + /// Who owns the run right now, from the ping reply. + owner: String, + /// The launch shape (`capture` or `stream`), from the ping reply. + launch: String, + }, + /// The socket file remains but nothing listens on it: the supervisor + /// died without cleaning up, and its agent group may still run. + Stale, + /// No socket at the recorded path, and the phase is stopped: a normal + /// end. + Gone, + /// No socket at the recorded path while the phase is not stopped: + /// nothing will advance it. A finding, not a normal state. + GoneWhileActive, + /// The state records no supervisor: nothing has been launched yet, or the + /// state predates the record. Never a finding. + NotLaunched, + /// The probe could not answer; a path-free summary of why (see + /// [`probe_error_summary`]). Neither ALIVE nor GONE is guessed. + ProbeFailed(String), +} + +impl PhaseLiveness { + /// The operator-facing text. The three states the operator acts on lead + /// with `ALIVE`, `STALE` and `GONE`. + pub(crate) fn describe(&self) -> String { + match self { + PhaseLiveness::Alive { owner, launch } => { + format!("ALIVE (owner: {owner}, launch: {launch})") + } + PhaseLiveness::Stale => { + "STALE — the socket remains but no supervisor answers".to_string() + } + PhaseLiveness::Gone => { + "GONE — the supervisor has exited and the phase is stopped".to_string() + } + PhaseLiveness::GoneWhileActive => { + "GONE — no supervisor runs this phase, but it is not stopped".to_string() + } + PhaseLiveness::NotLaunched => "not launched (no supervisor recorded)".to_string(), + PhaseLiveness::ProbeFailed(why) => format!("UNKNOWN — the probe failed: {why}"), + } + } + + /// A stable machine label for `doctor --json`. + pub(crate) fn label(&self) -> &'static str { match self { - Liveness::Healthy => "healthy", - Liveness::BetweenStages => "between stages", - Liveness::Stuck => "stuck — needs devflow resume", - Liveness::Unknown => "unknown (no monitor recorded)", + PhaseLiveness::Alive { .. } => "alive", + PhaseLiveness::Stale => "stale", + PhaseLiveness::Gone => "gone", + PhaseLiveness::GoneWhileActive => "gone_while_active", + PhaseLiveness::NotLaunched => "not_launched", + PhaseLiveness::ProbeFailed(_) => "probe_failed", } } } -/// Pure liveness predicate — no I/O. `monitor_pid` is matched `None` first -/// so a state written by a pre-18b binary (carrying no `monitor_pid`) can -/// never be misclassified as `Stuck` (T-18-11). -fn liveness(monitor_pid: Option, monitor_alive: bool, agent_alive: bool) -> Liveness { - match monitor_pid { - None => Liveness::Unknown, - Some(_) => match (monitor_alive, agent_alive) { - (true, true) => Liveness::Healthy, - (true, false) => Liveness::BetweenStages, - (false, _) => Liveness::Stuck, +/// Pure mapping from a phase's supervisor record and its probe result to a +/// [`PhaseLiveness`]. No I/O. No record (or no probe) is `NotLaunched`; GONE +/// splits on `stopped`; a probe error is `ProbeFailed`, never a guess. +fn phase_liveness( + record: Option<&SupervisorRecord>, + probe: Option>, + stopped: bool, +) -> PhaseLiveness { + let (Some(_), Some(probe)) = (record, probe) else { + return PhaseLiveness::NotLaunched; + }; + match probe { + Ok(supervisor::Liveness::Alive(reply)) => PhaseLiveness::Alive { + owner: reply.owner, + launch: reply.launch, }, + Ok(supervisor::Liveness::Stale) => PhaseLiveness::Stale, + Ok(supervisor::Liveness::Gone) if stopped => PhaseLiveness::Gone, + Ok(supervisor::Liveness::Gone) => PhaseLiveness::GoneWhileActive, + Err(error) => PhaseLiveness::ProbeFailed(probe_error_summary(&error)), } } -/// Recovery verbs discoverable from a phase's liveness (21a, D-03) — the -/// pure, testable counterpart to `status`'s old inline Stuck `println!`. -/// Always includes `devflow resume` for `Stuck`; additionally includes -/// `devflow advance` when the phase is gate-pending (the operator answers -/// the gate then advances — the primary footgun this closes; widening the -/// predicate further risks suggesting `advance` where nothing proves it is -/// right, per 21-CONTEXT.md's Review Incorporation). Empty for any other -/// liveness, so a healthy/between-stages/unknown phase prints nothing new. -fn recovery_hints(state: &State, liveness: Liveness) -> Vec { - if liveness != Liveness::Stuck { - return Vec::new(); +/// A [`supervisor::ProbeError`] as a path-free kind and cause (T-18-01): +/// `NotASocket`'s own text names the path, and a `NoReply` text may quote +/// whatever the peer sent, so neither is passed through. +fn probe_error_summary(error: &supervisor::ProbeError) -> String { + match error { + supervisor::ProbeError::Stat(cause) => { + format!("could not stat the socket ({})", cause.kind()) + } + supervisor::ProbeError::NotASocket(_) => "the recorded path is not a socket".to_string(), + supervisor::ProbeError::ForeignOwner { uid } => { + format!("the socket is owned by uid {uid}, not this user") + } + supervisor::ProbeError::Connect(cause) => { + format!("could not connect to the socket ({})", cause.kind()) + } + supervisor::ProbeError::NoReply(_) => "no valid ping reply".to_string(), + _ => "the probe failed".to_string(), } - let mut hints = vec![format!("devflow resume --phase {}", state.phase)]; - if state.gate_pending { - hints.push(format!("devflow advance --phase {}", state.phase)); +} + +/// Probe the socket `state.supervisor` records, once, and map the answer. +/// The only I/O half of [`phase_liveness`]; reads no pid. +fn probe_phase(state: &State) -> PhaseLiveness { + let record = state.supervisor.as_ref(); + let probe = record.map(|record| supervisor::probe(&record.socket_path)); + phase_liveness(record, probe, state.stopped) +} + +/// Recovery verbs discoverable from a phase's liveness (21a, D-03; 50-12) — +/// the pure, testable counterpart to `status`'s hints. STALE names +/// `devflow stop` (it ends the orphaned agent group and removes the socket) +/// and then `devflow resume`; GONE while active names `devflow resume`, plus +/// `devflow advance` when the phase is gate-pending (answer the gate, then +/// advance — widening that predicate risks suggesting `advance` where +/// nothing proves it is right, per 21-CONTEXT.md's Review Incorporation); a +/// failed probe names the socket to inspect. Empty for ALIVE, a stopped +/// GONE and NotLaunched. +fn recovery_hints(state: &State, liveness: &PhaseLiveness) -> Vec { + let phase = state.phase; + match liveness { + PhaseLiveness::Stale => vec![ + format!("devflow stop --phase {phase}"), + format!("devflow resume --phase {phase}"), + ], + PhaseLiveness::GoneWhileActive => { + let mut hints = vec![format!("devflow resume --phase {phase}")]; + if state.gate_pending { + hints.push(format!("devflow advance --phase {phase}")); + } + hints + } + PhaseLiveness::ProbeFailed(why) => { + let socket = state + .supervisor + .as_ref() + .map(|record| record.socket_path.display().to_string()) + .unwrap_or_default(); + vec![format!( + "the probe failed ({why}); inspect the socket at {socket}" + )] + } + PhaseLiveness::Alive { .. } | PhaseLiveness::Gone | PhaseLiveness::NotLaunched => { + Vec::new() + } } - hints } /// `status`'s in-stage progress line: real elapsed time since the phase's @@ -1114,19 +1232,15 @@ pub(crate) fn status(project_root: &Path) -> Result<(), CliError> { } None => println!(" agent_pid: none"), } - match state.monitor_pid { - Some(pid) => { - println!( - " monitor_pid: {pid} (running: {})", - agent::agent_running(pid) - ); - } - None => println!(" monitor_pid: none"), - } - let agent_alive = agent_pid.is_some_and(agent::agent_running); - let monitor_alive = state.monitor_pid.is_some_and(agent::agent_running); - let phase_liveness = liveness(state.monitor_pid, monitor_alive, agent_alive); - println!(" liveness: {}", phase_liveness.describe()); + // 50-12: liveness comes from the recorded socket, never a pid. + // `status` speaks only about the operator's own project, so it + // names the socket path (doctor does not, T-18-01). + let phase_liveness = probe_phase(state); + let socket = state.supervisor.as_ref().map_or_else( + || "none".to_string(), + |record| record.socket_path.display().to_string(), + ); + println!(" supervisor: {socket} — {}", phase_liveness.describe()); let phase_summary = last_events.remove(&state.phase); println!( "{}", @@ -1135,7 +1249,7 @@ pub(crate) fn status(project_root: &Path) -> Result<(), CliError> { phase_summary.as_ref().and_then(|s| s.stage_launched_ts) ) ); - for hint in recovery_hints(state, phase_liveness) { + for hint in recovery_hints(state, &phase_liveness) { println!(" → {hint}"); } if let Some(summary) = phase_summary { @@ -1868,32 +1982,685 @@ fn reap_stray_candidates( type AnsweredGate = (Stage, lock::HolderStatus, Option<(u32, Option)>); -/// End a running phase cleanly (23c) — the missing primitive -/// `23-ORPHAN-FORENSICS.md` names as the reason 54 processes accumulated -/// with no remedy but `kill(1)`. Answers `phase`'s open gate if it has one — -/// the primary path, since it is the only one that produces a clean final -/// state: the target unwinds through its own `abort()` and releases its -/// lock; otherwise signals the process recorded in `phase`'s per-phase lock -/// file (never `state.monitor_pid` — T-23-51: the monitor shell script's -/// `trap` closure only ever captures the **agent's** pid, never the -/// trailing `advance` invocation, so by the time `advance` is the shell's -/// foreground child, `monitor_pid` already names a process that exited long -/// ago; `lock::holder`'s recorded pid is the only correct target). +/// End a running phase cleanly (23c, D-04, D-06): the one teardown verb, at +/// every point of a run, that never claims a stop it did not perform. +/// +/// 1. An open gate with a confirmed live waiter is answered with an abort +/// ([`stop_via_gate`]); the waiter unwinds through its own `abort()`. +/// 2. Otherwise the run's supervisor socket is addressed +/// ([`stop_via_socket`]): the state's `supervisor` record names it, or, +/// when the state records none, is missing or does not load, the one live +/// sidecar-found socket for this root and phase. Its probe picks the arm: +/// ALIVE gets a `shutdown`, which ends a run that holds no per-phase lock +/// (999.138). +/// 3. Only when no live socket remains (GONE, or nothing recorded) does the +/// identity-confirmed per-phase lock holder get signalled +/// ([`stop_via_lock`]). +/// +/// No pid read from state is ever signalled. `stopped` is recorded only for +/// a stop performed or confirmed not live, and never into a state file that +/// does not load. pub(crate) fn stop(project_root: &Path, phase: PhaseId) -> Result<(), CliError> { + stop_in(project_root, phase, registry::cache_dir().as_deref()) +} + +/// [`stop`] with the cache dir whose `sock/` holds the sidecars it may +/// search, so tests never sweep the operator's real cache. +fn stop_in(project_root: &Path, phase: PhaseId, cache_dir: Option<&Path>) -> Result<(), CliError> { let (gate_answered, gate_holder) = stop_via_gate(project_root, phase)?; - let signal_sent = if gate_answered { - false - } else { - stop_via_lock(project_root, phase)? + if gate_answered { + return persist_stopped_state(project_root, phase, gate_holder, false); + } + match stop_via_socket(project_root, phase, cache_dir)? { + SocketStop::NoLiveSocket => { + let signal_sent = stop_via_lock(project_root, phase)?; + persist_stopped_state(project_root, phase, None, signal_sent) + } + // The supervisor may still be releasing the per-phase lock as it + // exits, so wait it out like a signalled holder. + SocketStop::Ended => persist_stopped_state(project_root, phase, None, true), + SocketStop::EndedStateUnloadable(state_file) => { + println!( + "stop: phase {phase}'s state file {} does not load, so `stopped` was not \ + recorded — repair or remove it before `devflow resume --phase {phase}`", + state_file.display() + ); + Ok(()) + } + } +} + +/// `devflow stop --phase N --launch ` (D-06): tear down only the +/// supervisor socket with that launch id, the form `devflow doctor` prints +/// for an orphan. The launch id is the address, so a deleted root's orphan +/// is reachable from any directory. +/// +/// The socket is the one whose sidecar carries ``, or, when its sidecar +/// is missing or does not parse, whose file name carries it (RV-05b). If this +/// project's state loads and records that same launch, the plain [`stop`] +/// runs. Otherwise only that socket is acted on: no gate is answered, no +/// lock holder signalled and no state written. +pub(crate) fn stop_launch( + project_root: &Path, + phase: PhaseId, + launch_id: &str, +) -> Result<(), CliError> { + let cache_dir = registry::cache_dir().ok_or_else(|| { + CliError::Message( + "stop: no cache directory (set HOME, XDG_CACHE_HOME or DEVFLOW_CACHE_DIR), so no \ + supervisor socket can be found by launch id" + .to_string(), + ) + })?; + stop_launch_in(project_root, phase, launch_id, &cache_dir) +} + +/// [`stop_launch`] against the sockets under `cache_dir`. +fn stop_launch_in( + project_root: &Path, + phase: PhaseId, + launch_id: &str, + cache_dir: &Path, +) -> Result<(), CliError> { + let Some(entry) = find_launch(cache_dir, launch_id)? else { + println!("stop: no supervisor socket with launch id {launch_id}; nothing to stop"); + return Ok(()); }; - persist_stopped_state( - project_root, - phase, - gate_answered.then_some(gate_holder).flatten(), - signal_sent, + check_launch_belongs(project_root, phase, launch_id, &entry, cache_dir)?; + let recorded = workflow::load_state(project_root, phase).is_ok_and(|state| { + state + .supervisor + .is_some_and(|record| record.launch_id == launch_id) + }); + if recorded { + return stop_in(project_root, phase, Some(cache_dir)); + } + stop_one_launch(launch_id, &entry) +} + +/// The one socket under `cache_dir` whose sidecar carries `launch_id`, or, +/// with no usable sidecar, whose file name does. Two such sockets are a +/// refusal: the id no longer names one launch. +fn find_launch( + cache_dir: &Path, + launch_id: &str, +) -> Result, CliError> { + let mut matches: Vec = supervisor::sweep_in(cache_dir) + .into_iter() + .filter(|entry| match &entry.sidecar { + Some(sidecar) => sidecar.launch_id == launch_id, + None => entry.launch_id_in_name.as_deref() == Some(launch_id), + }) + .collect(); + if matches.len() > 1 { + let paths: Vec = matches + .iter() + .map(|entry| entry.socket_path.display().to_string()) + .collect(); + return Err(CliError::Message(format!( + "stop: launch id {launch_id} names {} sockets ({}); nothing was changed", + matches.len(), + paths.join(", ") + ))); + } + Ok(matches.pop()) +} + +/// Refuse a launch that is not this project's phase `phase`, changing +/// nothing. A parsed sidecar decides: its phase must be `phase`, and its +/// project root must be this one or no longer exist (a deleted root's +/// orphan). With no usable sidecar, the socket must be one +/// [`supervisor::sweep_for_in`] attributes to this root and phase by its +/// file name. +fn check_launch_belongs( + project_root: &Path, + phase: PhaseId, + launch_id: &str, + entry: &supervisor::SweepEntry, + cache_dir: &Path, +) -> Result<(), CliError> { + let refuse = |why: String| { + Err(CliError::Message(format!( + "stop: launch {launch_id} {why}; nothing was changed" + ))) + }; + match &entry.sidecar { + Some(sidecar) if sidecar.phase != phase => { + refuse(format!("belongs to phase {}, not {phase}", sidecar.phase)) + } + Some(sidecar) if sidecar.project_root != project_root && sidecar.project_root.is_dir() => { + refuse(format!( + "belongs to project {}, not {}; run stop from that project", + sidecar.project_root.display(), + project_root.display() + )) + } + Some(_) => Ok(()), + None => { + let ours = supervisor::sweep_for_in(cache_dir, project_root, phase) + .iter() + .any(|candidate| candidate.socket_path == entry.socket_path); + if ours { + Ok(()) + } else { + refuse(format!( + "has no usable sidecar, and its socket file name {} names another project \ + or phase than {} phase {phase}", + entry.socket_path.display(), + project_root.display() + )) + } + } + } +} + +/// Act on one launch's socket alone: ALIVE → `shutdown` (a `handed_off` +/// answer is reported, never retried against the successor); STALE → the +/// backstop with its parsed sidecar and no state record, or a refusal when +/// it has none; GONE → nothing to stop; a probe error → a refusal. +fn stop_one_launch(launch_id: &str, entry: &supervisor::SweepEntry) -> Result<(), CliError> { + let subject = format!("launch {launch_id}"); + let socket = &entry.socket_path; + match supervisor::probe(socket) { + Ok(supervisor::Liveness::Gone) => { + println!( + "stop: {subject}: socket {} is gone; nothing to stop", + socket.display() + ); + Ok(()) + } + Err(error) => Err(CliError::Message(format!( + "stop: {subject}: could not probe socket {} ({}: {error}); nothing was signalled, \ + removed or recorded", + socket.display(), + probe_error_kind(&error) + ))), + Ok(supervisor::Liveness::Stale) => stop_stale_launch(&subject, entry), + Ok(supervisor::Liveness::Alive(_)) => shutdown_one_launch(&subject, entry), + } +} + +/// [`stop_one_launch`]'s ALIVE arm. A request that errors is probed once +/// more, as [`shutdown_supervisor`] does: STALE goes to the backstop, GONE +/// leaves nothing to stop. +fn shutdown_one_launch(subject: &str, entry: &supervisor::SweepEntry) -> Result<(), CliError> { + let socket = &entry.socket_path; + let error = match supervisor::request_shutdown(socket, shutdown_final_timeout()) { + Ok(supervisor::ShutdownOutcome::Stopped { + exit_code, + advance_suppressed, + }) => { + let advance = if advance_suppressed { + "advance suppressed" + } else { + "advance not suppressed" + }; + println!( + "stop: stopped {subject} through its supervisor (agent group ended, exit code \ + {exit_code}, {advance})" + ); + return Ok(()); + } + Ok(supervisor::ShutdownOutcome::HandedOff) => { + println!("stop: {subject} already handed off to its successor; nothing stopped"); + return Ok(()); + } + Ok(other) => { + return Err(CliError::Message(format!( + "stop: {subject}: the supervisor did not stop ({other:?}); nothing was recorded" + ))); + } + Err(error) => error, + }; + match supervisor::probe(socket) { + Ok(supervisor::Liveness::Stale) => stop_stale_launch(subject, entry), + Ok(supervisor::Liveness::Gone) => { + println!( + "stop: {subject}: socket {} went away under the shutdown request ({error}); \ + nothing left to stop", + socket.display() + ); + Ok(()) + } + _ => Err(CliError::Message(format!( + "stop: {subject}: the supervisor at {} did not answer the shutdown ({error}); \ + nothing was signalled or recorded", + socket.display() + ))), + } +} + +/// [`stop_one_launch`]'s STALE arm: the backstop with the sidecar the sweep +/// parsed and no state record, or, with no usable sidecar, a refusal that +/// signals and removes nothing (RV-05b): there is no pgid, start time or +/// boot id to validate. +fn stop_stale_launch(subject: &str, entry: &supervisor::SweepEntry) -> Result<(), CliError> { + let Some(sidecar) = &entry.sidecar else { + let why = entry.sidecar_error.as_ref().map_or_else( + || "its sidecar is missing".to_string(), + |error| format!("its sidecar is unusable ({error})"), + ); + return Err(stale_refusal(subject, &entry.socket_path, &why)); + }; + let socket = supervisor::SocketPath::new(entry.socket_path.clone()) + .map_err(|error| stale_refusal(subject, &entry.socket_path, &error.to_string()))?; + run_backstop(subject, &socket, sidecar, None, None) +} + +/// How long `stop` waits for a supervisor's final shutdown line: the +/// supervisor's own terminate-and-verify wait, twice, then its wait for the +/// per-phase lock before it records its stop and answers +/// ([`crate::pipeline_launch::SUPERVISOR_STOP_WAIT`]), plus time to persist. +/// Shorter, and a busy lock reads a supervisor that did stop as one that did +/// not answer. +fn shutdown_final_timeout() -> std::time::Duration { + 2 * agent::TERMINATE_VERIFY_WAIT + + crate::pipeline_launch::SUPERVISOR_STOP_WAIT + + std::time::Duration::from_secs(5) +} + +/// Shutdown requests `stop` sends in all when each is answered `handed_off`: +/// the first and 3 retries, each against the record a fresh state reload +/// names (crash row C13). +const MAX_SHUTDOWN_REQUESTS: usize = 4; + +/// What [`stop_via_socket`] did. +enum SocketStop { + /// No live supervisor socket: nothing recorded, or its probe says GONE. + NoLiveSocket, + /// The run was ended through its socket; record `stopped`. + Ended, + /// The run was ended through a sidecar-found socket, but the state file + /// at this path exists and does not load, so nothing can be recorded. + EndedStateUnloadable(PathBuf), +} + +/// The supervisor socket `stop` addresses for a phase, and its provenance. +struct LocatedSocket { + socket: PathBuf, + /// The state's record, when the socket came from it (the backstop + /// cross-checks the sidecar against it). + record: Option, + /// The state file, when it exists but does not load and the socket came + /// from its sidecar. + unloadable_state: Option, +} + +/// Find the socket for a bare `stop`: the loaded state's `supervisor` record; +/// or, when the state records none, is missing or does not load, the one +/// socket of this root and phase that a sweep finds not GONE. Several such +/// sockets are a refusal listing each, since `stop` cannot tell which is the +/// run. +/// +/// A loaded state with no record is swept because a `start` launcher saves +/// the record only after its spawned supervisor has published: a stop in +/// that window must end the supervisor through its socket, not signal the +/// launcher (the lock holder), whose death would leave the supervisor and +/// its agent running unrecorded. +fn locate_phase_socket( + project_root: &Path, + phase: PhaseId, + cache_dir: Option<&Path>, +) -> Result, CliError> { + let unloadable_state = match workflow::load_state(project_root, phase) { + Ok(state) => match state.supervisor { + Some(record) => { + return Ok(Some(LocatedSocket { + socket: record.socket_path.clone(), + record: Some(record), + unloadable_state: None, + })); + } + None => None, + }, + Err(workflow::WorkflowError::MissingState(_)) => None, + Err(_) => Some(workflow::state_path(project_root, phase)), + }; + let Some(cache_dir) = cache_dir else { + return Ok(None); + }; + let live: Vec = + supervisor::sweep_for_in(cache_dir, project_root, phase) + .into_iter() + .filter(|entry| { + entry.socket_present && !matches!(entry.liveness, Ok(supervisor::Liveness::Gone)) + }) + .collect(); + match live.as_slice() { + [] => Ok(None), + [entry] => Ok(Some(LocatedSocket { + socket: entry.socket_path.clone(), + record: None, + unloadable_state, + })), + several => Err(several_sockets_refusal( + phase, + unloadable_state.as_deref(), + several, + )), + } +} + +/// The refusal for several live sockets of one root and phase: each with its +/// launch id, liveness and the `devflow stop --phase N --launch ` form +/// that addresses it alone. Nothing is signalled, removed or recorded. +fn several_sockets_refusal( + phase: PhaseId, + unloadable_state: Option<&Path>, + entries: &[supervisor::SweepEntry], +) -> CliError { + use std::fmt::Write as _; + let why = unloadable_state.map_or_else( + || "no state file names one".to_string(), + |path| format!("its state file {} does not load", path.display()), + ); + let mut text = format!( + "stop: phase {phase} has {} live supervisor sockets for this project and {why}, so stop \ + cannot tell which one is the run; nothing was signalled, removed or recorded. Stop each \ + one by its launch id:", + entries.len() + ); + for entry in entries { + let _ = write!(text, "\n {}", sweep_entry_stop_line(phase, entry)); + } + CliError::Message(text) +} + +/// One line of [`several_sockets_refusal`]. A socket with no usable sidecar +/// is named by the launch id in its file name (RV-05b); one whose name is +/// outside the DevFlow scheme has no `--launch` form, so it is listed by its +/// path and its ping reply's launch id. +fn sweep_entry_stop_line(phase: PhaseId, entry: &supervisor::SweepEntry) -> String { + let liveness = match &entry.liveness { + Ok(supervisor::Liveness::Alive(_)) => "ALIVE", + Ok(supervisor::Liveness::Stale) => "STALE", + Ok(supervisor::Liveness::Gone) => "GONE", + Err(_) => "probe failed", + }; + let sidecar_note = match (&entry.sidecar, &entry.sidecar_error) { + (Some(_), _) => "", + (None, Some(_)) => ", sidecar does not parse", + (None, None) => ", sidecar missing", + }; + let launch_id = entry + .sidecar + .as_ref() + .map(|sidecar| sidecar.launch_id.clone()) + .or_else(|| entry.launch_id_in_name.clone()); + match launch_id { + Some(id) => format!( + "launch {id} ({liveness}{sidecar_note}): devflow stop --phase {phase} --launch {id}" + ), + None => { + let pinged = match &entry.liveness { + Ok(supervisor::Liveness::Alive(reply)) => reply.launch_id.as_str(), + _ => "unknown", + }; + format!( + "socket {} ({liveness}{sidecar_note}, ping names launch {pinged}): its name is \ + outside the DevFlow socket scheme, so `--launch` cannot address it", + entry.socket_path.display() + ) + } + } +} + +/// The socket arm of [`stop`]. Probes the located socket and acts on its +/// state: ALIVE → `shutdown`; STALE → [`stop_stale_socket`]; GONE → nothing +/// here (the caller falls back to the lock holder); a probe error → a +/// refusal that reaches no teardown (R-6). +/// +/// A `handed_off` reply means the run moved to its successor: the state is +/// reloaded and the record it names is asked, up to +/// [`MAX_SHUTDOWN_REQUESTS`] requests in all; then `stop` fails, recording +/// nothing (crash row C13). +fn stop_via_socket( + project_root: &Path, + phase: PhaseId, + cache_dir: Option<&Path>, +) -> Result { + let mut handed_off = 0; + for _ in 0..MAX_SHUTDOWN_REQUESTS { + let Some(located) = locate_phase_socket(project_root, phase, cache_dir)? else { + return Ok(SocketStop::NoLiveSocket); + }; + let ended = match supervisor::probe(&located.socket) { + Ok(supervisor::Liveness::Gone) => return Ok(SocketStop::NoLiveSocket), + Err(error) => return Err(probe_refusal(phase, &located.socket, &error)), + Ok(supervisor::Liveness::Stale) => { + stop_stale_socket(phase, &located)?; + true + } + Ok(supervisor::Liveness::Alive(_)) => match shutdown_supervisor(phase, &located)? { + ShutdownResult::Ended => true, + ShutdownResult::Relocate => false, + ShutdownResult::HandedOff => { + handed_off += 1; + false + } + }, + }; + if ended { + return Ok(match located.unloadable_state { + Some(state_file) => SocketStop::EndedStateUnloadable(state_file), + None => SocketStop::Ended, + }); + } + } + Err(CliError::Message(format!( + "stop: phase {phase}'s supervisor answered `handed_off` to {handed_off} of \ + {MAX_SHUTDOWN_REQUESTS} shutdown requests (the first and 3 retries, each against the \ + record a fresh state reload named); nothing was stopped or recorded — run `devflow \ + status` and `devflow stop --phase {phase}` again" + ))) +} + +/// What one shutdown attempt in [`stop_via_socket`] came to. +enum ShutdownResult { + /// The supervisor (or, after it died, the backstop) ended the run. + Ended, + /// The supervisor had handed the run to its successor. + HandedOff, + /// The socket went away under the request: locate the run again. + Relocate, +} + +/// Ask the ALIVE supervisor at `located` to stop. +/// +/// A request that fails to reach an answer (a connect refused, a reset or a +/// broken pipe) may have met a supervisor dying under it: its listener is +/// released while the connection waits in the backlog, which +/// `request_shutdown` reports as an error (6d46008 made `probe` re-connect +/// for this, not `request_shutdown`). So the socket is probed once more: +/// STALE goes to [`stop_stale_socket`], GONE re-locates the run, and only a +/// supervisor that still answers, or a failed probe, makes the error stand. +fn shutdown_supervisor( + phase: PhaseId, + located: &LocatedSocket, +) -> Result { + let error = match supervisor::request_shutdown(&located.socket, shutdown_final_timeout()) { + Ok(supervisor::ShutdownOutcome::Stopped { + exit_code, + advance_suppressed, + }) => { + let advance = if advance_suppressed { + "advance suppressed" + } else { + "advance not suppressed" + }; + println!( + "stop: stopped phase {phase} through its supervisor (agent group ended, exit \ + code {exit_code}, {advance})" + ); + return Ok(ShutdownResult::Ended); + } + Ok(supervisor::ShutdownOutcome::HandedOff) => return Ok(ShutdownResult::HandedOff), + Ok(supervisor::ShutdownOutcome::Refused(reason)) => { + return Err(CliError::Message(format!( + "stop: phase {phase}'s supervisor at {} refused the shutdown ({reason}); nothing \ + was stopped or recorded", + located.socket.display() + ))); + } + Ok(other) => { + return Err(CliError::Message(format!( + "stop: phase {phase}'s supervisor gave an unknown shutdown answer ({other:?}); \ + nothing was recorded" + ))); + } + Err(error) => error, + }; + match supervisor::probe(&located.socket) { + Ok(supervisor::Liveness::Stale) => { + println!( + "stop: phase {phase}'s supervisor went away under the shutdown request ({error}); \ + its socket is now STALE" + ); + stop_stale_socket(phase, located).map(|()| ShutdownResult::Ended) + } + Ok(supervisor::Liveness::Gone) => Ok(ShutdownResult::Relocate), + _ => Err(CliError::Message(format!( + "stop: phase {phase}'s supervisor at {} did not answer the shutdown ({error}); \ + nothing was signalled or recorded", + located.socket.display() + ))), + } +} + +/// A probe that could not answer ALIVE, STALE or GONE: refused, naming the +/// error kind and the socket path, and never handed to the backstop (R-6). +fn probe_refusal(phase: PhaseId, socket: &Path, error: &supervisor::ProbeError) -> CliError { + CliError::Message(format!( + "stop: could not probe phase {phase}'s supervisor socket {} ({}: {error}); nothing was \ + signalled, removed or recorded", + socket.display(), + probe_error_kind(error) + )) +} + +/// A short name for a [`supervisor::ProbeError`]'s kind. +fn probe_error_kind(error: &supervisor::ProbeError) -> &'static str { + match error { + supervisor::ProbeError::Stat(_) => "stat failed", + supervisor::ProbeError::NotASocket(_) => "not a socket", + supervisor::ProbeError::ForeignOwner { .. } => "owned by another user", + supervisor::ProbeError::Connect(_) => "connect failed", + supervisor::ProbeError::NoReply(_) => "no valid ping reply", + _ => "probe failed", + } +} + +/// A STALE socket: its supervisor is gone, and its agent group may not be. +/// The sidecar beside the socket is read and handed, with the state's record +/// when there is one, to the 50-09 backstop, which validates the group's +/// identity before it signals anything. A sidecar that cannot be read is a +/// refusal: nothing else can validate the group. +fn stop_stale_socket(phase: PhaseId, located: &LocatedSocket) -> Result<(), CliError> { + let subject = format!("phase {phase}"); + let socket = supervisor::SocketPath::new(located.socket.clone()) + .map_err(|error| stale_refusal(&subject, &located.socket, &error.to_string()))?; + let sidecar = read_stale_sidecar(&subject, &socket)?; + run_backstop( + &subject, + &socket, + &sidecar, + located.record.as_ref(), + Some(phase), ) } +/// The sidecar of a STALE socket, or the refusal that names why it cannot be +/// used. +fn read_stale_sidecar( + subject: &str, + socket: &supervisor::SocketPath, +) -> Result { + supervisor::read_sidecar(&socket.sidecar_path()).map_err(|error| { + let why = match &error { + supervisor::SidecarError::Io(io) if io.kind() == std::io::ErrorKind::NotFound => { + "is missing".to_string() + } + supervisor::SidecarError::Json(parse) => format!("does not parse: {parse}"), + other => format!("could not be read: {other}"), + }; + stale_refusal(subject, socket.as_path(), &format!("its sidecar {why}")) + }) +} + +/// The refusal for a STALE socket whose agent group nothing can validate: +/// nothing is signalled or removed, and the manual repair is named. +fn stale_refusal(subject: &str, socket: &Path, why: &str) -> CliError { + CliError::Message(format!( + "stop: {subject}: socket {} is STALE and {why}; nothing recorded can validate its agent \ + group, so stop signals nothing — end its agent yourself and remove {} and {}", + socket.display(), + socket.display(), + socket.with_extension("json").display() + )) +} + +/// Run the backstop on a STALE socket and report what it did. A group it +/// ended, or one it proved gone (previous boot, empty group, recycled +/// leader), is a stop; a refusal prints the backstop's own text (the +/// members, the residual F1 limit, the inspection command) and fails. +/// `resumable` names the phase a teardown that signalled nothing leaves +/// resumable (R-8); a targeted stop of a superseded launch passes `None`. +fn run_backstop( + subject: &str, + socket: &supervisor::SocketPath, + sidecar: &supervisor::Sidecar, + record: Option<&SupervisorRecord>, + resumable: Option, +) -> Result<(), CliError> { + use supervisor::backstop::{BackstopDecision, BackstopOutcome}; + let outcome = + supervisor::backstop::teardown_stale(socket, sidecar, record).map_err(|error| { + CliError::Message(format!( + "stop: {subject}: the stale-socket backstop did not act ({error}); nothing was \ + recorded" + )) + })?; + match outcome { + BackstopOutcome::Killed { members, .. } => { + let pids: Vec = members.iter().map(|m| m.pid.to_string()).collect(); + println!( + "stop: {subject}: its supervisor is gone (STALE socket); ended its agent group {} \ + ({} member(s): pid {}) and removed {} and its sidecar", + sidecar.agent_pgid, + members.len(), + pids.join(", "), + socket.as_path().display() + ); + Ok(()) + } + BackstopOutcome::RemovedNothingSignalled(decision) => { + let why = match decision { + BackstopDecision::PreviousBoot => "previous boot", + BackstopDecision::EmptyGroup => "empty group", + BackstopDecision::LeaderRecycled => "recycled leader", + _ => "nothing to signal", + }; + let resume = resumable.map_or_else(String::new, |phase| { + format!("; the phase is resumable: run `devflow resume --phase {phase}`") + }); + println!( + "stop: {subject}: nothing was running ({why}); removed the stale socket {} and its \ + sidecar{resume}", + socket.as_path().display() + ); + Ok(()) + } + BackstopOutcome::Refused(decision) => Err(CliError::Message(format!( + "stop: {subject}: {}", + decision.describe(socket, sidecar) + ))), + other => Err(CliError::Message(format!( + "stop: {subject}: the stale-socket backstop gave an unknown outcome ({other:?}); \ + nothing was recorded" + ))), + } +} + /// The primary path: answer `phase`'s open gate with a rejection whose note /// contains the abort keyword, so `GateAction::from_response` resolves to /// `GateAction::Abort` rather than `LoopBack(Code)` — looping back would @@ -2030,15 +2797,19 @@ fn stop_with_existing_response( } } -/// The fallback: signal the process recorded in `phase`'s per-phase lock -/// file — the process actually running `advance()` and holding the lock -/// across the gate wait. Fail-closed on identity (T-23-52): a live pid that -/// does not look like a devflow process is refused, not signalled, since -/// the lock may be stale with a recycled pid. Never reads -/// `state.monitor_pid` — see [`stop`]'s doc comment. +/// The fallback once no live supervisor socket remains (GONE, or none +/// recorded): signal the process recorded in `phase`'s per-phase lock file, +/// such as a foreground `advance()` holding the lock. Fail-closed on +/// identity (T-23-52): the pid is signalled only when it matches the start +/// time the lock recorded, since the lock may be stale with a recycled pid. +/// The lock file is the only place this reads a pid from; state is never +/// one. fn stop_via_lock(project_root: &Path, phase: PhaseId) -> Result { let Some((pid_str, _path)) = lock::holder(project_root, phase) else { - println!("stop: no lock held for phase {phase} — nothing is running `advance()`"); + println!( + "stop: no live supervisor socket and no lock held for phase {phase} — nothing is \ + running" + ); return Ok(false); }; let Ok(pid) = pid_str.parse::() else { @@ -2411,54 +3182,51 @@ fn default_logs_phase(project_root: &Path) -> Result { }) } -/// Refuse `start` while the phase's recorded monitor or agent is alive -/// (criterion 2, D-01: one writer per phase). `start` calls this under the -/// per-phase lock and before any gate cleanup, git mutation or `save_state`, -/// so a refusal writes nothing. `--force` does not bypass it. -/// -/// Liveness is pid liveness through the helpers `status` uses -/// (`agent_pid_from_file`, `agent::agent_running`), not identity: neither pid -/// records a start time. A recycled pid gives a fail-closed false refusal, -/// accepted by the operator with Option A (2026-09-22). +/// Refuse `start` while the phase's run is live (criterion 2, D-01: one +/// writer per phase). `start` calls this under the per-phase lock and before +/// any gate cleanup, git mutation or `save_state`, so a refusal writes +/// nothing. `--force` does not bypass it. See +/// [`refuse_launch_over_a_live_run`] for what counts as live. /// -/// No-state carve-out (operator-approved 2026-09-22): without a state file this -/// returns Ok even if the agent pid file names a live process, because -/// `recover --clean` leaves that file behind and a recycled pid would otherwise -/// block `start` with no DevFlow repair. A state file that does not load is -/// checked on the agent pid file only. +/// No-state carve-out (operator-approved 2026-09-22, unchanged by D-12): +/// without a state file the agent pid file is not read, so a live process it +/// names does not refuse, because `recover --clean` leaves that file behind +/// and a recycled pid would otherwise block `start` with no DevFlow repair. +/// The supervisor sockets of this root and phase are still swept. /// /// Out of scope: the lock-free window between agent exit and `advance` taking /// the lock stays backlog 999.136. This check says nothing about a gate waiter. fn refuse_start_over_a_live_run(project_root: &Path, phase: PhaseId) -> Result<(), CliError> { - refuse_launch_over_a_live_run(project_root, phase, LaunchVerb::Start) + refuse_launch_over_a_live_run( + project_root, + phase, + LaunchVerb::Start, + registry::cache_dir().as_deref(), + ) } /// Criterion 2, D-01 and D-02's `resume` half, promoted by the operator on /// 2026-09-23 as backlog 999.140: `pipeline_launch::resume` calls this just /// after it has acquired its phase lock and before it loads or writes state. /// The `devflow resume` CLI arm is resume's only production caller; no monitor -/// or advance path calls resume, so a recorded pid is never that process. An -/// agent that runs `devflow resume` for its own phase is refused correctly. -/// -/// Liveness is shared with `start` and uses the same pid-only Option A that -/// the operator accepted for start. A recycled pid can therefore refuse a -/// resume; that distinct resume cost is the known limit recorded in 48-20, -/// not an operator-approved permanent limit. Without a state file, resume -/// still reports its own missing-state error. For an unloadable state, this -/// checks only the agent pid file (999.139, backlog), and resume then cannot -/// launch because its own `load_state` fails immediately afterwards. +/// or advance path calls resume. An agent that runs `devflow resume` for its +/// own phase is refused correctly. /// -/// Production hints remain unchanged in this plan (backlog 999.142): status -/// and doctor for a dead monitor with a live agent, doctor's dead-agent and -/// gate-pending findings, cleanup's worktree messages, and `no_waiter_repair` -/// through the gate verbs, gate sweep, stop and doctor (48-20 H1-H10). The -/// agent-exit-to-advance window remains 999.136, and this check says nothing -/// about a gate waiter. +/// Liveness is shared with `start` ([`refuse_launch_over_a_live_run`]). +/// Without a state file, resume still reports its own missing-state error. +/// For a state that will not load, the guard finds the run through its +/// sidecar (R-12, 999.139), and resume then cannot launch anyway because its +/// own `load_state` fails immediately afterwards. pub(crate) fn refuse_resume_over_a_live_run( project_root: &Path, phase: PhaseId, ) -> Result<(), CliError> { - refuse_launch_over_a_live_run(project_root, phase, LaunchVerb::Resume) + refuse_launch_over_a_live_run( + project_root, + phase, + LaunchVerb::Resume, + registry::cache_dir().as_deref(), + ) } #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -2467,121 +3235,332 @@ enum LaunchVerb { Resume, } -fn refuse_launch_over_a_live_run( - project_root: &Path, - phase: PhaseId, - verb: LaunchVerb, -) -> Result<(), CliError> { - let loaded = workflow::load_state(project_root, phase); - if loaded.is_err() && !workflow::state_path(project_root, phase).exists() { - return Ok(()); - } - let monitor = loaded - .ok() - .and_then(|state| state.monitor_pid) +impl LaunchVerb { + fn label(self) -> &'static str { + match self { + LaunchVerb::Start => "start", + LaunchVerb::Resume => "resume", + } + } + + /// The command a refusal tells the operator to run again. + fn rerun(self, phase: PhaseId) -> String { + match self { + LaunchVerb::Start => "devflow start".to_string(), + LaunchVerb::Resume => format!("devflow resume --phase {phase}"), + } + } +} + +/// The shared start/resume guard (D-09, R-12, 999.139). Two halves, either of +/// which refuses: +/// +/// - **Supervisor** ([`possibly_live_sockets`]). The state loads → probe its +/// `supervisor` record: ALIVE is live; STALE is live unless +/// [`supervisor::backstop::assess`] proves its agent group gone +/// ([`stale_group_may_live`]); a probe error is live (fail closed, +/// T-50-40); GONE is not. The state is missing or will not load → +/// [`supervisor::sweep_for_in`] for this root and phase: an entry whose +/// sidecar parsed is classified the same way (with no record); an entry +/// with no usable sidecar, which 50-06 attributes by its file name or ping +/// reply, is live unless it probes GONE (RV-05: nothing recorded can prove +/// its group dead). A socket directory that exists but cannot be listed +/// refuses (the 50-06 threat flag: the sweep reads it as empty). With no +/// cache dir, no sweep runs, as for `stop` and `recover`. +/// - **Agent pid file** (unchanged, D-12): a live pid there is live. Pid +/// liveness, not identity: a recycled pid gives a fail-closed false +/// refusal (Option A, 2026-09-22). Not read without a state file (the +/// no-state carve-out on [`refuse_start_over_a_live_run`]). +/// +/// A loaded state with no record over a live supervisor (a launcher that +/// died before saving it, T-50-90/T-50-91) is accepted until Phase 50.1. +fn refuse_launch_over_a_live_run( + project_root: &Path, + phase: PhaseId, + verb: LaunchVerb, + cache_dir: Option<&Path>, +) -> Result<(), CliError> { + let state_file = workflow::state_path(project_root, phase); + let state = workflow::load_state(project_root, phase).ok(); + let no_state = state.is_none() && !state_file.exists(); + let sockets = possibly_live_sockets(project_root, phase, state.as_ref(), cache_dir).map_err( + |(dir, why)| { + unlistable_socket_dir_refusal(phase, verb, (&state_file, no_state, &dir), &why) + }, + )?; + let agent = (!no_state) + .then(|| agent_pid_from_file(project_root, phase)) + .flatten() .filter(|&pid| agent::agent_running(pid)); - let agent = agent_pid_from_file(project_root, phase).filter(|&pid| agent::agent_running(pid)); - let live: Vec<(&str, u32)> = [("monitor", monitor), ("agent", agent)] - .into_iter() - .filter_map(|(role, pid)| pid.map(|pid| (role, pid))) - .collect(); - if live.is_empty() { + if sockets.is_empty() && agent.is_none() { return Ok(()); } - Err(CliError::Message(live_run_refusal(phase, verb, &live))) + Err(CliError::Message(live_run_refusal( + phase, verb, &sockets, agent, + ))) } -/// The refusal text for [`refuse_start_over_a_live_run`]. It fires while -/// `start` holds the lock, so no other process holds it: `stop` would signal -/// nothing here (see `stop_via_lock`). -fn live_run_refusal(phase: PhaseId, verb: LaunchVerb, live: &[(&str, u32)]) -> String { - match verb { - LaunchVerb::Start => start_live_run_refusal(phase, live), - LaunchVerb::Resume => resume_live_run_refusal(phase, live), +/// The supervisor half of [`refuse_launch_over_a_live_run`], shared with +/// `recover --clean --phase N` ([`refuse_a_live_supervisor`]): the sockets of +/// this root and phase that may still run. `state` is the loaded state, or +/// `None` when it is missing or does not load, which sweeps the socket +/// directory under `cache_dir` instead. `Err` names that directory and why +/// it cannot be listed. +fn possibly_live_sockets( + project_root: &Path, + phase: PhaseId, + state: Option<&State>, + cache_dir: Option<&Path>, +) -> Result, (PathBuf, String)> { + if let Some(state) = state { + return Ok(state + .supervisor + .as_ref() + .and_then(recorded_live_socket) + .into_iter() + .collect()); + } + let Some(cache_dir) = cache_dir else { + return Ok(Vec::new()); + }; + if let Some(why) = socket_dir_unlistable(cache_dir) { + return Err((registry::socket_dir_in(cache_dir), why)); } + Ok(supervisor::sweep_for_in(cache_dir, project_root, phase) + .iter() + .filter_map(swept_live_socket) + .collect()) } -fn start_live_run_refusal(phase: PhaseId, live: &[(&str, u32)]) -> String { - let named = live - .iter() - .map(|(role, pid)| format!("{role} pid {pid}")) - .collect::>() - .join(", "); - let checks = live - .iter() - .map(|(role, pid)| format!("\n {role} pid {pid}: ps -ww -o pid=,lstart=,args= -p {pid}")) - .collect::(); - let padded = phase.padded(); - // Identity details below match the argv the monitors are spawned with: - // `pipe_owning_monitor_args` and the Legacy script plus `advance_tail` in - // devflow-core's monitor.rs (48-REVIEW CR-01). A Legacy monitor whose - // agent has exited runs `devflow advance` as a foreground child and - // defers SIGTERM until it returns (48-REVIEW WR-02). - format!( - "phase {phase}: a run is still live ({named} alive) — refusing to start; nothing was written\n\ - No process holds the phase lock right now, so `devflow stop --phase {phase}` would only mark the \ - state stopped: it signals nothing and does not end this run. stop ends a run through the process \ - that holds the lock: one parked at a gate, or this run's own `devflow advance` once it takes the \ - lock after this refusal.\n\ - Check each pid before signalling it. `ps -p` shows only the executable name, which cannot tell \ - phases apart; `-ww` keeps a long command line from being cut at the terminal width:{checks}\n\ - This phase's monitor is either a `__monitor` process with this project's path and `--phase {phase}` \ - in its args, or an `sh -c` script that names `.devflow/phase-{padded}-` files and ends in `advance` \ - with `--phase {phase}`. This phase's agent has that monitor as its parent (`ps -o ppid= -p `); \ - an agent whose monitor has exited cannot be tied to this phase that way, so do not signal it on the \ - pid alone.\n\ - To end this run now, send SIGTERM to the confirmed monitor first — signalling the agent first lets \ - the monitor launch the next stage. If the agent has already exited, an `sh` monitor may be running \ - `devflow advance` as a foreground child: the monitor defers SIGTERM until that child returns, and \ - the child may take the lock as soon as this start exits. Find it with \ - `ps -ww -o pid=,args= --ppid ` and signal that child, not only the monitor.\n\ - If the named processes have exited, run `devflow start` again. If a named pid is live but is not \ - this phase's process (a recycled pid), `start` refuses again: run \ - `devflow recover --clean --phase {phase}` first, then `devflow start`. Do neither while this \ - phase's processes are live: `recover --clean` clears state even while an agent runs, and a later \ - `start` would launch a second agent beside it." +/// A supervisor socket the launch guard counts as possibly live. +struct LiveSocket { + socket: PathBuf, + /// The record's or sidecar's launch id; for a socket with no usable + /// sidecar, its file name's, else its ping reply's, else `unknown`. + launch_id: String, + /// `ALIVE`, `STALE`, or why the socket could not be probed. + liveness: String, + /// Why the sidecar cannot be used, for a socket with none that parses: + /// `stop --launch` cannot validate it, so the refusal names bare `stop`. + sidecar_problem: Option, +} + +impl LiveSocket { + /// The refusal line that names this socket's repair. + fn repair_line(&self, phase: PhaseId) -> String { + let (id, liveness, socket) = (&self.launch_id, &self.liveness, self.socket.display()); + match &self.sidecar_problem { + None => format!( + "launch {id} ({liveness}, socket {socket}): run \ + `devflow stop --phase {phase} --launch {id}` first" + ), + Some(problem) => format!( + "socket {socket} ({liveness}, launch {id}, its sidecar {problem}): run \ + `devflow stop --phase {phase}` first — it shuts a supervisor that answers down \ + through its socket; one that does not answer cannot be validated without its \ + sidecar, so end its agent yourself and remove `{socket}` and `{}`", + self.socket.with_extension("json").display() + ), + } + } +} + +/// The loaded state's record as a [`LiveSocket`], or `None` when its socket +/// is GONE or STALE over a group the backstop proves gone. +fn recorded_live_socket(record: &SupervisorRecord) -> Option { + let live = |liveness: &str, sidecar_problem: Option| LiveSocket { + socket: record.socket_path.clone(), + launch_id: record.launch_id.clone(), + liveness: liveness.to_string(), + sidecar_problem, + }; + match supervisor::probe(&record.socket_path) { + Ok(supervisor::Liveness::Gone) => None, + Ok(supervisor::Liveness::Alive(_)) => Some(live("ALIVE", None)), + Err(error) => Some(live(&probe_failure(&error), None)), + Ok(supervisor::Liveness::Stale) => { + match supervisor::read_sidecar(&record.socket_path.with_extension("json")) { + Ok(sidecar) => { + let decision = supervisor::backstop::assess(&sidecar, Some(record)); + stale_group_may_live(&decision).then(|| live("STALE", None)) + } + Err(supervisor::SidecarError::Io(error)) + if error.kind() == std::io::ErrorKind::NotFound => + { + Some(live("STALE", Some("is missing".to_string()))) + } + Err(error) => Some(live("STALE", Some(format!("is unusable ({error})")))), + } + } + } +} + +/// A sweep entry for this root and phase as a [`LiveSocket`], or `None` +/// when it is GONE, or STALE with a parsed sidecar whose group the backstop +/// proves gone. An entry with no usable sidecar is live unless GONE (RV-05). +fn swept_live_socket(entry: &supervisor::SweepEntry) -> Option { + let liveness = match &entry.liveness { + Ok(supervisor::Liveness::Gone) => return None, + Ok(supervisor::Liveness::Alive(_)) => "ALIVE".to_string(), + Ok(supervisor::Liveness::Stale) => "STALE".to_string(), + Err(error) => probe_failure(error), + }; + let (launch_id, sidecar_problem) = match &entry.sidecar { + Some(sidecar) => { + let stale = matches!(entry.liveness, Ok(supervisor::Liveness::Stale)); + if stale && !stale_group_may_live(&supervisor::backstop::assess(sidecar, None)) { + return None; + } + (sidecar.launch_id.clone(), None) + } + None => { + let pinged = match &entry.liveness { + Ok(supervisor::Liveness::Alive(reply)) => Some(reply.launch_id.clone()), + _ => None, + }; + let problem = entry.sidecar_error.as_ref().map_or_else( + || "is missing".to_string(), + |error| format!("is unusable ({error})"), + ); + let id = entry.launch_id_in_name.clone().or(pinged); + (id.unwrap_or_else(|| "unknown".to_string()), Some(problem)) + } + }; + Some(LiveSocket { + socket: entry.socket_path.clone(), + launch_id, + liveness, + sidecar_problem, + }) +} + +/// A probe error as the liveness a refusal names. +fn probe_failure(error: &supervisor::ProbeError) -> String { + format!("probe failed ({}: {error})", probe_error_kind(error)) +} + +/// Whether a STALE socket's agent group may still run. Only the three +/// backstop decisions that prove the group gone (a previous boot, an empty +/// group, a recycled leader) say no; every refusal, an unprovable identity +/// (MAC-2), a live leader and any decision added later fail closed (T-50-40). +fn stale_group_may_live(decision: &supervisor::backstop::BackstopDecision) -> bool { + use supervisor::backstop::BackstopDecision; + !matches!( + decision, + BackstopDecision::PreviousBoot + | BackstopDecision::EmptyGroup + | BackstopDecision::LeaderRecycled ) } -fn resume_live_run_refusal(phase: PhaseId, live: &[(&str, u32)]) -> String { - let named = live - .iter() - .map(|(role, pid)| format!("{role} pid {pid}")) - .collect::>() - .join(", "); - let checks = live +/// The refusal for a state that is missing (`no_state`) or will not load +/// over a socket directory that cannot be listed: a live run may be named +/// only by its socket there, so the launch fails closed (50-06 threat flag). +fn unlistable_socket_dir_refusal( + phase: PhaseId, + verb: LaunchVerb, + (state_file, no_state, dir): (&Path, bool, &Path), + why: &str, +) -> CliError { + let label = verb.label(); + let state = if no_state { + format!("there is no state file {}", state_file.display()) + } else { + format!("the state file {} does not load", state_file.display()) + }; + CliError::Message(format!( + "phase {phase}: {state}, and the supervisor socket directory {} cannot be listed ({why}) \ + — refusing to {label}; nothing was written\n\ + A live run of this phase may be named only by its socket there, so {label} fails \ + closed. Make {} listable by this user (DevFlow creates it mode 0700), then run `{}` again.", + dir.display(), + dir.display(), + verb.rerun(phase) + )) +} + +/// The refusal text for [`refuse_launch_over_a_live_run`]. It fires while the +/// launcher holds the lock: each live socket is named with the `stop` line +/// that ends it (50-14), and a live agent pid with the identity checks the +/// operator must make before signalling it. +fn live_run_refusal( + phase: PhaseId, + verb: LaunchVerb, + sockets: &[LiveSocket], + agent: Option, +) -> String { + let mut named: Vec = sockets .iter() - .map(|(role, pid)| format!("\n {role} pid {pid}: ps -ww -o pid=,lstart=,args= -p {pid}")) - .collect::(); - let padded = phase.padded(); + .map(|live| format!("supervisor launch {} {}", live.launch_id, live.liveness)) + .collect(); + named.extend(agent.map(|pid| format!("agent pid {pid} alive"))); + let mut text = format!( + "phase {phase}: a run is still live ({}) — refusing to {}; nothing was written", + named.join(", "), + verb.label() + ); + if !sockets.is_empty() { + text.push_str( + "\n`devflow stop` ends a supervisor through its socket, whether or not the state \ + file loads; on a STALE socket it signals the agent group only after proving that \ + group is this launch's. Run each line below first:", + ); + for live in sockets { + text.push_str("\n "); + text.push_str(&live.repair_line(phase)); + } + } + if let Some(pid) = agent { + text.push('\n'); + text.push_str(&agent_refusal_guidance(phase, verb, pid)); + } + text.push_str(&format!( + "\nOnce every supervisor and process named above is gone, run `{}` again.", + verb.rerun(phase) + )); + text +} + +/// The live-run refusal's paragraph for a live agent its pid file names: +/// `stop` never signals that pid, so the operator confirms its identity +/// before signalling it, and a recycled pid has its own repair per verb. +fn agent_refusal_guidance(phase: PhaseId, verb: LaunchVerb, pid: u32) -> String { + let stopped_note = match verb { + LaunchVerb::Start => "", + LaunchVerb::Resume => { + " A state already marked stopped does not mean these processes have exited: \ + `devflow status` names `devflow resume` for a phase whose supervisor is GONE even \ + while its agent is still alive, and resume refuses until every named process has \ + exited." + } + }; + let recycled = match verb { + LaunchVerb::Start => format!( + "If a named pid is live but is not this phase's process (a recycled pid), `start` \ + refuses again: run `devflow recover --clean --phase {phase}` first, then \ + `devflow start`." + ), + LaunchVerb::Resume => format!( + "If a named pid is live but is not this phase's process (a recycled pid), `resume` \ + refuses again, and no DevFlow verb clears a recorded pid while keeping the phase's \ + work: `devflow recover --clean --phase {phase}` removes the phase state, after \ + which `resume` has nothing to resume, and `devflow start` refuses while the \ + phase's worktree or branch exists; `devflow start --force` recreates them from \ + the base and discards the phase's work." + ), + }; format!( - "phase {phase}: a run is still live ({named} alive) — refusing to resume; nothing was written\n\ - No process holds the phase lock right now, so `devflow stop --phase {phase}` would only mark the \ - state stopped: it signals nothing and does not end this run. stop ends a run through the process \ - that holds the lock: one parked at a gate, or this run's own `devflow advance` once it takes the \ - lock after this refusal.\n\ - A state already marked stopped does not mean these processes have exited. `devflow status` and \ - `devflow doctor` name `devflow resume` for a phase whose monitor is dead even while its agent is \ - still alive; resume refuses until every named process has exited.\n\ - Check each pid before signalling it. `ps -p` shows only the executable name, which cannot tell \ - phases apart; `-ww` keeps a long command line from being cut at the terminal width:{checks}\n\ - This phase's monitor is either a `__monitor` process with this project's path and `--phase {phase}` \ - in its args, or an `sh -c` script that names `.devflow/phase-{padded}-` files and ends in `advance` \ - with `--phase {phase}`. This phase's agent has that monitor as its parent (`ps -o ppid= -p `); \ - an agent whose monitor has exited cannot be tied to this phase that way, so do not signal it on the \ - pid alone.\n\ - To end this run now, send SIGTERM to the confirmed monitor first — signalling the agent first lets \ - the monitor launch the next stage. If the agent has already exited, an `sh` monitor may be running \ - `devflow advance` as a foreground child: the monitor defers SIGTERM until that child returns, and \ - the child may take the lock as soon as this resume exits. Find it with \ - `ps -ww -o pid=,args= --ppid ` and signal that child, not only the monitor.\n\ - If the named processes have exited, run `devflow resume --phase {phase}` again. If a named pid is \ - live but is not this phase's process (a recycled pid), `resume` refuses again, and no DevFlow verb \ - clears a recorded pid while keeping the phase's work: `devflow recover --clean --phase {phase}` \ - removes the phase state, after which `resume` has nothing to resume, and `devflow start` refuses \ - while the phase's worktree or branch exists; `devflow start --force` recreates them from the base \ - and discards the phase's work. Do neither while this phase's processes are live: `recover --clean` \ - clears state even while an agent runs, and a later `start` would launch a second agent beside it." + "The agent pid file names a live process, and `devflow stop` never signals a pid from \ + the agent pid file.{stopped_note} Check each pid before signalling it. `ps -p` shows \ + only the executable name, which cannot tell phases apart; `-ww` keeps a long command \ + line from being cut at the terminal width:\n \ + agent pid {pid}: ps -ww -o pid=,lstart=,args= -p {pid}\n\ + This phase's agent has its supervisor, a `__monitor` process with this project's path \ + and `--phase {phase}` in its args, as its parent (`ps -o ppid= -p `); an agent \ + whose supervisor has exited cannot be tied to this phase that way, so do not signal it \ + on the pid alone. {recycled} Do neither while this phase's processes are live: \ + `recover --clean` clears state even while an agent runs, and a later `start` would \ + launch a second agent beside it." ) } @@ -2729,6 +3708,43 @@ fn print_open_branches(project_root: &Path) { } } +/// `recover --clean --phase N` is a file cleaner, not a teardown (D-06): it +/// refuses while the phase's supervisor may still run, by the launch guard's +/// supervisor rule ([`possibly_live_sockets`]): an ALIVE socket, a STALE one +/// whose agent group the backstop does not prove gone, or a probe that +/// fails, found through the state's record, or, when the state is missing or +/// does not load, through a sweep of this root and phase (which refuses when +/// the socket directory cannot be listed). The agent pid file is not read. +/// Nothing is deleted. +fn refuse_a_live_supervisor( + project_root: &Path, + phase: PhaseId, + cache_dir: Option<&Path>, +) -> Result<(), CliError> { + let state = workflow::load_state(project_root, phase).ok(); + let sockets = possibly_live_sockets(project_root, phase, state.as_ref(), cache_dir).map_err( + |(dir, why)| { + CliError::Message(format!( + "phase {phase}: the supervisor socket directory {} cannot be listed ({why}), so a \ + live supervisor cannot be ruled out — recover --clean deleted nothing", + dir.display() + )) + }, + )?; + if sockets.is_empty() { + return Ok(()); + } + let mut text = format!( + "phase {phase} has a supervisor that may still run — recover --clean deleted nothing. \ + Run each line below first:" + ); + for live in &sockets { + text.push_str("\n "); + text.push_str(&live.repair_line(phase)); + } + Err(CliError::Message(text)) +} + pub(crate) fn recover_cmd( project_root: &Path, do_clean: bool, @@ -2736,9 +3752,12 @@ pub(crate) fn recover_cmd( ) -> Result<(), CliError> { if do_clean { match phase { - // Explicit phase: clear it regardless of staleness (14-CR-01's - // escape hatch for a wedged-but-fresh run). + // Explicit phase: cleaned when its lock holder is dead or its pid + // was recycled (a live pid whose start time differs from the one + // recorded, 999.128). A live, identity-confirmed holder still + // refuses, so a wedged live run needs `devflow stop` first. Some(phase) => { + refuse_a_live_supervisor(project_root, phase, registry::cache_dir().as_deref())?; let report = match recover::clean_phase_report(project_root, phase) { Ok(report) => report, Err(recover::RecoverError::Lock(lock::LockError::Contended { @@ -3091,6 +4110,14 @@ fn doctor_checks() -> Vec { /// Audit the environment and report what's installed, missing, or broken. pub(crate) fn doctor(project_root: &Path, json: bool) -> Result<(), CliError> { + doctor_in(project_root, json, registry::cache_dir().as_deref()) +} + +/// [`doctor`] against an explicit cache dir, whose `sock/` directory the +/// read-only supervisor sweep lists (50-12); `None` (no cache dir can be +/// resolved) skips the sweep without an error. Split out so tests sweep a +/// tempdir, never the operator's real cache. +fn doctor_in(project_root: &Path, json: bool, cache_dir: Option<&Path>) -> Result<(), CliError> { let checks = doctor_checks(); let facts = collect_phase_facts(project_root); @@ -3100,6 +4127,12 @@ pub(crate) fn doctor(project_root: &Path, json: bool) -> Result<(), CliError> { // scan, which never signals anything (T-25-62, doctor's read-only // contract). let stray_findings = collect_stray_process_findings(project_root); + // 50-12 (D-06): the read-only socket sweep — orphaned, superseded and + // deleted-root supervisors. It removes nothing and signals nothing. + let orphans = cache_dir + .map(|cache| collect_orphan_supervisors(project_root, cache)) + .unwrap_or_default(); + let sweep_unlistable = cache_dir.and_then(socket_dir_unlistable); if json { // WR-01 (18-fix): a single top-level JSON document — @@ -3113,7 +4146,13 @@ pub(crate) fn doctor(project_root: &Path, json: bool) -> Result<(), CliError> { // planning-doc check (D-05) and this plan's stray-process finding // (999.44) each extend this SAME object with one more key rather // than forking a second array. - let body = doctor_json_body(&checks, &facts, &doc_findings, &stray_findings); + let body = doctor_json_body( + &checks, + &facts, + &doc_findings, + &stray_findings, + (&orphans, sweep_unlistable.as_deref()), + ); println!( "{}", serde_json::to_string_pretty(&body).expect("doctor --json body must serialize") @@ -3139,6 +4178,10 @@ pub(crate) fn doctor(project_root: &Path, json: bool) -> Result<(), CliError> { print!("{}", render_reconciliation_text(&facts)); print!("{}", render_planning_doc_text(&doc_findings)); print!("{}", render_stray_process_text(&stray_findings)); + print!( + "{}", + render_orphan_supervisors_text(&orphans, sweep_unlistable.as_deref()) + ); } Ok(()) @@ -3700,11 +4743,13 @@ pub(crate) struct PhaseFacts { pub(crate) waiter: lock::HolderStatus, pub(crate) agent_pid: Option, pub(crate) agent_alive: bool, - /// The monitor pid recorded in `State.monitor_pid` (18b). `None` means - /// no monitor has been spawned for this state yet, or the state was - /// written by a binary predating the field — never treated as a problem. - pub(crate) monitor_pid: Option, - pub(crate) monitor_alive: bool, + /// The supervisor's liveness, from one probe of the socket + /// `state.supervisor` records (50-12) — never a pid. + pub(crate) supervisor: PhaseLiveness, + /// The launch id `state.supervisor` records; `None` when no supervisor + /// is recorded. It identifies the supervisor in findings in place of its + /// socket path (T-18-01). + pub(crate) supervisor_launch_id: Option, /// The most recent event's `event` field value, for display context. pub(crate) last_event: Option, /// The `stage` field of the most recent `stage_launched` event; `None` @@ -3714,8 +4759,8 @@ pub(crate) struct PhaseFacts { pub(crate) feature_branch_exists: bool, /// Whether this phase was intentionally halted by `devflow start --until /// ` (20c) — `State.stopped`. A stopped phase's dead agent - /// pid/stale monitor pid are expected, not a crash; both - /// `check_dead_agent` and `check_dead_monitor` must recognize this + /// pid/exited supervisor are expected, not a crash; both + /// `check_dead_agent` and `check_supervisor` must recognize this /// marker instead of reporting a `Problem` (review: Codex HIGH — the /// doctor gap is bigger than `check_dead_agent` alone). pub(crate) stopped: bool, @@ -3808,28 +4853,38 @@ fn check_dead_agent(facts: &PhaseFacts) -> Option { }) } -/// The recorded monitor pid is dead — nothing will call `devflow advance` -/// for this phase, whether or not the agent is also dead (an agent that -/// outlived its monitor is orphaned too, since nothing will advance it when -/// it exits either). Reuses `liveness` rather than re-deriving the matrix, -/// so the two copies can never drift (18b, T-18-11's `Unknown` guard applies -/// here transitively — an unrecorded monitor is silently `Unknown`, never a -/// finding). -fn check_dead_monitor(facts: &PhaseFacts) -> Option { - if facts.stopped - || liveness(facts.monitor_pid, facts.monitor_alive, facts.agent_alive) != Liveness::Stuck - { - return None; - } - let pid = facts.monitor_pid?; +/// The supervisor's liveness needs a repair (50-12): STALE (its socket +/// remains, nothing answers, its agent group may still run), GONE while the +/// phase is not stopped (nothing will advance it), or a probe that could not +/// answer. Reuses [`phase_liveness`]'s result from the facts rather than +/// re-deriving it, so `status` and `doctor` cannot drift. ALIVE, a stopped +/// GONE and NotLaunched are never findings (T-18-11: an unrecorded +/// supervisor is never a problem). Identifies the supervisor by phase and +/// launch id, never by its socket path (T-18-01). +fn check_supervisor(facts: &PhaseFacts) -> Option { + let phase = facts.phase; + let repair = match &facts.supervisor { + PhaseLiveness::Stale => { + format!("devflow stop --phase {phase} && devflow resume --phase {phase}") + } + PhaseLiveness::GoneWhileActive => format!("devflow resume --phase {phase}"), + // `status` prints the operator's own socket path, which doctor + // must not. + PhaseLiveness::ProbeFailed(_) => "devflow status".to_string(), + PhaseLiveness::Alive { .. } | PhaseLiveness::Gone | PhaseLiveness::NotLaunched => { + return None; + } + }; + let launch = facts.supervisor_launch_id.as_deref().unwrap_or("?"); Some(PhaseFinding { - phase: facts.phase, + phase, severity: Severity::Problem, detail: format!( - "phase {}: monitor pid {pid} recorded but not running at stage {}", - facts.phase, facts.stage + "phase {phase}: supervisor launch {launch} at stage {} is {}", + facts.stage, + facts.supervisor.describe() ), - repair: Some(format!("devflow resume --phase {}", facts.phase)), + repair: Some(repair), }) } @@ -3884,7 +4939,7 @@ fn reconcile_phase(facts: &PhaseFacts) -> Vec { check_open_gate_without_waiter(facts), check_orphan_gate(facts), check_dead_agent(facts), - check_dead_monitor(facts), + check_supervisor(facts), check_stage_event_drift(facts), check_missing_branch(facts), ] @@ -3928,8 +4983,11 @@ fn build_phase_facts( let waiter = lock::holder_status(project_root, phase); let agent_pid = agent_pid_from_file(project_root, phase); let agent_alive = agent_pid.is_some_and(agent::agent_running); - let monitor_pid = state.monitor_pid; - let monitor_alive = monitor_pid.is_some_and(agent::agent_running); + let supervisor = probe_phase(&state); + let supervisor_launch_id = state + .supervisor + .as_ref() + .map(|record| record.launch_id.clone()); let last_event = last_events.remove(&phase).map(|summary| summary.event); let last_launched_stage = last_event.as_ref().and_then(last_launched_stage_from_event); let last_event_name = last_event @@ -3953,8 +5011,8 @@ fn build_phase_facts( waiter, agent_pid, agent_alive, - monitor_pid, - monitor_alive, + supervisor, + supervisor_launch_id, last_event: last_event_name, last_launched_stage, open_gate_stages, @@ -4037,12 +5095,29 @@ fn render_reconciliation_json(facts: &[PhaseFacts]) -> serde_json::Value { "detail": finding.detail, "repair": finding.repair, "last_event": phase_facts.last_event, + "supervisor": supervisor_json(phase_facts), }) }) .collect(), ) } +/// A phase's supervisor facts for `doctor --json` (50-12): its liveness +/// label, the owner and launch shape an ALIVE ping reported, and the +/// recorded launch id. Never the socket path (T-18-01, T-50-32). +fn supervisor_json(facts: &PhaseFacts) -> serde_json::Value { + let (owner, launch) = match &facts.supervisor { + PhaseLiveness::Alive { owner, launch } => (Some(owner), Some(launch)), + _ => (None, None), + }; + serde_json::json!({ + "liveness": facts.supervisor.label(), + "owner": owner, + "launch": launch, + "launch_id": facts.supervisor_launch_id, + }) +} + /// Build `doctor --json`'s `"environment"` array from the pre-existing /// tool/env checks (WR-01, 18-fix). Extracted so it can be composed with /// `render_reconciliation_json`'s array into ONE JSON document instead of @@ -4073,17 +5148,24 @@ fn checks_json_value(checks: &[Check]) -> serde_json::Value { /// [...], "planning_doc_staleness": [...]}` — 21b's addition (D-05) extends /// this SAME object with a third key rather than forking a second reporter /// or printing a second top-level array. +/// 50-12 adds `"orphan_supervisors"` (the socket sweep's findings) and +/// `"orphan_supervisor_sweep_error"` (`null`, or why the socket directory +/// could not be listed, so an empty array is never mistaken for a clean +/// sweep). fn doctor_json_body( checks: &[Check], facts: &[PhaseFacts], doc_findings: &[PlanningDocFinding], stray_findings: &[StrayProcessFinding], + (orphans, sweep_unlistable): (&[OrphanSupervisorFinding], Option<&str>), ) -> serde_json::Value { serde_json::json!({ "environment": checks_json_value(checks), "reconciliation": render_reconciliation_json(facts), "planning_doc_staleness": render_planning_doc_findings_json(doc_findings), "stray_processes": render_stray_process_findings_json(stray_findings), + "orphan_supervisors": render_orphan_supervisors_json(orphans), + "orphan_supervisor_sweep_error": sweep_unlistable, }) } @@ -4333,12 +5415,19 @@ fn stray_layer_label(layer: agent::StrayLayer) -> &'static str { } } -/// The set of pids that a live registry entry still reaches: every -/// [`devflow_core::state::State::monitor_pid`] recorded under any of -/// `roots` (via [`workflow::list_states`]) plus every lock holder pid for -/// those same roots' phases (via [`lock::holder_identity`]). A pid in this -/// set is by definition not a stray (CR-01, 25-15-PLAN.md) — a live -/// registry entry, lock file, or state file still names it. +/// The set of pids that a live registry entry still reaches: every lock +/// holder pid for the phases of `roots` (listed via +/// [`workflow::list_states`], read via [`lock::holder_identity`]). A pid in +/// this set is by definition not a stray (CR-01, 25-15-PLAN.md) — a live +/// lock file still names it. +/// +/// Lock holders only (50-22). A 3.x supervisor needs no entry: the `/proc` +/// census this set filters ([`agent::discover_stray_devflow_processes`]) +/// matches only the 2.x monitor-wrapper and `devflow advance` argv shapes +/// and cannot match a supervisor's (50-12, pinned by +/// `agent::tests::the_stray_census_matches_only_the_2x_process_shapes` and +/// by `census_never_sees_a_live_supervisor` in the supervisor e2e), so a +/// supervisor's pid can never reach a reap through this filter. /// /// Deliberately excludes the per-phase agent pid file: the `/proc` census /// this set filters ([`agent::discover_stray_devflow_processes`]) can only @@ -4353,8 +5442,8 @@ fn stray_layer_label(layer: agent::StrayLayer) -> &'static str { /// read-only path). /// /// RESIDUAL (T-25-15-08, not eliminated): this is a point-in-time read. A -/// root registered, or a `monitor_pid` written, between this read and a -/// caller's later signal is not covered by it. The untouched +/// root registered, or a lock taken, between this read and a caller's later +/// signal is not covered by it. The untouched /// `agent::is_same_process` identity re-confirmation and /// [`agent::STRAY_MIN_AGE`] age floor downstream remain the last line of /// defence in that window. @@ -4369,9 +5458,6 @@ pub(crate) fn registry_reachable_pids(roots: &[PathBuf]) -> HashSet { continue; } for state in workflow::list_states(root) { - if let Some(pid) = state.monitor_pid { - reachable.insert(pid); - } if let Some((pid, _start_time)) = lock::holder_identity(root, state.phase) { reachable.insert(pid); } @@ -4489,6 +5575,13 @@ pub(crate) fn build_stray_process_findings( /// into the reachable set via [`unreachable_stray_candidates`] so a project /// the machine registry has not recorded still has its own live monitors /// protected from being reported as orphans. +/// +/// The census matches only the two 2.x process shapes (the `sh -c` monitor +/// wrapper and its `devflow advance` child), which a 3.x binary never +/// starts, so this finding covers 2.x leftovers only (50-12). A 3.x +/// supervisor is found through its socket by [`check_supervisor`] and +/// [`collect_orphan_supervisors`] (which also takes over the deleted-root +/// class) and ended with `devflow stop --phase N --launch `. fn collect_stray_process_findings(project_root: &Path) -> Vec { build_stray_process_findings(&unreachable_stray_candidates(&[project_root.to_path_buf()])) } @@ -4534,6 +5627,245 @@ fn render_stray_process_text(findings: &[StrayProcessFinding]) -> String { out } +/// One supervisor socket `doctor`'s read-only sweep reports (50-12, D-06). +pub(crate) struct OrphanSupervisorFinding { + /// The path-free finding: detail and `devflow stop --phase N --launch + /// ` repair. + pub(crate) finding: PhaseFinding, + /// The launch id that addresses exactly this socket. + pub(crate) launch_id: String, + /// `alive` or `stale`. + pub(crate) liveness: &'static str, + /// Whether the project root the socket names still exists. `true` for a + /// sidecar-less socket, which is attributed to this (existing) root. + pub(crate) root_exists: bool, +} + +/// Every ALIVE or STALE supervisor socket under `cache_dir` that nothing +/// claims (50-12, D-06): (1) this root's sockets whose phase has no loadable +/// state, or whose state records no supervisor or another launch +/// (superseded); (2) machine-wide, sockets whose sidecar names a project +/// root that no longer exists (the 999.44 deleted-root class the 2.x `/proc` +/// census used to cover); (3) (R5-I3) sockets of this root whose sidecar is +/// missing or does not parse, attributed by file name through +/// [`supervisor::sweep_for_in`] so 50-06's digest rule stays in one place. +/// +/// Not listed, by design: a socket named outside the 50-05 scheme (it +/// carries no launch id, so no verb addresses it), and a sidecar-less socket +/// of a deleted root (only the digest survives, so nothing attributes it). +/// +/// Read-only (D-06, T-50-31): it sweeps, probes and loads state, and removes +/// nothing and signals nothing. Every finding's repair is `devflow stop +/// --phase N --launch ` (50-14), never the bare stop, which addresses +/// the state's own record and would stop a live successor (T-50-51). +fn collect_orphan_supervisors( + project_root: &Path, + cache_dir: &Path, +) -> Vec { + let mut findings = Vec::new(); + for entry in supervisor::sweep_in(cache_dir) { + let liveness = match &entry.liveness { + Ok(supervisor::Liveness::Alive(_)) => "alive", + Ok(supervisor::Liveness::Stale) => "stale", + _ => continue, + }; + if let Some(sidecar) = &entry.sidecar { + // A root that cannot be examined counts as present: only a + // proven-missing root is the deleted-root class. + let root_exists = sidecar.project_root.try_exists().unwrap_or(true); + let reason = if !root_exists { + Some(OrphanReason::RootDeleted) + } else if sidecar.project_root == project_root { + unclaimed_by_state(project_root, sidecar.phase, &sidecar.launch_id) + } else { + None + }; + if let Some(reason) = reason { + let launch = (sidecar.phase, sidecar.launch_id.as_str()); + findings.push(orphan_finding(launch, liveness, root_exists, &reason)); + } + continue; + } + // R5-I3: no usable sidecar, so the file name is the only address. + let (Some(launch_id), Some(phase)) = ( + entry.launch_id_in_name.as_deref(), + phase_in_socket_name(&entry.socket_path), + ) else { + continue; + }; + let ours = supervisor::sweep_for_in(cache_dir, project_root, phase) + .iter() + .any(|candidate| candidate.socket_path == entry.socket_path); + let Some(unclaimed) = ours + .then(|| unclaimed_by_state(project_root, phase, launch_id)) + .flatten() + else { + continue; + }; + let reason = OrphanReason::NoUsableSidecar { + broken: entry.sidecar_error.is_some(), + unclaimed: Box::new(unclaimed), + }; + findings.push(orphan_finding((phase, launch_id), liveness, true, &reason)); + } + findings +} + +/// Why [`collect_orphan_supervisors`] lists a socket. +enum OrphanReason { + /// This root's phase has no state file. + NoState, + /// This root's phase has a state file that does not load. + StateUnloadable, + /// The phase's state records no supervisor. + NoRecord, + /// The phase's state records another launch (the R-10 aftermath). + Superseded, + /// The sidecar names a project root that no longer exists. + RootDeleted, + /// The sidecar is missing (`broken: false`) or does not parse, and the + /// state does not claim the launch for `unclaimed`'s reason. + NoUsableSidecar { + broken: bool, + unclaimed: Box, + }, +} + +impl OrphanReason { + /// Path-free text for the finding's detail (T-18-01). + fn describe(&self) -> String { + match self { + OrphanReason::NoState => "its phase has no state".to_string(), + OrphanReason::StateUnloadable => "its phase's state does not load".to_string(), + OrphanReason::NoRecord => "its phase's state records no supervisor".to_string(), + OrphanReason::Superseded => { + "its phase's state records another launch (superseded)".to_string() + } + OrphanReason::RootDeleted => "its project root no longer exists".to_string(), + OrphanReason::NoUsableSidecar { broken, unclaimed } => format!( + "its sidecar {}, and {}", + if *broken { + "does not parse" + } else { + "is missing" + }, + unclaimed.describe() + ), + } + } +} + +/// Whether this root's state for `phase` claims `launch_id`; the reason +/// when it does not. Reads the state file only. +fn unclaimed_by_state( + project_root: &Path, + phase: PhaseId, + launch_id: &str, +) -> Option { + match workflow::load_state(project_root, phase) { + Err(workflow::WorkflowError::MissingState(_)) => Some(OrphanReason::NoState), + Err(_) => Some(OrphanReason::StateUnloadable), + Ok(state) => match state.supervisor { + None => Some(OrphanReason::NoRecord), + Some(record) if record.launch_id != launch_id => Some(OrphanReason::Superseded), + Some(_) => None, + }, + } +} + +/// The phase segment of a socket file name in the 50-05 scheme +/// (`<016x digest>--.sock`): the text between the +/// first and second `-`. `None` when it does not parse as a phase. +fn phase_in_socket_name(socket: &Path) -> Option { + let stem = socket.file_stem()?.to_str()?; + stem.split('-').nth(1)?.parse().ok() +} + +/// One orphan finding for `(phase, launch_id)`, path-free, with the +/// `--launch` repair that tears down exactly that socket. +fn orphan_finding( + (phase, launch_id): (PhaseId, &str), + liveness: &'static str, + root_exists: bool, + reason: &OrphanReason, +) -> OrphanSupervisorFinding { + OrphanSupervisorFinding { + finding: PhaseFinding { + phase, + severity: Severity::Problem, + detail: format!( + "phase {phase}: supervisor launch {launch_id} is {} but {}", + liveness.to_uppercase(), + reason.describe() + ), + repair: Some(format!("devflow stop --phase {phase} --launch {launch_id}")), + }, + launch_id: launch_id.to_string(), + liveness, + root_exists, + } +} + +/// Whether doctor's socket sweep could look (50-12): `None` when the socket +/// directory is absent or listable, else why it cannot be listed. +/// [`supervisor::sweep_in`] answers an unlistable directory with no entries, +/// so without this check doctor would report "nothing" when it saw nothing. +fn socket_dir_unlistable(cache_dir: &Path) -> Option { + match std::fs::read_dir(registry::socket_dir_in(cache_dir)) { + Err(error) if error.kind() != std::io::ErrorKind::NotFound => { + Some(error.kind().to_string()) + } + _ => None, + } +} + +/// Build `doctor --json`'s `"orphan_supervisors"` array (50-12). +fn render_orphan_supervisors_json(findings: &[OrphanSupervisorFinding]) -> serde_json::Value { + serde_json::Value::Array( + findings + .iter() + .map(|orphan| { + serde_json::json!({ + "phase": orphan.finding.phase, + "launch_id": orphan.launch_id, + "liveness": orphan.liveness, + "root_exists": orphan.root_exists, + "severity": orphan.finding.severity.label(), + "detail": orphan.finding.detail, + "repair": orphan.finding.repair, + }) + }) + .collect(), + ) +} + +/// Build `doctor`'s text section for the socket sweep (50-12). Like +/// [`render_stray_process_text`] it prints nothing when there is nothing to +/// report, but a socket directory it could not list is reported, never +/// shown as "nothing". +fn render_orphan_supervisors_text( + findings: &[OrphanSupervisorFinding], + unlistable: Option<&str>, +) -> String { + if findings.is_empty() && unlistable.is_none() { + return String::new(); + } + let mut out = String::from("\nsupervisor sockets no state claims:\n"); + if let Some(why) = unlistable { + out.push_str(&format!( + " warn: the socket directory could not be listed ({why}); its sockets are not \ + reported\n" + )); + } + for orphan in findings { + out.push_str(&format!(" {}\n", orphan.finding.detail)); + if let Some(repair) = &orphan.finding.repair { + out.push_str(&format!(" repair: {repair}\n")); + } + } + out +} + #[cfg(test)] mod tests { use super::*; @@ -4555,6 +5887,13 @@ mod tests { ); } + /// [`stop`] for a root with no state file, whose socket search would + /// otherwise sweep the operator's real cache: an empty tempdir cache. + fn stop_with_an_empty_cache(root: &Path, phase: PhaseId) -> Result<(), CliError> { + let cache = tempfile::tempdir().unwrap(); + stop_in(root, phase, Some(cache.path())) + } + #[test] fn stop_via_gate_with_no_waiter_at_a_non_ship_gate_writes_nothing() { let dir = tempfile::tempdir().unwrap(); @@ -5450,30 +6789,91 @@ mod tests { assert!(default_logs_phase(dir.path()).is_err()); } - /// 18b: a state with no recorded monitor is never reported as stuck, - /// regardless of the (unreliable, since no monitor was ever recorded) - /// liveness bits passed alongside it. - #[test] - fn liveness_unknown_when_no_monitor_recorded() { - assert_eq!(liveness(None, false, false), Liveness::Unknown); - assert_eq!(liveness(None, false, true), Liveness::Unknown); - assert_eq!(liveness(None, true, false), Liveness::Unknown); - assert_eq!(liveness(None, true, true), Liveness::Unknown); - } - - /// 18b: the full four-row matrix for a recorded monitor pid. A dead - /// agent with a dead monitor OR a live monitor with a dead agent are - /// different states — only the former is `Stuck` (nothing will call - /// `devflow advance`); the latter is a normal between-stages moment. An - /// agent that outlived its monitor is also `Stuck` — orphaned, since - /// nothing will advance it when it exits either. + /// A ping reply as a supervisor sends it; built through serde because + /// `PingReply` is `#[non_exhaustive]`. + fn ping_reply(owner: &str, launch: &str) -> devflow_core::supervisor::protocol::PingReply { + serde_json::from_value(serde_json::json!({ + "launch_id": "0123456789abcdef", + "phase": "7", + "stage": "code", + "launch": launch, + "owner": owner, + "agent_pid": 10, + "agent_pgid": 10, + "agent_start_time": 20, + "boot_id": null, + "supervisor_pid": 11, + })) + .expect("a complete ping reply") + } + + /// 50-12: the pure mapping over every probe outcome, with and without + /// `stopped`. Each row of one `stopped` value is a distinct answer, and + /// GONE splits by `stopped`: an abandoned active phase is not a stopped + /// one. #[test] - fn liveness_matrix_covers_all_four_rows() { - let pid = Some(4242); - assert_eq!(liveness(pid, true, true), Liveness::Healthy); - assert_eq!(liveness(pid, true, false), Liveness::BetweenStages); - assert_eq!(liveness(pid, false, false), Liveness::Stuck); - assert_eq!(liveness(pid, false, true), Liveness::Stuck); + fn liveness_matrix_covers_every_probe_outcome() { + use devflow_core::supervisor::{Liveness as Probe, ProbeError}; + let record = devflow_core::state::SupervisorRecord::new( + PathBuf::from("/cache/sock/x.sock"), + "0123456789abcdef".to_string(), + 10, + 20, + "boot".to_string(), + ); + for stopped in [false, true] { + let rows = [ + phase_liveness(None, None, stopped), + phase_liveness(Some(&record), Some(Ok(Probe::Gone)), stopped), + phase_liveness(Some(&record), Some(Ok(Probe::Stale)), stopped), + phase_liveness( + Some(&record), + Some(Ok(Probe::Alive(ping_reply("agent_running", "stream")))), + stopped, + ), + phase_liveness( + Some(&record), + Some(Err(ProbeError::NoReply("timed out".to_string()))), + stopped, + ), + ]; + assert_eq!(rows[0], PhaseLiveness::NotLaunched, "stopped={stopped}"); + let gone = if stopped { + PhaseLiveness::Gone + } else { + PhaseLiveness::GoneWhileActive + }; + assert_eq!(rows[1], gone, "stopped={stopped}"); + assert_eq!(rows[2], PhaseLiveness::Stale, "stopped={stopped}"); + assert_eq!( + rows[3], + PhaseLiveness::Alive { + owner: "agent_running".to_string(), + launch: "stream".to_string(), + }, + "stopped={stopped}" + ); + match &rows[4] { + PhaseLiveness::ProbeFailed(why) => assert!( + why.contains("no valid ping reply") && !why.contains('/'), + "a path-free probe failure, got {why:?}" + ), + other => panic!("a probe error must be ProbeFailed, got {other:?}"), + } + for (at, row) in rows.iter().enumerate() { + for other in &rows[at + 1..] { + assert_ne!( + row, other, + "stopped={stopped}: two outcomes share one answer" + ); + } + } + } + assert_ne!( + phase_liveness(Some(&record), Some(Ok(Probe::Gone)), true), + phase_liveness(Some(&record), Some(Ok(Probe::Gone)), false), + "GONE while stopped must differ from GONE while active" + ); } /// 18b: a corrupt pid (0, or above `i32::MAX`) must never read as alive @@ -5486,10 +6886,10 @@ mod tests { assert!(!agent::agent_running(u32::MAX)); } - /// 18b: persisting `monitor_pid` for one phase must not disturb a - /// concurrently-active sibling phase's `monitor_pid` (concurrency edge). + /// 18b: persisting the supervisor record for one phase must not disturb + /// a concurrently-active sibling phase's record (concurrency edge). #[test] - fn monitor_pid_persisted_for_one_phase_does_not_disturb_a_sibling() { + fn supervisor_record_persisted_for_one_phase_does_not_disturb_a_sibling() { let dir = tempfile::tempdir().unwrap(); let root = dir.path(); @@ -5499,7 +6899,14 @@ mod tests { Mode::Auto, root.to_path_buf(), ); - phase7.monitor_pid = Some(111); + let record7 = devflow_core::state::SupervisorRecord::new( + root.join("seven.sock"), + "seven00000000000".to_string(), + 111, + 1, + String::new(), + ); + phase7.supervisor = Some(record7.clone()); workflow::save_state(&phase7).unwrap(); let mut phase8 = State::new( @@ -5508,28 +6915,41 @@ mod tests { Mode::Auto, root.to_path_buf(), ); - phase8.monitor_pid = Some(222); + let record8 = devflow_core::state::SupervisorRecord::new( + root.join("eight.sock"), + "eight00000000000".to_string(), + 222, + 2, + String::new(), + ); + phase8.supervisor = Some(record8.clone()); workflow::save_state(&phase8).unwrap(); let reloaded7 = workflow::load_state(root, PhaseId::new(7)).unwrap(); let reloaded8 = workflow::load_state(root, PhaseId::new(8)).unwrap(); - assert_eq!(reloaded7.monitor_pid, Some(111)); - assert_eq!(reloaded8.monitor_pid, Some(222)); + assert_eq!(reloaded7.supervisor, Some(record7)); + assert_eq!(reloaded8.supervisor, Some(record8)); } /// 18b (idempotency edge): running `devflow status` twice must produce - /// byte-identical `.devflow/` state — the new monitor liveness probe is + /// byte-identical `.devflow/` state — the supervisor liveness probe is /// purely a read, same as the existing agent liveness probe it sits - /// beside. Also exercises the `u32::MAX` boundary pid (precision edge, - /// via `agent::agent_running`'s existing hardening) so the probe can - /// only ever report `Stuck`, never a false `Healthy`. + /// beside. The planted record names a socket that does not exist and the + /// `u32::MAX` boundary agent group (precision edge), so the probe can + /// only read it GONE, never a false ALIVE. #[test] fn status_reading_monitor_liveness_writes_no_state_and_no_event() { let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(66); let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); - state.monitor_pid = Some(u32::MAX); + state.supervisor = Some(devflow_core::state::SupervisorRecord::new( + root.join("never-bound.sock"), + "statusidem000000".to_string(), + u32::MAX, + 1, + String::new(), + )); workflow::save_state(&state).unwrap(); let state_path = workflow::state_path(root, phase); @@ -6152,7 +7572,7 @@ mod tests { stop_via_lock's identity guard treats as unconfirmable" ); - let err = stop(root, phase) + let err = stop_with_an_empty_cache(root, phase) .expect_err("a legacy lock with no recorded start time must be refused"); let message = err.to_string(); assert!( @@ -6196,7 +7616,8 @@ mod tests { std::fs::create_dir_all(lock_path.parent().unwrap()).unwrap(); std::fs::write(&lock_path, format!("{pid}\n{wrong_start}")).unwrap(); - let err = stop(root, phase).expect_err("a start-time mismatch must be refused"); + let err = stop_with_an_empty_cache(root, phase) + .expect_err("a start-time mismatch must be refused"); let message = err.to_string(); assert!( message.contains(&pid.to_string()), @@ -6246,7 +7667,7 @@ mod tests { std::fs::create_dir_all(lock_path.parent().unwrap()).unwrap(); std::fs::write(&lock_path, format!("{child_pid}\n{child_start}")).unwrap(); - let result = stop(root, phase); + let result = stop_with_an_empty_cache(root, phase); let _ = child.kill(); let _ = child.wait(); @@ -6278,24 +7699,6 @@ mod tests { stop(root, phase).expect("stop against a stale lock must succeed, not error"); } - /// T-23-51: a live `monitor_pid` recorded in `State` — with no open gate - /// and no lock file — must never be treated as a signalling target. - /// `stop` only ever looks at the lock file. - #[test] - fn stop_never_treats_monitor_pid_as_a_signalling_target() { - let dir = tempfile::tempdir().unwrap(); - let root = dir.path(); - let phase = PhaseId::new(203); - let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); - state.monitor_pid = Some(std::process::id()); - workflow::save_state(&state).unwrap(); - - stop(root, phase).expect( - "stop must succeed — a recorded monitor_pid alone must never be signalled or \ - treated as a blocker", - ); - } - /// Task 3 behavior: no open gate, no lock file, and no persisted state /// at all — a phase that was never started — is a successful no-op. #[test] @@ -6303,7 +7706,8 @@ mod tests { let dir = tempfile::tempdir().unwrap(); let root = dir.path(); - stop(root, PhaseId::new(201)).expect("stop against nothing present must succeed"); + stop_with_an_empty_cache(root, PhaseId::new(201)) + .expect("stop against nothing present must succeed"); } /// 14-CR-03: a capture file SHORTER than the follower's offset means the @@ -6832,11 +8236,13 @@ mod tests { assert!(row.contains('?'), "row must render the unknown age: {row}"); } - /// 21a: `recovery_hints` returns a `resume` hint for a stuck phase, - /// additionally an `advance` hint when the phase is gate-pending - /// (answer the gate, then advance), and nothing for a non-stuck phase. + /// 21a / 50-12: `recovery_hints` returns a `resume` hint for a phase + /// whose supervisor is GONE while it is not stopped, additionally an + /// `advance` hint when the phase is gate-pending (answer the gate, then + /// advance), `stop` then `resume` for a STALE socket, and nothing for an + /// ALIVE, stopped or unlaunched phase. #[test] - fn recovery_hints_includes_resume_for_stuck() { + fn recovery_hints_includes_resume_for_gone_while_active() { let dir = tempfile::tempdir().unwrap(); let state = State::new( PhaseId::new(7), @@ -6845,13 +8251,13 @@ mod tests { dir.path().to_path_buf(), ); - let hints = recovery_hints(&state, Liveness::Stuck); + let hints = recovery_hints(&state, &PhaseLiveness::GoneWhileActive); assert_eq!(hints, vec!["devflow resume --phase 7".to_string()]); } #[test] - fn recovery_hints_includes_advance_when_stuck_and_gate_pending() { + fn recovery_hints_includes_advance_when_gone_while_active_and_gate_pending() { let dir = tempfile::tempdir().unwrap(); let mut state = State::new( PhaseId::new(7), @@ -6861,7 +8267,7 @@ mod tests { ); state.gate_pending = true; - let hints = recovery_hints(&state, Liveness::Stuck); + let hints = recovery_hints(&state, &PhaseLiveness::GoneWhileActive); assert_eq!( hints, @@ -6873,7 +8279,26 @@ mod tests { } #[test] - fn recovery_hints_empty_for_healthy() { + fn recovery_hints_names_stop_then_resume_for_stale() { + let dir = tempfile::tempdir().unwrap(); + let state = State::new( + PhaseId::new(7), + AgentKind::Claude, + Mode::Auto, + dir.path().to_path_buf(), + ); + + assert_eq!( + recovery_hints(&state, &PhaseLiveness::Stale), + vec![ + "devflow stop --phase 7".to_string(), + "devflow resume --phase 7".to_string(), + ] + ); + } + + #[test] + fn recovery_hints_empty_for_alive_stopped_and_unlaunched() { let dir = tempfile::tempdir().unwrap(); let state = State::new( PhaseId::new(7), @@ -6881,8 +8306,17 @@ mod tests { Mode::Auto, dir.path().to_path_buf(), ); + let alive = PhaseLiveness::Alive { + owner: "agent_running".to_string(), + launch: "stream".to_string(), + }; - assert!(recovery_hints(&state, Liveness::Healthy).is_empty()); + for liveness in [alive, PhaseLiveness::Gone, PhaseLiveness::NotLaunched] { + assert!( + recovery_hints(&state, &liveness).is_empty(), + "{liveness:?} needs no repair" + ); + } } /// 21a / 999.30 IN-01: the shared one-pass event summary's @@ -6930,7 +8364,7 @@ mod tests { root, PhaseId::new(7), "stage_launched", - serde_json::json!({"stage": "code", "agent": "claude", "monitor_pid": 1}), + serde_json::json!({"stage": "code", "agent": "claude", "supervisor_pid": 1}), ); // events::emit always stamps `ts` with the current time; rewrite it // to a fixed, known-past value so the assertion is deterministic @@ -6970,6 +8404,72 @@ mod tests { ); } + /// Run `body` on a new thread without `CAP_DAC_OVERRIDE` and + /// `CAP_DAC_READ_SEARCH` in its effective set (WINDOWS.md #19). Either + /// lets uid 0 list a mode-0o000 directory, so the pinned container (root) + /// could not produce an unlistable socket directory; without them root is + /// refused like any other owner. Capabilities are per-thread and + /// `capset(2)` changes only the calling thread's, so the harness keeps + /// its own, and they end with the thread. A non-root thread holds neither + /// capability, so nothing changes there. + fn without_dac_override(body: impl FnOnce() -> T + Send) -> T { + std::thread::scope(|scope| { + scope + .spawn(|| { + drop_dac_override_on_this_thread(); + body() + }) + .join() + .unwrap_or_else(|panic| std::panic::resume_unwind(panic)) + }) + } + + /// Clear `CAP_DAC_OVERRIDE` and `CAP_DAC_READ_SEARCH` from the calling + /// thread's effective capability set; the permitted set is kept. + #[cfg(target_os = "linux")] + fn drop_dac_override_on_this_thread() { + // ``, version 3: one header, two data structs. + #[repr(C)] + struct CapHeader { + version: u32, + pid: libc::c_int, + } + #[repr(C)] + #[derive(Clone, Copy)] + struct CapData { + effective: u32, + permitted: u32, + inheritable: u32, + } + const LINUX_CAPABILITY_VERSION_3: u32 = 0x2008_0522; + const CAP_DAC_OVERRIDE: u32 = 1; + const CAP_DAC_READ_SEARCH: u32 = 2; + + let mut header = CapHeader { + version: LINUX_CAPABILITY_VERSION_3, + pid: 0, + }; + let mut data = [CapData { + effective: 0, + permitted: 0, + inheritable: 0, + }; 2]; + // SAFETY: `header` and `data` have capget(2)'s version-3 layout (one + // header, two data structs); pid 0 is the calling thread. + let got = unsafe { libc::syscall(libc::SYS_capget, &raw mut header, data.as_mut_ptr()) }; + assert_eq!(got, 0, "capget: {}", std::io::Error::last_os_error()); + data[0].effective &= !((1 << CAP_DAC_OVERRIDE) | (1 << CAP_DAC_READ_SEARCH)); + // SAFETY: as above; capset(2) with pid 0 changes only this thread. + let set = unsafe { libc::syscall(libc::SYS_capset, &raw const header, data.as_ptr()) }; + assert_eq!(set, 0, "capset: {}", std::io::Error::last_os_error()); + } + + /// No per-thread capabilities to drop off Linux: there a root run still + /// lists a 0o000 directory, and the tests that use this fail their own + /// precondition rather than pass without asserting. + #[cfg(not(target_os = "linux"))] + fn drop_dac_override_on_this_thread() {} + /// Unit tests for the pure `doctor` reconciliation core (18a). Each test /// builds a `PhaseFacts` directly — no repository, no I/O — proving /// `reconcile_phase` is a predicate over facts alone. @@ -6977,19 +8477,590 @@ mod tests { mod doctor_reconciliation { use super::*; - /// A fully-agreeing baseline: `reconcile_phase` over this returns - /// zero findings. Each test overrides only the field(s) needed to - /// trigger the one check it's proving. - fn agreeing_facts(phase: PhaseId) -> PhaseFacts { - PhaseFacts { - phase, - stage: Stage::Code, - gate_pending: false, - waiter: lock::HolderStatus::NoHolder, + /// The launch id every fixture's supervisor record carries. + const SUPERVISOR_LAUNCH_ID: &str = "0123456789abcdef"; + + /// Every string anywhere inside `value`. + fn strings_in(value: &serde_json::Value, out: &mut Vec) { + match value { + serde_json::Value::String(text) => out.push(text.clone()), + serde_json::Value::Array(items) => { + items.iter().for_each(|item| strings_in(item, out)); + } + serde_json::Value::Object(map) => { + map.values().for_each(|item| strings_in(item, out)); + } + _ => {} + } + } + + /// 50-12: a STALE supervisor, one GONE while its phase is not + /// stopped, and a failed probe are each a `Problem` naming the + /// launch, never its socket path; STALE's repair stops then resumes, + /// GONE's resumes, and a failed probe points at `devflow status` + /// (which prints the operator's own socket path). + #[test] + fn check_supervisor_reports_stale_and_gone_while_active() { + let stale = check_supervisor(&PhaseFacts { + supervisor: PhaseLiveness::Stale, + ..agreeing_facts(PhaseId::new(21)) + }) + .expect("a STALE supervisor is a finding"); + assert_eq!(stale.severity, Severity::Problem); + assert_eq!( + stale.repair.as_deref(), + Some("devflow stop --phase 21 && devflow resume --phase 21") + ); + assert!( + stale.detail.contains("STALE") && stale.detail.contains(SUPERVISOR_LAUNCH_ID), + "{}", + stale.detail + ); + + let gone = check_supervisor(&PhaseFacts { + supervisor: PhaseLiveness::GoneWhileActive, + ..agreeing_facts(PhaseId::new(22)) + }) + .expect("GONE while active is a finding"); + assert_eq!(gone.severity, Severity::Problem); + assert_eq!(gone.repair.as_deref(), Some("devflow resume --phase 22")); + assert!(gone.detail.contains("GONE"), "{}", gone.detail); + + let record = devflow_core::state::SupervisorRecord::new( + PathBuf::from("/home/someone/.cache/devflow/sock/x.sock"), + SUPERVISOR_LAUNCH_ID.to_string(), + 10, + 20, + "boot".to_string(), + ); + let failed = phase_liveness( + Some(&record), + Some(Err(devflow_core::supervisor::ProbeError::NotASocket( + record.socket_path.clone(), + ))), + false, + ); + let failed = check_supervisor(&PhaseFacts { + supervisor: failed, + ..agreeing_facts(PhaseId::new(23)) + }) + .expect("a failed probe is a finding"); + assert_eq!(failed.severity, Severity::Problem); + assert!( + failed.detail.contains("not a socket") && !failed.detail.contains('/'), + "a failed probe names its kind and no path: {}", + failed.detail + ); + assert_eq!(failed.repair.as_deref(), Some("devflow status")); + + let reconciled = reconcile_phase(&PhaseFacts { + supervisor: PhaseLiveness::Stale, + ..agreeing_facts(PhaseId::new(21)) + }); + assert!( + reconciled.iter().any(|f| f.detail == stale.detail), + "reconcile_phase runs check_supervisor" + ); + } + + /// 50-12 control: ALIVE, GONE while stopped, and NotLaunched are not + /// findings, so the checks above can only fire for the three states + /// that need a repair. + #[test] + fn check_supervisor_is_silent_when_alive_or_stopped() { + let alive = agreeing_facts(PhaseId::new(24)); + assert!(check_supervisor(&alive).is_none()); + assert!(reconcile_phase(&alive).is_empty()); + let stopped = PhaseFacts { + supervisor: PhaseLiveness::Gone, + stopped: true, + ..agreeing_facts(PhaseId::new(25)) + }; + assert!(check_supervisor(&stopped).is_none()); + let unlaunched = PhaseFacts { + supervisor: PhaseLiveness::NotLaunched, + supervisor_launch_id: None, + ..agreeing_facts(PhaseId::new(26)) + }; + assert!(check_supervisor(&unlaunched).is_none()); + } + + /// 50-12 (T-50-32): `doctor --json` phase facts carry the + /// supervisor's liveness, owner, launch and launch id, and no string + /// in the reconciliation section holds a path. + #[test] + fn doctor_json_reports_supervisor_liveness_without_paths() { + let record = devflow_core::state::SupervisorRecord::new( + PathBuf::from("/home/someone/.cache/devflow/sock/x.sock"), + SUPERVISOR_LAUNCH_ID.to_string(), + 10, + 20, + "boot".to_string(), + ); + let failed = phase_liveness( + Some(&record), + Some(Err(devflow_core::supervisor::ProbeError::NotASocket( + record.socket_path.clone(), + ))), + false, + ); + let facts = vec![ + agreeing_facts(PhaseId::new(27)), + PhaseFacts { + supervisor: failed, + ..agreeing_facts(PhaseId::new(28)) + }, + ]; + let body = render_reconciliation_json(&facts); + let entries = body.as_array().expect("an array"); + let alive = &entries[0]["supervisor"]; + assert_eq!(alive["liveness"], "alive", "{body}"); + assert_eq!(alive["owner"], "agent_running", "{body}"); + assert_eq!(alive["launch"], "stream", "{body}"); + assert_eq!(alive["launch_id"], SUPERVISOR_LAUNCH_ID, "{body}"); + let failed = &entries + .iter() + .find(|entry| entry["phase"] == 28) + .expect("phase 28's finding")["supervisor"]; + assert_eq!(failed["liveness"], "probe_failed", "{body}"); + assert!(failed["owner"].is_null(), "{body}"); + let mut strings = Vec::new(); + strings_in(&body, &mut strings); + assert!(!strings.is_empty(), "the scan must see strings: {body}"); + assert!( + strings.iter().all(|text| !text.contains('/')), + "no string in the reconciliation section may hold a path: {strings:?}" + ); + } + + /// A short cache dir directly under `/tmp`, so every socket path + /// below it fits `sun_path`. + fn short_cache() -> tempfile::TempDir { + tempfile::Builder::new() + .prefix("dfs") + .tempdir_in("/tmp") + .expect("create a short cache dir under /tmp") + } + + /// Publish the socket for `(root, phase, launch_id)` under `cache`. + /// Dropping the returned listener leaves the socket STALE. + fn publish_launch( + cache: &Path, + root: &Path, + phase: u32, + launch_id: &str, + ) -> (supervisor::SocketPath, std::os::unix::net::UnixListener) { + let socket = + supervisor::SocketPath::for_launch(cache, root, PhaseId::new(phase), launch_id) + .expect("socket path fits sun_path"); + let listener = supervisor::publish(&socket).expect("publish the socket"); + (socket, listener) + } + + /// A complete sidecar for `(root, phase, launch_id)`, as a supervisor + /// writes it. + fn write_sidecar_for(socket: &supervisor::SocketPath, root: &Path, phase: u32, id: &str) { + let sidecar = serde_json::json!({ + "launch_id": id, + "project_root": root, + "phase": PhaseId::new(phase), + "stage": "code", + "agent": "claude", + "launch": "stream", + "agent_pid": 1, + "agent_pgid": 1, + "agent_start_time": 0, + "boot_id": "boot", + "supervisor_pid": 1, + "supervisor_start_time": 0, + "created_at": "2026-10-17T00:00:00Z", + }); + std::fs::write(socket.sidecar_path(), sidecar.to_string()).unwrap(); + supervisor::read_sidecar(&socket.sidecar_path()).expect("the planted sidecar parses"); + } + + /// A loadable state for `phase` whose supervisor record carries + /// `launch_id`. + fn save_state_with_launch(root: &Path, phase: u32, launch_id: &str) { + let mut state = State::new( + PhaseId::new(phase), + AgentKind::Claude, + Mode::Auto, + root.to_path_buf(), + ); + state.supervisor = Some(devflow_core::state::SupervisorRecord::new( + PathBuf::from("/elsewhere.sock"), + launch_id.to_string(), + 1, + 0, + "boot".to_string(), + )); + workflow::save_state(&state).unwrap(); + } + + /// Answers every ping on `listener` as a supervisor would, until + /// [`PingResponder::stop`]. + struct PingResponder { + done: std::sync::Arc, + socket: PathBuf, + thread: Option>, + } + + impl PingResponder { + fn start( + socket: &supervisor::SocketPath, + listener: std::os::unix::net::UnixListener, + launch_id: &str, + phase: u32, + ) -> Self { + use std::io::{BufRead as _, Write as _}; + let done = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let reply = serde_json::json!({ + "ok": true, + "launch_id": launch_id, + "phase": phase.to_string(), + "stage": "code", + "launch": "stream", + "owner": "agent_running", + "agent_pid": 1, + "agent_pgid": 1, + "agent_start_time": 0, + "boot_id": null, + "supervisor_pid": 1, + }); + let stop = std::sync::Arc::clone(&done); + let thread = std::thread::spawn(move || { + for stream in listener.incoming() { + if stop.load(std::sync::atomic::Ordering::SeqCst) { + break; + } + let Ok(stream) = stream else { continue }; + let mut line = String::new(); + if std::io::BufReader::new(&stream) + .read_line(&mut line) + .is_ok() + { + let _ = (&stream).write_all(format!("{reply}\n").as_bytes()); + } + } + }); + Self { + done, + socket: socket.as_path().to_path_buf(), + thread: Some(thread), + } + } + + fn stop(&mut self) { + self.done.store(true, std::sync::atomic::Ordering::SeqCst); + // Wake the blocked accept so the thread sees the flag. + let _ = std::os::unix::net::UnixStream::connect(&self.socket); + if let Some(thread) = self.thread.take() { + thread.join().expect("the ping responder thread"); + } + } + } + + impl Drop for PingResponder { + fn drop(&mut self) { + self.stop(); + } + } + + /// Every file under `dir` as (name, length, mtime), sorted. + fn snapshot(dir: &Path) -> Vec<(String, u64, std::time::SystemTime)> { + let mut files: Vec<_> = std::fs::read_dir(dir) + .expect("list the socket dir") + .map(|entry| { + let entry = entry.unwrap(); + let meta = std::fs::symlink_metadata(entry.path()).unwrap(); + ( + entry.file_name().to_string_lossy().into_owned(), + meta.len(), + meta.modified().unwrap(), + ) + }) + .collect(); + files.sort(); + files + } + + /// 50-12 (D-06, R5-I3, T-50-31, T-50-51, T-50-84): doctor's sweep + /// lists the ALIVE and STALE sockets that no loadable state of this + /// root claims — no state, an unparsable state, a superseded launch, + /// a missing or broken sidecar attributed by file name — and, + /// machine-wide, those whose project root is gone. Each names the + /// `--launch` teardown, never the bare stop. Another root's sockets + /// and the ones a state claims are not listed, and nothing under + /// `sock/` changes. + #[test] + fn doctor_reports_an_orphan_socket_without_removing_it() { + let cache = short_cache(); + let cache = cache.path(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let other = tempfile::tempdir().unwrap(); + let other_root = other.path(); + let deleted = tempfile::tempdir().unwrap(); + let deleted_root = deleted.path().to_path_buf(); + + // (a) STALE, sidecar for phase 7, whose state file is corrupt. + let (a, _) = publish_launch(cache, root, 7, "aaaa0000aaaa0007"); + write_sidecar_for(&a, root, 7, "aaaa0000aaaa0007"); + let state7 = workflow::state_path(root, PhaseId::new(7)); + std::fs::create_dir_all(state7.parent().unwrap()).unwrap(); + std::fs::write(&state7, "{").unwrap(); + // (b) STALE, superseded: phase 8's state names its successor. + let (b, _) = publish_launch(cache, root, 8, "aaaa0000aaaa0008"); + write_sidecar_for(&b, root, 8, "aaaa0000aaaa0008"); + save_state_with_launch(root, 8, "ffff0000ffff0008"); + // (c) STALE, another root that exists: not ours. + let (c, _) = publish_launch(cache, other_root, 7, "aaaa0000aaaa000c"); + write_sidecar_for(&c, other_root, 7, "aaaa0000aaaa000c"); + // (d) STALE, a project root deleted after the launch. + let (d, _) = publish_launch(cache, &deleted_root, 5, "aaaa0000aaaa000d"); + write_sidecar_for(&d, &deleted_root, 5, "aaaa0000aaaa000d"); + drop(deleted); + assert!(!deleted_root.exists(), "the root of (d) is deleted"); + // (e) ALIVE, this root's phase 9, no state, sidecar `{`. + let (e, listener) = publish_launch(cache, root, 9, "bbbb0000bbbb0000"); + std::fs::write(e.sidecar_path(), "{").unwrap(); + let mut responder = PingResponder::start(&e, listener, "bbbb0000bbbb0000", 9); + // (f) STALE, this root's phase 10, no sidecar, no state. + let (_f, _) = publish_launch(cache, root, 10, "cccc0000cccc0000"); + // (g) control: STALE, the other root's phase 9, no sidecar. + let (_g, _) = publish_launch(cache, other_root, 9, "dddd0000dddd0000"); + // (h) control: STALE, no sidecar, whose state claims it. + let (_h, _) = publish_launch(cache, root, 11, "eeee0000eeee0000"); + save_state_with_launch(root, 11, "eeee0000eeee0000"); + // (i) control: STALE with a sidecar, whose state claims it. + let (i, _) = publish_launch(cache, root, 12, "aaaa0000aaaa0012"); + write_sidecar_for(&i, root, 12, "aaaa0000aaaa0012"); + save_state_with_launch(root, 12, "aaaa0000aaaa0012"); + + let sock_dir = registry::socket_dir_in(cache); + let before = snapshot(&sock_dir); + let findings = collect_orphan_supervisors(root, cache); + assert_eq!( + snapshot(&sock_dir), + before, + "the sweep must change nothing under sock/" + ); + responder.stop(); + + let mut listed: Vec<(PhaseId, &str, &str, bool)> = findings + .iter() + .map(|orphan| { + ( + orphan.finding.phase, + orphan.launch_id.as_str(), + orphan.liveness, + orphan.root_exists, + ) + }) + .collect(); + listed.sort(); + assert_eq!( + listed, + vec![ + (PhaseId::new(5), "aaaa0000aaaa000d", "stale", false), + (PhaseId::new(7), "aaaa0000aaaa0007", "stale", true), + (PhaseId::new(8), "aaaa0000aaaa0008", "stale", true), + (PhaseId::new(9), "bbbb0000bbbb0000", "alive", true), + (PhaseId::new(10), "cccc0000cccc0000", "stale", true), + ], + "exactly (a), (b), (d), (e) and (f) are listed" + ); + for orphan in &findings { + let finding = &orphan.finding; + let repair = finding + .repair + .as_deref() + .expect("every orphan names a repair"); + assert_eq!( + repair, + format!( + "devflow stop --phase {} --launch {}", + finding.phase, orphan.launch_id + ), + "the repair addresses exactly this socket" + ); + assert!( + repair.contains("--launch") + && repair != format!("devflow stop --phase {}", finding.phase), + "never the bare stop, which would stop the state's own record: {repair}" + ); + assert!( + !finding.detail.contains('/'), + "a path-free detail: {}", + finding.detail + ); + assert_eq!(finding.severity, Severity::Problem); + } + let detail_of = |id: &str| { + findings + .iter() + .find(|orphan| orphan.launch_id == id) + .map(|orphan| orphan.finding.detail.clone()) + .unwrap() + }; + assert!(detail_of("aaaa0000aaaa000d").contains("no longer exists")); + assert!(detail_of("bbbb0000bbbb0000").contains("does not parse")); + assert!(detail_of("cccc0000cccc0000").contains("missing")); + assert!(detail_of("aaaa0000aaaa0008").contains("another launch")); + + // Control: removing (a)'s files changes the snapshot, so the + // comparison above can fail. + supervisor::remove_launch_files(&a).unwrap(); + assert_ne!( + snapshot(&sock_dir), + before, + "the snapshot must see a removal" + ); + } + + /// 50-12: `supervisor::sweep_in` answers a socket directory it + /// cannot list with no entries, so doctor checks that case itself and + /// reports it rather than printing nothing. Controls: an absent and a + /// listable directory report nothing. + #[test] + fn doctor_reports_an_unlistable_socket_dir_instead_of_nothing() { + use std::os::unix::fs::PermissionsExt as _; + let cache = tempfile::tempdir().unwrap(); + assert_eq!(socket_dir_unlistable(cache.path()), None, "absent"); + assert_eq!(render_orphan_supervisors_text(&[], None), ""); + let sock = registry::socket_dir_in(cache.path()); + std::fs::create_dir_all(&sock).unwrap(); + assert_eq!(socket_dir_unlistable(cache.path()), None, "listable"); + + std::fs::set_permissions(&sock, std::fs::Permissions::from_mode(0o000)).unwrap(); + // Without DAC override, so uid 0 (the pinned container) is + // refused too (WINDOWS.md #19). + let unlistable = without_dac_override(|| socket_dir_unlistable(cache.path())); + std::fs::set_permissions(&sock, std::fs::Permissions::from_mode(0o700)).unwrap(); + let why = unlistable.expect("a directory without read permission is unlistable"); + assert_eq!( + why, + std::io::ErrorKind::PermissionDenied.to_string(), + "the 0o000 directory itself refuses the listing" + ); + assert!(!why.contains('/'), "path-free: {why}"); + let text = render_orphan_supervisors_text(&[], Some(&why)); + assert!( + text.contains("could not be listed") && text.contains(&why), + "{text}" + ); + let body = doctor_json_body(&[], &[], &[], &[], (&[], Some(&why))); + assert_eq!(body["orphan_supervisor_sweep_error"], why.as_str()); + } + + /// The comment-stripped body of `fn collect_orphan_supervisors(` in + /// `source` (cut at its first `mod tests {` line), from that line to + /// the next line that is exactly `}`; `None` when absent. + fn orphan_sweep_body(source: &str) -> Option> { + let lines: Vec<&str> = source + .lines() + .take_while(|line| !line.contains("mod tests {")) + .collect(); + let start = lines + .iter() + .position(|line| line.contains("fn collect_orphan_supervisors("))?; + let end = start + lines[start..].iter().position(|line| *line == "}")?; + Some( + lines[start..=end] + .iter() + .copied() + .filter(|line| !line.trim_start().starts_with("//")) + .collect(), + ) + } + + /// Why `source`'s `collect_orphan_supervisors` breaks R6-G4: absent + /// or under 5 lines, an acting call, or no `sweep_for_in(` call. + fn orphan_sweep_violations(source: &str) -> Vec { + const ACTING: [&str; 5] = [ + "remove_launch_files", + "remove_file", + "teardown_stale", + "request_shutdown", + "kill(", + ]; + let Some(body) = orphan_sweep_body(source) else { + return vec!["fn collect_orphan_supervisors( not found".to_string()]; + }; + let mut violations = Vec::new(); + if body.len() < 5 { + violations.push(format!("body has {} lines, under 5", body.len())); + } + for line in &body { + for call in ACTING { + if line.contains(call) { + violations.push(format!("acting call {call}: {line}")); + } + } + } + if !body.iter().any(|line| line.contains("sweep_for_in(")) { + violations.push("no sweep_for_in( call".to_string()); + } + violations + } + + /// 50-12 R6-G4 (T-50-31, T-50-84): doctor's orphan sweep only reads, + /// and attributes a sidecar-less socket through `sweep_for_in`, so + /// 50-06's digest rule stays in one place. Planted controls run the + /// same check. + #[test] + fn the_orphan_sweep_only_reads_and_attributes_through_sweep_for_in() { + let reads = "fn collect_orphan_supervisors(a: &Path, b: &Path) -> Vec {\n let mut out = Vec::new();\n for entry in supervisor::sweep_for_in(b, a, p) {\n out.push(entry);\n }\n out\n}\n"; + assert!( + orphan_sweep_violations(reads).is_empty(), + "a read-only fn passes" + ); + let acts = "fn collect_orphan_supervisors(a: &Path, b: &Path) -> Vec {\n let mut out = Vec::new();\n for entry in supervisor::sweep_for_in(b, a, p) {\n supervisor::remove_launch_files(&entry);\n out.push(entry);\n }\n out\n}\n"; + let found = orphan_sweep_violations(acts); + assert!( + found.len() == 1 && found[0].contains("remove_launch_files"), + "an acting call fails: {found:?}" + ); + let reimplements = "fn collect_orphan_supervisors(a: &Path, b: &Path) -> Vec {\n let mut out = Vec::new();\n // sweep_for_in(b, a, p) re-derived below\n for entry in supervisor::sweep_in(b) {\n out.push(entry);\n }\n out\n}\n"; + let found = orphan_sweep_violations(reimplements); + assert!( + found.len() == 1 && found[0].contains("no sweep_for_in"), + "a commented-out sweep_for_in fails: {found:?}" + ); + let absent = "fn other() {\n}\n"; + assert_eq!( + orphan_sweep_violations(absent), + vec!["fn collect_orphan_supervisors( not found".to_string()] + ); + + let source = std::fs::read_to_string( + Path::new(env!("CARGO_MANIFEST_DIR")).join("src/commands.rs"), + ) + .expect("read src/commands.rs"); + assert_eq!( + orphan_sweep_violations(&source), + Vec::::new(), + "collect_orphan_supervisors must only read and must call sweep_for_in" + ); + } + + /// A fully-agreeing baseline: `reconcile_phase` over this returns + /// zero findings. Each test overrides only the field(s) needed to + /// trigger the one check it's proving. + fn agreeing_facts(phase: PhaseId) -> PhaseFacts { + PhaseFacts { + phase, + stage: Stage::Code, + gate_pending: false, + waiter: lock::HolderStatus::NoHolder, agent_pid: Some(4242), agent_alive: true, - monitor_pid: Some(4343), - monitor_alive: true, + supervisor: PhaseLiveness::Alive { + owner: "agent_running".to_string(), + launch: "stream".to_string(), + }, + supervisor_launch_id: Some(SUPERVISOR_LAUNCH_ID.to_string()), last_event: Some("stage_launched".into()), last_launched_stage: Some(Stage::Code), open_gate_stages: Vec::new(), @@ -7165,27 +9236,27 @@ mod tests { assert!(findings[0].repair.is_none()); } - /// 18b: a dead monitor with a dead agent is `Stuck` — nothing will - /// call `devflow advance` for this phase — and reports a `Problem` - /// finding with a `devflow resume --phase N` repair. + /// 18b / 50-12: a supervisor GONE while its phase is not stopped, + /// with a dead agent — nothing will call `devflow advance` for this + /// phase — reports a `Problem` finding with a `devflow resume --phase + /// N` repair. #[test] - fn reconcile_reports_stuck_when_monitor_and_agent_are_both_dead() { + fn reconcile_reports_gone_supervisor_and_dead_agent() { let facts = PhaseFacts { - monitor_pid: Some(5150), - monitor_alive: false, + supervisor: PhaseLiveness::GoneWhileActive, agent_pid: Some(4242), agent_alive: false, ..agreeing_facts(PhaseId::new(8)) }; let findings = reconcile_phase(&facts); - let monitor_finding = findings + let supervisor_finding = findings .iter() - .find(|f| f.detail.contains("monitor pid")) - .expect("expected a monitor finding when monitor and agent are both dead"); - assert_eq!(monitor_finding.severity, Severity::Problem); - assert!(monitor_finding.detail.contains("monitor pid 5150")); + .find(|f| f.detail.contains("supervisor launch")) + .expect("expected a supervisor finding when the supervisor is gone"); + assert_eq!(supervisor_finding.severity, Severity::Problem); + assert!(supervisor_finding.detail.contains(SUPERVISOR_LAUNCH_ID)); assert_eq!( - monitor_finding.repair.as_deref(), + supervisor_finding.repair.as_deref(), Some("devflow resume --phase 8") ); } @@ -7214,17 +9285,14 @@ mod tests { } /// 20c (D-09 + review: Codex HIGH — the doctor gap is bigger than - /// `check_dead_agent`): the same stopped phase may also carry a - /// stale `monitor_pid` (though the stop path clears it — this - /// proves the guard holds even if that clear were ever bypassed). - /// `check_dead_monitor` must also recognize `facts.stopped` and - /// report ZERO findings. + /// `check_dead_agent`): the same stopped phase's supervisor has + /// exited. `check_supervisor` must also recognize `facts.stopped` + /// and report ZERO findings. #[test] - fn reconcile_phase_ignores_dead_monitor_when_stopped() { + fn reconcile_phase_ignores_gone_supervisor_when_stopped() { let facts = PhaseFacts { stage: Stage::Plan, - monitor_pid: Some(5150), - monitor_alive: false, + supervisor: PhaseLiveness::Gone, agent_pid: Some(4242), agent_alive: false, stopped: true, @@ -7233,45 +9301,48 @@ mod tests { let findings = reconcile_phase(&facts); assert!( findings.iter().all(|f| f.severity != Severity::Problem), - "a --until-stopped phase must yield zero Problem findings even with a \ - stale monitor_pid, got: {:?}", + "a --until-stopped phase must yield zero Problem findings even with an \ + exited supervisor, got: {:?}", findings.iter().map(|f| &f.detail).collect::>() ); } - /// 18b (T-18-11): an unrecorded monitor is unknown, not a problem — - /// a state file written by a pre-18b binary must never render as - /// stuck. + /// 18b (T-18-11): an unrecorded supervisor is not launched, not a + /// problem — a state file written before the record existed must + /// never render as a finding. #[test] - fn reconcile_is_silent_when_monitor_pid_is_unrecorded() { + fn reconcile_is_silent_when_no_supervisor_is_recorded() { let facts = PhaseFacts { - monitor_pid: None, - monitor_alive: false, + supervisor: PhaseLiveness::NotLaunched, + supervisor_launch_id: None, ..agreeing_facts(PhaseId::new(9)) }; assert!( reconcile_phase(&facts).is_empty(), - "an unrecorded monitor must never produce a finding" + "an unrecorded supervisor must never produce a finding" ); } - /// 18b: a live monitor with a dead agent is a normal between-stages - /// moment (the monitor hasn't advanced the phase yet), not a monitor - /// finding. `check_dead_agent`'s own pre-existing finding for the - /// dead agent pid is unrelated to this check and out of this plan's - /// scope. + /// 18b / 50-12: an ALIVE supervisor with a dead agent is a normal + /// between-stages moment (the supervisor runs the advance itself), + /// not a supervisor finding. `check_dead_agent`'s own pre-existing + /// finding for the dead agent pid is unrelated to this check. #[test] - fn reconcile_is_silent_when_monitor_alive_and_agent_dead() { + fn reconcile_is_silent_when_supervisor_alive_and_agent_dead() { let facts = PhaseFacts { - monitor_pid: Some(5150), - monitor_alive: true, + supervisor: PhaseLiveness::Alive { + owner: "advancing".to_string(), + launch: "stream".to_string(), + }, agent_alive: false, ..agreeing_facts(PhaseId::new(10)) }; let findings = reconcile_phase(&facts); assert!( - findings.iter().all(|f| !f.detail.contains("monitor pid")), - "a live monitor with a dead agent must not produce a monitor finding" + findings + .iter() + .all(|f| !f.detail.contains("supervisor launch")), + "a live supervisor with a dead agent must not produce a supervisor finding" ); } @@ -7284,8 +9355,7 @@ mod tests { gate_pending: true, agent_pid: Some(999_999), agent_alive: false, - monitor_pid: Some(999_998), - monitor_alive: false, + supervisor: PhaseLiveness::GoneWhileActive, last_launched_stage: Some(Stage::Validate), open_gate_stages: Vec::new(), feature_branch_exists: false, @@ -7298,14 +9368,15 @@ mod tests { vec![ Severity::Problem, // check_gate_pending_without_gate Severity::Problem, // check_dead_agent - Severity::Problem, // check_dead_monitor + Severity::Problem, // check_supervisor Severity::Warn, // check_stage_event_drift Severity::Warn, // check_missing_branch ] ); assert!(findings[0].detail.contains("gate_pending is true")); assert!(findings[1].detail.contains("agent pid 999999")); - assert!(findings[2].detail.contains("monitor pid 999998")); + assert!(findings[2].detail.contains("supervisor launch")); + assert!(findings[2].detail.contains("GONE")); assert!( findings[3] .detail @@ -7409,7 +9480,7 @@ mod tests { workflow::save_state(&state).unwrap(); let facts = collect_phase_facts(root); - let body = doctor_json_body(&checks, &facts, &[], &[]); + let body = doctor_json_body(&checks, &facts, &[], &[], (&[], None)); let serialized = serde_json::to_string(&body).unwrap(); let reparsed: serde_json::Value = serde_json::from_str(&serialized) .expect("doctor --json must be single-document JSON, not two concatenated arrays"); @@ -7432,10 +9503,14 @@ mod tests { "999.44: must carry the stray-process findings under a FOURTH key, \ never a second concatenated array: {reparsed}" ); + assert!( + reparsed.get("orphan_supervisors").is_some(), + "50-12: must carry the socket sweep's findings under their own key: {reparsed}" + ); assert_eq!( reparsed.as_object().unwrap().len(), - 4, - "doctor --json must have exactly four top-level keys: {reparsed}" + 6, + "doctor --json must have exactly six top-level keys: {reparsed}" ); assert!(reparsed["environment"].is_array()); assert!(reparsed["reconciliation"].is_array()); @@ -7483,8 +9558,18 @@ mod tests { .lines() .count(); - doctor(root, false).unwrap(); - doctor(root, false).unwrap(); + // 50-12: a hermetic cache whose STALE socket no state claims, so + // the run also exercises the socket sweep, which must leave it. + let cache = short_cache(); + let (orphan, _) = publish_launch(cache.path(), root, 91, "aaaa0000aaaa0091"); + write_sidecar_for(&orphan, root, 91, "aaaa0000aaaa0091"); + + doctor_in(root, false, Some(cache.path())).unwrap(); + doctor_in(root, false, Some(cache.path())).unwrap(); + assert!( + orphan.as_path().exists() && orphan.sidecar_path().exists(), + "doctor must not remove a socket or sidecar it reports" + ); let after_len = std::fs::metadata(&state_path).unwrap().len(); let after_modified = std::fs::metadata(&state_path).unwrap().modified().unwrap(); @@ -7600,9 +9685,9 @@ mod tests { detail: "detail".to_string(), repair: None, }]; - let body = doctor_json_body(&checks, &facts, &[], &stray_findings); + let body = doctor_json_body(&checks, &facts, &[], &stray_findings, (&[], None)); let obj = body.as_object().unwrap(); - assert_eq!(obj.len(), 4, "must be exactly four top-level keys: {body}"); + assert_eq!(obj.len(), 6, "must be exactly six top-level keys: {body}"); assert!(obj.contains_key("stray_processes")); let strays = obj["stray_processes"].as_array().unwrap(); assert_eq!(strays.len(), 1); @@ -7676,9 +9761,11 @@ mod tests { // Also exercise doctor() itself, end to end, twice — not just // the pure finding collector — proving the full read-only path. - doctor(dir.path(), false).unwrap(); + // A hermetic, empty cache: never sweep the operator's real one. + let cache = tempfile::tempdir().unwrap(); + doctor_in(dir.path(), false, Some(cache.path())).unwrap(); assert!(agent::agent_running(pid), "doctor() itself must not signal"); - doctor(dir.path(), false).unwrap(); + doctor_in(dir.path(), false, Some(cache.path())).unwrap(); assert!( agent::agent_running(pid), "doctor() must remain read-only across repeated runs" @@ -7771,18 +9858,20 @@ mod tests { } /// Task 1's tracer (CR-01, 999.44/DEN-68): one discovery pass names - /// a live `monitor_pid`-recorded pid, a live lock-holder pid, and a - /// genuine orphan under no registered root at all. Only the orphan - /// may survive `unreachable_stray_candidates`'s filter. All three - /// pids are real, spawned processes crossed through + /// a live lock-holder pid and a genuine orphan under no registered root + /// at all. Only the orphan may survive `unreachable_stray_candidates`'s + /// filter. Both pids are real, spawned processes crossed through /// `wait_for_exec_visibility` — not synthesised numbers — so the /// census and the filter are both exercised for real, in one pass. + /// + /// A third, state-recorded fixture (a pid in the state's monitor-pid + /// field) was removed with that input (50-22): lock holders are the + /// only reachable source, because the census cannot match a 3.x + /// supervisor (`agent::tests::the_stray_census_matches_only_the_2x_process_shapes`). #[test] fn reachable_pids_are_excluded_from_both_the_findings_and_the_reap_candidates() { - let mut state_child = spawn_wrapper_shaped_fixture(); let mut lock_child = spawn_wrapper_shaped_fixture(); let mut orphan_child = spawn_wrapper_shaped_fixture(); - let state_pid = state_child.id(); let lock_pid = lock_child.id(); let orphan_pid = orphan_child.id(); @@ -7790,22 +9879,19 @@ mod tests { let project_root_guard = tempfile::tempdir().unwrap(); let project_root = project_root_guard.path().to_path_buf(); - // Phase 1: a real `State` naming `state_pid` as its monitor. - let mut state = State::new( + // Phase 1: a real `State` with no lock, the registered phase. + let state = State::new( PhaseId::new(1), AgentKind::Claude, Mode::Auto, project_root.clone(), ); - state.monitor_pid = Some(state_pid); workflow::save_state(&state).unwrap(); // Phase 2: a real `State` (so `list_states` enumerates the // phase at all — mirrors real operation, where a phase with an - // active lock also has a persisted state), deliberately with NO - // `monitor_pid`, so this phase's only path into the reachable - // set is the lock file below, not the monitor_pid path Phase 1 - // already exercises. + // active lock also has a persisted state), whose only path into + // the reachable set is the lock file below. let state_2 = State::new( PhaseId::new(2), AgentKind::Claude, @@ -7843,10 +9929,6 @@ mod tests { .collect(); let reachable = registry_reachable_pids(®istered_roots); - assert!( - reachable.contains(&state_pid), - "state_pid must be reachable via its recorded monitor_pid" - ); assert!( reachable.contains(&lock_pid), "lock_pid must be reachable via the lock file's holder_identity" @@ -7860,11 +9942,7 @@ mod tests { // Without this, the test could pass vacuously because the // census never saw the fixtures at all (T-25-15-11). let census = agent::discover_stray_devflow_processes(); - for (pid, label) in [ - (state_pid, "state"), - (lock_pid, "lock"), - (orphan_pid, "orphan"), - ] { + for (pid, label) in [(lock_pid, "lock"), (orphan_pid, "orphan")] { assert!( census.iter().any(|p| p.pid == pid), "the {label} fixture (pid {pid}) must be part of the real /proc census" @@ -7876,10 +9954,6 @@ mod tests { retained.iter().any(|p| p.pid == orphan_pid), "the orphan must survive the filter" ); - assert!( - !retained.iter().any(|p| p.pid == state_pid), - "the state-named pid must be filtered out" - ); assert!( !retained.iter().any(|p| p.pid == lock_pid), "the lock-held pid must be filtered out" @@ -7890,26 +9964,21 @@ mod tests { findings.iter().any(|f| f.pid == orphan_pid), "the orphan must produce a finding" ); - assert!( - !findings.iter().any(|f| f.pid == state_pid), - "the state-named pid must produce no finding" - ); assert!( !findings.iter().any(|f| f.pid == lock_pid), "the lock-held pid must produce no finding" ); - // Reap all three with a VERIFIED signal, never a bare one + // Reap both with a VERIFIED signal, never a bare one // (mirrors `reap_strays_e2e.rs:202-223`'s own teardown), plus a // final `wait()` on each to reclaim the zombie regardless. - for pid in [state_pid, lock_pid, orphan_pid] { + for pid in [lock_pid, orphan_pid] { agent::terminate_and_verify( pid, agent::TERMINATE_VERIFY_WAIT, agent::TERMINATE_VERIFY_POLL, ); } - let _ = state_child.wait(); let _ = lock_child.wait(); let _ = orphan_child.wait(); } @@ -7930,13 +9999,21 @@ mod tests { let mut child = spawn_wrapper_shaped_fixture(); let pid = child.id(); - let mut state = State::new(PhaseId::new(1), AgentKind::Claude, Mode::Auto, root.clone()); - state.monitor_pid = Some(pid); + let state = State::new(PhaseId::new(1), AgentKind::Claude, Mode::Auto, root.clone()); workflow::save_state(&state).unwrap(); + // The pid's path into the reachable set: a lock file naming it as + // holder, in the `.devflow/lock-{phase:02}` shape (pid, start time). + let start_time = agent::process_start_time(pid) + .expect("must read the lock fixture's own recorded start time"); + std::fs::write( + root.join(".devflow").join("lock-01"), + format!("{pid}\n{start_time}"), + ) + .unwrap(); assert!( registry_reachable_pids(std::slice::from_ref(&root)).contains(&pid), - "the pid must be reachable while its root and state file still exist" + "the pid must be reachable while its root, state file and lock file still exist" ); std::fs::remove_dir_all(&root).unwrap(); @@ -8368,13 +10445,13 @@ mod tests { detail: "detail".to_string(), repair: None, }]; - let body = doctor_json_body(&checks, &facts, &doc_findings, &[]); + let body = doctor_json_body(&checks, &facts, &doc_findings, &[], (&[], None)); let obj = body.as_object().unwrap(); assert_eq!( obj.len(), - 4, + 6, "must be exactly {{environment, reconciliation, planning_doc_staleness, \ - stray_processes}}: {body}" + stray_processes, orphan_supervisors, orphan_supervisor_sweep_error}}: {body}" ); assert!(obj.contains_key("environment")); assert!(obj.contains_key("reconciliation")); @@ -8540,4 +10617,910 @@ mod tests { ); } } + + // ----------------------------------------------------------------------- + // 50-14: `stop` through the run's supervisor socket (crash row C13, and + // a supervisor that dies under its shutdown request). + // ----------------------------------------------------------------------- + + /// A 0700 socket directory under a short tempdir in `/tmp`, so every + /// socket path below it fits `sun_path`. The tempdir removes it on drop. + fn stop_socket_dir() -> (tempfile::TempDir, PathBuf) { + use std::os::unix::fs::PermissionsExt as _; + let base = tempfile::Builder::new() + .prefix("dfs") + .tempdir_in("/tmp") + .expect("create a short socket base under /tmp"); + let dir = base.path().join("sock"); + std::fs::create_dir(&dir).unwrap(); + std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap(); + (base, dir) + } + + /// Bind a listener at `path` with mode 0600, as a published supervisor + /// socket is. + fn bind_supervisor_socket(path: &Path) -> std::os::unix::net::UnixListener { + use std::os::unix::fs::PermissionsExt as _; + let listener = std::os::unix::net::UnixListener::bind(path).expect("bind the socket"); + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600)).unwrap(); + listener + } + + /// Wait up to `within` for `fd` to be readable (a queued connection on a + /// listener, data on a stream); whether it became readable. + fn readable_within(fd: &impl std::os::fd::AsRawFd, within: std::time::Duration) -> bool { + let mut entry = libc::pollfd { + fd: fd.as_raw_fd(), + events: libc::POLLIN, + revents: 0, + }; + let millis = libc::c_int::try_from(within.as_millis()).unwrap_or(libc::c_int::MAX); + loop { + // SAFETY: one initialized pollfd naming a descriptor the caller + // holds open for the whole call. + let ready = unsafe { libc::poll(&raw mut entry, 1, millis) }; + if ready >= 0 + || std::io::Error::last_os_error().kind() != std::io::ErrorKind::Interrupted + { + return ready == 1; + } + } + } + + /// A well-formed ping reply line for a scripted supervisor. + fn scripted_ping_reply(launch_id: &str, phase: PhaseId, owner: &str) -> String { + serde_json::json!({ + "ok": true, + "launch_id": launch_id, + "phase": phase.to_string(), + "stage": "code", + "launch": "stream", + "owner": owner, + "agent_pid": 1, + "agent_pgid": 1, + "agent_start_time": 0, + "boot_id": null, + "supervisor_pid": 1, + }) + .to_string() + } + + /// Accept one connection on `listener` within `within`, or `None`. + fn accept_within( + listener: &std::os::unix::net::UnixListener, + within: std::time::Duration, + ) -> Option { + if !readable_within(listener, within) { + return None; + } + Some(listener.accept().expect("accept a queued connection").0) + } + + /// Read one request line from `stream`, or `None` when none arrives + /// within the read timeout. + fn read_request_line(stream: &std::os::unix::net::UnixStream) -> Option { + use std::io::BufRead as _; + stream + .set_read_timeout(Some(std::time::Duration::from_secs(5))) + .unwrap(); + let mut line = String::new(); + std::io::BufReader::new(stream).read_line(&mut line).ok()?; + (!line.is_empty()).then(|| line.trim_end().to_string()) + } + + /// The lines a scripted supervisor sends for one `shutdown` request, and + /// whether it then unlinks its socket and stops serving. + struct ShutdownScript { + lines: Vec, + unlink_after: bool, + } + + /// A scripted supervisor: serves `socket` on a thread, logs each + /// connection's one request line, answers `ping` with + /// [`scripted_ping_reply`] and the n-th `shutdown` (1-based) with + /// `on_shutdown(n)`. Dropping it stops the thread and removes the socket. + struct ScriptedSupervisor { + socket: PathBuf, + log: std::sync::Arc>>, + done: std::sync::Arc, + thread: Option>, + } + + impl ScriptedSupervisor { + fn serve( + socket: &Path, + launch_id: &str, + phase: PhaseId, + owner: &str, + mut on_shutdown: impl FnMut(usize) -> ShutdownScript + Send + 'static, + ) -> Self { + use std::io::Write as _; + use std::sync::atomic::Ordering; + let listener = bind_supervisor_socket(socket); + let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let done = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let ping = scripted_ping_reply(launch_id, phase, owner); + let (thread_log, thread_done, path) = (log.clone(), done.clone(), socket.to_path_buf()); + let thread = std::thread::spawn(move || { + let mut shutdowns = 0; + while !thread_done.load(Ordering::SeqCst) { + let Some(mut stream) = + accept_within(&listener, std::time::Duration::from_millis(50)) + else { + continue; + }; + let Some(request) = read_request_line(&stream) else { + continue; + }; + thread_log.lock().unwrap().push(request.clone()); + if request.contains("\"ping\"") { + let _ = writeln!(stream, "{ping}"); + continue; + } + shutdowns += 1; + let script = on_shutdown(shutdowns); + for line in &script.lines { + let _ = writeln!(stream, "{line}"); + } + if script.unlink_after { + let _ = std::fs::remove_file(&path); + return; + } + } + }); + Self { + socket: socket.to_path_buf(), + log, + done, + thread: Some(thread), + } + } + + /// How many `shutdown` requests this supervisor has received. + fn shutdowns(&self) -> usize { + self.log + .lock() + .unwrap() + .iter() + .filter(|line| line.contains("\"shutdown\"")) + .count() + } + } + + impl Drop for ScriptedSupervisor { + fn drop(&mut self) { + self.done.store(true, std::sync::atomic::Ordering::SeqCst); + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + let _ = std::fs::remove_file(&self.socket); + } + } + + /// Save a state for `phase` whose supervisor record names `socket` and + /// `launch_id`. + fn save_state_naming(root: &Path, phase: PhaseId, socket: &Path, launch_id: &str) { + let mut state = match workflow::load_state(root, phase) { + Ok(state) => state, + Err(_) => State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()), + }; + state.supervisor = Some(SupervisorRecord::new( + socket.to_path_buf(), + launch_id.to_string(), + 1, + 0, + "boot".to_string(), + )); + workflow::save_state(&state).unwrap(); + } + + const HANDED_OFF_LINE: &str = r#"{"ok":false,"error":"handed_off"}"#; + const ACCEPTED_LINE: &str = r#"{"ok":true,"accepted":true,"owner":"agent_running"}"#; + const STOPPED_LINE: &str = + r#"{"ok":true,"stopped":true,"exit_code":143,"advance_suppressed":true}"#; + + /// Crash row C13 (V-57): the state names supervisor A; A answers the + /// shutdown by saving the state naming B (N+1's record is saved before N + /// is marked handed off) and replying `handed_off`. `stop` reloads the + /// state and stops B: A is asked once, B once, and `stopped` is recorded. + #[test] + fn stop_retries_a_handed_off_reply_against_the_reloaded_record() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let phase = PhaseId::new(1401); + let (_base, sock) = stop_socket_dir(); + let (a_path, b_path) = (sock.join("a.sock"), sock.join("b.sock")); + save_state_naming(&root, phase, &a_path, "launch-a"); + + let b = ScriptedSupervisor::serve(&b_path, "launch-b", phase, "agent_running", |_| { + ShutdownScript { + lines: vec![ACCEPTED_LINE.to_string(), STOPPED_LINE.to_string()], + unlink_after: false, + } + }); + let (a_root, a_b_path) = (root.clone(), b_path.clone()); + let a = ScriptedSupervisor::serve(&a_path, "launch-a", phase, "handed_off", move |_| { + save_state_naming(&a_root, phase, &a_b_path, "launch-b"); + ShutdownScript { + lines: vec![HANDED_OFF_LINE.to_string()], + unlink_after: false, + } + }); + + let result = stop(&root, phase); + assert!( + result.is_ok(), + "stop must reach B through the reloaded record: {result:?}" + ); + assert_eq!(a.shutdowns(), 1, "A is asked exactly once"); + assert_eq!(b.shutdowns(), 1, "B is asked exactly once"); + let state = workflow::load_state(&root, phase).unwrap(); + assert!(state.stopped, "the stop B performed is recorded"); + } + + /// C13 control (V-57): every reload names A again, so `stop` gives up + /// after the first request and 3 retries, with an error naming + /// `handed_off` and the attempt count, and records nothing. A unlinks its + /// socket after a fifth request, so a loop past the cap ends as Gone and + /// fails the count instead of hanging. + #[test] + fn stop_gives_up_after_three_handed_off_retries_and_records_nothing() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let phase = PhaseId::new(1402); + let (_base, sock) = stop_socket_dir(); + let a_path = sock.join("a.sock"); + save_state_naming(&root, phase, &a_path, "launch-a"); + let state_file = workflow::state_path(&root, phase); + let before = std::fs::read(&state_file).unwrap(); + + let a = ScriptedSupervisor::serve(&a_path, "launch-a", phase, "handed_off", |count| { + ShutdownScript { + lines: vec![HANDED_OFF_LINE.to_string()], + unlink_after: count >= 5, + } + }); + + let error = stop(&root, phase) + .expect_err("a supervisor that keeps answering handed_off is not a stop") + .to_string(); + assert!( + error.contains("handed_off") && error.contains('4'), + "the error names handed_off and the attempt count: {error}" + ); + assert_eq!(a.shutdowns(), 4, "the first request plus exactly 3 retries"); + assert_eq!( + std::fs::read(&state_file).unwrap(), + before, + "nothing is recorded" + ); + } + + /// The sidecar and state record of a launch recorded in an earlier boot: + /// the backstop proves its group gone without reading a process, so it + /// removes the files and signals nothing. + fn plant_previous_boot_launch(root: &Path, phase: PhaseId, socket: &Path) { + let sidecar = serde_json::json!({ + "launch_id": "aaaa0000aaaa000f", + "project_root": root, + "phase": phase, + "stage": "code", + "agent": "claude", + "launch": "stream", + "agent_pid": 99_999_999, + "agent_pgid": 99_999_999, + "agent_start_time": 1, + "boot_id": "an-earlier-boot", + "supervisor_pid": 99_999_998, + "supervisor_start_time": 1, + "created_at": "2026-10-17T00:00:00Z", + }); + std::fs::write(socket.with_extension("json"), sidecar.to_string()).unwrap(); + let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); + state.supervisor = Some(SupervisorRecord::new( + socket.to_path_buf(), + "aaaa0000aaaa000f".to_string(), + 99_999_999, + 1, + "an-earlier-boot".to_string(), + )); + workflow::save_state(&state).unwrap(); + } + + /// The race 6d46008 left to `stop`: the supervisor answers the probe's + /// ping, then its listener is released with the shutdown's connect still + /// queued, so `request_shutdown` reads a reset or a broken pipe. That is + /// a supervisor that died, not a failed stop: `stop` probes once more, + /// reads STALE, runs the backstop (which removes the files and signals + /// nothing for this earlier-boot launch) and records `stopped`. + #[test] + fn stop_acts_on_a_supervisor_that_dies_under_its_shutdown_request() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let phase = PhaseId::new(1403); + let (_base, sock) = stop_socket_dir(); + let socket = sock.join("r.sock"); + plant_previous_boot_launch(&root, phase, &socket); + let listener = bind_supervisor_socket(&socket); + let ping = scripted_ping_reply("aaaa0000aaaa000f", phase, "agent_running"); + let server = std::thread::spawn(move || { + use std::io::Write as _; + let wait = std::time::Duration::from_secs(30); + let Some(mut first) = accept_within(&listener, wait) else { + return "no probe connected"; + }; + if read_request_line(&first).is_none_or(|line| !line.contains("\"ping\"")) { + return "the first request was not a ping"; + } + let _ = writeln!(first, "{ping}"); + drop(first); + // Release the listener with the shutdown's connect queued in it + // and never accepted. + if !readable_within(&listener, wait) { + return "no shutdown connect was queued"; + } + drop(listener); + "released" + }); + + let result = stop(&root, phase); + assert_eq!(server.join().expect("server thread"), "released"); + assert!( + result.is_ok(), + "a supervisor that died under the shutdown is torn down, not an error: {result:?}" + ); + assert!( + !socket.exists() && !socket.with_extension("json").exists(), + "the backstop removed the stale socket and its sidecar" + ); + assert!( + workflow::load_state(&root, phase).unwrap().stopped, + "the confirmed-not-live stop is recorded" + ); + } + + /// Control for the race: a listener that stays up but closes the + /// shutdown (and every later connection) with its request unread is not + /// a dead supervisor. `stop` fails, removes nothing and records nothing. + #[test] + fn stop_reports_a_shutdown_closed_by_a_live_listener() { + /// Stops an unbounded re-probe from hanging this test. + const ACCEPT_AT_MOST: usize = 6; + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let phase = PhaseId::new(1404); + let (_base, sock) = stop_socket_dir(); + let socket = sock.join("l.sock"); + plant_previous_boot_launch(&root, phase, &socket); + let state_file = workflow::state_path(&root, phase); + let before = std::fs::read(&state_file).unwrap(); + let listener = bind_supervisor_socket(&socket); + let ping = scripted_ping_reply("aaaa0000aaaa000f", phase, "agent_running"); + let (stop_server, stopped) = std::sync::mpsc::channel::<()>(); + let server = std::thread::spawn(move || { + use std::io::Write as _; + let mut accepted = 0; + while accepted < ACCEPT_AT_MOST && stopped.try_recv().is_err() { + let Some(mut stream) = + accept_within(&listener, std::time::Duration::from_millis(50)) + else { + continue; + }; + accepted += 1; + if accepted == 1 { + let _ = read_request_line(&stream); + let _ = writeln!(stream, "{ping}"); + continue; + } + // Close with the request unread, which resets the client. + let _ = readable_within(&stream, std::time::Duration::from_secs(5)); + drop(stream); + } + (listener, accepted) + }); + + let result = stop(&root, phase); + stop_server.send(()).expect("stop the server"); + let (listener, accepted) = server.join().expect("server thread"); + drop(listener); + assert!( + accepted >= 2, + "stop never sent its shutdown ({accepted} connects)" + ); + assert!( + result.is_err(), + "a live listener that closes the shutdown is not a stop: {result:?}" + ); + assert!( + socket.exists() && socket.with_extension("json").exists(), + "nothing is removed" + ); + assert_eq!( + std::fs::read(&state_file).unwrap(), + before, + "nothing is recorded" + ); + } + + /// A `sleep` child recorded as the holder of `root`'s lock for `phase`, + /// with its own pid and start time: the identity `stop_via_lock` + /// signals. Returns the child and the lock file. + fn hold_lock_by_a_child(root: &Path, phase: PhaseId) -> (std::process::Child, PathBuf) { + let child = std::process::Command::new("sleep") + .arg("30") + .spawn() + .expect("spawn sleep"); + let pid = child.id(); + let mut start = None; + for _ in 0..100 { + start = agent::process_start_time(pid); + if start.is_some() { + break; + } + std::thread::sleep(std::time::Duration::from_millis(10)); + } + let start = start.expect("read the child's start time"); + let lock_path = root + .join(".devflow") + .join(format!("lock-{padded}", padded = phase.padded())); + std::fs::create_dir_all(lock_path.parent().unwrap()).unwrap(); + std::fs::write(&lock_path, format!("{pid}\n{start}")).unwrap(); + (child, lock_path) + } + + /// `stop` during a `start` launcher's publish wait: the state loads but + /// records no supervisor yet, the spawned supervisor already serves its + /// socket, and the launcher holds the lock. `stop` must end that + /// supervisor through its socket, not signal the launcher (which would + /// die and leave the supervisor and its agent running unrecorded). The + /// scripted supervisor drops the lock as it stops, as the launcher does + /// once its supervisor is gone. + #[test] + fn stop_ends_an_unrecorded_supervisor_through_its_socket_not_the_launcher() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let phase = PhaseId::new(1405); + workflow::save_state(&State::new( + phase, + AgentKind::Claude, + Mode::Auto, + root.clone(), + )) + .unwrap(); + let (base, _sock) = stop_socket_dir(); + let launch_id = "aaaa0000aaaa1405"; + let socket = supervisor::SocketPath::for_launch(base.path(), &root, phase, launch_id) + .expect("socket path fits sun_path"); + let (mut launcher, lock_path) = hold_lock_by_a_child(&root, phase); + let released = lock_path.clone(); + let unrecorded = + ScriptedSupervisor::serve(socket.as_path(), launch_id, phase, "agent_running", { + move |_| { + let _ = std::fs::remove_file(&released); + ShutdownScript { + lines: vec![ACCEPTED_LINE.to_string(), STOPPED_LINE.to_string()], + unlink_after: true, + } + } + }); + + let result = stop_in(&root, phase, Some(base.path())); + let launcher_alive = launcher.try_wait().unwrap().is_none(); + let _ = launcher.kill(); + let _ = launcher.wait(); + assert_eq!( + unrecorded.shutdowns(), + 1, + "stop must shut the unrecorded supervisor down through its socket: {result:?}" + ); + assert!(launcher_alive, "the launcher must not be signalled"); + assert!(result.is_ok(), "the stop happened: {result:?}"); + assert!( + workflow::load_state(&root, phase).unwrap().stopped, + "the stop is recorded" + ); + } + + /// The several-sockets refusal also covers a loaded state with no + /// record: with two live sockets for the phase, `stop` cannot tell which + /// is the run, so it asks neither and signals no lock holder. + #[test] + fn stop_refuses_several_unrecorded_sockets_and_signals_nothing() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + let phase = PhaseId::new(1406); + workflow::save_state(&State::new( + phase, + AgentKind::Claude, + Mode::Auto, + root.clone(), + )) + .unwrap(); + let (base, _sock) = stop_socket_dir(); + let serve = |launch_id: &str| { + let socket = supervisor::SocketPath::for_launch(base.path(), &root, phase, launch_id) + .expect("socket path fits sun_path"); + ScriptedSupervisor::serve(socket.as_path(), launch_id, phase, "agent_running", |_| { + ShutdownScript { + lines: vec![ACCEPTED_LINE.to_string(), STOPPED_LINE.to_string()], + unlink_after: true, + } + }) + }; + let (a, b) = (serve("aaaa0000aaaa140a"), serve("aaaa0000aaaa140b")); + let (mut launcher, _lock_path) = hold_lock_by_a_child(&root, phase); + + let result = stop_in(&root, phase, Some(base.path())); + let launcher_alive = launcher.try_wait().unwrap().is_none(); + let _ = launcher.kill(); + let _ = launcher.wait(); + let Err(CliError::Message(text)) = &result else { + panic!("several live sockets must refuse: {result:?}"); + }; + assert!( + text.contains("2 live supervisor sockets") && text.contains("no state file names one"), + "{text}" + ); + assert_eq!((a.shutdowns(), b.shutdowns()), (0, 0), "neither is asked"); + assert!(launcher_alive, "no lock holder is signalled"); + } + + /// 50-15: the start/resume guard over a state file that will not load + /// (D-09). These drive the guard in-process against a tempdir cache, so + /// nothing reads the operator's real `~/.cache/devflow`. + mod launch_guard { + use super::*; + use supervisor::backstop::BackstopDecision; + + /// A project root whose phase state file exists and will not load. + fn root_with_an_unloadable_state(phase: PhaseId) -> tempfile::TempDir { + let root = tempfile::tempdir().unwrap(); + let state = workflow::state_path(root.path(), phase); + std::fs::create_dir_all(state.parent().unwrap()).unwrap(); + std::fs::write(&state, b"{ this does not parse").unwrap(); + assert!(workflow::load_state(root.path(), phase).is_err()); + root + } + + /// The 50-06 threat flag: `sweep_in` answers a socket directory it + /// cannot list with no entries, which must not read as "nothing + /// live" here. The guard fails closed and names the directory. + #[test] + fn the_launch_guard_refuses_an_unlistable_socket_dir_over_an_unloadable_state() { + use std::os::unix::fs::PermissionsExt as _; + let phase = PhaseId::new(1501); + let root = root_with_an_unloadable_state(phase); + let cache = tempfile::tempdir().unwrap(); + let sock = registry::socket_dir_in(cache.path()); + std::fs::create_dir_all(&sock).unwrap(); + std::fs::set_permissions(&sock, std::fs::Permissions::from_mode(0o000)).unwrap(); + // Both reads run without DAC override, so uid 0 (the pinned + // container) is refused too (WINDOWS.md #19). + let (listable, result) = without_dac_override(|| { + ( + std::fs::read_dir(&sock).is_ok(), + refuse_launch_over_a_live_run( + root.path(), + phase, + LaunchVerb::Start, + Some(cache.path()), + ), + ) + }); + std::fs::set_permissions(&sock, std::fs::Permissions::from_mode(0o700)).unwrap(); + assert!( + !listable, + "precondition: a 0o000 directory cannot be listed" + ); + let Err(CliError::Message(text)) = result else { + panic!("an unlistable socket directory must refuse: {result:?}"); + }; + for fragment in [ + sock.display().to_string(), + format!( + "cannot be listed ({})", + std::io::ErrorKind::PermissionDenied + ), + "refusing to start; nothing was written".to_string(), + ] { + assert!(text.contains(&fragment), "missing {fragment:?}: {text}"); + } + } + + /// Control for the test above: a socket directory that does not + /// exist means no launch has published a socket, so nothing is live. + #[test] + fn the_launch_guard_proceeds_over_an_unloadable_state_when_the_socket_dir_is_missing() { + let phase = PhaseId::new(1502); + let root = root_with_an_unloadable_state(phase); + let cache = tempfile::tempdir().unwrap(); + assert!(!registry::socket_dir_in(cache.path()).exists()); + let result = refuse_launch_over_a_live_run( + root.path(), + phase, + LaunchVerb::Resume, + Some(cache.path()), + ); + assert!( + result.is_ok(), + "a missing socket dir is not a refusal: {result:?}" + ); + } + + /// T-50-40: only the three backstop decisions that prove a STALE + /// socket's agent group gone let a launch through; every refusal, + /// an unprovable identity and a live leader count as live. + #[test] + fn only_a_provably_gone_group_lets_a_stale_socket_through_the_launch_guard() { + for gone in [ + BackstopDecision::PreviousBoot, + BackstopDecision::EmptyGroup, + BackstopDecision::LeaderRecycled, + ] { + assert!( + !stale_group_may_live(&gone), + "{gone:?} proves the group gone" + ); + } + for live in [ + BackstopDecision::LeaderPresentSignal, + BackstopDecision::RefuseLeaderGone(Vec::new()), + BackstopDecision::RefuseMismatch("agent_pgid".to_string()), + BackstopDecision::RefuseUnsafeTarget, + BackstopDecision::RefuseIdentityUnavailable, + ] { + assert!(stale_group_may_live(&live), "{live:?} must fail closed"); + } + } + + /// An ALIVE scripted supervisor at this root and phase's socket under + /// `cache`, with no sidecar, so the sweep attributes it by file name. + fn serve_unrecorded(cache: &Path, root: &Path, phase: PhaseId) -> ScriptedSupervisor { + let launch_id = "aaaa0000aaaa1503"; + let socket = supervisor::SocketPath::for_launch(cache, root, phase, launch_id) + .expect("socket path fits sun_path"); + ScriptedSupervisor::serve(socket.as_path(), launch_id, phase, "agent_running", |_| { + ShutdownScript { + lines: Vec::new(), + unlink_after: false, + } + }) + } + + /// A launch whose state file is missing is still found through its + /// socket, as `stop` finds it: `recover --clean` leaves a live + /// supervisor's socket behind without a state file. + #[test] + fn the_launch_guard_finds_a_live_socket_when_the_state_file_is_missing() { + let phase = PhaseId::new(1503); + let root = tempfile::tempdir().unwrap(); + assert!(!workflow::state_path(root.path(), phase).exists()); + let (base, _sock) = stop_socket_dir(); + let _alive = serve_unrecorded(base.path(), root.path(), phase); + let result = refuse_launch_over_a_live_run( + root.path(), + phase, + LaunchVerb::Start, + Some(base.path()), + ); + let Err(CliError::Message(text)) = result else { + panic!("a live socket with no state file must refuse: {result:?}"); + }; + for fragment in [ + "supervisor launch aaaa0000aaaa1503 ALIVE", + "refusing to start; nothing was written", + ] { + assert!(text.contains(fragment), "missing {fragment:?}: {text}"); + } + } + + /// The unlistable-directory refusal also covers a missing state file: + /// with neither a state nor a listing, nothing can rule a run out. + #[test] + fn the_launch_guard_refuses_an_unlistable_socket_dir_when_the_state_file_is_missing() { + use std::os::unix::fs::PermissionsExt as _; + let phase = PhaseId::new(1504); + let root = tempfile::tempdir().unwrap(); + let cache = tempfile::tempdir().unwrap(); + let sock = registry::socket_dir_in(cache.path()); + std::fs::create_dir_all(&sock).unwrap(); + std::fs::set_permissions(&sock, std::fs::Permissions::from_mode(0o000)).unwrap(); + let (listable, result) = without_dac_override(|| { + ( + std::fs::read_dir(&sock).is_ok(), + refuse_launch_over_a_live_run( + root.path(), + phase, + LaunchVerb::Start, + Some(cache.path()), + ), + ) + }); + std::fs::set_permissions(&sock, std::fs::Permissions::from_mode(0o700)).unwrap(); + assert!( + !listable, + "precondition: a 0o000 directory cannot be listed" + ); + let Err(CliError::Message(text)) = result else { + panic!("an unlistable socket directory must refuse: {result:?}"); + }; + for fragment in [ + "there is no state file".to_string(), + sock.display().to_string(), + "refusing to start; nothing was written".to_string(), + ] { + assert!(text.contains(&fragment), "missing {fragment:?}: {text}"); + } + } + + /// Controls for the two tests above: a fresh start (no state, and no + /// socket directory or an empty one) proceeds, and the no-state + /// carve-out still ignores a live pid in the agent pid file. + #[test] + fn the_launch_guard_proceeds_with_no_state_and_no_live_socket() { + let phase = PhaseId::new(1505); + let root = tempfile::tempdir().unwrap(); + let cache = tempfile::tempdir().unwrap(); + let guard = || { + refuse_launch_over_a_live_run( + root.path(), + phase, + LaunchVerb::Start, + Some(cache.path()), + ) + }; + assert!(guard().is_ok(), "no socket directory: {:?}", guard()); + std::fs::create_dir_all(registry::socket_dir_in(cache.path())).unwrap(); + assert!(guard().is_ok(), "an empty socket directory: {:?}", guard()); + let pid_file = agent_result::agent_pid_path(root.path(), phase); + std::fs::create_dir_all(pid_file.parent().unwrap()).unwrap(); + std::fs::write(&pid_file, std::process::id().to_string()).unwrap(); + assert!( + guard().is_ok(), + "the no-state carve-out ignores a live agent pid: {:?}", + guard() + ); + } + } + + /// A sidecar whose launch id is not `new_launch_id`'s format is read as + /// unusable, so no copy-paste `--launch` line ever carries that id: the + /// stop line names the id from the socket's file name instead. + #[test] + fn a_hostile_sidecar_launch_id_never_reaches_a_stop_line() { + let (base, _sock) = stop_socket_dir(); + let root = tempfile::tempdir().unwrap(); + let phase = PhaseId::new(1506); + let name_id = "aaaa0000aaaa1506"; + let hostile = "x;touch${IFS}pwned"; + let socket = supervisor::SocketPath::for_launch(base.path(), root.path(), phase, name_id) + .expect("socket path fits sun_path"); + drop(bind_supervisor_socket(socket.as_path())); + let sidecar = serde_json::json!({ + "launch_id": hostile, + "project_root": root.path(), + "phase": phase, + "stage": "code", + "agent": "claude", + "launch": "stream", + "agent_pid": 1, + "agent_pgid": 1, + "agent_start_time": 1, + "boot_id": "boot", + "supervisor_pid": 1, + "supervisor_start_time": 1, + "created_at": "2026-10-17T00:00:00Z", + }); + std::fs::write(socket.sidecar_path(), sidecar.to_string()).unwrap(); + let entries = supervisor::sweep_for_in(base.path(), root.path(), phase); + let [entry] = entries.as_slice() else { + panic!("one entry for this root and phase: {entries:?}"); + }; + let line = sweep_entry_stop_line(phase, entry); + assert!(!line.contains(hostile), "{line}"); + assert!( + line.contains(&format!("--launch {name_id}")) && line.contains("does not parse"), + "{line}" + ); + let live = swept_live_socket(entry).expect("a STALE socket without a usable sidecar"); + let repair = live.repair_line(phase); + assert!(!repair.contains(hostile), "{repair}"); + } + + /// `recover --clean --phase N` uses the launch guard's supervisor rule: + /// a STALE socket whose group the backstop cannot prove gone, or a probe + /// that fails, refuses and deletes nothing, as an ALIVE one does. + mod recover_guard { + use super::*; + + /// A loadable state for `phase` whose record names `socket`, with a + /// sidecar beside it that records `agent_pgid` in this boot. + fn plant_stale_launch(root: &Path, phase: PhaseId, socket: &Path, agent_pgid: u32) { + let boot = supervisor::boot_id().expect("a readable boot id"); + let sidecar = serde_json::json!({ + "launch_id": "aaaa0000aaaa1601", + "project_root": root, + "phase": phase, + "stage": "code", + "agent": "claude", + "launch": "stream", + "agent_pid": agent_pgid, + "agent_pgid": agent_pgid, + "agent_start_time": 1, + "boot_id": boot, + "supervisor_pid": 99_999_998, + "supervisor_start_time": 1, + "created_at": "2026-10-17T00:00:00Z", + }); + std::fs::write(socket.with_extension("json"), sidecar.to_string()).unwrap(); + let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); + state.supervisor = Some(SupervisorRecord::new( + socket.to_path_buf(), + "aaaa0000aaaa1601".to_string(), + agent_pgid, + 1, + boot, + )); + workflow::save_state(&state).unwrap(); + } + + /// A STALE socket over pgid 1 (the backstop refuses it as an unsafe + /// target, so its group is not proved gone) refuses `recover`. + #[test] + fn recover_refuses_a_stale_socket_whose_group_may_live() { + let dir = tempfile::tempdir().unwrap(); + let phase = PhaseId::new(1601); + let (_base, sock) = stop_socket_dir(); + let socket = sock.join("s.sock"); + plant_stale_launch(dir.path(), phase, &socket, 1); + drop(bind_supervisor_socket(&socket)); + assert!(matches!( + supervisor::probe(&socket), + Ok(supervisor::Liveness::Stale) + )); + let result = refuse_a_live_supervisor(dir.path(), phase, None); + let Err(CliError::Message(text)) = result else { + panic!("a STALE socket whose group may live must refuse: {result:?}"); + }; + assert!( + text.contains("recover --clean deleted nothing") && text.contains("STALE"), + "{text}" + ); + } + + /// A recorded socket path whose probe fails (a regular file) refuses. + #[test] + fn recover_refuses_a_recorded_socket_whose_probe_fails() { + let dir = tempfile::tempdir().unwrap(); + let phase = PhaseId::new(1602); + let (_base, sock) = stop_socket_dir(); + let socket = sock.join("f.sock"); + plant_stale_launch(dir.path(), phase, &socket, 1); + std::fs::write(&socket, b"not a socket").unwrap(); + let result = refuse_a_live_supervisor(dir.path(), phase, None); + let Err(CliError::Message(text)) = result else { + panic!("a failed probe must refuse: {result:?}"); + }; + assert!( + text.contains("recover --clean deleted nothing") && text.contains("probe failed"), + "{text}" + ); + } + + /// Controls: a GONE socket, and a STALE one whose group the backstop + /// proves gone (a previous boot), let `recover` through. + #[test] + fn recover_proceeds_over_a_gone_or_provably_dead_socket() { + let dir = tempfile::tempdir().unwrap(); + let phase = PhaseId::new(1603); + let (_base, sock) = stop_socket_dir(); + let socket = sock.join("g.sock"); + plant_previous_boot_launch(dir.path(), phase, &socket); + let gone = refuse_a_live_supervisor(dir.path(), phase, None); + assert!(gone.is_ok(), "a GONE socket is not live: {gone:?}"); + drop(bind_supervisor_socket(&socket)); + let stale = refuse_a_live_supervisor(dir.path(), phase, None); + assert!(stale.is_ok(), "a previous boot's group is gone: {stale:?}"); + } + } } diff --git a/crates/devflow-cli/src/main.rs b/crates/devflow-cli/src/main.rs index 4ecdddf1..8b871c5d 100644 --- a/crates/devflow-cli/src/main.rs +++ b/crates/devflow-cli/src/main.rs @@ -26,7 +26,7 @@ mod commands; use commands::{ cleanup, doctor, evidence, gate_list, gate_respond, gate_show, gate_sweep, history_cmd, list, logs, recover_cmd, reference, release_check, release_verify, resolve_gate_target, start, - status, stop, test_cmd, + status, stop, stop_launch, test_cmd, }; use devflow_core::phase_id::PhaseId; @@ -90,8 +90,8 @@ enum Command { #[arg(long)] yes_ship: bool, /// Force the pre-31 single-document Claude launch: the prompt - /// positionally in argv, `--output-format json`, and the `sh` monitor - /// (D-11, `31-CONTEXT.md`). Off by default. + /// positionally in argv, `--output-format json`, and the capture-shape + /// supervisor (D-11, `31-CONTEXT.md`). Off by default. /// /// It exists so an operator can recover from a defect in the /// `stream-json` transport without waiting for a release. Its use is @@ -128,12 +128,13 @@ enum Command { #[arg(long)] stage: Option, }, - /// Internal: the pipe-owning monitor's own process entry point (Phase 31). + /// Internal: the stage supervisor's own process entry point (50-11). /// - /// `monitor::spawn_monitor`'s `PipeOwning` arm re-execs this binary as this - /// subcommand, detached. Everything after `--` is the supervised child's - /// program and argv. Hidden for the same reason `advance` is: it is an - /// implementation detail of the monitor chain, never an operator verb. + /// `monitor::spawn_monitor` re-execs this binary as this subcommand, + /// detached and in its own process group, for both launch shapes. + /// Everything after `--` is the supervised agent's program and argv. + /// Hidden for the same reason `advance` is: it is an implementation + /// detail of the monitor chain, never an operator verb. #[command(name = "__monitor", hide = true)] Monitor { /// Project root whose `.devflow/` capture files this monitor writes. @@ -149,17 +150,28 @@ enum Command { /// worktree mode is active, else the project root). #[arg(long)] workdir: PathBuf, - /// File holding the stage prompt. A file, not argv: argv has a hard - /// length ceiling and DevFlow stage prompts routinely exceed it. - #[arg(long)] - prompt_file: PathBuf, - /// Seconds of stream silence before the idle timeout fires. - #[arg(long)] - idle_timeout_secs: u64, /// Which agent's transport the monitor supervises (selects the /// first-turn schema and the close-rule marker predicate). #[arg(long)] agent: AgentKind, + /// Supervision shape: `capture` (stdin null, stdout to the capture file) or `stream`. + #[arg(long, value_enum)] + launch: pipeline_launch::LaunchShape, + /// Socket path this supervisor publishes, under the DevFlow cache dir. + #[arg(long)] + socket: PathBuf, + /// Per-launch id the launcher waits to see in this supervisor's ping reply. + #[arg(long)] + launch_id: String, + /// Adapter env var NAME to re-apply to the agent (repeatable; the value stays off argv). + #[arg(long = "keep-env")] + keep_env: Vec, + /// File holding the stage prompt (stream shape; a file, not argv, for length). + #[arg(long)] + prompt_file: Option, + /// Seconds of stream silence before the idle timeout fires (stream shape). + #[arg(long)] + idle_timeout_secs: Option, /// The supervised child's program and arguments, after `--`. #[arg(trailing_var_arg = true, allow_hyphen_values = true, required = true)] argv: Vec, @@ -340,18 +352,22 @@ enum Command { #[arg(default_value = ".")] project: PathBuf, }, - /// End a running phase cleanly (23c): answers its open gate with a - /// rejection if one is open — the target unwinds through its own abort - /// path, no signal sent — otherwise signals the process recorded in its - /// per-phase lock file (`.devflow/lock-{phase:02}`), never - /// `state.monitor_pid` (the PID `devflow status` displays, and the - /// wrong one — see `commands::stop`'s doc comment). Idempotent: safe to + /// End a running phase cleanly: answers its open gate with a rejection + /// when a live waiter holds it; otherwise addresses the run's supervisor + /// socket — `shutdown` when ALIVE, the guarded backstop when STALE, the + /// per-phase lock holder when GONE. `--launch ` tears down only the + /// socket with that launch id (the form `devflow doctor` prints for an + /// orphan). Never signals a pid recorded in state. Idempotent: safe to /// run against an already-stopped, never-started, or already-dead /// phase. Stop { /// Phase to stop. #[arg(long)] phase: PhaseId, + /// Tear down only the supervisor socket with this launch id; answers + /// no gate, signals no lock holder and writes no state. + #[arg(long)] + launch: Option, /// Project root. Overrides the positional argument when supplied. #[arg(long)] root: Option, @@ -474,18 +490,21 @@ enum GateCmd { /// machine-wide regardless of this flag (CR-01, 25-15). #[arg(long)] root: Option, - /// Also discover and clear devflow processes shaped like a monitor - /// wrapper or an `advance` child (999.44), by scanning the OS + /// Also discover and clear devflow 2.x processes shaped like a + /// monitor wrapper or an `advance` child (999.44), by scanning the OS /// process table directly rather than trusting a lock file the - /// process itself wrote. Matches only two structural argv shapes, - /// owned by the calling user, older than the minimum age, and NOT - /// named by any registered root's state file or lock file (CR-01, - /// 25-15) — a pid a live registry entry still reaches is never - /// touched. Discovery itself stays registry-independent, so this - /// still catches a process whose project root no longer exists on - /// disk (`devflow stop`/the default sweep above cannot see - /// either). Off by default: preview with `--dry-run` before - /// authorising it for real. + /// process itself wrote. Both shapes are 2.x only: a 3.x binary + /// starts neither, so this never matches a 3.x supervisor, which + /// `devflow doctor` reports through its socket and + /// `devflow stop --phase N --launch ` ends. Matches only two + /// structural argv shapes, owned by the calling user, older than the + /// minimum age, and NOT named by any registered root's state file or + /// lock file (CR-01, 25-15) — a pid a live registry entry still + /// reaches is never touched. Discovery itself stays + /// registry-independent, so this still catches a 2.x process whose + /// project root no longer exists on disk (`devflow stop`/the default + /// sweep above cannot see either). Off by default: preview with + /// `--dry-run` before authorising it for real. #[arg(long)] reap_strays: bool, }, @@ -590,19 +609,30 @@ fn run() -> Result<(), CliError> { phase, stage, workdir, + agent, + launch, + socket, + launch_id, + keep_env, prompt_file, idle_timeout_secs, - agent, argv, } => run_monitor( - &project_root(project)?, - phase, - stage, - &workdir, - &prompt_file, - idle_timeout_secs, - agent, - &argv, + &pipeline_launch::MonitorInvocation { + project_root: project_root(project)?, + phase, + stage, + workdir, + agent, + launch, + socket, + launch_id, + keep_env, + prompt_file, + idle_timeout_secs, + argv, + }, + true, ), Command::Resume { phase, @@ -714,9 +744,16 @@ fn run() -> Result<(), CliError> { } => ship_override(&project_root(project)?, phase, force), Command::Stop { phase, + launch, root, project, - } => stop(&project_root(root.unwrap_or(project))?, phase), + } => { + let root = project_root(root.unwrap_or(project))?; + match launch { + Some(launch_id) => stop_launch(&root, phase, &launch_id), + None => stop(&root, phase), + } + } Command::Evidence { phase, json, diff --git a/crates/devflow-cli/src/pipeline_gate.rs b/crates/devflow-cli/src/pipeline_gate.rs index 7c3ad9c3..e99cdb69 100644 --- a/crates/devflow-cli/src/pipeline_gate.rs +++ b/crates/devflow-cli/src/pipeline_gate.rs @@ -21,7 +21,7 @@ use crate::CliError; use crate::config_parse::{foreground_gate_timeout_secs, gate_timeout_secs}; -use crate::pipeline_launch::launch_stage; +use crate::pipeline_launch::{ADVANCE_STOP_REASON, GateWaitEnd, launch_stage}; use crate::pipeline_outcomes::{run_checkout_hooks, truncate_reason}; use devflow_core::gates::{self, GateAction, GateError, GateResponse, Gates}; use devflow_core::hooks; @@ -68,10 +68,13 @@ pub(crate) fn transition( if state.stop_until == Some(from) { state.stopped = true; state.stop_reason = Some(format!("stopped after {from} completed (--until {from})")); - // T-20-03b / doctor gap (D-09): clear monitor_pid/gate_pending so - // neither check_dead_agent nor check_dead_monitor misreports this - // intentional stop as a crashed agent or dead monitor. - state.monitor_pid = None; + // T-20-03b / doctor gap (D-09): clear the supervisor record and + // gate_pending so neither check_dead_agent nor check_dead_monitor + // misreports this intentional stop as a crashed agent or dead + // monitor (50-13): the supervisor running this advance finishes once + // it returns, so the record would otherwise name a socket that is + // gone. + state.supervisor = None; state.gate_pending = false; workflow::save_state(state)?; events::emit( @@ -230,6 +233,21 @@ pub(crate) fn finish_workflow_with_gate_timeout( state: &mut State, gate_timeout_secs: u64, ) -> Result<(), CliError> { + #[cfg(test)] + crate::pipeline_launch::test_hold("before_finalization"); + // Safe point 6 (CR-01, 50-REVIEW.md): a stop taken during the advance + // after the Ship gate was answered is honoured before the merge, version + // bump, tag and push. The consumed approval is cleared with it, so a + // resumed run asks again. Outside a supervisor's advance (the foreground + // `ship_override`) no stop token is engaged and this proceeds. + if crate::pipeline_launch::stop_requested_at_safe_point() { + let _ = Gates::cleanup(project_root, state.phase, Stage::Ship); + state.gate_pending = false; + crate::pipeline_launch::record_advance_stop(state, 6)?; + return Err(CliError::Message(format!( + "finalization did not run: {ADVANCE_STOP_REASON}" + ))); + } loop { if run_checkout_hooks(project_root, state, &hooks::hooks_after_ship(), Stage::Ship) { break; @@ -418,6 +436,19 @@ pub(crate) fn run_gate_with_timeout( // `Gates::write_gate` above, and `poll_response` would never observe a // response written after it returns). if let Some(response) = auto_response { + // Safe point 5 (CR-01, 50-REVIEW.md): a response already on disk + // wins over a stop in the wait below, so a stop taken during the + // advance is honoured before the pre-authorized one is written. As + // at safe point 4, the gate is cleared, the stop recorded and an + // error unwinds every caller. + if crate::pipeline_launch::stop_requested_at_safe_point() { + let _ = Gates::cleanup(project_root, state.phase, stage); + state.gate_pending = false; + crate::pipeline_launch::record_advance_stop(state, 5)?; + return Err(CliError::Message(format!( + "the gate for stage {stage} was not auto-answered: {ADVANCE_STOP_REASON}" + ))); + } match Gates::respond(project_root, state.phase, stage, response) { Ok(_) => {} // A human or 23b's stale-gate sweep may have answered first; @@ -426,8 +457,13 @@ pub(crate) fn run_gate_with_timeout( Err(err) => return Err(err.into()), } } - match Gates::poll_response(project_root, state.phase, stage, timeout_secs) { - Some(response) => { + match crate::pipeline_launch::wait_for_gate_response( + project_root, + state.phase, + stage, + timeout_secs, + ) { + GateWaitEnd::Answered(response) => { state.gate_pending = false; // WR-10 (49-REVIEW.md): the operator may have edited the run's // notify command on disk while this gate waited; keep that edit @@ -455,7 +491,25 @@ pub(crate) fn run_gate_with_timeout( ); Ok(action) } - None => { + GateWaitEnd::Stopped => { + // Safe point 4 (50-13, R-2): the supervisor running this advance + // was stopped during the wait. The gate has no waiter any more, so + // it is cleared (no stale request or response outlives the run), + // the stop is recorded, and an error unwinds every caller through + // `?`, which releases the per-phase lock. Deliberately not + // `GateAction::Abort`: `abort()` deletes the state file, and a + // stopped run keeps its state for `devflow resume`. + let _ = Gates::cleanup(project_root, state.phase, stage); + state.gate_pending = false; + if let Ok(on_disk) = workflow::load_state(project_root, state.phase) { + state.gate_notify_cmd = on_disk.gate_notify_cmd; + } + crate::pipeline_launch::record_advance_stop(state, 4)?; + Err(CliError::Message(format!( + "the gate wait for stage {stage} ended: {ADVANCE_STOP_REASON}" + ))) + } + GateWaitEnd::TimedOut => { events::emit( project_root, state.phase, @@ -727,7 +781,7 @@ mod tests { if !devflow_core::test_support::in_child_test($name) { let path_dir = $path_dir; let output = - devflow_core::test_support::run_test_in_child($name, path_dir.path(), &[]); + crate::test_support::run_hermetic_test_in_child($name, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test( &output, $name, ); @@ -888,6 +942,36 @@ mod tests { assert!(devflow_core::ship_evidence::is_stopped_at(&evidence)); } + /// 50-13: the `--until` stop clears the supervisor record, because the + /// supervisor running this advance finishes once the advance returns. + /// The record is set first, so a stop that leaves it standing fails here. + #[test] + fn until_stop_clears_the_supervisor_record() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let phase = PhaseId::new(1313); + let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); + state.stage = Stage::Plan; + state.stop_until = Some(Stage::Plan); + state.supervisor = Some(devflow_core::state::SupervisorRecord::new( + root.join("until.sock"), + "until00000000000".to_string(), + 4242, + 1, + String::new(), + )); + workflow::save_state(&state).unwrap(); + + transition(root, &mut state, Stage::Code).unwrap(); + + let reloaded = workflow::load_state(root, phase).unwrap(); + assert!(reloaded.stopped, "the --until stop marks the run stopped"); + assert_eq!( + reloaded.supervisor, None, + "the --until stop clears the supervisor record" + ); + } + #[test] fn terminal_merge_failure_reopens_actionable_gate_and_never_reports_finished() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/devflow-cli/src/pipeline_launch.rs b/crates/devflow-cli/src/pipeline_launch.rs index ec7e147f..60af8533 100644 --- a/crates/devflow-cli/src/pipeline_launch.rs +++ b/crates/devflow-cli/src/pipeline_launch.rs @@ -40,7 +40,9 @@ use devflow_core::outcome_policy::{self, Action}; use devflow_core::phase_id::PhaseId; use devflow_core::prompt; use devflow_core::stage::Stage; +use devflow_core::state::SupervisorRecord; use devflow_core::state::{AgentKind, ApprovedCheckpoint, CheckpointApproval, State}; +use devflow_core::supervisor::runtime::{AgentOutcome, OwnerPhase, SafePoint, StopToken}; use devflow_core::verify::CheckpointDeclaration; use devflow_core::{ agent_result, agents, canary, events, gsd_config, lock, mode, monitor, verify, workflow, @@ -573,14 +575,15 @@ where /// Abort a launch the canary refused, leaving no half-launched stage behind. /// -/// Clearing `monitor_pid` applies WR-04's rationale (see -/// [`spawn_agent_and_record`]) to an aborted launch: `transition()` has already +/// Clearing `supervisor` applies WR-04's rationale (see +/// [`record_failed_launch`]) to an aborted launch: `transition()` has already /// advanced `state.stage` and saved it by the time this runs, so a refusal that -/// left the PREVIOUS stage's monitor pid standing would make `liveness()` -/// report `Stuck → devflow resume` and send the operator after the wrong -/// problem entirely. The right remedy is the message below. +/// left the PREVIOUS stage's supervisor record standing would read as a run +/// gone while active and send the operator to `devflow resume`, after the +/// wrong problem entirely. With no record, the run reads not launched, and the +/// right remedy is the message below. fn refuse_launch(state: &mut State, message: String) -> Result<(), CliError> { - state.monitor_pid = None; + state.supervisor = None; workflow::save_state(state)?; Err(CliError::Message(message)) } @@ -900,101 +903,686 @@ impl Drop for AutoChainGuard { } } -/// The detached pipe-owning monitor's own process body (Phase 31): supervise -/// the child, then advance the stage machine exactly as the shell monitor's -/// `devflow advance` tail did. -/// -/// Runs in the `__monitor` process, never in the operator's CLI. -/// -/// `envs` is deliberately empty here — the adapter's extra env was applied to -/// THIS process by `spawn_monitor` and rides down by inheritance. That is -/// sufficient only because the sole adapter routed through the pipe-owning arm -/// (Claude) declares no extra env; see the note at `spawn_monitor`'s -/// `PipeOwning` arm before widening it. -#[allow( - clippy::too_many_arguments, - reason = "the hidden CLI dispatch supplies these independently parsed monitor arguments" -)] -pub(crate) fn run_monitor( +/// The `__monitor` subcommand's `--launch` value: which supervision shape the +/// supervisor wires the agent's stdio for. +#[derive(Debug, Clone, Copy, PartialEq, Eq, clap::ValueEnum)] +pub(crate) enum LaunchShape { + /// Stdin `/dev/null`, stdout to the phase capture file + /// (`MonitorLaunch::Legacy`). + Capture, + /// The supervisor owns both pipes and delivers the prompt as the first + /// turn (`MonitorLaunch::PipeOwning`). + Stream, +} + +/// One `__monitor` invocation, as `main` (or the test build's harness entry) +/// parsed it from the argv `monitor::spawn_monitor` built. +#[derive(Debug, Clone)] +pub(crate) struct MonitorInvocation { + /// Project root whose `.devflow/` files the supervisor writes. + pub(crate) project_root: PathBuf, + /// Phase whose stage machine to advance once the agent completes. + pub(crate) phase: PhaseId, + /// Stage the launch ran; passed to advance. + pub(crate) stage: Option, + /// The agent's working directory. + pub(crate) workdir: PathBuf, + /// Which coding agent this is. + pub(crate) agent: AgentKind, + /// The supervision shape. + pub(crate) launch: LaunchShape, + /// Where the supervisor publishes its socket. + pub(crate) socket: PathBuf, + /// The per-launch id the launcher waits to see in a ping reply. + pub(crate) launch_id: String, + /// Adapter environment variable NAMES to re-apply to the agent from this + /// process's environment (values never travel on argv, T-50-41). + pub(crate) keep_env: Vec, + /// The stream shape's prompt file. + pub(crate) prompt_file: Option, + /// The stream shape's idle timeout in seconds. + pub(crate) idle_timeout_secs: Option, + /// The agent's program and argv. + pub(crate) argv: Vec, +} + +/// The stop reason a run ended through its supervisor records. +pub(crate) const SUPERVISOR_STOP_REASON: &str = + "stopped through the supervisor (socket shutdown or SIGTERM/SIGINT)"; + +/// The stop reason an advance records when it stops at a safe point +/// (50-13, R-2). +pub(crate) const ADVANCE_STOP_REASON: &str = "stopped through the supervisor during advance"; + +/// The bound on the two waits a stop can put between two supervisors +/// (50-13, T-50-54): a stopped supervisor's wait for the per-phase lock +/// before it records its stop, and safe point 3's wait for an unpublished +/// next supervisor's final `stopped` line. `2 × TERMINATE_VERIFY_WAIT + 5 s`: +/// the next supervisor's kill grace plus margin. `stop`'s own wait for a +/// final line covers it (`commands::shutdown_final_timeout`). +pub(crate) const SUPERVISOR_STOP_WAIT: Duration = + Duration::from_secs(2 * devflow_core::agent::TERMINATE_VERIFY_WAIT.as_secs() + 5); + +/// What a supervisor's advance reads on this thread (50-13): its stop token, +/// for the advance's safe points, and the successor the advance recorded. +struct EngagedAdvance { + token: StopToken, + successor: Option, +} + +thread_local! { + /// Set only while [`StopTokenGuard`] is alive. Thread-local, not a + /// process-global static, so parallel in-crate tests never share it; an + /// advance run by anything other than a supervisor (the hidden + /// `devflow advance`, a test calling `advance_with` directly) sees `None` + /// and every safe point proceeds. + static ENGAGED_ADVANCE: std::cell::RefCell> = + const { std::cell::RefCell::new(None) }; +} + +/// Makes a supervisor's stop token visible to the safe points of the +/// advance running on this thread, for as long as the guard lives (RAII, +/// the [`AutoChainGuard`] shape). Dropping it clears the token and the +/// successor slot. +struct StopTokenGuard { + /// Keeps the guard on the thread whose slot it set. + _not_send: std::marker::PhantomData<*const ()>, +} + +impl StopTokenGuard { + fn engage(token: StopToken) -> Self { + ENGAGED_ADVANCE.with(|slot| { + *slot.borrow_mut() = Some(EngagedAdvance { + token, + successor: None, + }); + }); + Self { + _not_send: std::marker::PhantomData, + } + } +} + +impl Drop for StopTokenGuard { + fn drop(&mut self) { + ENGAGED_ADVANCE.with(|slot| slot.borrow_mut().take()); + } +} + +/// Run `f` on the engaged stop token, or return `None` when no supervisor's +/// advance is running on this thread. +fn with_stop_token(f: impl FnOnce(&StopToken) -> R) -> Option { + let token = ENGAGED_ADVANCE.with(|slot| slot.borrow().as_ref().map(|e| e.token.clone())); + token.map(|token| f(&token)) +} + +/// Remember `record` as the successor this advance published, so +/// [`run_monitor`] marks this supervisor handed off once the advance has +/// returned. A no-op when no supervisor's advance is engaged. +fn record_successor(record: SupervisorRecord) { + ENGAGED_ADVANCE.with(|slot| { + if let Some(engaged) = slot.borrow_mut().as_mut() { + engaged.successor = Some(record); + } + }); +} + +/// Take the successor [`record_successor`] stored, if any. +fn take_successor() -> Option { + ENGAGED_ADVANCE.with(|slot| { + slot.borrow_mut() + .as_mut() + .and_then(|engaged| engaged.successor.take()) + }) +} + +/// An advance safe point (50-13, R-2): when the engaged supervisor's stop +/// has been requested, record the stop in `state` and return `true`, so the +/// caller returns at once without launching anything (the per-phase lock +/// guard drops on the way out, and the supervisor's `finish` answers a +/// parked caller `stopped`). `false` with no stop requested, or outside a +/// supervisor's advance. +fn stopped_at_safe_point(state: &mut State, safe_point: u8) -> Result { + if with_stop_token(StopToken::safe_point) != Some(SafePoint::StopHere) { + return Ok(false); + } + record_advance_stop(state, safe_point)?; + Ok(true) +} + +/// The test half of an advance safe point, for a caller that must clear +/// something before it calls [`record_advance_stop`]: `true` when the +/// engaged supervisor's stop has been requested (the run is then marked +/// ended as a stop), `false` with none or outside a supervisor's advance. +pub(crate) fn stop_requested_at_safe_point() -> bool { + with_stop_token(StopToken::safe_point) == Some(SafePoint::StopHere) +} + +/// Persist a stop an advance honoured at `safe_point`: `stopped` with +/// [`ADVANCE_STOP_REASON`], and an `advance_stopped` event naming the point. +pub(crate) fn record_advance_stop(state: &mut State, safe_point: u8) -> Result<(), CliError> { + state.stopped = true; + state.stop_reason = Some(ADVANCE_STOP_REASON.to_string()); + workflow::save_state(state)?; + events::emit( + &state.project_root, + state.phase, + "advance_stopped", + serde_json::json!({ + "stage": state.stage.to_string(), + "safe_point": safe_point, + }), + ); + info!( + "phase {}: {ADVANCE_STOP_REASON} (safe point {safe_point})", + state.phase + ); + Ok(()) +} + +/// Safe point 3's shutdown of a next supervisor that was spawned and +/// answered its ping but is not published: ask it to stop through its own +/// socket, bounded by [`SUPERVISOR_STOP_WAIT`]. It records nothing itself +/// (the state does not name it), so this launcher's stop is the record. +fn shut_down_unpublished(record: &SupervisorRecord) { + match devflow_core::supervisor::request_shutdown(&record.socket_path, SUPERVISOR_STOP_WAIT) { + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) => info!( + "stopped the unpublished next supervisor at {}", + record.socket_path.display() + ), + Ok(other) => warn!( + "the unpublished next supervisor at {} answered {other:?} to shutdown", + record.socket_path.display() + ), + Err(err) => warn!( + "could not stop the unpublished next supervisor at {}: {err}", + record.socket_path.display() + ), + } +} + +/// How an advance's gate wait ended (safe point 4). +#[derive(Debug)] +pub(crate) enum GateWaitEnd { + /// The gate was answered. + Answered(devflow_core::gates::GateResponse), + /// No answer arrived within the timeout. + TimedOut, + /// The supervisor running this advance was stopped during the wait. + Stopped, +} + +/// An advance's gate wait (50-13, R-2, safe point 4). Inside a supervisor's +/// advance the run is `GateWait` for the wait and `Advancing` again after +/// it, and the poll is cancelled by the supervisor's stop token, so a +/// SIGTERM/SIGINT or socket stop ends the wait within about a second as +/// [`GateWaitEnd::Stopped`]. Anywhere else (the hidden `devflow advance`, a +/// foreground gate) it is `Gates::poll_response`, unchanged. +pub(crate) fn wait_for_gate_response( project_root: &Path, phase: PhaseId, - stage: Option, - workdir: &Path, - prompt_file: &Path, - idle_timeout_secs: u64, - agent: AgentKind, - argv: &[String], -) -> Result<(), CliError> { - let prompt = std::fs::read_to_string(prompt_file).map_err(|err| { - CliError::Message(format!( - "monitor could not read the prompt file {}: {err}", - prompt_file.display() - )) - })?; - let Some((program, args)) = argv.split_first() else { - return Err(CliError::Message( - "monitor was given no child program to supervise".to_string(), - )); + stage: Stage, + timeout_secs: u64, +) -> GateWaitEnd { + let Some(token) = with_stop_token(StopToken::clone) else { + return Gates::poll_response(project_root, phase, stage, timeout_secs) + .map_or(GateWaitEnd::TimedOut, GateWaitEnd::Answered); + }; + token.enter(OwnerPhase::GateWait); + let polled = Gates::poll_response_cancellable(project_root, phase, stage, timeout_secs, || { + token.stop_requested() + }); + token.enter(OwnerPhase::Advancing); + match polled { + Some(response) => GateWaitEnd::Answered(response), + None if token.safe_point() == SafePoint::StopHere => GateWaitEnd::Stopped, + None => GateWaitEnd::TimedOut, + } +} + +/// How long a [`test_hold`] waits for its release file. +#[cfg(test)] +const TEST_HOLD_WAIT: Duration = Duration::from_secs(30); + +/// Test-only hold point (50-13): inside a supervisor's own advance (a stop +/// token engaged), when `DEVFLOW_TEST_HOLD` is `:` for this +/// `point`, write `.held` and wait (bounded by [`TEST_HOLD_WAIT`]) until +/// `` exists, so a test can land a stop, a signal or a kill in a chosen +/// window. A launch made outside an advance (the test's own first launch) +/// never holds. Compiled out of non-test builds, together with every call. +#[cfg(test)] +pub(crate) fn test_hold(point: &str) { + if with_stop_token(|_| ()).is_none() { + return; + } + let Ok(hold) = std::env::var("DEVFLOW_TEST_HOLD") else { + return; }; + let Some((at, path)) = hold.split_once(':') else { + return; + }; + if at != point { + return; + } + let _ = std::fs::write(format!("{path}.held"), ""); + let release = Path::new(path); + let deadline = std::time::Instant::now() + TEST_HOLD_WAIT; + while !release.exists() && std::time::Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(20)); + } +} + +/// The `__monitor` process body (50-11): supervise the stage's agent under +/// an addressed supervisor (`monitor::run_supervised_stage`), then, only +/// when the agent completed on its own, advance the stage machine +/// in-process (D-02). A stopped run is recorded as stopped and never +/// advances. +/// +/// The handoff (50-13, R-1): the advance runs with this supervisor's stop +/// token engaged for its safe points and, when it launches the next stage, +/// records the next supervisor under the per-phase lock before it returns. +/// Only after it has returned (the lock released) is this supervisor marked +/// handed off, so every stop that reaches it during the advance is a +/// flagged `Advancing` stop (RV-04), and only then does `finish` unlink its +/// socket: the state never names a socket that is gone. +/// +/// `advance` is `true` on every production path; the test build's +/// supervise-only monitor harness entry (50-10) passes `false` so an +/// in-crate launch supervises its agent without moving the stage machine on. +/// +/// Runs in the detached `__monitor` process, never in the operator's CLI. +pub(crate) fn run_monitor(invocation: &MonitorInvocation, advance: bool) -> Result<(), CliError> { + let MonitorInvocation { + project_root, + phase, + stage, + workdir, + launch, + .. + } = invocation; + let (project_root, phase, stage) = (project_root.as_path(), *phase, *stage); + let spec = stage_spec(invocation)?; // D-01/D-06: hold GSD's `workflow._auto_chain_active` at the value this - // launch requires for as long as the child runs, and return it to `false` - // when this function returns — by `?` on the monitor's `Err` OR by falling - // through to `advance` below. Both exits are covered because the guard is - // bound to a named variable in THIS scope; `let _ = ...` would drop it - // immediately and the flag's true-window would collapse to nothing. + // launch requires for as long as the agent runs, and return it to + // `false` before anything else happens: on the `?` below when the + // supervisor fails, and right after the agent's run otherwise. // - // The target is `workdir`, not `project_root`: `.planning/config.json` is a - // tracked file inside the worktree the agent's cwd is set to, and that copy - // is the one GSD's `check auto-mode` reads. + // The target is `workdir`, not `project_root`: `.planning/config.json` is + // a tracked file inside the worktree the agent's cwd is set to, and that + // copy is the one GSD's `check auto-mode` reads. // - // F-4 — no agent or launch-shape condition belongs in the predicate. - // `run_monitor` is the body of the hidden `__monitor` subcommand, which - // `monitor::spawn_monitor` re-execs ONLY on its `MonitorLaunch::PipeOwning` - // arm. Being inside this function already implies a stream launch (Claude - // and Antigravity today, round-3 D-10) — the old "Claude + stream launch" - // claim became false when the predicate widened — so re-checking - // `state.agent` or `state.legacy_claude_launch` here would be a second, - // driftable notion of the same fact. The consequence — a Legacy-arm or - // non-stream-agent launch never gets the flag — is accepted and is turned - // into a loud preflight refusal by plan `35.1-03`, not left silent. + // Conditioned on the launch shape explicitly: since 50-11 both shapes + // reach this function, and only a stream launch can bound the flag's + // lifetime to the agent's (35.1-03's preflight refuses the capture shape + // in `Mode::Auto` for exactly that reason). A capture launch never gets + // the flag. + // + // The guard is dropped BEFORE the advance, not after it: the advance + // launches the next stage and waits for that stage's supervisor to + // publish, and that supervisor has already set the flag for its own + // agent. Clearing it after the advance would clear the next stage's + // value. // // A state that will not load is NOT fatal here: warn, skip the guard, and // let `advance` surface the real state error afterwards with its own // context. - let _auto_chain_guard = match workflow::load_state(project_root, phase) { - Ok(state) => Some(AutoChainGuard::engage( - workdir, - auto_chain_flag_eligible(state.stage, state.mode), - )), - Err(err) => { - warn!( - "monitor could not load state for phase {phase} ({err}) — running \ - without the GSD chain-flag guard" - ); - None + let auto_chain_guard = match launch { + LaunchShape::Capture => None, + LaunchShape::Stream => match workflow::load_state(project_root, phase) { + Ok(state) => Some(AutoChainGuard::engage( + workdir, + auto_chain_flag_eligible(state.stage, state.mode), + )), + Err(err) => { + warn!( + "monitor could not load state for phase {phase} ({err}) — running \ + without the GSD chain-flag guard" + ); + None + } + }, + }; + + let run = monitor::run_supervised_stage(spec) + .map_err(|err| CliError::Message(format!("stage supervisor failed: {err}")))?; + drop(auto_chain_guard); + + let result = match run.outcome { + AgentOutcome::Completed(_) if advance => { + let stop_guard = StopTokenGuard::engage(run.supervisor.token()); + let advanced = advance_for_stage(project_root, Some(phase), stage); + // The advance has returned and released the per-phase lock. Only + // a run whose advance published a successor is handed off. + if let Some(successor) = take_successor() { + let token = run.supervisor.token(); + #[cfg(test)] + test_hold("before_handoff_mark"); + token.enter(OwnerPhase::HandedOff); + // RV-04 / R5-I1 (50-23): a stop that reached this supervisor + // after safe point 3 was flagged (it stayed `Advancing`) and + // must not be dropped. The one owner mutex orders every stop + // request before the marking above (flagged, so seen here) or + // after it (answered `handed_off`, acting on nothing), so one + // check right after the marking drops no flagged stop. + if token.stop_requested() { + forward_stop_to_successor(&token, project_root, phase, &successor); + } + } + drop(stop_guard); + advanced + } + AgentOutcome::Completed(_) => { + wait_for_launch_record(project_root, phase, &invocation.launch_id); + Ok(()) + } + _ => { + record_supervisor_stop(project_root, phase, &invocation.launch_id); + Ok(()) } }; + // Never a `stopped` line for a completed run: `finish` decides the final + // line from the recorded outcome, so a shutdown that arrived while + // advancing is answered `handed_off`. + run.supervisor.finish(); + result +} - monitor::run_pipe_owning_monitor( +/// Forward a stop that reached this supervisor after safe point 3 to the +/// successor its advance recorded (50-23, RV-04), through the successor's +/// socket, bounded by [`SUPERVISOR_STOP_WAIT`]. Writes no state: the +/// successor is the recorded supervisor and records its own stop. +/// +/// A confirmed `Stopped` takes a safe point on `token`, which returns +/// `StopHere`, so `finish` answers a parked shutdown caller `stopped` (the +/// stop did happen, through the successor) and that caller does not also +/// retry against the successor. Anything else leaves `finish` answering +/// `handed_off`, so a socket caller retries against the state's record, and +/// names the `devflow stop --phase N --launch ` repair. Both outcomes +/// are recorded as a `stop_forwarded` event, because this process's stderr +/// is `/dev/null` in production. The stop's source is not consulted: the +/// token does not keep it (T-50-75). +fn forward_stop_to_successor( + token: &StopToken, + project_root: &Path, + phase: PhaseId, + successor: &SupervisorRecord, +) { + let launch_id = &successor.launch_id; + let outcome = + devflow_core::supervisor::request_shutdown(&successor.socket_path, SUPERVISOR_STOP_WAIT); + let confirmed = matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ); + let repair = format!("devflow stop --phase {phase} --launch {launch_id}"); + let detail = match &outcome { + Ok(outcome) => format!("{outcome:?}"), + Err(err) => err.to_string(), + }; + if confirmed { + let _ = token.safe_point(); + eprintln!("forwarded a stop received after safe point 3 to launch {launch_id}: stopped"); + } else { + eprintln!( + "a stop reached this supervisor after safe point 3 and forwarding it to launch \ + {launch_id} did not confirm a stop ({detail}); run `{repair}`" + ); + } + events::emit( project_root, phase, - workdir, - &prompt, - std::time::Duration::from_secs(idle_timeout_secs), - program, - args, - &[], - agent, - ) - .map_err(|err| CliError::Message(format!("pipe-owning monitor failed: {err}")))?; + "stop_forwarded", + serde_json::json!({ + "launch_id": launch_id, + "confirmed": confirmed, + "outcome": detail, + "repair": (!confirmed).then_some(repair), + }), + ); +} + +/// The [`monitor::StageSpec`] for `invocation`: the stream shape reads its +/// prompt file, and each `--keep-env` name takes its value from this +/// process's environment. +fn stage_spec(invocation: &MonitorInvocation) -> Result { + let Some((program, args)) = invocation.argv.split_first() else { + return Err(CliError::Message( + "monitor was given no child program to supervise".to_string(), + )); + }; + let io = match invocation.launch { + LaunchShape::Capture => monitor::StageIo::Capture, + LaunchShape::Stream => { + let (Some(prompt_file), Some(idle_secs)) = + (&invocation.prompt_file, invocation.idle_timeout_secs) + else { + return Err(CliError::Message( + "a stream launch needs --prompt-file and --idle-timeout-secs".to_string(), + )); + }; + let prompt = std::fs::read_to_string(prompt_file).map_err(|err| { + CliError::Message(format!( + "monitor could not read the prompt file {}: {err}", + prompt_file.display() + )) + })?; + monitor::StageIo::Stream { + prompt, + idle_timeout: Duration::from_secs(idle_secs), + } + } + }; + let socket = devflow_core::supervisor::SocketPath::new(invocation.socket.clone()) + .map_err(|err| CliError::Message(format!("monitor socket path refused: {err}")))?; + let mut spec = monitor::StageSpec::new( + invocation.project_root.clone(), + invocation.phase, + invocation.agent, + program.clone(), + io, + socket, + invocation.launch_id.clone(), + ); + spec.stage = invocation.stage; + spec.workdir = invocation.workdir.clone(); + spec.args = args.to_vec(); + spec.envs = kept_env(&invocation.keep_env); + Ok(spec) +} + +/// Each named variable with its value from this process's environment, +/// where `spawn_monitor` put it. A name that is unset or not UTF-8 is +/// skipped with a warning. +fn kept_env(names: &[String]) -> Vec<(String, String)> { + names + .iter() + .filter_map( + |name| match std::env::var_os(name).map(|value| value.into_string()) { + Some(Ok(value)) => Some((name.clone(), value)), + _ => { + warn!( + "--keep-env {name}: not set to a UTF-8 value in the monitor's environment" + ); + None + } + }, + ) + .collect() +} + +/// Record a run its supervisor stopped: `state.stopped = true` with +/// [`SUPERVISOR_STOP_REASON`], under the per-phase lock. Never advances. +/// +/// Only when the state names THIS launch (50-13, T-50-54): checked without +/// the lock first, then again under it. A next supervisor that was never +/// published (safe point 3 stops it while its launcher holds the lock and +/// records the stop itself) and a superseded supervisor (the state names its +/// successor, whose run it must not mark stopped) both write nothing, so +/// neither waits on a lock its stopper holds. +/// +/// Never fails (T-50-68): a state that will not load, a lock wait that +/// times out or a save that fails is logged, naming the state path and why +/// `stopped` was not recorded, and the caller still reaches `finish`, which +/// answers a parked stop caller `stopped`. +fn record_supervisor_stop(project_root: &Path, phase: PhaseId, launch_id: &str) { + let path = workflow::state_path(project_root, phase); + let names_this_launch = |state: &State| { + state + .supervisor + .as_ref() + .is_some_and(|record| record.launch_id == launch_id) + }; + let not_recorded = |why: &str| { + warn!( + "phase {phase}: `stopped` was not recorded in {}: {why}", + path.display() + ); + }; + match workflow::load_state(project_root, phase) { + Err(err) => return not_recorded(&format!("the state does not load ({err})")), + Ok(state) if !names_this_launch(&state) => { + info!( + "phase {phase}: the state does not name launch {launch_id}; its stop is \ + not this supervisor's to record" + ); + return; + } + Ok(_) => {} + } + let _lock = match lock::acquire_blocking(project_root, phase, SUPERVISOR_STOP_WAIT) { + Ok(guard) => guard, + Err(err) => return not_recorded(&format!("the per-phase lock was not acquired ({err})")), + }; + let mut state = match workflow::load_state(project_root, phase) { + Ok(state) if names_this_launch(&state) => state, + Ok(_) => { + info!("phase {phase}: launch {launch_id} was superseded before its stop was recorded"); + return; + } + Err(err) => return not_recorded(&format!("the state does not load ({err})")), + }; + state.stopped = true; + state.stop_reason = Some(SUPERVISOR_STOP_REASON.to_string()); + if let Err(err) = workflow::save_state(&state) { + return not_recorded(&format!("the save failed ({err})")); + } + info!("phase {phase}: {SUPERVISOR_STOP_REASON}"); +} + +/// How long the supervise-only entry waits for its launcher's record. +const LAUNCH_RECORD_WAIT: Duration = Duration::from_secs(15); + +/// The supervise-only path (test build): keep serving until the launcher has +/// recorded THIS launch in the state file (bounded), so a launcher still +/// waiting for the publish cannot miss a socket that finished first. A +/// production supervisor gets the same ordering from `advance`, which takes +/// the per-phase lock its launcher holds until after the publish. +fn wait_for_launch_record(project_root: &Path, phase: PhaseId, launch_id: &str) { + let deadline = std::time::Instant::now() + LAUNCH_RECORD_WAIT; + while std::time::Instant::now() < deadline { + let recorded = workflow::load_state(project_root, phase) + .ok() + .and_then(|state| state.supervisor) + .is_some_and(|record| record.launch_id == launch_id); + if recorded { + return; + } + std::thread::sleep(Duration::from_millis(20)); + } + warn!("launch {launch_id} was not recorded within {LAUNCH_RECORD_WAIT:?}"); +} + +/// Resolve, once per launch, the cache directory and the monitor executable +/// a stage launch uses (50-10). +/// +/// The released binary always re-execs itself as `__monitor` and registers +/// under [`devflow_core::registry::cache_dir`]. There is no environment +/// override of either: the test-build branch below is selected at compile +/// time. +#[cfg(not(test))] +pub(crate) fn launch_context() -> Result { + let cache_dir = devflow_core::registry::cache_dir().ok_or_else(|| { + CliError::Message( + "cannot resolve DevFlow's cache directory for this launch: set HOME, \ + XDG_CACHE_HOME or DEVFLOW_CACHE_DIR" + .to_string(), + ) + })?; + let exe = std::env::current_exe().map_err(|err| { + CliError::Message(format!( + "could not resolve the devflow executable for the stage monitor: {err}" + )) + })?; + Ok(monitor::LaunchContext::new( + cache_dir, + monitor::MonitorExec::Subcommand { exe }, + )) +} + +/// Selects the advance-running monitor harness entry for a `cfg(test)` stage +/// launch when set to `1` (50-10). Read only by the test build of +/// [`launch_context`]. +#[cfg(test)] +pub(crate) const TEST_MONITOR_FULL_ENV: &str = "DEVFLOW_TEST_MONITOR_FULL"; + +/// The test build of [`launch_context`] (50-10, D-07). +/// +/// Inside this crate's unit-test binary, `current_exe()` is libtest, which +/// rejects `__monitor`'s arguments, so the monitor is re-entered at an +/// `#[ignore]`d harness entry in `test_support` that runs the same +/// `run_monitor` body. The cache dir must come from the test process's own +/// `DEVFLOW_CACHE_DIR`, which only a child test process given one through +/// `run_test_in_child`'s `extra_env` has: an in-crate test that launches a +/// stage any other way fails here, loudly, instead of registering under the +/// developer's real HOME. +#[cfg(test)] +pub(crate) fn launch_context() -> Result { + let cache_dir = std::env::var_os("DEVFLOW_CACHE_DIR") + .map(PathBuf::from) + .ok_or_else(|| { + CliError::Message( + "in-crate test launched a stage without a hermetic DEVFLOW_CACHE_DIR; \ + run it through run_test_in_child with extra_env" + .to_string(), + ) + })?; + let exe = std::env::current_exe() + .map_err(|err| CliError::Message(format!("could not resolve the test binary: {err}")))?; + let test_name = if std::env::var_os(TEST_MONITOR_FULL_ENV).is_some_and(|value| value == "1") { + "test_support::monitor_entry_full" + } else { + "test_support::monitor_entry_supervise_only" + }; + Ok(monitor::LaunchContext::new( + cache_dir, + monitor::MonitorExec::HarnessEntry { + exe, + test_name: test_name.to_string(), + }, + )) +} - advance_for_stage(project_root, Some(phase), stage) +/// End a launch that did not happen (50-23, T-50-62): clear +/// `state.supervisor` and save. +/// +/// WR-04's rationale, moved from the monitor pid to the socket record: +/// `transition()` has already advanced and saved `state.stage` by the time a +/// launch runs, so a record still naming the previous stage's supervisor, +/// which is ending or gone, reads GONE-while-active and sends the operator to +/// `devflow resume`, the wrong remedy. With no record the run reads not +/// launched, and the launch's own error message is the remedy. +/// +/// Called only on a failure path, never before the save that records a new +/// record (50-13's rule). +fn record_failed_launch(state: &mut State) -> Result<(), CliError> { + state.supervisor = None; + workflow::save_state(state)?; + Ok(()) } -/// The tail of [`launch_stage_inner`]: clear the stale monitor pid, validate -/// the agent binary, archive the prior capture, spawn the monitor, and -/// record the launch. Extracted (28-03, Task 2) so +/// The tail of [`launch_stage_inner`]: validate the agent binary, archive +/// the prior capture, spawn the monitor, and record the launch. Extracted (28-03, Task 2) so /// [`relaunch_checkpoint_session`] can share this EXACT tail for a /// checkpoint auto-decide resume — a resume is a continuation of the same /// stage's agent run, not a fresh stage entry, so it must not duplicate this @@ -1012,60 +1600,57 @@ fn spawn_agent_and_record( archived_stage: Option, launch: monitor::MonitorLaunch, ) -> Result<(), CliError> { - // WR-04 (18-fix): clear the prior stage's monitor pid up front, before - // any fallible step below (`ensure_agent_binary`) can return early via - // `?`. Without this, a failed relaunch left `state.stage` already - // advanced (by `transition()`, before this function was ever called) - // alongside a stale `monitor_pid` still naming the PREVIOUS stage's - // (now-dead) monitor — `liveness()` then misreports `Stuck → devflow - // resume`, even when the real remedy is unrelated. The real pid is set - // again below once `monitor::spawn_monitor` actually succeeds. - state.monitor_pid = None; + // Persist the caller's launch bookkeeping (`launch_stage_inner`'s + // `checkpoint_resumes` reset and Validate dispatch stamp) before + // anything is spawned, as this save always has. It once also cleared + // the previous stage's monitor pid (WR-04, 18-fix); that field is gone + // from the CLI (50-22), and the record keeps its own rule below. workflow::save_state(state)?; - ensure_agent_binary(program)?; - - // 17d (Task 2, D-17-D-19) originally placed the self-dogfood - // build-staleness gate here, before spawn_monitor. 25b (D-03) moved it - // out: it is now adjudicated exactly once, in `commands::start`, after - // `state.worktree_path` is set and before this whole launch path is ever - // entered — so a phase that modifies DevFlow's own source can progress - // past every stage boundary instead of being re-blocked at each one. See - // `commands::start` for the new call site and its accompanying D-04/D-05 - // trade-off notes. - if let Some(stamp) = agent_result::archive_phase_files( - &state.project_root, - state - .worktree_path - .as_deref() - .unwrap_or(&state.project_root), - state.phase, - capture_retention(&state.project_root), - ) - .map_err(|err| { - CliError::Message(format!( - "could not archive phase {} capture before rollover: {err}", - state.phase - )) - })? { - events::emit( - &state.project_root, - state.phase, - "capture_archived", - serde_json::json!({ - "stage": archived_stage.unwrap_or(state.stage).to_string(), - "to_stage": state.stage.to_string(), - "stamp": stamp, - }), - ); - } - let pid = monitor::spawn_monitor(state, program, args, extra_env, launch) - .map_err(|err| CliError::Message(format!("could not spawn monitor: {err}")))?; + // WR-04 (18-fix, 50-23 T-50-62): every error before the publish save + // ends the launch with no supervisor record (see `record_failed_launch`), + // so a failed relaunch never leaves `state.stage` advanced beside a + // record naming the previous stage's supervisor; the success path below + // never clears the record before the save that replaces it. + let SpawnedLaunch { + published, + ctx, + launch_label, + } = match spawn_next_supervisor(state, program, args, extra_env, archived_stage, launch) { + Ok(Some(spawned)) => spawned, + // A stop honoured at safe point 2 or 3 (50-13), already recorded. + Ok(None) => return Ok(()), + Err(err) => { + return Err(match record_failed_launch(state) { + Ok(()) => err, + Err(clear_err) => CliError::Message(format!( + "{err}; clearing the supervisor record after it also failed: {clear_err}" + )), + }); + } + }; + let pid = published.supervisor_pid; + let socket_path = published.record.socket_path.display().to_string(); + let launch_id = published.record.launch_id.clone(); // `transition()` calls `workflow::save_state` BEFORE `launch_stage`, so a - // pid recorded only in memory here is lost unless it is written again - // (18b). - state.monitor_pid = Some(pid); - workflow::save_state(state)?; + // record held only in memory here is lost unless it is written again + // (18b). The ping-confirmed supervisor record replaces any previous one + // in this same save; it is never cleared first. + // + // The handoff (50-13, R-1): this save is the publish. The previous + // supervisor, when this launch is its advance, stays alive and + // `Advancing` through it (RV-04: never handed off here) and only + // unlinks its own socket after the advance has returned, so a reader + // between the two always finds a live address. + #[cfg(test)] + test_hold("before_record_save"); + let record = published.record.clone(); + match publish_record(state, published, workflow::save_state)? { + PublishOutcome::HandedOff => record_successor(record), + PublishOutcome::SuccessorAlreadyEnded => {} + } + #[cfg(test)] + test_hold("after_publish"); match devflow_core::ship::consume_cron_instructions(&state.project_root, state.phase) { Ok(Some(path_kind)) => { let path_kind = match path_kind { @@ -1087,10 +1672,10 @@ fn spawn_agent_and_record( Err(err) => warn!(phase = %state.phase, "could not consume cron instructions: {err}"), } // 23b: register this (project_root, phase) in the machine-global - // registry on the same code path that just recorded monitor_pid, so a + // registry on the same code path that just recorded the supervisor, so a // running phase cannot be missing from `devflow gate list --all-roots`. // Best-effort observability — never a reason to fail a launch. - let _ = devflow_core::registry::register(&state.project_root, state.phase); + let _ = devflow_core::registry::register_in(&ctx.cache_dir, &state.project_root, state.phase); events::emit( &state.project_root, state.phase, @@ -1098,39 +1683,277 @@ fn spawn_agent_and_record( serde_json::json!({ "stage": state.stage.to_string(), "agent": state.agent.to_string(), - "monitor_pid": pid, + "supervisor_pid": pid, + "socket_path": socket_path, + "launch_id": launch_id, + "launch": launch_label, }), ); println!( - "stage {} → launched {} (monitor pid {pid})", + "stage {} → launched {} (supervisor pid {pid}, {launch_label} supervisor at {socket_path})", state.stage, agents::driver_for(state.agent).name() ); Ok(()) } -/// Render the re-scan gate's context: which plan files carry a human-only -/// checkpoint DevFlow did not record at this phase's Code preflight. +/// The stop reason a launcher records when the next stage's supervisor it +/// just published had already ended (50-23, T-50-64). +pub(crate) const NEXT_STOPPED_BEFORE_SAVE_REASON: &str = + "the next stage's supervisor was stopped before its record was saved"; + +/// How long [`successor_runs_on`] keeps re-probing a STALE successor: a +/// finishing supervisor drops its listener just before it unlinks. +const STALE_REPROBE_WINDOW: Duration = Duration::from_millis(200); + +/// The interval of [`successor_runs_on`]'s STALE re-probe. +const STALE_REPROBE_INTERVAL: Duration = Duration::from_millis(20); + +/// How [`publish_record`] ended. +enum PublishOutcome { + /// The record is saved and the next supervisor is running it. + HandedOff, + /// The record is saved, but the next supervisor had already ended, so + /// the run is recorded `stopped`. + SuccessorAlreadyEnded, +} + +/// Record the next stage's supervisor (`published`) in `state` with one +/// `save`, the publish (50-13, R-1 step 4; 50-23). /// -/// Names the plan FILES rather than quoting the task bodies. A task element -/// can be arbitrarily long and the gate context is truncated for the notify -/// payload, so quoting bodies would push the actionable part — which file to -/// look at — past the cut. -fn rescan_gate_context(unapproved: &[&CheckpointDeclaration], mode: Mode) -> String { - let mut plan_files: Vec<&str> = unapproved - .iter() - .map(|declaration| declaration.plan_file.as_str()) - .collect(); - plan_files.sort_unstable(); - plan_files.dedup(); - let next_step = match mode { - Mode::Auto => { - "approve, or switch this phase to Supervise for its remaining stages to repair, or abort" - } - Mode::Supervise => { - "approve then make a separate human Code decision, repair under Supervise, or abort" - } - }; +/// - The save fails: shut the new supervisor down through its socket +/// (bounded by [`SUPERVISOR_STOP_WAIT`]), so no live supervisor is left +/// unrecorded, and return the error ([`shut_down_unrecorded`]). +/// - The save succeeds: probe the new socket once ([`successor_runs_on`]). +/// A successor still running is handed off. One that already ended (a +/// stop sent straight to it takes 50-13's no-write path while the state +/// still names this launcher's run) is recorded here: `stopped` with +/// [`NEXT_STOPPED_BEFORE_SAVE_REASON`], saved again under the per-phase +/// lock the caller holds. +/// +/// A successor that has accepted a stop but is still in its kill grace +/// answers owner `agent_running` and is handed off: this save lands before +/// that probe, and it loads the state only after its outcome is decided, so +/// it finds its own launch id and records its own stop. +/// +/// Signals nothing by pid: every step goes through `published`'s socket. +fn publish_record>( + state: &mut State, + published: monitor::Published, + mut save: impl FnMut(&State) -> Result<(), E>, +) -> Result { + let record = published.record; + state.supervisor = Some(record.clone()); + if let Err(err) = save(state) { + return Err(shut_down_unrecorded(state, &record, err.into())); + } + if successor_runs_on(&record.socket_path) { + return Ok(PublishOutcome::HandedOff); + } + state.stopped = true; + state.stop_reason = Some(NEXT_STOPPED_BEFORE_SAVE_REASON.to_string()); + save(state).map_err(Into::into)?; + info!( + "phase {}: {NEXT_STOPPED_BEFORE_SAVE_REASON} (launch {})", + state.phase, record.launch_id + ); + Ok(PublishOutcome::SuccessorAlreadyEnded) +} + +/// After a failed publish save (`save_err`): ask the unrecorded supervisor +/// at `record` to stop, and return the launch's error. +/// +/// A confirmed `Stopped` clears the record (best effort) and the error says +/// the supervisor was shut down. Anything else (RV-08: `HandedOff`, +/// `Refused`, a probe error or a timeout) means it may still be running: +/// nothing is cleared or saved, so the in-memory record keeps naming it and +/// no save erases the handle the file holds, and the error names its socket, +/// its launch id and the `devflow stop --phase N --launch ` repair, which +/// finds it by its sidecar whatever the state says (50-14). +fn shut_down_unrecorded( + state: &mut State, + record: &SupervisorRecord, + save_err: CliError, +) -> CliError { + let failed = format!( + "could not save the record of the next stage's supervisor (launch {}): {save_err}", + record.launch_id + ); + match devflow_core::supervisor::request_shutdown(&record.socket_path, SUPERVISOR_STOP_WAIT) { + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) => { + let cleared = record_failed_launch(state) + .err() + .map(|err| format!("; clearing its record also failed: {err}")) + .unwrap_or_default(); + CliError::Message(format!( + "{failed}; that supervisor was shut down through its socket{cleared}" + )) + } + other => { + let outcome = match other { + Ok(outcome) => format!("{outcome:?}"), + Err(err) => err.to_string(), + }; + CliError::Message(format!( + "{failed}; shutting it down at {} did not confirm a stop ({outcome}), so it may \ + still be running; run `devflow stop --phase {} --launch {}` to end it", + record.socket_path.display(), + state.phase, + record.launch_id + )) + } + } +} + +/// Whether the successor whose record was just saved is still running and +/// can be handed off: Alive with owner `agent_running`, `advancing` or +/// `gate_wait`. Gone, or Alive with any other owner (it finished without +/// advancing), is `false`. STALE is re-probed for +/// [`STALE_REPROBE_WINDOW`]: Gone within it is `false`, and a socket still +/// STALE is `true` (a crashed successor stays recorded and reads STALE, a +/// real finding with its own repair). A probe error is `true`: the record +/// stays and `status` reports the error. +fn successor_runs_on(socket: &Path) -> bool { + use devflow_core::supervisor::{Liveness, probe}; + let running = [ + OwnerPhase::AgentRunning, + OwnerPhase::Advancing, + OwnerPhase::GateWait, + ]; + let deadline = std::time::Instant::now() + STALE_REPROBE_WINDOW; + loop { + match probe(socket) { + Ok(Liveness::Alive(reply)) => { + return running.iter().any(|phase| phase.as_str() == reply.owner); + } + Ok(Liveness::Gone) => return false, + Ok(Liveness::Stale) if std::time::Instant::now() < deadline => { + std::thread::sleep(STALE_REPROBE_INTERVAL); + } + Ok(Liveness::Stale) => return true, + Err(err) => { + warn!( + "could not probe the next supervisor at {} after its publish: {err}", + socket.display() + ); + return true; + } + } + } +} + +/// What [`spawn_next_supervisor`] hands back to [`spawn_agent_and_record`] +/// for the publish: the ping-confirmed supervisor, the launch context it +/// was spawned under, and the launch shape's label. +struct SpawnedLaunch { + published: monitor::Published, + ctx: monitor::LaunchContext, + launch_label: &'static str, +} + +/// The fallible steps of [`spawn_agent_and_record`] before the publish save: +/// validate the agent binary, archive the prior capture, resolve the launch +/// context, and spawn the next supervisor, with 50-13's safe points 2 and 3 +/// around the spawn. `Ok(None)` when a stop was honoured at one of them (the +/// stop is already recorded). Every `Err` is a launch that did not happen, +/// and its caller clears the supervisor record ([`record_failed_launch`]). +fn spawn_next_supervisor( + state: &mut State, + program: &str, + args: &[String], + extra_env: &[(String, String)], + archived_stage: Option, + launch: monitor::MonitorLaunch, +) -> Result, CliError> { + ensure_agent_binary(program)?; + + // 17d (Task 2, D-17-D-19) originally placed the self-dogfood + // build-staleness gate here, before spawn_monitor. 25b (D-03) moved it + // out: it is now adjudicated exactly once, in `commands::start`, after + // `state.worktree_path` is set and before this whole launch path is ever + // entered — so a phase that modifies DevFlow's own source can progress + // past every stage boundary instead of being re-blocked at each one. See + // `commands::start` for the new call site and its accompanying D-04/D-05 + // trade-off notes. + if let Some(stamp) = agent_result::archive_phase_files( + &state.project_root, + state + .worktree_path + .as_deref() + .unwrap_or(&state.project_root), + state.phase, + capture_retention(&state.project_root), + ) + .map_err(|err| { + CliError::Message(format!( + "could not archive phase {} capture before rollover: {err}", + state.phase + )) + })? { + events::emit( + &state.project_root, + state.phase, + "capture_archived", + serde_json::json!({ + "stage": archived_stage.unwrap_or(state.stage).to_string(), + "to_stage": state.stage.to_string(), + "stamp": stamp, + }), + ); + } + let ctx = launch_context()?; + let launch_label = match &launch { + monitor::MonitorLaunch::PipeOwning { .. } => "stream", + monitor::MonitorLaunch::Legacy => "capture", + }; + #[cfg(test)] + test_hold("before_spawn"); + // Safe point 2 (50-13): a stop taken during the advance ends it before + // the next stage's supervisor is spawned. + if stopped_at_safe_point(state, 2)? { + return Ok(None); + } + let published = monitor::spawn_monitor(state, program, args, extra_env, launch, &ctx) + .map_err(|err| CliError::Message(format!("could not spawn monitor: {err}")))?; + #[cfg(test)] + test_hold("before_publish"); + // Safe point 3 (50-13): the next supervisor answered its ping but is not + // published. On a stop, shut it down through its socket and never save + // its record; the state keeps this run's record, marked stopped. + if with_stop_token(StopToken::safe_point) == Some(SafePoint::StopHere) { + shut_down_unpublished(&published.record); + record_advance_stop(state, 3)?; + return Ok(None); + } + Ok(Some(SpawnedLaunch { + published, + ctx, + launch_label, + })) +} + +/// Render the re-scan gate's context: which plan files carry a human-only +/// checkpoint DevFlow did not record at this phase's Code preflight. +/// +/// Names the plan FILES rather than quoting the task bodies. A task element +/// can be arbitrarily long and the gate context is truncated for the notify +/// payload, so quoting bodies would push the actionable part — which file to +/// look at — past the cut. +fn rescan_gate_context(unapproved: &[&CheckpointDeclaration], mode: Mode) -> String { + let mut plan_files: Vec<&str> = unapproved + .iter() + .map(|declaration| declaration.plan_file.as_str()) + .collect(); + plan_files.sort_unstable(); + plan_files.dedup(); + let next_step = match mode { + Mode::Auto => { + "approve, or switch this phase to Supervise for its remaining stages to repair, or abort" + } + Mode::Supervise => { + "approve then make a separate human Code decision, repair under Supervise, or abort" + } + }; format!( "[checkpoint re-scan] {} human-only checkpoint(s) in {} were added or changed after this \ phase's Code preflight — a human must review them before continuing ({next_step})", @@ -1652,6 +2475,31 @@ pub(crate) fn advance_with( // different file. A duplicate advance of THIS phase queues behind the // lock, then stage binding below handles a launch that has moved on. let mut state = workflow::load_state(project_root, phase)?; + // Safe point 1 (50-13): a stop taken while this supervisor waited for + // the lock ends the advance before it evaluates anything. + if stopped_at_safe_point(&mut state, 1)? { + return Ok(()); + } + #[cfg(test)] + test_hold("after_safe_point_1"); + // 999.138 residual: a stopped run never advances. `resume` clears + // `stopped` before it launches, so no caller advances a stopped state + // legitimately. + if state.stopped { + events::emit( + project_root, + phase, + "advance_failed", + serde_json::json!({ + "reason": "stopped", + "stop_reason": state.stop_reason, + }), + ); + return Err(CliError::Message(format!( + "phase {phase} is stopped ({}); run `devflow resume --phase {phase}` to continue it", + state.stop_reason.as_deref().unwrap_or("no reason recorded") + ))); + } // CR-03 (49-REVIEW.md): a hand-run `advance` updates the run's notify // command; the detached chain carries no variable, so its `advance` // keeps the persisted one. @@ -1975,7 +2823,7 @@ mod tests { if !devflow_core::test_support::in_child_test($name) { let path_dir = agent_free_dir_with_agent_stub($program); let output = - devflow_core::test_support::run_test_in_child($name, path_dir.path(), &[]); + crate::test_support::run_hermetic_test_in_child($name, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test( &output, $name, ); @@ -2116,26 +2964,28 @@ mod tests { } /// 18b: after `launch_stage` spawns a monitor, the persisted state file - /// for that phase carries the monitor's pid — `transition()` saves state - /// BEFORE calling `launch_stage`, so the pid must be saved again inside - /// `launch_stage` or it is lost. + /// for that phase carries the published supervisor record — `transition()` + /// saves state BEFORE calling `launch_stage`, so the record must be saved + /// again inside `launch_stage` or it is lost. /// /// **Premise moved off `STREAM_JSON_STAGES` membership deliberately /// (34-06); do not "simplify" the opt-out away.** This test's subject is - /// pid persistence, not which launch path runs, so it previously relied on + /// record persistence, not which launch path runs, so it previously relied on /// its stage being ABSENT from `STREAM_JSON_STAGES` — an incidental /// premise that 34-05's widening destroys, at which point `canary_gate` /// invokes the real `ClaudeCanaryLauncher` and the launch fails on a - /// delivery refusal that has nothing to do with pid persistence. Pinning + /// delivery refusal that has nothing to do with record persistence. Pinning /// the legacy path via the opt-out makes the premise explicit and stable /// under any contents of the constant, exactly as /// `canary_gate_only_applies_to_the_stream_launch_path` does. #[test] - fn launch_stage_persists_monitor_pid_for_reload() { - const NAME: &str = "pipeline_launch::tests::launch_stage_persists_monitor_pid_for_reload"; + fn launch_stage_persists_the_supervisor_record_for_reload() { + const NAME: &str = + "pipeline_launch::tests::launch_stage_persists_the_supervisor_record_for_reload"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2150,7 +3000,7 @@ mod tests { // legacy opt-out set below makes this a launch shape whose Code stage // cannot bound the chain flag's lifetime, and // `preflight_unattended_launch_check` refuses exactly that combination - // in `Mode::Auto`. This test's subject — pid persistence — is + // in `Mode::Auto`. This test's subject — record persistence — is // mode-independent, so supervise keeps the premise the 34-06 note // below establishes without asking the preflight to permit a launch // D-07 exists to refuse. @@ -2172,153 +3022,667 @@ mod tests { // reap it before `dir` drops below. Bound here — this test's LAST // `&mut state` use — the guard outranks every panicking checkpoint // that follows: `result.unwrap()`, the `assert!` on - // `state.monitor_pid.is_some()`, `workflow::load_state(...).unwrap()`, - // and the `assert_eq!` on the reloaded pid. Binding here also covers + // `state.supervisor.is_some()`, `workflow::load_state(...).unwrap()`, + // and the `assert_eq!` on the reloaded record. Binding here also covers // the narrower case of a launch that spawned the monitor and then // failed a later `?` inside `launch_stage_inner` — `result` would be - // `Err` but the pid would nonetheless be live (G-25-2, 25-17). + // `Err` but the supervisor would nonetheless be live (G-25-2, 25-17). let _reap_guard = ReapMonitorOnDrop::after_launch(&state); result.unwrap(); assert!( - state.monitor_pid.is_some(), - "launch_stage must record the monitor pid on the in-memory state" + state.supervisor.is_some(), + "launch_stage must record the published supervisor on the in-memory state" ); let reloaded = workflow::load_state(root, phase).unwrap(); assert_eq!( - reloaded.monitor_pid, state.monitor_pid, - "the monitor pid recorded by launch_stage must be persisted to disk, \ + reloaded.supervisor, state.supervisor, + "the supervisor record published by launch_stage must be persisted to disk, \ since transition() saves state before launch_stage runs" ); } - /// 20c (review: Codex MEDIUM — resume semantics): a phase halted by - /// `--until ` has `stopped`/`stop_reason`/`stop_until` persisted. - /// `resume` must clear all three BEFORE relaunching — otherwise - /// `transition()`'s `stop_until == Some(from)` check would immediately - /// re-stop the phase the next time it advances, and the phase would - /// stay marked `stopped` forever despite the operator's explicit - /// resume. Asserts on the persisted state (not just `resume`'s exit - /// code), since `transition()` saves state before `launch_stage` runs. + + /// Poll `done` every 50 ms until it holds or `limit` passes; returns its + /// last answer. + fn poll_until(limit: Duration, mut done: impl FnMut() -> bool) -> bool { + let deadline = std::time::Instant::now() + limit; + loop { + if done() { + return true; + } + if std::time::Instant::now() >= deadline { + return false; + } + std::thread::sleep(Duration::from_millis(50)); + } + } + + /// Names the file a V-07 child writes its recorded socket path to. + const V07_OUT_ENV: &str = "DEVFLOW_TEST_V07_OUT"; + + /// A `claude` stand-in that answers `--version`, then waits for `go` in + /// its working directory (the project root) and exits 0, with `sleep` + /// linked beside it so the wait works under the replaced PATH. + fn go_waiting_claude_dir() -> tempfile::TempDir { + let path_dir = agent_free_dir_with_agent_stub("claude"); + std::fs::write( + path_dir.path().join("claude"), + "#!/bin/sh\ncase \"$1\" in --version) exit 0 ;; esac\n\ + while [ ! -e go ]; do sleep 0.05; done\nexit 0\n", + ) + .unwrap(); + let real_sleep = ["/usr/bin/sleep", "/bin/sleep"] + .into_iter() + .map(Path::new) + .find(|path| path.is_file()) + .expect("sleep must exist to run this test"); + std::os::unix::fs::symlink(real_sleep, path_dir.path().join("sleep")).unwrap(); + path_dir + } + + /// V-07 (criterion 3): a launch persists `state.supervisor.socket_path` + /// under its context's cache dir (`DEVFLOW_CACHE_DIR/sock/`), never under + /// the project root, and the reloaded state holds exactly the published + /// record. Control: the same launch in a child with another cache dir + /// records a path under THAT dir, so the path follows the context. /// - /// **Premise moved off `STREAM_JSON_STAGES` membership deliberately - /// (34-06); do not "simplify" the opt-out away.** The subject is resume's - /// stop-marker semantics, not which launch path the relaunch takes. The - /// opt-out is persisted here rather than set on a local binding because - /// `resume()` loads its own `State` from disk; `apply_legacy_launch_opt_out` - /// ORs the persisted value, so it survives the reload. + /// The first child also runs the advance-running harness entry + /// (`test_support::monitor_entry_full`, `DEVFLOW_TEST_MONITOR_FULL=1`) and + /// proves it is reached: with the state moved on to Plan while the agent + /// runs, its in-process advance records `advance_failed` naming the + /// launched stage (`define`). The second child, on the supervise-only + /// entry, must record no advance event at all. #[test] - fn resume_clears_stop_marker_and_advances_past_stop_point() { + fn launch_publishes_the_socket_path_under_the_cache_dir_in_state() { const NAME: &str = - "pipeline_launch::tests::resume_clears_stop_marker_and_advances_past_stop_point"; + "pipeline_launch::tests::launch_publishes_the_socket_path_under_the_cache_dir_in_state"; if !devflow_core::test_support::in_child_test(NAME) { - let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); - devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); + let mut recorded = Vec::new(); + for full in [true, false] { + let path_dir = go_waiting_claude_dir(); + let cache = crate::test_support::hermetic_cache_dir(); + let out = path_dir.path().join("v07-socket-path"); + let mut env: Vec<(&str, &std::ffi::OsStr)> = vec![ + ("DEVFLOW_CACHE_DIR", cache.path().as_os_str()), + (V07_OUT_ENV, out.as_os_str()), + ]; + if full { + env.push((TEST_MONITOR_FULL_ENV, std::ffi::OsStr::new("1"))); + } + let output = + devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &env); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test( + &output, NAME, + ); + let socket = PathBuf::from(std::fs::read_to_string(&out).unwrap()); + assert!( + socket.starts_with(cache.path().join("sock")), + "{} must be under {}", + socket.display(), + cache.path().display() + ); + recorded.push((cache, socket)); + } + let (a, b) = (&recorded[0], &recorded[1]); + assert_ne!(a.0.path(), b.0.path(), "two distinct cache dirs"); + assert!( + !a.1.starts_with(b.0.path()) && !b.1.starts_with(a.0.path()), + "control: each launch's socket follows its own cache dir: {} and {}", + a.1.display(), + b.1.display() + ); return; } let _guard = env_lock(); + let cache = PathBuf::from( + std::env::var_os("DEVFLOW_CACHE_DIR").expect("the parent passes DEVFLOW_CACHE_DIR"), + ); + let out = + PathBuf::from(std::env::var_os(V07_OUT_ENV).expect("the parent passes the out file")); + let full = std::env::var_os(TEST_MONITOR_FULL_ENV).is_some_and(|value| value == "1"); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); init_repo(root); - - let phase = PhaseId::new(66); - let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); - state.stage = Stage::Plan; - state.stop_until = Some(Stage::Plan); - state.stopped = true; - state.stop_reason = Some("stopped after plan completed (--until plan)".to_string()); - // Not because this test wants legacy behaviour, but because its - // subject is orthogonal to the launch path (34-06). + let phase = PhaseId::new(1107); + let mut state = State::new( + phase, + AgentKind::Claude, + Mode::Supervise, + root.to_path_buf(), + ); + // The capture shape, for the same reason as + // `launch_stage_persists_the_supervisor_record_for_reload`: no delivery canary. state.legacy_claude_launch = true; workflow::save_state(&state).unwrap(); - let result = resume(root, phase, None, false); - - // `resume()` loads its own `State` from the state file and never writes the - // spawned pid back into this test's local `state`, so binding the guard from - // that local binding would capture `None` and silently reap nothing. Read the - // pid back from disk. `ReapMonitorOnDrop` captures `Option` by value, so - // the guard does not borrow the temporary it is built from. - // - // Bound here, ahead of `result.unwrap()` below: a `resume()` that spawned the - // monitor and then failed a later `?` leaves `result` as `Err` with the pid - // nonetheless live. - let reloaded_for_reap = workflow::load_state(root, phase).ok(); - let _reap_guard = reloaded_for_reap - .as_ref() - .map(ReapMonitorOnDrop::after_launch); - + let result = launch_stage(&mut state, None, None); + let _reap_guard = ReapMonitorOnDrop::after_launch(&state); result.unwrap(); - let reloaded = workflow::load_state(root, phase).unwrap(); + let record = state + .supervisor + .clone() + .expect("the launch records its supervisor"); assert!( - !reloaded.stopped, - "resume must clear stopped so the phase is no longer marked halted" - ); - assert_eq!( - reloaded.stop_reason, None, - "resume must clear stop_reason alongside stopped" - ); - assert_eq!( - reloaded.stop_until, None, - "resume must clear stop_until so the phase does not immediately re-stop \ - the next time it advances past Plan" + record.socket_path.starts_with(cache.join("sock")), + "{} must be under {}/sock", + record.socket_path.display(), + cache.display() ); assert!( - reloaded.monitor_pid.is_some(), - "resume() must have spawned a monitor whose pid is recorded in state — if this \ - fails, the reap guard above is silently reaping nothing and this test has \ - stopped covering the launch path it was written to cover" + !record.socket_path.starts_with(root), + "the socket must not live in the project tree" ); - } + assert!(record.socket_path.as_os_str().len() <= 107); + let reloaded = workflow::load_state(root, phase).unwrap(); + assert_eq!( + reloaded.supervisor.as_ref(), + Some(&record), + "the reloaded state holds exactly the published record" + ); + match devflow_core::supervisor::probe(&record.socket_path).unwrap() { + devflow_core::supervisor::Liveness::Alive(reply) => { + assert_eq!(reply.launch_id, record.launch_id); + assert_eq!(reply.launch, "capture"); + } + other => panic!("the recorded socket must answer ALIVE, got {other:?}"), + } + std::fs::write(&out, record.socket_path.display().to_string()).unwrap(); - /// A phase parked at an open, already-answered Code gate: the state says - /// `gate_pending`, the request file is on disk, and an unconsumed response - /// sits beside it. Returns the (request, response) bytes as planted. - fn seed_code_gate_with_pending_answer( - root: &Path, - phase: PhaseId, - mode: Mode, - ) -> (Vec, Vec) { - let mut state = State::new(phase, AgentKind::Claude, mode, root.to_path_buf()); - state.stage = Stage::Code; - state.gate_pending = true; - state.stopped = true; - state.stop_reason = Some("stage failed; gate open".to_string()); - state.legacy_claude_launch = true; - workflow::save_state(&state).unwrap(); + let events_path = devflow_core::events::events_path(root); + let advance_events = || { + std::fs::read_to_string(&events_path) + .unwrap_or_default() + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .filter(|event| { + event["event"] + .as_str() + .is_some_and(|name| name.starts_with("advance_")) + }) + .collect::>() + }; + if full { + let mut moved = workflow::load_state(root, phase).unwrap(); + moved.stage = Stage::Plan; + workflow::save_state(&moved).unwrap(); + } + std::fs::write(root.join("go"), "").unwrap(); + let gone = poll_until(Duration::from_secs(30), || { + devflow_core::supervisor::probe(&record.socket_path) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }); + assert!(gone, "the supervisor finishes once its agent exits"); + let events = advance_events(); + if full { + let failed = events + .iter() + .find(|event| event["event"] == "advance_failed") + .unwrap_or_else(|| panic!("monitor_entry_full must run the advance: {events:?}")); + assert_eq!(failed["reason"], "stage mismatch", "{failed}"); + assert_eq!( + failed["expected"], "define", + "the advance is passed the launched stage: {failed}" + ); + } else { + assert!( + events.is_empty(), + "control: the supervise-only entry never advances: {events:?}" + ); + } + } - Gates::write_gate( - root, - phase, - Stage::Code, - "[never-silent] stage code failed — human review needed", - ) - .unwrap(); - let response_path = Gates::response_path(root, phase, Stage::Code); + /// A `claude` stand-in that leaves `agent-ran` in its working directory + /// (the project root) and exits 0 without reading its stdin. + fn marker_writing_claude_dir() -> tempfile::TempDir { + let path_dir = agent_free_dir_with_agent_stub("claude"); std::fs::write( - &response_path, - r#"{"approved":false,"note":"abort: stale answer nobody polled","responded_by":"test"}"#, + path_dir.path().join("claude"), + "#!/bin/sh\necho ran > agent-ran\nexit 0\n", ) .unwrap(); - ( - std::fs::read(Gates::gate_path(root, phase, Stage::Code)).unwrap(), - std::fs::read(&response_path).unwrap(), - ) + path_dir } - /// SURV-02 / Phase 48 criterion 3 (48-CONTEXT D-05 "verified correction"): - /// `resume` relaunches the saved stage and never reads a pending answer. - /// The gate here is a stage-failure gate, not a preflight refusal: the - /// relaunch's preflight PASSES (Supervise mode), so nothing re-fires the - /// gate, and the planted answer must survive byte-identical — not read, - /// not acked, not deleted. + /// 50-10 tracer (D-07, T-50-27): an in-crate stream launch really runs its + /// agent, because `launch_context()`'s test branch re-enters this test + /// binary at the `test_support::monitor_entry_supervise_only` harness + /// entry, which runs the real `run_monitor` body; and the launch registers + /// only under the cache dir its parent handed it. /// - /// Scope: asserts on the state immediately after `resume()` returns. The - /// detached monitor it spawns is reaped at drop; what that monitor would - /// do with the answer later is outside this test. + /// The control launches the same shape through + /// `MonitorExec::Subcommand` at this test binary, which is what every + /// in-crate stream launch did before 50-10: libtest rejects `__monitor`'s + /// arguments, so the agent never runs. The same "agent ran" check that + /// passes above must fail there, or it proves nothing. + #[test] + fn an_in_crate_stream_launch_runs_its_monitor_through_the_harness_entry() { + const NAME: &str = "pipeline_launch::tests::an_in_crate_stream_launch_runs_its_monitor_through_the_harness_entry"; + if !devflow_core::test_support::in_child_test(NAME) { + // The child gets a parent-owned `/tmp/dfu…` DEVFLOW_CACHE_DIR. + let path_dir = marker_writing_claude_dir(); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); + return; + } + let cache = PathBuf::from( + std::env::var_os("DEVFLOW_CACHE_DIR").expect("the parent passes DEVFLOW_CACHE_DIR"), + ); + let prompt = || monitor::MonitorLaunch::PipeOwning { + prompt: "50-10 tracer".to_string(), + }; + + // The harness-entry launch. + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + init_repo(root); + let phase = PhaseId::new(1010); + let mut state = State::new( + phase, + AgentKind::Claude, + Mode::Supervise, + root.to_path_buf(), + ); + state.stage = Stage::Code; + workflow::save_state(&state).unwrap(); + let launched = spawn_agent_and_record(&mut state, "claude", &[], &[], None, prompt()); + let _reap = ReapMonitorOnDrop::after_launch(&state); + launched.unwrap(); + let exit_path = devflow_core::agent_result::exit_code_path(root, phase); + let exited = poll_until(Duration::from_secs(30), || { + std::fs::read_to_string(&exit_path).is_ok_and(|code| !code.trim().is_empty()) + }); + assert!( + root.join("agent-ran").is_file(), + "the harness entry must run the agent: no marker at {} (exit file written: {exited})", + root.join("agent-ran").display() + ); + assert_eq!( + std::fs::read_to_string(&exit_path).unwrap().trim(), + "0", + "the supervised agent's exit code must be recorded" + ); + let entry = devflow_core::registry::entry_path_in(&cache, &state.project_root, phase); + assert!( + entry.is_file(), + "the launch must register under its own cache dir: no {}", + entry.display() + ); + + // The control: the pre-50-10 re-exec of this test binary. + let control_dir = tempfile::tempdir().unwrap(); + let control_root = control_dir.path(); + let control_phase = PhaseId::new(1011); + let mut control = State::new( + control_phase, + AgentKind::Claude, + Mode::Supervise, + control_root.to_path_buf(), + ); + control.stage = Stage::Code; + let subcommand = monitor::LaunchContext::new( + cache.clone(), + monitor::MonitorExec::Subcommand { + exe: std::env::current_exe().unwrap(), + }, + ); + // Since 50-11 `spawn_monitor` waits for the publish, so the control + // monitor (libtest given `__monitor`'s arguments) exiting by itself + // comes back as `NotReady` rather than as a pid to poll. + let control_result = + monitor::spawn_monitor(&control, "claude", &[], &[], prompt(), &subcommand); + assert!( + matches!(control_result, Err(monitor::MonitorError::NotReady(_))), + "the control monitor (libtest given __monitor's arguments) must exit by itself \ + without publishing: {control_result:?}" + ); + assert!( + !control_root.join("agent-ran").exists(), + "control: re-execing the test binary as __monitor must NOT run the agent" + ); + assert!( + !devflow_core::agent_result::exit_code_path(control_root, control_phase).exists(), + "control: no monitor body ran, so no exit file may exist" + ); + } + + /// The CLI source files under `src/`, found from `CARGO_MANIFEST_DIR`. + fn cli_source_files() -> Vec { + fn walk(dir: &Path, found: &mut Vec) { + for entry in std::fs::read_dir(dir).unwrap() { + let path = entry.unwrap().path(); + if path.is_dir() { + walk(&path, found); + } else if path.extension().is_some_and(|ext| ext == "rs") { + found.push(path); + } + } + } + let mut found = Vec::new(); + walk( + &Path::new(env!("CARGO_MANIFEST_DIR")).join("src"), + &mut found, + ); + found.sort(); + assert!( + found + .iter() + .any(|path| path.ends_with("src/pipeline_launch.rs")), + "the scan must see this file, or it scanned the wrong directory: {found:?}" + ); + found + } + + /// `source` with every line whose first non-blank characters are `//` + /// dropped. + fn code_lines(source: &str) -> impl Iterator { + source + .lines() + .filter(|line| !line.trim_start().starts_with("//")) + } + + /// Lines of `source` (comments dropped) that register a root through the + /// env-reading `registry::register(`. `register_in(` does not match: `_` + /// follows `register` there. + fn unscoped_register_calls(source: &str) -> Vec { + let needle = concat!("registry::", "register("); + code_lines(source) + .filter(|line| line.contains(needle)) + .map(str::to_string) + .collect() + } + + /// T-50-27: every root registration in the CLI goes through + /// `register_in(ctx.cache_dir, …)`, so a test's cache dir is the only + /// place an in-crate launch can register. + #[test] + fn cli_source_registers_roots_only_through_register_in() { + let planted = concat!( + " let _ = devflow_core::registry::", + "register(&root, phase);\n" + ); + assert_eq!( + unscoped_register_calls(planted).len(), + 1, + "planted control: a live unscoped register call must be flagged" + ); + let commented = concat!( + " // let _ = devflow_core::registry::", + "register(&root, phase);\n" + ); + assert!( + unscoped_register_calls(commented).is_empty(), + "planted control: the same call behind // must not be flagged" + ); + let scoped = concat!( + " let _ = devflow_core::registry::", + "register_in(&ctx.cache_dir, &root, phase);\n" + ); + assert!( + unscoped_register_calls(scoped).is_empty(), + "register_in( must not be flagged" + ); + + let mut offenders = Vec::new(); + for path in cli_source_files() { + let source = std::fs::read_to_string(&path).unwrap(); + for line in unscoped_register_calls(&source) { + offenders.push(format!("{}: {}", path.display(), line.trim())); + } + } + assert!( + offenders.is_empty(), + "CLI source must register roots only through registry::register_in: {offenders:#?}" + ); + } + + /// The 48-09 D-09 ENV_MUTEX-guarded `set_var`/`remove_var` calls already + /// in this crate's tests (preflight 19, pipeline_launch 7, pipeline_gate + /// 6, pipeline_outcomes 4; counted 2026-10-14 and again 2026-10-16). + const GUARDED_ENV_MUTATION_CEILING: usize = 36; + + /// The process-env mutators, spelled through `concat!` so this file adds + /// nothing to its own scan. + const ENV_MUTATORS: [&str; 2] = [concat!("env::", "set_var"), concat!("env::", "remove_var")]; + + /// The variables 50-10's cache isolation and harness-entry selection must + /// reach a child only through `run_test_in_child`'s `extra_env`. + const HERMETIC_ENV_LITERALS: [&str; 6] = [ + concat!("\"DEVFLOW_", "CACHE_DIR\""), + concat!("\"HO", "ME\""), + concat!("\"XDG_", "CACHE_HOME\""), + concat!("\"XDG_", "CONFIG_HOME\""), + concat!("\"DEVFLOW_", "MONITOR_ARGV\""), + concat!("\"DEVFLOW_", "TEST_MONITOR_FULL\""), + ]; + + /// `source` with `//` lines dropped and the rest joined, so a call split + /// across lines still reads as one. + fn joined_code(source: &str) -> String { + code_lines(source).collect::>().join("\n") + } + + /// Calls in comment-stripped `source` that mutate one of + /// [`HERMETIC_ENV_LITERALS`]: a mutator, `(`, optional whitespace, then + /// the variable's string literal. + fn hermetic_env_mutations(source: &str) -> Vec { + let code = joined_code(source); + let mut found = Vec::new(); + for mutator in ENV_MUTATORS { + let call = format!("{mutator}("); + for (at, _) in code.match_indices(&call) { + let argument = code[at + call.len()..].trim_start(); + if let Some(literal) = HERMETIC_ENV_LITERALS + .iter() + .find(|literal| argument.starts_with(**literal)) + { + found.push(format!("{mutator}({literal}")); + } + } + } + found + } + + /// Every `set_var`/`remove_var` in comment-stripped `source`. + fn env_mutation_count(source: &str) -> usize { + let code = joined_code(source); + ENV_MUTATORS + .iter() + .map(|mutator| code.matches(mutator).count()) + .sum() + } + + /// 50-10 Task 2: the hermetic variables reach a child test process only + /// through `extra_env`, never through a process-global mutation, and this + /// plan adds no other process-env mutation to the crate (the 36 guarded + /// 48-09 calls stay as they are). + #[test] + fn cli_source_sets_no_hermetic_env_var_and_adds_no_process_env_mutation() { + let split_call = concat!( + " std::env::set", + "_var(\n", + " \"DEVFLOW_", + "CACHE_DIR\", dir);\n" + ); + assert_eq!( + hermetic_env_mutations(split_call).len(), + 1, + "planted control: a two-line mutation of the cache variable must be flagged" + ); + let other_var = concat!( + " std::env::set", + "_var(\n", + " \"DEVFLOW_", + "OTHER\", dir);\n" + ); + assert!( + hermetic_env_mutations(other_var).is_empty(), + "planted control: a mutation of an unrelated variable must not be flagged" + ); + let three_calls = concat!( + " std::env::set", + "_var(\"A\", \"1\");\n", + " std::env::remove", + "_var(\"A\");\n", + " // std::env::set", + "_var(\"A\", \"2\");\n" + ); + assert_eq!( + env_mutation_count(three_calls), + 2, + "planted control: one set, one remove and one commented-out set count 2" + ); + + let mut hermetic = Vec::new(); + let mut total = 0; + for path in cli_source_files() { + let source = std::fs::read_to_string(&path).unwrap(); + for call in hermetic_env_mutations(&source) { + hermetic.push(format!("{}: {call}", path.display())); + } + total += env_mutation_count(&source); + } + assert!( + hermetic.is_empty(), + "the hermetic variables must reach a child only through extra_env: {hermetic:#?}" + ); + assert!( + total <= GUARDED_ENV_MUTATION_CEILING, + "CLI source holds {total} set_var/remove_var calls, more than the \ + {GUARDED_ENV_MUTATION_CEILING} ENV_MUTEX-guarded 48-09 calls" + ); + } + /// 20c (review: Codex MEDIUM — resume semantics): a phase halted by + /// `--until ` has `stopped`/`stop_reason`/`stop_until` persisted. + /// `resume` must clear all three BEFORE relaunching — otherwise + /// `transition()`'s `stop_until == Some(from)` check would immediately + /// re-stop the phase the next time it advances, and the phase would + /// stay marked `stopped` forever despite the operator's explicit + /// resume. Asserts on the persisted state (not just `resume`'s exit + /// code), since `transition()` saves state before `launch_stage` runs. + /// + /// **Premise moved off `STREAM_JSON_STAGES` membership deliberately + /// (34-06); do not "simplify" the opt-out away.** The subject is resume's + /// stop-marker semantics, not which launch path the relaunch takes. The + /// opt-out is persisted here rather than set on a local binding because + /// `resume()` loads its own `State` from disk; `apply_legacy_launch_opt_out` + /// ORs the persisted value, so it survives the reload. + #[test] + fn resume_clears_stop_marker_and_advances_past_stop_point() { + const NAME: &str = + "pipeline_launch::tests::resume_clears_stop_marker_and_advances_past_stop_point"; + if !devflow_core::test_support::in_child_test(NAME) { + let path_dir = agent_free_dir_with_agent_stub("claude"); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); + return; + } + let _guard = env_lock(); + + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + init_repo(root); + + let phase = PhaseId::new(66); + let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); + state.stage = Stage::Plan; + state.stop_until = Some(Stage::Plan); + state.stopped = true; + state.stop_reason = Some("stopped after plan completed (--until plan)".to_string()); + // Not because this test wants legacy behaviour, but because its + // subject is orthogonal to the launch path (34-06). + state.legacy_claude_launch = true; + workflow::save_state(&state).unwrap(); + + let result = resume(root, phase, None, false); + + // `resume()` loads its own `State` from the state file and never writes the + // spawned pid back into this test's local `state`, so binding the guard from + // that local binding would capture `None` and silently reap nothing. Read the + // pid back from disk. `ReapMonitorOnDrop` captures `Option` by value, so + // the guard does not borrow the temporary it is built from. + // + // Bound here, ahead of `result.unwrap()` below: a `resume()` that spawned the + // monitor and then failed a later `?` leaves `result` as `Err` with the pid + // nonetheless live. + let reloaded_for_reap = workflow::load_state(root, phase).ok(); + let _reap_guard = reloaded_for_reap + .as_ref() + .map(ReapMonitorOnDrop::after_launch); + + result.unwrap(); + + let reloaded = workflow::load_state(root, phase).unwrap(); + assert!( + !reloaded.stopped, + "resume must clear stopped so the phase is no longer marked halted" + ); + assert_eq!( + reloaded.stop_reason, None, + "resume must clear stop_reason alongside stopped" + ); + assert_eq!( + reloaded.stop_until, None, + "resume must clear stop_until so the phase does not immediately re-stop \ + the next time it advances past Plan" + ); + assert!( + reloaded.supervisor.is_some(), + "resume() must have spawned a monitor whose supervisor is recorded in state — if this \ + fails, the reap guard above is silently reaping nothing and this test has \ + stopped covering the launch path it was written to cover" + ); + } + + /// A phase parked at an open, already-answered Code gate: the state says + /// `gate_pending`, the request file is on disk, and an unconsumed response + /// sits beside it. Returns the (request, response) bytes as planted. + fn seed_code_gate_with_pending_answer( + root: &Path, + phase: PhaseId, + mode: Mode, + ) -> (Vec, Vec) { + let mut state = State::new(phase, AgentKind::Claude, mode, root.to_path_buf()); + state.stage = Stage::Code; + state.gate_pending = true; + state.stopped = true; + state.stop_reason = Some("stage failed; gate open".to_string()); + state.legacy_claude_launch = true; + workflow::save_state(&state).unwrap(); + + Gates::write_gate( + root, + phase, + Stage::Code, + "[never-silent] stage code failed — human review needed", + ) + .unwrap(); + let response_path = Gates::response_path(root, phase, Stage::Code); + std::fs::write( + &response_path, + r#"{"approved":false,"note":"abort: stale answer nobody polled","responded_by":"test"}"#, + ) + .unwrap(); + ( + std::fs::read(Gates::gate_path(root, phase, Stage::Code)).unwrap(), + std::fs::read(&response_path).unwrap(), + ) + } + + /// SURV-02 / Phase 48 criterion 3 (48-CONTEXT D-05 "verified correction"): + /// `resume` relaunches the saved stage and never reads a pending answer. + /// The gate here is a stage-failure gate, not a preflight refusal: the + /// relaunch's preflight PASSES (Supervise mode), so nothing re-fires the + /// gate, and the planted answer must survive byte-identical — not read, + /// not acked, not deleted. + /// + /// Scope: asserts on the state immediately after `resume()` returns. The + /// detached monitor it spawns is reaped at drop; what that monitor would + /// do with the answer later is outside this test. #[test] fn resume_relaunches_without_consuming_a_pending_gate_answer() { const NAME: &str = @@ -2348,7 +3712,7 @@ mod tests { "resume must relaunch the saved stage" ); assert!( - reloaded.monitor_pid.is_some(), + reloaded.supervisor.is_some(), "resume must have spawned a monitor for the saved stage" ); assert_eq!( @@ -2447,7 +3811,8 @@ mod tests { const NAME: &str = "pipeline_launch::tests::resume_preserves_unfired_until_cap"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2459,7 +3824,7 @@ mod tests { let phase = PhaseId::new(67); // `Mode::Supervise` for the same 35.1-03 reason as - // `launch_stage_persists_monitor_pid_for_reload` above: the legacy + // `launch_stage_persists_the_supervisor_record_for_reload` above: the legacy // opt-out set below is refused in `Mode::Auto` by // `preflight_unattended_launch_check`, and this test's subject — the // unfired `stop_until` cap surviving a resume — is mode-independent. @@ -2505,8 +3870,8 @@ mod tests { "an unfired cap must not itself flip stopped to true — resume only relaunches" ); assert!( - reloaded.monitor_pid.is_some(), - "resume() must still have spawned a monitor whose pid is recorded in state" + reloaded.supervisor.is_some(), + "resume() must still have spawned a monitor whose supervisor is recorded in state" ); } @@ -2524,7 +3889,8 @@ mod tests { const NAME: &str = "pipeline_launch::tests::resume_without_a_cap_is_unchanged"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2569,8 +3935,8 @@ mod tests { assert!(!reloaded.stopped); assert_eq!(reloaded.stop_reason, None); assert!( - reloaded.monitor_pid.is_some(), - "resume() must still relaunch and record a monitor pid with no cap present" + reloaded.supervisor.is_some(), + "resume() must still relaunch and record a supervisor with no cap present" ); } @@ -2579,7 +3945,8 @@ mod tests { const NAME: &str = "pipeline_launch::tests::resume_with_agent_hands_off_and_relaunches_under_the_new_driver"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("codex"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2612,7 +3979,7 @@ mod tests { let reloaded = workflow::load_state(root, phase).unwrap(); assert_eq!(reloaded.agent, AgentKind::Codex); - assert!(reloaded.monitor_pid.is_some()); + assert!(reloaded.supervisor.is_some()); assert!(!devflow_core::ship::cron_instructions_path(root, phase).exists()); let events = std::fs::read_to_string(devflow_core::events::events_path(root)).unwrap(); let handoff = events @@ -2632,7 +3999,8 @@ mod tests { "pipeline_launch::tests::resume_without_agent_leaves_the_saved_agent_untouched"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2730,7 +4098,8 @@ mod tests { "pipeline_launch::tests::resume_with_same_agent_is_an_ordinary_idempotent_resume"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2770,7 +4139,8 @@ mod tests { const NAME: &str = "pipeline_launch::tests::resume_with_agent_allows_plan_stage"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("codex"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2796,11 +4166,13 @@ mod tests { /// Whole-state comparison keeps this handoff check current when State gains fields. #[test] - fn resume_with_agent_preserves_every_state_field_except_agent_and_monitor_pid() { - const NAME: &str = "pipeline_launch::tests::resume_with_agent_preserves_every_state_field_except_agent_and_monitor_pid"; + fn resume_with_agent_preserves_every_state_field_except_agent() { + const NAME: &str = + "pipeline_launch::tests::resume_with_agent_preserves_every_state_field_except_agent"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("codex"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2860,9 +4232,23 @@ mod tests { plan files, so the set is legitimately empty" ); + // 50-11: the relaunch also records its ping-confirmed supervisor, + // asserted rather than merely excluded for the same reason. + let supervisor_before = before + .as_object_mut() + .unwrap() + .remove("supervisor") + .unwrap(); + let supervisor_after = after.as_object_mut().unwrap().remove("supervisor").unwrap(); + assert_eq!(supervisor_before, serde_json::Value::Null); + assert!( + supervisor_after["socket_path"].is_string() + && supervisor_after["launch_id"].is_string(), + "the relaunch must record its published supervisor: {supervisor_after}" + ); + for value in [&mut before, &mut after] { value.as_object_mut().unwrap().remove("agent"); - value.as_object_mut().unwrap().remove("monitor_pid"); } assert_eq!(before, after); } @@ -2873,7 +4259,8 @@ mod tests { "pipeline_launch::tests::resume_with_agent_from_a_rate_limited_state_relaunches"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_dir_with_agent_stub("codex"); - let output = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let output = + crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, NAME); return; } @@ -2907,7 +4294,7 @@ mod tests { let reloaded = workflow::load_state(root, phase).unwrap(); assert_eq!(reloaded.agent, AgentKind::Codex); assert!(!reloaded.stopped); - assert!(reloaded.monitor_pid.is_some()); + assert!(reloaded.supervisor.is_some()); assert!(!devflow_core::ship::cron_instructions_path(root, phase).exists()); } @@ -2945,7 +4332,7 @@ mod tests { "a failed resume must leave the phase marked stopped, not a false-active zombie" ); assert!(reloaded.stop_reason.is_some_and(|r| r.contains("claude"))); - assert!(reloaded.monitor_pid.is_none()); + assert!(reloaded.supervisor.is_none()); return; } @@ -3043,20 +4430,24 @@ mod tests { }); } - /// WR-04 (18-fix): an early failure in `launch_stage_inner` — before - /// `monitor::spawn_monitor` ever runs — must not leave a stale - /// `monitor_pid` behind. Pre-fix, `state.monitor_pid` still named the - /// PREVIOUS stage's (now-dead) monitor after `ensure_agent_binary` - /// returned early via `?`, and `liveness()`/`doctor` then misreported - /// `Stuck → devflow resume` — the wrong remedy for what's actually an - /// agent-binary/staleness failure. PATH is neutralized to a `git`-only - /// directory under `ENV_MUTEX`, mirroring `transition_resets_infra_failures`, - /// so `ensure_agent_binary("claude")` reliably fails without touching a - /// real agent CLI and without racing other PATH-mutating tests. + /// WR-04 (18-fix, 50-23): an early failure in `launch_stage_inner` — + /// before `monitor::spawn_monitor` ever runs — must not leave the + /// PREVIOUS stage's supervisor record behind. Pre-fix, the state still + /// named the previous stage's (now-dead) monitor after the launch + /// returned early via `?`, and `status`/`doctor` then misreported it as + /// gone while active, naming `devflow resume` — the wrong remedy for + /// what is actually a launch failure. + /// + /// Which early failure this reaches (checked by hand mutants, 50-22): the + /// child's PATH holds only `git` and an `exit 0` stub `claude`, so for an + /// `Auto` Code launch the delivery canary reads `Absent` and the launch + /// ends in [`refuse_launch`], before `ensure_agent_binary` runs. The + /// other early-failure path, [`record_failed_launch`], is pinned by + /// `a_failed_monitor_spawn_clears_the_supervisor_record`. #[test] - fn launch_stage_inner_clears_monitor_pid_on_early_failure() { - const NAME: &str = - "pipeline_launch::tests::launch_stage_inner_clears_monitor_pid_on_early_failure"; + fn launch_stage_inner_clears_the_supervisor_record_on_early_failure() { + const NAME: &str = "pipeline_launch::tests::\ + launch_stage_inner_clears_the_supervisor_record_on_early_failure"; enter_agent_free_child!(NAME, "claude"); let _guard = env_lock(); @@ -3065,26 +4456,26 @@ mod tests { let phase = PhaseId::new(93); let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); state.stage = Stage::Code; - // A stale pid from a prior stage's now-dead monitor — this is what - // must be cleared, not carried forward into the new stage. - state.monitor_pid = Some(999_999); + // 50-23: the prior stage's supervisor record, which must be cleared, + // not carried forward into the new stage, or `status` reads + // GONE-while-active and names `devflow resume`. + state.supervisor = Some(fabricated_prior_record(root)); workflow::save_state(&state).unwrap(); let result = launch_stage_inner(&mut state, None, None); assert!( result.is_err(), - "ensure_agent_binary must fail against the neutralized, agent-free PATH" + "the launch must fail early: the stub `claude` cannot confirm the canary" ); assert_eq!( - state.monitor_pid, None, - "an early launch failure must clear the stale monitor_pid in-memory, not carry it \ - forward from the previous stage" + state.supervisor, None, + "an early launch failure must clear the prior supervisor record in memory" ); let reloaded = workflow::load_state(root, phase).unwrap(); assert_eq!( - reloaded.monitor_pid, None, - "the monitor_pid clear must be persisted to state.json, not just in-memory" + reloaded.supervisor, None, + "the supervisor record clear must be persisted to state.json" ); } @@ -3747,8 +5138,9 @@ mod tests { let phase = PhaseId::new(122); let mut state = canary_state(root, phase); - // The previous stage's monitor pid, as `transition()` would leave it. - state.monitor_pid = Some(4_294_967_000); + // The previous stage's supervisor record, as `transition()` would + // leave it (50-23). + state.supervisor = Some(fabricated_prior_record(root)); workflow::save_state(&state).unwrap(); let calls = std::cell::Cell::new(0usize); @@ -3770,15 +5162,15 @@ mod tests { got: {message}" ); - assert!( - state.monitor_pid.is_none(), - "a refused launch must not leave the previous stage's monitor pid standing — \ - liveness() would report Stuck and point at `devflow resume`, which cannot help" + assert_eq!( + state.supervisor, None, + "a refused launch must not leave the previous stage's supervisor record standing: \ + it reads GONE-while-active and sends the operator to `devflow resume`" ); let reloaded = workflow::load_state(root, phase).unwrap(); - assert!( - reloaded.monitor_pid.is_none(), - "the cleared pid must be persisted, not only cleared in memory" + assert_eq!( + reloaded.supervisor, None, + "the cleared supervisor record must be persisted, not only cleared in memory" ); assert_eq!( reloaded.canary, @@ -3961,8 +5353,8 @@ mod tests { ); assert_eq!(state.canary, Some(canary::CanaryOutcome::Absent)); assert!( - state.monitor_pid.is_none(), - "a refused launch must record no monitor pid" + state.supervisor.is_none(), + "a refused launch must record no supervisor" ); assert_eq!( stage_launched_count(root, phase), @@ -4157,7 +5549,7 @@ mod tests { workflow::save_state(&state).unwrap(); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4222,7 +5614,7 @@ mod tests { workflow::save_state(&state).unwrap(); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4467,7 +5859,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4568,7 +5960,7 @@ mod tests { Mode::Supervise, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4644,7 +6036,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4679,11 +6071,16 @@ mod tests { let writer = std::thread::spawn(move || { let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2); loop { + // Check the response before the state. The park saves + // `stopped` before it removes the response, so a response the + // park removed is always seen with `stopped` set. Reading the + // state first let a whole park land between the two reads. + let response_gone = !writer_response_path.exists(); let state = workflow::load_state(&writer_root, phase); if state.as_ref().is_ok_and(|state| state.stopped) { return; } - if !writer_response_path.exists() { + if response_gone { write_gate_response(&writer_root, phase, Stage::Code, false, Some("abort")); writer_third_gate_response_written .store(true, std::sync::atomic::Ordering::SeqCst); @@ -4799,7 +6196,7 @@ mod tests { Mode::Supervise, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4909,7 +6306,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -4973,7 +6370,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5010,7 +6407,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5048,7 +6445,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5093,7 +6490,7 @@ mod tests { Mode::Auto, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5143,7 +6540,7 @@ mod tests { Mode::Supervise, ); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5338,7 +6735,7 @@ mod tests { workflow::save_state(&state).unwrap(); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5413,7 +6810,7 @@ mod tests { workflow::save_state(&state).unwrap(); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -5450,7 +6847,7 @@ mod tests { assert!( reloaded_for_reap .as_ref() - .is_some_and(|reloaded| reloaded.monitor_pid.is_some()), + .is_some_and(|reloaded| reloaded.supervisor.is_some()), "the checkpoint must relaunch the session, not only record the event" ); let gate_fired = events_of_kind(root, "gate_fired"); @@ -6330,4 +7727,2293 @@ mod tests { the guard for ANY pipe-owning launch (Claude + Antigravity today)" ); } + + // ---- 50-13: the stage handoff, advance safe points and the gate wait ---- + + /// The `:` hold variable `test_hold` reads (50-13). + const HOLD_ENV: &str = "DEVFLOW_TEST_HOLD"; + + /// The marker a handoff fixture's first agent prints when it completes. + const SUCCESS_MARKER: &str = r#"DEVFLOW_RESULT: {"status":"success"}"#; + + /// How long a handoff child waits for any one step of the chain. + const HANDOFF_STEP_WAIT: Duration = Duration::from_secs(45); + + /// A PATH dir whose fake `claude` answers `--version`. Its first run + /// (stage N) waits for `go` in its working directory (the project root) + /// and then prints `first_marker` and exits 0. Every later run (stage + /// N+1) waits for `go2` and then prints a success marker. Both waits are + /// bounded (about 60 s), so no fake agent outlives a failed test by long, + /// and the fixture's `stop_until` ends the chain after N+1. + fn handoff_claude_dir(first_marker: &str) -> tempfile::TempDir { + let path_dir = agent_free_dir_with_agent_stub("claude"); + let script = format!( + "#!/bin/sh\ncase \"$1\" in --version) exit 0 ;; esac\n\ + if [ ! -e first-done ]; then\n : > first-done\n gate=go\n marker='{first_marker}'\n\ + else\n gate=go2\n marker='{SUCCESS_MARKER}'\nfi\n\ + i=0\nwhile [ ! -e \"$gate\" ] && [ \"$i\" -lt 1200 ]; do sleep 0.05; i=$((i+1)); done\n\ + printf '%s\\n' \"$marker\"\nexit 0\n" + ); + std::fs::write(path_dir.path().join("claude"), script).unwrap(); + let real_sleep = ["/usr/bin/sleep", "/bin/sleep"] + .into_iter() + .map(Path::new) + .find(|path| path.is_file()) + .expect("sleep must exist to run this test"); + std::os::unix::fs::symlink(real_sleep, path_dir.path().join("sleep")).unwrap(); + path_dir + } + + /// The parent half of a handoff test: run `name` in a hermetic child on + /// the advance-running monitor entry, with `DEVFLOW_TEST_HOLD` naming + /// `hold` (when given) and a release path in a parent-owned dir. + fn run_handoff_child(name: &str, hold: Option<&str>, first_marker: &str) { + let path_dir = handoff_claude_dir(first_marker); + let hold_dir = tempfile::Builder::new() + .prefix("dfh") + .tempdir_in("/tmp") + .unwrap(); + let hold_value = hold.map(|point| { + std::ffi::OsString::from(format!( + "{point}:{}", + hold_dir.path().join("release").display() + )) + }); + let mut env: Vec<(&str, &std::ffi::OsStr)> = vec![ + (TEST_MONITOR_FULL_ENV, std::ffi::OsStr::new("1")), + ("DEVFLOW_GATE_TIMEOUT_SECS", std::ffi::OsStr::new("120")), + ]; + if let Some(value) = &hold_value { + env.push((HOLD_ENV, value.as_os_str())); + } + let output = run_hermetic_test_in_child(name, path_dir.path(), &env); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, name); + } + + /// One launched stage N in a handoff child: the repo, its phase, N's + /// record and the hermetic cache dir. Dropping it releases any hold and + /// reaps every supervisor whose socket is still in the cache dir, so a + /// failing assertion leaves no `__monitor` or fake agent behind. + struct HandoffRun { + _dir: tempfile::TempDir, + root: PathBuf, + phase: PhaseId, + n: devflow_core::state::SupervisorRecord, + cache: PathBuf, + } + + impl HandoffRun { + /// Launch stage Define for `phase` (capture shape, Supervise mode, + /// `--until plan`) and return once N is recorded. + fn launch(phase: PhaseId) -> Self { + Self::launch_with(phase, |_, _| {}) + } + + /// [`launch`](Self::launch), with `configure` applied to the repo + /// and the state before it is saved and stage N launched. + fn launch_with(phase: PhaseId, configure: impl FnOnce(&Path, &mut State)) -> Self { + let cache = PathBuf::from( + std::env::var_os("DEVFLOW_CACHE_DIR").expect("the parent passes DEVFLOW_CACHE_DIR"), + ); + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + init_repo(&root); + std::fs::create_dir_all(root.join(".planning")).unwrap(); + std::fs::write( + root.join(".planning/ROADMAP.md"), + format!("# Roadmap\n\n### Phase {phase}: handoff fixture\n"), + ) + .unwrap(); + let mut state = State::new(phase, AgentKind::Claude, Mode::Supervise, root.clone()); + state.legacy_claude_launch = true; + state.stop_until = Some(Stage::Plan); + configure(&root, &mut state); + workflow::save_state(&state).unwrap(); + let launched = launch_stage(&mut state, None, None); + let n = state.supervisor.clone(); + let run = Self { + _dir: dir, + root, + phase, + n: n.clone().unwrap_or_else(|| { + panic!("the launch must record stage N's supervisor: {launched:?}") + }), + cache, + }; + launched.unwrap(); + run + } + + /// The release path `test_hold` waits on, from `DEVFLOW_TEST_HOLD`. + fn release_path() -> Option { + let hold = std::env::var(HOLD_ENV).ok()?; + let (_, path) = hold.split_once(':')?; + Some(PathBuf::from(path)) + } + + /// Let stage N's agent finish. + fn go(&self) { + std::fs::write(self.root.join("go"), "").unwrap(); + } + + /// Block until a supervisor reports it reached the configured hold. + fn wait_held(&self) { + let release = Self::release_path().expect("this child runs with a hold"); + let held = PathBuf::from(format!("{}.held", release.display())); + assert!( + poll_until(HANDOFF_STEP_WAIT, || held.exists()), + "the {} hold was never reached; events:\n{}", + std::env::var(HOLD_ENV).unwrap_or_default(), + self.events() + ); + } + + /// The run's whole event log, for failure messages. + fn events(&self) -> String { + std::fs::read_to_string(devflow_core::events::events_path(&self.root)) + .unwrap_or_default() + } + + /// Release the configured hold. + fn release(&self) { + if let Some(release) = Self::release_path() { + std::fs::write(release, "").unwrap(); + } + } + + fn state(&self) -> State { + workflow::load_state(&self.root, self.phase).expect("the run's state loads") + } + + /// Every `.sock` in the cache dir's `sock/` other than N's. + fn other_sockets(&self) -> Vec { + std::fs::read_dir(self.cache.join("sock")) + .map(|entries| { + entries + .filter_map(|entry| Some(entry.ok()?.path())) + .filter(|path| path.extension().is_some_and(|ext| ext == "sock")) + .filter(|path| *path != self.n.socket_path) + .collect() + }) + .unwrap_or_default() + } + } + + impl Drop for HandoffRun { + fn drop(&mut self) { + self.release(); + // Two passes: a stage N still advancing when the first pass + // stops it may have launched N+1 by the second. + for _ in 0..2 { + let sockets: Vec = std::fs::read_dir(self.cache.join("sock")) + .map(|entries| { + entries + .filter_map(|entry| Some(entry.ok()?.path())) + .filter(|path| path.extension().is_some_and(|ext| ext == "sock")) + .collect() + }) + .unwrap_or_default(); + for socket in sockets { + let Ok(sidecar) = + devflow_core::supervisor::read_sidecar(&socket.with_extension("json")) + else { + continue; + }; + let mut holder = self.state_holder(); + holder.supervisor = Some(devflow_core::state::SupervisorRecord::new( + socket, + sidecar.launch_id, + sidecar.agent_pgid, + sidecar.agent_start_time, + sidecar.boot_id, + )); + drop(ReapMonitorOnDrop::after_launch(&holder)); + } + } + } + } + + impl HandoffRun { + /// A throwaway `State` to carry one record into `ReapMonitorOnDrop`. + fn state_holder(&self) -> State { + State::new( + self.phase, + AgentKind::Claude, + Mode::Supervise, + self.root.clone(), + ) + } + } + + /// What a handoff observer saw in one poll: the state's recorded + /// supervisor and how its socket probed. + #[derive(Debug, Clone, PartialEq, Eq)] + enum Seen { + Alive(String), + Stale, + Gone, + NoRecord, + Unloadable, + ProbeFailed, + } + + fn observe(root: &Path, phase: PhaseId) -> Seen { + let Ok(state) = workflow::load_state(root, phase) else { + return Seen::Unloadable; + }; + let Some(record) = state.supervisor else { + return Seen::NoRecord; + }; + match devflow_core::supervisor::probe(&record.socket_path) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => Seen::Alive(reply.launch_id), + Ok(devflow_core::supervisor::Liveness::Stale) => Seen::Stale, + Ok(devflow_core::supervisor::Liveness::Gone) => Seen::Gone, + Err(_) => Seen::ProbeFailed, + } + } + + /// A thread that polls [`observe`] every 5 ms (and whether `watched` + /// exists) until stopped. + struct Observer { + stop: std::sync::Arc, + handle: std::thread::JoinHandle>, + } + + impl Observer { + fn start(root: &Path, phase: PhaseId, watched: &Path) -> Self { + let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let (root, watched, flag) = ( + root.to_path_buf(), + watched.to_path_buf(), + std::sync::Arc::clone(&stop), + ); + let handle = std::thread::spawn(move || { + let mut seen = Vec::new(); + while !flag.load(std::sync::atomic::Ordering::Acquire) { + seen.push((observe(&root, phase), watched.exists())); + std::thread::sleep(Duration::from_millis(5)); + } + seen + }); + Self { stop, handle } + } + + fn finish(self) -> Vec<(Seen, bool)> { + self.stop.store(true, std::sync::atomic::Ordering::Release); + self.handle.join().expect("the observer thread") + } + } + + /// Every observation between the first `Alive(from)` and the first + /// `Alive(to)` that reads `Gone`, `Stale` or no record: the handoff's + /// gaps. An error when either endpoint was never observed. + fn handoff_gaps(seen: &[(Seen, bool)], from: &str, to: &str) -> Result, String> { + let first = |id: &str| { + seen.iter() + .position(|(observed, _)| *observed == Seen::Alive(id.to_string())) + }; + let start = first(from).ok_or_else(|| format!("never saw {from} alive"))?; + let end = first(to).ok_or_else(|| format!("never saw {to} alive"))?; + if end < start { + return Err(format!("saw {to} alive before {from}")); + } + Ok(seen[start..end] + .iter() + .map(|(observed, _)| observed.clone()) + .filter(|observed| matches!(observed, Seen::Gone | Seen::Stale | Seen::NoRecord)) + .collect()) + } + + /// V-13 (R-1, T-50-34): a stage handoff publishes N+1 before N unlinks. + /// Stage N's agent finishes, N's in-process advance launches N+1 and + /// records it, and N is parked at the `after_publish` hold (lock still + /// held). At the hold the state names N+1, N+1 probes Alive, and N's + /// socket still exists and answers with owner `advancing` (RV-04: N is + /// not marked handed off inside the advance). Released, N unlinks and + /// N+1 stays Alive. A 5 ms observer must see no Gone, Stale or missing + /// record between N's first Alive and N+1's first Alive; it can miss a + /// gap shorter than 5 ms, so that check is supplementary and + /// `the_handoff_observer_detects_a_pre_clear_gap` shows it sees one. + #[test] + fn stage_handoff_publishes_the_next_socket_before_the_old_one_is_unlinked() { + const NAME: &str = "pipeline_launch::tests::stage_handoff_publishes_the_next_socket_before_the_old_one_is_unlinked"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("after_publish"), SUCCESS_MARKER); + return; + } + let _guard = env_lock(); + let run = HandoffRun::launch(PhaseId::new(1301)); + let observer = Observer::start(&run.root, run.phase, &run.n.socket_path); + run.go(); + run.wait_held(); + + let next = run + .state() + .supervisor + .expect("at the after_publish hold the state records a supervisor"); + assert_ne!( + next.launch_id, run.n.launch_id, + "at the after_publish hold the state names N+1, not N" + ); + match devflow_core::supervisor::probe(&next.socket_path) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => { + assert_eq!(reply.launch_id, next.launch_id); + } + other => panic!("N+1 must probe Alive at the hold, got {other:?}"), + } + assert!( + run.n.socket_path.exists(), + "N's socket file must still exist at the after_publish hold" + ); + match devflow_core::supervisor::probe(&run.n.socket_path) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => assert_eq!( + reply.owner, "advancing", + "RV-04: N stays Advancing until its advance returns" + ), + other => panic!("N must still answer at the hold, got {other:?}"), + } + + run.release(); + assert!( + poll_until(HANDOFF_STEP_WAIT, || { + devflow_core::supervisor::probe(&run.n.socket_path) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }), + "N unlinks its socket once its advance has returned" + ); + match devflow_core::supervisor::probe(&next.socket_path) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => { + assert_eq!(reply.launch_id, next.launch_id); + } + other => panic!("N+1 must stay Alive after N exits, got {other:?}"), + } + let seen = observer.finish(); + let gaps = handoff_gaps(&seen, &run.n.launch_id, &next.launch_id) + .unwrap_or_else(|why| panic!("observer: {why}: {seen:?}")); + assert!( + gaps.is_empty(), + "state read GONE or no record across the handoff: {gaps:?}" + ); + } + + /// A live in-process supervisor serving `launch_id` at a short socket + /// path in `dir`, for the observer control. + fn fake_supervisor( + dir: &Path, + root: &Path, + phase: PhaseId, + launch_id: &str, + ) -> ( + devflow_core::supervisor::runtime::Supervisor, + devflow_core::state::SupervisorRecord, + ) { + let path = dir.join(format!("{launch_id}.sock")); + let socket = devflow_core::supervisor::SocketPath::new(path.clone()).unwrap(); + let sidecar: devflow_core::supervisor::Sidecar = + serde_json::from_value(serde_json::json!({ + "launch_id": launch_id, + "project_root": root, + "phase": phase, + "stage": null, + "agent": "claude", + "launch": "capture", + "agent_pid": 0, + "agent_pgid": 0, + "agent_start_time": 0, + "boot_id": "", + "supervisor_pid": std::process::id(), + "supervisor_start_time": 0, + "created_at": "2026-10-17T00:00:00Z", + })) + .unwrap(); + let supervisor = devflow_core::supervisor::runtime::Supervisor::serve(socket, sidecar) + .expect("serve the fake supervisor"); + let record = devflow_core::state::SupervisorRecord::new( + path, + launch_id.to_string(), + 0, + 0, + String::new(), + ); + (supervisor, record) + } + + /// The observer control for V-13: the same observer and gap rule applied + /// to a simulated pre-clear handoff (a save with no supervisor between + /// N's record and N+1's, the WR-04 pitfall) reports the gap; the same + /// sequence without the pre-clear reports none. Both endpoints are real + /// in-process supervisors answering pings. + #[test] + fn the_handoff_observer_detects_a_pre_clear_gap() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let sock_dir = tempfile::Builder::new() + .prefix("dfs") + .tempdir_in("/tmp") + .unwrap(); + let phase = PhaseId::new(1302); + // Launch ids in `new_launch_id`'s format, the only one a sidecar takes. + let (obs_n, obs_next) = ("0000000000000b5a", "0000000000000b5b"); + let (sup_n, rec_n) = fake_supervisor(sock_dir.path(), root, phase, obs_n); + let (sup_next, rec_next) = fake_supervisor(sock_dir.path(), root, phase, obs_next); + let save = |record: Option<&devflow_core::state::SupervisorRecord>| { + let mut state = State::new(phase, AgentKind::Claude, Mode::Supervise, root.into()); + state.supervisor = record.cloned(); + workflow::save_state(&state).unwrap(); + }; + let hold = Duration::from_millis(80); + + for pre_clear in [true, false] { + save(Some(&rec_n)); + let observer = Observer::start(root, phase, &rec_n.socket_path); + std::thread::sleep(hold); + if pre_clear { + save(None); + std::thread::sleep(hold); + } + save(Some(&rec_next)); + std::thread::sleep(hold); + let seen = observer.finish(); + let gaps = handoff_gaps(&seen, obs_n, obs_next) + .unwrap_or_else(|why| panic!("observer: {why}: {seen:?}")); + if pre_clear { + assert!( + gaps.contains(&Seen::NoRecord), + "the observer must see the pre-clear gap: {seen:?}" + ); + } else { + assert!(gaps.is_empty(), "control: no pre-clear, no gap: {gaps:?}"); + } + } + sup_n.finish(); + sup_next.finish(); + } + + /// The RV-04 order rule, as a pure scan over `src`: `spawn_agent_and_record` + /// records N+1 as the successor and never marks N handed off, and + /// `run_monitor` takes the successor and marks the hand-off. Each function + /// is its comment-stripped lines from the column-0 line starting with its + /// signature through the next line that is exactly `}`. + fn handoff_order_violations(src: &str) -> Vec { + fn body<'a>(src: &'a str, signature: &str, floor: usize) -> Option> { + let mut lines = src.lines(); + let first = lines.by_ref().find(|line| line.starts_with(signature))?; + let mut body = vec![first]; + for line in lines { + if line.trim_start().starts_with("//") { + continue; + } + body.push(line); + if line == "}" { + break; + } + } + (body.len() >= floor).then_some(body) + } + let has = |lines: &[&str], needle: &str| lines.iter().any(|line| line.contains(needle)); + let mut violations = Vec::new(); + match body(src, "fn spawn_agent_and_record(", 40) { + None => violations.push(SCAN_SPAWN_NOT_FOUND.to_string()), + Some(lines) => { + if has(&lines, "OwnerPhase::HandedOff") { + violations.push(SCAN_SPAWN_HANDS_OFF.to_string()); + } + if !has(&lines, "record_successor(") { + violations.push(SCAN_SPAWN_NO_SUCCESSOR.to_string()); + } + } + } + match body(src, "pub(crate) fn run_monitor(", 10) { + None => violations.push(SCAN_RUN_NOT_FOUND.to_string()), + Some(lines) => { + if !has(&lines, "OwnerPhase::HandedOff") { + violations.push(SCAN_RUN_NO_HAND_OFF.to_string()); + } + if !has(&lines, "take_successor(") { + violations.push(SCAN_RUN_NO_TAKE.to_string()); + } + } + } + violations + } + + const SCAN_SPAWN_NOT_FOUND: &str = + "spawn_agent_and_record not found (or under 40 comment-stripped lines)"; + const SCAN_SPAWN_HANDS_OFF: &str = "spawn_agent_and_record enters the handed-off phase"; + const SCAN_SPAWN_NO_SUCCESSOR: &str = "spawn_agent_and_record never calls record_successor"; + const SCAN_RUN_NOT_FOUND: &str = "run_monitor not found (or under 10 comment-stripped lines)"; + const SCAN_RUN_NO_HAND_OFF: &str = "run_monitor never enters the handed-off phase"; + const SCAN_RUN_NO_TAKE: &str = "run_monitor never calls take_successor"; + + /// RV-04 (permanent source scan): N is marked handed off only in + /// `run_monitor`, after its advance returned and only when the advance + /// recorded a successor; `spawn_agent_and_record` records the successor + /// and leaves N `Advancing`. Planted controls prove the scan can flag + /// each violation and passes a correct layout. + #[test] + fn the_handed_off_marking_stays_in_run_monitor_after_the_advance() { + let real = handoff_order_violations(include_str!("pipeline_launch.rs")); + assert!(real.is_empty(), "pipeline_launch.rs: {real:?}"); + + let pad = |n: usize| " let _filler = 0;\n".repeat(n); + let spawn = |extra: &str| format!("fn spawn_agent_and_record(\n{}{extra}}}\n", pad(40)); + let run = |extra: &str| format!("pub(crate) fn run_monitor(\n{}{extra}}}\n", pad(10)); + let good_spawn = spawn(" record_successor(record);\n"); + let good_run = run( + " if let Some(next) = take_successor() {\n token.enter(OwnerPhase::HandedOff);\n }\n", + ); + let only = |violation: &str| vec![violation.to_string()]; + + assert_eq!( + handoff_order_violations(&format!("{good_spawn}{good_run}")), + Vec::::new(), + "a correct pair" + ); + assert_eq!( + handoff_order_violations(&format!( + "{}{good_run}", + spawn(" record_successor(record);\n token.enter(OwnerPhase::HandedOff);\n") + )), + only(SCAN_SPAWN_HANDS_OFF) + ); + assert_eq!( + handoff_order_violations(&format!( + "{}{good_run}", + spawn( + " record_successor(record);\n // token.enter(OwnerPhase::HandedOff);\n" + ) + )), + Vec::::new(), + "a comment line is not code" + ); + assert_eq!( + handoff_order_violations(&format!("{}{good_run}", spawn(""))), + only(SCAN_SPAWN_NO_SUCCESSOR) + ); + assert_eq!( + handoff_order_violations(&format!( + "{good_spawn}{}", + run(" let _next = take_successor();\n") + )), + only(SCAN_RUN_NO_HAND_OFF) + ); + assert_eq!( + handoff_order_violations(&format!( + "{good_spawn}{}", + run(" token.enter(OwnerPhase::HandedOff);\n") + )), + only(SCAN_RUN_NO_TAKE) + ); + assert_eq!( + handoff_order_violations(&good_run), + only(SCAN_SPAWN_NOT_FOUND) + ); + assert_eq!( + handoff_order_violations(&good_spawn), + only(SCAN_RUN_NOT_FOUND) + ); + assert_eq!( + handoff_order_violations(&format!( + "fn spawn_agent_and_record(\n{} record_successor(record);\n}}\n{good_run}", + pad(5) + )), + only(SCAN_SPAWN_NOT_FOUND), + "a body under its floor" + ); + assert_eq!( + handoff_order_violations(&format!( + "{good_spawn}pub(crate) fn run_monitor(\n take_successor();\n token.enter(OwnerPhase::HandedOff);\n}}\n" + )), + only(SCAN_RUN_NOT_FOUND), + "a run_monitor under its floor" + ); + } + + /// The run's events named `name`. + fn events_named(root: &Path, name: &str) -> Vec { + std::fs::read_to_string(devflow_core::events::events_path(root)) + .unwrap_or_default() + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .filter(|event| event["event"].as_str() == Some(name)) + .collect() + } + + /// Send `shutdown` to the supervisor at `socket` on the wire, wait for + /// its `accepted` line (returned first), run `then`, and return the final + /// line read within `final_wait`. Unlike `request_shutdown`, the caller + /// acts at a known point: after the supervisor has taken the stop. + fn shutdown_accepted_then( + socket: &Path, + then: impl FnOnce(), + final_wait: Duration, + ) -> (serde_json::Value, serde_json::Value) { + use std::io::{BufRead as _, Write as _}; + let stream = std::os::unix::net::UnixStream::connect(socket).expect("connect to N"); + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .unwrap(); + (&stream) + .write_all(b"{\"op\":\"shutdown\"}\n") + .expect("send shutdown"); + let mut reader = std::io::BufReader::new(&stream); + let mut line = String::new(); + reader.read_line(&mut line).expect("the accepted line"); + let accepted: serde_json::Value = serde_json::from_str(&line) + .unwrap_or_else(|err| panic!("accepted line {line:?}: {err}")); + then(); + stream.set_read_timeout(Some(final_wait)).unwrap(); + line.clear(); + reader.read_line(&mut line).expect("the final line"); + let last: serde_json::Value = + serde_json::from_str(&line).unwrap_or_else(|err| panic!("final line {line:?}: {err}")); + (accepted, last) + } + + /// Whether any live process is left in group `pgid`. Zombies do not + /// count: a CI job container's PID 1 never reaps an orphaned one + /// (`test_support::live_group_members`). + fn group_alive(pgid: u32) -> bool { + !devflow_core::test_support::live_group_members(pgid).is_empty() + } + + /// The body of the held-advance pair: N parks at `before_spawn` (safe + /// point 2 follows the hold), then a stop is (or is not) taken before the + /// release. + fn held_advance_before_spawn(phase: PhaseId, stop: bool) { + let _guard = env_lock(); + let run = HandoffRun::launch(phase); + run.go(); + run.wait_held(); + let release = || run.release(); + if stop { + let (accepted, last) = + shutdown_accepted_then(&run.n.socket_path, release, Duration::from_secs(30)); + assert_eq!(accepted["owner"], "advancing", "{accepted}"); + assert_eq!(last["stopped"], true, "N answers stopped: {last}"); + assert!( + poll_until(HANDOFF_STEP_WAIT, || { + devflow_core::supervisor::probe(&run.n.socket_path) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }), + "N's socket is gone after the stop" + ); + assert!( + poll_until(Duration::from_secs(5), || matches!( + lock::holder_status(&run.root, run.phase), + lock::HolderStatus::NoHolder + )), + "the per-phase lock is released" + ); + let state = run.state(); + assert!(state.stopped, "state.stopped after a stop at safe point 2"); + assert_eq!(state.stop_reason.as_deref(), Some(ADVANCE_STOP_REASON)); + assert_eq!( + stage_launched_count(&run.root, run.phase), + 1, + "no next stage is launched: {}", + run.events() + ); + let stopped = events_named(&run.root, "advance_stopped"); + assert_eq!(stopped.len(), 1, "{}", run.events()); + assert_eq!(stopped[0]["safe_point"], 2, "{}", stopped[0]); + assert!( + run.other_sockets().is_empty(), + "no next supervisor was spawned" + ); + } else { + release(); + assert!( + poll_until(HANDOFF_STEP_WAIT, || stage_launched_count( + &run.root, run.phase + ) == 2), + "control: without a stop the held advance launches the next stage: {}", + run.events() + ); + let next = run + .state() + .supervisor + .expect("control: the next stage is recorded"); + assert_ne!(next.launch_id, run.n.launch_id); + assert!( + matches!( + devflow_core::supervisor::probe(&next.socket_path), + Ok(devflow_core::supervisor::Liveness::Alive(_)) + ), + "control: the next supervisor is Alive" + ); + assert!(!run.state().stopped, "control: not stopped"); + } + } + + /// R-2, safe point 2: a stop taken while N is advancing is honoured right + /// before the next stage's spawn. Nothing is launched, the state is + /// stopped with the safe-point reason, the lock is released and N's + /// socket is gone. + #[test] + fn a_stop_during_advance_stops_at_the_next_safe_point_and_launches_nothing() { + const NAME: &str = "pipeline_launch::tests::a_stop_during_advance_stops_at_the_next_safe_point_and_launches_nothing"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_spawn"), SUCCESS_MARKER); + return; + } + held_advance_before_spawn(PhaseId::new(1303), true); + } + + /// The control for the held-advance pair: the same hold and no stop, so + /// the next stage is launched and Alive. + #[test] + fn without_a_stop_the_held_advance_launches_the_next_stage() { + const NAME: &str = + "pipeline_launch::tests::without_a_stop_the_held_advance_launches_the_next_stage"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_spawn"), SUCCESS_MARKER); + return; + } + held_advance_before_spawn(PhaseId::new(1304), false); + } + + /// R-2, safe point 3 (T-50-54): a stop taken while N holds the lock + /// between N+1's ping and its publish shuts N+1 down through its socket + /// and does not publish it. The state keeps N's record and is stopped with + /// N's safe-point reason; no agent of N+1 survives; and N answers + /// `stopped` in under `TERMINATE_VERIFY_WAIT + 3 s`, below both bounded + /// waits a lock deadlock would sit out (N's final timeout and N+1's lock + /// wait), so a stopped N+1 that waited on N's lock fails the timing. + #[test] + fn a_stop_between_the_next_spawn_and_its_publish_shuts_the_next_supervisor_down() { + const NAME: &str = "pipeline_launch::tests::a_stop_between_the_next_spawn_and_its_publish_shuts_the_next_supervisor_down"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_publish"), SUCCESS_MARKER); + return; + } + let _guard = env_lock(); + let run = HandoffRun::launch(PhaseId::new(1305)); + run.go(); + run.wait_held(); + let next_socket = match run.other_sockets().as_slice() { + [one] => one.clone(), + other => panic!("exactly one next supervisor is spawned at the hold: {other:?}"), + }; + let next = devflow_core::supervisor::read_sidecar(&next_socket.with_extension("json")) + .expect("N+1's sidecar"); + assert!(group_alive(next.agent_pgid), "N+1's agent runs at the hold"); + + let mut released_at = None; + let (accepted, last) = shutdown_accepted_then( + &run.n.socket_path, + || { + released_at = Some(std::time::Instant::now()); + run.release(); + }, + Duration::from_secs(30), + ); + let took = released_at.expect("released").elapsed(); + assert_eq!(accepted["owner"], "advancing", "{accepted}"); + assert_eq!(last["stopped"], true, "N answers stopped: {last}"); + let bound = devflow_core::agent::TERMINATE_VERIFY_WAIT + Duration::from_secs(3); + assert!( + took < bound, + "N's stop took {took:?}, not under {bound:?}: a lock deadlock between N and N+1" + ); + assert!( + poll_until(HANDOFF_STEP_WAIT, || { + devflow_core::supervisor::probe(&next_socket) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }), + "N+1 was shut down through its socket" + ); + assert!( + poll_until(Duration::from_secs(5), || !group_alive(next.agent_pgid)), + "no agent of N+1 survives" + ); + let state = run.state(); + assert_eq!( + state.supervisor.as_ref().map(|record| &record.launch_id), + Some(&run.n.launch_id), + "N+1 was never published: the state keeps N's record" + ); + assert!(state.stopped, "state.stopped after a stop at safe point 3"); + assert_eq!(state.stop_reason.as_deref(), Some(ADVANCE_STOP_REASON)); + assert_eq!( + stage_launched_count(&run.root, run.phase), + 1, + "{}", + run.events() + ); + let stopped = events_named(&run.root, "advance_stopped"); + assert_eq!(stopped.len(), 1, "{}", run.events()); + assert_eq!(stopped[0]["safe_point"], 3, "{}", stopped[0]); + } + + /// Write a capture whose Layer 1 marker reads success, with exit code 0. + fn write_success_capture(root: &Path, phase: PhaseId) { + let stdout = devflow_core::agent_result::stdout_path(root, phase); + std::fs::create_dir_all(stdout.parent().unwrap()).unwrap(); + std::fs::write(&stdout, format!("work done\n{SUCCESS_MARKER}\n")).unwrap(); + std::fs::write( + devflow_core::agent_result::exit_code_path(root, phase), + "0\n", + ) + .unwrap(); + } + + /// 999.138 residual (A6): `advance_with` refuses a stopped state with an + /// `advance_failed {reason: "stopped"}` event and an error naming + /// `devflow resume`. Control: the same state with `stopped: false` + /// advances (here to its `--until define` stop, so nothing launches). + #[test] + fn advance_refuses_a_stopped_state() { + let _guard = env_lock(); + for stopped in [true, false] { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let phase = PhaseId::new(1307); + let mut state = State::new( + phase, + AgentKind::Claude, + Mode::Supervise, + root.to_path_buf(), + ); + state.stop_until = Some(Stage::Define); + state.stopped = stopped; + workflow::save_state(&state).unwrap(); + write_success_capture(root, phase); + + let result = advance_with(root, phase, Some(Stage::Define), Duration::from_secs(5)); + let failed = events_named(root, "advance_failed"); + if stopped { + let err = result.expect_err("a stopped state must not advance"); + assert!( + err.to_string().contains("devflow resume --phase 1307"), + "the error names the repair: {err}" + ); + assert_eq!(failed.len(), 1, "{failed:?}"); + assert_eq!(failed[0]["reason"], "stopped", "{}", failed[0]); + assert!( + events_named(root, "advance_evaluated").is_empty(), + "a refused advance evaluates nothing" + ); + } else { + result.expect("control: an unstopped state advances"); + assert!(failed.is_empty(), "control: {failed:?}"); + assert_eq!(events_named(root, "advance_evaluated").len(), 1); + let finished = events_named(root, "workflow_finished"); + assert_eq!(finished.len(), 1, "control reaches its --until stop"); + } + } + } + + /// T-50-68: a supervisor stopped while its state file will not load still + /// answers `stopped` (not an error, a closed connection or the final + /// timeout), its socket goes, its agent group goes, and the state bytes + /// are unchanged: the no-write path. Control: the same stop on a run + /// whose state loads records `stopped`, so the unchanged bytes are the + /// no-write path and not a harness that cannot see a write. + #[test] + fn a_stopped_supervisor_whose_state_will_not_load_still_answers_stopped() { + const NAME: &str = "pipeline_launch::tests::a_stopped_supervisor_whose_state_will_not_load_still_answers_stopped"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, None, SUCCESS_MARKER); + return; + } + let _guard = env_lock(); + let wait = 2 * devflow_core::agent::TERMINATE_VERIFY_WAIT + Duration::from_secs(5); + for corrupt in [true, false] { + let run = HandoffRun::launch(PhaseId::new(if corrupt { 1308 } else { 1309 })); + let state_file = workflow::state_path(&run.root, run.phase); + if corrupt { + std::fs::write(&state_file, b"{ this is not json").unwrap(); + } + let before = std::fs::read(&state_file).unwrap(); + let outcome = devflow_core::supervisor::request_shutdown(&run.n.socket_path, wait); + assert!( + matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "corrupt={corrupt}: the stop reads Stopped, got {outcome:?}" + ); + assert!( + poll_until(HANDOFF_STEP_WAIT, || { + devflow_core::supervisor::probe(&run.n.socket_path) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }), + "corrupt={corrupt}: the socket is gone" + ); + assert!( + poll_until(Duration::from_secs(5), || !group_alive(run.n.agent_pgid)), + "corrupt={corrupt}: the agent group is gone" + ); + if corrupt { + assert_eq!( + std::fs::read(&state_file).unwrap(), + before, + "the state that will not load is left byte for byte" + ); + } else { + let state = run.state(); + assert!(state.stopped, "control: a loadable state records stopped"); + assert_eq!(state.stop_reason.as_deref(), Some(SUPERVISOR_STOP_REASON)); + } + } + } + + /// The marker a gate fixture's first agent prints: a failed Define, + /// which the never-silent path parks at a human gate. + const FAILED_MARKER: &str = r#"DEVFLOW_RESULT: {"status":"failed","reason":"gate fixture"}"#; + + /// Signal stage N's supervisor by the pid in its sidecar, after + /// confirming that pid is still the recorded process. + fn signal_supervisor(run: &HandoffRun, signal: libc::c_int) { + let sidecar = + devflow_core::supervisor::read_sidecar(&run.n.socket_path.with_extension("json")) + .expect("N's sidecar"); + let pid = sidecar.supervisor_pid; + assert!( + devflow_core::agent::is_same_process(pid, sidecar.supervisor_start_time), + "the sidecar's supervisor {pid} is the recorded process" + ); + let signed = libc::pid_t::try_from(pid).expect("pid fits pid_t"); + assert!(signed > 1, "refusing to signal pid {signed}"); + // SAFETY: `signed > 1` names the supervisor just identity-checked. + assert_eq!(unsafe { libc::kill(signed, signal) }, 0, "signal {pid}"); + } + + /// Launch a gate fixture and return once N is parked at its Define gate: + /// the gate file exists and N holds the lock (its owner phase is the + /// control's assertion, not a precondition here). + fn launch_parked_at_gate(phase: PhaseId) -> HandoffRun { + let run = HandoffRun::launch(phase); + run.go(); + let gate = Gates::gate_path(&run.root, phase, Stage::Define); + assert!( + poll_until(HANDOFF_STEP_WAIT, || gate.exists() + && events_named(&run.root, "gate_fired").len() == 1), + "N parks at the Define gate: {}", + run.events() + ); + assert!( + matches!( + lock::holder_status(&run.root, phase), + lock::HolderStatus::Live { .. } + ), + "N holds the per-phase lock across the gate wait" + ); + run + } + + /// V-15 (R-2, T-50-33): SIGTERM to a supervisor parked at a gate aborts + /// the wait (safe point 4): within 5 s the lock has no holder, the state + /// is stopped with the safe-point reason, the socket is gone and no next + /// stage was launched. + #[test] + fn a_signal_during_a_gate_wait_aborts_and_releases_the_lock() { + const NAME: &str = + "pipeline_launch::tests::a_signal_during_a_gate_wait_aborts_and_releases_the_lock"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, None, FAILED_MARKER); + return; + } + let _guard = env_lock(); + let run = launch_parked_at_gate(PhaseId::new(1310)); + signal_supervisor(&run, libc::SIGTERM); + let ended = poll_until(Duration::from_secs(5), || { + matches!( + lock::holder_status(&run.root, run.phase), + lock::HolderStatus::NoHolder + ) && devflow_core::supervisor::probe(&run.n.socket_path) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }); + assert!( + ended, + "within 5 s of SIGTERM the lock is released and the socket gone: {}", + run.events() + ); + let state = run.state(); + assert!(state.stopped, "state.stopped after a signal at the gate"); + assert_eq!(state.stop_reason.as_deref(), Some(ADVANCE_STOP_REASON)); + assert_eq!( + stage_launched_count(&run.root, run.phase), + 1, + "{}", + run.events() + ); + let stopped = events_named(&run.root, "advance_stopped"); + assert_eq!(stopped.len(), 1, "{}", run.events()); + assert_eq!(stopped[0]["safe_point"], 4, "{}", stopped[0]); + } + + /// The V-15 control: with no signal the gate wait holds (lock held, N in + /// `gate_wait`) after 3 s, and answering the gate lets it proceed: an + /// abort answer releases the lock and ends the supervisor. + #[test] + fn without_a_signal_the_gate_wait_holds_until_answered() { + const NAME: &str = + "pipeline_launch::tests::without_a_signal_the_gate_wait_holds_until_answered"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, None, FAILED_MARKER); + return; + } + let _guard = env_lock(); + let run = launch_parked_at_gate(PhaseId::new(1311)); + std::thread::sleep(Duration::from_secs(3)); + assert!( + matches!( + lock::holder_status(&run.root, run.phase), + lock::HolderStatus::Live { .. } + ), + "control: still parked with the lock held after 3 s" + ); + assert!( + matches!( + devflow_core::supervisor::probe(&run.n.socket_path), + Ok(devflow_core::supervisor::Liveness::Alive(ref reply)) if reply.owner == "gate_wait" + ), + "control: N still answers gate_wait" + ); + assert!(!run.state().stopped, "control: not stopped while waiting"); + Gates::respond( + &run.root, + run.phase, + Stage::Define, + &devflow_core::gates::GateResponse { + approved: false, + note: Some("abort: control answer".to_string()), + responded_by: Some("test".to_string()), + }, + ) + .unwrap(); + let ended = poll_until(HANDOFF_STEP_WAIT, || { + matches!( + lock::holder_status(&run.root, run.phase), + lock::HolderStatus::NoHolder + ) && devflow_core::supervisor::probe(&run.n.socket_path) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + }); + assert!( + ended, + "control: the answered gate proceeds: {}", + run.events() + ); + assert_eq!(events_named(&run.root, "workflow_aborted").len(), 1); + assert!(events_named(&run.root, "advance_stopped").is_empty()); + } + + // ---- CR-01 (50-REVIEW.md): a stop before a `--yes-ship` merge ---- + + /// Whether `branch` exists in the repo at `root`. + fn branch_exists(root: &Path, branch: &str) -> bool { + devflow_core::test_support::git_command(root) + .args(["rev-parse", "--verify", "--quiet", branch]) + .status() + .unwrap() + .success() + } + + /// The body of the `--yes-ship` pair: stage N is Ship with `--yes-ship` + /// and a feature branch finalization can merge, and its advance is held + /// at the configured point. With `stop`, a socket stop is taken there + /// (owner `advancing`, so it is only flagged) before the release: the + /// advance must ship nothing, record the stop at `safe_point` and answer + /// the caller `stopped`. Without it (the control) the run ships. + fn held_yes_ship_advance(phase: PhaseId, stop: bool, safe_point: u8) { + let _guard = env_lock(); + let branch = format!("feature/phase-{}", phase.padded()); + let run = HandoffRun::launch_with(phase, |root, state| { + assert!( + devflow_core::test_support::git_command(root) + .args(["branch", &branch, "develop"]) + .status() + .unwrap() + .success() + ); + state.stage = Stage::Ship; + state.yes_ship = true; + state.stop_until = None; + }); + run.go(); + run.wait_held(); + if stop { + let (accepted, last) = shutdown_accepted_then( + &run.n.socket_path, + || run.release(), + Duration::from_secs(30), + ); + assert_eq!(accepted["owner"], "advancing", "{accepted}"); + assert!( + events_named(&run.root, "workflow_shipped").is_empty(), + "a stop accepted before the --yes-ship merge must ship nothing: {}", + run.events() + ); + assert!( + branch_exists(&run.root, &branch), + "no after-ship hook may run: the feature branch is kept" + ); + let state = run.state(); + assert!( + state.stopped, + "state.stopped after a stop at safe point {safe_point}" + ); + assert_eq!(state.stop_reason.as_deref(), Some(ADVANCE_STOP_REASON)); + assert!(!state.gate_pending, "no gate is left pending"); + assert!(!Gates::response_path(&run.root, run.phase, Stage::Ship).exists()); + let stopped = events_named(&run.root, "advance_stopped"); + assert_eq!(stopped.len(), 1, "{}", run.events()); + assert_eq!(stopped[0]["safe_point"], safe_point, "{}", stopped[0]); + assert_eq!( + last["stopped"], true, + "the stop's caller reads stopped: {last}" + ); + } else { + run.release(); + assert!( + poll_until(HANDOFF_STEP_WAIT, || events_named( + &run.root, + "workflow_shipped" + ) + .len() + == 1), + "control: without a stop the --yes-ship run ships: {}", + run.events() + ); + assert!( + !branch_exists(&run.root, &branch), + "control: the after-ship hooks ran (the feature branch is cleaned up)" + ); + assert!(events_named(&run.root, "advance_stopped").is_empty()); + } + } + + /// CR-01, safe point 5: a stop accepted after safe point 1 and before + /// `--yes-ship` writes its approval stops the run there. The pre-written + /// approval would otherwise beat the stop in the gate wait and let + /// finalization merge, bump, tag and push. + #[test] + fn a_stop_after_safe_point_1_is_honoured_before_the_yes_ship_approval() { + const NAME: &str = "pipeline_launch::tests::a_stop_after_safe_point_1_is_honoured_before_the_yes_ship_approval"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("after_safe_point_1"), SUCCESS_MARKER); + return; + } + held_yes_ship_advance(PhaseId::new(1314), true, 5); + } + + /// The control for safe point 5: the same hold and no stop, and the + /// `--yes-ship` run ships. + #[test] + fn without_a_stop_after_safe_point_1_the_yes_ship_run_ships() { + const NAME: &str = + "pipeline_launch::tests::without_a_stop_after_safe_point_1_the_yes_ship_run_ships"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("after_safe_point_1"), SUCCESS_MARKER); + return; + } + held_yes_ship_advance(PhaseId::new(1315), false, 5); + } + + /// CR-01, safe point 6: a stop accepted after the Ship gate was answered + /// and before finalization stops the run before the merge. + #[test] + fn a_stop_before_finalization_is_honoured_before_the_merge() { + const NAME: &str = + "pipeline_launch::tests::a_stop_before_finalization_is_honoured_before_the_merge"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_finalization"), SUCCESS_MARKER); + return; + } + held_yes_ship_advance(PhaseId::new(1316), true, 6); + } + + /// The control for safe point 6: the same hold and no stop, and + /// finalization ships. + #[test] + fn without_a_stop_before_finalization_the_run_ships() { + const NAME: &str = + "pipeline_launch::tests::without_a_stop_before_finalization_the_run_ships"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_finalization"), SUCCESS_MARKER); + return; + } + held_yes_ship_advance(PhaseId::new(1317), false, 6); + } + + /// R-10: a SIGKILL of N between N+1's publish and N's unlink (N parked + /// at `after_publish`) leaves the state naming the live N+1, and N's + /// leftover socket probes Stale with a sidecar whose launch id differs + /// from the state's. + #[test] + fn a_kill_between_the_next_publish_and_the_old_unlink_leaves_state_on_the_live_next_supervisor() + { + const NAME: &str = "pipeline_launch::tests::a_kill_between_the_next_publish_and_the_old_unlink_leaves_state_on_the_live_next_supervisor"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("after_publish"), SUCCESS_MARKER); + return; + } + let _guard = env_lock(); + let run = HandoffRun::launch(PhaseId::new(1312)); + run.go(); + run.wait_held(); + signal_supervisor(&run, libc::SIGKILL); + assert!( + poll_until(HANDOFF_STEP_WAIT, || matches!( + devflow_core::supervisor::probe(&run.n.socket_path), + Ok(devflow_core::supervisor::Liveness::Stale) + )), + "N's leftover socket probes Stale after the kill" + ); + let state = run.state(); + let next = state.supervisor.expect("the state names a supervisor"); + assert_ne!(next.launch_id, run.n.launch_id, "the state names N+1"); + match devflow_core::supervisor::probe(&next.socket_path) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => { + assert_eq!(reply.launch_id, next.launch_id); + } + other => panic!("N+1 must be Alive after N's kill, got {other:?}"), + } + let leftover = + devflow_core::supervisor::read_sidecar(&run.n.socket_path.with_extension("json")) + .expect("N's sidecar is left behind"); + assert_ne!( + leftover.launch_id, next.launch_id, + "N's leftover sidecar names a launch other than the state's" + ); + assert_eq!(leftover.launch_id, run.n.launch_id); + } + + // ---- 50-23: every launch failure path has a defined end state ---- + + /// A previous stage's supervisor record whose socket does not exist and + /// whose agent pgid does not fit `pid_t`, so no reaper can signal it. + fn fabricated_prior_record(dir: &Path) -> SupervisorRecord { + SupervisorRecord::new( + dir.join("prior-stage-never-bound.sock"), + "priorstage0000000000".to_string(), + 4_294_967_000, + 1, + String::new(), + ) + } + + /// Names which run of `a_failed_monitor_spawn_clears_the_supervisor_record` + /// a child is (`fail` or `ok`). Not a DevFlow setting, so it carries no + /// `DEVFLOW_` prefix the doc check would require documented. + const SPAWN_CASE_ENV: &str = "P50_23_SPAWN_CASE"; + + /// WR-04's supervisor equivalent (T-50-62): a launch whose + /// `spawn_monitor` fails before spawning anything (its socket path is + /// over `sun_path`, so `SocketPath::for_launch` refuses it) clears the + /// previous stage's record in memory and on disk, and leaves no socket or + /// sidecar under the cache dir. Control: the same launch under a short + /// cache dir records a new record with a new launch id. + #[test] + fn a_failed_monitor_spawn_clears_the_supervisor_record() { + const NAME: &str = + "pipeline_launch::tests::a_failed_monitor_spawn_clears_the_supervisor_record"; + if !devflow_core::test_support::in_child_test(NAME) { + for case in ["fail", "ok"] { + let path_dir = go_waiting_claude_dir(); + let short = crate::test_support::hermetic_cache_dir(); + let cache = if case == "fail" { + let long = short.path().join("c".repeat(80)); + std::fs::create_dir_all(&long).unwrap(); + long + } else { + short.path().to_path_buf() + }; + let env: Vec<(&str, &std::ffi::OsStr)> = vec![ + ("DEVFLOW_CACHE_DIR", cache.as_os_str()), + (SPAWN_CASE_ENV, std::ffi::OsStr::new(case)), + ]; + let output = + devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &env); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test( + &output, NAME, + ); + } + return; + } + let _guard = env_lock(); + let cache = PathBuf::from( + std::env::var_os("DEVFLOW_CACHE_DIR").expect("the parent passes DEVFLOW_CACHE_DIR"), + ); + let fail = std::env::var(SPAWN_CASE_ENV).expect("the parent names the case") == "fail"; + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + init_repo(root); + let phase = PhaseId::new(2301); + let mut state = State::new( + phase, + AgentKind::Claude, + Mode::Supervise, + root.to_path_buf(), + ); + state.legacy_claude_launch = true; + let prior = fabricated_prior_record(root); + state.supervisor = Some(prior.clone()); + workflow::save_state(&state).unwrap(); + + let result = launch_stage(&mut state, None, None); + let _reap_guard = ReapMonitorOnDrop::after_launch(&state); + let reloaded = workflow::load_state(root, phase).unwrap(); + if fail { + let err = result.expect_err("a socket path over sun_path cannot be bound"); + assert!( + err.to_string().contains("sun_path"), + "the error names the socket path bound: {err}" + ); + assert_eq!( + state.supervisor, None, + "the failed spawn clears the prior record in memory" + ); + assert_eq!( + reloaded.supervisor, None, + "the failed spawn clears the prior record on disk" + ); + let left: Vec = + std::fs::read_dir(devflow_core::registry::socket_dir_in(&cache)) + .map(|entries| entries.filter_map(|e| Some(e.ok()?.path())).collect()) + .unwrap_or_default(); + assert!(left.is_empty(), "no socket or sidecar is left: {left:?}"); + } else { + result.expect("control: the launch succeeds under a short cache dir"); + let record = state + .supervisor + .clone() + .expect("control: the launch records its supervisor"); + assert_ne!( + record.launch_id, prior.launch_id, + "control: a new launch id replaces the prior one" + ); + assert_eq!(reloaded.supervisor.as_ref(), Some(&record)); + let outcome = devflow_core::supervisor::request_shutdown( + &record.socket_path, + SUPERVISOR_STOP_WAIT, + ); + assert!( + matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "control: the launched supervisor stops through its socket: {outcome:?}" + ); + } + } + + /// `source`'s non-test lines: the part before the first line that is + /// exactly `mod tests {`, without its whole-line `//` comments. A + /// trailing comment stays, as part of its code line. + fn non_test_code_lines(source: &str) -> Vec<&str> { + source + .lines() + .take_while(|line| *line != "mod tests {") + .filter(|line| !line.trim_start().starts_with("//")) + .collect() + } + + /// The lines of the item whose header line starts with `header`, through + /// the next line starting with `closing`; `None` when no line starts with + /// `header`. + fn item_lines<'a>(lines: &[&'a str], header: &str, closing: &str) -> Option> { + let start = lines.iter().position(|line| line.starts_with(header))?; + let mut item = vec![lines[start]]; + for line in &lines[start + 1..] { + item.push(*line); + if line.starts_with(closing) { + break; + } + } + Some(item) + } + + const CLEAR_SPAWN_NOT_FOUND: &str = + "spawn_agent_and_record not found in the non-test code (or under 40 lines)"; + const CLEAR_SPAWN_CLEARS: &str = + "spawn_agent_and_record clears the supervisor record in its own body"; + const CLEAR_SPAWN_NO_HELPER: &str = "spawn_agent_and_record never calls record_failed_launch"; + const CLEAR_REFUSE_NOT_FOUND: &str = + "refuse_launch not found in the non-test code (or under 3 lines)"; + const CLEAR_REFUSE_NO_CLEAR: &str = "refuse_launch does not clear the supervisor record"; + + /// 50-13's rule as a pure scan over `source` (R6-G4): in the non-test, + /// comment-stripped code, `spawn_agent_and_record` never clears the + /// supervisor record itself (only its failure helper does) and calls + /// `record_failed_launch`, and `refuse_launch` clears the record. + fn launch_record_clear_violations(source: &str) -> Vec { + let lines = non_test_code_lines(source); + let has = |item: &[&str], needle: &str| item.iter().any(|line| line.contains(needle)); + let mut violations = Vec::new(); + match item_lines(&lines, "fn spawn_agent_and_record(", "}").filter(|item| item.len() >= 40) + { + None => violations.push(CLEAR_SPAWN_NOT_FOUND.to_string()), + Some(item) => { + if has(&item, "supervisor = None") { + violations.push(CLEAR_SPAWN_CLEARS.to_string()); + } + if !has(&item, "record_failed_launch(") { + violations.push(CLEAR_SPAWN_NO_HELPER.to_string()); + } + } + } + match item_lines(&lines, "fn refuse_launch(", "}").filter(|item| item.len() >= 3) { + None => violations.push(CLEAR_REFUSE_NOT_FOUND.to_string()), + Some(item) if !has(&item, "supervisor = None") => { + violations.push(CLEAR_REFUSE_NO_CLEAR.to_string()); + } + Some(_) => {} + } + violations + } + + /// T-50-62 (permanent source scan, R6-G4): the supervisor record is + /// cleared only on a launch failure path. The real file yields no + /// violation; planted sources, each padded to the floors so it carries + /// one defect, yield exactly that defect's violation. + #[test] + fn the_supervisor_record_is_cleared_only_on_launch_failure_paths() { + let real = launch_record_clear_violations(include_str!("pipeline_launch.rs")); + assert!(real.is_empty(), "pipeline_launch.rs: {real:?}"); + + let pad = |n: usize| " let _filler = 0;\n".repeat(n); + let spawn = |body: &str| format!("fn spawn_agent_and_record(\n{}{body}}}\n", pad(40)); + let refuse = |body: &str| { + format!( + "fn refuse_launch(state: &mut State) {{\n{}{body}}}\n", + pad(2) + ) + }; + let good_spawn = spawn(" return Err(failed(record_failed_launch(state), err));\n"); + let good_refuse = refuse(" state.supervisor = None;\n"); + let only = |violation: &str| vec![violation.to_string()]; + + assert_eq!( + launch_record_clear_violations(&format!("{good_spawn}{good_refuse}")), + Vec::::new(), + "a correct layout" + ); + assert_eq!( + launch_record_clear_violations(&format!( + "{}{good_refuse}", + spawn(" record_failed_launch(state)?;\n state.supervisor = None;\n") + )), + only(CLEAR_SPAWN_CLEARS) + ); + assert_eq!( + launch_record_clear_violations(&format!( + "{}{good_refuse}", + spawn(" record_failed_launch(state)?; // never: state.supervisor = None\n") + )), + only(CLEAR_SPAWN_CLEARS), + "a trailing comment is part of its code line" + ); + assert_eq!( + launch_record_clear_violations(&format!( + "{}{good_refuse}", + spawn(" // record_failed_launch(state)?;\n") + )), + only(CLEAR_SPAWN_NO_HELPER), + "a commented-out call is not a call" + ); + assert_eq!( + launch_record_clear_violations(&format!("{good_spawn}{}", refuse(""))), + only(CLEAR_REFUSE_NO_CLEAR) + ); + assert_eq!( + launch_record_clear_violations(&format!("mod tests {{\n{good_spawn}{good_refuse}")), + vec![ + CLEAR_SPAWN_NOT_FOUND.to_string(), + CLEAR_REFUSE_NOT_FOUND.to_string() + ], + "functions only after the test-module cut are not found" + ); + assert_eq!( + launch_record_clear_violations(&format!( + "fn spawn_agent_and_record(\n{} record_failed_launch(state)?;\n}}\n{good_refuse}", + pad(5) + )), + only(CLEAR_SPAWN_NOT_FOUND), + "a body under its floor" + ); + } + + /// A launch context's supervisor, spawned straight through + /// `spawn_monitor` on the supervise-only harness entry for a fake agent + /// that waits for `go` (never created), with a reaper for it. The state + /// on disk holds `prior` and is left untouched. + struct PublishFixture { + _dir: tempfile::TempDir, + root: PathBuf, + phase: PhaseId, + state: State, + prior: SupervisorRecord, + published: Option, + reaper: Option, + } + + impl PublishFixture { + fn spawn(phase: PhaseId) -> Self { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_path_buf(); + init_repo(&root); + let mut state = State::new(phase, AgentKind::Claude, Mode::Supervise, root.clone()); + state.legacy_claude_launch = true; + let prior = fabricated_prior_record(&root); + state.supervisor = Some(prior.clone()); + workflow::save_state(&state).unwrap(); + let ctx = launch_context().expect("the parent passes DEVFLOW_CACHE_DIR"); + let published = monitor::spawn_monitor( + &state, + "claude", + &[], + &[], + monitor::MonitorLaunch::Legacy, + &ctx, + ) + .expect("the next supervisor publishes"); + let mut holder = state.clone(); + holder.supervisor = Some(published.record.clone()); + let reaper = Some(ReapMonitorOnDrop::after_launch(&holder)); + Self { + _dir: dir, + root, + phase, + state, + prior, + published: Some(published), + reaper, + } + } + + fn take(&mut self) -> monitor::Published { + self.published.take().expect("published once") + } + + fn reloaded(&self) -> State { + workflow::load_state(&self.root, self.phase).expect("the state loads") + } + } + + impl Drop for PublishFixture { + fn drop(&mut self) { + drop(self.reaper.take()); + } + } + + /// A save that always fails, as a full disk would. + fn failing_save(_: &State) -> Result<(), CliError> { + Err(CliError::Workflow( + devflow_core::workflow::WorkflowError::Io(std::io::Error::other( + "injected save failure: no space left on device", + )), + )) + } + + fn probes_alive(socket: &Path) -> bool { + matches!( + devflow_core::supervisor::probe(socket), + Ok(devflow_core::supervisor::Liveness::Alive(_)) + ) + } + + fn probes_gone(socket: &Path) -> bool { + devflow_core::supervisor::probe(socket) + .is_ok_and(|liveness| liveness == devflow_core::supervisor::Liveness::Gone) + } + + /// The parent half of a publish-save test: a child on the supervise-only + /// entry whose fake agent waits for `go` in its working directory. + fn run_publish_child(name: &str) { + let path_dir = go_waiting_claude_dir(); + let output = run_hermetic_test_in_child(name, path_dir.path(), &[]); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, name); + } + + /// RV-08 (V-38, T-50-63): a failed publish save whose shutdown of the new + /// supervisor is not confirmed (its recorded socket refuses the connect) + /// keeps the new record in memory, saves nothing over the file's record, + /// and names `devflow stop --phase N --launch `. The new supervisor + /// is still running. Its opposite is the next test, whose confirmed + /// shutdown clears the record and names no repair. + #[test] + fn a_failed_publish_save_whose_shutdown_is_not_confirmed_names_the_stop_launch_repair() { + const NAME: &str = "pipeline_launch::tests::a_failed_publish_save_whose_shutdown_is_not_confirmed_names_the_stop_launch_repair"; + if !devflow_core::test_support::in_child_test(NAME) { + run_publish_child(NAME); + return; + } + let _guard = env_lock(); + let mut fixture = PublishFixture::spawn(PhaseId::new(2302)); + let mut published = fixture.take(); + let real = published.record.socket_path.clone(); + let new_id = published.record.launch_id.clone(); + let refused = fixture.root.join("refused.sock"); + drop(std::os::unix::net::UnixListener::bind(&refused).expect("bind then drop")); + published.record.socket_path = refused.clone(); + + let err = publish_record(&mut fixture.state, published, failing_save) + .err() + .expect("a failed save is an error"); + let text = err.to_string(); + assert!( + text.contains("injected save failure"), + "names the save error: {text}" + ); + let repair = format!("devflow stop --phase {} --launch {new_id}", fixture.phase); + assert!(text.contains(&repair), "names `{repair}`: {text}"); + assert!( + text.contains(&refused.display().to_string()), + "names the socket: {text}" + ); + assert_eq!( + fixture.state.supervisor.as_ref().map(|r| &r.launch_id), + Some(&new_id), + "the in-memory record still names the new launch" + ); + assert_eq!( + fixture.reloaded().supervisor.as_ref(), + Some(&fixture.prior), + "nothing was saved over the file's record" + ); + assert!(probes_alive(&real), "the new supervisor is still running"); + let outcome = devflow_core::supervisor::request_shutdown(&real, SUPERVISOR_STOP_WAIT); + assert!( + matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "the operator's handle still ends it: {outcome:?}" + ); + } + + /// R-1 step 4 (T-50-63): a failed publish save shuts the new supervisor + /// down through its socket before returning the error: its socket reads + /// Gone, its agent group is gone, and the record does not name it. + #[test] + fn a_failed_publish_save_shuts_the_new_supervisor_down() { + const NAME: &str = + "pipeline_launch::tests::a_failed_publish_save_shuts_the_new_supervisor_down"; + if !devflow_core::test_support::in_child_test(NAME) { + run_publish_child(NAME); + return; + } + let _guard = env_lock(); + let mut fixture = PublishFixture::spawn(PhaseId::new(2303)); + let published = fixture.take(); + let record = published.record.clone(); + + let err = publish_record(&mut fixture.state, published, failing_save) + .err() + .expect("a failed save is an error"); + let text = err.to_string(); + assert!( + text.contains("injected save failure"), + "names the save error: {text}" + ); + assert!( + text.contains("shut down"), + "names the shutdown of the new supervisor: {text}" + ); + assert!( + !text.contains("--launch"), + "a confirmed shutdown needs no repair line: {text}" + ); + assert!( + poll_until(Duration::from_secs(5), || probes_gone(&record.socket_path)), + "the new supervisor's socket reads Gone" + ); + assert!( + poll_until(Duration::from_secs(5), || !group_alive(record.agent_pgid)), + "the new supervisor's agent group is gone" + ); + assert_ne!( + fixture.state.supervisor.as_ref().map(|r| &r.launch_id), + Some(&record.launch_id), + "the record does not name the shut-down supervisor" + ); + } + + /// The control for the failed-save pair: with `workflow::save_state` the + /// publish hands off, the reloaded state names the new launch and the + /// new supervisor probes Alive. + #[test] + fn a_successful_publish_save_records_and_keeps_the_new_supervisor() { + const NAME: &str = "pipeline_launch::tests::a_successful_publish_save_records_and_keeps_the_new_supervisor"; + if !devflow_core::test_support::in_child_test(NAME) { + run_publish_child(NAME); + return; + } + let _guard = env_lock(); + let mut fixture = PublishFixture::spawn(PhaseId::new(2304)); + let published = fixture.take(); + let record = published.record.clone(); + + let outcome = publish_record(&mut fixture.state, published, workflow::save_state) + .expect("control: the save succeeds"); + assert!( + matches!(outcome, PublishOutcome::HandedOff), + "control: a running successor is handed off" + ); + assert_eq!( + fixture.reloaded().supervisor.as_ref(), + Some(&record), + "control: the reloaded state names the new launch" + ); + assert!(!fixture.reloaded().stopped, "control: not stopped"); + assert!( + probes_alive(&record.socket_path), + "control: the new supervisor is running" + ); + let shutdown = + devflow_core::supervisor::request_shutdown(&record.socket_path, SUPERVISOR_STOP_WAIT); + assert!( + matches!( + shutdown, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "{shutdown:?}" + ); + } + + /// The next supervisor a held stage N spawned: the one sweep entry whose + /// launch id is not N's, with its sidecar. + fn next_supervisor(run: &HandoffRun) -> (PathBuf, devflow_core::supervisor::Sidecar) { + let found: Vec<_> = devflow_core::supervisor::sweep_in(&run.cache) + .into_iter() + .filter_map(|entry| { + let sidecar = entry.sidecar?; + (sidecar.launch_id != run.n.launch_id).then_some((entry.socket_path, sidecar)) + }) + .collect(); + match <[_; 1]>::try_from(found) { + Ok([one]) => one, + Err(other) => panic!("exactly one next supervisor at the hold: {other:?}"), + } + } + + /// Release N, then wait for its socket to read Gone (its save, probe and + /// `finish` are done). + fn release_and_wait_for_n(run: &HandoffRun) { + run.release(); + assert!( + poll_until(SUPERVISOR_STOP_WAIT, || probes_gone(&run.n.socket_path)), + "N finishes once released: {}", + run.events() + ); + } + + /// T-50-64: a stop sent straight to N+1 while N is held before its + /// publish save (so N+1 ends on 50-13's no-write path, the state still + /// naming N) is recorded by N's post-save re-probe: `stopped` with this + /// plan's reason, the state naming N+1, which reads Gone with no agent + /// left. + #[test] + fn a_stop_sent_straight_to_the_next_supervisor_before_its_record_is_saved_is_recorded() { + const NAME: &str = "pipeline_launch::tests::a_stop_sent_straight_to_the_next_supervisor_before_its_record_is_saved_is_recorded"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_publish"), SUCCESS_MARKER); + return; + } + let _guard = env_lock(); + let run = HandoffRun::launch(PhaseId::new(2305)); + run.go(); + run.wait_held(); + let (next_socket, next) = next_supervisor(&run); + let outcome = + devflow_core::supervisor::request_shutdown(&next_socket, SUPERVISOR_STOP_WAIT); + assert!( + matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "N+1 stops: {outcome:?}" + ); + release_and_wait_for_n(&run); + + let state = run.state(); + assert_eq!( + state.supervisor.as_ref().map(|r| &r.launch_id), + Some(&next.launch_id), + "the state names N+1" + ); + assert!( + state.stopped, + "the stop is recorded (stop_reason {:?})", + state.stop_reason + ); + assert_eq!( + state.stop_reason.as_deref(), + Some(NEXT_STOPPED_BEFORE_SAVE_REASON) + ); + assert!(probes_gone(&next_socket), "N+1 reads Gone"); + assert!( + poll_until(Duration::from_secs(5), || !group_alive(next.agent_pgid)), + "no agent of N+1 survives" + ); + } + + /// The control: the same held publish with no stop hands off, so the + /// state names a running N+1 and is not stopped. + #[test] + fn without_a_stop_to_the_next_supervisor_the_held_publish_hands_off() { + const NAME: &str = "pipeline_launch::tests::without_a_stop_to_the_next_supervisor_the_held_publish_hands_off"; + if !devflow_core::test_support::in_child_test(NAME) { + run_handoff_child(NAME, Some("before_publish"), SUCCESS_MARKER); + return; + } + let _guard = env_lock(); + let run = HandoffRun::launch(PhaseId::new(2306)); + run.go(); + run.wait_held(); + let (next_socket, next) = next_supervisor(&run); + release_and_wait_for_n(&run); + + let state = run.state(); + assert_eq!( + state.supervisor.as_ref().map(|r| &r.launch_id), + Some(&next.launch_id), + "control: the state names N+1" + ); + assert!( + !state.stopped, + "control: not stopped ({:?})", + state.stop_reason + ); + assert!(probes_alive(&next_socket), "control: N+1 runs on"); + } + + /// The stopped-outcome hold variable (R-6E): the path a stopped + /// supervisor waits on between its agent's exit and its outcome decision. + const STOPPED_OUTCOME_HOLD_ENV: &str = "DEVFLOW_TEST_STOPPED_OUTCOME_HOLD"; + + /// Names which run of the kill-grace test a child is (`a` or `b`). Not a + /// DevFlow setting, so no `DEVFLOW_` prefix. + const KILL_GRACE_RUN_ENV: &str = "P50_23_KILL_GRACE_RUN"; + + /// The marker N+1's fake agent leaves when SIGTERM reaches it. + const TERM_SEEN: &str = "term-seen"; + + /// [`handoff_claude_dir`] with a stage N+1 agent that traps SIGTERM by + /// leaving [`TERM_SEEN`] in its working directory and running on, so its + /// supervisor's kill grace ends in SIGKILL. + fn term_trapping_handoff_claude_dir() -> tempfile::TempDir { + let path_dir = handoff_claude_dir(SUCCESS_MARKER); + let script = format!( + "#!/bin/sh\ncase \"$1\" in --version) exit 0 ;; esac\n\ + if [ ! -e first-done ]; then\n : > first-done\n gate=go\n\ + else\n trap ': > {TERM_SEEN}' TERM\n gate=go2\nfi\n\ + i=0\nwhile [ ! -e \"$gate\" ] && [ \"$i\" -lt 1200 ]; do sleep 0.05; i=$((i+1)); done\n\ + printf '%s\\n' '{SUCCESS_MARKER}'\nexit 0\n" + ); + std::fs::write(path_dir.path().join("claude"), script).unwrap(); + path_dir + } + + /// Creates its path when dropped, so a failing kill-grace run never + /// leaves N+1 parked at the stopped-outcome hold. + struct ReleaseOnDrop(PathBuf); + + impl Drop for ReleaseOnDrop { + fn drop(&mut self) { + let _ = std::fs::write(&self.0, ""); + } + } + + /// Wait (bounded) for `path` to exist. + fn wait_for_file(path: &Path, within: Duration) -> bool { + poll_until(within, || path.exists()) + } + + /// R6-E (V-50, T-50-64): a stop N+1 has accepted but not carried out + /// when N saves its record is recorded by N+1 itself. Ordered only by + /// events: N held at `before_publish`, the stop sent to N+1, N+1's + /// agent marker, N+1 parked at the stopped-outcome hold (owner still + /// `agent_running`), then the two releases. Run (a) releases N first: + /// N saves, probes N+1 (running) and hands off with no stop written, + /// then N+1 records its own stop. Run (b), the in-test control, releases + /// N+1 first: it ends on the no-write path and N's re-probe records this + /// plan's reason. The release order alone selects the end state. + #[test] + fn a_next_supervisor_still_in_its_kill_grace_at_publish_records_its_own_stop() { + const NAME: &str = "pipeline_launch::tests::a_next_supervisor_still_in_its_kill_grace_at_publish_records_its_own_stop"; + if !devflow_core::test_support::in_child_test(NAME) { + for run in ["a", "b"] { + let path_dir = term_trapping_handoff_claude_dir(); + let hold_dir = tempfile::Builder::new() + .prefix("dfh") + .tempdir_in("/tmp") + .unwrap(); + let n_hold = std::ffi::OsString::from(format!( + "before_publish:{}", + hold_dir.path().join("release").display() + )); + let k_file = hold_dir.path().join("k-release"); + let env: Vec<(&str, &std::ffi::OsStr)> = vec![ + (TEST_MONITOR_FULL_ENV, std::ffi::OsStr::new("1")), + ("DEVFLOW_GATE_TIMEOUT_SECS", std::ffi::OsStr::new("120")), + (HOLD_ENV, n_hold.as_os_str()), + (STOPPED_OUTCOME_HOLD_ENV, k_file.as_os_str()), + (KILL_GRACE_RUN_ENV, std::ffi::OsStr::new(run)), + ]; + let output = run_hermetic_test_in_child(NAME, path_dir.path(), &env); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test( + &output, NAME, + ); + } + return; + } + let _guard = env_lock(); + let release_first_n = std::env::var(KILL_GRACE_RUN_ENV).expect("the run") == "a"; + let k_file = PathBuf::from( + std::env::var_os(STOPPED_OUTCOME_HOLD_ENV).expect("the parent passes the hold"), + ); + let k_held = PathBuf::from(format!("{}.held", k_file.display())); + let step = SUPERVISOR_STOP_WAIT; + let run = HandoffRun::launch(PhaseId::new(if release_first_n { 2307 } else { 2308 })); + let _release_k = ReleaseOnDrop(k_file.clone()); + run.go(); + run.wait_held(); + let (next_socket, next) = next_supervisor(&run); + let stopper = { + let socket = next_socket.clone(); + std::thread::spawn(move || { + devflow_core::supervisor::request_shutdown(&socket, Duration::from_secs(60)) + }) + }; + assert!( + wait_for_file(&run.root.join(TERM_SEEN), Duration::from_secs(5)), + "N+1 accepted the stop and signalled its agent group" + ); + assert!( + wait_for_file(&k_held, step), + "stopped-outcome hold never reached" + ); + let mut owner = None; + let answered = poll_until(step, || { + match devflow_core::supervisor::probe(&next_socket) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => { + owner = Some(reply.owner); + true + } + Ok(other) => panic!("N+1 must be Alive at the hold, found {other:?}"), + Err(_) => false, + } + }); + assert!(answered, "N+1 answers a ping while parked at the hold"); + assert_eq!( + owner.as_deref(), + Some("agent_running"), + "expected owner agent_running, found {owner:?}" + ); + + let ended = |socket: &Path| poll_until(step, || probes_gone(socket)); + if release_first_n { + release_and_wait_for_n(&run); + let state = run.state(); + assert_eq!( + state.supervisor.as_ref().map(|r| &r.launch_id), + Some(&next.launch_id), + "N saved N+1's record" + ); + assert!( + !state.stopped, + "N handed off and wrote no stop: found stopped {} with stop_reason {:?}", + state.stopped, state.stop_reason + ); + std::fs::write(&k_file, "").unwrap(); + assert!(ended(&next_socket), "N+1 finishes once released"); + } else { + std::fs::write(&k_file, "").unwrap(); + assert!(ended(&next_socket), "N+1 finishes once released"); + release_and_wait_for_n(&run); + } + let outcome = stopper.join().expect("the stopper thread"); + assert!( + matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "the stop sent to N+1 reads Stopped: {outcome:?}" + ); + let state = run.state(); + assert_eq!( + state.supervisor.as_ref().map(|r| &r.launch_id), + Some(&next.launch_id), + "the state names N+1" + ); + assert!(state.stopped, "the stop is recorded"); + let expected = if release_first_n { + SUPERVISOR_STOP_REASON + } else { + NEXT_STOPPED_BEFORE_SAVE_REASON + }; + assert_eq!(state.stop_reason.as_deref(), Some(expected)); + assert!( + poll_until(Duration::from_secs(5), || !group_alive(next.agent_pgid)), + "no agent of N+1 survives" + ); + } + + // ---- 50-23 Task 3: a stop after safe point 3 is forwarded (RV-04) ---- + + /// Names which run of the forward test a child is (`a` to `d`). Not a + /// DevFlow setting, so no `DEVFLOW_` prefix. + const FORWARD_RUN_ENV: &str = "P50_23_FORWARD_RUN"; + + /// The parent half of a forward test: [`run_handoff_child`] with the run + /// label passed to the child. + fn run_forward_child(name: &str, hold: &str, label: &str) { + let path_dir = handoff_claude_dir(SUCCESS_MARKER); + let hold_dir = tempfile::Builder::new() + .prefix("dfh") + .tempdir_in("/tmp") + .unwrap(); + let hold_value = std::ffi::OsString::from(format!( + "{hold}:{}", + hold_dir.path().join("release").display() + )); + let env: Vec<(&str, &std::ffi::OsStr)> = vec![ + (TEST_MONITOR_FULL_ENV, std::ffi::OsStr::new("1")), + ("DEVFLOW_GATE_TIMEOUT_SECS", std::ffi::OsStr::new("120")), + (HOLD_ENV, hold_value.as_os_str()), + (FORWARD_RUN_ENV, std::ffi::OsStr::new(label)), + ]; + let output = run_hermetic_test_in_child(name, path_dir.path(), &env); + devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&output, name); + } + + /// RV-04 (V-35, T-50-73) and R5-I1 (V-48, T-50-83): a stop that reaches + /// stage N after safe point 3 (N is still `Advancing`, so it is flagged) + /// is forwarded by N to the successor it recorded, and N+1 stops and + /// records its own stop. Four runs, each a fresh child: (a) SIGTERM while + /// N is held at `before_record_save`; (b) at `after_publish`; (c) a + /// socket stop at `before_record_save`, whose caller is answered + /// `stopped`, not `handed_off`; (d) SIGTERM while N is held at + /// `before_handoff_mark`, after its advance returned and before it marks + /// itself handed off. + #[test] + fn a_stop_reaching_the_supervisor_after_safe_point_3_is_forwarded_to_the_next_supervisor() { + const NAME: &str = "pipeline_launch::tests::a_stop_reaching_the_supervisor_after_safe_point_3_is_forwarded_to_the_next_supervisor"; + if !devflow_core::test_support::in_child_test(NAME) { + for (label, hold) in [ + ("a", "before_record_save"), + ("b", "after_publish"), + ("c", "before_record_save"), + ("d", "before_handoff_mark"), + ] { + run_forward_child(NAME, hold, label); + } + return; + } + let _guard = env_lock(); + let label = std::env::var(FORWARD_RUN_ENV).expect("the parent names the run"); + let phase = match label.as_str() { + "a" => 2309, + "b" => 2310, + "c" => 2311, + _ => 2312, + }; + let run = HandoffRun::launch(PhaseId::new(phase)); + run.go(); + run.wait_held(); + if label == "c" { + let (accepted, last) = + shutdown_accepted_then(&run.n.socket_path, || run.release(), HANDOFF_STEP_WAIT); + assert_eq!(accepted["owner"], "advancing", "run {label}: {accepted}"); + assert_eq!( + last["stopped"], true, + "run {label}: N answers the forwarded stop `stopped`, so its caller does not \ + retry: {last}" + ); + } else { + signal_supervisor(&run, libc::SIGTERM); + // The control thread polls the signal pipe, so the stop is + // flagged well within this. + std::thread::sleep(Duration::from_millis(500)); + run.release(); + } + assert!( + poll_until(HANDOFF_STEP_WAIT, || probes_gone(&run.n.socket_path)), + "run {label}: N finishes: {}", + run.events() + ); + let next = run + .state() + .supervisor + .expect("the state names a supervisor"); + assert_ne!( + next.launch_id, run.n.launch_id, + "run {label}: the state names N+1" + ); + assert!( + poll_until(HANDOFF_STEP_WAIT, || probes_gone(&next.socket_path)), + "run {label}: N+1 was stopped through the forward: {}", + run.events() + ); + let state = run.state(); + assert!( + state.stopped, + "run {label}: N+1 recorded its stop (stop_reason {:?}): {}", + state.stop_reason, + run.events() + ); + assert_eq!( + state.stop_reason.as_deref(), + Some(SUPERVISOR_STOP_REASON), + "run {label}: N+1's own stop reason" + ); + assert!( + poll_until(Duration::from_secs(5), || !group_alive(next.agent_pgid)), + "run {label}: no agent of N+1 survives" + ); + let forwarded = events_named(&run.root, "stop_forwarded"); + assert_eq!(forwarded.len(), 1, "run {label}: {}", run.events()); + assert_eq!(forwarded[0]["confirmed"], true, "{}", forwarded[0]); + assert_eq!(forwarded[0]["launch_id"], next.launch_id.as_str()); + } + + /// The control for the forward test: the same `before_record_save` hold + /// and no stop leaves N+1 running and the run not stopped. + #[test] + fn without_a_stop_after_safe_point_3_the_next_supervisor_runs_on() { + const NAME: &str = + "pipeline_launch::tests::without_a_stop_after_safe_point_3_the_next_supervisor_runs_on"; + if !devflow_core::test_support::in_child_test(NAME) { + run_forward_child(NAME, "before_record_save", "control"); + return; + } + let _guard = env_lock(); + let run = HandoffRun::launch(PhaseId::new(2313)); + run.go(); + run.wait_held(); + run.release(); + assert!( + poll_until(HANDOFF_STEP_WAIT, || probes_gone(&run.n.socket_path)), + "control: N finishes: {}", + run.events() + ); + let state = run.state(); + let next = state + .supervisor + .clone() + .expect("control: the state names N+1"); + assert_ne!(next.launch_id, run.n.launch_id); + assert!( + !state.stopped, + "control: not stopped ({:?})", + state.stop_reason + ); + assert!(probes_alive(&next.socket_path), "control: N+1 runs on"); + assert!( + events_named(&run.root, "stop_forwarded").is_empty(), + "control: nothing was forwarded" + ); + let outcome = + devflow_core::supervisor::request_shutdown(&next.socket_path, SUPERVISOR_STOP_WAIT); + assert!( + matches!( + outcome, + Ok(devflow_core::supervisor::ShutdownOutcome::Stopped { .. }) + ), + "control: N+1 stops through its socket: {outcome:?}" + ); + } + + const ORDER_RUN_NOT_FOUND: &str = "run_monitor not found in the non-test code"; + const ORDER_NO_TAKE: &str = "run_monitor never calls take_successor"; + const ORDER_NO_HOLD: &str = "no before_handoff_mark hold after take_successor"; + const ORDER_NO_MARK: &str = "no handed-off marking after the before_handoff_mark hold"; + const ORDER_NO_CHECK: &str = "no stop_requested() check after the handed-off marking"; + + /// R5-I1 as a pure scan over `source`'s non-test, comment-stripped code: + /// in `run_monitor`, in this order, the first `take_successor(` line, a + /// later `test_hold("before_handoff_mark")`, a later + /// `OwnerPhase::HandedOff` and a later `stop_requested()`. Names the + /// missing function or the first step it cannot find. + fn handoff_order_violation(source: &str) -> Option { + let lines = non_test_code_lines(source); + let Some(body) = item_lines(&lines, "pub(crate) fn run_monitor(", "}") else { + return Some(ORDER_RUN_NOT_FOUND.to_string()); + }; + let steps = [ + ("take_successor(", ORDER_NO_TAKE), + ("test_hold(\"before_handoff_mark\")", ORDER_NO_HOLD), + ("OwnerPhase::HandedOff", ORDER_NO_MARK), + ("stop_requested()", ORDER_NO_CHECK), + ]; + let mut from = 0; + for (needle, missing) in steps { + match body[from..].iter().position(|line| line.contains(needle)) { + Some(at) => from += at + 1, + None => return Some(missing.to_string()), + } + } + None + } + + /// R5-I1 (permanent source scan, R6-G4): `run_monitor` checks for a stop + /// right after its handed-off marking, with the `before_handoff_mark` + /// hold just before that marking. Every planted source carries a decoy + /// marking before `take_successor(` and a decoy `stop_requested()` check + /// in another function. + #[test] + fn run_monitor_checks_for_a_stop_right_after_the_handed_off_marking() { + assert_eq!( + handoff_order_violation(include_str!("pipeline_launch.rs")), + None, + "pipeline_launch.rs" + ); + let take = " let next = take_successor();\n"; + let hold = " test_hold(\"before_handoff_mark\");\n"; + let mark = " token.enter(OwnerPhase::HandedOff);\n"; + let check = " if token.stop_requested() {}\n"; + let planted = |steps: &[&str]| { + format!( + "pub(crate) fn run_monitor(\n decoy.enter(OwnerPhase::HandedOff);\n{}}}\nfn other() {{\n if token.stop_requested() {{}}\n}}\n", + steps.concat() + ) + }; + let violation = |steps: &[&str]| handoff_order_violation(&planted(steps)); + let missing = |text: &str| Some(text.to_string()); + + assert_eq!( + violation(&[take, hold, mark, check]), + None, + "the planned layout" + ); + assert_eq!( + violation(&[take, check, hold, mark, check]), + None, + "the hint's check-plus-re-check layout" + ); + assert_eq!( + violation(&[take, check, mark]), + missing(ORDER_NO_HOLD), + "the pre-revision layout" + ); + assert_eq!( + violation(&[take, hold, check, mark]), + missing(ORDER_NO_CHECK), + "a hold whose only check precedes the marking (run (d)'s RED state)" + ); + assert_eq!( + violation(&[take, mark, hold, check]), + missing(ORDER_NO_MARK), + "a hold after the marking" + ); + assert_eq!( + violation(&[take, hold, mark, " // if token.stop_requested() {}\n"]), + missing(ORDER_NO_CHECK), + "a commented-out post-marking check" + ); + assert_eq!( + violation(&[take, hold, check]), + missing(ORDER_NO_MARK), + "no marking" + ); + assert_eq!( + handoff_order_violation(&format!("fn other() {{\n{take}{hold}{mark}{check}}}\n")), + missing(ORDER_RUN_NOT_FOUND), + "no run_monitor" + ); + } } diff --git a/crates/devflow-cli/src/pipeline_outcomes.rs b/crates/devflow-cli/src/pipeline_outcomes.rs index 2143e46d..c0fe1abd 100644 --- a/crates/devflow-cli/src/pipeline_outcomes.rs +++ b/crates/devflow-cli/src/pipeline_outcomes.rs @@ -1509,7 +1509,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::validate_gaps_does_not_advance_to_ship"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -1540,7 +1540,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::validate_missing_verdict_does_not_advance"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -1573,7 +1573,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::validate_pass_advances"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -1607,7 +1607,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::external_verify_agreement_advances_to_ship"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -1672,7 +1672,7 @@ mod tests { "pipeline_outcomes::tests::external_verify_disagreement_gates_immediately"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -1725,7 +1725,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::external_verify_no_verdict_gates_immediately"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2024,7 +2024,7 @@ mod tests { "pipeline_outcomes::tests::grafted_failure_shape_gates_instead_of_shipping"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2100,7 +2100,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::resource_killed_on_code_bumps_infra_failures_not_consecutive_failures"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2147,7 +2147,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::resource_killed_on_validate_bumps_infra_not_consecutive_failures"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2239,7 +2239,7 @@ mod tests { "pipeline_outcomes::tests::consecutive_failures_reaches_ceiling_across_cycles"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2306,7 +2306,7 @@ mod tests { "pipeline_outcomes::tests::healthy_multi_wave_progress_does_not_reach_the_ceiling"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2375,7 +2375,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::repeated_failure_without_new_commits_still_reaches_the_ceiling"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2566,7 +2566,7 @@ mod tests { // CYCLE 2 — `git` runs again and reports the same real count as // before. Nothing was committed in between, so this is not progress. let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child( + let out = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, root.as_os_str())], @@ -2638,7 +2638,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::phase_validate_failure_ceiling_gates_despite_trivial_commit_progress"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2722,7 +2722,7 @@ mod tests { "pipeline_outcomes::tests::validate_gate_message_leads_with_the_per_phase_total"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2812,7 +2812,7 @@ mod tests { .expect("sh must be resolvable on the parent PATH"); std::os::unix::fs::symlink(real_sh, path_dir.path().join("sh")) .expect("add sh to the isolated child PATH"); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2879,7 +2879,7 @@ mod tests { "pipeline_outcomes::tests::loop_back_reason_is_distinct_when_no_commit_baseline_exists"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -2946,7 +2946,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::phase_validate_failures_reset_on_operator_approval_at_the_ceiling_gate"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3035,7 +3035,7 @@ mod tests { "pipeline_outcomes::tests::a_passing_validate_at_the_ceiling_explains_why_it_gated"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3122,7 +3122,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::the_ceiling_reset_records_the_total_it_spent"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3209,7 +3209,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::phase_validate_failures_increment_saturates"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3575,7 +3575,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::evaluate_agent_result_with_real_git_and_empty_branch_still_reports_failed"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3631,7 +3631,7 @@ mod tests { "pipeline_outcomes::tests::mid_arc_loop_back_issues_plain_execute_command"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3672,7 +3672,7 @@ mod tests { "pipeline_outcomes::tests::genuine_gaps_loop_back_still_issues_gaps_only"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3737,7 +3737,7 @@ mod tests { "pipeline_outcomes::tests::worktree_mode_genuine_gaps_loop_back_issues_gaps_only"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3814,7 +3814,7 @@ mod tests { "pipeline_outcomes::tests::worktree_mode_mid_arc_loop_back_issues_plain_execute"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3870,7 +3870,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::worktree_mode_main_checkout_only_artifact_is_the_or_both_roots_discriminator"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -3934,7 +3934,7 @@ mod tests { "pipeline_outcomes::tests::stale_verification_artifact_dispatches_full_execute"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4012,7 +4012,7 @@ mod tests { "pipeline_outcomes::tests::verification_written_this_run_dispatches_gaps_only"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4273,7 +4273,7 @@ mod tests { "pipeline_outcomes::tests::an_uncaptured_baseline_does_not_claim_an_inherited_artifact"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4376,7 +4376,7 @@ mod tests { "pipeline_outcomes::tests::an_idempotent_rewrite_is_authored_not_inherited"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4473,7 +4473,7 @@ mod tests { "pipeline_outcomes::tests::ambiguous_gate_loop_back_respects_the_mid_arc_check"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4530,7 +4530,7 @@ mod tests { "pipeline_outcomes::tests::failure_gate_loop_back_respects_the_mid_arc_check"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4587,7 +4587,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::ship_loop_back_still_issues_gaps_only_when_verification_absent"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4640,7 +4640,7 @@ mod tests { "pipeline_outcomes::tests::external_verify_cycles_reach_ceiling_without_unbounded_loop"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4755,7 +4755,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::consecutive_failures_increment_saturates"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -4872,7 +4872,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::rate_limited_with_unparseable_retry_hint_gates_instead_of_stalling_silently"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -5007,7 +5007,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::ship_agent_failed_fires_gate"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -5229,7 +5229,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::handle_ship_outcome_without_yes_ship_writes_gate_but_no_response"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } @@ -5332,7 +5332,7 @@ mod tests { }; std::os::unix::fs::symlink(resolve("sh"), path_dir.path().join("sh")).unwrap(); let touch = resolve("touch"); - let out = devflow_core::test_support::run_test_in_child( + let out = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(TOUCH_ENV, touch.as_os_str())], @@ -5419,7 +5419,7 @@ mod tests { const NAME: &str = "pipeline_outcomes::tests::stage_failure_retry_cleans_stale_response"; if !devflow_core::test_support::in_child_test(NAME) { let path_dir = agent_free_git_only_path_dir(); - let out = devflow_core::test_support::run_test_in_child(NAME, path_dir.path(), &[]); + let out = crate::test_support::run_hermetic_test_in_child(NAME, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test(&out, NAME); return; } diff --git a/crates/devflow-cli/src/preflight.rs b/crates/devflow-cli/src/preflight.rs index 9078a516..df734a4c 100644 --- a/crates/devflow-cli/src/preflight.rs +++ b/crates/devflow-cli/src/preflight.rs @@ -1008,11 +1008,13 @@ fn unattended_config_condition(launch_root: &Path) -> ConditionState { /// C2: the Code stage would launch on the pipe-owning arm AND the driver has /// a real dogfooded run (round-3 F5/D-04). /// -/// The chain-flag guard lives in a Rust process's stack frame; the legacy arm -/// is a detached shell script with no frame to hang a `Drop` on -/// (`35.1-RESEARCH.md` Pitfall 4). This turns that accepted gap from a silent -/// stall hours into an unattended run into a refusal before the first agent is -/// spawned. +/// The chain-flag guard is engaged by the stage supervisor (`run_monitor`) +/// for a stream launch only, the one shape that bounds the flag's lifetime to +/// the agent's. Since 50-11 a capture launch runs under the same Rust +/// supervisor but never engages it (before that the legacy arm was a detached +/// shell script with no frame to hang a `Drop` on, `35.1-RESEARCH.md` +/// Pitfall 4). This turns that accepted gap from a silent stall hours into an +/// unattended run into a refusal before the first agent is spawned. /// /// Asks [`stream_launch_enabled`] — the launch path's OWN predicate — /// rather than re-deriving the answer, so the preflight and the launch cannot @@ -1048,8 +1050,8 @@ fn unattended_launch_shape_condition(state: &State) -> ConditionState { // The reason names the CAUSE that applied, and then the consequence in // terms true of every cause. ConditionState::DoesNotHold(format!( - "{} — the chain-flag guard binds only inside the pipe-owning monitor, which this \ - launch shape never starts", + "{} — so Code would launch on the capture shape, whose supervisor never engages the \ + chain-flag guard (only a stream-shape supervisor does)", causes.join(" and ") )) } @@ -1487,7 +1489,7 @@ mod tests { if !devflow_core::test_support::in_child_test($name) { let path_dir = $path_dir; let output = - devflow_core::test_support::run_test_in_child($name, path_dir.path(), &[]); + crate::test_support::run_hermetic_test_in_child($name, path_dir.path(), &[]); devflow_core::test_support::assert_child_ran_exactly_one_passing_test( &output, $name, ); @@ -2202,7 +2204,7 @@ mod tests { // from the constant; 34-05's widening puts it on the stream path, // where `canary_gate` invokes the real `ClaudeCanaryLauncher` and the // launch fails on a delivery refusal unrelated to launch counting. - // The `launches == 1` and `monitor_pid.is_some()` assertions below + // The `launches == 1` and `supervisor.is_some()` assertions below // are untouched, so a launch must still genuinely happen. state.legacy_claude_launch = true; workflow::save_state(&state).unwrap(); @@ -2245,7 +2247,7 @@ mod tests { exactly once, not {launches}" ); assert!( - state.monitor_pid.is_some(), + state.supervisor.is_some(), "this test is expected to drive a real monitor::spawn_monitor \ through run_preflight's Advance arm — None here means WR-05's \ premise was wrong and the guard above reaped nothing" @@ -2276,7 +2278,7 @@ mod tests { // (34-06); do not "simplify" this away. Same reasoning as the Advance // arm's sibling above: the subject is the launch COUNT through the // LoopBack arm, not which launch path is taken, and the `launches == 1` - // and `monitor_pid.is_some()` assertions below are untouched. + // and `supervisor.is_some()` assertions below are untouched. state.legacy_claude_launch = true; workflow::save_state(&state).unwrap(); @@ -2324,7 +2326,7 @@ mod tests { exactly once, not {launches}" ); assert!( - state.monitor_pid.is_some(), + state.supervisor.is_some(), "this test is expected to drive a real monitor::spawn_monitor \ through run_preflight's LoopBack arm — None here means WR-05's \ premise was wrong and the guard above reaped nothing" @@ -2436,7 +2438,7 @@ mod tests { "a human Advance must reset the retry counter" ); assert!( - state.monitor_pid.is_some(), + state.supervisor.is_some(), "this test's Advance arm is expected to drive a real \ monitor::spawn_monitor via launch_stage_inner — None here means \ this residual finding's premise was wrong and the guard above \ @@ -4657,4 +4659,40 @@ mod tests { "the driver still resolves the agy program (unchanged path)" ); } + + /// Since 50-11 both launch shapes run under the same Rust supervisor, + /// and only a stream launch engages the chain-flag guard. The refusal + /// says so, for every cause, and no longer says the capture shape + /// "never starts" a pipe-owning monitor. + #[test] + fn unattended_launch_shape_refusal_names_the_capture_supervisor() { + let dir = tempfile::tempdir().unwrap(); + let mut legacy = State::new( + PhaseId::new(43), + AgentKind::Claude, + Mode::Auto, + dir.path().to_path_buf(), + ); + legacy.legacy_claude_launch = true; + let codex = State::new( + PhaseId::new(44), + AgentKind::Codex, + Mode::Auto, + dir.path().to_path_buf(), + ); + for state in [legacy, codex] { + let ConditionState::DoesNotHold(reason) = unattended_launch_shape_condition(&state) + else { + panic!("{} --mode auto must be refused", state.agent); + }; + assert!( + reason.ends_with( + " — so Code would launch on the capture shape, whose supervisor never \ + engages the chain-flag guard (only a stream-shape supervisor does)" + ), + "{reason}" + ); + assert!(!reason.contains("never starts"), "{reason}"); + } + } } diff --git a/crates/devflow-cli/src/staleness.rs b/crates/devflow-cli/src/staleness.rs index b20a9440..bd6a3c82 100644 --- a/crates/devflow-cli/src/staleness.rs +++ b/crates/devflow-cli/src/staleness.rs @@ -1196,7 +1196,7 @@ mod tests { // identical "self-dogfood stale build blocked" error produced above. workflow::save_state(&state).unwrap(); let path_dir = agent_free_dir_with_agent_stub("claude"); - let output = devflow_core::test_support::run_test_in_child( + let output = crate::test_support::run_hermetic_test_in_child( NAME, path_dir.path(), &[(ROOT_ENV, project_root.as_os_str())], diff --git a/crates/devflow-cli/src/test_support.rs b/crates/devflow-cli/src/test_support.rs index 1f0e7dbf..05644d9a 100644 --- a/crates/devflow-cli/src/test_support.rs +++ b/crates/devflow-cli/src/test_support.rs @@ -15,7 +15,7 @@ use devflow_core::agents; use devflow_core::phase_id::PhaseId; -use devflow_core::state::State; +use devflow_core::state::{State, SupervisorRecord}; use std::path::{Path, PathBuf}; use std::sync::Mutex; @@ -372,6 +372,9 @@ pub(crate) fn child_no_git_root() -> Option { pub(crate) fn run_test_without_git(test_name: &str, root: &Path) -> std::process::Output { // Deliberately empty. Nothing is written into it, by design. let empty = tempfile::tempdir().unwrap(); + // 50-10 (D-07): a launch inside this child registers here, never under + // the developer's real HOME. + let cache = hermetic_cache_dir(); let exe = std::env::current_exe().expect("current_exe for child test re-invocation"); let output = std::process::Command::new(&exe) .arg(test_name) @@ -379,10 +382,42 @@ pub(crate) fn run_test_without_git(test_name: &str, root: &Path) -> std::process .arg("--test-threads=1") .env("PATH", empty.path()) .env(CHILD_NO_GIT_ROOT, root) + .env("DEVFLOW_CACHE_DIR", cache.path()) .output() .expect("spawn child test process with an empty PATH"); - // Explicit, not incidental: the empty directory must outlive the child. + // Explicit, not incidental: both directories must outlive the child. drop(empty); + drop(cache); + output +} + +/// A parent-owned DevFlow cache dir for one child test process (50-10, D-07): +/// short and under `/tmp`, so 50-11's socket paths inside it stay under the +/// 107-byte `sun_path` limit. +pub(crate) fn hermetic_cache_dir() -> tempfile::TempDir { + tempfile::Builder::new() + .prefix("dfu") + .tempdir_in("/tmp") + .expect("create a hermetic DevFlow cache dir under /tmp") +} + +/// [`devflow_core::test_support::run_test_in_child`] with a fresh +/// [`hermetic_cache_dir`] passed as the child's `DEVFLOW_CACHE_DIR` (50-10, +/// D-07). Every in-crate test that may launch a stage runs its child through +/// this, so the launch registers (and, from 50-11, binds its socket) inside +/// that dir and never under the developer's real HOME. The cache dir lives +/// until the child has exited. A `DEVFLOW_CACHE_DIR` in `extra_env` wins. +pub(crate) fn run_hermetic_test_in_child( + test_name: &str, + path_dir: &Path, + extra_env: &[(&str, &std::ffi::OsStr)], +) -> std::process::Output { + let cache = hermetic_cache_dir(); + let mut env: Vec<(&str, &std::ffi::OsStr)> = + vec![("DEVFLOW_CACHE_DIR", cache.path().as_os_str())]; + env.extend_from_slice(extra_env); + let output = devflow_core::test_support::run_test_in_child(test_name, path_dir, &env); + drop(cache); output } @@ -496,28 +531,60 @@ pub(crate) fn stage_launched_count(root: &Path, phase: PhaseId) -> usize { .count() } -/// The single reap mechanism: escalate through `terminate_and_verify` -/// (bounded TERM-then-KILL — 999.44 measured 15 of 15 orphaned monitor -/// wrappers surviving a bare `SIGTERM` against this exact process shape, so -/// an unescalated signal would leave the leak in place while appearing to -/// fix it), then return a VERIFIED liveness answer. +/// How long a reaper waits for a supervisor's final shutdown line. +const REAP_SHUTDOWN_WAIT: std::time::Duration = std::time::Duration::from_secs(15); + +/// The single reap mechanism (D-07): end a launched run through its +/// supervisor's socket, then return a VERIFIED answer about its agent group. +/// +/// 1. `supervisor::request_shutdown` on the recorded socket, bounded by +/// [`REAP_SHUTDOWN_WAIT`]. A supervisor that answers ends its agent group +/// itself. +/// 2. If the recorded agent group's leader is still the process the record +/// names (`agent::is_same_process` against its start time), SIGKILL the +/// whole group. This covers a socket that does not answer (a SIGKILLed +/// supervisor, or a record whose socket never existed); the identity +/// check keeps a recycled pid from ever being signalled. +/// 3. Poll until the leader is gone (a zombie counts as gone) or +/// `agent::TERMINATE_VERIFY_WAIT` passes. /// /// Asserts nothing and panics on no path — the caller decides what a `false` -/// means. This matters because one of its two callers -/// ([`ReapMonitorOnDrop::drop`]) may run during a stack unwind, where a -/// panic would call `abort()` and kill the whole test binary instead of -/// merely failing the one test in flight. -fn reap_monitor_pid(pid: u32) -> bool { - devflow_core::agent::terminate_and_verify( - pid, - devflow_core::agent::TERMINATE_VERIFY_WAIT, - devflow_core::agent::TERMINATE_VERIFY_POLL, - ); - !devflow_core::agent::agent_running(pid) +/// means. This matters because one of its callers ([`ReapMonitorOnDrop::drop`]) +/// may run during a stack unwind, where a panic would call `abort()` and kill +/// the whole test binary instead of merely failing the one test in flight. +fn reap_supervised_run(record: &SupervisorRecord) -> bool { + let _ = devflow_core::supervisor::request_shutdown(&record.socket_path, REAP_SHUTDOWN_WAIT); + let leader = record.agent_pgid; + let alive = || { + devflow_core::agent::agent_running(leader) + && devflow_core::agent::is_same_process(leader, record.agent_start_time) + }; + if alive() { + signal_group(leader, libc::SIGKILL); + } + let deadline = std::time::Instant::now() + devflow_core::agent::TERMINATE_VERIFY_WAIT; + while alive() && std::time::Instant::now() < deadline { + std::thread::sleep(devflow_core::agent::TERMINATE_VERIFY_POLL); + } + !alive() } -/// Reap the detached monitor wrapper a real `launch_stage_inner` call just -/// spawned, verifying its death rather than assuming it (WR-03, 999.46). +/// Send `signal` to process group `pgid`. Refuses pgid 0 or 1, a value that +/// does not fit `pid_t`, and this process's own group. +fn signal_group(pgid: u32, signal: libc::c_int) { + let Ok(signed) = libc::pid_t::try_from(pgid) else { + return; + }; + // SAFETY: getpgrp has no preconditions and cannot fail. + if signed > 1 && signed != unsafe { libc::getpgrp() } { + // SAFETY: `signed > 1` and not our own group: this names one process + // group whose leader the caller has just identity-checked. + unsafe { libc::kill(-signed, signal) }; + } +} + +/// Reap the supervised run a real `launch_stage_inner` call just launched, +/// verifying its agent group's end rather than assuming it (WR-03, 999.46). /// /// This plain-call form runs ONLY if every preceding statement in its caller /// returned normally — a panic anywhere between the launch and this call @@ -529,61 +596,47 @@ fn reap_monitor_pid(pid: u32) -> bool { /// `tests::trailing_reap_call_is_skipped_when_a_later_assertion_panics`, /// which exists precisely to demonstrate this failure mode. /// -/// Any test that drives a real `launch_stage_inner` causes it to spawn a -/// detached monitor wrapper (`monitor::spawn_monitor`) that is DESIGNED to -/// outlive the call that spawned it — that is exactly what lets a real -/// `devflow start` invocation return while the monitor keeps watching the -/// agent. A test that drives the same call path gets the same detached -/// wrapper, and since nothing else in a test's lifetime ever signals it, the -/// TEST is the only thing that ever could — so the test owns reaping it. -/// -/// The pid to reap comes from `state.monitor_pid` on the same `&mut State` -/// the caller handed to `launch_stage_inner` — the only on-disk-free handle a -/// test has on what it just caused to exist (`pipeline_launch.rs` writes the -/// spawned pid there immediately after `monitor::spawn_monitor` returns). -/// This function never scans `/proc`, never guesses, and never signals a pid -/// it did not read from that same handle. -/// -/// Must be called BEFORE the caller's `TempDir` guard drops — reaping after -/// the project root has already been deleted is 999.44's reproduction shape -/// with extra steps, not a fix for it. -/// -/// Tolerates `state.monitor_pid == None` by returning quietly, with no -/// `unwrap`, no `expect` and no panic: an early-failing `launch_stage_inner` -/// clears the field before any fallible step (`pipeline_launch.rs:70`), and a -/// teardown helper that panics on that path would mask the launch's own -/// failure rather than merely fail to clean up after it. +/// Any test that drives a real `launch_stage_inner` causes it to start a +/// detached supervisor that is DESIGNED to outlive the call that started it; +/// the TEST is the only thing that ever could end it, so the test owns +/// reaping it. +/// +/// The handle comes from `state.supervisor` on the same `&State` the caller +/// handed to the launch (`spawn_agent_and_record` records the ping-confirmed +/// record there). This function never scans `/proc`, never guesses, and +/// signals only the group that record names, after its identity check. +/// +/// Must be called BEFORE the caller's `TempDir` guard drops. Tolerates a +/// state with no supervisor record by returning quietly: an early-failing +/// launch never recorded one, and a teardown helper that panics on that path +/// would mask the launch's own failure. pub(crate) fn reap_spawned_monitor(state: &State) { - let Some(pid) = state.monitor_pid else { + let Some(record) = state.supervisor.as_ref() else { return; }; assert!( - reap_monitor_pid(pid), - "monitor wrapper pid {pid}, spawned by this test's own launch_stage_inner call, must be \ - verified dead after reaping — not merely assumed dead" + reap_supervised_run(record), + "agent group {}, launched by this test's own launch_stage_inner call, must be \ + verified gone after reaping through {} — not merely assumed gone", + record.agent_pgid, + record.socket_path.display() ); } -/// RAII guard that reaps a spawned monitor wrapper on EVERY exit path of the -/// scope it is bound in — including a path on which a later assertion -/// panics. This is the fix for G-25-2 / WINDOWS.md item 3: a plain trailing -/// call to [`reap_spawned_monitor`] only runs on the success path, since Rust +/// RAII guard that reaps a launched run on EVERY exit path of the scope it +/// is bound in — including a path on which a later assertion panics. This is +/// the fix for G-25-2 / WINDOWS.md item 3: a plain trailing call to +/// [`reap_spawned_monitor`] only runs on the success path, since Rust /// abandons the rest of a function's statements the instant a panic begins /// unwinding, so it is the language's own `Drop` guarantee — not a call /// ordering convention — that makes the reap unconditional. pub(crate) struct ReapMonitorOnDrop { - pid: Option, + record: Option, } impl ReapMonitorOnDrop { - /// Capture `state.monitor_pid` BY VALUE, not `&'a State`. - /// - /// By value: at every call site this guard is bound strictly AFTER the - /// test's final `&mut state` use, so a borrow would not conflict with - /// anything at that point — but a `&'a State` guard would still tie the - /// guard's lifetime to the `State` binding's own scope, which is exactly - /// the kind of incidental coupling a teardown guard should not have. A - /// bare `Option` keeps the guard independent of `State` entirely. + /// Capture `state.supervisor` BY VALUE, not `&'a State`, so the guard + /// stays independent of the `State` binding's own scope. /// /// Named `after_launch`, not `new`: bound before the launch call /// returns, it captures `None` and silently does nothing — the @@ -591,17 +644,17 @@ impl ReapMonitorOnDrop { /// rather than leaving it to a doc comment alone. pub(crate) fn after_launch(state: &State) -> Self { Self { - pid: state.monitor_pid, + record: state.supervisor.clone(), } } } impl Drop for ReapMonitorOnDrop { fn drop(&mut self) { - let Some(pid) = self.pid else { + let Some(record) = self.record.as_ref() else { return; }; - if !reap_monitor_pid(pid) { + if !reap_supervised_run(record) { // The check below is the double-panic interlock — the ONLY // thing standing between "an assertion failed" and "the whole // test binary aborted". A `Drop` that panics while an unwind is @@ -622,19 +675,97 @@ impl Drop for ReapMonitorOnDrop { use std::io::Write as _; let _ = writeln!( std::io::stderr(), - "ReapMonitorOnDrop: monitor wrapper pid {pid} still alive after reap \ - during an unwind — not re-panicking because a panic is already in flight" + "ReapMonitorOnDrop: agent group {} still alive after reap \ + during an unwind — not re-panicking because a panic is already in flight", + record.agent_pgid ); } else { panic!( - "monitor wrapper pid {pid}, spawned by this test's own launch call, must be \ - verified dead after reaping — not merely assumed dead" + "agent group {}, launched by this test's own launch call, must be \ + verified gone after reaping through {} — not merely assumed gone", + record.agent_pgid, + record.socket_path.display() ); } } } } +/// Parse the `__monitor` argv a [`MonitorExec::HarnessEntry`] re-exec put in +/// [`MONITOR_ARGV_ENV`] and run it as `main` dispatches `__monitor`, with or +/// without the advance tail. Returns without doing anything when the variable +/// is absent, so a run of ignored tests that reaches an entry is a no-op. +/// +/// [`MonitorExec::HarnessEntry`]: devflow_core::monitor::MonitorExec::HarnessEntry +/// [`MONITOR_ARGV_ENV`]: devflow_core::monitor::MONITOR_ARGV_ENV +fn run_monitor_harness_entry(advance: bool) { + use clap::Parser as _; + let Some(encoded) = std::env::var_os(devflow_core::monitor::MONITOR_ARGV_ENV) else { + return; + }; + let encoded = encoded + .into_string() + .expect("the monitor argv must be UTF-8 JSON"); + let after_subcommand: Vec = + serde_json::from_str(&encoded).expect("the monitor argv must be a JSON array of strings"); + let argv = ["devflow", "__monitor"] + .into_iter() + .map(str::to_string) + .chain(after_subcommand); + let cli = crate::Cli::try_parse_from(argv).expect("the monitor argv must parse as __monitor"); + let crate::Command::Monitor { + project, + phase, + stage, + workdir, + agent, + launch, + socket, + launch_id, + keep_env, + prompt_file, + idle_timeout_secs, + argv, + } = cli.command + else { + panic!("the monitor argv parsed as a command other than __monitor"); + }; + let invocation = crate::pipeline_launch::MonitorInvocation { + project_root: crate::project_root(project).expect("monitor project root"), + phase, + stage, + workdir, + agent, + launch, + socket, + launch_id, + keep_env, + prompt_file, + idle_timeout_secs, + argv, + }; + crate::pipeline_launch::run_monitor(&invocation, advance) + .expect("the harness-entry monitor must supervise its agent"); +} + +/// The test build's full monitor entry (50-10): `run_monitor` exactly as the +/// `__monitor` subcommand runs it, advance tail included. Selected by +/// `pipeline_launch::launch_context` when `DEVFLOW_TEST_MONITOR_FULL=1`. +#[test] +#[ignore = "monitor re-exec entry; run only by spawn_monitor under cfg(test)"] +fn monitor_entry_full() { + run_monitor_harness_entry(true); +} + +/// The test build's default monitor entry (50-10): `run_monitor`'s +/// supervision without its advance tail, so an in-crate launch supervises its +/// fake agent and leaves the stage machine where the test put it. +#[test] +#[ignore = "monitor re-exec entry; run only by spawn_monitor under cfg(test)"] +fn monitor_entry_supervise_only() { + run_monitor_harness_entry(false); +} + #[cfg(test)] mod tests { use super::*; @@ -755,9 +886,13 @@ mod tests { impl ChildGuard { fn spawn() -> Self { + use std::os::unix::process::CommandExt as _; + // Its own process group, as a launched agent is (R-3), so the + // reapers' identity-confirmed group fallback has a real subject. Self( Command::new("sleep") .arg("300") + .process_group(0) .spawn() .expect("sleep must be spawnable to run this test"), ) @@ -775,9 +910,11 @@ mod tests { } } - /// A `State` whose only relevant field is `monitor_pid`. No I/O occurs — - /// the project root is never read, since nothing in the reap path - /// touches disk. + /// A `State` whose only relevant field is `supervisor`: a record whose + /// socket does not exist, naming `pid` (its own group leader) with its + /// real start time, so the reapers' shutdown finds nothing to ask and + /// they reach the identity-confirmed group fallback. No I/O occurs + /// beyond reading `pid`'s start time and boot id. fn state_holding(pid: u32) -> State { let mut state = State::new( PhaseId::new(0), @@ -785,7 +922,13 @@ mod tests { Mode::Auto, PathBuf::from("/nonexistent"), ); - state.monitor_pid = Some(pid); + state.supervisor = Some(SupervisorRecord::new( + PathBuf::from("/nonexistent/reap-fixture.sock"), + "reapfixture00000".to_string(), + pid, + devflow_core::agent::process_start_time(pid).expect("the subject's start time"), + devflow_core::supervisor::boot_id().unwrap_or_default(), + )); state } diff --git a/crates/devflow-cli/tests/auto_chain_flag_e2e.rs b/crates/devflow-cli/tests/auto_chain_flag_e2e.rs index d7684851..9dcc5d06 100644 --- a/crates/devflow-cli/tests/auto_chain_flag_e2e.rs +++ b/crates/devflow-cli/tests/auto_chain_flag_e2e.rs @@ -13,6 +13,12 @@ //! the guard scoped to the wrong stack frame — that is RESEARCH Pitfall 2, and //! it is the entire reason this is an end-to-end test rather than a unit test //! of the writer. +//! +//! Every `devflow` child is built by `common::hermetic_devflow`, so the +//! monitor's registry entries land in the fixture's own tempdir `HOME`, never +//! in the operator's real `~/.cache/devflow` (999.141). + +mod common; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; @@ -22,11 +28,7 @@ use devflow_core::workflow::save_state; use std::fs; use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; -use std::process::{Command, Output}; - -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} +use std::process::Output; /// Hermetic git invocation pinned to `root` (999.37) — never a bare /// `Command::new("git")`. @@ -146,6 +148,9 @@ struct Fixture { observation: PathBuf, prompt_file: PathBuf, script: PathBuf, + /// Every `devflow` child's `HOME` and cache. Declared last so it drops + /// after the rest of the fixture. + home: common::HermeticHome, } impl Fixture { @@ -191,12 +196,23 @@ impl Fixture { observation, prompt_file, script, + home: common::HermeticHome::new(), } } /// Run the REAL binary's hidden `__monitor` subcommand over `argv`. fn run_monitor(&self, argv: &[&str]) -> Output { - Command::new(devflow_bin()) + // The 3.x `__monitor` argv (50-11): a per-launch id and a socket + // under the hermetic cache dir this file's child process gets. + let launch_id = devflow_core::supervisor::new_launch_id(); + let socket = devflow_core::supervisor::SocketPath::for_launch( + &self.home.cache_dir(), + &self.root, + self.phase, + &launch_id, + ) + .expect("the hermetic socket path fits sun_path"); + common::hermetic_devflow(&self.home) .arg("__monitor") .arg("--project") .arg(&self.root) @@ -210,6 +226,12 @@ impl Fixture { .arg("30") .arg("--agent") .arg("claude") + .arg("--launch") + .arg("stream") + .arg("--launch-id") + .arg(&launch_id) + .arg("--socket") + .arg(socket.as_path()) .arg("--") .args(argv) .output() @@ -247,6 +269,10 @@ fn auto_mode_code_stage_child_observes_the_flag_set() { let fixture = Fixture::new(Mode::Auto, false); let output = fixture.run_observer(); + // The foreground monitor halts at the Code boundary (`stop_until`), so + // it should launch nothing. If that halt ever regresses, the next stage's + // detached monitor and agent are reaped here instead of outliving the test. + let _run_reaper = common::LaunchedRunReaper::new(&fixture.root, fixture.phase, Vec::new()); assert!( output.status.success(), "monitor exited {:?}\nstdout:\n{}\nstderr:\n{}", @@ -287,6 +313,10 @@ fn supervise_mode_code_stage_child_observes_the_flag_clear() { let before = fs::read(&fixture.config).unwrap(); let output = fixture.run_observer(); + // The foreground monitor halts at the Code boundary (`stop_until`), so + // it should launch nothing. If that halt ever regresses, the next stage's + // detached monitor and agent are reaped here instead of outliving the test. + let _run_reaper = common::LaunchedRunReaper::new(&fixture.root, fixture.phase, Vec::new()); assert!( output.status.success(), "monitor exited {:?}\nstdout:\n{}\nstderr:\n{}", @@ -311,7 +341,7 @@ fn supervise_mode_code_stage_child_observes_the_flag_clear() { } /// The in-process half of criterion 2: the flag is cleared even when the run -/// dies before `advance` is ever reached — `run_pipe_owning_monitor`'s +/// dies before `advance` is ever reached — `run_supervised_stage`'s /// early-return `Err` path, driven here by a child program that cannot be /// spawned at all. (`35.1-02` owns the kill half.) /// @@ -323,6 +353,10 @@ fn guard_clears_the_flag_when_the_supervised_child_fails() { let fixture = Fixture::new(Mode::Auto, true); let output = fixture.run_monitor(&["definitely-not-a-real-program-35-1-01"]); + // The foreground monitor halts at the Code boundary (`stop_until`), so + // it should launch nothing. If that halt ever regresses, the next stage's + // detached monitor and agent are reaped here instead of outliving the test. + let _run_reaper = common::LaunchedRunReaper::new(&fixture.root, fixture.phase, Vec::new()); assert!( !output.status.success(), "a child that cannot be spawned must fail the monitor, not pass silently" diff --git a/crates/devflow-cli/tests/auto_chain_leak_repair_e2e.rs b/crates/devflow-cli/tests/auto_chain_leak_repair_e2e.rs index 9cc5ecba..c60090f5 100644 --- a/crates/devflow-cli/tests/auto_chain_leak_repair_e2e.rs +++ b/crates/devflow-cli/tests/auto_chain_leak_repair_e2e.rs @@ -20,6 +20,8 @@ //! scaffolding below is copied from `auto_chain_flag_e2e.rs` rather than //! shared. Keep the two in step. +mod common; + use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; use devflow_core::stage::Stage; @@ -32,10 +34,6 @@ use std::path::{Path, PathBuf}; use std::process::{Child, Command, Output}; use std::time::{Duration, Instant}; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - /// Hermetic git invocation pinned to `root` (999.37) — never a bare /// `Command::new("git")`. fn git(root: &Path, args: &[&str]) { @@ -205,6 +203,9 @@ struct Fixture { script: PathBuf, child_pid_file: PathBuf, reaper: Reaper, + /// Every `devflow` child's `HOME` and cache. Declared last so it drops + /// after `reaper` has reaped the monitor. + home: common::HermeticHome, } impl Fixture { @@ -245,6 +246,7 @@ impl Fixture { script, child_pid_file, reaper: Reaper::default(), + home: common::HermeticHome::new(), } } @@ -252,7 +254,17 @@ impl Fixture { /// sleeper, as a child of this test process so it can be signalled /// directly. fn spawn_monitor(&self) -> Child { - let child = Command::new(devflow_bin()) + // The 3.x `__monitor` argv (50-11): a per-launch id and a socket + // under the hermetic cache dir this file's child process gets. + let launch_id = devflow_core::supervisor::new_launch_id(); + let socket = devflow_core::supervisor::SocketPath::for_launch( + &self.home.cache_dir(), + &self.root, + self.phase, + &launch_id, + ) + .expect("the hermetic socket path fits sun_path"); + let child = common::hermetic_devflow(&self.home) .arg("__monitor") .arg("--project") .arg(&self.root) @@ -268,6 +280,12 @@ impl Fixture { .arg("120") .arg("--agent") .arg("claude") + .arg("--launch") + .arg("stream") + .arg("--launch-id") + .arg(&launch_id) + .arg("--socket") + .arg(socket.as_path()) .arg("--") .arg("sh") .arg(&self.script) @@ -298,7 +316,7 @@ impl Fixture { /// process outlives the test binary" criterion forbids. See the SUMMARY for /// what this does NOT establish. fn run_resume(&self, git_only_path: &Path) -> Output { - Command::new(devflow_bin()) + common::hermetic_devflow(&self.home) .arg("resume") .arg("--phase") .arg(self.phase.to_string()) @@ -379,6 +397,7 @@ fn auto_chain_flag_survives_sigkill_and_is_repaired_on_the_next_start() { let path_dir = git_only_path(); let mut monitor = fixture.spawn_monitor(); + let _run_reaper = common::LaunchedRunReaper::new(&fixture.root, fixture.phase, Vec::new()); // The fixture must actually engage the guard before anything downstream // means anything. A timeout here is "the fixture never set the flag", which diff --git a/crates/devflow-cli/tests/common/mod.rs b/crates/devflow-cli/tests/common/mod.rs new file mode 100644 index 00000000..feec5f39 --- /dev/null +++ b/crates/devflow-cli/tests/common/mod.rs @@ -0,0 +1,325 @@ +//! Helpers shared by the `devflow` CLI integration tests (999.141, D-07). +//! +//! Each test target includes this file with `mod common;`. It holds helpers +//! only and no `#[test]`: every target that declares `mod common;` compiles +//! it, so a test here would be added to each of those targets' counts. +//! +//! The rule it exists for: an e2e child that runs the `devflow` binary gets a +//! tempdir `HOME`, `XDG_CACHE_HOME`, `XDG_CONFIG_HOME` and `DEVFLOW_CACHE_DIR` +//! and has the repo-local `GIT_*` variables removed, so its registry entries +//! (and, from 50-11, its sockets) never land in the operator's real +//! `~/.cache/devflow`. Spawn `devflow` through [`hermetic_devflow`] only; +//! `test_home_isolation.rs` scans every suite that declares `mod common;` for +//! a raw spawn. Isolation is set on each child `Command`, never by mutating +//! the test process's own environment. + +#![allow( + dead_code, + reason = "each test target uses a different subset of the shared helpers" +)] + +use devflow_core::phase_id::PhaseId; +use std::fs; +use std::os::unix::fs::{PermissionsExt, symlink}; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command}; +use std::time::{Duration, Instant}; + +/// The compiled `devflow` binary under test. The only place in `tests/` that +/// names Cargo's binary path variable once migration completes. +pub fn devflow_bin() -> &'static str { + env!("CARGO_BIN_EXE_devflow") +} + +/// A throwaway `HOME` for one or more `devflow` children, removed on drop. +/// +/// Created directly under `/tmp` with a short prefix, never under a possibly +/// long `$TMPDIR`, so a socket path below it stays under the 107-byte +/// `sun_path` limit (R-5). +pub struct HermeticHome { + dir: tempfile::TempDir, +} + +impl HermeticHome { + /// Create a fresh, empty home under `/tmp`. + pub fn new() -> Self { + let dir = tempfile::Builder::new() + .prefix("dfh") + .tempdir_in("/tmp") + .expect("create hermetic HOME under /tmp"); + Self { dir } + } + + /// The home directory itself (the child's `HOME`). + pub fn home(&self) -> &Path { + self.dir.path() + } + + /// The child's DevFlow cache dir (`home/.cache/devflow`), where its + /// registry `roots/` entries land. + pub fn cache_dir(&self) -> PathBuf { + self.home().join(".cache").join("devflow") + } + + /// The child's `XDG_CONFIG_HOME` (`home/.config`). + pub fn config_dir(&self) -> PathBuf { + self.home().join(".config") + } +} + +impl Default for HermeticHome { + fn default() -> Self { + Self::new() + } +} + +/// A `devflow` command whose `HOME`, `XDG_CACHE_HOME`, `XDG_CONFIG_HOME` and +/// `DEVFLOW_CACHE_DIR` all point inside `home`, with every repo-local and +/// repo-redirecting `GIT_*` variable removed. `home` must outlive the child +/// and any monitor it launches. +pub fn hermetic_devflow(home: &HermeticHome) -> Command { + let mut command = Command::new(devflow_bin()); + command + .env("HOME", home.home()) + .env("XDG_CACHE_HOME", home.home().join(".cache")) + .env("XDG_CONFIG_HOME", home.config_dir()) + .env("DEVFLOW_CACHE_DIR", home.cache_dir()); + for name in devflow_core::test_support::REPO_LOCAL_GIT_VARS + .iter() + .chain(devflow_core::test_support::ALSO_REDIRECTING_GIT_VARS) + { + command.env_remove(name); + } + command +} + +/// A directory of stub programs to put first on a child's `PATH`. Removed on +/// drop. +pub struct FakeBin { + _dir: tempfile::TempDir, + pub path: PathBuf, +} + +/// A fake `claude` whose body is `agent_body`, plus `git` and `sh` links. +pub fn fake_bin_dir(agent_body: &str) -> FakeBin { + fake_bin_dir_for("claude", agent_body) +} + +/// A fake agent named `program` (for example `claude` or `codex`) whose body +/// is `agent_body`, plus `git` and `sh` links. +pub fn fake_bin_dir_for(program: &str, agent_body: &str) -> FakeBin { + let dir = tempfile::tempdir().unwrap(); + let agent = dir.path().join(program); + fs::write(&agent, format!("#!/bin/sh\n{agent_body}\n")).unwrap(); + let mut permissions = fs::metadata(&agent).unwrap().permissions(); + permissions.set_mode(0o755); + fs::set_permissions(&agent, permissions).unwrap(); + for linked in ["git", "sh"] { + symlink(format!("/usr/bin/{linked}"), dir.path().join(linked)).unwrap(); + } + FakeBin { + path: dir.path().to_path_buf(), + _dir: dir, + } +} + +/// Poll `predicate` every 25ms until it is true, panicking with `what` once +/// `timeout` elapses. +pub fn wait_for(mut predicate: impl FnMut() -> bool, timeout: Duration, what: &str) { + let started = Instant::now(); + while !predicate() { + assert!( + started.elapsed() < timeout, + "timed out after {timeout:?} waiting for {what}" + ); + std::thread::sleep(Duration::from_millis(25)); + } +} + +/// Kill `child` if it is still running, then wait on it. +pub fn kill_and_reap(child: &mut Child) { + if child.try_wait().expect("poll child").is_none() { + let _ = child.kill(); + } + let _ = child.wait(); +} + +/// How long [`LaunchedRunReaper`] waits for a supervisor's final shutdown line. +const REAP_SHUTDOWN_WAIT: Duration = Duration::from_secs(15); + +/// How long [`LaunchedRunReaper`] waits for the phase's supervisors to exit on +/// their own before it ends the ones still alive. +const REAP_EXIT_WAIT: Duration = Duration::from_secs(15); + +/// Ends, on drop, the run recorded in the phase state, through its +/// supervisor's socket (D-07): a `shutdown` request first; then, if the +/// recorded agent group's leader is still the process the record names +/// (`agent::is_same_process` against its start time), a SIGKILL to that +/// group. Only when the state holds no `supervisor` record does it fall back +/// to the agent named in the agent-pid file. A pid in `spare` is never +/// signalled. Bind it before (or right after) the spawn that launches a run, +/// and keep the run's [`HermeticHome`] alive until this has dropped. +/// +/// It then waits, bounded by [`REAP_EXIT_WAIT`], until no `devflow +/// __monitor` whose argv names this root and phase is alive, and ends any +/// still alive after that wait. The state cannot name every one: a stop that +/// lands while an advance is launching the next stage leaves that next +/// supervisor spawned but never recorded, with the stop flagged. It records +/// its own stop under the per-phase lock a moment later, and taking that lock +/// recreates `/.devflow/` when the root's `TempDir` has already been +/// removed, leaving `/tmp/.tmpXXXXXX/.devflow/.lock-NN.coord` behind. +pub struct LaunchedRunReaper { + root: PathBuf, + phase: PhaseId, + spare: Vec, +} + +impl LaunchedRunReaper { + pub fn new(root: &Path, phase: PhaseId, spare: Vec) -> Self { + Self { + root: root.to_path_buf(), + phase, + spare, + } + } + + /// Every live `devflow __monitor` whose argv names this root (as given or + /// canonicalized) after `--project` and this phase after `--phase`, with + /// its start time: each supervisor this run started, recorded or not. + fn live_supervisors(&self) -> Vec<(u32, u64)> { + use std::os::unix::ffi::OsStrExt as _; + let canonical = fs::canonicalize(&self.root).unwrap_or_else(|_| self.root.clone()); + let roots = [ + self.root.as_os_str().as_bytes(), + canonical.as_os_str().as_bytes(), + ]; + let Ok(entries) = fs::read_dir("/proc") else { + return Vec::new(); + }; + entries + .flatten() + .filter_map(|entry| { + let pid = entry.file_name().to_str()?.parse::().ok()?; + let cmdline = fs::read(entry.path().join("cmdline")).ok()?; + let args: Vec<&[u8]> = cmdline.split(|byte| *byte == 0).collect(); + let value_after = |flag: &str| { + args.windows(2) + .find(|pair| pair[0] == flag.as_bytes()) + .map(|pair| pair[1]) + }; + let phase = value_after("--phase") + .and_then(|value| std::str::from_utf8(value).ok()) + .and_then(|value| value.parse::().ok()); + let ours = args.contains(&b"__monitor".as_slice()) + && value_after("--project").is_some_and(|value| roots.contains(&value)) + && phase == Some(self.phase); + if !ours { + return None; + } + let start = devflow_core::agent::process_start_time(pid)?; + devflow_core::agent::agent_running(pid).then_some((pid, start)) + }) + .collect() + } + + /// Wait for this phase's supervisors to exit, then end any left, so none + /// of them writes under the root once this returns. + fn wait_for_supervisors_to_exit(&self) { + let deadline = Instant::now() + REAP_EXIT_WAIT; + while Instant::now() < deadline { + if self.live_supervisors().is_empty() { + return; + } + std::thread::sleep(Duration::from_millis(50)); + } + for (pid, start) in self.live_supervisors() { + if !self.spare.contains(&pid) && devflow_core::agent::is_same_process(pid, start) { + let _ = devflow_core::agent::terminate_and_verify( + pid, + devflow_core::agent::TERMINATE_VERIFY_WAIT, + devflow_core::agent::TERMINATE_VERIFY_POLL, + ); + } + } + } + + /// The shutdown described on [`LaunchedRunReaper`]: the recorded + /// supervisor, else the agent in the agent-pid file. + fn end_recorded_run(&self) { + let record = devflow_core::workflow::load_state(&self.root, self.phase) + .ok() + .and_then(|state| state.supervisor); + if let Some(record) = record { + let _ = + devflow_core::supervisor::request_shutdown(&record.socket_path, REAP_SHUTDOWN_WAIT); + let leader = record.agent_pgid; + if !self.spare.contains(&leader) + && devflow_core::agent::agent_running(leader) + && devflow_core::agent::is_same_process(leader, record.agent_start_time) + { + signal_group(leader, libc::SIGKILL); + } + return; + } + let agent = fs::read_to_string(devflow_core::agent_result::agent_pid_path( + &self.root, self.phase, + )) + .ok() + .and_then(|pid| pid.trim().parse::().ok()); + if let Some(pid) = agent + && !self.spare.contains(&pid) + && devflow_core::agent::agent_running(pid) + { + let _ = devflow_core::agent::terminate_and_verify( + pid, + devflow_core::agent::TERMINATE_VERIFY_WAIT, + devflow_core::agent::TERMINATE_VERIFY_POLL, + ); + } + } +} + +impl Drop for LaunchedRunReaper { + fn drop(&mut self) { + self.end_recorded_run(); + self.wait_for_supervisors_to_exit(); + } +} + +/// Send `signal` to process group `pgid`. Refuses pgid 0 or 1, a value that +/// does not fit `pid_t`, and this process's own group. +pub fn signal_group(pgid: u32, signal: libc::c_int) { + let Ok(signed) = libc::pid_t::try_from(pgid) else { + return; + }; + // SAFETY: getpgrp has no preconditions and cannot fail. + if signed > 1 && signed != unsafe { libc::getpgrp() } { + // SAFETY: `signed > 1` and not our own group: one process group + // whose leader the caller has just identity-checked. + unsafe { libc::kill(-signed, signal) }; + } +} + +/// Every file below `dir`, recursively, sorted. Empty when `dir` is missing. +pub fn entries_under(dir: &Path) -> Vec { + let mut files = Vec::new(); + collect_files(dir, &mut files); + files.sort(); + files +} + +fn collect_files(dir: &Path, files: &mut Vec) { + let Ok(entries) = fs::read_dir(dir) else { + return; + }; + for entry in entries { + let entry = entry.expect("read directory entry"); + let path = entry.path(); + // `file_type` does not follow symlinks, so a link loop cannot recurse. + if entry.file_type().expect("read entry type").is_dir() { + collect_files(&path, files); + } else { + files.push(path); + } + } +} diff --git a/crates/devflow-cli/tests/doctor_antigravity.rs b/crates/devflow-cli/tests/doctor_antigravity.rs index a01cf198..22c821eb 100644 --- a/crates/devflow-cli/tests/doctor_antigravity.rs +++ b/crates/devflow-cli/tests/doctor_antigravity.rs @@ -7,13 +7,10 @@ //! `devflow` binary, and PATH is scoped to each child via `Command::env` so //! no test-process-global environment mutation can poison parallel tests. +mod common; + use std::os::unix::fs::PermissionsExt; use std::path::Path; -use std::process::Command; - -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} fn write_stub_agy(root: &Path) -> std::path::PathBuf { let bin = root.join("bin"); @@ -33,11 +30,12 @@ fn write_stub_agy(root: &Path) -> std::path::PathBuf { /// antigravity check present with the stub's version line. #[test] fn doctor_reports_antigravity_present_with_stub_on_path() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let bin = write_stub_agy(root); - let out = Command::new(devflow_bin()) + let out = common::hermetic_devflow(&home) .arg("doctor") .arg(root) .env("PATH", &bin) @@ -59,12 +57,13 @@ fn doctor_reports_antigravity_present_with_stub_on_path() { /// missing/warn — never a hard failure, never a false green. #[test] fn doctor_reports_antigravity_absent_without_agy() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let empty = root.join("empty-bin"); std::fs::create_dir_all(&empty).unwrap(); - let out = Command::new(devflow_bin()) + let out = common::hermetic_devflow(&home) .arg("doctor") .arg(root) .env("PATH", &empty) diff --git a/crates/devflow-cli/tests/gate_notify_persist_e2e.rs b/crates/devflow-cli/tests/gate_notify_persist_e2e.rs index 6ab10fed..6bd38dcf 100644 --- a/crates/devflow-cli/tests/gate_notify_persist_e2e.rs +++ b/crates/devflow-cli/tests/gate_notify_persist_e2e.rs @@ -21,6 +21,8 @@ //! The variable is set or removed on each spawned child only (TEST-01), so the //! developer's own shell hook never reaches these runs. +mod common; + use devflow_core::gates::{GATE_NOTIFY_CMD_ENV, Gates}; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; @@ -32,10 +34,6 @@ use std::path::{Path, PathBuf}; use std::process::{Child, Command}; use std::time::{Duration, Instant}; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - fn git(root: &Path, args: &[&str]) { let output = devflow_core::test_support::git_command(root) .args(args) @@ -109,29 +107,16 @@ fn fake_bin_dir() -> FakeBin { } /// Owns a spawned `devflow` process so a panicking assertion never leaks a -/// parked process for the length of its gate timeout, and the private cache -/// directory it registers in (`Drop::drop` reaps the child before the field -/// is removed). -struct ReapOnDrop( - Child, - #[expect(dead_code, reason = "held only so the cache dir outlives the child")] - tempfile::TempDir, -); - -/// A private `DEVFLOW_CACHE_DIR` for one spawned child, so its root -/// registration never lands in the developer's real `~/.cache/devflow`. -fn isolated_cache(command: &mut Command) -> tempfile::TempDir { - let cache = tempfile::tempdir().unwrap(); - command.env("DEVFLOW_CACHE_DIR", cache.path()); - cache -} - -impl Drop for ReapOnDrop { +/// parked process for the length of its gate timeout: dropping it kills and +/// reaps the child through [`common::kill_and_reap`]. The child is built by +/// [`common::hermetic_devflow`], so its root registration lands in the test's +/// [`common::HermeticHome`], which each test binds before this guard and so +/// outlives it. +struct ParkedChild(Child); + +impl Drop for ParkedChild { fn drop(&mut self) { - if matches!(self.0.try_wait(), Ok(None)) { - let _ = self.0.kill(); - } - let _ = self.0.wait(); + common::kill_and_reap(&mut self.0); } } @@ -148,7 +133,7 @@ fn apply_notify_env(command: &mut Command, notify: NotifyEnv<'_>) { }; } -fn wait_until_parked(child: &mut ReapOnDrop, gate: &Path, what: &str) { +fn wait_until_parked(child: &mut ParkedChild, gate: &Path, what: &str) { let waited = Instant::now(); while !gate.exists() { if let Some(status) = child.0.try_wait().expect("poll child") { @@ -181,24 +166,26 @@ fn wait_for_notify_fired(root: &Path) { } fn spawn_parked_start( + home: &common::HermeticHome, root: &Path, phase: PhaseId, fake_bin: &FakeBin, notify: NotifyEnv<'_>, -) -> ReapOnDrop { - spawn_parked_start_with(root, phase, fake_bin, notify, true) +) -> ParkedChild { + spawn_parked_start_with(home, root, phase, fake_bin, notify, true) } fn spawn_parked_start_with( + home: &common::HermeticHome, root: &Path, phase: PhaseId, fake_bin: &FakeBin, notify: NotifyEnv<'_>, legacy_launch: bool, -) -> ReapOnDrop { +) -> ParkedChild { let existing = std::env::var_os("PATH").unwrap_or_default(); let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); - let mut command = Command::new(devflow_bin()); + let mut command = common::hermetic_devflow(home); command .args([ "start", @@ -217,8 +204,7 @@ fn spawn_parked_start_with( .env("DEVFLOW_GATE_TIMEOUT_SECS", "30") .env(TEST_MARKER_ENV, root); apply_notify_env(&mut command, notify); - let cache = isolated_cache(&mut command); - let mut start = ReapOnDrop(command.spawn().expect("spawn foreground start"), cache); + let mut start = ParkedChild(command.spawn().expect("spawn foreground start")); wait_until_parked( &mut start, &Gates::gate_path(root, phase, Stage::Define), @@ -228,23 +214,23 @@ fn spawn_parked_start_with( } fn spawn_parked_advance( + home: &common::HermeticHome, root: &Path, phase: PhaseId, persisted: Option<&str>, notify: NotifyEnv<'_>, -) -> ReapOnDrop { +) -> ParkedChild { let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); state.stage = Stage::Code; state.gate_notify_cmd = persisted.map(str::to_owned); devflow_core::workflow::save_state(&state).unwrap(); - let mut command = Command::new(devflow_bin()); + let mut command = common::hermetic_devflow(home); command .args(["advance", "--phase", &phase.to_string()]) .arg(root) .env("DEVFLOW_GATE_TIMEOUT_SECS", "30"); apply_notify_env(&mut command, notify); - let cache = isolated_cache(&mut command); - let mut advance = ReapOnDrop(command.spawn().expect("spawn foreground advance"), cache); + let mut advance = ParkedChild(command.spawn().expect("spawn foreground advance")); wait_until_parked( &mut advance, &Gates::gate_path(root, phase, Stage::Code), @@ -261,6 +247,7 @@ fn persisted_cmd(root: &Path, phase: PhaseId) -> Option { #[test] fn start_persists_the_notify_command_from_the_environment() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(94); @@ -269,26 +256,28 @@ fn start_persists_the_notify_command_from_the_environment() { let cmd = format!("touch {}", sentinel.path().join("fired").display()); let fake_bin = fake_bin_dir(); - let _start = spawn_parked_start(root, phase, &fake_bin, NotifyEnv::Set(&cmd)); + let _start = spawn_parked_start(&home, root, phase, &fake_bin, NotifyEnv::Set(&cmd)); assert_eq!(persisted_cmd(root, phase).as_deref(), Some(cmd.as_str())); } #[test] fn start_with_an_empty_notify_variable_persists_nothing() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(94); init_start_repo(root, phase); let fake_bin = fake_bin_dir(); - let _start = spawn_parked_start(root, phase, &fake_bin, NotifyEnv::Set("")); + let _start = spawn_parked_start(&home, root, phase, &fake_bin, NotifyEnv::Set("")); assert_eq!(persisted_cmd(root, phase), None); } #[test] fn advance_gate_runs_the_persisted_command_when_the_variable_is_unset() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(95); @@ -297,7 +286,7 @@ fn advance_gate_runs_the_persisted_command_when_the_variable_is_unset() { let sentinel = sentinel_dir.path().join("fired"); let cmd = format!("touch {}", sentinel.display()); - let _advance = spawn_parked_advance(root, phase, Some(&cmd), NotifyEnv::Unset); + let _advance = spawn_parked_advance(&home, root, phase, Some(&cmd), NotifyEnv::Unset); wait_for_notify_fired(root); assert!(sentinel.exists(), "the persisted notify command must run"); @@ -310,6 +299,7 @@ fn advance_gate_runs_the_persisted_command_when_the_variable_is_unset() { #[test] fn advance_gate_prefers_a_set_variable_over_the_persisted_command() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(95); @@ -321,6 +311,7 @@ fn advance_gate_prefers_a_set_variable_over_the_persisted_command() { let override_cmd = format!("touch {}", overriding.display()); let _advance = spawn_parked_advance( + &home, root, phase, Some(&persisted_hook), @@ -342,6 +333,7 @@ fn advance_gate_prefers_a_set_variable_over_the_persisted_command() { #[test] fn advance_gate_with_an_empty_variable_skips_the_persisted_command() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(95); @@ -350,7 +342,7 @@ fn advance_gate_with_an_empty_variable_skips_the_persisted_command() { let sentinel = sentinel_dir.path().join("fired"); let cmd = format!("touch {}", sentinel.display()); - let _advance = spawn_parked_advance(root, phase, Some(&cmd), NotifyEnv::Set("")); + let _advance = spawn_parked_advance(&home, root, phase, Some(&cmd), NotifyEnv::Set("")); wait_for_notify_fired(root); assert!( @@ -368,6 +360,7 @@ fn advance_gate_with_an_empty_variable_skips_the_persisted_command() { /// persisted command, so a command exported (or emptied) after `start` /// reaches the chain it launches. fn resume_with(notify: NotifyEnv<'_>) -> Option { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(97); @@ -379,7 +372,7 @@ fn resume_with(notify: NotifyEnv<'_>) -> Option { let _reaper = MarkedReaper(root.to_string_lossy().into_owned()); let existing = std::env::var_os("PATH").unwrap_or_default(); let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); - let mut command = Command::new(devflow_bin()); + let mut command = common::hermetic_devflow(&home); command .args([ "resume", @@ -393,8 +386,8 @@ fn resume_with(notify: NotifyEnv<'_>) -> Option { .env("DEVFLOW_GATE_TIMEOUT_SECS", "30") .env(TEST_MARKER_ENV, root); apply_notify_env(&mut command, notify); - let cache = isolated_cache(&mut command); - let mut resume = ReapOnDrop(command.spawn().expect("spawn resume"), cache); + let mut resume = ParkedChild(command.spawn().expect("spawn resume")); + let _run_reaper = common::LaunchedRunReaper::new(root, phase, Vec::new()); let waited = Instant::now(); loop { let saved = persisted_cmd(root, phase); @@ -467,8 +460,13 @@ fn marked_processes(marker: &str) -> Vec<(u32, String, Vec)> { /// Approve the parked Define gate, wait for the monitor chain to launch the /// fake agent, and return the agent's environment from `/proc`. Every marked /// process is killed before returning. -fn agent_environment_after_define(root: &Path, phase: PhaseId, fake_bin: &FakeBin) -> Vec { - let approve = Command::new(devflow_bin()) +fn agent_environment_after_define( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &FakeBin, +) -> Vec { + let approve = common::hermetic_devflow(home) .args(["gate", "approve", &phase.to_string(), "define"]) .arg(root) .output() @@ -495,7 +493,7 @@ fn agent_environment_after_define(root: &Path, phase: PhaseId, fake_bin: &FakeBi ); std::thread::sleep(Duration::from_millis(20)); }; - let _ = Command::new(devflow_bin()) + let _ = common::hermetic_devflow(home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output(); @@ -503,20 +501,23 @@ fn agent_environment_after_define(root: &Path, phase: PhaseId, fake_bin: &FakeBi } fn assert_chain_has_no_notify_variable(legacy_launch: bool) { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(96); init_start_repo(root, phase); let fake_bin = fake_bin_dir(); let _start = spawn_parked_start_with( + &home, root, phase, &fake_bin, NotifyEnv::Set("touch /nonexistent/notify-sentinel"), legacy_launch, ); + let _run_reaper = common::LaunchedRunReaper::new(root, phase, Vec::new()); - let environment = agent_environment_after_define(root, phase, &fake_bin); + let environment = agent_environment_after_define(&home, root, phase, &fake_bin); // Control: a variable set on `start` the same way does reach the agent, // so an absent entry below means stripped, not unread. diff --git a/crates/devflow-cli/tests/gate_sweep_e2e.rs b/crates/devflow-cli/tests/gate_sweep_e2e.rs index 605655d1..d928ee7b 100644 --- a/crates/devflow-cli/tests/gate_sweep_e2e.rs +++ b/crates/devflow-cli/tests/gate_sweep_e2e.rs @@ -9,19 +9,16 @@ //! `release_check.rs`) — this drives the real compiled binary via //! `Command::new`, not an in-process call. +mod common; + use devflow_core::gates::{GateAction, GateFile, Gates}; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; use devflow_core::stage::Stage; use devflow_core::state::{AgentKind, State}; use std::path::Path; -use std::process::Command; use std::time::{Duration, Instant}; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - /// Hermetic `git` invocation for fixture setup (999.37) — never a bare /// `Command::new("git")`. fn git(root: &Path, args: &[&str]) { @@ -162,6 +159,7 @@ const AGED_WELL_PAST_DEFAULT_THRESHOLD_SECS: u64 = 30 * 24 * 60 * 60; /// it up and resolves to `GateAction::Abort`. #[test] fn sweep_reaps_an_aged_gate_and_a_real_poller_resolves_to_abort() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(91); @@ -184,7 +182,7 @@ fn sweep_reaps_an_aged_gate_and_a_real_poller_resolves_to_abort() { std::thread::scope(|scope| { let poller = scope.spawn(move || Gates::poll_response(root, phase, stage, 30)); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["gate", "sweep", "--root"]) .arg(root) .output() @@ -216,6 +214,7 @@ fn sweep_reaps_an_aged_gate_and_a_real_poller_resolves_to_abort() { /// change. #[test] fn sweep_leaves_a_fresh_gate_untouched() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(92); @@ -224,7 +223,7 @@ fn sweep_leaves_a_fresh_gate_untouched() { // Freshly written — `unix_now()`'s age is ~0s, well under the 6h default. Gates::write_gate(root, phase, stage, "review gaps").unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["gate", "sweep", "--root"]) .arg(root) .output() @@ -254,7 +253,8 @@ fn sweep_leaves_a_fresh_gate_untouched() { /// in the subcommand's own `--help`. #[test] fn sweep_help_documents_max_age_and_dry_run() { - let output = Command::new(devflow_bin()) + let home = common::HermeticHome::new(); + let output = common::hermetic_devflow(&home) .args(["gate", "sweep", "--help"]) .output() .expect("run devflow gate sweep --help"); @@ -283,6 +283,7 @@ fn sweep_help_documents_max_age_and_dry_run() { /// OS process via `Command::new`, not driven in-process on a thread. #[test] fn sweep_ends_a_real_advance_process_through_its_own_abort_path() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(95); @@ -300,7 +301,7 @@ fn sweep_ends_a_real_advance_process_through_its_own_abort_path() { // block for `gate_timeout_secs()`'s multi-day production default. // Set per-`Command` via `.env(...)`, not via a process-global env // mutation (999.37) — so this cannot race any other test. - let mut child = Command::new(devflow_bin()) + let mut child = common::hermetic_devflow(&home) .args(["advance", "--phase", &phase.to_string()]) .arg(root) .env("DEVFLOW_GATE_TIMEOUT_SECS", "15") @@ -371,3 +372,221 @@ fn sweep_ends_a_real_advance_process_through_its_own_abort_path() { // forms (must be zero), not asserted at runtime — there is no API // surface here that could send one. } + +// --------------------------------------------------------------------------- +// 50-03 (999.141, T-50-05): the machine-wide stray pass stays dry-run in every +// suite on the hermetic builder. +// +// `gate sweep --reap-strays` scans the whole machine's `/proc` and reaps every +// DevFlow-shaped process the caller owns, sparing only the ones the registry +// knows about. On `common::hermetic_devflow` the registry holds nothing but +// the test's own roots, so a real (non-dry-run) pass from a test would treat +// every live run on the developer's machine as a stray. This scan refuses one. +// --------------------------------------------------------------------------- + +/// The `"--reap-strays"` argument literal, quotes included. Spelled through +/// `concat!` split inside the token so this file's own source never holds it. +const REAP_STRAYS_LITERAL: &str = concat!("\"--reap-", "strays\""); + +/// The `"--dry-run"` argument literal, quotes included, spelled the same way. +const DRY_RUN_LITERAL: &str = concat!("\"--dry-", "run\""); + +/// `src` with every `//` comment (line, doc and trailing) removed. A `//` +/// inside a string, raw string or char literal is not a comment. A local copy +/// of `test_home_isolation.rs`'s scanner: integration test binaries cannot +/// share private helpers, and `common/mod.rs` holds no scan code. +fn code_only(src: &str) -> String { + let chars: Vec = src.chars().collect(); + let mut out = String::with_capacity(src.len()); + let mut index = 0; + while index < chars.len() { + index = match chars[index] { + '/' if chars.get(index + 1) == Some(&'/') => skip_to_line_end(&chars, index), + '"' => copy_string(&chars, index, &mut out), + 'r' if raw_string_hashes(&chars, index).is_some() => { + copy_raw_string(&chars, index, &mut out) + } + '\'' => copy_char_or_lifetime(&chars, index, &mut out), + other => { + out.push(other); + index + 1 + } + }; + } + out +} + +fn skip_to_line_end(chars: &[char], mut index: usize) -> usize { + while index < chars.len() && chars[index] != '\n' { + index += 1; + } + index +} + +/// Copies a `"…"` literal starting at `start`, honouring `\` escapes. +fn copy_string(chars: &[char], start: usize, out: &mut String) -> usize { + out.push('"'); + let mut index = start + 1; + while index < chars.len() { + out.push(chars[index]); + match chars[index] { + '\\' if index + 1 < chars.len() => { + out.push(chars[index + 1]); + index += 2; + } + '"' => return index + 1, + _ => index += 1, + } + } + index +} + +fn is_ident_char(c: char) -> bool { + c.is_alphanumeric() || c == '_' +} + +/// The `#` count of a raw string (`r"`, `r#"`, `br"`…) starting at the `r` at +/// `index`, or `None` when that `r` is part of an identifier or not followed +/// by `#*"`. +fn raw_string_hashes(chars: &[char], index: usize) -> Option { + let before = index.checked_sub(1).map(|i| chars[i]); + let prefix_ok = match before { + None => true, + Some('b') => index < 2 || !is_ident_char(chars[index - 2]), + Some(c) => !is_ident_char(c), + }; + if !prefix_ok { + return None; + } + let hashes = chars[index + 1..].iter().take_while(|&&c| c == '#').count(); + (chars.get(index + 1 + hashes) == Some(&'"')).then_some(hashes) +} + +fn copy_raw_string(chars: &[char], start: usize, out: &mut String) -> usize { + let hashes = raw_string_hashes(chars, start).expect("caller checked the raw-string opener"); + let body = start + 2 + hashes; + out.extend(&chars[start..body]); + let mut index = body; + while index < chars.len() { + let closes = chars[index] == '"' + && chars[index + 1..] + .iter() + .take(hashes) + .filter(|&&c| c == '#') + .count() + == hashes; + if closes { + out.extend(&chars[index..=index + hashes]); + return index + hashes + 1; + } + out.push(chars[index]); + index += 1; + } + index +} + +/// Copies a char literal (`'x'`, `'\''`, `'\u{1F600}'`) or, for a lifetime +/// or label, only its quote. +fn copy_char_or_lifetime(chars: &[char], start: usize, out: &mut String) -> usize { + let end = if chars.get(start + 1) == Some(&'\\') { + chars + .get(start + 2..) + .unwrap_or_default() + .iter() + .position(|&c| c == '\'') + .map(|offset| start + 2 + offset) + } else if chars.get(start + 2) == Some(&'\'') { + Some(start + 2) + } else { + None + }; + match end { + Some(end) => { + out.extend(&chars[start..=end]); + end + 1 + } + None => { + out.push('\''); + start + 1 + } + } +} + +/// The `;`-separated statements of comment-stripped `code` that pass +/// `"--reap-strays"` without `"--dry-run"`, trimmed. A test that adds +/// `--dry-run` in a separate statement is flagged too: the conservative +/// direction. +fn real_reap_strays_statements(code: &str) -> Vec { + code.split(';') + .filter(|statement| { + statement.contains(REAP_STRAYS_LITERAL) && !statement.contains(DRY_RUN_LITERAL) + }) + .map(|statement| statement.trim().to_string()) + .collect() +} + +#[test] +fn no_suite_on_the_common_builder_runs_a_real_reap_strays_pass() { + // (1) Planted controls: a real pass is flagged, a dry-run pass and a + // commented-out real pass are not. + let planted = concat!( + "let out = common::hermetic_devflow(&home).args([\"gate\", \"sweep\", \"--reap-", + "strays\"]).output();" + ); + assert_eq!( + real_reap_strays_statements(&code_only(planted)).len(), + 1, + "a planted real --reap-strays pass must be flagged" + ); + let planted_dry_run = concat!( + "let out = common::hermetic_devflow(&home).args([\"gate\", \"sweep\", \"--reap-", + "strays\", \"--dry-", + "run\"]).output();" + ); + assert_eq!( + real_reap_strays_statements(&code_only(planted_dry_run)), + Vec::::new(), + "a planted --reap-strays pass with --dry-run must pass" + ); + let commented = format!("// {planted}\n"); + assert_eq!( + real_reap_strays_statements(&code_only(&commented)), + Vec::::new(), + "a commented-out real --reap-strays pass must pass" + ); + + // (2) Every suite directly under `tests/` that declares `mod common;`; + // this file always does. + let tests_dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests"); + let mut paths: Vec<_> = std::fs::read_dir(&tests_dir) + .unwrap_or_else(|err| panic!("read {}: {err}", tests_dir.display())) + .map(|entry| entry.expect("read tests dir entry").path()) + .filter(|path| path.is_file() && path.extension().is_some_and(|ext| ext == "rs")) + .collect(); + paths.sort(); + let mut scanned = 0; + let mut offenders = Vec::new(); + for path in paths { + let source = std::fs::read_to_string(&path) + .unwrap_or_else(|err| panic!("read {}: {err}", path.display())); + let code = code_only(&source); + if !code.lines().any(|line| line.trim() == "mod common;") { + continue; + } + scanned += 1; + for statement in real_reap_strays_statements(&code) { + offenders.push(format!("{}: {statement}", path.display())); + } + } + assert!( + scanned > 0, + "no suite under {} declares `mod common;`, so the scan checked nothing", + tests_dir.display() + ); + assert!( + offenders.is_empty(), + "a suite on the hermetic builder runs a real (non-dry-run) --reap-strays pass, \ + whose safety set would shrink to the test's own roots:\n{}", + offenders.join("\n") + ); +} diff --git a/crates/devflow-cli/tests/gate_wedge_e2e.rs b/crates/devflow-cli/tests/gate_wedge_e2e.rs index a2227284..d5b85169 100644 --- a/crates/devflow-cli/tests/gate_wedge_e2e.rs +++ b/crates/devflow-cli/tests/gate_wedge_e2e.rs @@ -7,6 +7,8 @@ //! - the `advance_*` arms cover a foreground `devflow advance` parked at a //! Code gate. +mod common; + use devflow_core::gates::Gates; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; @@ -18,10 +20,6 @@ use std::path::{Path, PathBuf}; use std::process::{Child, Command}; use std::time::{Duration, Instant}; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - fn git(root: &Path, args: &[&str]) { let output = devflow_core::test_support::git_command(root) .args(args) @@ -62,11 +60,11 @@ fn wait_for(mut predicate: impl FnMut() -> bool, what: &str) { } } -fn spawn_gated_advance(root: &Path, phase: PhaseId) -> Child { +fn spawn_gated_advance(home: &common::HermeticHome, root: &Path, phase: PhaseId) -> Child { let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); state.stage = Stage::Code; devflow_core::workflow::save_state(&state).unwrap(); - let child = Command::new(devflow_bin()) + let child = common::hermetic_devflow(home) .args(["advance", "--phase", &phase.to_string()]) .arg(root) .env("DEVFLOW_GATE_TIMEOUT_SECS", "15") @@ -83,8 +81,8 @@ fn spawn_gated_advance(root: &Path, phase: PhaseId) -> Child { child } -fn reject(root: &Path, phase: PhaseId) -> std::process::Output { - Command::new(devflow_bin()) +fn reject(home: &common::HermeticHome, root: &Path, phase: PhaseId) -> std::process::Output { + common::hermetic_devflow(home) .args([ "gate", "reject", @@ -142,10 +140,15 @@ fn fake_bin_dir(agent_body: &str) -> FakeBin { /// A foreground `devflow start --mode auto` with the fake agent first on the /// child's PATH. PATH is set on the child only (TEST-01). -fn start_child(root: &Path, phase: PhaseId, fake_bin: &FakeBin) -> Command { +fn start_child( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &FakeBin, +) -> Command { let existing = std::env::var_os("PATH").unwrap_or_default(); let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); - let mut command = Command::new(devflow_bin()); + let mut command = common::hermetic_devflow(home); command .args([ "start", @@ -180,13 +183,14 @@ impl Drop for ReapOnDrop { /// Spawns `start` (owned by `ReapOnDrop` from the moment it exists) and waits /// up to 20 s for it to park at the Define preflight gate. fn spawn_parked_start( + home: &common::HermeticHome, root: &Path, phase: PhaseId, fake_bin: &FakeBin, gate_timeout_secs: &str, ) -> ReapOnDrop { let mut start = ReapOnDrop( - start_child(root, phase, fake_bin) + start_child(home, root, phase, fake_bin) .env("DEVFLOW_GATE_TIMEOUT_SECS", gate_timeout_secs) .spawn() .expect("spawn foreground start"), @@ -205,8 +209,8 @@ fn spawn_parked_start( start } -fn reject_define(root: &Path, phase: PhaseId) -> std::process::Output { - Command::new(devflow_bin()) +fn reject_define(home: &common::HermeticHome, root: &Path, phase: PhaseId) -> std::process::Output { + common::hermetic_devflow(home) .args([ "gate", "reject", @@ -245,15 +249,16 @@ fn phase_gate_entries(root: &Path, phase: PhaseId) -> Vec { #[test] fn advance_wedge_arm_killed_advance_leaves_a_code_gate_that_reject_reports_honestly() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(4816); init_repo(root, phase); - let mut child = spawn_gated_advance(root, phase); + let mut child = spawn_gated_advance(&home, root, phase); child.kill().expect("kill foreground advance"); child.wait().expect("reap foreground advance"); - let output = reject(root, phase); + let output = reject(&home, root, phase); assert!( !output.status.success(), "wedge reject unexpectedly succeeded: {}", @@ -273,7 +278,7 @@ fn advance_wedge_arm_killed_advance_leaves_a_code_gate_that_reject_reports_hones "unexpected output: {output_text}" ); - let recovery = Command::new(devflow_bin()) + let recovery = common::hermetic_devflow(&home) .args(["recover", "--clean", "--phase", &phase.to_string()]) .arg(root) .output() @@ -323,13 +328,14 @@ fn advance_wedge_arm_killed_advance_leaves_a_code_gate_that_reject_reports_hones #[test] fn advance_self_resolving_arm_live_advance_consumes_the_code_rejection() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(4817); init_repo(root, phase); - let mut child = spawn_gated_advance(root, phase); + let mut child = spawn_gated_advance(&home, root, phase); let before = Instant::now(); - let output = reject(root, phase); + let output = reject(&home, root, phase); assert!( output.status.success(), "live reject failed: {}", @@ -359,6 +365,7 @@ fn advance_self_resolving_arm_live_advance_consumes_the_code_rejection() { #[test] fn dry_run_sweep_reports_no_waiter_gates_as_left_alone() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(4818); @@ -375,7 +382,7 @@ fn dry_run_sweep_reports_no_waiter_gates_as_left_alone() { std::fs::write(lock_path, format!("{pid}\n{start}")).unwrap(); Gates::write_gate(root, live_ship_phase, Stage::Ship, "live poller").unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "gate", "sweep", @@ -425,12 +432,13 @@ fn dry_run_sweep_reports_no_waiter_gates_as_left_alone() { #[test] fn start_wedge_arm_interrupted_start_leaves_a_define_gate_nothing_answers() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(93); init_start_repo(root, phase); let fake_bin = fake_bin_dir("sleep 60"); - let mut start = spawn_parked_start(root, phase, &fake_bin, "30"); + let mut start = spawn_parked_start(&home, root, phase, &fake_bin, "30"); assert_eq!( devflow_core::lock::holder_identity(root, phase).map(|(pid, _)| pid), @@ -445,7 +453,7 @@ fn start_wedge_arm_interrupted_start_leaves_a_define_gate_nothing_answers() { "the interrupted start must leave its Define gate behind" ); - let output = reject_define(root, phase); + let output = reject_define(&home, root, phase); let output_text = format!( "{}{}", String::from_utf8_lossy(&output.stdout), @@ -502,7 +510,7 @@ fn start_wedge_arm_interrupted_start_leaves_a_define_gate_nothing_answers() { ); } - let recovery = Command::new(devflow_bin()) + let recovery = common::hermetic_devflow(&home) .args(["recover", "--clean", "--phase", &phase.to_string()]) .arg(root) .output() @@ -539,12 +547,13 @@ fn start_wedge_arm_interrupted_start_leaves_a_define_gate_nothing_answers() { #[test] fn start_self_resolving_arm_live_start_consumes_the_define_rejection() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(94); init_start_repo(root, phase); let fake_bin = fake_bin_dir("sleep 60"); - let mut start = spawn_parked_start(root, phase, &fake_bin, "60"); + let mut start = spawn_parked_start(&home, root, phase, &fake_bin, "60"); assert_eq!( devflow_core::lock::holder_identity(root, phase).map(|(pid, _)| pid), @@ -553,7 +562,7 @@ fn start_self_resolving_arm_live_start_consumes_the_define_rejection() { ); let before = Instant::now(); - let output = reject_define(root, phase); + let output = reject_define(&home, root, phase); let stdout = String::from_utf8_lossy(&output.stdout); assert!( output.status.success(), diff --git a/crates/devflow-cli/tests/help_snapshot.rs b/crates/devflow-cli/tests/help_snapshot.rs index 72722239..f1258e40 100644 --- a/crates/devflow-cli/tests/help_snapshot.rs +++ b/crates/devflow-cli/tests/help_snapshot.rs @@ -5,12 +5,12 @@ //! ```bash //! cargo run -q -p devflow -- --help > crates/devflow-cli/tests/snapshots/devflow-help.txt //! ``` +//! +//! The child is built by `common::hermetic_devflow`, so it never reads the +//! operator's real config or cache (999.141); the help text does not depend +//! on either. -use std::process::Command; - -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} +mod common; /// Normalize trailing whitespace per line so editor/clap cosmetic drift /// doesn't produce false diffs. @@ -25,7 +25,8 @@ fn normalized(text: &str) -> String { #[test] fn help_output_matches_committed_snapshot() { - let output = Command::new(devflow_bin()) + let home = common::HermeticHome::new(); + let output = common::hermetic_devflow(&home) .arg("--help") .output() .expect("run devflow --help"); diff --git a/crates/devflow-cli/tests/hermetic_spawn_sites.rs b/crates/devflow-cli/tests/hermetic_spawn_sites.rs new file mode 100644 index 00000000..08f08233 --- /dev/null +++ b/crates/devflow-cli/tests/hermetic_spawn_sites.rs @@ -0,0 +1,231 @@ +//! Permanent source scan for 999.141 / D-07 (50-21, R6-G4): every e2e suite +//! spawns `devflow` through `common::hermetic_devflow`, so no suite's child +//! reaches the operator's real `~/.cache/devflow` (and, from 50-11, no suite +//! binds a socket there). +//! +//! It replaces the shell counts the migration plans used as gates. It spawns +//! nothing and does not include the shared test helpers. The scanned tokens +//! are spelled through `concat!`, so this file never holds the binary's env +//! name or a raw spawn as written and cannot match itself. +//! +//! The comment rule is deliberately simple: a line whose trimmed start is +//! `//` is a comment and is skipped; every other line is code. + +use std::fs; +use std::path::{Path, PathBuf}; + +/// Suites migrated to the hermetic builder whose spawn sites this scan pins. +const HERMETIC_SUITES: &[&str] = &[ + "auto_chain_flag_e2e.rs", + "yes_ship_config.rs", + "release_check.rs", + "reap_strays_e2e.rs", + "help_snapshot.rs", +]; + +/// The only files under `tests/` (paths relative to it, `/`-separated) whose +/// code may name the binary's env var. `common/mod.rs` is the one legitimate +/// site. `recover_clean_e2e.rs` is 50-04's migration and is tolerated so the +/// scan holds whichever of 50-04 and 50-21 lands first. +const BINARY_NAMERS: &[&str] = &["common/mod.rs", "recover_clean_e2e.rs"]; + +/// `crates/devflow-cli/tests`. +fn tests_dir() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("tests") +} + +/// The lines of `source` whose trimmed start is not `//`. +fn code_lines(source: &str) -> Vec<&str> { + source + .lines() + .filter(|line| !line.trim_start().starts_with("//")) + .collect() +} + +/// Cargo's env var naming the compiled `devflow` binary, assembled so this +/// file never spells it. +fn binary_env_name() -> &'static str { + concat!("CARGO_BIN_EXE_", "devflow") +} + +/// Occurrences, in the code lines of `source`, of a raw `devflow` spawn +/// (`Command::new` over the local or the shared `devflow_bin()`) or of the +/// binary's env name. +fn raw_spawn_sites(source: &str) -> usize { + let tokens = [ + concat!("Command::new(", "devflow_bin())"), + concat!("Command::new(common::", "devflow_bin())"), + binary_env_name(), + ]; + code_lines(source) + .iter() + .map(|line| { + tokens + .iter() + .map(|token| line.matches(token).count()) + .sum::() + }) + .sum() +} + +/// The source of `tests/`. Panics naming the path when it cannot be +/// read, so a missing or renamed suite fails rather than reading as clean. +fn read_suite(name: &str) -> String { + let path = tests_dir().join(name); + fs::read_to_string(&path) + .unwrap_or_else(|err| panic!("cannot read suite {}: {err}", path.display())) +} + +/// Whether any code line of `source` names the binary's env var. +fn names_binary(source: &str) -> bool { + code_lines(source) + .iter() + .any(|line| line.contains(binary_env_name())) +} + +/// Every `.rs` file below `dir`, recursively, as a path relative to `base` +/// joined with `/`. Panics on an unreadable directory or entry, so a walk +/// that cannot see a file fails instead of reading as clean. +fn rust_files_under(base: &Path, dir: &Path, files: &mut Vec<(String, PathBuf)>) { + let entries = fs::read_dir(dir) + .unwrap_or_else(|err| panic!("cannot read directory {}: {err}", dir.display())); + for entry in entries { + let entry = + entry.unwrap_or_else(|err| panic!("cannot read an entry of {}: {err}", dir.display())); + let path = entry.path(); + // `file_type` does not follow symlinks, so a link loop cannot recurse. + let file_type = entry + .file_type() + .unwrap_or_else(|err| panic!("cannot stat {}: {err}", path.display())); + if file_type.is_dir() { + rust_files_under(base, &path, files); + } else if path.extension().is_some_and(|ext| ext == "rs") { + let relative = path + .strip_prefix(base) + .expect("walked path is under the walk base") + .components() + .map(|part| part.as_os_str().to_string_lossy().into_owned()) + .collect::>() + .join("/"); + files.push((relative, path)); + } + } +} + +#[test] +fn migrated_suites_spawn_devflow_only_through_the_hermetic_builder() { + // Planted controls first, on inline strings: the matcher must flag each + // raw form and pass a comment and the hermetic builder. + let raw_spawn = concat!("let out = Command::new(", "devflow_bin()).output();"); + assert_eq!( + raw_spawn_sites(raw_spawn), + 1, + "a raw local spawn must count" + ); + let shared_spawn = concat!( + "let out = Command::new(common::", + "devflow_bin()).output();" + ); + assert_eq!( + raw_spawn_sites(shared_spawn), + 1, + "a raw spawn over the shared devflow_bin() must count" + ); + let env_line = format!("let bin = env!(\"{}\");", binary_env_name()); + assert_eq!( + raw_spawn_sites(&env_line), + 1, + "a code line naming the binary's env var must count" + ); + let commented = format!("// {raw_spawn}"); + assert_eq!( + raw_spawn_sites(&commented), + 0, + "a commented-out spawn must not count" + ); + let hermetic = "let out = common::hermetic_devflow(&home).output();"; + assert_eq!( + raw_spawn_sites(hermetic), + 0, + "the hermetic builder must not count" + ); + + // Then every listed suite, read from disk. + let mut failures = Vec::new(); + for name in HERMETIC_SUITES { + let source = read_suite(name); + let raw = raw_spawn_sites(&source); + if raw != 0 { + failures.push(format!( + "{name}: {raw} raw devflow spawn site(s) or binary env name(s); expected 0" + )); + } + // Real-data control: the scan read the migrated file, not an empty + // or unrelated string. + let uses_builder = code_lines(&source) + .iter() + .any(|line| line.contains("common::hermetic_devflow(")); + if !uses_builder { + failures.push(format!( + "{name}: no `common::hermetic_devflow(` call in its code lines" + )); + } + } + assert!( + failures.is_empty(), + "suites not spawning devflow only through the hermetic builder:\n{}", + failures.join("\n") + ); +} + +#[test] +fn only_common_names_the_devflow_binary() { + // Planted controls first, on inline strings run through the same matcher. + let naming = format!(" let bin = env!(\"{}\");", binary_env_name()); + assert!( + names_binary(&naming), + "a code line naming the binary's env var must be detected" + ); + let commented = format!("// {naming}"); + assert!( + !names_binary(&commented), + "the same line behind `// ` must not be detected" + ); + + let base = tests_dir(); + let mut files = Vec::new(); + rust_files_under(&base, &base, &mut files); + let mut namers: Vec = files + .into_iter() + .filter(|(_, path)| { + let source = fs::read_to_string(path) + .unwrap_or_else(|err| panic!("cannot read {}: {err}", path.display())); + names_binary(&source) + }) + .map(|(relative, _)| relative) + .collect(); + namers.sort(); + + // Real-data control: the walk and the matcher find the one legitimate + // site. + assert!( + namers.iter().any(|path| path == "common/mod.rs"), + "common/mod.rs was not found naming the binary under {}; found {namers:?}", + base.display() + ); + let unexpected: Vec<&String> = namers + .iter() + .filter(|path| !BINARY_NAMERS.contains(&path.as_str())) + .collect(); + assert!( + unexpected.is_empty(), + "files under {} name the devflow binary outside common/mod.rs; spawn it through \ + common::hermetic_devflow instead:\n{}", + base.display(), + unexpected + .iter() + .map(|path| path.as_str()) + .collect::>() + .join("\n") + ); +} diff --git a/crates/devflow-cli/tests/log_format_env.rs b/crates/devflow-cli/tests/log_format_env.rs index 8ee165bd..908ab74f 100644 --- a/crates/devflow-cli/tests/log_format_env.rs +++ b/crates/devflow-cli/tests/log_format_env.rs @@ -19,17 +19,14 @@ //! Tests below now assert log lines on stderr and assert their absence from //! stdout. +mod common; + use devflow_core::gates::Gates; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; use devflow_core::stage::Stage; use devflow_core::state::{AgentKind, State}; use std::path::PathBuf; -use std::process::Command; - -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} /// Set up a project dir with a legacy `.devflow/state.json` (phase 1, no /// matching per-phase file), so `devflow status` triggers the migration's @@ -57,8 +54,13 @@ fn project_with_legacy_state() -> tempfile::TempDir { dir } -fn run_status(dir: &PathBuf, log_format: Option<&str>, rust_log: Option<&str>) -> (String, String) { - let mut cmd = Command::new(devflow_bin()); +fn run_status( + home: &common::HermeticHome, + dir: &PathBuf, + log_format: Option<&str>, + rust_log: Option<&str>, +) -> (String, String) { + let mut cmd = common::hermetic_devflow(home); cmd.arg("status").arg(dir); // Start from a clean env so ambient RUST_LOG/DEVFLOW_LOG_FORMAT from the @@ -85,8 +87,14 @@ fn run_status(dir: &PathBuf, log_format: Option<&str>, rust_log: Option<&str>) - #[test] fn rust_log_debug_is_honored_under_json_log_format() { + let home = common::HermeticHome::new(); let dir = project_with_legacy_state(); - let (stdout, stderr) = run_status(&dir.path().to_path_buf(), Some("json"), Some("debug")); + let (stdout, stderr) = run_status( + &home, + &dir.path().to_path_buf(), + Some("json"), + Some("debug"), + ); assert!( stderr.contains("migrated legacy state.json"), @@ -104,12 +112,13 @@ fn rust_log_debug_is_honored_under_json_log_format() { #[test] fn rust_log_default_suppresses_debug_under_json_log_format() { + let home = common::HermeticHome::new(); let dir = project_with_legacy_state(); // No RUST_LOG set — tracing-subscriber's EnvFilter::from_default_env() // defaults to ERROR-level-only when the env var is absent, so this DEBUG // line must NOT appear. If it does, RUST_LOG is being ignored (i.e. a // fixed, too-verbose level is hardcoded again), the exact WR-01 bug class. - let (stdout, stderr) = run_status(&dir.path().to_path_buf(), Some("json"), None); + let (stdout, stderr) = run_status(&home, &dir.path().to_path_buf(), Some("json"), None); assert!( !stderr.contains("migrated legacy state.json"), @@ -145,9 +154,10 @@ fn project_with_open_gate(phase: PhaseId, stage: Stage) -> tempfile::TempDir { #[test] fn rust_log_unset_still_shows_info_level_logs_by_default() { + let home = common::HermeticHome::new(); let dir = project_with_open_gate(PhaseId::new(15), Stage::Ship); - let mut cmd = Command::new(devflow_bin()); + let mut cmd = common::hermetic_devflow(&home); cmd.arg("gate") .arg("approve") .arg("15") diff --git a/crates/devflow-cli/tests/phase7_cli.rs b/crates/devflow-cli/tests/phase7_cli.rs index 6603ee6c..d31eea4b 100644 --- a/crates/devflow-cli/tests/phase7_cli.rs +++ b/crates/devflow-cli/tests/phase7_cli.rs @@ -1,3 +1,5 @@ +mod common; + use devflow_core::phase_id::PhaseId; use devflow_core::stage::Stage; use devflow_core::workflow::load_state; @@ -12,10 +14,6 @@ struct FakeBin { path: PathBuf, } -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - fn git(root: &Path, args: &[&str]) -> Output { // Hermetic: pinning cwd alone does not stop an inherited GIT_DIR from // retargeting the real repository (999.37). @@ -93,8 +91,8 @@ fn path_with_fake_bin(fake_bin: &Path) -> String { format!("{}:{}", fake_bin.display(), existing.to_string_lossy()) } -fn run_devflow(root: &Path, fake_bin: &Path, args: &[&str]) -> Output { - run_devflow_inner(root, fake_bin, args, false) +fn run_devflow(home: &common::HermeticHome, root: &Path, fake_bin: &Path, args: &[&str]) -> Output { + run_devflow_inner(home, root, fake_bin, args, false) } /// [`run_devflow`] with D-11's legacy-launch opt-out forced on for the spawned @@ -123,12 +121,23 @@ fn run_devflow(root: &Path, fake_bin: &Path, args: &[&str]) -> Output { /// coverage rather than stabilising it; see /// `parallel_creates_two_worktrees_and_spawns_two_monitors`, which deliberately /// does not use this helper. -fn run_devflow_legacy_launch(root: &Path, fake_bin: &Path, args: &[&str]) -> Output { - run_devflow_inner(root, fake_bin, args, true) +fn run_devflow_legacy_launch( + home: &common::HermeticHome, + root: &Path, + fake_bin: &Path, + args: &[&str], +) -> Output { + run_devflow_inner(home, root, fake_bin, args, true) } -fn run_devflow_inner(root: &Path, fake_bin: &Path, args: &[&str], legacy_launch: bool) -> Output { - let mut command = Command::new(devflow_bin()); +fn run_devflow_inner( + home: &common::HermeticHome, + root: &Path, + fake_bin: &Path, + args: &[&str], + legacy_launch: bool, +) -> Output { + let mut command = common::hermetic_devflow(home); command .args(args) .arg(root) @@ -266,6 +275,7 @@ fn seed_feature_branch(root: &Path, phase: PhaseId) { #[test] fn parallel_creates_two_worktrees_and_spawns_two_monitors() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -324,6 +334,7 @@ esac // test's subject is two worktrees and two monitors, which is // mode-independent; the agent diversity is the point and is preserved. let output = run_devflow( + &home, root, &fake_bin.path, &[ @@ -336,6 +347,8 @@ esac "supervise", ], ); + let _reaper7 = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); + let _reaper8 = common::LaunchedRunReaper::new(root, PhaseId::new(8), Vec::new()); let stdout = String::from_utf8_lossy(&output.stdout); assert!(stdout.contains("phase 7")); assert!(stdout.contains("phase 8")); @@ -382,6 +395,7 @@ esac #[test] fn start_defaults_to_worktree() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -397,6 +411,7 @@ fn start_defaults_to_worktree() { // puts the phase, not which transport its agent is launched over. See // `run_devflow_legacy_launch`. run_devflow_legacy_launch( + &home, root, &fake_bin.path, &[ @@ -412,6 +427,7 @@ fn start_defaults_to_worktree() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(11), Vec::new()); // start returns before the detached monitor finishes; wait for the // worktree directory like the other integration tests do. @@ -441,6 +457,7 @@ fn start_defaults_to_worktree() { /// phase's worktree, which always starts at ahead=0, behind=0. #[test] fn start_worktree_mode_ignores_main_checkout_divergence() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -473,6 +490,7 @@ fn start_worktree_mode_ignores_main_checkout_divergence() { // divergence check's scope, not the launch transport. See // `run_devflow_legacy_launch`. run_devflow_legacy_launch( + &home, root, &fake_bin.path, &[ @@ -488,6 +506,7 @@ fn start_worktree_mode_ignores_main_checkout_divergence() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(13), Vec::new()); wait_for(&root.join(".worktrees/phase-13")); assert!(root.join(".worktrees/phase-13").is_dir()); @@ -500,6 +519,7 @@ fn start_worktree_mode_ignores_main_checkout_divergence() { #[test] fn start_no_worktree_uses_feature_branch() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -512,6 +532,7 @@ fn start_no_worktree_uses_feature_branch() { // `--no-worktree` keeps the phase on its feature branch, not which // transport its agent is launched over. See `run_devflow_legacy_launch`. run_devflow_legacy_launch( + &home, root, &fake_bin.path, &[ @@ -528,6 +549,7 @@ fn start_no_worktree_uses_feature_branch() { "--no-worktree", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(12), Vec::new()); // start returns before the detached monitor finishes; wait for the // agent pid file that the monitor writes on the feature-branch path. @@ -555,6 +577,7 @@ fn start_no_worktree_uses_feature_branch() { /// case this plan adds. #[test] fn start_until_plan_halts_cleanly() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -569,6 +592,7 @@ fn start_until_plan_halts_cleanly() { // into `state.legacy_claude_launch` by `start`, so the detached monitor // that launches Plan reads it back. See `run_devflow_legacy_launch`. run_devflow_legacy_launch( + &home, root, &fake_bin.path, &[ @@ -586,6 +610,7 @@ fn start_until_plan_halts_cleanly() { "plan", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(44), Vec::new()); let state = wait_for_stopped(root, PhaseId::new(44)); assert_eq!( @@ -596,14 +621,14 @@ fn start_until_plan_halts_cleanly() { ); assert!(state.stopped, "stop marker must be set"); assert_eq!( - state.monitor_pid, None, - "the stop path must clear monitor_pid so no monitor is left behind" + state.supervisor, None, + "the stop path must clear the supervisor record so no monitor is left behind" ); assert!( state.stop_reason.is_some(), "a human-readable stop_reason must be recorded" ); - // The stop path cleared monitor_pid; the guard still binds (no-op) so + // The stop path cleared the supervisor record; the guard still binds (no-op) so // every monitor-spawning test in this file follows the same teardown // shape (41-02 Task 1 systematic pass). let _reap = MonitorReapGuard::after_launch(&state); @@ -615,6 +640,7 @@ fn start_until_plan_halts_cleanly() { /// any stage runs, not silently accepted as if it intercepted anything. #[test] fn start_until_ship_is_rejected() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -623,7 +649,7 @@ fn start_until_ship_is_rejected() { "#!/bin/sh\nprintf 'DEVFLOW_RESULT: {\"status\":\"success\"}\\n'\n", )]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", "45", "--agent", "claude", "--mode", "auto", "--until", "ship", ]) @@ -653,6 +679,7 @@ fn start_until_ship_is_rejected() { /// `start` is ever dispatched. #[test] fn start_until_unknown_stage_is_rejected_by_clap() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -661,7 +688,7 @@ fn start_until_unknown_stage_is_rejected_by_clap() { "#!/bin/sh\nprintf 'DEVFLOW_RESULT: {\"status\":\"success\"}\\n'\n", )]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", "46", "--agent", "claude", "--mode", "auto", "--until", "bogus", ]) @@ -688,12 +715,13 @@ fn start_until_unknown_stage_is_rejected_by_clap() { /// `--until` had not been passed. #[test] fn start_dry_run_annotates_until_stage() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", @@ -738,12 +766,13 @@ fn start_dry_run_annotates_until_stage() { /// annotation — the full pipeline runs to Ship. #[test] fn start_dry_run_without_until_has_no_stop_annotation() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", @@ -770,6 +799,7 @@ fn start_dry_run_without_until_has_no_stop_annotation() { #[test] fn status_prints_cron_hint_when_cron_instructions_exist() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -781,7 +811,7 @@ fn status_prints_cron_hint_when_cron_instructions_exist() { devflow_core::ship::write_cron_instructions(root, &instructions).unwrap(); let fake_bin = fake_bin_dir(&[]); - let output = run_devflow(root, &fake_bin.path, &["status"]); + let output = run_devflow(&home, root, &fake_bin.path, &["status"]); let stdout = String::from_utf8_lossy(&output.stdout); // D-10 (44-03) replaced the old `--from-devflow ` invocation with a @@ -804,6 +834,7 @@ fn status_prints_cron_hint_when_cron_instructions_exist() { #[test] fn reference_and_cleanup_worktree_cli_flow() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -835,13 +866,14 @@ esac )]); // reference — creates static snapshot - let out = run_devflow(root, &fake_bin.path, &["reference"]); + let out = run_devflow(&home, root, &fake_bin.path, &["reference"]); let stdout = String::from_utf8_lossy(&out.stdout); assert!(stdout.contains("reference worktree")); assert!(root.join(".worktrees/reference").is_dir()); // start --worktree — creates phase worktree let out = run_devflow( + &home, root, &fake_bin.path, &[ @@ -855,6 +887,7 @@ esac "--worktree", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(8), Vec::new()); let stdout = String::from_utf8_lossy(&out.stdout); assert!(stdout.contains("phase 8")); assert!( @@ -863,7 +896,7 @@ esac ); // status — lists active worktrees - let out = run_devflow(root, &fake_bin.path, &["status"]); + let out = run_devflow(&home, root, &fake_bin.path, &["status"]); let stdout = String::from_utf8_lossy(&out.stdout); assert!( stdout.contains(".worktrees/reference"), @@ -902,6 +935,7 @@ esac let _reap = MonitorReapGuard::after_launch(&settled); run_devflow( + &home, root, &fake_bin.path, &[ @@ -917,7 +951,7 @@ esac wait_for_state_cleared(root, PhaseId::new(8)); // cleanup — removes worktrees - let out = run_devflow(root, &fake_bin.path, &["cleanup", "--force"]); + let out = run_devflow(&home, root, &fake_bin.path, &["cleanup", "--force"]); let stdout = String::from_utf8_lossy(&out.stdout); assert!(stdout.contains("removed")); @@ -932,6 +966,7 @@ esac /// monitor is created. #[test] fn start_codex_without_context_fails_preflight() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -939,7 +974,7 @@ fn start_codex_without_context_fails_preflight() { // (14-CR-05) passes and the CONTEXT.md artifact check must fire next. let fake_bin = fake_bin_dir(&[("codex", "#!/bin/sh\nexit 0\n")]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", "42", "--agent", "codex", "--mode", "auto", ]) @@ -970,14 +1005,15 @@ fn start_codex_without_context_fails_preflight() { /// 20b instance 1 (D-06, review: Codex HIGH fail-closed-on-live-agent): /// `cleanup --force` must refuse to remove a worktree whose agent pid is -/// genuinely alive, even when the persisted `State` carries `monitor_pid = -/// None` — a classification `liveness()` reports as `Unknown`, NOT `Healthy`. +/// genuinely alive, even when the persisted `State` carries no supervisor +/// record — a run with no addressable monitor, NOT a healthy one. /// A guard that only refuses on `Healthy`/`BetweenStages` would still delete /// this worktree out from under a live agent. Against unmodified `cleanup` /// (no liveness check at all today) this test FAILS: cleanup removes the /// worktree unconditionally and exits 0. #[test] fn cleanup_force_refuses_on_live_agent_unknown_monitor() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -996,7 +1032,7 @@ fn cleanup_force_refuses_on_live_agent_unknown_monitor() { fs::create_dir_all(pid_path.parent().unwrap()).unwrap(); fs::write(&pid_path, std::process::id().to_string()).unwrap(); - // Persist a State with monitor_pid = None (Unknown liveness) and + // Persist a State with no supervisor record (no addressable monitor) and // worktree_path pointing at the created worktree (the worktree->phase // join key, review: Codex MEDIUM). let mut state = devflow_core::state::State::new( @@ -1009,7 +1045,7 @@ fn cleanup_force_refuses_on_live_agent_unknown_monitor() { devflow_core::workflow::save_state(&state).unwrap(); let fake_bin = fake_bin_dir(&[]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["cleanup", "--force"]) .arg(root) .env("PATH", path_with_fake_bin(&fake_bin.path)) @@ -1020,16 +1056,18 @@ fn cleanup_force_refuses_on_live_agent_unknown_monitor() { assert!( !output.status.success(), "cleanup --force must refuse to remove a live agent's worktree even \ - under Unknown liveness (monitor_pid = None)" + with no supervisor record" ); let combined = format!( "{}{}", String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); + // 50-14: `stop` ends a live run; `resume` refuses while one is live. assert!( - combined.contains("devflow resume"), - "refusal must name `devflow resume` as the unblocking action, got:\n{combined}" + combined.contains("run `devflow stop --phase 8`") && !combined.contains("devflow resume"), + "refusal must name `devflow stop --phase 8` (not `devflow resume`) as the \ + unblocking action, got:\n{combined}" ); assert!( wt_path.is_dir(), @@ -1044,6 +1082,7 @@ fn cleanup_force_refuses_on_live_agent_unknown_monitor() { /// classification alone. #[test] fn cleanup_force_refuses_on_dead_monitor_live_agent() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1061,8 +1100,8 @@ fn cleanup_force_refuses_on_dead_monitor_live_agent() { fs::create_dir_all(pid_path.parent().unwrap()).unwrap(); fs::write(&pid_path, std::process::id().to_string()).unwrap(); - // Persist a State with a DEAD monitor pid (`liveness()` classifies this - // Stuck, since monitor_alive is false), plus worktree_path. + // Persist a State whose supervisor record names a socket that does not + // exist (a dead monitor: the probe reads GONE), plus worktree_path. let mut state = devflow_core::state::State::new( phase, devflow_core::state::AgentKind::Claude, @@ -1070,11 +1109,17 @@ fn cleanup_force_refuses_on_dead_monitor_live_agent() { root.to_path_buf(), ); state.worktree_path = Some(wt_path.clone()); - state.monitor_pid = Some(0x7FFF_FFFE); // essentially never a live pid + state.supervisor = Some(devflow_core::state::SupervisorRecord::new( + root.join("dead-monitor-never-bound.sock"), + "deadmonitor00000".to_string(), + 0x7FFF_FFFE, // essentially never a live process group + 1, + String::new(), + )); devflow_core::workflow::save_state(&state).unwrap(); let fake_bin = fake_bin_dir(&[]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["cleanup", "--force"]) .arg(root) .env("PATH", path_with_fake_bin(&fake_bin.path)) @@ -1092,9 +1137,11 @@ fn cleanup_force_refuses_on_dead_monitor_live_agent() { String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr) ); + // 50-14: `stop` ends a live run; `resume` refuses while one is live. assert!( - combined.contains("devflow resume"), - "refusal must name `devflow resume` as the unblocking action, got:\n{combined}" + combined.contains("run `devflow stop --phase 9`") && !combined.contains("devflow resume"), + "refusal must name `devflow stop --phase 9` (not `devflow resume`) as the \ + unblocking action, got:\n{combined}" ); assert!( wt_path.is_dir(), @@ -1103,7 +1150,7 @@ fn cleanup_force_refuses_on_dead_monitor_live_agent() { } /// CR-02 (phase 20 review): a phase halted via `devflow start --until -/// ` clears `monitor_pid` and its agent has already exited by +/// ` clears the supervisor record and its agent has already exited by /// design — `Liveness::Unknown` with `agent_alive == false` sails straight /// through the live-agent refusal, so an ordinary `devflow cleanup` (no /// `--force`) must not delete the worktree of a phase the operator parked @@ -1112,6 +1159,7 @@ fn cleanup_force_refuses_on_dead_monitor_live_agent() { /// same phase; `cleanup` must recognize `state.stopped` too. #[test] fn cleanup_keeps_worktree_for_until_stopped_phase_without_force() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1125,7 +1173,7 @@ fn cleanup_keeps_worktree_for_until_stopped_phase_without_force() { devflow_core::worktree::add(root, &wt_path, &branch, &branch, false).unwrap(); // No agent pid file (the stage's agent has already exited normally) and - // monitor_pid = None (cleared by the --until stop path) — Unknown + // no supervisor record (cleared by the --until stop path) — Unknown // liveness, agent_alive == false. Only `state.stopped` distinguishes // this from a genuinely dead, safe-to-remove phase. let mut state = devflow_core::state::State::new( @@ -1140,7 +1188,7 @@ fn cleanup_keeps_worktree_for_until_stopped_phase_without_force() { devflow_core::workflow::save_state(&state).unwrap(); let fake_bin = fake_bin_dir(&[]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["cleanup"]) .arg(root) .env("PATH", path_with_fake_bin(&fake_bin.path)) @@ -1173,6 +1221,7 @@ fn cleanup_keeps_worktree_for_until_stopped_phase_without_force() { /// still be able to discard a stopped phase's worktree. #[test] fn cleanup_force_removes_worktree_for_until_stopped_phase() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1197,7 +1246,7 @@ fn cleanup_force_removes_worktree_for_until_stopped_phase() { devflow_core::workflow::save_state(&state).unwrap(); let fake_bin = fake_bin_dir(&[]); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["cleanup", "--force"]) .arg(root) .env("PATH", path_with_fake_bin(&fake_bin.path)) @@ -1220,6 +1269,7 @@ fn cleanup_force_removes_worktree_for_until_stopped_phase() { /// the second run finds the worktree already gone and does not error. #[test] fn cleanup_is_idempotent_when_worktree_already_removed() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1232,8 +1282,9 @@ fn cleanup_is_idempotent_when_worktree_already_removed() { .join(format!("phase-{padded}", padded = phase.padded())); devflow_core::worktree::add(root, &wt_path, &branch, &branch, false).unwrap(); - // Dead agent, dead monitor (Stuck liveness) — a genuinely dead phase, - // safe for cleanup to proceed. + // Dead agent, dead monitor (a supervisor record whose socket does not + // exist, so the probe reads GONE) — a genuinely dead phase, safe for + // cleanup to proceed. let mut state = devflow_core::state::State::new( phase, devflow_core::state::AgentKind::Claude, @@ -1241,11 +1292,17 @@ fn cleanup_is_idempotent_when_worktree_already_removed() { root.to_path_buf(), ); state.worktree_path = Some(wt_path.clone()); - state.monitor_pid = Some(0x7FFF_FFFE); + state.supervisor = Some(devflow_core::state::SupervisorRecord::new( + root.join("dead-monitor-never-bound.sock"), + "deadmonitor00000".to_string(), + 0x7FFF_FFFE, + 1, + String::new(), + )); devflow_core::workflow::save_state(&state).unwrap(); let fake_bin = fake_bin_dir(&[]); - let first = Command::new(devflow_bin()) + let first = common::hermetic_devflow(&home) .args(["cleanup", "--force"]) .arg(root) .env("PATH", path_with_fake_bin(&fake_bin.path)) @@ -1259,7 +1316,7 @@ fn cleanup_is_idempotent_when_worktree_already_removed() { ); assert!(!wt_path.is_dir(), "worktree must be removed on first run"); - let second = Command::new(devflow_bin()) + let second = common::hermetic_devflow(&home) .args(["cleanup", "--force"]) .arg(root) .env("PATH", path_with_fake_bin(&fake_bin.path)) @@ -1296,7 +1353,7 @@ fn pi_stub(launch: &str) -> String { /// Wait until a phase's chain settles in a state the tests can reap from: /// either the never-silent gate fired (`gate_pending`) or the machine /// stopped (`--until` / completion). Returns that state — whose -/// `monitor_pid` is the LAST monitor the chain spawned, the one a +/// `supervisor` record names the LAST monitor the chain spawned, the one a /// `MonitorReapGuard` must capture (a guard bound to an early state read /// reaps an already-exited pid and leaks the gate-waiting monitor, HYG-01). fn wait_for_settled(root: &Path, phase: PhaseId) -> devflow_core::state::State { @@ -1330,12 +1387,14 @@ fn wait_for_gate(root: &Path, phase: PhaseId) -> devflow_core::state::State { /// `Failed` and gates instead of advancing to Code. #[test] fn pi_marker_less_run_does_not_advance() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("pi", &pi_stub("printf 'fake pi, no marker\\n'\nexit 0\n"))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1348,6 +1407,7 @@ fn pi_marker_less_run_does_not_advance() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_gate(root, PhaseId::new(7)); assert_eq!( @@ -1363,12 +1423,14 @@ fn pi_marker_less_run_does_not_advance() { /// not Success, so the pipeline gates instead of advancing. #[test] fn pi_nonzero_exit_does_not_advance() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("pi", &pi_stub("printf 'fake pi, no marker\\n'\nexit 1\n"))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1381,6 +1443,7 @@ fn pi_nonzero_exit_does_not_advance() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_gate(root, PhaseId::new(7)); assert_eq!( @@ -1397,12 +1460,14 @@ fn pi_nonzero_exit_does_not_advance() { /// the monitor reaps it and gates (never a silent advance). #[test] fn pi_hung_process_is_detected_not_left_running() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("pi", &pi_stub("exec sleep 30\n"))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1415,6 +1480,7 @@ fn pi_hung_process_is_detected_not_left_running() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let pid_path = root.join(".devflow/phase-07-agent-pid"); let pid = wait_for_pid(&pid_path); @@ -1524,26 +1590,39 @@ enum StubMode { /// of the binary crate's `ReapMonitorOnDrop`, built on the PUBLIC /// `devflow_core::agent` surface (integration tests cannot reach /// `devflow-cli`'s test_support). TERM->KILL escalation with VERIFIED death, -/// keyed to `state.monitor_pid`, bound strictly after the final +/// keyed to the pid of the supervisor `state.supervisor` records (read from +/// its socket by [`live_supervisor_pid`]), bound strictly after the final /// `&mut State` use. 41-02 Task 1 turns this into the systematic pass. struct MonitorReapGuard { pid: Option, } +/// The pid of the supervisor `state` records, from its socket's ping reply. +/// The state carries no monitor pid (50-22); before that field went, it held +/// this same pid. `None` when the state has no record or its socket does not +/// answer as alive: nothing is left for a guard to reap. +fn live_supervisor_pid(state: &devflow_core::state::State) -> Option { + let record = state.supervisor.as_ref()?; + match devflow_core::supervisor::probe(&record.socket_path) { + Ok(devflow_core::supervisor::Liveness::Alive(reply)) => Some(reply.supervisor_pid), + _ => None, + } +} + impl MonitorReapGuard { fn after_launch(state: &devflow_core::state::State) -> Self { // Register the pid THIS guard is responsible for (the settled-state // monitor it will reap), and mark a guard in flight for the audit's // ordering barrier. `state` must be the SETTLED state - // (`wait_for_settled` / `wait_for_gate`), whose `monitor_pid` is the - // chain's LAST monitor — the one that would leak if not reaped. - if let Some(pid) = state.monitor_pid { + // (`wait_for_settled` / `wait_for_gate`), whose supervisor record + // names the chain's LAST monitor — the one that would leak if not + // reaped. + let pid = live_supervisor_pid(state); + if let Some(pid) = pid { registry().lock().unwrap().insert(pid); ACTIVE_GUARDS.fetch_add(1, std::sync::atomic::Ordering::SeqCst); } - Self { - pid: state.monitor_pid, - } + Self { pid } } } @@ -1589,12 +1668,14 @@ impl Drop for MonitorReapGuard { /// no commits, so it gates instead of advancing to Code. #[test] fn marker_less_antigravity_never_advances() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("agy", &antigravity_stub(StubMode::Quiet))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1607,6 +1688,7 @@ fn marker_less_antigravity_never_advances() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_gate(root, PhaseId::new(7)); let _reap = MonitorReapGuard::after_launch(&state); @@ -1625,12 +1707,14 @@ fn marker_less_antigravity_never_advances() { /// stopped, NOT gated. #[test] fn antigravity_parses_devflow_result_from_stream() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("agy", &antigravity_stub(StubMode::MarkerStream))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1645,6 +1729,7 @@ fn antigravity_parses_devflow_result_from_stream() { "plan", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_stopped(root, PhaseId::new(7)); let _reap = MonitorReapGuard::after_launch(&state); @@ -1664,12 +1749,14 @@ fn antigravity_parses_devflow_result_from_stream() { /// advances a commit-gated stage, only the marker does (ANTG-03). #[test] fn antigravity_init_without_marker_gates_at_plan() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("agy", &antigravity_stub(StubMode::InitOnly))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1682,6 +1769,7 @@ fn antigravity_init_without_marker_gates_at_plan() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_gate(root, PhaseId::new(7)); let _reap = MonitorReapGuard::after_launch(&state); @@ -1704,8 +1792,17 @@ fn antigravity_init_without_marker_gates_at_plan() { /// or seen its monitor exit naturally), the registry must DRAIN. Polls /// bounded so a slow legitimate test does not flake; a genuinely leaked /// monitor keeps the registry non-empty and the audit panics naming the pid. +/// +/// The audit reads process-wide counters the other tests in this binary bump, +/// so it only means something when they share its process, as under +/// `cargo test` (what `scripts/check.sh` runs). cargo-nextest runs every test +/// in its own process and sets `NEXTEST`, where `REAPED` is always 0 and the +/// audit could only time out; there it returns without asserting. #[test] fn suite_reap_audit() { + if std::env::var_os("NEXTEST").is_some() { + return; + } let deadline = std::time::Instant::now() + Duration::from_secs(60); // Ordering barrier: wait for a SUSTAINED clean state, not a single empty // poll. A clean state means: no guard in flight (ACTIVE_GUARDS == 0), at @@ -1755,6 +1852,7 @@ fn suite_reap_audit() { /// normal guard. #[test] fn unguarded_monitor_is_detected_by_the_registry() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1763,6 +1861,7 @@ fn unguarded_monitor_is_detected_by_the_registry() { let fake_bin = fake_bin_dir(&[("pi", &pi_stub("exec sleep 30\n"))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1775,11 +1874,11 @@ fn unguarded_monitor_is_detected_by_the_registry() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = load_state(root, PhaseId::new(7)).expect("load state"); - let pid = state - .monitor_pid - .expect("a launched stage must record a monitor pid"); + let pid = live_supervisor_pid(&state) + .expect("a launched stage must record a supervisor that answers its socket"); assert!( devflow_core::agent::agent_running(pid), "premise: with no guard the monitor stays alive — nothing reaps it" @@ -1822,6 +1921,7 @@ fi /// commit-gated stage. Define advances on exit 0; Plan gates. #[test] fn hermes_marker_less_run_does_not_advance() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1831,6 +1931,7 @@ fn hermes_marker_less_run_does_not_advance() { )]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1843,6 +1944,7 @@ fn hermes_marker_less_run_does_not_advance() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_gate(root, PhaseId::new(7)); assert_eq!( @@ -1857,6 +1959,7 @@ fn hermes_marker_less_run_does_not_advance() { /// HRMS-03: a Hermes run that exits non-zero must not advance its stage. #[test] fn hermes_nonzero_exit_does_not_advance() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); @@ -1866,6 +1969,7 @@ fn hermes_nonzero_exit_does_not_advance() { )]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1878,6 +1982,7 @@ fn hermes_nonzero_exit_does_not_advance() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let state = wait_for_gate(root, PhaseId::new(7)); assert_eq!( @@ -1893,12 +1998,14 @@ fn hermes_nonzero_exit_does_not_advance() { /// does not falsely advance, and gates when killed. #[test] fn hermes_hung_process_is_detected_not_left_running() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo(root); let fake_bin = fake_bin_dir(&[("hermes", &hermes_stub("exec sleep 30\n"))]); run_devflow( + &home, root, &fake_bin.path, &[ @@ -1911,6 +2018,7 @@ fn hermes_hung_process_is_detected_not_left_running() { "supervise", ], ); + let _reaper = common::LaunchedRunReaper::new(root, PhaseId::new(7), Vec::new()); let pid_path = root.join(".devflow/phase-07-agent-pid"); let pid = wait_for_pid(&pid_path); diff --git a/crates/devflow-cli/tests/reap_strays_e2e.rs b/crates/devflow-cli/tests/reap_strays_e2e.rs index f2ac81de..286fbeb5 100644 --- a/crates/devflow-cli/tests/reap_strays_e2e.rs +++ b/crates/devflow-cli/tests/reap_strays_e2e.rs @@ -43,16 +43,17 @@ //! `gate_sweep_e2e.rs` document — the one CLI-level assertion this file //! does make (`devflow stop` reporting the deleted root as unreachable) is //! driven via the real compiled binary, exactly like those files. +//! +//! That one `devflow` child is built by `common::hermetic_devflow`, so it +//! never reads or writes the operator's real `~/.cache/devflow` (999.141). + +mod common; use devflow_core::agent; use std::path::Path; use std::process::Command; use std::time::{Duration, Instant}; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - /// Poll `predicate` until it's true, panicking with `what` if `timeout_secs` /// elapses first — mirrors `stop_e2e.rs`'s `wait_for`. fn wait_for(mut predicate: impl FnMut() -> bool, timeout_secs: u64, what: &str) { @@ -125,6 +126,7 @@ fn spawn_monitor_wrapper_fixture(cwd: &Path) -> std::process::Child { /// discovery) and clear it with verified death. #[test] fn reap_clears_a_process_whose_root_was_deleted_which_devflow_stop_cannot_see() { + let home = common::HermeticHome::new(); let (outer, root) = nested_root_fixture(); let mut child = spawn_monitor_wrapper_fixture(&root); let pid = child.id(); @@ -159,7 +161,7 @@ fn reap_clears_a_process_whose_root_was_deleted_which_devflow_stop_cannot_see() // deleted path, so it reports the process as unreachable -- neither // clearing it nor pretending the machine is otherwise clean by // signalling something else. - let stop_output = Command::new(devflow_bin()) + let stop_output = common::hermetic_devflow(&home) .args(["stop", "--phase", "900", "--root"]) .arg(&root) .output() diff --git a/crates/devflow-cli/tests/recover_clean_e2e.rs b/crates/devflow-cli/tests/recover_clean_e2e.rs index 257e9b39..38c06ef1 100644 --- a/crates/devflow-cli/tests/recover_clean_e2e.rs +++ b/crates/devflow-cli/tests/recover_clean_e2e.rs @@ -3,20 +3,18 @@ //! exits non-zero, and a sweep that cleared nothing does not claim a cleanup. //! Each refusal test has a control that must still clean and say so. +mod common; + use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; use devflow_core::state::{AgentKind, State}; use std::path::Path; -use std::process::{Command, Output}; +use std::process::Output; use std::time::{SystemTime, UNIX_EPOCH}; /// A PID that is essentially certain not to map to a live process. const DEAD_PID: u32 = 0x7FFF_FFFE; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - fn init_repo(root: &Path) { let output = devflow_core::test_support::git_command(root) .args(["init", "-q"]) @@ -45,7 +43,8 @@ fn save_state_aged(root: &Path, phase: PhaseId, age_secs: u64) { } fn recover_clean(root: &Path, phase: Option) -> Output { - let mut command = Command::new(devflow_bin()); + let home = common::HermeticHome::new(); + let mut command = common::hermetic_devflow(&home); command.args(["recover", "--clean"]); if let Some(phase) = phase { command.args(["--phase", &phase.to_string()]); @@ -114,6 +113,54 @@ fn explicit_clean_without_a_lock_holder_cleans_and_says_so() { assert!(!devflow_core::workflow::state_path(root, phase).exists()); } +/// 999.128 (D-08), R-13: the repair `stop`/`doctor` name at a recycled-pid +/// gate must work. A lock record naming a live pid whose start time differs +/// from the recorded one belongs to a holder that exited, so an explicit clean +/// reclaims it and cleans the phase. +/// `explicit_clean_with_a_held_phase_lock_fails_and_cleans_nothing` is the +/// live-holder control. +#[test] +fn explicit_clean_at_a_recycled_lock_holder_reclaims_and_cleans() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + init_repo(root); + let phase = PhaseId::new(65); + save_state_aged(root, phase, 60); + let gate = devflow_core::gates::Gates::write_gate( + root, + phase, + devflow_core::stage::Stage::Code, + "wedged at a recycled pid", + ) + .unwrap(); + // This test process is alive; recording a start time one tick off its + // real one makes the record name a process instance that no longer exists. + let pid = std::process::id(); + let start = devflow_core::agent::process_start_time(pid).expect("test pid start time"); + let lock_path = root + .join(".devflow") + .join(format!("lock-{}", phase.padded())); + std::fs::write(&lock_path, format!("{pid}\n{}", start.saturating_add(1))).unwrap(); + assert_eq!( + devflow_core::lock::holder_status(root, phase), + devflow_core::lock::HolderStatus::Recycled { pid } + ); + + let output = recover_clean(root, Some(phase)); + + let text = combined(&output); + assert!( + output.status.success(), + "a clean at a recycled-pid lock must succeed: {text}" + ); + assert!( + text.contains("cleaned up workflow state for phase 65"), + "the clean must say what it cleaned: {text}" + ); + assert!(!devflow_core::workflow::state_path(root, phase).exists()); + assert!(!gate.exists(), "the phase's gate file must be removed"); +} + /// Fix-review finding C-3: `clean_phase` is idempotent, so an explicit clean /// of a phase with nothing on disk succeeded and printed "cleaned up". /// `explicit_clean_without_a_lock_holder_cleans_and_says_so` is the control. diff --git a/crates/devflow-cli/tests/release_check.rs b/crates/devflow-cli/tests/release_check.rs index 900a6a8c..a48ab921 100644 --- a/crates/devflow-cli/tests/release_check.rs +++ b/crates/devflow-cli/tests/release_check.rs @@ -4,27 +4,25 @@ //! `pub(crate)` inside `devflow-cli`, and driving the actual CLI is what //! proves the `--check` gate and the self-pin comparison end-to-end). -use std::path::Path; -use std::process::{Command, Output}; +mod common; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} +use std::path::Path; +use std::process::Output; /// Runs `devflow release ` with an ISOLATED `HOME` (a fresh -/// empty directory, no `.gitconfig`) and no inherited `SSH_AUTH_SOCK`/ +/// empty directory, no `.gitconfig`, set by `common::hermetic_devflow` along +/// with the XDG and DevFlow cache dirs) and no inherited `SSH_AUTH_SOCK`/ /// `SSH_AGENT_PID` — so the release check reads git config through the /// fixture's own repo-local config, never the OPERATOR's global /// `~/.gitconfig`. Without this isolation, these tests would be /// non-deterministic on any machine whose global config sets git signing /// or other release-relevant values. fn run_release(project: &Path, args: &[&str]) -> Output { - let isolated_home = tempfile::tempdir().unwrap(); - Command::new(devflow_bin()) + let isolated_home = common::HermeticHome::new(); + common::hermetic_devflow(&isolated_home) .arg("release") .args(args) .arg(project) - .env("HOME", isolated_home.path()) .env_remove("SSH_AUTH_SOCK") .env_remove("SSH_AGENT_PID") .output() diff --git a/crates/devflow-cli/tests/snapshots/devflow-help.txt b/crates/devflow-cli/tests/snapshots/devflow-help.txt index ac8e35cc..8a07484c 100644 --- a/crates/devflow-cli/tests/snapshots/devflow-help.txt +++ b/crates/devflow-cli/tests/snapshots/devflow-help.txt @@ -18,7 +18,7 @@ Commands: doctor Audit the environment and report what's installed, missing, or broken release Read-only release-cut preflight: self-pin, develop/main divergence, crates.io publish order, and changelog-version agreement ship Manually drive a phase through Ship when the monitor that would have consumed its already-written Ship gate response is dead - stop End a running phase cleanly (23c): answers its open gate with a rejection if one is open — the target unwinds through its own abort path, no signal sent — otherwise signals the process recorded in its per-phase lock file (`.devflow/lock-{phase:02}`), never `state.monitor_pid` (the PID `devflow status` displays, and the wrong one — see `commands::stop`'s doc comment). Idempotent: safe to run against an already-stopped, never-started, or already-dead phase + stop End a running phase cleanly: answers its open gate with a rejection when a live waiter holds it; otherwise addresses the run's supervisor socket — `shutdown` when ALIVE, the guarded backstop when STALE, the per-phase lock holder when GONE. `--launch ` tears down only the socket with that launch id (the form `devflow doctor` prints for an orphan). Never signals a pid recorded in state. Idempotent: safe to run against an already-stopped, never-started, or already-dead phase evidence Report DevFlow's own structural record of whether a phase shipped (23-06) — a read-only oracle sourced from the append-only event log, not from any agent-authored attestation document. See `devflow_core::ship_evidence` for the full contract help Print this message or the help of the given subcommand(s) diff --git a/crates/devflow-cli/tests/start_lock_e2e.rs b/crates/devflow-cli/tests/start_lock_e2e.rs index 87b50be0..ae94a066 100644 --- a/crates/devflow-cli/tests/start_lock_e2e.rs +++ b/crates/devflow-cli/tests/start_lock_e2e.rs @@ -4,21 +4,23 @@ //! the compiled CLI as a separate process. That makes the refusal observable //! at the same boundary an operator uses, rather than coupling the test to //! `commands::start`'s implementation details. +//! +//! Every `devflow` child runs through `common::hermetic_devflow`, so its +//! registry entries land in a tempdir `HOME`, never the operator's real +//! `~/.cache/devflow` (999.141). + +mod common; use devflow_core::gates::{GateResponse, Gates}; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; use devflow_core::stage::Stage; use devflow_core::state::{AgentKind, State}; +use devflow_core::supervisor::{self, Liveness, ShutdownOutcome}; use std::fs; -use std::os::unix::fs::{PermissionsExt, symlink}; use std::path::{Path, PathBuf}; use std::process::{Child, Command, Output}; -use std::time::{Duration, Instant}; - -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} +use std::time::Duration; fn git(root: &Path, args: &[&str]) { let output = devflow_core::test_support::git_command(root) @@ -50,31 +52,16 @@ fn init_repo(root: &Path, phase: PhaseId) { git(root, &["commit", "-q", "-m", "base"]); } -struct FakeBin { - _dir: tempfile::TempDir, - path: PathBuf, -} - -fn fake_bin_dir(agent_body: &str) -> FakeBin { - let dir = tempfile::tempdir().unwrap(); - let claude = dir.path().join("claude"); - fs::write(&claude, format!("#!/bin/sh\n{agent_body}\n")).unwrap(); - let mut permissions = fs::metadata(&claude).unwrap().permissions(); - permissions.set_mode(0o755); - fs::set_permissions(&claude, permissions).unwrap(); - for program in ["git", "sh"] { - symlink(format!("/usr/bin/{program}"), dir.path().join(program)).unwrap(); - } - FakeBin { - path: dir.path().to_path_buf(), - _dir: dir, - } -} - -fn child(root: &Path, phase: PhaseId, fake_bin: &FakeBin, mode: &str) -> Command { +fn child( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &common::FakeBin, + mode: &str, +) -> Command { let existing = std::env::var_os("PATH").unwrap_or_default(); let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); - let mut command = Command::new(devflow_bin()); + let mut command = common::hermetic_devflow(home); command .args([ "start", @@ -93,31 +80,14 @@ fn child(root: &Path, phase: PhaseId, fake_bin: &FakeBin, mode: &str) -> Command command } -fn wait_for(mut predicate: impl FnMut() -> bool, timeout: Duration, what: &str) { - let started = Instant::now(); - while !predicate() { - assert!( - started.elapsed() < timeout, - "timed out after {timeout:?} waiting for {what}" - ); - std::thread::sleep(Duration::from_millis(25)); - } -} - -fn kill_and_reap(child: &mut Child) { - if child.try_wait().expect("poll start child").is_none() { - let _ = child.kill(); - } - let _ = child.wait(); -} - #[test] fn start_refuses_while_another_process_holds_the_phase_lock() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(90); init_repo(root, phase); - let fake_bin = fake_bin_dir("exit 0"); + let fake_bin = common::fake_bin_dir("exit 0"); let holder = devflow_core::lock::acquire(root, phase).expect("test holds phase lock"); let holder_pid = std::process::id(); let worktrees_before = devflow_core::test_support::git_command(root) @@ -126,7 +96,7 @@ fn start_refuses_while_another_process_holds_the_phase_lock() { .expect("list baseline worktrees"); assert!(worktrees_before.status.success()); - let output = child(root, phase, &fake_bin, "supervise") + let output = child(&home, root, phase, &fake_bin, "supervise") .output() .expect("run devflow start"); let stderr = String::from_utf8_lossy(&output.stderr); @@ -172,13 +142,15 @@ fn start_refuses_while_another_process_holds_the_phase_lock() { #[test] fn start_proceeds_when_no_process_holds_the_phase_lock() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(91); init_repo(root, phase); - let fake_bin = fake_bin_dir("exit 0"); + let fake_bin = common::fake_bin_dir("exit 0"); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); - let output = child(root, phase, &fake_bin, "supervise") + let output = child(&home, root, phase, &fake_bin, "supervise") .output() .expect("run devflow start"); assert!( @@ -200,11 +172,12 @@ fn start_proceeds_when_no_process_holds_the_phase_lock() { #[test] fn start_clears_leftover_gate_files_after_taking_the_lock() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(92); init_repo(root, phase); - let fake_bin = fake_bin_dir("sleep 60"); + let fake_bin = common::fake_bin_dir("sleep 60"); let response_path = Gates::response_path(root, phase, Stage::Define); fs::create_dir_all(response_path.parent().unwrap()).unwrap(); fs::write( @@ -217,12 +190,13 @@ fn start_clears_leftover_gate_files_after_taking_the_lock() { .unwrap(), ) .unwrap(); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); - let mut process = child(root, phase, &fake_bin, "auto") + let mut process = child(&home, root, phase, &fake_bin, "auto") .env("DEVFLOW_GATE_TIMEOUT_SECS", "30") .spawn() .expect("spawn devflow start"); - wait_for( + common::wait_for( || { Gates::gate_path(root, phase, Stage::Define).exists() || process.try_wait().expect("poll start child").is_some() @@ -249,7 +223,7 @@ fn start_clears_leftover_gate_files_after_taking_the_lock() { !response_path.exists(), "the planted response must be removed" ); - kill_and_reap(&mut process); + common::kill_and_reap(&mut process); } // --------------------------------------------------------------------------- @@ -299,25 +273,130 @@ impl Drop for LiveProcess { } } +/// A fake `claude` that answers `--version` and otherwise sleeps until it is +/// ended, so a launched run stays live for the whole test. +const SLEEPING_CLAUDE: &str = "if [ \"$1\" = \"--version\" ]; then echo \"2.1.220 (Claude Code)\"; exit 0; fi\nexec sleep 120"; + +/// How long [`LiveRun`]'s drop waits for its supervisor's final shutdown line. +const LIVE_RUN_SHUTDOWN_WAIT: Duration = Duration::from_secs(15); + +/// A run a real `devflow start` launched (capture shape, supervise mode) with +/// a sleeping fake agent: its supervisor answers ALIVE on `socket`, and +/// `sidecar` is what it published, read before any test corrupts it. +/// +/// Dropping it ends the run through that socket, then SIGKILLs the agent +/// group and the supervisor if each is still the process the sidecar names +/// (`agent::is_same_process`), so a run whose state or sidecar a test +/// corrupted is still reaped. Bind a `LaunchedRunReaper` before +/// [`LiveRun::launch`] as the backstop for a panic inside it. +struct LiveRun { + socket: PathBuf, + sidecar: supervisor::Sidecar, + _fake_bin: common::FakeBin, +} + +impl LiveRun { + fn launch(home: &common::HermeticHome, root: &Path, phase: PhaseId) -> Self { + let fake_bin = common::fake_bin_dir(SLEEPING_CLAUDE); + let output = child(home, root, phase, &fake_bin, "supervise") + .env("DEVFLOW_GATE_TIMEOUT_SECS", GATE_TIMEOUT_SECS) + .output() + .expect("run devflow start for the live-run fixture"); + assert!( + output.status.success(), + "the live-run fixture's start failed\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + let record = devflow_core::workflow::load_state(root, phase) + .expect("the fixture's start saved its state") + .supervisor + .expect("the fixture's start recorded state.supervisor"); + let sidecar = supervisor::read_sidecar(&record.socket_path.with_extension("json")) + .expect("read the fixture's sidecar"); + let run = Self { + socket: record.socket_path, + sidecar, + _fake_bin: fake_bin, + }; + assert_eq!(run.sidecar.launch_id, record.launch_id, "sidecar launch id"); + assert!( + matches!(supervisor::probe(&run.socket), Ok(Liveness::Alive(_))), + "the fixture's supervisor must answer ALIVE" + ); + let pid_file = devflow_core::agent_result::agent_pid_path(root, phase); + common::wait_for( + || pid_file.exists(), + Duration::from_secs(10), + "the fixture's agent pid file", + ); + run + } + + fn launch_id(&self) -> &str { + &self.sidecar.launch_id + } + + fn sidecar_path(&self) -> PathBuf { + self.socket.with_extension("json") + } + + /// Remove the agent pid file, so the guard's pid half sees nothing and + /// only the supervisor socket can name this run: 999.139's shape, where + /// the pid half let `start --force` launch a second agent. + fn hide_agent_pid_file(&self, root: &Path, phase: PhaseId) { + fs::remove_file(devflow_core::agent_result::agent_pid_path(root, phase)) + .expect("remove the fixture's agent pid file"); + } +} + +impl Drop for LiveRun { + fn drop(&mut self) { + // No `expect` here: a panic inside drop during unwinding aborts. + let _ = supervisor::request_shutdown(&self.socket, LIVE_RUN_SHUTDOWN_WAIT); + let group = (self.sidecar.agent_pgid, self.sidecar.agent_start_time); + if devflow_core::agent::agent_running(group.0) + && devflow_core::agent::is_same_process(group.0, group.1) + { + common::signal_group(group.0, libc::SIGKILL); + } + let monitor = ( + self.sidecar.supervisor_pid, + self.sidecar.supervisor_start_time, + ); + if devflow_core::agent::agent_running(monitor.0) + && devflow_core::agent::is_same_process(monitor.0, monitor.1) + && let Ok(pid) = libc::pid_t::try_from(monitor.0) + { + // SAFETY: one process, identity-checked against its start time. + unsafe { libc::kill(pid, libc::SIGKILL) }; + } + } +} + /// The planted Code gate files, returned so a control can assert they are gone. struct PlantedGate { request: PathBuf, response: PathBuf, } -fn plant_leftover_run( - root: &Path, - phase: PhaseId, - monitor_pid: Option, - agent_pid: Option, -) -> PlantedGate { +/// A leftover Code-stage state with no supervisor record, an optional agent +/// pid file, and a planted Code gate. The launch guard reads no +/// `monitor_pid` (50-15), so none is planted; a test that needs a live +/// supervisor launches one with [`LiveRun::launch`]. +fn plant_leftover_run(root: &Path, phase: PhaseId, agent_pid: Option) -> PlantedGate { let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); state.stage = Stage::Code; - state.monitor_pid = monitor_pid; devflow_core::workflow::save_state(&state).expect("plant leftover state"); if let Some(pid) = agent_pid { write_agent_pid(root, phase, pid); } + plant_code_gate(root, phase) +} + +/// A Code gate request and an aborting response, as a leftover run leaves +/// them. +fn plant_code_gate(root: &Path, phase: PhaseId) -> PlantedGate { let request = Gates::write_gate(root, phase, Stage::Code, "planted leftover Code gate") .expect("plant Code gate request"); let response = Gates::response_path(root, phase, Stage::Code); @@ -374,8 +453,14 @@ fn snapshot_paths(snapshot: &[(PathBuf, Vec)]) -> Vec<&Path> { snapshot.iter().map(|(path, _)| path.as_path()).collect() } -fn run_start(root: &Path, phase: PhaseId, fake_bin: &FakeBin, extra: &[&str]) -> Output { - child(root, phase, fake_bin, "supervise") +fn run_start( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &common::FakeBin, + extra: &[&str], +) -> Output { + child(home, root, phase, fake_bin, "supervise") .args(extra) .env("DEVFLOW_GATE_TIMEOUT_SECS", GATE_TIMEOUT_SECS) .output() @@ -389,17 +474,78 @@ fn events_contain_workflow_started(root: &Path) -> bool { .contains("\"workflow_started\"") } -/// Drives `start` against a planted live run and asserts the refusal plus -/// the absence of every side effect. `refusal` is the first assertion's -/// message and names the arm. +/// Sentences the live-run refusals carried for the deleted shell monitor, or +/// that 50-14's `stop` made false. Each must be gone, not merely joined by +/// new text, from every arm of both the start and the resume refusal. +fn removed_refusal_sentences(phase: PhaseId) -> Vec { + vec![ + "would only mark the state stopped".to_string(), + "it signals nothing and does not end this run".to_string(), + "monitor pid".to_string(), + "an `sh -c` script".to_string(), + format!("`.devflow/phase-{}-`", phase.padded()), + "send SIGTERM to the confirmed monitor first".to_string(), + "signalling the agent first lets the monitor launch the next stage".to_string(), + "may be running `devflow advance` as a foreground child".to_string(), + "`ps -ww -o pid=,args= --ppid `".to_string(), + "once it takes the lock after this refusal".to_string(), + ] +} + +/// The fragments a refusal must carry for a live supervisor socket that +/// `stop --launch` can address: its launch id and liveness, its socket, and +/// the exact repair line (50-14's `--launch` form works whether or not the +/// state loads). +fn supervisor_arm_fragments(phase: PhaseId, run: &LiveRun) -> Vec { + let id = run.launch_id(); + vec![ + format!("supervisor launch {id} ALIVE"), + format!("socket {}", run.socket.display()), + format!("run `devflow stop --phase {phase} --launch {id}` first"), + "`devflow stop` ends a supervisor through its socket, whether or not the state file loads" + .to_string(), + "on a STALE socket it signals the agent group only after proving that group is this launch's" + .to_string(), + ] +} + +/// The fragments a refusal must carry for a live agent its pid file names. +/// Every sentence that keeps the operator from signalling the wrong process +/// or launching a second agent is pinned, not only the repair commands +/// (T-48-19-06, T-48-19-07). +fn agent_arm_fragments(phase: PhaseId, pid: u32) -> Vec { + vec![ + format!("agent pid {pid} alive"), + "`devflow stop` never signals a pid from the agent pid file".to_string(), + "Check each pid before signalling it".to_string(), + // Identity, not the bare executable name `ps -p` prints (48-REVIEW + // CR-01); `-ww` keeps a long `args` from being cut. + "`ps -p` shows only the executable name".to_string(), + format!("agent pid {pid}: ps -ww -o pid=,lstart=,args= -p {pid}"), + "`__monitor`".to_string(), + format!("`--phase {phase}`"), + "`ps -o ppid= -p `".to_string(), + "an agent whose supervisor has exited cannot be tied to this phase that way".to_string(), + "so do not signal it on the pid alone".to_string(), + "Do neither while this phase's processes are live".to_string(), + "a later `start` would launch a second agent beside it".to_string(), + ] +} + +/// Drives `start` against a live run and asserts the refusal plus the +/// absence of every side effect. `present` and `absent` are the arm's own +/// fragments; the verb's common fragments and every removed sentence are +/// checked here. `refusal` is the first assertion's message and names the +/// arm. fn assert_start_refuses_live_run( + home: &common::HermeticHome, root: &Path, phase: PhaseId, extra: &[&str], - live_pid: u32, + (present, absent): (&[String], &[String]), refusal: &str, ) { - let fake_bin = fake_bin_dir("exit 0"); + let fake_bin = common::fake_bin_dir("exit 0"); let before = phase_file_snapshot(root, phase); // Negative control on the snapshot itself: it must cover the planted // state and both planted gate files, or byte-identity would prove nothing. @@ -415,8 +561,10 @@ fn assert_start_refuses_live_run( snapshot_paths(&before) ); } + let branches_before = phase_branches(root, phase); + let started_before = workflow_started_count(root); - let output = run_start(root, phase, &fake_bin, extra); + let output = run_start(home, root, phase, &fake_bin, extra); let stdout = String::from_utf8_lossy(&output.stdout); let stderr = String::from_utf8_lossy(&output.stderr); assert!( @@ -424,57 +572,29 @@ fn assert_start_refuses_live_run( "{refusal}\nstdout: {stdout}\nstderr: {stderr}" ); - for fragment in [ + let common = [ phase.to_string(), - live_pid.to_string(), "refusing to start; nothing was written".to_string(), - format!("devflow stop --phase {phase}"), - "would only mark the state stopped".to_string(), - // stop is not a no-op once this run's own advance holds the lock - // (48-REVIEW WR-02). - "once it takes the lock after this refusal".to_string(), - // Identity, not the bare executable name `ps -p` prints (48-REVIEW - // CR-01). `-ww`: at a terminal `args` is cut at the screen width, - // which drops the Legacy script's phase paths. - format!("ps -ww -o pid=,lstart=,args= -p {live_pid}"), - "`ps -p` shows only the executable name".to_string(), - format!("`--phase {phase}`"), - "`__monitor`".to_string(), - format!("`.devflow/phase-{}-`", phase.padded()), - "`ps -o ppid= -p `".to_string(), - // A Legacy monitor in its advance tail defers SIGTERM behind a - // foreground `devflow advance` child (48-REVIEW WR-02). - "may be running `devflow advance` as a foreground child".to_string(), - "`ps -ww -o pid=,args= --ppid `".to_string(), - "signal that child, not only the monitor".to_string(), - // A live recycled pid refuses start again; only recover --clean - // then start gets past it (48-REVIEW WR-01). - "If the named processes have exited, run `devflow start` again".to_string(), - format!("run `devflow recover --clean --phase {phase}` first, then `devflow start`"), - // Every sentence that keeps the operator from signalling the wrong - // process or launching a second agent is pinned, not only the - // repair commands (T-48-19-06, T-48-19-07): a text that drops these - // but keeps the commands must fail. - "Check each pid before signalling it".to_string(), - "so do not signal it on the pid alone".to_string(), - "send SIGTERM to the confirmed monitor first".to_string(), - "signalling the agent first lets the monitor launch the next stage".to_string(), - "If a named pid is live but is not this phase's process (a recycled pid), `start` refuses again" + "Once every supervisor and process named above is gone, run `devflow start` again." .to_string(), - "Do neither while this phase's processes are live".to_string(), - "a later `start` would launch a second agent beside it".to_string(), - ] { + ]; + for fragment in common.iter().chain(present) { assert!( - stderr.contains(&fragment), + stderr.contains(fragment.as_str()), "refusal must contain {fragment:?}\nstderr: {stderr}" ); } // The superseded guidance must be gone, not merely joined by new text. - for stale in [ - format!("ps -p {live_pid}"), - "or after `ps -p` shows they are no longer this phase's processes".to_string(), - "(stop does end a run that holds the lock, for example one parked at a gate.)".to_string(), - ] { + for stale in removed_refusal_sentences(phase) + .into_iter() + .chain([ + "ps -p ".to_string(), + "or after `ps -p` shows they are no longer this phase's processes".to_string(), + "(stop does end a run that holds the lock, for example one parked at a gate.)" + .to_string(), + ]) + .chain(absent.iter().cloned()) + { assert!( !stderr.contains(&stale), "refusal must not contain superseded guidance {stale:?}\nstderr: {stderr}" @@ -495,6 +615,20 @@ fn assert_start_refuses_live_run( "a refused start must not leave {}", lock.display() ); + assert_eq!( + phase_branches(root, phase), + branches_before, + "a refused start must not create or change the phase branch" + ); + assert_eq!( + workflow_started_count(root), + started_before, + "a refused start must not emit workflow_started" + ); +} + +/// `git branch --list feature/phase-`'s output. +fn phase_branches(root: &Path, phase: PhaseId) -> String { let branches = devflow_core::test_support::git_command(root) .args([ "branch", @@ -504,63 +638,94 @@ fn assert_start_refuses_live_run( .output() .expect("list phase branch"); assert!(branches.status.success()); - assert!( - String::from_utf8_lossy(&branches.stdout).trim().is_empty(), - "a refused start must not create the phase branch: {}", - String::from_utf8_lossy(&branches.stdout) - ); - assert!( - !events_contain_workflow_started(root), - "a refused start must not emit workflow_started" - ); + String::from_utf8_lossy(&branches.stdout).into_owned() +} + +/// How many `workflow_started` events the project's event log holds. +fn workflow_started_count(root: &Path) -> usize { + fs::read_to_string(devflow_core::events::events_path(root)) + .unwrap_or_default() + .matches("\"workflow_started\"") + .count() } #[test] fn start_refuses_a_phase_whose_recorded_monitor_is_alive() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(95); init_repo(root, phase); - let monitor = LiveProcess::spawn(); - plant_leftover_run(root, phase, Some(monitor.pid()), None); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + // Only the supervisor can refuse: the agent pid file is gone. + run.hide_agent_pid_file(root, phase); + plant_code_gate(root, phase); + assert_eq!(workflow_started_count(root), 1, "the live run's own start"); assert_start_refuses_live_run( + &home, root, phase, &[], - monitor.pid(), - "start must refuse a phase whose recorded monitor is alive", + ( + &supervisor_arm_fragments(phase, &run), + &["agent pid".to_string()], + ), + "start must refuse a phase whose recorded supervisor is alive", + ); + assert!( + matches!(supervisor::probe(&run.socket), Ok(Liveness::Alive(_))), + "a refused start must leave the live supervisor running" ); } #[test] fn start_refuses_a_phase_whose_recorded_agent_is_alive() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(96); init_repo(root, phase); let agent = LiveProcess::spawn(); - plant_leftover_run(root, phase, None, Some(agent.pid())); + plant_leftover_run(root, phase, Some(agent.pid())); + let mut present = agent_arm_fragments(phase, agent.pid()); + present.extend([ + // A live recycled pid refuses start again; only recover --clean + // then start gets past it (48-REVIEW WR-01). + "If a named pid is live but is not this phase's process (a recycled pid), `start` refuses again" + .to_string(), + format!("run `devflow recover --clean --phase {phase}` first, then `devflow start`"), + ]); assert_start_refuses_live_run( + &home, root, phase, &["--force"], - agent.pid(), + ( + &present, + &[ + "supervisor launch".to_string(), + format!("devflow stop --phase {phase} --launch"), + ], + ), "start must refuse a phase whose recorded agent is alive", ); } #[test] fn start_replaces_a_leftover_state_whose_recorded_processes_are_dead() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(97); init_repo(root, phase); - let planted = plant_leftover_run(root, phase, Some(NEVER_LIVE_PID), Some(NEVER_LIVE_PID)); - let fake_bin = fake_bin_dir("exit 0"); + let planted = plant_leftover_run(root, phase, Some(NEVER_LIVE_PID)); + let fake_bin = common::fake_bin_dir("exit 0"); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); - let output = run_start(root, phase, &fake_bin, &[]); + let output = run_start(&home, root, phase, &fake_bin, &[]); assert!( output.status.success(), "a leftover state whose recorded processes are dead must be replaced as today\n\ @@ -586,6 +751,7 @@ fn start_replaces_a_leftover_state_whose_recorded_processes_are_dead() { #[test] fn start_proceeds_when_no_state_exists_even_if_the_agent_pid_file_names_a_live_process() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(98); @@ -596,9 +762,10 @@ fn start_proceeds_when_no_state_exists_even_if_the_agent_pid_file_names_a_live_p !devflow_core::workflow::state_path(root, phase).exists(), "the carve-out fixture must have no phase state" ); - let fake_bin = fake_bin_dir("exit 0"); + let fake_bin = common::fake_bin_dir("exit 0"); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![agent.pid()]); - let output = run_start(root, phase, &fake_bin, &[]); + let output = run_start(&home, root, phase, &fake_bin, &[]); assert!( output.status.success(), "without phase state, start must proceed even if the agent pid file names a live process\n\ @@ -616,13 +783,30 @@ fn start_proceeds_when_no_state_exists_even_if_the_agent_pid_file_names_a_live_p // 48-20 (criterion 2, D-01, D-02): `resume` must not overwrite a live run. // --------------------------------------------------------------------------- -const START_REFUSAL_GOLDEN_PHASE_94: &str = "phase 94: a run is still live (monitor pid {pid} alive) — refusing to start; nothing was written\n\ - No process holds the phase lock right now, so `devflow stop --phase 94` would only mark the state stopped: it signals nothing and does not end this run. stop ends a run through the process that holds the lock: one parked at a gate, or this run's own `devflow advance` once it takes the lock after this refusal.\n\ - Check each pid before signalling it. `ps -p` shows only the executable name, which cannot tell phases apart; `-ww` keeps a long command line from being cut at the terminal width:\n\ - \x20\x20monitor pid {pid}: ps -ww -o pid=,lstart=,args= -p {pid}\n\ - This phase's monitor is either a `__monitor` process with this project's path and `--phase 94` in its args, or an `sh -c` script that names `.devflow/phase-94-` files and ends in `advance` with `--phase 94`. This phase's agent has that monitor as its parent (`ps -o ppid= -p `); an agent whose monitor has exited cannot be tied to this phase that way, so do not signal it on the pid alone.\n\ - To end this run now, send SIGTERM to the confirmed monitor first — signalling the agent first lets the monitor launch the next stage. If the agent has already exited, an `sh` monitor may be running `devflow advance` as a foreground child: the monitor defers SIGTERM until that child returns, and the child may take the lock as soon as this start exits. Find it with `ps -ww -o pid=,args= --ppid ` and signal that child, not only the monitor.\n\ - If the named processes have exited, run `devflow start` again. If a named pid is live but is not this phase's process (a recycled pid), `start` refuses again: run `devflow recover --clean --phase 94` first, then `devflow start`. Do neither while this phase's processes are live: `recover --clean` clears state even while an agent runs, and a later `start` would launch a second agent beside it."; +/// The whole start refusal for a live run at phase 94 whose state loads: its +/// supervisor answers ALIVE and its agent pid file names its live agent. +/// `{launch}`, `{socket}` and `{pid}` are that run's. +const START_REFUSAL_GOLDEN_PHASE_94: &str = concat!( + "phase 94: a run is still live (supervisor launch {launch} ALIVE, agent pid {pid} alive) — ", + "refusing to start; nothing was written\n", + "`devflow stop` ends a supervisor through its socket, whether or not the state file loads; ", + "on a STALE socket it signals the agent group only after proving that group is this ", + "launch's. Run each line below first:\n", + " launch {launch} (ALIVE, socket {socket}): run `devflow stop --phase 94 --launch {launch}` first\n", + "The agent pid file names a live process, and `devflow stop` never signals a pid from the ", + "agent pid file. Check each pid before signalling it. `ps -p` shows only the executable name, ", + "which cannot tell phases apart; `-ww` keeps a long command line from being cut at the ", + "terminal width:\n", + " agent pid {pid}: ps -ww -o pid=,lstart=,args= -p {pid}\n", + "This phase's agent has its supervisor, a `__monitor` process with this project's path and ", + "`--phase 94` in its args, as its parent (`ps -o ppid= -p `); an agent whose supervisor ", + "has exited cannot be tied to this phase that way, so do not signal it on the pid alone. ", + "If a named pid is live but is not this phase's process (a recycled pid), `start` refuses ", + "again: run `devflow recover --clean --phase 94` first, then `devflow start`. Do neither ", + "while this phase's processes are live: `recover --clean` clears state even while an agent ", + "runs, and a later `start` would launch a second agent beside it.\n", + "Once every supervisor and process named above is gone, run `devflow start` again.", +); fn events_contain(root: &Path, event: &str) -> bool { fs::read_to_string(devflow_core::events::events_path(root)) @@ -630,13 +814,10 @@ fn events_contain(root: &Path, event: &str) -> bool { .contains(&format!("\"{event}\"")) } -fn plant_resumable_run( - root: &Path, - phase: PhaseId, - monitor_pid: Option, - agent_pid: Option, - stopped: bool, -) { +/// A resumable Code-stage state with no supervisor record and an optional +/// agent pid file. No `monitor_pid` is planted: the launch guard reads none +/// (50-15). +fn plant_resumable_run(root: &Path, phase: PhaseId, agent_pid: Option, stopped: bool) { let mut state = State::new( phase, AgentKind::Claude, @@ -644,7 +825,6 @@ fn plant_resumable_run( root.to_path_buf(), ); state.stage = Stage::Code; - state.monitor_pid = monitor_pid; state.legacy_claude_launch = true; state.stopped = stopped; state.stop_reason = @@ -655,10 +835,17 @@ fn plant_resumable_run( } } -fn resume_command(root: &Path, phase: PhaseId, fake_bin: &FakeBin, cache_dir: &Path) -> Command { +/// `devflow resume` for `phase`; its `DEVFLOW_CACHE_DIR` is `home`'s cache dir, +/// set by `common::hermetic_devflow`. +fn resume_command( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &common::FakeBin, +) -> Command { let existing = std::env::var_os("PATH").unwrap_or_default(); let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); - let mut command = Command::new(devflow_bin()); + let mut command = common::hermetic_devflow(home); command .args([ "resume", @@ -669,62 +856,26 @@ fn resume_command(root: &Path, phase: PhaseId, fake_bin: &FakeBin, cache_dir: &P .arg(root) .current_dir(root) .env("PATH", path) - .env("DEVFLOW_GATE_TIMEOUT_SECS", GATE_TIMEOUT_SECS) - .env("DEVFLOW_CACHE_DIR", cache_dir); + .env("DEVFLOW_GATE_TIMEOUT_SECS", GATE_TIMEOUT_SECS); command } -struct LaunchedRunReaper { - root: PathBuf, - phase: PhaseId, - spare: Vec, -} - -impl LaunchedRunReaper { - fn new(root: &Path, phase: PhaseId, spare: Vec) -> Self { - Self { - root: root.to_path_buf(), - phase, - spare, - } - } -} - -impl Drop for LaunchedRunReaper { - fn drop(&mut self) { - let monitor = devflow_core::workflow::load_state(&self.root, self.phase) - .ok() - .and_then(|state| state.monitor_pid); - let agent = fs::read_to_string(devflow_core::agent_result::agent_pid_path( - &self.root, self.phase, - )) - .ok() - .and_then(|pid| pid.trim().parse::().ok()); - for pid in [monitor, agent].into_iter().flatten() { - if !self.spare.contains(&pid) && devflow_core::agent::agent_running(pid) { - let _ = devflow_core::agent::terminate_and_verify( - pid, - devflow_core::agent::TERMINATE_VERIFY_WAIT, - devflow_core::agent::TERMINATE_VERIFY_POLL, - ); - } - } - } -} - struct RefusalCase { - role: &'static str, - live_pid: u32, required: Vec, - never_named: Vec, + /// The arm's own fragments. + present: Vec, + /// Text this arm must not contain. + absent: Vec, + /// A live agent pid the refusal must leave running. + spared_agent: Option, refusal: &'static str, } fn assert_resume_refuses_live_run( + home: &common::HermeticHome, root: &Path, phase: PhaseId, - fake_bin: &FakeBin, - cache_dir: &Path, + fake_bin: &common::FakeBin, case: &RefusalCase, ) { let before = phase_file_snapshot(root, phase); @@ -736,8 +887,9 @@ fn assert_resume_refuses_live_run( snapshot_paths(&before) ); } + let launched_before = stage_launched_count(root); - let output = resume_command(root, phase, fake_bin, cache_dir) + let output = resume_command(home, root, phase, fake_bin) .output() .expect("run devflow resume"); let stdout = String::from_utf8_lossy(&output.stdout); @@ -749,55 +901,31 @@ fn assert_resume_refuses_live_run( ); let padded = phase.padded(); - for fragment in [ + let common = [ + "refusing to resume; nothing was written".to_string(), format!( - "a run is still live ({} pid {} alive) — refusing to resume; nothing was written", - case.role, case.live_pid + "Once every supervisor and process named above is gone, run `devflow resume --phase {phase}` again." ), - format!("`devflow stop --phase {phase}` would only mark the state stopped: it signals nothing and does not end this run"), - "once it takes the lock after this refusal".to_string(), - "A state already marked stopped does not mean these processes have exited".to_string(), - "`devflow status` and `devflow doctor` name `devflow resume` for a phase whose monitor is dead even while its agent is still alive".to_string(), - "resume refuses until every named process has exited".to_string(), - "Check each pid before signalling it".to_string(), - "`ps -p` shows only the executable name".to_string(), - format!("{} pid {}: ps -ww -o pid=,lstart=,args= -p {}", case.role, case.live_pid, case.live_pid), - "`__monitor`".to_string(), - format!("`--phase {phase}`"), - format!("`.devflow/phase-{padded}-`"), - "`ps -o ppid= -p `".to_string(), - "so do not signal it on the pid alone".to_string(), - "send SIGTERM to the confirmed monitor first".to_string(), - "signalling the agent first lets the monitor launch the next stage".to_string(), - "may be running `devflow advance` as a foreground child".to_string(), - "the child may take the lock as soon as this resume exits".to_string(), - "`ps -ww -o pid=,args= --ppid `".to_string(), - "signal that child, not only the monitor".to_string(), - format!("If the named processes have exited, run `devflow resume --phase {phase}` again"), - "If a named pid is live but is not this phase's process (a recycled pid), `resume` refuses again".to_string(), - "no DevFlow verb clears a recorded pid while keeping the phase's work".to_string(), - format!("`devflow recover --clean --phase {phase}` removes the phase state, after which `resume` has nothing to resume, and `devflow start` refuses while the phase's worktree or branch exists"), - "`devflow start --force` recreates them from the base and discards the phase's work".to_string(), - "Do neither while this phase's processes are live".to_string(), - "a later `start` would launch a second agent beside it".to_string(), - ] { + ]; + for fragment in common.iter().chain(&case.present) { assert!( - stderr.contains(&fragment), + stderr.contains(fragment.as_str()), "refusal must contain {fragment:?}\nstderr: {stderr}" ); } - for stale in [ - "refusing to start".to_string(), - "run `devflow start` again".to_string(), - "as soon as this start exits".to_string(), - format!("ps -p {}", case.live_pid), - ] - .into_iter() - .chain(case.never_named.iter().cloned()) + for stale in removed_refusal_sentences(phase) + .into_iter() + .chain([ + "refusing to start".to_string(), + "run `devflow start` again".to_string(), + "as soon as this start exits".to_string(), + "ps -p ".to_string(), + ]) + .chain(case.absent.iter().cloned()) { assert!( !stderr.contains(&stale), - "refusal must not contain start-only or dead-pid text {stale:?}\nstderr: {stderr}" + "refusal must not contain start-only or superseded text {stale:?}\nstderr: {stderr}" ); } @@ -810,77 +938,101 @@ fn assert_resume_refuses_live_run( !root.join(format!(".devflow/lock-{padded}")).exists(), "a refused resume must not leave a phase lock" ); - assert!( - !events_contain(root, "stage_launched"), + assert_eq!( + stage_launched_count(root), + launched_before, "a refused resume must not emit stage_launched" ); - assert!( - devflow_core::agent::agent_running(case.live_pid), - "a refused resume must not signal the recorded {}", - case.role - ); + if let Some(pid) = case.spared_agent { + assert!( + devflow_core::agent::agent_running(pid), + "a refused resume must not signal the recorded agent {pid}" + ); + } +} + +/// How many `stage_launched` events the project's event log holds. +fn stage_launched_count(root: &Path) -> usize { + fs::read_to_string(devflow_core::events::events_path(root)) + .unwrap_or_default() + .matches("\"stage_launched\"") + .count() +} + +/// The resume agent arm's fragments beyond [`agent_arm_fragments`]. +fn resume_agent_fragments(phase: PhaseId) -> Vec { + vec![ + "A state already marked stopped does not mean these processes have exited".to_string(), + "`devflow status` names `devflow resume` for a phase whose supervisor is GONE even while its agent is still alive".to_string(), + "resume refuses until every named process has exited".to_string(), + "If a named pid is live but is not this phase's process (a recycled pid), `resume` refuses again".to_string(), + "no DevFlow verb clears a recorded pid while keeping the phase's work".to_string(), + format!("`devflow recover --clean --phase {phase}` removes the phase state, after which `resume` has nothing to resume, and `devflow start` refuses while the phase's worktree or branch exists"), + "`devflow start --force` recreates them from the base and discards the phase's work".to_string(), + ] } #[test] fn resume_refuses_a_phase_whose_recorded_monitor_is_alive() { let repo = tempfile::tempdir().unwrap(); - let cache = tempfile::tempdir().unwrap(); + let home = common::HermeticHome::new(); let root = repo.path(); let phase = PhaseId::new(85); init_repo(root, phase); - let monitor = LiveProcess::spawn(); - let fake_bin = fake_bin_dir("exec sleep 60"); - let _reaper = LaunchedRunReaper::new(root, phase, vec![monitor.pid()]); - plant_resumable_run( - root, - phase, - Some(monitor.pid()), - Some(NEVER_LIVE_PID), - false, - ); + let fake_bin = common::fake_bin_dir("exec sleep 60"); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + // Only the supervisor can refuse: the agent pid file is gone. + run.hide_agent_pid_file(root, phase); assert_resume_refuses_live_run( + &home, root, phase, &fake_bin, - cache.path(), &RefusalCase { - role: "monitor", - live_pid: monitor.pid(), - required: vec![ - devflow_core::workflow::state_path(root, phase), - devflow_core::agent_result::agent_pid_path(root, phase), - ], - never_named: vec![format!("agent pid {NEVER_LIVE_PID}")], - refusal: "resume must refuse a phase whose recorded monitor is alive", + required: vec![devflow_core::workflow::state_path(root, phase)], + present: supervisor_arm_fragments(phase, &run), + absent: vec!["agent pid".to_string()], + spared_agent: None, + refusal: "resume must refuse a phase whose recorded supervisor is alive", }, ); + assert!( + matches!(supervisor::probe(&run.socket), Ok(Liveness::Alive(_))), + "a refused resume must leave the live supervisor running" + ); } #[test] fn resume_refuses_a_stopped_phase_whose_recorded_agent_is_alive() { let repo = tempfile::tempdir().unwrap(); - let cache = tempfile::tempdir().unwrap(); + let home = common::HermeticHome::new(); let root = repo.path(); let phase = PhaseId::new(86); init_repo(root, phase); let agent = LiveProcess::spawn(); - let fake_bin = fake_bin_dir("exec sleep 60"); - let _reaper = LaunchedRunReaper::new(root, phase, vec![agent.pid()]); - plant_resumable_run(root, phase, Some(NEVER_LIVE_PID), Some(agent.pid()), true); + let fake_bin = common::fake_bin_dir("exec sleep 60"); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![agent.pid()]); + plant_resumable_run(root, phase, Some(agent.pid()), true); + let mut present = agent_arm_fragments(phase, agent.pid()); + present.extend(resume_agent_fragments(phase)); let case = RefusalCase { - role: "agent", - live_pid: agent.pid(), required: vec![ devflow_core::workflow::state_path(root, phase), devflow_core::agent_result::agent_pid_path(root, phase), ], - never_named: vec![format!("monitor pid {NEVER_LIVE_PID}")], + present, + absent: vec![ + "supervisor launch".to_string(), + format!("devflow stop --phase {phase} --launch"), + ], + spared_agent: Some(agent.pid()), refusal: "resume must refuse a phase whose recorded agent is alive", }; - assert_resume_refuses_live_run(root, phase, &fake_bin, cache.path(), &case); - assert_resume_refuses_live_run(root, phase, &fake_bin, cache.path(), &case); + assert_resume_refuses_live_run(&home, root, phase, &fake_bin, &case); + assert_resume_refuses_live_run(&home, root, phase, &fake_bin, &case); assert!( devflow_core::workflow::load_state(root, phase) .expect("load planted state") @@ -893,20 +1045,14 @@ fn resume_refuses_a_stopped_phase_whose_recorded_agent_is_alive() { fn resume_relaunches_a_phase_whose_recorded_processes_are_dead() { for (phase, stopped) in [(PhaseId::new(87), false), (PhaseId::new(88), true)] { let repo = tempfile::tempdir().unwrap(); - let cache = tempfile::tempdir().unwrap(); + let home = common::HermeticHome::new(); let root = repo.path(); init_repo(root, phase); - let fake_bin = fake_bin_dir("exec sleep 60"); - let _reaper = LaunchedRunReaper::new(root, phase, vec![]); - plant_resumable_run( - root, - phase, - Some(NEVER_LIVE_PID), - Some(NEVER_LIVE_PID), - stopped, - ); + let fake_bin = common::fake_bin_dir("exec sleep 60"); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + plant_resumable_run(root, phase, Some(NEVER_LIVE_PID), stopped); - let output = resume_command(root, phase, &fake_bin, cache.path()) + let output = resume_command(&home, root, phase, &fake_bin) .output() .expect("run devflow resume"); assert!( @@ -917,7 +1063,10 @@ fn resume_relaunches_a_phase_whose_recorded_processes_are_dead() { ); let state = devflow_core::workflow::load_state(root, phase).expect("load relaunched state"); assert!(!state.stopped, "resume must clear the stop mark"); - assert_ne!(state.monitor_pid, Some(NEVER_LIVE_PID)); + assert!( + state.supervisor.is_some(), + "the relaunch must record its supervisor: {state:?}" + ); assert!( events_contain(root, "stage_launched"), "a resumed dead leftover must emit stage_launched" @@ -928,12 +1077,12 @@ fn resume_relaunches_a_phase_whose_recorded_processes_are_dead() { #[test] fn resume_without_phase_state_reports_the_missing_state_not_a_live_run() { let repo = tempfile::tempdir().unwrap(); - let cache = tempfile::tempdir().unwrap(); + let home = common::HermeticHome::new(); let root = repo.path(); let phase = PhaseId::new(84); init_repo(root, phase); let agent = LiveProcess::spawn(); - let fake_bin = fake_bin_dir("exec sleep 60"); + let fake_bin = common::fake_bin_dir("exec sleep 60"); write_agent_pid(root, phase, agent.pid()); assert!( !devflow_core::workflow::state_path(root, phase).exists(), @@ -941,7 +1090,7 @@ fn resume_without_phase_state_reports_the_missing_state_not_a_live_run() { ); let before = phase_file_snapshot(root, phase); - let output = resume_command(root, phase, &fake_bin, cache.path()) + let output = resume_command(&home, root, phase, &fake_bin) .output() .expect("run devflow resume"); let stderr = String::from_utf8_lossy(&output.stderr); @@ -961,39 +1110,434 @@ fn resume_without_phase_state_reports_the_missing_state_not_a_live_run() { #[test] fn start_refusal_text_stays_byte_identical_for_a_live_monitor() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); let phase = PhaseId::new(94); init_repo(root, phase); - let monitor = LiveProcess::spawn(); - let fake_bin = fake_bin_dir("exit 0"); - plant_leftover_run(root, phase, Some(monitor.pid()), None); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + let agent = run.sidecar.agent_pid; + let fake_bin = common::fake_bin_dir("exit 0"); - let output = run_start(root, phase, &fake_bin, &[]); + let output = run_start(&home, root, phase, &fake_bin, &[]); let stderr = String::from_utf8_lossy(&output.stderr); assert!( !output.status.success(), - "start must still refuse a phase whose recorded monitor is alive\nstderr: {stderr}" + "start must still refuse a phase whose recorded supervisor is alive\nstderr: {stderr}" ); let expected = format!( "error: {}\n", - START_REFUSAL_GOLDEN_PHASE_94.replace("{pid}", &monitor.pid().to_string()) + START_REFUSAL_GOLDEN_PHASE_94 + .replace("{launch}", run.launch_id()) + .replace("{socket}", &run.socket.display().to_string()) + .replace("{pid}", &agent.to_string()) ); assert!( stderr.contains(&expected), - "the start refusal must stay byte-identical to 48-19's text\nstderr: {stderr}" + "the start refusal must match the golden byte for byte\nexpected: {expected}\nstderr: {stderr}" ); assert!( !stderr.contains(&expected.replacen("refusing to start", "refusing to resume", 1)), "the golden comparison must be sensitive to the launch verb\nstderr: {stderr}" ); + for placeholder in ["{launch}", "{socket}", "{pid}"] { + assert!( + START_REFUSAL_GOLDEN_PHASE_94.contains(placeholder), + "the golden must carry {placeholder}, or the substitution proves nothing" + ); + } + assert!(START_REFUSAL_GOLDEN_PHASE_94.ends_with( + "Once every supervisor and process named above is gone, run `devflow start` again." + )); +} + +// --------------------------------------------------------------------------- +// 50-15 (D-09, R-12, 999.139): a state file that will not load must not hide +// a live run from `start` or `resume`. Each fixture is a real launched run +// whose agent pid file is removed, so only its supervisor socket can refuse. +// --------------------------------------------------------------------------- + +/// Bytes no `State` deserializes. +const CORRUPT_JSON: &[u8] = b"{ this does not parse"; + +/// Overwrite `phase`'s state file with invalid JSON; returns the bytes. +fn corrupt_state(root: &Path, phase: PhaseId) -> Vec { + let path = devflow_core::workflow::state_path(root, phase); + fs::write(&path, CORRUPT_JSON).expect("corrupt the state file"); assert!( - START_REFUSAL_GOLDEN_PHASE_94.starts_with( - "phase 94: a run is still live (monitor pid {pid} alive) — refusing to start; nothing was written" - ) + devflow_core::workflow::load_state(root, phase).is_err(), + "the corrupted state must not load" ); + CORRUPT_JSON.to_vec() +} + +/// Every `.sock` file under `home`'s socket directory, sorted. +fn sockets_under(home: &common::HermeticHome) -> Vec { + let mut sockets: Vec = fs::read_dir(home.cache_dir().join("sock")) + .map(|entries| { + entries + .map(|entry| entry.expect("read sock dir entry").path()) + .filter(|path| path.extension().is_some_and(|ext| ext == "sock")) + .collect() + }) + .unwrap_or_default(); + sockets.sort(); + sockets +} + +/// `devflow start --force` for `phase`, as the 48-19 refusal tests run it. +fn force_start(home: &common::HermeticHome, root: &Path, phase: PhaseId) -> Output { + let fake_bin = common::fake_bin_dir("exit 0"); + run_start(home, root, phase, &fake_bin, &["--force"]) +} + +/// Run `launch` (a `start` or `resume` over a corrupt state) and assert it is +/// refused with no side effect: no new socket, the corrupt state bytes +/// unchanged, no `workflow_started` or `stage_launched` appended. Returns +/// its stderr. +fn assert_refused_over_a_corrupt_state( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + launch: impl FnOnce() -> Output, +) -> String { + let state_file = devflow_core::workflow::state_path(root, phase); + let state_before = fs::read(&state_file).expect("read the corrupt state"); + let sockets_before = sockets_under(home); + let started_before = workflow_started_count(root); + let launched_before = stage_launched_count(root); + + let output = launch(); + let stderr = String::from_utf8_lossy(&output.stderr).into_owned(); assert!( - START_REFUSAL_GOLDEN_PHASE_94 - .ends_with("a later `start` would launch a second agent beside it.") + !output.status.success(), + "a launch over a live run whose state will not load must refuse\nstdout: {}\nstderr: {stderr}", + String::from_utf8_lossy(&output.stdout) + ); + assert_eq!( + sockets_under(home), + sockets_before, + "a refused launch must publish no second socket\nstderr: {stderr}" + ); + assert_eq!( + fs::read(&state_file).expect("read the state after the refusal"), + state_before, + "a refused launch must leave the corrupt state bytes unchanged" + ); + assert_eq!( + workflow_started_count(root), + started_before, + "a refused launch must not emit workflow_started" ); + assert_eq!( + stage_launched_count(root), + launched_before, + "a refused launch must not emit stage_launched" + ); + stderr +} + +/// The backticked `devflow stop --phase …` command in `stderr`, split into +/// its words without the leading `devflow`. +fn printed_stop_command(stderr: &str, phase: PhaseId) -> Vec { + let head = format!("`devflow stop --phase {phase}"); + let at = stderr + .find(&head) + .unwrap_or_else(|| panic!("no {head}… command in the refusal\nstderr: {stderr}")); + let command = &stderr[at + 1..]; + let end = command + .find('`') + .expect("the printed command closes its backtick"); + command[..end] + .split_whitespace() + .skip(1) + .map(str::to_string) + .collect() +} + +/// Run a printed `devflow stop` command exactly as printed, from `root`. +fn run_printed_stop(home: &common::HermeticHome, root: &Path, words: &[String]) -> Output { + common::hermetic_devflow(home) + .args(words) + .current_dir(root) + .output() + .expect("run the printed devflow stop") +} + +/// Whether process group `pgid` has no live member left. Zombies do not +/// count: a CI job container's PID 1 never reaps an orphaned one +/// (`test_support::live_group_members`). +fn group_is_gone(pgid: u32) -> bool { + devflow_core::test_support::live_group_members(pgid).is_empty() +} + +#[test] +fn start_refuses_over_a_live_supervisor_when_the_state_will_not_load() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let phase = PhaseId::new(70); + init_repo(root, phase); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + run.hide_agent_pid_file(root, phase); + corrupt_state(root, phase); + + let stderr = + assert_refused_over_a_corrupt_state(&home, root, phase, || force_start(&home, root, phase)); + let id = run.launch_id(); + for fragment in [ + "refusing to start; nothing was written".to_string(), + format!("supervisor launch {id} ALIVE"), + format!("run `devflow stop --phase {phase} --launch {id}` first"), + ] { + assert!( + stderr.contains(&fragment), + "refusal must contain {fragment:?}\nstderr: {stderr}" + ); + } + assert!( + !stderr.contains("agent pid"), + "only the socket may refuse here (the pid file is gone)\nstderr: {stderr}" + ); + assert!( + matches!(supervisor::probe(&run.socket), Ok(Liveness::Alive(_))), + "a refused start must leave the live supervisor running" + ); +} + +#[test] +fn the_corrupt_state_refusal_names_a_stop_that_ends_the_run() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let phase = PhaseId::new(71); + init_repo(root, phase); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + run.hide_agent_pid_file(root, phase); + let corrupt = corrupt_state(root, phase); + let stderr = + assert_refused_over_a_corrupt_state(&home, root, phase, || force_start(&home, root, phase)); + let words = printed_stop_command(&stderr, phase); + assert_eq!( + words, + [ + "stop", + "--phase", + &phase.to_string(), + "--launch", + run.launch_id() + ], + "the refusal must print the `--launch` form\nstderr: {stderr}" + ); + + let stop = run_printed_stop(&home, root, &words); + assert!( + stop.status.success(), + "the printed stop must end the run\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&stop.stdout), + String::from_utf8_lossy(&stop.stderr) + ); + common::wait_for( + || matches!(supervisor::probe(&run.socket), Ok(Liveness::Gone)), + Duration::from_secs(5), + "the stopped supervisor's socket to probe Gone", + ); + common::wait_for( + || group_is_gone(run.sidecar.agent_pgid), + Duration::from_secs(5), + "the stopped run's agent group to be gone", + ); + assert_eq!( + fs::read(devflow_core::workflow::state_path(root, phase)).unwrap(), + corrupt, + "stop must leave the corrupt state bytes unchanged" + ); + + let again = force_start(&home, root, phase); + let again_stderr = String::from_utf8_lossy(&again.stderr); + assert!( + !again_stderr.contains("a run is still live"), + "once the run is stopped, start must not refuse over it\nstderr: {again_stderr}" + ); +} + +#[test] +fn resume_refuses_over_a_live_supervisor_when_the_state_will_not_load() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let phase = PhaseId::new(72); + init_repo(root, phase); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + run.hide_agent_pid_file(root, phase); + corrupt_state(root, phase); + let fake_bin = common::fake_bin_dir("exec sleep 60"); + + let stderr = assert_refused_over_a_corrupt_state(&home, root, phase, || { + resume_command(&home, root, phase, &fake_bin) + .output() + .expect("run devflow resume") + }); + let id = run.launch_id(); + for fragment in [ + "refusing to resume; nothing was written".to_string(), + format!("supervisor launch {id} ALIVE"), + format!("run `devflow stop --phase {phase} --launch {id}` first"), + ] { + assert!( + stderr.contains(&fragment), + "refusal must contain {fragment:?}\nstderr: {stderr}" + ); + } + assert!( + matches!(supervisor::probe(&run.socket), Ok(Liveness::Alive(_))), + "a refused resume must leave the live supervisor running" + ); +} + +#[test] +fn start_proceeds_over_a_corrupt_state_with_no_live_supervisor() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let phase = PhaseId::new(73); + init_repo(root, phase); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + run.hide_agent_pid_file(root, phase); + match supervisor::request_shutdown(&run.socket, LIVE_RUN_SHUTDOWN_WAIT) + .expect("shut the fixture's run down") + { + ShutdownOutcome::Stopped { .. } => {} + other => panic!("the fixture's shutdown must stop the run, got {other:?}"), + } + common::wait_for( + || !run.socket.exists() && !run.sidecar_path().exists(), + Duration::from_secs(5), + "the shut-down run's socket and sidecar to be removed", + ); + corrupt_state(root, phase); + let started_before = workflow_started_count(root); + + let output = force_start(&home, root, phase); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + output.status.success(), + "with no live supervisor, start over a corrupt state proceeds as today\nstdout: {}\nstderr: {stderr}", + String::from_utf8_lossy(&output.stdout) + ); + assert!( + !stderr.contains("a run is still live"), + "no refusal over a dead run\nstderr: {stderr}" + ); + assert_eq!( + workflow_started_count(root), + started_before + 1, + "the start over the corrupt state must emit workflow_started" + ); +} + +#[test] +fn start_refuses_when_the_state_and_the_live_sidecar_both_will_not_load() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let phase = PhaseId::new(74); + init_repo(root, phase); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + run.hide_agent_pid_file(root, phase); + let (pgid, started) = (run.sidecar.agent_pgid, run.sidecar.agent_start_time); + corrupt_state(root, phase); + fs::write(run.sidecar_path(), CORRUPT_JSON).expect("corrupt the live sidecar"); + assert!(supervisor::read_sidecar(&run.sidecar_path()).is_err()); + + let stderr = + assert_refused_over_a_corrupt_state(&home, root, phase, || force_start(&home, root, phase)); + for fragment in [ + "refusing to start; nothing was written".to_string(), + format!("socket {} (ALIVE", run.socket.display()), + format!("launch {}", run.launch_id()), + "does not parse".to_string(), + format!("run `devflow stop --phase {phase}` first"), + "one that does not answer cannot be validated without its sidecar".to_string(), + ] { + assert!( + stderr.contains(&fragment), + "refusal must contain {fragment:?}\nstderr: {stderr}" + ); + } + assert!( + !stderr.contains("--launch"), + "a socket with no usable sidecar is not addressed by `--launch`\nstderr: {stderr}" + ); + let words = printed_stop_command(&stderr, phase); + assert_eq!(words, ["stop", "--phase", &phase.to_string()]); + + assert!( + devflow_core::agent::is_same_process(pgid, started), + "the agent group's leader must still be the launched agent before the stop" + ); + let stop = run_printed_stop(&home, root, &words); + assert!( + stop.status.success(), + "the printed stop must end the run\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&stop.stdout), + String::from_utf8_lossy(&stop.stderr) + ); + common::wait_for( + || matches!(supervisor::probe(&run.socket), Ok(Liveness::Gone)), + Duration::from_secs(5), + "the stopped supervisor's socket to probe Gone", + ); + common::wait_for( + || group_is_gone(pgid), + Duration::from_secs(5), + "the stopped run's agent group to be gone", + ); +} + +#[test] +fn a_live_socket_named_for_another_root_does_not_block_a_corrupt_state_start() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path(); + let other_repo = tempfile::tempdir().unwrap(); + let other = other_repo.path(); + let phase = PhaseId::new(75); + init_repo(root, phase); + init_repo(other, phase); + let _reaper = common::LaunchedRunReaper::new(root, phase, vec![]); + let _other_reaper = common::LaunchedRunReaper::new(other, phase, vec![]); + let run = LiveRun::launch(&home, root, phase); + run.hide_agent_pid_file(root, phase); + corrupt_state(root, phase); + fs::write(run.sidecar_path(), CORRUPT_JSON).expect("corrupt the live sidecar"); + fs::create_dir_all(other.join(".devflow")).unwrap(); + corrupt_state(other, phase); + + let output = force_start(&home, other, phase); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + !stderr.contains("devflow stop --phase"), + "a socket named for another root must not refuse this root's start\nstderr: {stderr}" + ); + assert!( + !stderr.contains("a run is still live"), + "no live-run refusal for another root's socket\nstderr: {stderr}" + ); + assert!( + matches!(supervisor::probe(&run.socket), Ok(Liveness::Alive(_))), + "the other root's start must leave this root's supervisor running" + ); + match supervisor::request_shutdown(&run.socket, LIVE_RUN_SHUTDOWN_WAIT) + .expect("shut this root's run down") + { + ShutdownOutcome::Stopped { .. } => {} + other => panic!("the shutdown must stop the run, got {other:?}"), + } } diff --git a/crates/devflow-cli/tests/start_reachability_e2e.rs b/crates/devflow-cli/tests/start_reachability_e2e.rs index 8c90e4ab..9ec368b8 100644 --- a/crates/devflow-cli/tests/start_reachability_e2e.rs +++ b/crates/devflow-cli/tests/start_reachability_e2e.rs @@ -10,14 +10,12 @@ //! stderr, and filesystem effects — never on `preflight.rs`/`commands.rs` //! source text (test-signal-rejection pattern 4). +mod common; + use std::fs; use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; -use std::process::{Command, Output}; - -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} +use std::process::Output; /// Hermetic git invocation pinned to `root` (999.37) — never a bare /// `Command::new("git")`. @@ -115,12 +113,13 @@ fn path_with_fake_bin(fake_bin: &Path) -> String { #[test] fn start_refuses_a_phase_promoted_only_on_the_working_branch_and_scaffolds_nothing() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo_with_phase_24_promoted_only_on_feature_branch(root); let fake_bin = fake_bin_dir(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", "24", "--agent", "claude", "--mode", "auto", ]) @@ -185,12 +184,13 @@ fn start_refuses_a_phase_promoted_only_on_the_working_branch_and_scaffolds_nothi /// branch is created. #[test] fn start_refuses_before_creating_the_feature_branch_in_no_worktree_mode() { + let home = common::HermeticHome::new(); let repo = tempfile::tempdir().unwrap(); let root = repo.path(); init_repo_with_phase_24_promoted_only_on_feature_branch(root); let fake_bin = fake_bin_dir(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args([ "start", "--phase", diff --git a/crates/devflow-cli/tests/stop_e2e.rs b/crates/devflow-cli/tests/stop_e2e.rs index 8f3f7bac..6959427c 100644 --- a/crates/devflow-cli/tests/stop_e2e.rs +++ b/crates/devflow-cli/tests/stop_e2e.rs @@ -10,21 +10,20 @@ //! `gate_sweep_e2e.rs`) — this drives the real compiled binary via //! `Command::new`, not an in-process call. +mod common; + use devflow_core::gates::{GateResponse, Gates}; use devflow_core::mode::Mode; use devflow_core::phase_id::PhaseId; use devflow_core::stage::Stage; -use devflow_core::state::{AgentKind, State}; +use devflow_core::state::{AgentKind, State, SupervisorRecord}; +use devflow_core::supervisor::{self, Liveness}; #[cfg(unix)] use std::os::unix::ffi::OsStrExt; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::process::{Child, Command}; use std::time::{Duration, Instant}; -fn devflow_bin() -> &'static str { - env!("CARGO_BIN_EXE_devflow") -} - /// Hermetic `git` invocation for fixture setup (999.37) — never a bare /// `Command::new("git")`. fn git(root: &Path, args: &[&str]) { @@ -97,15 +96,9 @@ fn write_live_lock(root: &Path, phase: PhaseId, child: &Child, recorded_start: u std::fs::write(lock, format!("{}\n{recorded_start}", child.id())).unwrap(); } -fn kill_and_reap(child: &mut Child) { - if child.try_wait().expect("poll holder").is_none() { - let _ = child.kill(); - } - let _ = child.wait(); -} - #[test] fn stop_marks_stopped_after_the_signalled_lock_holder_exits() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(103); @@ -118,7 +111,7 @@ fn stop_marks_stopped_after_the_signalled_lock_holder_exits() { let start = devflow_core::agent::process_start_time(holder.id()).expect("holder start time"); write_live_lock(root, phase, &holder, start); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -141,6 +134,7 @@ fn stop_marks_stopped_after_the_signalled_lock_holder_exits() { #[test] fn stop_writes_no_state_while_the_lock_holder_survives_the_signal() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(104); @@ -155,7 +149,7 @@ fn stop_writes_no_state_while_the_lock_holder_survives_the_signal() { let start = devflow_core::agent::process_start_time(holder.id()).expect("holder start time"); write_live_lock(root, phase, &holder, start); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -170,39 +164,66 @@ fn stop_writes_no_state_while_the_lock_holder_survives_the_signal() { assert!(stderr.contains(&holder.id().to_string()) && stderr.contains("not marked")); assert_eq!(std::fs::read(&state_path).unwrap(), before); assert!(holder.try_wait().expect("poll holder").is_none()); - kill_and_reap(&mut holder); + common::kill_and_reap(&mut holder); } +/// T-50-38 (renamed from `stop_never_signals_the_recorded_monitor_pid`): +/// `stop` never signals a pid read from state; it addresses the socket. The +/// state plants a live test-owned process as the agent group of a +/// `supervisor` record whose socket file does not exist, so the probe reads +/// GONE: `stop` reports nothing running and the planted process is still +/// alive afterwards. Also covers T-23-51's unit test (removed in 50-22): a +/// pid recorded in state alone is never a signalling target or a blocker. #[test] -fn stop_never_signals_the_recorded_monitor_pid() { +fn stop_never_signals_a_recorded_pid_it_addresses_the_socket() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(105); - let mut monitor = Command::new("sleep") + let mut planted = Command::new("sleep") .arg("60") .spawn() - .expect("spawn monitor"); + .expect("spawn the planted process"); + let start = devflow_core::agent::process_start_time(planted.id()).expect("start time"); let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); - state.monitor_pid = Some(monitor.id()); + state.supervisor = Some(SupervisorRecord::new( + home.cache_dir().join("sock").join("missing.sock"), + "launch-missing".to_string(), + planted.id(), + start, + supervisor::boot_id().unwrap_or_default(), + )); devflow_core::workflow::save_state(&state).unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() .expect("run devflow stop"); - assert!(output.status.success()); - assert!(monitor.try_wait().expect("poll monitor").is_none()); + let text = describe(&output); + assert!(output.status.success(), "{text}"); + assert!( + String::from_utf8_lossy(&output.stdout).contains("nothing is running"), + "a GONE socket and no lock read as nothing running: {text}" + ); + assert!( + planted + .try_wait() + .expect("poll the planted process") + .is_none(), + "the planted process named only by state is never signalled" + ); assert!( devflow_core::workflow::load_state(root, phase) .unwrap() .stopped ); - kill_and_reap(&mut monitor); + common::kill_and_reap(&mut planted); } #[test] fn stop_refuses_to_signal_a_lock_holder_whose_start_time_does_not_match() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(106); @@ -218,7 +239,7 @@ fn stop_refuses_to_signal_a_lock_holder_whose_start_time_does_not_match() { devflow_core::agent::process_start_time(holder.id()).expect("holder start time") + 1; write_live_lock(root, phase, &holder, wrong_start); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -227,7 +248,7 @@ fn stop_refuses_to_signal_a_lock_holder_whose_start_time_does_not_match() { assert!(String::from_utf8_lossy(&output.stderr).contains("refusing to signal")); assert!(holder.try_wait().expect("poll holder").is_none()); assert_eq!(std::fs::read(&state_path).unwrap(), before); - kill_and_reap(&mut holder); + common::kill_and_reap(&mut holder); } /// Saves a fresh state for `phase`, starts a live `sleep` as the process the @@ -253,6 +274,7 @@ fn gated_phase_with_foreign_holder( /// gate for the real recovery path. #[test] fn stop_at_a_ship_gate_with_a_recycled_holder_refuses_without_writing_a_response() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(108); @@ -267,7 +289,7 @@ fn stop_at_a_ship_gate_with_a_recycled_holder_refuses_without_writing_a_response write_live_lock(root, phase, &holder, wrong_start); Gates::write_gate(root, phase, Stage::Ship, "merge").unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -292,7 +314,7 @@ fn stop_at_a_ship_gate_with_a_recycled_holder_refuses_without_writing_a_response ); assert!(holder.try_wait().expect("poll holder").is_none()); assert_eq!(std::fs::read(&state_path).unwrap(), before); - kill_and_reap(&mut holder); + common::kill_and_reap(&mut holder); } /// Finding B: a legacy single-line lock makes the holder's identity @@ -300,6 +322,7 @@ fn stop_at_a_ship_gate_with_a_recycled_holder_refuses_without_writing_a_response /// response that no confirmed process can consume. #[test] fn stop_at_a_gate_with_an_unconfirmable_holder_refuses_unevidenced_success() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(109); @@ -315,7 +338,7 @@ fn stop_at_a_gate_with_an_unconfirmable_holder_refuses_unevidenced_success() { std::fs::write(lock, holder.id().to_string()).unwrap(); Gates::write_gate(root, phase, Stage::Code, "paused").unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -345,7 +368,7 @@ fn stop_at_a_gate_with_an_unconfirmable_holder_refuses_unevidenced_success() { ); assert!(holder.try_wait().expect("poll holder").is_none()); assert_eq!(std::fs::read(&state_path).unwrap(), before); - kill_and_reap(&mut holder); + common::kill_and_reap(&mut holder); } /// Finding C-3: an old gate request is not evidence that the current live @@ -353,6 +376,7 @@ fn stop_at_a_gate_with_an_unconfirmable_holder_refuses_unevidenced_success() { /// writing an abort response and reporting an unevidenced successful stop. #[test] fn stop_does_not_treat_a_stale_gate_as_the_live_holders_wait() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(112); @@ -369,7 +393,7 @@ fn stop_does_not_treat_a_stale_gate_as_the_live_holders_wait() { write_live_lock(root, phase, &holder, start); Gates::write_gate(root, phase, Stage::Code, "stale request").unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -381,7 +405,7 @@ fn stop_does_not_treat_a_stale_gate_as_the_live_holders_wait() { String::from_utf8_lossy(&output.stderr) ); let Some(status) = holder.try_wait().expect("poll holder") else { - kill_and_reap(&mut holder); + common::kill_and_reap(&mut holder); panic!("stop must signal the live holder instead of trusting the stale gate"); }; assert!(!status.success(), "SIGTERM must end the live holder"); @@ -403,12 +427,13 @@ fn stop_does_not_treat_a_stale_gate_as_the_live_holders_wait() { /// live holder exists. #[test] fn gate_approve_without_a_live_holder_does_not_claim_advance() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(110); Gates::write_gate(root, phase, Stage::Ship, "approve merge").unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["gate", "approve", &phase.to_string(), "--project"]) .arg(root) .output() @@ -476,6 +501,7 @@ fn wait_for_child_exit( /// class `23-ORPHAN-FORENSICS.md` documented, now remedied. #[test] fn stop_ends_a_gated_phase_through_its_own_abort_path_with_no_signal_sent() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(95); @@ -490,7 +516,7 @@ fn stop_ends_a_gated_phase_through_its_own_abort_path_with_no_signal_sent() { // failed stop still lets `wait_for_child_exit`'s cleanup `wait()` // return promptly. Set per-`Command` via `.env(...)`, never via a // process-global env mutation (999.37). - let mut child = Command::new(devflow_bin()) + let mut child = common::hermetic_devflow(&home) .args(["advance", "--phase", &phase.to_string()]) .arg(root) .env("DEVFLOW_GATE_TIMEOUT_SECS", "15") @@ -518,7 +544,7 @@ fn stop_ends_a_gated_phase_through_its_own_abort_path_with_no_signal_sent() { blocking on it" ); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -566,6 +592,7 @@ fn stop_ends_a_gated_phase_through_its_own_abort_path_with_no_signal_sent() { /// may race a second writer). #[test] fn stop_marks_state_stopped_and_records_reason() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(96); @@ -576,7 +603,7 @@ fn stop_marks_state_stopped_and_records_reason() { state.gate_pending = true; devflow_core::workflow::save_state(&state).unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -603,6 +630,7 @@ fn stop_marks_state_stopped_and_records_reason() { /// `--until` halt) must survive — appended to, not overwritten. #[test] fn stop_preserves_pre_existing_stop_reason() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(97); @@ -612,7 +640,7 @@ fn stop_preserves_pre_existing_stop_reason() { state.stop_reason = Some("stopped after plan completed (--until plan)".to_string()); devflow_core::workflow::save_state(&state).unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -636,6 +664,7 @@ fn stop_preserves_pre_existing_stop_reason() { /// halt stage for `devflow start --until`) and must be left untouched. #[test] fn stop_leaves_stop_until_unchanged() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(98); @@ -644,7 +673,7 @@ fn stop_leaves_stop_until_unchanged() { state.stop_until = Some(Stage::Plan); devflow_core::workflow::save_state(&state).unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -665,7 +694,8 @@ fn stop_leaves_stop_until_unchanged() { /// verification map. #[test] fn stop_help_documents_phase_flag() { - let output = Command::new(devflow_bin()) + let home = common::HermeticHome::new(); + let output = common::hermetic_devflow(&home) .args(["stop", "--help"]) .output() .expect("run devflow stop --help"); @@ -689,6 +719,7 @@ fn stop_help_documents_phase_flag() { /// one wrote. #[test] fn stop_is_idempotent_against_an_already_answered_gate() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(99); @@ -700,7 +731,7 @@ fn stop_is_idempotent_against_an_already_answered_gate() { devflow_core::workflow::save_state(&state).unwrap(); let run_stop = || { - Command::new(devflow_bin()) + common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -735,6 +766,7 @@ fn stop_is_idempotent_against_an_already_answered_gate() { /// success no-op here, not a clobber. #[test] fn stop_against_a_hand_written_response_is_a_success_no_op() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(100); @@ -754,7 +786,7 @@ fn stop_against_a_hand_written_response_is_a_success_no_op() { let response_path = Gates::response_path(root, phase, Stage::Ship); let before = std::fs::read(&response_path).unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -777,11 +809,12 @@ fn stop_against_a_hand_written_response_is_a_success_no_op() { /// not an error. #[test] fn stop_against_a_root_with_no_state_is_a_success() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(101); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -800,6 +833,7 @@ fn stop_against_a_root_with_no_state_is_a_success() { /// fail-closed guarantee is provably unweakened by this plan. #[test] fn stop_then_cleanup_composes_refuse_then_force() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(102); @@ -832,7 +866,7 @@ fn stop_then_cleanup_composes_refuse_then_force() { // No open gate, no lock — the lock-holder fallback is a no-op, and the // only observable effect of this stop is state.stopped/stop_reason. - let stopped = Command::new(devflow_bin()) + let stopped = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root) .output() @@ -850,7 +884,7 @@ fn stop_then_cleanup_composes_refuse_then_force() { // cleanup without --force must still decline — the fail-closed // guarantee this plan's design depends on and does not weaken. - let refused = Command::new(devflow_bin()) + let refused = common::hermetic_devflow(&home) .args(["cleanup"]) .arg(root) .output() @@ -867,7 +901,7 @@ fn stop_then_cleanup_composes_refuse_then_force() { // cleanup --force is the documented escape hatch — the two verbs // compose in that order. - let forced = Command::new(devflow_bin()) + let forced = common::hermetic_devflow(&home) .args(["cleanup", "--force"]) .arg(root) .output() @@ -893,6 +927,7 @@ fn stop_then_cleanup_composes_refuse_then_force() { /// root and the stop is asserted to have landed THERE. #[test] fn stop_accepts_the_project_root_as_a_positional_argument() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let root = dir.path(); let phase = PhaseId::new(103); @@ -900,7 +935,7 @@ fn stop_accepts_the_project_root_as_a_positional_argument() { let state = State::new(phase, AgentKind::Claude, Mode::Auto, root.to_path_buf()); devflow_core::workflow::save_state(&state).unwrap(); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string()]) .arg(root) .output() @@ -940,6 +975,7 @@ fn stop_accepts_the_project_root_as_a_positional_argument() { /// measurement of PRECEDENCE rather than of the walk. #[test] fn stop_root_flag_takes_precedence_over_the_positional() { + let home = common::HermeticHome::new(); let dir_a = tempfile::tempdir().unwrap(); let dir_b = tempfile::tempdir().unwrap(); let root_a = dir_a.path(); @@ -959,7 +995,7 @@ fn stop_root_flag_takes_precedence_over_the_positional() { // Ordering 1 — flag names A, positional names B. seed(root_a); seed(root_b); - let first = Command::new(devflow_bin()) + let first = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root_a) .arg(root_b) @@ -979,7 +1015,7 @@ fn stop_root_flag_takes_precedence_over_the_positional() { // Ordering 2 — roles reversed: flag names B, positional names A. seed(root_a); seed(root_b); - let second = Command::new(devflow_bin()) + let second = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg(root_b) .arg(root_a) @@ -1008,6 +1044,7 @@ fn stop_root_flag_takes_precedence_over_the_positional() { /// discriminate. #[test] fn stop_against_a_nonexistent_positional_root_names_the_offending_path() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let missing = dir.path().join("no-such-project-root"); let phase = PhaseId::new(105); @@ -1017,7 +1054,7 @@ fn stop_against_a_nonexistent_positional_root_names_the_offending_path() { "fixture error: the offending path must NOT exist" ); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string()]) .arg(&missing) .output() @@ -1058,10 +1095,11 @@ fn stop_against_a_nonexistent_positional_root_names_the_offending_path() { /// is the precise legibility failure the plan was worried about. #[test] fn stop_with_an_empty_project_root_is_refused_identically_for_both_spellings() { + let home = common::HermeticHome::new(); let phase = PhaseId::new(106); // Spelling 1 — empty POSITIONAL. - let positional = Command::new(devflow_bin()) + let positional = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string()]) .arg("") .output() @@ -1085,7 +1123,7 @@ fn stop_with_an_empty_project_root_is_refused_identically_for_both_spellings() { ); // Spelling 2 — empty value supplied to the long FLAG. - let flag = Command::new(devflow_bin()) + let flag = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string(), "--root"]) .arg("") .output() @@ -1120,6 +1158,7 @@ fn stop_with_an_empty_project_root_is_refused_identically_for_both_spellings() { #[cfg(unix)] #[test] fn stop_positional_root_survives_a_non_utf8_path() { + let home = common::HermeticHome::new(); let dir = tempfile::tempdir().unwrap(); let phase = PhaseId::new(107); @@ -1138,7 +1177,7 @@ fn stop_positional_root_survives_a_non_utf8_path() { "fixture error: the offending path must NOT exist" ); - let output = Command::new(devflow_bin()) + let output = common::hermetic_devflow(&home) .args(["stop", "--phase", &phase.to_string()]) .arg(&missing) .output() @@ -1154,3 +1193,1087 @@ fn stop_positional_root_survives_a_non_utf8_path() { "a non-UTF-8 root must survive clap and reach project_root:\n{stderr}" ); } + +// --------------------------------------------------------------------------- +// 50-14: `stop` addresses the run's supervisor socket (D-04, D-06). +// +// These tests launch real supervised runs with `devflow start --mode +// supervise` under a `HermeticHome`, so every socket and sidecar lives under +// the test's tempdir cache. A pid a test signals comes from a sidecar and is +// confirmed with `agent::is_same_process` against its recorded start time. +// --------------------------------------------------------------------------- + +/// How long a test waits for a supervised run to reach a state. +const RUN_WAIT: Duration = Duration::from_secs(60); + +/// A repository on `develop` with a roadmap entry for `phase`, the shape +/// `supervisor_e2e.rs` launches a real supervised run from. +fn init_supervised_repo(root: &Path, phase: PhaseId) { + git(root, &["init", "-q"]); + git(root, &["config", "user.email", "devflow@example.com"]); + git(root, &["config", "user.name", "DevFlow Tests"]); + git(root, &["config", "commit.gpgsign", "false"]); + git(root, &["config", "core.hooksPath", "/dev/null"]); + git(root, &["checkout", "-q", "-b", "develop"]); + std::fs::create_dir_all(root.join(".planning")).unwrap(); + std::fs::write(root.join("README.md"), "base\n").unwrap(); + std::fs::write( + root.join(".planning/ROADMAP.md"), + format!("# Roadmap\n\n### Phase {phase}: stop fixture\n"), + ) + .unwrap(); + git(root, &["add", "README.md", ".planning/ROADMAP.md"]); + git(root, &["commit", "-q", "-m", "base"]); +} + +/// The fake `claude`'s health check and delivery canary, as in +/// `supervisor_e2e.rs`; `body` runs for the real stage turn. +fn fake_claude(body: &str) -> String { + format!( + r#"if [ "$1" = "--version" ]; then echo "2.1.220 (Claude Code)"; exit 0; fi +read -r turn +case "$turn" in + *DEVFLOW_DELIVERY_CANARY_*) + token=$(printf '%s' "$turn" | grep -o 'DEVFLOW_DELIVERY_CANARY_[0-9a-f]*' | head -1) + printf '{{"type":"result","subtype":"success","is_error":false,"session_id":"s-fake","result":"%s"}}\n' "$token" + exit 0 + ;; +esac +{body}"# + ) +} + +/// What a [`triggered_agent`] does once released: one empty commit, the +/// success marker, exit 0. +const SUCCEEDS: &str = r#"git commit -q --allow-empty -m "fake stage work" +printf 'work done\nDEVFLOW_RESULT: {"status":"success","commits":1}\n' +exit 0"#; + +/// What a [`triggered_agent`] does once released: a failed Define, which the +/// never-silent path parks at a human gate. +const FAILS_TO_A_GATE: &str = r#"printf 'DEVFLOW_RESULT: {"status":"failed","reason":"gate fixture"}\n' +exit 0"#; + +/// A fake agent that keeps running until `trigger` exists, then runs +/// `released` ([`SUCCEEDS`] or [`FAILS_TO_A_GATE`]). +fn triggered_agent(trigger: &Path, released: &str) -> String { + fake_claude(&format!( + "while [ ! -e '{trigger}' ]; do sleep 0.05; done\n{released}", + trigger = trigger.display() + )) +} + +/// The `stage_launched` events recorded for `phase`. +fn stage_launched_count(root: &Path, phase: PhaseId) -> usize { + std::fs::read_to_string(devflow_core::events::events_path(root)) + .unwrap_or_default() + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .filter(|event| { + phase.matches_json(event.get("phase")) + && event["event"].as_str() == Some("stage_launched") + }) + .count() +} + +/// The pids of the live members of process group `pgid`. Zombies do not +/// count: a CI job container's PID 1 never reaps an orphaned one +/// (`test_support::live_group_members`). +fn group_members(pgid: u32) -> Vec { + devflow_core::test_support::live_group_members(pgid) +} + +/// Poll `done` every 25 ms until it holds or `within` elapses. +fn holds_within(within: Duration, mut done: impl FnMut() -> bool) -> bool { + let deadline = Instant::now() + within; + loop { + if done() { + return true; + } + if Instant::now() >= deadline { + return false; + } + std::thread::sleep(Duration::from_millis(25)); + } +} + +/// Ends, on drop, the run a captured supervisor record names, whether or not +/// its state still loads (a test may corrupt it): a socket shutdown, then a +/// SIGKILL to the agent group when its leader is still the recorded process. +struct RecordReaper(SupervisorRecord); + +impl Drop for RecordReaper { + fn drop(&mut self) { + let _ = supervisor::request_shutdown(&self.0.socket_path, Duration::from_secs(15)); + let leader = self.0.agent_pgid; + if devflow_core::agent::agent_running(leader) + && devflow_core::agent::is_same_process(leader, self.0.agent_start_time) + { + common::signal_group(leader, libc::SIGKILL); + } + } +} + +/// One live supervised run of a [`triggered_agent`], started with +/// `devflow start --mode supervise --until plan`. Its reapers are declared +/// first so they drop before the home and repository they need. +struct LiveRun { + _record_reaper: RecordReaper, + _launch_reaper: common::LaunchedRunReaper, + home: common::HermeticHome, + _repo: tempfile::TempDir, + _fake_bin: common::FakeBin, + root: PathBuf, + phase: PhaseId, + trigger: PathBuf, + record: SupervisorRecord, +} + +impl LiveRun { + /// A run whose agent succeeds once released. + fn start(phase: PhaseId) -> Self { + Self::start_with(phase, SUCCEEDS) + } + + /// A run whose agent runs `released` once its trigger exists. + fn start_with(phase: PhaseId, released: &str) -> Self { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + init_supervised_repo(&root, phase); + let trigger = root.join(".git").join("devflow-test-trigger"); + let fake_bin = common::fake_bin_dir(&triggered_agent(&trigger, released)); + let launch_reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + let existing = std::env::var_os("PATH").unwrap_or_default(); + let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); + let output = common::hermetic_devflow(&home) + .args([ + "start", + "--phase", + &phase.to_string(), + "--agent", + "claude", + "--mode", + "supervise", + "--no-worktree", + "--until", + "plan", + ]) + .arg(&root) + .current_dir(&root) + .env("PATH", path) + .output() + .expect("run devflow start"); + assert!( + output.status.success(), + "start failed\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + let record = devflow_core::workflow::load_state(&root, phase) + .expect("start saved state") + .supervisor + .expect("start records state.supervisor"); + Self { + _record_reaper: RecordReaper(record.clone()), + _launch_reaper: launch_reaper, + home, + _repo: repo, + _fake_bin: fake_bin, + root, + phase, + trigger, + record, + } + } + + /// `devflow stop --phase N` plus `extra`, run from the project root. + fn stop(&self, extra: &[&str]) -> std::process::Output { + common::hermetic_devflow(&self.home) + .args(["stop", "--phase", &self.phase.to_string()]) + .args(extra) + .current_dir(&self.root) + .output() + .expect("run devflow stop") + } + + fn state_file(&self) -> PathBuf { + devflow_core::workflow::state_path(&self.root, self.phase) + } + + fn probe(&self) -> Liveness { + supervisor::probe(&self.record.socket_path).expect("probe the recorded socket") + } +} + +/// `stop`'s stdout and stderr, for an assertion message. +fn describe(output: &std::process::Output) -> String { + format!( + "exit {:?}\nstdout: {}\nstderr: {}", + output.status.code(), + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ) +} + +/// The agent's work once released: ignore SIGTERM (its `sleep` children +/// inherit that), mark that it does, then idle about 60 s. +const IGNORES_TERM: &str = "trap '' TERM +: > .git/devflow-test-ignores-term +i=0 +while [ \"$i\" -lt 1200 ]; do sleep 0.05; i=$((i+1)); done +exit 0"; + +/// A stopped supervisor records its stop under the per-phase lock, waiting +/// up to 11 s for it, before it sends the final `stopped` line, and its kill +/// of an agent that ignores SIGTERM first takes the 3 s SIGTERM grace. So +/// `stop`'s wait for that line must cover both: here this test holds the +/// lock for 12.5 s after the shutdown is sent, past an 11 s wait and within +/// the supervisor's 14 s. A wait too short reads the still-answering +/// supervisor as one that "did not answer the shutdown" although it stopped. +#[test] +fn stop_waits_out_a_supervisor_whose_stop_record_waits_on_the_lock() { + const HOLD_AFTER_SHUTDOWN: Duration = Duration::from_millis(12_500); + let run = LiveRun::start_with(PhaseId::new(156), IGNORES_TERM); + std::fs::write(&run.trigger, "").unwrap(); + assert!( + holds_within(Duration::from_secs(10), || run + .root + .join(".git/devflow-test-ignores-term") + .exists()), + "precondition: the agent ignores SIGTERM" + ); + let lock = devflow_core::lock::acquire(&run.root, run.phase).expect("hold the phase lock"); + let stop = common::hermetic_devflow(&run.home) + .args(["stop", "--phase", &run.phase.to_string()]) + .current_dir(&run.root) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .expect("spawn devflow stop"); + std::thread::sleep(HOLD_AFTER_SHUTDOWN); + drop(lock); + let output = stop.wait_with_output().expect("wait for devflow stop"); + let text = describe(&output); + assert!( + output.status.success(), + "stop must report the stop that happened: {text}" + ); + assert!( + String::from_utf8_lossy(&output.stdout).contains(&format!( + "stopped phase {} through its supervisor", + run.phase + )), + "{text}" + ); + assert!( + devflow_core::workflow::load_state(&run.root, run.phase) + .expect("the state loads") + .stopped, + "the stop is recorded: {text}" + ); +} + +/// V-17 (999.138): a live run whose agent holds no per-phase lock is ended +/// through its supervisor's socket: the agent group is gone, the exit file +/// reads 143, no next stage is launched, the socket is Gone, and the state +/// records the stop. +#[test] +fn stop_ends_a_lock_free_live_run_through_the_socket() { + let run = LiveRun::start(PhaseId::new(140)); + assert!( + matches!( + devflow_core::lock::holder_status(&run.root, run.phase), + devflow_core::lock::HolderStatus::NoHolder + ), + "precondition: the running agent holds no per-phase lock" + ); + assert!( + matches!(run.probe(), Liveness::Alive(_)), + "precondition: the run is ALIVE" + ); + let pgid = run.record.agent_pgid; + assert!( + !group_members(pgid).is_empty(), + "precondition: the agent group is running" + ); + + let output = run.stop(&[]); + let text = describe(&output); + assert!(output.status.success(), "stop must succeed: {text}"); + assert!( + String::from_utf8_lossy(&output.stdout).contains(&format!( + "stopped phase {} through its supervisor", + run.phase + )), + "stop says it stopped the phase through its supervisor: {text}" + ); + assert!( + holds_within(Duration::from_secs(5), || group_members(pgid).is_empty()), + "the agent group {pgid} is gone" + ); + let exit = std::fs::read_to_string(devflow_core::agent_result::exit_code_path( + &run.root, run.phase, + )) + .expect("the supervisor records the agent's exit code"); + assert_eq!(exit.trim(), "143", "the agent ended by the stop"); + std::thread::sleep(Duration::from_secs(1)); + assert_eq!( + stage_launched_count(&run.root, run.phase), + 1, + "no next stage is launched after the stop" + ); + assert_eq!(run.probe(), Liveness::Gone, "the socket is gone"); + let state = devflow_core::workflow::load_state(&run.root, run.phase).unwrap(); + assert!(state.stopped, "state.stopped after the stop"); + assert!( + state + .stop_reason + .as_deref() + .is_some_and(|reason| reason.contains("devflow stop")), + "the stop reason names `devflow stop`: {:?}", + state.stop_reason + ); +} + +/// The V-17 control: the same fixture with no `stop`, whose agent is +/// released by its trigger, completes and advances, so a following +/// `stage_launched` appears. V-17's "no next stage" is therefore the stop's +/// doing, not a fixture that never advances. +#[test] +fn without_stop_a_lock_free_run_completes_and_advances() { + let run = LiveRun::start(PhaseId::new(141)); + assert_eq!(stage_launched_count(&run.root, run.phase), 1); + std::fs::write(&run.trigger, "go\n").unwrap(); + assert!( + holds_within(RUN_WAIT, || stage_launched_count(&run.root, run.phase) >= 2), + "a released agent must complete and launch the next stage:\n{}", + std::fs::read_to_string(devflow_core::events::events_path(&run.root)).unwrap_or_default() + ); +} + +/// D-09's repair verb: a live run whose state file will not load is ended +/// through its one sidecar-found socket. `stop` exits 0, says the stop was +/// not recorded because the state does not load (naming the file), and +/// leaves the state bytes unchanged. V-17 is its loadable-state control. +#[test] +fn stop_ends_a_live_run_whose_state_will_not_load() { + let run = LiveRun::start(PhaseId::new(142)); + let pgid = run.record.agent_pgid; + assert!( + !group_members(pgid).is_empty(), + "precondition: agent running" + ); + let state_file = run.state_file(); + std::fs::write(&state_file, b"{ this is not json").unwrap(); + let before = std::fs::read(&state_file).unwrap(); + + let output = run.stop(&[]); + let text = describe(&output); + assert!(output.status.success(), "stop must succeed: {text}"); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + stdout.contains(&format!( + "stopped phase {} through its supervisor", + run.phase + )), + "stop says it stopped the phase: {text}" + ); + assert!( + stdout.contains(&state_file.display().to_string()) + && stdout.contains("does not load") + && stdout.contains("was not recorded"), + "stop says `stopped` was not recorded and names the state file: {text}" + ); + assert!( + holds_within(Duration::from_secs(5), || group_members(pgid).is_empty()), + "the agent group {pgid} is gone" + ); + assert_eq!(run.probe(), Liveness::Gone, "the socket is gone"); + assert_eq!( + std::fs::read(&state_file).unwrap(), + before, + "the state that will not load is left byte for byte" + ); +} + +/// SIGKILL a run's supervisor by the pid in its sidecar, after confirming +/// that pid is still the recorded process, and return once it is gone and +/// its socket probes STALE. Returns the sidecar. +fn kill_supervisor(run: &LiveRun) -> supervisor::Sidecar { + let sidecar = supervisor::read_sidecar(&run.record.socket_path.with_extension("json")) + .expect("read the sidecar"); + let pid = sidecar.supervisor_pid; + assert!( + devflow_core::agent::is_same_process(pid, sidecar.supervisor_start_time), + "the sidecar's supervisor {pid} is the recorded process" + ); + let signed = libc::pid_t::try_from(pid).expect("pid fits pid_t"); + assert!(signed > 1, "refusing to signal pid {signed}"); + // SAFETY: `signed > 1` names the supervisor just identity-checked. + assert_eq!( + unsafe { libc::kill(signed, libc::SIGKILL) }, + 0, + "SIGKILL {pid}" + ); + assert!( + holds_within(RUN_WAIT, || !devflow_core::agent::agent_running(pid)), + "the supervisor {pid} is gone after SIGKILL" + ); + assert!( + holds_within(Duration::from_secs(10), || matches!( + supervisor::probe(&run.record.socket_path), + Ok(Liveness::Stale) + )), + "a SIGKILLed supervisor leaves its socket STALE" + ); + sidecar +} + +/// A STALE socket goes through the 50-09 backstop: the supervisor of a live +/// run is SIGKILLed, its agent group outlives it, and `stop` ends that group +/// through the validated pidfd path, reports the members it ended, removes +/// the socket and sidecar, and records `stopped`. +#[test] +fn stop_tears_down_a_stale_supervisor_through_the_backstop() { + let run = LiveRun::start(PhaseId::new(143)); + let pgid = run.record.agent_pgid; + kill_supervisor(&run); + assert!( + !group_members(pgid).is_empty(), + "precondition: the orphaned agent group {pgid} outlives its supervisor" + ); + + let output = run.stop(&[]); + let text = describe(&output); + assert!(output.status.success(), "stop must succeed: {text}"); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + stdout.contains("ended its agent group") && stdout.contains(&pgid.to_string()), + "stop reports the group and the members it ended: {text}" + ); + assert!( + holds_within(Duration::from_secs(5), || group_members(pgid).is_empty()), + "the agent group {pgid} is gone" + ); + let socket = &run.record.socket_path; + assert!( + !socket.exists() && !socket.with_extension("json").exists(), + "the socket and sidecar are removed" + ); + assert_eq!(run.probe(), Liveness::Gone); + assert!( + devflow_core::workflow::load_state(&run.root, run.phase) + .unwrap() + .stopped, + "the performed stop is recorded" + ); +} + +/// R-10: a supervise-mode run parked at a gate, holding the per-phase lock, +/// whose supervisor is SIGKILLed. `stop` writes no gate response (no live +/// waiter), tears the STALE socket down to GONE (the agent had already +/// exited, so nothing is signalled), names `devflow resume --phase N` (R-8), +/// and leaves the lock reclaimable. +#[test] +fn stop_after_a_supervisor_killed_at_a_gate_reaps_and_reports_gone() { + let run = LiveRun::start_with(PhaseId::new(144), FAILS_TO_A_GATE); + std::fs::write(&run.trigger, "go\n").unwrap(); + let gate = Gates::gate_path(&run.root, run.phase, Stage::Define); + assert!( + holds_within(RUN_WAIT, || gate.exists() + && matches!( + devflow_core::lock::holder_status(&run.root, run.phase), + devflow_core::lock::HolderStatus::Live { .. } + )), + "precondition: the run parks at the Define gate holding the lock:\n{}", + std::fs::read_to_string(devflow_core::events::events_path(&run.root)).unwrap_or_default() + ); + kill_supervisor(&run); + + let output = run.stop(&[]); + let text = describe(&output); + assert!(output.status.success(), "stop must succeed: {text}"); + assert!( + String::from_utf8_lossy(&output.stdout).contains(&format!( + "the phase is resumable: run `devflow resume --phase {}`", + run.phase + )), + "a teardown that signalled nothing says the phase is resumable (R-8): {text}" + ); + assert!( + !Gates::response_path(&run.root, run.phase, Stage::Define).exists(), + "no gate response is written without a live waiter" + ); + assert_eq!(run.probe(), Liveness::Gone, "the socket is torn down"); + assert!( + matches!( + devflow_core::lock::holder_status(&run.root, run.phase), + devflow_core::lock::HolderStatus::NoHolder + | devflow_core::lock::HolderStatus::Recycled { .. } + ), + "the dead holder's lock is reclaimable" + ); + drop( + devflow_core::lock::acquire(&run.root, run.phase) + .expect("the per-phase lock can be taken after the stop"), + ); +} + +/// R-6: a probe error is reported and never reaches the backstop. The +/// recorded socket path holds a regular file; the sidecar beside it records +/// a live test-owned group with its real identity, which a backstop run +/// would end. `stop` exits non-zero naming the path problem, the group is +/// untouched, and nothing is removed or recorded. +#[test] +fn stop_reports_a_probe_error_and_never_runs_the_backstop() { + use std::os::unix::process::CommandExt as _; + let home = common::HermeticHome::new(); + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().canonicalize().unwrap(); + let phase = PhaseId::new(145); + let sock_dir = home.cache_dir().join("sock"); + std::fs::create_dir_all(&sock_dir).unwrap(); + let socket = sock_dir.join("not-a-socket.sock"); + std::fs::write(&socket, "a regular file\n").unwrap(); + let mut group = Command::new("sleep") + .arg("60") + .process_group(0) + .spawn() + .expect("spawn a test-owned group"); + let pgid = group.id(); + let start = devflow_core::agent::process_start_time(pgid).expect("start time"); + let boot = supervisor::boot_id().expect("a readable boot id"); + let sidecar = serde_json::json!({ + "launch_id": "aaaa0000aaaa000e", + "project_root": root, + "phase": phase, + "stage": "code", + "agent": "claude", + "launch": "stream", + "agent_pid": pgid, + "agent_pgid": pgid, + "agent_start_time": start, + "boot_id": boot, + "supervisor_pid": 1, + "supervisor_start_time": 0, + "created_at": "2026-10-18T00:00:00Z", + }); + std::fs::write(socket.with_extension("json"), sidecar.to_string()).unwrap(); + let mut state = State::new(phase, AgentKind::Claude, Mode::Auto, root.clone()); + state.supervisor = Some(SupervisorRecord::new( + socket.clone(), + "aaaa0000aaaa000e".to_string(), + pgid, + start, + boot, + )); + devflow_core::workflow::save_state(&state).unwrap(); + let state_file = devflow_core::workflow::state_path(&root, phase); + let before = std::fs::read(&state_file).unwrap(); + + let output = common::hermetic_devflow(&home) + .args(["stop", "--phase", &phase.to_string(), "--root"]) + .arg(&root) + .output() + .expect("run devflow stop"); + let text = describe(&output); + assert!( + !output.status.success(), + "a probe error is a refusal: {text}" + ); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&socket.display().to_string()) && stderr.contains("not a socket"), + "the refusal names the path and its problem: {text}" + ); + assert!( + group.try_wait().expect("poll the group").is_none(), + "the group the sidecar records is untouched" + ); + assert!(socket.is_file() && socket.with_extension("json").is_file()); + assert_eq!(std::fs::read(&state_file).unwrap(), before); + common::signal_group(pgid, libc::SIGKILL); + let _ = group.wait(); +} + +/// What a planted orphan socket has beside it. +enum OrphanSidecar { + /// The live run's sidecar, round-tripped through serde with this launch + /// id and the identity of a test child already reaped. + Parsed, + /// A sidecar file that does not parse. + Broken, + /// No sidecar file. + Missing, +} + +/// Plant a STALE socket for the run's root and phase with launch id `id` +/// (a listener bound and dropped) and the sidecar `kind` describes. Returns +/// the socket path. +fn plant_stale_orphan(run: &LiveRun, id: &str, kind: OrphanSidecar) -> PathBuf { + let socket = + supervisor::SocketPath::for_launch(&run.home.cache_dir(), &run.root, run.phase, id) + .expect("the orphan socket path fits sun_path"); + drop(supervisor::publish(&socket).expect("publish the orphan socket")); + let path = socket.as_path().to_path_buf(); + assert!( + matches!(supervisor::probe(&path), Ok(Liveness::Stale)), + "the planted orphan {id} is STALE" + ); + match kind { + OrphanSidecar::Parsed => { + let mut reaped = Command::new("sleep") + .arg("60") + .spawn() + .expect("spawn a child to reap"); + let start = + devflow_core::agent::process_start_time(reaped.id()).expect("its start time"); + common::kill_and_reap(&mut reaped); + let live = supervisor::read_sidecar(&run.record.socket_path.with_extension("json")) + .expect("the live run's sidecar"); + let mut value = serde_json::to_value(&live).unwrap(); + value["launch_id"] = serde_json::json!(id); + value["agent_pid"] = serde_json::json!(reaped.id()); + value["agent_pgid"] = serde_json::json!(reaped.id()); + value["agent_start_time"] = serde_json::json!(start); + std::fs::write(socket.sidecar_path(), value.to_string()).unwrap(); + supervisor::read_sidecar(&socket.sidecar_path()).expect("the planted sidecar parses"); + } + OrphanSidecar::Broken => std::fs::write(socket.sidecar_path(), "{").unwrap(), + OrphanSidecar::Missing => {} + } + path +} + +/// The socket and sidecar of `socket` both exist. +fn launch_files_present(socket: &Path) -> bool { + socket.exists() && socket.with_extension("json").exists() +} + +/// The run's supervisor answers ping with its own launch id. +fn alive_as(run: &LiveRun) -> bool { + matches!(run.probe(), Liveness::Alive(reply) if reply.launch_id == run.record.launch_id) +} + +/// D-06 / T-50-53: `stop --launch ` tears down exactly the socket with +/// that launch id. A non-matching id and a phase mismatch change nothing; +/// the matching id removes the superseded orphan A and leaves the live run B +/// running, its state unstopped and no gate answered. +#[test] +fn stop_with_a_launch_id_tears_down_only_that_socket() { + let run = LiveRun::start(PhaseId::new(146)); + let orphan = plant_stale_orphan(&run, "0000000000000aaa", OrphanSidecar::Parsed); + let other_phase = PhaseId::new(147).to_string(); + + let nope = run.stop(&["--launch", "nope"]); + assert!(nope.status.success(), "{}", describe(&nope)); + assert!( + String::from_utf8_lossy(&nope.stdout).contains("nothing to stop"), + "a non-matching id says nothing matched: {}", + describe(&nope) + ); + assert!(launch_files_present(&orphan) && alive_as(&run)); + + let mismatch = common::hermetic_devflow(&run.home) + .args([ + "stop", + "--phase", + &other_phase, + "--launch", + "0000000000000aaa", + ]) + .current_dir(&run.root) + .output() + .expect("run devflow stop"); + assert!( + !mismatch.status.success(), + "a phase mismatch is refused: {}", + describe(&mismatch) + ); + assert!(launch_files_present(&orphan) && alive_as(&run)); + + let output = run.stop(&["--launch", "0000000000000aaa"]); + let text = describe(&output); + assert!(output.status.success(), "{text}"); + assert!( + String::from_utf8_lossy(&output.stdout).contains("launch 0000000000000aaa"), + "stop names the launch it tore down: {text}" + ); + assert!( + !orphan.exists() && !orphan.with_extension("json").exists(), + "A's socket and sidecar are removed: {text}" + ); + assert!(alive_as(&run), "B still answers with its own launch id"); + assert!( + !group_members(run.record.agent_pgid).is_empty(), + "B's agent group is alive" + ); + let state = devflow_core::workflow::load_state(&run.root, run.phase).unwrap(); + assert!(!state.stopped, "a targeted stop records nothing"); + assert!( + !Gates::response_path(&run.root, run.phase, Stage::Define).exists(), + "a targeted stop answers no gate" + ); +} + +/// The control for the targeted form: the same fixture with a bare +/// `devflow stop --phase N` stops the run the state records (B) and leaves +/// the superseded socket A, so the two forms address different sockets. +#[test] +fn bare_stop_addresses_the_state_record_and_leaves_a_superseded_socket() { + let run = LiveRun::start(PhaseId::new(148)); + let orphan = plant_stale_orphan(&run, "0000000000000aaa", OrphanSidecar::Parsed); + + let output = run.stop(&[]); + assert!(output.status.success(), "{}", describe(&output)); + assert_eq!(run.probe(), Liveness::Gone, "B is stopped"); + assert!( + devflow_core::workflow::load_state(&run.root, run.phase) + .unwrap() + .stopped + ); + assert!( + launch_files_present(&orphan), + "the superseded socket A is untouched" + ); +} + +/// D-06: `recover --clean` is a file cleaner and refuses a run whose +/// supervisor is ALIVE, deleting nothing. +#[test] +fn recover_clean_refuses_while_the_supervisor_is_alive() { + let run = LiveRun::start(PhaseId::new(149)); + let gate = + devflow_core::gates::Gates::write_gate(&run.root, run.phase, Stage::Define, "planted gate") + .expect("plant a gate file"); + let gate_bytes = std::fs::read(&gate).unwrap(); + let state_bytes = std::fs::read(run.state_file()).unwrap(); + + let output = common::hermetic_devflow(&run.home) + .args(["recover", "--clean", "--phase", &run.phase.to_string()]) + .current_dir(&run.root) + .output() + .expect("run devflow recover"); + let text = describe(&output); + assert!(!output.status.success(), "a live run is refused: {text}"); + assert!( + String::from_utf8_lossy(&output.stderr) + .contains(&format!("devflow stop --phase {}", run.phase)), + "the refusal names the stop verb: {text}" + ); + assert_eq!(std::fs::read(run.state_file()).unwrap(), state_bytes); + assert_eq!(std::fs::read(&gate).unwrap(), gate_bytes); + assert!(alive_as(&run), "the run is untouched"); +} + +/// The `devflow stop ... --launch ` line `stop` printed for `id`, as +/// arguments after the program name. +fn printed_launch_line(output: &std::process::Output, id: &str) -> Vec { + let stderr = String::from_utf8_lossy(&output.stderr); + let wanted = format!("--launch {id}"); + let line = stderr + .lines() + .find(|line| line.trim_end().ends_with(&wanted)) + .unwrap_or_else(|| panic!("no printed line ends with `{wanted}`:\n{stderr}")); + let command = &line[line + .find("devflow stop") + .expect("the line holds the command")..]; + command + .split_whitespace() + .skip(1) + .map(str::to_string) + .collect() +} + +/// T-50-65: several live sockets of one root and phase with a state file +/// that does not load are a refusal listing each `--launch` form, acting on +/// neither. The printed line for B, run as printed, then stops B and leaves +/// the corrupt state bytes unchanged. +#[test] +fn stop_refuses_several_sockets_when_the_state_will_not_load() { + let run = LiveRun::start(PhaseId::new(150)); + let orphan = plant_stale_orphan(&run, "0000000000000aaa", OrphanSidecar::Parsed); + std::fs::write(run.state_file(), b"{ this is not json").unwrap(); + let corrupt = std::fs::read(run.state_file()).unwrap(); + let b_id = run.record.launch_id.clone(); + + let refused = run.stop(&[]); + let text = describe(&refused); + assert!(!refused.status.success(), "several sockets: {text}"); + let stderr = String::from_utf8_lossy(&refused.stderr); + for id in [b_id.as_str(), "0000000000000aaa"] { + assert!( + stderr.contains(&format!("devflow stop --phase {} --launch {id}", run.phase)), + "the refusal lists the --launch form for {id}: {text}" + ); + } + assert!(alive_as(&run) && launch_files_present(&orphan), "{text}"); + + let args = printed_launch_line(&refused, &b_id); + let output = common::hermetic_devflow(&run.home) + .args(&args) + .current_dir(&run.root) + .output() + .expect("run the printed line"); + assert!( + output.status.success(), + "the printed repair works: {args:?}\n{}", + describe(&output) + ); + assert_eq!(run.probe(), Liveness::Gone, "B is stopped"); + assert!(launch_files_present(&orphan), "A is untouched"); + assert_eq!(std::fs::read(run.state_file()).unwrap(), corrupt); +} + +/// RV-05b (V-39): with both the state and B's sidecar unparseable, the +/// refusal names B by the launch id in its file name; a non-matching id and +/// a wrong phase change nothing; the printed `--launch` line stops B and its +/// agent group, leaving A and the corrupt state bytes alone. +#[test] +fn stop_with_a_launch_id_stops_a_live_socket_whose_sidecar_will_not_parse() { + let run = LiveRun::start(PhaseId::new(151)); + let orphan = plant_stale_orphan(&run, "0000000000000aaa", OrphanSidecar::Parsed); + let sidecar_path = run.record.socket_path.with_extension("json"); + let sidecar = supervisor::read_sidecar(&sidecar_path).expect("B's sidecar"); + let (pgid, start) = (sidecar.agent_pgid, sidecar.agent_start_time); + let b_id = run.record.launch_id.clone(); + assert!( + run.record + .socket_path + .file_stem() + .and_then(std::ffi::OsStr::to_str) + .is_some_and(|stem| stem.ends_with(&format!("-{b_id}"))), + "B's launch id is the last segment of its file name" + ); + std::fs::write(run.state_file(), b"{ this is not json").unwrap(); + std::fs::write(&sidecar_path, b"{ nor is this").unwrap(); + let corrupt = std::fs::read(run.state_file()).unwrap(); + + let refused = run.stop(&[]); + let text = describe(&refused); + assert!(!refused.status.success(), "{text}"); + let stderr = String::from_utf8_lossy(&refused.stderr); + let b_line = format!("devflow stop --phase {} --launch {b_id}", run.phase); + assert!( + stderr + .lines() + .any(|line| line.contains(&b_line) && line.contains("sidecar does not parse")), + "B is listed by its file name's launch id, marked unparseable: {text}" + ); + assert!( + stderr.contains(&format!( + "devflow stop --phase {} --launch 0000000000000aaa", + run.phase + )), + "{text}" + ); + assert!(alive_as(&run) && launch_files_present(&orphan)); + + let nope = run.stop(&["--launch", "nope"]); + assert!(nope.status.success(), "{}", describe(&nope)); + assert!(String::from_utf8_lossy(&nope.stdout).contains("nothing to stop")); + assert!(alive_as(&run), "a non-matching id leaves B running"); + + let wrong_phase = common::hermetic_devflow(&run.home) + .args([ + "stop", + "--phase", + &PhaseId::new(152).to_string(), + "--launch", + &b_id, + ]) + .current_dir(&run.root) + .output() + .expect("run devflow stop"); + assert!( + !wrong_phase.status.success(), + "B's file name names phase {}: {}", + run.phase, + describe(&wrong_phase) + ); + assert!(alive_as(&run), "a wrong phase leaves B running"); + + assert!( + devflow_core::agent::is_same_process(pgid, start), + "precondition: B's agent leader {pgid} is the recorded process" + ); + let args = printed_launch_line(&refused, &b_id); + let output = common::hermetic_devflow(&run.home) + .args(&args) + .current_dir(&run.root) + .output() + .expect("run the printed line"); + assert!(output.status.success(), "{args:?}\n{}", describe(&output)); + assert_eq!(run.probe(), Liveness::Gone, "B is stopped"); + assert!( + holds_within(Duration::from_secs(5), || group_members(pgid).is_empty()), + "B's agent group {pgid} is gone" + ); + assert!(launch_files_present(&orphan), "A is untouched"); + assert_eq!(std::fs::read(run.state_file()).unwrap(), corrupt); +} + +/// RV-05b (V-40): a STALE socket with no usable sidecar has no identity to +/// validate, so `--launch` refuses it with nothing signalled or removed, +/// for a sidecar that does not parse and for a missing one. Control: a STALE +/// socket whose sidecar parses, in the same fixture, is torn down. +#[test] +fn stop_with_a_launch_id_refuses_a_stale_socket_without_a_usable_sidecar() { + let run = LiveRun::start(PhaseId::new(153)); + let a = plant_stale_orphan(&run, "0000000000000aaa", OrphanSidecar::Parsed); + let c = plant_stale_orphan(&run, "0000000000000ccc", OrphanSidecar::Broken); + let d = plant_stale_orphan(&run, "0000000000000ddd", OrphanSidecar::Missing); + + for (id, socket, why) in [ + ("0000000000000ccc", &c, "does not parse"), + ("0000000000000ddd", &d, "missing"), + ] { + let output = run.stop(&["--launch", id]); + let text = describe(&output); + assert!(!output.status.success(), "{id} is refused: {text}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains("STALE") && stderr.contains(why) && stderr.contains("signals nothing"), + "{id}: the refusal says STALE, why the sidecar is unusable, and that nothing is \ + signalled: {text}" + ); + assert!(socket.exists(), "{id}'s socket is kept"); + assert!(alive_as(&run), "B is untouched"); + } + assert_eq!(std::fs::read(c.with_extension("json")).unwrap(), b"{"); + assert!(!d.with_extension("json").exists()); + + let output = run.stop(&["--launch", "0000000000000aaa"]); + assert!(output.status.success(), "{}", describe(&output)); + assert!(!a.exists() && !a.with_extension("json").exists()); + assert!(alive_as(&run), "B is untouched"); +} + +/// Plant a STALE socket and a parsed sidecar for `(project_root, phase, +/// id)` under `home`'s cache, recording a test child already reaped as its +/// agent group. Returns the socket path. +fn plant_unattached_orphan( + home: &common::HermeticHome, + project_root: &Path, + phase: PhaseId, + id: &str, +) -> PathBuf { + let socket = supervisor::SocketPath::for_launch(&home.cache_dir(), project_root, phase, id) + .expect("the orphan socket path fits sun_path"); + drop(supervisor::publish(&socket).expect("publish the orphan socket")); + let mut reaped = Command::new("sleep") + .arg("60") + .spawn() + .expect("spawn a child to reap"); + let start = devflow_core::agent::process_start_time(reaped.id()).expect("its start time"); + common::kill_and_reap(&mut reaped); + let sidecar = serde_json::json!({ + "launch_id": id, + "project_root": project_root, + "phase": phase, + "stage": "code", + "agent": "claude", + "launch": "stream", + "agent_pid": reaped.id(), + "agent_pgid": reaped.id(), + "agent_start_time": start, + "boot_id": supervisor::boot_id().expect("a readable boot id"), + "supervisor_pid": 1, + "supervisor_start_time": 0, + "created_at": "2026-10-18T00:00:00Z", + }); + std::fs::write(socket.sidecar_path(), sidecar.to_string()).unwrap(); + supervisor::read_sidecar(&socket.sidecar_path()).expect("the planted sidecar parses"); + socket.as_path().to_path_buf() +} + +/// 50-12's repair for a deleted root's orphan works by launch id alone, +/// from any directory, with no project state: `devflow doctor` prints +/// `devflow stop --phase N --launch ` for it, and that line, run from an +/// unrelated directory, tears it down. Control: an orphan whose sidecar +/// names another project that still exists is refused there and kept. +#[test] +fn stop_with_a_launch_id_reaches_a_deleted_roots_orphan_from_any_directory() { + let home = common::HermeticHome::new(); + let dirs = tempfile::tempdir().unwrap(); + let base = dirs.path().canonicalize().unwrap(); + let (gone_root, other_root, elsewhere) = ( + base.join("gone"), + base.join("other"), + base.join("elsewhere"), + ); + for dir in [&gone_root, &other_root, &elsewhere] { + std::fs::create_dir(dir).unwrap(); + } + let phase = PhaseId::new(154); + let deleted = plant_unattached_orphan(&home, &gone_root, phase, "0000000000000eee"); + let foreign = plant_unattached_orphan(&home, &other_root, phase, "0000000000000fff"); + std::fs::remove_dir(&gone_root).unwrap(); + let repair = format!("devflow stop --phase {phase} --launch 0000000000000eee"); + + let doctor = common::hermetic_devflow(&home) + .arg("doctor") + .current_dir(&elsewhere) + .output() + .expect("run devflow doctor"); + assert!( + String::from_utf8_lossy(&doctor.stdout).contains(&repair), + "doctor prints `{repair}` for the deleted root's orphan: {}", + describe(&doctor) + ); + + let refused = common::hermetic_devflow(&home) + .args([ + "stop", + "--phase", + &phase.to_string(), + "--launch", + "0000000000000fff", + ]) + .current_dir(&elsewhere) + .output() + .expect("run devflow stop"); + assert!( + !refused.status.success() + && String::from_utf8_lossy(&refused.stderr).contains("belongs to project"), + "another existing project's orphan is refused: {}", + describe(&refused) + ); + assert!(launch_files_present(&foreign), "the refused orphan is kept"); + + let args: Vec<&str> = repair.split_whitespace().skip(1).collect(); + let output = common::hermetic_devflow(&home) + .args(&args) + .current_dir(&elsewhere) + .output() + .expect("run the doctor's repair"); + assert!(output.status.success(), "{}", describe(&output)); + assert!( + !deleted.exists() && !deleted.with_extension("json").exists(), + "the deleted root's orphan is torn down: {}", + describe(&output) + ); +} + +/// `--launch` with the launch the state itself records runs the plain stop, +/// every arm included: the run is stopped through its socket and the plain +/// stop's own reason is recorded beside the supervisor's, which a targeted +/// stop (it writes no state) never adds. +#[test] +fn stop_with_the_recorded_launch_id_runs_the_plain_stop() { + let run = LiveRun::start(PhaseId::new(155)); + let output = run.stop(&["--launch", &run.record.launch_id]); + assert!(output.status.success(), "{}", describe(&output)); + assert_eq!(run.probe(), Liveness::Gone, "the recorded run is stopped"); + let state = devflow_core::workflow::load_state(&run.root, run.phase).unwrap(); + assert!(state.stopped, "the run is recorded stopped"); + assert!( + state + .stop_reason + .as_deref() + .is_some_and(|reason| reason.contains("stopped via `devflow stop`")), + "the plain stop's own reason is recorded: {:?}", + state.stop_reason + ); +} diff --git a/crates/devflow-cli/tests/supervisor_e2e.rs b/crates/devflow-cli/tests/supervisor_e2e.rs new file mode 100644 index 00000000..492cd807 --- /dev/null +++ b/crates/devflow-cli/tests/supervisor_e2e.rs @@ -0,0 +1,1316 @@ +//! End-to-end proof that a stage launch runs under an addressed Rust +//! supervisor (50-11, criteria 2 and 3). +//! +//! The launch path is `commands::start` → `launch_stage` → +//! `monitor::spawn_monitor` → the hidden `__monitor` subcommand, all inside +//! the `devflow` BINARY crate (no `lib.rs`), so integration tests cannot call +//! any of it in-process. Like `stop_e2e.rs`, this drives the real compiled +//! binary through `common::hermetic_devflow` and observes what an operator +//! can: the persisted state, the socket it names, and the files the +//! supervisor writes. +//! +//! Every `devflow` child gets a tempdir `HOME` and `DEVFLOW_CACHE_DIR`, so the +//! sockets these launches bind live under that tempdir, never under the +//! operator's real `~/.cache/devflow`. + +mod common; + +use devflow_core::phase_id::PhaseId; +use devflow_core::supervisor::{self, Liveness, ShutdownOutcome}; +use std::fs; +use std::path::Path; +use std::process::Output; +use std::time::{Duration, Instant}; + +/// How long a shutdown waits for the supervisor's final line. +const SHUTDOWN_WAIT: Duration = Duration::from_secs(30); + +/// Hermetic `git` for fixture setup (999.37), never a bare `Command::new("git")`. +fn git(root: &Path, args: &[&str]) { + let output = devflow_core::test_support::git_command(root) + .args(args) + .output() + .expect("spawn git"); + assert!( + output.status.success(), + "git {args:?} failed: {}", + String::from_utf8_lossy(&output.stderr) + ); +} + +/// A repository on `develop` with a roadmap entry for `phase`. +fn init_repo(root: &Path, phase: PhaseId) { + git(root, &["init", "-q"]); + git(root, &["config", "user.email", "devflow@example.com"]); + git(root, &["config", "user.name", "DevFlow Tests"]); + git(root, &["config", "commit.gpgsign", "false"]); + git(root, &["config", "core.hooksPath", "/dev/null"]); + git(root, &["checkout", "-q", "-b", "develop"]); + fs::create_dir_all(root.join(".planning")).unwrap(); + fs::write(root.join("README.md"), "base\n").unwrap(); + fs::write( + root.join(".planning/ROADMAP.md"), + format!("# Roadmap\n\n### Phase {phase}: supervisor fixture\n"), + ) + .unwrap(); + git(root, &["add", "README.md", ".planning/ROADMAP.md"]); + git(root, &["commit", "-q", "-m", "base"]); +} + +/// A fake `claude` that answers `--version` (the launch's health check), the +/// delivery canary's planted token (so a stream launch passes +/// `canary_gate`), and otherwise stays running until it is ended. Under the +/// capture shape its stdin is `/dev/null`, so the `read` yields nothing and +/// it goes straight to waiting. +const WAITING_CLAUDE: &str = r#"if [ "$1" = "--version" ]; then echo "2.1.220 (Claude Code)"; exit 0; fi +read -r turn +case "$turn" in + *DEVFLOW_DELIVERY_CANARY_*) + token=$(printf '%s' "$turn" | grep -o 'DEVFLOW_DELIVERY_CANARY_[0-9a-f]*' | head -1) + printf '{"type":"result","subtype":"success","is_error":false,"session_id":"s-fake","result":"%s"}\n' "$token" + exit 0 + ;; +esac +sleep 120"#; + +/// One launch shape as `devflow start` selects it. +#[derive(Debug, Clone, Copy)] +enum Shape { + /// The default Claude launch: the stream shape. + Stream, + /// `--legacy-claude-launch`: the capture shape. + Capture, +} + +impl Shape { + fn label(self) -> &'static str { + match self { + Shape::Stream => "stream", + Shape::Capture => "capture", + } + } + + fn start_flags(self) -> &'static [&'static str] { + match self { + Shape::Stream => &[], + Shape::Capture => &["--legacy-claude-launch"], + } + } +} + +/// `devflow start` for `phase` in supervise mode, with `fake_bin` first on PATH. +fn run_start( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &common::FakeBin, + shape: Shape, +) -> Output { + run_start_with(home, root, phase, fake_bin, shape, &[]) +} + +/// [`run_start`] with `extra` start flags (for example `--until plan`). +fn run_start_with( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &common::FakeBin, + shape: Shape, + extra: &[&str], +) -> Output { + let existing = std::env::var_os("PATH").unwrap_or_default(); + let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); + common::hermetic_devflow(home) + .args([ + "start", + "--phase", + &phase.to_string(), + "--agent", + "claude", + "--mode", + "supervise", + "--no-worktree", + ]) + .args(shape.start_flags()) + .args(extra) + .arg(root) + .current_dir(root) + .env("PATH", path) + .output() + .expect("run devflow start") +} + +/// Criterion 3 for one shape: after `devflow start` returns, the state names +/// a socket under the hermetic cache dir that answers ALIVE with the +/// shape's label, its sidecar names this root and phase, and a socket +/// shutdown stops the run and leaves the socket Gone. +fn start_publishes_for(shape: Shape, phase: PhaseId) { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + init_repo(&root, phase); + let fake_bin = common::fake_bin_dir(WAITING_CLAUDE); + let _reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + + // Control: nothing names a supervisor before the launch, so an Alive + // answer below can only come from this launch. + assert!( + devflow_core::workflow::load_state(&root, phase).is_err(), + "{shape:?}: no state may exist before start" + ); + + let output = run_start(&home, &root, phase, &fake_bin, shape); + assert!( + output.status.success(), + "{shape:?}: start failed\nstdout: {}\nstderr: {}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + + let state = devflow_core::workflow::load_state(&root, phase).expect("start saved state"); + let record = state + .supervisor + .clone() + .unwrap_or_else(|| panic!("{shape:?}: start must record state.supervisor: {state:?}")); + let socket = record.socket_path.clone(); + let sock_dir = home.cache_dir().join("sock"); + assert!( + socket.starts_with(&sock_dir), + "{shape:?}: socket {} must be under the child's cache dir {}", + socket.display(), + sock_dir.display() + ); + assert!( + !socket.starts_with(&root), + "{shape:?}: the socket must not live in the project tree" + ); + assert!( + socket.as_os_str().len() <= 107, + "{shape:?}: socket path is {} bytes", + socket.as_os_str().len() + ); + let reply = match supervisor::probe(&socket).expect("probe the recorded socket") { + Liveness::Alive(reply) => reply, + other => panic!("{shape:?}: the recorded socket must answer ALIVE, got {other:?}"), + }; + assert_eq!(reply.launch, shape.label(), "{shape:?}: launch label"); + assert_eq!(reply.launch_id, record.launch_id, "{shape:?}: launch id"); + assert_eq!(reply.agent_pgid, record.agent_pgid, "{shape:?}: agent pgid"); + + let sidecar_path = socket.with_extension("json"); + let sidecar = supervisor::read_sidecar(&sidecar_path).expect("read the sidecar"); + assert_eq!(sidecar.project_root, root, "{shape:?}: sidecar root"); + assert_eq!(sidecar.phase, phase, "{shape:?}: sidecar phase"); + assert_eq!(sidecar.launch, shape.label(), "{shape:?}: sidecar launch"); + assert_eq!( + sidecar.supervisor_pid, reply.supervisor_pid, + "{shape:?}: the recorded socket answers for the supervisor its sidecar names" + ); + + match supervisor::request_shutdown(&socket, SHUTDOWN_WAIT).expect("shutdown the run") { + ShutdownOutcome::Stopped { .. } => {} + other => panic!("{shape:?}: shutdown must stop the run, got {other:?}"), + } + assert_eq!( + supervisor::probe(&socket).expect("probe after shutdown"), + Liveness::Gone, + "{shape:?}: the socket is gone once the supervisor stopped" + ); +} + +#[test] +fn start_publishes_a_live_supervisor_socket_for_each_launch_shape() { + start_publishes_for(Shape::Stream, PhaseId::new(61)); + start_publishes_for(Shape::Capture, PhaseId::new(62)); +} + +/// How long the tests below wait for a supervisor to act on a signal, or a +/// stage chain to reach its stop point. +const RUN_WAIT: Duration = Duration::from_secs(60); + +/// The stop reason `run_monitor` records for a run its supervisor stopped. +const SUPERVISOR_STOP_REASON: &str = + "stopped through the supervisor (socket shutdown or SIGTERM/SIGINT)"; + +/// A fake `claude` that answers `--version` and the canary like +/// [`WAITING_CLAUDE`], commits one empty commit as its "work", prints the +/// success marker and exits 0 on its own. +const FINISHING_CLAUDE: &str = r#"if [ "$1" = "--version" ]; then echo "2.1.220 (Claude Code)"; exit 0; fi +read -r turn +case "$turn" in + *DEVFLOW_DELIVERY_CANARY_*) + token=$(printf '%s' "$turn" | grep -o 'DEVFLOW_DELIVERY_CANARY_[0-9a-f]*' | head -1) + printf '{"type":"result","subtype":"success","is_error":false,"session_id":"s-fake","result":"%s"}\n' "$token" + exit 0 + ;; +esac +git commit -q --allow-empty -m "fake stage work" +printf 'work done\nDEVFLOW_RESULT: {"status":"success","commits":1}\n' +exit 0"#; + +/// The `stage_launched` events recorded for `phase`. +fn stage_launched_count(root: &Path, phase: PhaseId) -> usize { + fs::read_to_string(devflow_core::events::events_path(root)) + .unwrap_or_default() + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .filter(|event| { + phase.matches_json(event.get("phase")) + && event["event"].as_str() == Some("stage_launched") + }) + .count() +} + +/// The pids of the live members of process group `pgid`. Zombies do not +/// count: a CI job container's PID 1 never reaps an orphaned one +/// (`test_support::live_group_members`). +fn group_members(pgid: u32) -> Vec { + devflow_core::test_support::live_group_members(pgid) +} + +/// Poll `done` every 25 ms until it holds or `within` elapses. +fn holds_within(within: Duration, mut done: impl FnMut() -> bool) -> bool { + let deadline = Instant::now() + within; + loop { + if done() { + return true; + } + if Instant::now() >= deadline { + return false; + } + std::thread::sleep(Duration::from_millis(25)); + } +} + +/// D-05 at the CLI level (999.134) for one shape: SIGTERM to the live +/// supervisor named in its sidecar ends the run. The agent group is gone, +/// `state.stopped` carries the supervisor stop reason, no further +/// `stage_launched` is recorded, and the socket is Gone. +fn sigterm_stops_for(shape: Shape, phase: PhaseId) { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + init_repo(&root, phase); + let fake_bin = common::fake_bin_dir(WAITING_CLAUDE); + let _reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + + let output = run_start(&home, &root, phase, &fake_bin, shape); + assert!( + output.status.success(), + "{shape:?}: start failed\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + let record = devflow_core::workflow::load_state(&root, phase) + .expect("start saved state") + .supervisor + .unwrap_or_else(|| panic!("{shape:?}: start must record state.supervisor")); + let socket = record.socket_path.clone(); + let sidecar = supervisor::read_sidecar(&socket.with_extension("json")).expect("sidecar"); + assert_eq!( + stage_launched_count(&root, phase), + 1, + "{shape:?}: one launch" + ); + assert!( + !group_members(record.agent_pgid).is_empty(), + "{shape:?}: the agent group is running before the signal" + ); + + // The pid to signal comes from the sidecar and is confirmed against its + // recorded start time first, so a recycled pid is never signalled. + let pid = sidecar.supervisor_pid; + assert!( + devflow_core::agent::is_same_process(pid, sidecar.supervisor_start_time), + "{shape:?}: the sidecar's supervisor {pid} is the recorded process" + ); + let signed = libc::pid_t::try_from(pid).expect("pid fits pid_t"); + assert!(signed > 1, "refusing to signal pid {signed}"); + // SAFETY: `signed > 1` names the supervisor just identity-checked. + assert_eq!( + unsafe { libc::kill(signed, libc::SIGTERM) }, + 0, + "SIGTERM {pid}" + ); + + assert!( + holds_within(RUN_WAIT, || !devflow_core::agent::agent_running(pid)), + "{shape:?}: the supervisor {pid} exits after SIGTERM" + ); + assert_eq!( + supervisor::probe(&socket).expect("probe after the signal"), + Liveness::Gone, + "{shape:?}: the socket is gone" + ); + assert_eq!( + group_members(record.agent_pgid), + Vec::::new(), + "{shape:?}: the agent group {} is gone", + record.agent_pgid + ); + let state = devflow_core::workflow::load_state(&root, phase).expect("reload state"); + assert!(state.stopped, "{shape:?}: state.stopped after the signal"); + assert_eq!( + state.stop_reason.as_deref(), + Some(SUPERVISOR_STOP_REASON), + "{shape:?}: the supervisor stop reason" + ); + assert_eq!( + stage_launched_count(&root, phase), + 1, + "{shape:?}: no stage_launched after the signal" + ); +} + +/// The control for [`sigterm_stops_for`]: the same launch, whose fake agent +/// exits 0 on its own, advances in-process and launches the next stage, so a +/// following `stage_launched` names Plan (the launched stage was passed to the +/// advance). `--until plan` stops the chain at the Plan boundary. +fn finishing_run_advances_for(shape: Shape, phase: PhaseId) { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + init_repo(&root, phase); + let fake_bin = common::fake_bin_dir(FINISHING_CLAUDE); + let _reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + + let output = run_start_with(&home, &root, phase, &fake_bin, shape, &["--until", "plan"]); + assert!( + output.status.success(), + "{shape:?}: start failed\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!( + holds_within(RUN_WAIT, || stage_launched_count(&root, phase) >= 2), + "{shape:?} control: a completed agent must advance and launch the next stage\n{}", + fs::read_to_string(devflow_core::events::events_path(&root)).unwrap_or_default() + ); + let events = fs::read_to_string(devflow_core::events::events_path(&root)).unwrap(); + let launched: Vec = events + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .filter(|event| event["event"].as_str() == Some("stage_launched")) + .collect(); + assert_eq!(launched[0]["stage"], "define", "{shape:?}: {launched:?}"); + assert_eq!(launched[1]["stage"], "plan", "{shape:?}: {launched:?}"); + assert_eq!( + launched[1]["launch"], + shape.label(), + "{shape:?}: {launched:?}" + ); + assert!( + holds_within(RUN_WAIT, || devflow_core::workflow::load_state( + &root, phase + ) + .is_ok_and(|state| state.stopped)), + "{shape:?} control: the chain stops at --until plan" + ); +} + +#[test] +fn sigterm_to_a_live_supervisor_ends_the_run_without_advancing() { + sigterm_stops_for(Shape::Stream, PhaseId::new(63)); + sigterm_stops_for(Shape::Capture, PhaseId::new(64)); + finishing_run_advances_for(Shape::Stream, PhaseId::new(65)); + finishing_run_advances_for(Shape::Capture, PhaseId::new(66)); +} + +/// Re-homed from `devflow-core/tests/monitor_e2e.rs` (50-11): a library test +/// binary cannot re-exec `__monitor`, so the capture-result check runs on the +/// real binary. A capture-shape run whose fake agent prints a +/// `DEVFLOW_RESULT` marker and exits 0: Layers 1–2 evaluate to Success from +/// the files the supervisor wrote, and the supervisor's own advance recorded +/// the same verdict. +#[test] +fn capture_supervisor_owns_a_fake_agent_and_records_devflow_result() { + let phase = PhaseId::new(67); + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + init_repo(&root, phase); + let fake_bin = common::fake_bin_dir(FINISHING_CLAUDE); + let _reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + + let output = run_start_with( + &home, + &root, + phase, + &fake_bin, + Shape::Capture, + &["--until", "define"], + ); + assert!( + output.status.success(), + "start failed\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!( + holds_within(RUN_WAIT, || devflow_core::workflow::load_state( + &root, phase + ) + .is_ok_and(|state| state.stopped)), + "the run stops at --until define once its supervisor advanced" + ); + + let agent_pid: u32 = + fs::read_to_string(devflow_core::agent_result::agent_pid_path(&root, phase)) + .expect("the supervisor records the agent pid") + .trim() + .parse() + .expect("a pid"); + assert!(agent_pid > 0); + let stdout = fs::read_to_string(devflow_core::agent_result::stdout_path(&root, phase)).unwrap(); + assert!( + stdout.contains("DEVFLOW_RESULT"), + "captured stdout missing marker: {stdout:?}" + ); + let exit = fs::read_to_string(devflow_core::agent_result::exit_code_path(&root, phase)) + .expect("the supervisor records the exit code"); + assert_eq!(exit.trim(), "0"); + + let state = devflow_core::workflow::load_state(&root, phase).unwrap(); + let git_flow = devflow_core::config::git_flow_for_run(&root, state.base_branch.as_deref()); + let result = devflow_core::agent_result::evaluate_agent_result(&root, &state, &git_flow) + .expect("evaluate the supervisor's files"); + assert_eq!( + result.status, + devflow_core::agent_result::AgentStatus::Success, + "{result:?}" + ); + let events = fs::read_to_string(devflow_core::events::events_path(&root)).unwrap(); + assert!( + events + .lines() + .any(|line| line.contains("\"advance_evaluated\"") + && line.contains("\"status\":\"success\"")), + "the supervisor's in-process advance evaluated the stage as success:\n{events}" + ); +} + +/// `devflow status` for `root`, as an operator runs it; its stdout. +fn run_status(home: &common::HermeticHome, root: &Path) -> String { + let output = common::hermetic_devflow(home) + .arg("status") + .arg(root) + .current_dir(root) + .output() + .expect("run devflow status"); + assert!( + output.status.success(), + "status failed\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8_lossy(&output.stdout).into_owned() +} + +/// 50-12 criterion 1: `devflow status` answers ALIVE, STALE and GONE from the +/// recorded socket, and each non-ALIVE answer names the verb that repairs it. +/// The run is made STALE by a SIGKILL of its supervisor (the pid comes from +/// the sidecar and is identity-checked first), and GONE by removing the stale +/// socket file. The reaper ends the agent group the SIGKILL orphaned. +#[test] +fn status_reports_each_liveness_state_with_its_repair_verb() { + let phase = PhaseId::new(68); + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + init_repo(&root, phase); + let fake_bin = common::fake_bin_dir(WAITING_CLAUDE); + let _reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + + let output = run_start(&home, &root, phase, &fake_bin, Shape::Stream); + assert!( + output.status.success(), + "start failed\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + let record = devflow_core::workflow::load_state(&root, phase) + .expect("start saved state") + .supervisor + .expect("start must record state.supervisor"); + let socket = record.socket_path.clone(); + let stop_verb = format!("devflow stop --phase {phase}"); + let resume_verb = format!("devflow resume --phase {phase}"); + + let alive = run_status(&home, &root); + assert!( + alive.contains("ALIVE") && alive.contains("owner: agent_running"), + "a live run reads ALIVE with its owner:\n{alive}" + ); + assert!( + alive.contains(&format!("supervisor: {}", socket.display())), + "status names the recorded socket:\n{alive}" + ); + assert!( + !alive.contains(&stop_verb) && !alive.contains(&resume_verb), + "an ALIVE run needs no repair:\n{alive}" + ); + + let sidecar = supervisor::read_sidecar(&socket.with_extension("json")).expect("sidecar"); + let pid = sidecar.supervisor_pid; + census_never_sees_a_live_supervisor(pid); + + assert!( + devflow_core::agent::is_same_process(pid, sidecar.supervisor_start_time), + "the sidecar's supervisor {pid} is the recorded process" + ); + let signed = libc::pid_t::try_from(pid).expect("pid fits pid_t"); + assert!(signed > 1, "refusing to signal pid {signed}"); + // SAFETY: `signed > 1` names the supervisor just identity-checked. + assert_eq!( + unsafe { libc::kill(signed, libc::SIGKILL) }, + 0, + "SIGKILL {pid}" + ); + assert!( + holds_within(RUN_WAIT, || !devflow_core::agent::agent_running(pid)), + "the supervisor {pid} is gone after SIGKILL" + ); + assert_eq!( + supervisor::probe(&socket).expect("probe the orphaned socket"), + Liveness::Stale, + "a SIGKILLed supervisor leaves its socket STALE" + ); + + let stale = run_status(&home, &root); + assert!(stale.contains("STALE"), "the run reads STALE:\n{stale}"); + let stop_at = stale + .find(&stop_verb) + .unwrap_or_else(|| panic!("STALE names `{stop_verb}`:\n{stale}")); + let resume_at = stale + .find(&resume_verb) + .unwrap_or_else(|| panic!("STALE names `{resume_verb}` after the stop:\n{stale}")); + assert!(stop_at < resume_at, "stop comes before resume:\n{stale}"); + assert!( + !stale.contains("ALIVE"), + "a STALE run is not ALIVE:\n{stale}" + ); + + fs::remove_file(&socket).expect("remove the stale socket"); + let gone = run_status(&home, &root); + assert!(gone.contains("GONE"), "the run reads GONE:\n{gone}"); + assert!( + gone.contains(&resume_verb), + "GONE while active names `{resume_verb}`:\n{gone}" + ); + assert!( + !gone.contains(&stop_verb), + "GONE names no stop, there is nothing left to stop:\n{gone}" + ); +} + +/// A 2.x-shaped monitor wrapper (`sh -c` with the trap line the 2.x script +/// carried), in its own process group, which drop SIGKILLs and reaps. +struct PlantedWrapper(std::process::Child); + +impl PlantedWrapper { + fn spawn() -> Self { + use std::os::unix::process::CommandExt as _; + // The trailing `:` keeps `sh` from exec-ing `sleep` in its place, so + // the wrapper's own argv stays visible in `/proc`. + let child = std::process::Command::new("sh") + .arg("-c") + .arg("cleanup() { :; }\ntrap cleanup TERM INT\nsleep 30\n:") + .process_group(0) + .spawn() + .expect("spawn the planted 2.x wrapper"); + Self(child) + } +} + +impl Drop for PlantedWrapper { + fn drop(&mut self) { + common::signal_group(self.0.id(), libc::SIGKILL); + let _ = self.0.wait(); + } +} + +/// 50-12 (T-50-52): the `/proc` stray census, run against a real 3.x +/// supervisor, never lists it. The control is a planted 2.x-shaped wrapper +/// that the same census run must list, so an empty census cannot pass. +fn census_never_sees_a_live_supervisor(supervisor_pid: u32) { + let wrapper = PlantedWrapper::spawn(); + let wrapper_pid = wrapper.0.id(); + let mut census = Vec::new(); + assert!( + holds_within(RUN_WAIT, || { + census = devflow_core::agent::discover_stray_devflow_processes(); + census.iter().any(|stray| stray.pid == wrapper_pid) + }), + "control: the census must list the planted 2.x wrapper {wrapper_pid}" + ); + assert!( + devflow_core::agent::agent_running(supervisor_pid), + "the supervisor {supervisor_pid} is alive while the census runs" + ); + assert!( + census.iter().all(|stray| stray.pid != supervisor_pid), + "the 2.x census must never list the 3.x supervisor {supervisor_pid}: {census:?}" + ); + drop(wrapper); +} + +// --------------------------------------------------------------------------- +// 50-15 (R-1, V-12): two `devflow start` launches racing for one phase yield +// exactly one live supervisor and one agent group. +// --------------------------------------------------------------------------- + +/// Bounds any gate a launched run parks at, as the 48-19 refusal tests do. +const RACE_GATE_TIMEOUT_SECS: &str = "15"; + +/// A spawned `devflow start` for `phase` with the arguments the 48-19 +/// live-run refusal tests use (supervise mode, `--no-worktree`, the capture +/// shape), so a loser fails only at the per-phase lock or at the live-run +/// guard, never at an earlier refusal. +fn spawn_start( + home: &common::HermeticHome, + root: &Path, + phase: PhaseId, + fake_bin: &common::FakeBin, +) -> std::process::Child { + let existing = std::env::var_os("PATH").unwrap_or_default(); + let path = format!("{}:{}", fake_bin.path.display(), existing.to_string_lossy()); + common::hermetic_devflow(home) + .args([ + "start", + "--phase", + &phase.to_string(), + "--agent", + "claude", + "--mode", + "supervise", + "--no-worktree", + "--legacy-claude-launch", + ]) + .arg(root) + .current_dir(root) + .env("PATH", path) + .env("DEVFLOW_GATE_TIMEOUT_SECS", RACE_GATE_TIMEOUT_SECS) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .expect("spawn devflow start") +} + +/// What a sweep of `home`'s socket directory sees: the ALIVE sockets, and the +/// distinct agent pgids its parsed sidecars name. +fn live_supervisors(home: &common::HermeticHome) -> (Vec, Vec) { + let entries = supervisor::sweep_in(&home.cache_dir()); + let alive = entries + .iter() + .filter(|entry| matches!(entry.liveness, Ok(Liveness::Alive(_)))) + .map(|entry| entry.socket_path.clone()) + .collect(); + let mut pgids: Vec = entries + .iter() + .filter_map(|entry| entry.sidecar.as_ref().map(|sidecar| sidecar.agent_pgid)) + .collect(); + pgids.sort_unstable(); + pgids.dedup(); + (alive, pgids) +} + +#[test] +fn two_concurrent_starts_yield_exactly_one_live_supervisor() { + let home = common::HermeticHome::new(); + let repo = tempfile::tempdir().unwrap(); + let root = repo.path().canonicalize().unwrap(); + let phase = PhaseId::new(63); + init_repo(&root, phase); + let fake_bin = common::fake_bin_dir(WAITING_CLAUDE); + let _reaper = common::LaunchedRunReaper::new(&root, phase, vec![]); + + let first = spawn_start(&home, &root, phase, &fake_bin); + let second = spawn_start(&home, &root, phase, &fake_bin); + let outputs = [ + first.wait_with_output().expect("wait for the first start"), + second + .wait_with_output() + .expect("wait for the second start"), + ]; + + let report = |output: &Output| { + format!( + "status {:?}\nstdout: {}\nstderr: {}", + output.status, + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ) + }; + let winners = outputs + .iter() + .filter(|output| output.status.success()) + .count(); + assert_eq!( + winners, + 1, + "exactly one racing start must succeed\nfirst: {}\nsecond: {}", + report(&outputs[0]), + report(&outputs[1]) + ); + let loser = outputs + .iter() + .find(|output| !output.status.success()) + .expect("one start lost"); + let stderr = String::from_utf8_lossy(&loser.stderr); + let lock_refusal = stderr.contains("holds the per-phase lock"); + let live_refusal = stderr.contains(&format!("devflow stop --phase {phase}")) + && stderr.contains("a run is still live"); + assert!( + lock_refusal || live_refusal, + "the loser must be refused at the lock or at the live-run guard, nothing else\n{}", + report(loser) + ); + println!( + "V-12 loser refusal: {}", + if lock_refusal { + "per-phase lock" + } else { + "live-supervisor guard" + } + ); + + let (alive, pgids) = live_supervisors(&home); + assert_eq!( + alive.len(), + 1, + "exactly one socket may probe ALIVE: {alive:?}\nloser: {}", + report(loser) + ); + assert_eq!( + pgids.len(), + 1, + "the sidecars must name exactly one agent group: {pgids:?}" + ); + assert!( + !group_members(pgids[0]).is_empty(), + "the one agent group {} must be live", + pgids[0] + ); +} + +#[test] +fn two_starts_for_different_phases_yield_two_live_supervisors() { + let home = common::HermeticHome::new(); + let repos = [tempfile::tempdir().unwrap(), tempfile::tempdir().unwrap()]; + let roots: Vec = repos + .iter() + .map(|repo| repo.path().canonicalize().unwrap()) + .collect(); + let phases = [PhaseId::new(64), PhaseId::new(65)]; + for (root, phase) in roots.iter().zip(phases) { + init_repo(root, phase); + } + let fake_bin = common::fake_bin_dir(WAITING_CLAUDE); + let _reapers: Vec = roots + .iter() + .zip(phases) + .map(|(root, phase)| common::LaunchedRunReaper::new(root, phase, vec![])) + .collect(); + + let first = spawn_start(&home, &roots[0], phases[0], &fake_bin); + let second = spawn_start(&home, &roots[1], phases[1], &fake_bin); + for (child, phase) in [first, second].into_iter().zip(phases) { + let output = child.wait_with_output().expect("wait for start"); + assert!( + output.status.success(), + "phase {phase}: a start with no rival for its phase must succeed\nstderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + } + + let (alive, pgids) = live_supervisors(&home); + assert_eq!( + alive.len(), + 2, + "the counting must see both live supervisors: {alive:?}" + ); + assert_eq!( + pgids.len(), + 2, + "the sidecars must name two distinct agent groups: {pgids:?}" + ); +} + +// ---- 50-26: WR-03 on the real `devflow __monitor` binary ----------------- +// +// A completed stream run whose agent left a process holding its stdout must +// reach its advance, and a `devflow stop` sent while it is `advancing` must +// end it at safe point 1. Each run is a `__monitor` started directly (no +// launcher), over a seeded Code-stage state that halts at `--until code`. + +/// A repository on `develop` with `feature/phase-NN` one commit ahead of it. +/// Named apart from [`init_repo`] (develop only, with a ROADMAP entry, used by +/// the `start` tests): the commit ahead is what lets Layer 2 classify the +/// Code run `Success`, so the advance reaches `transition`'s `--until` halt +/// (as `auto_chain_flag_e2e.rs`'s `init_repo` makes it). +fn init_repo_with_feature_branch(root: &Path, phase: PhaseId) { + git(root, &["init", "-q"]); + git(root, &["config", "user.email", "devflow@example.com"]); + git(root, &["config", "user.name", "DevFlow Tests"]); + git(root, &["config", "commit.gpgsign", "false"]); + git(root, &["config", "core.hooksPath", "/dev/null"]); + git(root, &["checkout", "-q", "-b", "develop"]); + fs::write(root.join("README.md"), "base\n").unwrap(); + git(root, &["add", "README.md"]); + git(root, &["commit", "-q", "-m", "base"]); + + let branch = format!("feature/phase-{padded}", padded = phase.padded()); + git(root, &["checkout", "-q", "-b", &branch]); + fs::write(root.join("work.txt"), "agent work\n").unwrap(); + git(root, &["add", "work.txt"]); + git(root, &["commit", "-q", "-m", "agent work"]); +} + +/// Mirrors `pipeline_launch::ADVANCE_STOP_REASON`, as +/// [`SUPERVISOR_STOP_REASON`] mirrors the supervisor's; `devflow stop` +/// appends its own clause after it. +const ADVANCE_STOP_REASON: &str = "stopped through the supervisor during advance"; + +/// The Claude top-level result line the held-stdout agent prints. +const HELD_RESULT_LINE: &str = r#"{"type":"result","subtype":"success","is_error":false,"session_id":"s-fake","result":"DEVFLOW_RESULT: {\"status\":\"success\"}"}"#; + +/// The holder record ` ` at `record`, when it exists and parses. +fn read_holder_record(record: &Path) -> Option<(u32, u64)> { + let text = fs::read_to_string(record).ok()?; + let mut fields = text.split_whitespace(); + let pid = fields.next()?.parse().ok()?; + let start = fields.next()?.parse().ok()?; + fields.next().is_none().then_some((pid, start)) +} + +/// A `devflow __monitor` started directly over a seeded Code-stage state +/// (`Mode::Supervise`, `stop_until = Some(Stage::Code)`), with production's +/// 120 s idle timeout, supervising `sh