test/e2e: replay the control-plane outage as a chaos scenario, and repair the harness three ways

[neochaotic]

The production incident was an ordering failure and nothing in the repository reproduced it. Scenario C proves a durable outcome record is recovered with the control plane up; the failure needed it down across the pods' termination, so that the restarted leader wakes to task instances that are running with stale heartbeats and pods that are terminal, and a reaper can judge them before the reconciler has read their records.

Scenario D injects C's fault on three parallel tasks and SIGKILLs the scheduler before they finish. Every step is fail-closed: a pod that exits 0 means the fault seam did not fire, and an attempt that settles during the outage means the outage never covered the report window. Scenarios are selectable with LEOFLOW_CHAOS_ONLY so D can be iterated without paying for the others' sleeps.

Measured on k3d against this branch, rebased onto main at d5cef2d (with #925, #926, #927 and #932 all in):

PASS D: all three pods terminated during the outage, records written, no report delivered
PASS D: all three attempts are still unsettled during the outage (states: 3 running)
PASS D: run recovered to SUCCESS 35s after the restart, all three settled from their durable records
PASS D: recovery (35s) beat the settling grace (180s)
PASS D: every attempt settled in place (max try_number=1)
PASS D: the reap pass ran inside the grace and was held (reap_settling_skip=1)
PASS D: the liveness valve never opened (reap_settling_valve_open=0)
PASS D: no reaper fired at all during recovery (pod_lost=0, agent_lost=0)

What this proves, stated precisely. The maintenance cycle's ordering: reconcile runs before reap inside one cycle, so by the time the reap pass is consulted the same cycle has already settled all three attempts from their records. That ordering is what broke in production.

What it does not prove, so the assertions are not oversold. It does not prove the settling gate refusing a real candidate: the skip is recorded once per tick before any candidate is listed, and the first tick lands one interval after boot, by which point the reconcile has already settled everything — build the same tree with SettlingGrace at 0 and every behavioural assertion here still passes, only that instrument goes red. Proving the gate itself needs a scenario that costs the reconciler a cycle so a candidate still exists at reap time. It also ends before the grace opens, so "no reaper fired" is scoped to that window and says nothing about the window in which the reapers are finally allowed to act. And it does not prove #915's presence rule or execution_timeout (#930).

Writing it surfaced three defects in the harness, none in production code, and review found four more in the assertions. All of them were the same failure: a test that had quietly stopped testing anything.

The fault seam had been undeliverable since 2026-08-31. Dispatch strips every LEOFLOW_-prefixed key an author declares, so a DAG cannot redirect the agent's credential exchange (#828, fixed in #829) — and the seam rode exactly that channel, so the hook was never installed and the task took the ordinary report path. Scenario C is fail-closed and would have caught it, but its message accused a stale agent image, the wrong cause and the wrong remedy. Both scenarios now carry the seam as an ENV in their own DAG image, which nothing in the pod spec overrides. Scenario C passes again, which is how the diagnosis was confirmed.

The setup ignored its own errors. A control plane that died at boot sailed past a fixed sleep 5 and surfaced minutes later as an unrelated-looking "no run id". The api is now waited for, and a failure prints both role logs and names the likely cause; the token mint and every push fail loudly, C's push included.

The harness was not re-runnable. A DAG version is immutable and keyed by the repository's describe output, and the cluster is recreated per run while the control-plane database is not — so editing a scenario's DAG body and re-running was rejected with 409, and would otherwise have risked exercising the previous body. Each setup now deregisters its DAG first.

The assertions themselves had three defects. The grace comparison could not fail: the success wait was 150s against a grace of 180, so a measured success was always below it and the past-the-grace branch was dead code. The unsettled precondition passed vacuously when its own read failed, verifying nothing inside a green run. And the Running check counted lines rather than distinct tasks. Every curl now carries a timeout, without which no deadline in the script means anything, and the four counters come from one snapshot with a loud failure when the metric is absent.

One behavioural note the scenario had to encode: its tasks must not return a value. The agent pushes a return value before writing the durable outcome record, and that push has no deadline of its own, so with the control plane down an agent with a return value blocks in the gRPC reconnect and its pod stays Running through the entire outage. That is correct behaviour and the opposite of what this scenario needs. It also means that after #897 a task finishing during an outage keeps its pod alive and retries, so killing the control plane alone no longer reproduces the field failure — the seam is what forces the pod to end inside the outage.

Test-only change, hence skip-changelog. Not gated in CI (destructive, about 12 minutes; D alone about 5).