fix(vm-agent): bind prompt-cancel grace watchdog to the cancelled attempt

[simple-agent-manager]

Summary

Fixes the VM agent's prompt-cancel grace watchdog, which killed the next prompt and failed its task (Prompt cancel grace elapsed after 5s). This is Fix Plan section A of idea 01M31M9G3T4SEWT9ZW1BM4QKZ3. The control-plane amplifier (section B: per-session delivery single-flight, one stop per urgent delivery) is a separate follow-up PR.

• Root cause: cancelPrompt(true) armed a 5 s timer keyed only by a numeric prompt ID and never disarmed it. If a follow-up prompt was accepted inside the window, the timer fired against it: promptAttemptForID fabricated an attempt for the stale ID and overwrote h.promptAttempt, and the stop was classified fatal_error because activePromptID had moved on. That killed the new prompt's agent and failed the task. This is a regression from PR feat: add durable session sleep and recovery for Claude Code and Codex #1785.
• Production impact (2026-08-30 → 09-29): 91 cancels; in 4 the next prompt started within 5 s, and all 4 were killed. The watchdog had 0 true positives. 3 of the 4 were steering messages killing the agent they were meant to steer.
• Fix:
  • The watchdog is armed with the exact *promptAttempt and selects on attempt.done, host teardown, or the timer.
  • Force-stop acts only while that attempt is current and non-terminal.
  • The fabricate-an-attempt branch is deleted.
  • A genuinely stuck requested cancel now finishes cancelled and restarts the agent through the existing intentional prompt-cancel stop. It never produces HostError / Agent prompt failed. If a concurrent restart already cleared the process, it defers to that owner.
  • Hard prompt timeouts keep the fatal path.
• Observability: Prompt cancel requested, ACP Prompt started/cancelled/completed and ACP prompt force-stopped now carry promptId. Control-plane prompts also carry deliveryId (AcceptPrompt gains a deliveryID arg). Force-stop adds currentPromptId and cancelRequested.
• Rule 18: the first commit (579d8f6ca) is a pure move that splits session_host.go from 1295 lines to 744 (cancel, attempt arbiter, settings, stderr and MCP builders get their own files).

Validation

• pnpm lint — N/A for Go-only change; gofmt -l clean and go vet ./... clean
• pnpm typecheck — N/A (no TypeScript changed)
• pnpm test — go test -race ./... for packages/vm-agent: all packages ok (CI runs the same command)
• Additional validation run: go test -race -count=10 ./internal/acp/ — the new tests passed every run. One pre-existing, unrelated debounce test flaked once (TestHarnessActivityReportCoalescesACPToolCallBursts) and is filed as tasks/backlog/2026-09-29-flaky-harness-activity-coalesce-test.md
• N/A: no sweep/cron/alarm candidate selection changed

Discrimination (rule 62):

• Restoring the pre-fix semantics (no attempt.done disarm; force-stop re-pointed at the current attempt) made TestCancelGraceWatchdogNeverTouchesTheNextPrompt/{ws,http} fail with prompt B completing fatal_error, the incident signature. It also made TestCancelGraceWatchdogDisarmsWhenAttemptSettles fail.
• Disabling the stuck-cancel settle branch made TestCancelGraceWatchdogSettlesAGenuinelyStuckCancel/{ws,http} report fatal_error.
• Removing the restart from the settle path made TestCancelGraceStuckViewerCancelRestartsAgentBackToReady time out, because the host never returns to Ready.
• The test-engineer reviewer reproduced the first two independently.

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — run 36530902316 on this branch, success
• Live app verified via Playwright — not with Playwright: it isn't installed in this workspace. I authenticated with the staging token via POST /api/auth/token-login and drove the same endpoints the app's chat UI calls (task submit, session cancel, session prompt, session messages). See Exceptions.
• Existing workflows confirmed working — task submission, node provisioning, workspace creation, agent start, LoadSession restart after cancel, and durable follow-up delivery all completed normally
• New feature/fix verified on staging — see evidence
• Infrastructure verification completed — fresh VM provisioned, healthy heartbeat, agent build confirmed, cleaned up
• Mobile and desktop verification notes added for UI changes — N/A: no UI changes

Staging Verification Evidence

• Fresh node 01M3NYCH7N2JK1QAACABAKF4NJ reported agent_version = 4ef45be85 (this branch) with healthy heartbeats. The test was a Claude Code VM task (01M3NYC8SHR8N7JFR0HY137SV1).
• I ran three HTTP control-plane cancels of live turns, each immediately followed by a durable follow-up. Cycle 2 hit the incident window:
  • prompt 10 was cancelled at 06:52:44.101;
  • follow-up prompt 11 started at 06:52:48.976, 4.875 s later, inside the 5 s grace;
  • pre-fix, the stale timer would have fired about 125 ms into prompt 11 (the 09-28 incident fired 158 ms in);
  • instead, prompt 11 ran for 17 s until the next deliberate cancel.
• Workspace logs in observability platform_errors: 0 warn/error rows, 0 ACP prompt force-stopped, and 0 grace or "did not settle" rows. The task finished in_progress / awaiting_followup, and the last follow-up answered PONG.
• The new identity fields are live, for example Prompt cancel requested {"deliveryId":"01M3NYZ8B19D7DBGQSCPJ521JT","promptId":10}.
• Cleanup: the session was stopped and the node deleted; /api/nodes returns [].

UI Compliance Checklist (Required for UI changes)

N/A: no UI changes.

UI Screenshot Evidence

N/A: no UI changes.

End-to-End Verification (Required for multi-component changes)

• Data flow traced from user input to final outcome with code path citations
• Capability test exercises the complete happy path across system boundaries
• All spec/doc assumptions about existing behavior verified against code
• Manual verification steps documented below

Data Flow Trace

1. Stop button → VM agent: apps/api/src/routes/chat-cancel.ts (POST /:sessionId/cancel) → cancelAgentSessionOnNode → VM handleCancelAgentSession (internal/server/workspaces.go) → SessionHost.CancelPromptFromControlPlane (internal/acp/session_host_cancel.go) → cancelPrompt(true) → watchPromptCancelGrace(attempt).
2. Browser WS cancel: Gateway.handleMessage session/cancel (gateway.go) → CancelPrompt → the same watchdog.
3. Follow-up: routes/chat-prompt-route.ts durable delivery → VM prompt-delivery handler → AcceptPrompt(..., deliveryID, observer) → beginPromptForDelivery → AcceptedPrompt.Run.
4. Grace deadline: triggerPromptForceStopIfStuck(attempt) (session_host_prompt_state.go) no-ops unless h.promptAttempt == attempt and the attempt is non-terminal. A stuck requested cancel goes to settleStuckPromptCancel → StopProcessForPromptCancel → monitorProcessExit restart (session_host_process.go).

Untested Gaps

• The ACP WebSocket session/cancel path is covered by Go tests through Gateway.handleMessage, but was not exercised on staging. Staging exercised the HTTP control-plane path, which is the one in all 4 production incidents.
• The HTTP handler itself (handleCancelAgentSession) is a thin IsPrompting() guard in front of CancelPromptFromControlPlane. The Go tests call the latter directly; staging exercised the full HTTP path.

Post-Mortem (Required for bug fix PRs)

What broke

A stop or urgent message sent to a busy agent could kill the agent's next turn and mark the task failed with "Prompt cancel grace elapsed after 5s". Claude Code restarts in about 4.2–6.2 s, so this was roughly a coin flip per urgent message.

Root cause

PR #1785 (00169b016, 2026-08-12) replaced the force-stop's activePromptID != promptID early return with an ID-based attempt lookup. That lookup fabricated an attempt for any unknown ID while a prompt was in flight, and the grace timer was never disarmed when its own attempt finished.

Class of bug

A watchdog keyed by a reusable, moving identity (a sequence ID interpreted against "current state") instead of the owned object it guards. Once the guarded thing ended, the watchdog outlived it and applied its verdict to a successor.

Why it wasn't caught

The tests hand-built promptInFlight=true with no real attempt, so they depended on the fabricate seam and never ran the "next prompt accepted inside the grace window" ordering. No test owned the timer, so the load-bearing midpoint was never reached (rule 62). Logs carried no prompt identity, so the 09-21 report could only guess at the cause.

Process fix included in this PR

• Tests now enter through the real WS gateway and control-plane AcceptPrompt / cancel paths, with an injectable grace timer that owns the ordering.
• Lifecycle logs carry prompt and delivery identity, so a recurrence can be diagnosed from platform_errors alone.
• No rule file changed: .claude/rules/62-tests-must-observe-the-real-trigger.md already covers this class, and the tests now comply with it.

Post-mortem file

tasks/archive/2026-09-29-bind-prompt-cancel-watchdog-to-attempt.md and idea 01M31M9G3T4SEWT9ZW1BM4QKZ3.

Specialist Review Evidence (Required for agent-authored PRs)

• All local reviewers completed and findings addressed before merge
• If any reviewer did NOT complete: needs-human-review label added and merge deferred to human — N/A: all completed

Reviewer	Status	Outcome
go-specialist	ADDRESSED	No concurrency, lock-ordering or goroutine-lifetime defects. MED-HIGH: stuck-cancel test didn't run the real monitor → added TestCancelGraceStuckViewerCancelRestartsAgentBackToReady (real monitorProcessExit → Ready → follow-up; discriminating). MED: nil-process settle → now defers to the in-flight restart, plus a test (4ef45be85). One observed full-suite failure coincided with the test-engineer's concurrent revert experiment; a clean re-run of 10× passed every cancel-grace test.
test-engineer	ADDRESSED	Independently reproduced both discrimination claims; -race -count=10 clean. LOW: cross-reference comment added.
task-completion-validator	PASS	Checks A–F pass; LOW: tick acceptance criteria (done).

CodeRabbit Review Evidence (Required for agent-authored PRs)

• coderabbit-review label applied after local review, staging, and CI gates passed
• The required wait completed after trusted review requests
• CodeRabbit submitted no review and therefore had no findings to resolve
• The observed skip/non-response is recorded below

CodeRabbit Notes

CodeRabbit was requested and given substantially more than the required 15-minute wait, but it did not review this PR.

• The label-triggered CodeRabbit run reported Review skipped / Bot user detected and submitted no review.
• The trusted owner-authored workflow requests at 07:09:41Z, 07:43:42Z, and 12:20:06Z also produced no submitted review, inline comment, review thread, or findings.
• The PR still has zero submitted reviews from CodeRabbit. Under the best-effort rule merged in PR docs: make CodeRabbit review best-effort instead of merge-blocking #2189 (219e1c05), this recorded skip satisfies the CodeRabbit gate; no waiver or needs-human-review label is required.

Exceptions (If any)

• Scope: "Live app verified via Playwright" checkbox.
• Rationale: this is a VM-agent-only change with no UI surface, and Playwright is not installed in this workspace. The live staging app was exercised through the identical authenticated HTTP endpoints the chat UI calls (token-login cookie session), on a real provisioned VM.
• Expiration: this PR only.

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

github.com/coder/acp-go-sdk@v0.13.5 source (client_gen.go Prompt, connection.go waitForResponse / sendMessage), read to confirm that Prompt returns on ctx cancel but first writes session/cancel with context.Background(). That write can block on a wedged agent's stdin, which is the real stuck-cancel case.

Codebase Impact Analysis

• packages/vm-agent/internal/acp: session_host_cancel.go, session_host_prompt_state.go, session_host_attempt.go, session_host_prompt.go, gateway.go (comment), and the split files.
• packages/vm-agent/internal/server/workspaces.go: passes deliveryID to AcceptPrompt.
• packages/vm-agent/internal/config/config.go: comment only.
• The control-plane task callback is unchanged: cancelled already maps to a cancellation, not failed (server.go makeTaskCompletionCallback).
• Rollout (rule 54): agent-only and additive. Old nodes keep the bug until they are replaced; the control-plane mitigation is in section B.

Documentation & Specs

• .claude/skills/env-reference/SKILL.md: ACP_PROMPT_CANCEL_GRACE_PERIOD semantics.
• Doc comments on DefaultPromptCancelGracePeriod, GatewayConfig.PromptCancelGracePeriod and config.ACPPromptCancelGrace.
• No public docs describe the watchdog.

Constitution & Risk Check

• Principle XI: the grace period stays configurable (ACP_PROMPT_CANCEL_GRACE_PERIOD); no new hardcoded values.
• Risk: a stuck cancel now restarts the agent instead of parking it in error. That is the same restart path the control-plane Stop already uses on every cancel.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: raphaeltm/simple-agent-manager/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 7bf7b7be-09be-45d1-9bd3-ced62bb543ba

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[raphaeltm]

@coderabbitai review

[raphaeltm]

@coderabbitai review

[raphaeltm]

@coderabbitai review

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud