This repository was archived by the owner on Oct 13, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 13, 2026. It is now read-only.
Prism jobs go silent: review changes orphan the controller, stall transfer races interrupt, cancel cannot finish #132
Description
Outcome
Prism jobs never go silent. Every job either keeps advancing or reaches a reported state the planner is woken for. A stalled worker moves to the next route, a review asking for changes goes back to a worker, and cancel works from Prism.
Evidence and causes (2026-09-27, Router 0.37.0, policy 2.10.0, T3 planner; jobs ao-27-t3-attribution-2 and -3)
1. A review asking for changes orphans the job (highest impact; since 0.36.0 / #126, commit 7984bd5)
_review_gatereturns{"action": "review-changes-requested"}(controller.py ~3057) after persisting the correction envelope.run_controller_processonly keeps looping for actions incontinuing(controller.py ~4204), and that tuple lacksreview-changes-requested. So the loop breaks with the job stillrunning.runningis not inTERMINAL_REPORT_STATUSES, so no terminal report is sent, the lease is released, and nothing ever relaunches the controller.- Observed on
-3:review_verdictrequest_changesat 19:01:29.896,controller_exitedat 19:01:29.905. Status stayedrunningwithowner_pidnull, and the planner was never woken. The user found the dispatcher thread idle 13 minutes later. - The same gap applies to
{"action": "resumed"}from the recovered-envelope path in_step_inner(~3890); not verified live. - Every job whose first review requests changes is affected.
2. Stall to route transfer races the interrupt, so the job blocks instead of moving route
- controller.py ~862-880 dispatches
thread.turn.interrupt, then immediately reads one snapshot. T3 applies the interrupt asynchronously. On-2the interrupt was requested at 18:03:07.781 and the turn becameinterruptedat 18:03:07.839 (T3projection_turns). Router's single read saw a live turn, soidle_confirmed=false, thenroute_transfer_abort_failedat 18:03:08.05, with 4 of 5 attempts unused. - Fix: poll the snapshot (for example every 0.5 s, up to about 15 s) until the turn is
completed,errororinterrupted. Treat a thread whose session isstoppedwith no active turn as idle. _interrupt_saved_t3_threads(core.py ~1861) has the same interrupt-then-read-once pattern.
3. Cancel cannot finish outside Prism; jobs stick in cancelling and keep their workspace
cancelandrecover --request-idfrom a shell recordt3_turn_interrupt_failed: T3 client unavailable.discover_tokenneedsT3_SERVER_TOKENor at3CLI; neither exists outside the Prism-launched environment (not3on PATH on this Mac)._finalize_stoppedthen keepscancellingwith the misleading reasoncancellation_pending: live child process group remains. Workspace ownership stays, becauseACTIVE_WORKSPACE_STATUSESincludescancelling.- Prism exposes no cancel tool, so no working cancel path exists for T3 jobs.
-2and-3are stillcancellingnow. - Fix: add a
prism_canceltool in t3code that runs cancel and recover with Prism's token. The Router should also report "T3 unreachable" rather than "live child process group". Once every saved turn is already terminal in the last known state, releasing the workspace needs no T3 call.
4. The 60 s silence threshold is likely too aggressive for hidden-reasoning models (inconclusive)
T3_SILENCE_SECS = 60. Reasoning parts are not T3 activities. In OpenCode's database, the-2worker's final part was areasoningpart created 20:01:22 local and never updated until the abort at 20:03:07 (MessageAbortedError). It could be a provider hang or long hidden thinking; the evidence cannot tell them apart.- The user's measured OpenCode baseline: healthy sessions reach about 160 s of silence, and the watchdog kills at 180 s.
- Fix: count OpenCode reasoning-part updates as liveness where readable, or raise the silence window to about 180 s for OpenCode routes.
Acceptance criteria
-
review-changes-requested(andresumed) continue the loop. Any other non-terminal, non-blocked exit is marked blocked with a named reason (controller_exit_unhandled: <action>), so the planner is always woken. Regression test: a review withrequest_changesgoes to the correction worker, then re-review, without relaunch. - Stall transfer polls until the turn is settled. Regression test using a fake server that settles the interrupt after a delay.
- A cancelled T3 job reaches
cancelledand frees its workspace through a Prism cancel path. The block reason names T3 unreachability accurately. - The silence threshold for OpenCode routes is aligned with measured liveness, or reasoning updates count as activity.
Contributes to the Objective
Reliable dispatch without the human on the critical path: jobs no longer vanish or need manual recovery.
Activity
[-]Stalled T3 worker blocks the job instead of moving route; cancelling a blocked job never releases the workspace[/-][+]Prism jobs go silent: review changes orphan the controller, stall transfer races interrupt, cancel cannot finish[/+]lukemaj commentedon Sep 27, 2026
Elon record (2026-09-27, after the user's requirement: stream liveness including thinking, no fixed timeouts, exact-second detection at 99.999% certainty)
Wanted result: every Prism job either advances or wakes the planner. A dead worker is detected within seconds of its stream stopping; a working one is never killed. Cancel works.
New evidence:
-2, healthy stretches that looked like 92-108 s of silence in native events were streaming reasoning segments every 5-15 s, with a max gap of about 18 s. The killed turn's last segment was created 18:01:22.100 (87 chars), with nothing after until the abort at 18:03:07. It really was dead.T3_SILENCE_SECS = 60on segment timestamps.Cut:
continuingallowlist inrun_controller_process, the source of bug 1: continuation becomes status-based.Smallest surviving solution:
controller_exit_unhandled: <action>, so the planner is always woken.completed,errororinterrupted(0.5 s steps, capped). A stopped session with no active turn counts as settled. Apply this in the stall path and in_interrupt_saved_t3_threads.nowminuslastStreamAtwhile no tool is open (Stream clock for Prism: exact-second worker liveness from every token, plus prism_cancel chromeria#55). A turn is stale when silence exceeds the route's measured healthy tail and a confirming probe agrees. Thresholds are policy data per route and phase (time to first token versus mid-stream), derived from the per-turn statistics T3 records. Until a route has enough samples, its threshold stays conservative at the largest observed healthy gap times a margin, never lower than the evidence. Every turn records its max gap and time to first token in the ledger, which Observer uses.prism_cancel(t3code#55). Router finalizescancelledwhen T3 reports every saved turn terminal, and names T3 unreachability accurately.Order: 1, 2 and the Router half of 4 now. 3 after t3code#55 lands and Chromeria is rebuilt; until then, raise the T3 window to the measured evidence instead of 60 s.
Falsifier: a healthy turn is interrupted as stale (review each
stalledagainst the T3 record), or a dead turn runs past threshold plus one poll.lukemaj commentedon Sep 27, 2026
Correction to the Elon record above: the
-2worker was probably not dead.pmset -g logshows clamshell sleep from 20:01:26 to 20:03:05 +0200 (99 s), 4 s after its last reasoning segment (20:01:22). Router killed the turn at 20:03:07, 2 s after wake, and counted the sleep as silence. The same cause explains the "45 s late" detection. The fix branch no longer counts sleep time as silence.Prism jobs never go silent (#132); release 0.38.0
Prism jobs never go silent (#132); release 0.38.0
lukemaj commentedon Sep 27, 2026
Router side delivered in PR #135 (not merged).
fix/132-prism-silent-jobs25c0e15fc0380c71ea68ab7d7a8ac24bcae9fb16(one commit oncf2bda5)python3 scripts/test.py && python3 -m compileall -q runner scripts testsgives 322 tests OK.versionctl release-checkOK. The new regression tests intests/test_issue132.pyfail on 0.37.0 the same way the live jobs failed.gpt-6-luna) gave APPROVE on25c0e15with no findings, after two blockers in the settle check were fixed.Covered: bug 1 (the controller keeps stepping while the job is active; an unexpected stop blocks as
controller_exit_unhandled: <action>), bug 2 (after an interrupt, poll until the turn settles, in the stall path and in cancel), the Router half of bug 3 (cancel finalizes without T3 when every saved turn's last known state is finished; the reason names T3 unreachability), and bug 4 interim (measured windows per driver: Codex 300 s, OpenCode 210 s, Claude 90 s, others 300 s, plus at3_turn_streamledger event per turn).New finding on bug 4: the "45 s late" detection on
-2was a 99 s macOS clamshell sleep (20:01:26 to 20:03:05 local, frompmset -g log). The worker's last evidence was 4 s before the sleep, and the Router counted the sleep as silence. Another job was interrupted 1 s earlier for the same reason, and T3 wrote no event from any thread during the sleep. So the-2worker was probably not dead. Host sleep no longer counts as silence.Remaining, so the PR says "Refs" and not "Closes":
prism_cancel(toolboxmd/chromeria#55), which-2and-3still need because they predate the recorded turn states; and Elon item 3, stream-clock liveness from T3'slastStreamAt/openTool.lukemaj commentedon Sep 28, 2026
Another occurrence, 2026-09-27, job
prism-muk1zqbl-09879193(lukemaj/transcribe-queue#3), routemuse-spark-xhigh-free, turn impl_5: blocked withroute_transfer_abort_failed: session not confirmed idle after stalledafter 101.7 s of silence (last_part: message:system), 1 of 5 attempts used.New consequence worth covering here: the stalled worker had already finished an independent Luna review (
request_changes, 7 majors) and written it to$TMPDIR/opencode/luna-review-issue3.logat 19:57; it went silent while parsing it at 20:00:30. The job blocked and the verdict never reached the dispatcher or planner. A later reviewer approved a nearby SHA without those findings, and the PR merged. Suggest: on a stalled turn, salvage the worker's latest artifacts and text into the terminal report before blocking or moving route.Role and capability causes from the same job are in #138.
lukemaj commentedon Sep 28, 2026
Plan for the remaining Elon item 3 (stream-clock liveness), 2026-09-28.
Chromeria #56 is merged and installed;
/api/prism/livenessis live. Revision to the record above: the stale decision itself moves into T3 (toolboxmd/chromeria#62), becausespawn_threadchildren have no Router and one detector beats two. Router consumes it:fix/132-prism-silent-jobs(PR Keep Prism jobs advancing with T3 stream liveness (#132); 0.41.0 #135) onto current main (0.40.0) and bump to the next version.GET /api/prism/liveness?threadId=every 1 s for the running worker thread. When the response carriesstale: true, treat the turn as stalled (existing interrupt, settle, retry-on-next-route path). When the field is absent (older Chromeria) or the route is unreachable, fall back to the interim per-driver windows from Keep Prism jobs advancing with T3 stream liveness (#132); 0.41.0 #135.prism-muk1zqblcase above).silenceMs,thresholdMs,thresholdSourceandreason, for the falsifier review.planner_t3_thread) is told on every stale event.Falsifier unchanged: a healthy turn interrupted as stale, or a dead turn running past threshold plus one poll.
wip: rebase #132 controller and interrupt fixes onto 0.40.0
Finish #132 stream liveness integration; prepare 0.41.0
lukemaj commentedon Sep 28, 2026
PR #135 updated for the 2026-09-28 liveness plan, ready for the dispatcher's independent review.
cecc4b30be7adbe32f50066e6cce8d9e83ada065(main 0.40.0). Main's Wait for T3 instead of blocking when Prism is unreachable #136 waiting behavior and Start Chromeria when T3 is down instead of only waiting #140 launch recovery are preserved.fix/132-prism-silent-jobs; existing workspacemodel-router-worktrees/132-prism-silent-jobs, clean.0ab66fb7bf52f828cc65788740c2c9fd2eff4192.61c239648aa34dc6a6a326005dc26312220fcc0c(rebase checkpoint),78e131e754e7a6baa3da573668c73213006a9604(liveness and salvage checkpoint),0ab66fb7bf52f828cc65788740c2c9fd2eff4192(final integration and version).T3 owns staleness. Router polls every second, acts on
stale: truewithin one poll, and never interrupts for elapsed silence whilestale: false. Missing-field/unreachable-route fallback retains #135's driver windows and host-sleep exclusion. Confirmed T3 staleness interrupts, waits for settling, then moves to the next route. No new watcher or Router-owned timeout was added.Before interrupting, Router durably salvages the current turn's text and named text artifacts, records
silenceMs,thresholdMs,thresholdSource,reasonandstaleSince, and notifiesplanner_t3_thread. Salvage survives route moves and later success in the terminal report. Artifact copy/excerpt limits and unavailable references are explicit in RUNNER.md and reports.Proof for this candidate's tree
1f4e89fc831a1c9b16f1d6f067ce505d74de9925:python3 scripts/test.py && python3 -m compileall -q runner scripts tests: 360 tests OK; compileall exit 0.versionctl release-check: v0.41.0 candidate OK at0ab66fb7bf52f828cc65788740c2c9fd2eff4192.Independent review verdict: pending on this exact head, owned by the dispatcher through Prism's review role per the planner's direction. GitHub currently has no formal reviews or
review/independentstatus for this candidate; the earlier head's approval is not reused. Next step: review this SHA, publish findings and the exact-SHA status, then update this handoff with the reviewer identity and verdict.Verified pushed head and open PR against main; GitHub reports MERGEABLE. The PR retains
Refs #132. Chromeria #62 is covered with fakes as requested, not live-verified; existing production jobs were not modified. No merge, tag, release or installation. Website/SEO impact: none, because this is runner behavior and operational documentation.lukemaj commentedon Sep 28, 2026
Live false positive on installed Router 0.40.0, 2026-09-28, job
mr-132-stream-liveness: the dispatcher resume turn (gpt-6-luna@max, threadsub.3cecf5f2-….3866cdbc) completed normally at 18:29:44Z (T3prism.stream-stats: outcome completed, durationMs 248202, maxGapMs 7085, 1287 events). Router blocked the job at 18:30:43Z witht3_resume_failed: stalled, 59 s after the turn had already completed, and the dispatcher's review action was lost. Another case for the stream-clock liveness this PR adds: a completed turn must never read as stalled.Merge pull request #135 from toolboxmd/fix/132-prism-silent-jobs
lukemaj commentedon Oct 1, 2026
Superseded by toolboxmd/chromeria#92: the controller, stall transfer and runner cancel are deleted. Stopping a child is interrupt_thread (toolboxmd/chromeria#95, #102); report-back survives restarts (#94, #103).