Skip to content

Preserve ready work and back off peer coordination - #4845

Open
huangruiteng wants to merge 5 commits into
mainfrom
codex/fix-ready-work-backoff
Open

huangruiteng wants to merge 5 commits into
mainfrom
codex/fix-ready-work-backoff

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

A ready deferred Todo could disappear when the host supplied runtime capabilities: resume planning recomputed candidates from the bounded display lane and dropped ready rows retained only in the dedicated candidate lane. The resulting false peer_coordination_blocked state could strand work. Its scheduler and automation projections also disagreed about whether the recurring heartbeat should remain alive.

This change:

  • reevaluates the explicit deferred-candidate lane together with the bounded display lane while preserving the display bound;
  • deduplicates compact and lossless projections by canonical Todo id;
  • classifies peer_coordination_blocked as a recoverable wait instead of a terminal stop;
  • keeps the heartbeat active without quota spend and uses the existing TypeScript-owned scheduler backoff transaction with a 10/20/30/60 minute progression;
  • reevaluates the peer/runtime/local frontier at each bounded poll and reuses the existing scheduler identity-reset contract when the projected decision changes.

Goal stopped, quota paused, and validated terminal no-follow-up remain the stop cases. The TypeScript migration RFC is already in its transaction-payoff phase, so this PR reuses the existing coarse scheduler transition owner and removes the special Python stop path instead of adding another leaf reducer or bridge.

A replay of the captured finance-research-goal snapshot against base 61b8311b7 and head 216fc90cc verifies both defects: the ready candidate remains selectable when runtime capabilities are supplied, and the blocked coordination response now projects keep_active=true, pause_allowed=false, backoff_until_reassigned, cadence class peer_coordination_wait, and 10/20/30/60 stateful backoff.

Validation:

  • 528 focused Python tests passed across scheduler arbitration/liveness, quota pause precedence, semantic vocabulary, task orchestration, and scheduler state behavior.
  • 13 TypeScript scheduler/resume-planning tests passed.
  • task-orchestration-smoke.py passed.
  • automation-liveness-state-machine-smoke.py passed.
  • Control-plane TypeScript typecheck passed.
  • Ruff 0.15.22 passed for changed Python files.
  • Real captured Finance Goal candidate and blocked-heartbeat base/head replays passed.

Entry points:

  • CLI/managed Turn and Codex App heartbeat consume the corrected shared typed projections.
  • No frontend or Lark companion change is needed: no schema or editable configuration field was added, and both surfaces already consume the shared projection/readback rather than owning a scheduler or resume-planning implementation.

No local runtime state, credentials, logs, or private research artifacts are included.

Closes #4843

@huangruiteng
huangruiteng force-pushed the codex/fix-ready-work-backoff branch 2 times, most recently from b168e3a to 6403d38 Compare September 21, 2026 05:08
@huangruiteng huangruiteng changed the title Preserve ready work and align heartbeat stop liveness Preserve ready work and back off peer coordination Sep 21, 2026
@huangruiteng

Copy link
Copy Markdown
Collaborator Author

补充 exact-head 行为证据与 TS 边界判断(head 6403d38300563517841e05f36745bdd7a6c97de2):

为什么是 backoff,而不是 stop

peer_coordination_blocked 的恢复输入会在本 Agent 之外变化:peer capability/readiness、运行环境、协调配置和本地 frontier 都可能异步改变。停止或删除 heartbeat 会移除唯一观察者,只能依赖人工恢复;因此本 PR 将它定义为 no-spend recoverable wait。真正的 stop 仍只用于 Goal stopped、quota paused 和已验证的 terminal no-followup。

同一份真实 Finance quota should-run 快照的 base/head 对照:

base 65afc487:
  liveness = keep_active=true / pause_allowed=false / action=keep_active
  scheduler = return_to_owner_until_material_change
  host_action = pause_or_delete_current_heartbeat

head 6403d383:
  liveness = keep_active=true / pause_allowed=false
  automation_action = keep_active_peer_coordination_backoff
  scheduler = backoff_until_reassigned
  cadence = 10 -> 20 -> 30 -> 60 minutes
  host_action = update_current_heartbeat_rrule
  apply_needed = true

这既消除了 base 的“保持活跃 + 停止宿主”矛盾,也保留了异步恢复观察面。

被隐藏 ready Todo 的 base/head 对照

同一个 typed resume-planning fixture 中,todo_ready 只存在于 lossless deferred_resume_candidates,不在有界展示 deferred_items

base: candidates = [todo_visible]
head: candidates = [todo_ready, todo_visible]
head display lane remains [todo_visible]

因此修复保留候选真相,同时不放大展示窗口。

TS 重构裁决

没有新建 leaf-only TS reducer。该 PR 让 peer wait 复用现有 scheduler.state_transition.evaluate TypeScript 事务,继续由同一个 TS owner 负责 progression、reset/identity、host apply/ACK/failure retention;Python 只做 interaction disposition 与参数适配,并删除专用 stop 分支。这符合 TS migration RFC 的 transaction-payoff 阶段,避免再造第二个 cadence owner。

验证:408 个相关 Python tests、13 个 TS tests、两个 control-plane smoke、control-plane typecheck、Ruff 0.15.22、git diff --check 和上述真实快照 replay 均通过。当前仍等待 GitHub required checks 与独立维护者 review;本 PR 不自合并。

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
@huangruiteng
huangruiteng force-pushed the codex/fix-ready-work-backoff branch from 6403d38 to 216fc90 Compare September 21, 2026 05:20
@huangruiteng

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (61b8311b77e43a984d3612a5e8ff8d600327ab03). The exact head is now 216fc90ccba0ddca0db3ada99ced2bfced74aed9; the earlier exact-head comment is superseded only by this history rewrite.

The base/head Finance replay remains unchanged:

  • base: contradictory keep_active plus pause_or_delete_current_heartbeat, and the lossless ready candidate is dropped;
  • head: keep_active_peer_coordination_backoff, backoff_until_reassigned, 10/20/30/60 RRULE progression, and the ready candidate remains selectable while the bounded display lane stays bounded.

Post-rebase validation: 528 Python tests, 13 TS tests, two smokes, control-plane typecheck, Ruff and diff check all pass. Independent review and refreshed required checks remain outstanding; no self-merge.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

Exact head reviewed: 216fc90ccba0ddca0db3ada99ced2bfced74aed9

动机

这次修复的主问题成立,而且优先级高:当宿主传入运行时 capability 时,旧路径只从有界展示 lane 重算,可能丢掉只存在于 deferred_resume_candidates 的 ready Todo;随后产生的 peer_coordination_blocked 又会被不同投影当成终止状态,最终把本来可恢复的工作搁置。把 ready candidate 保留下来、把 peer coordination 改成 no-spend 的可恢复等待,是正确的产品方向。

改动思路

整体架构选择是对的:

  • readiness 继续由 TypeScript 的 typed resume-condition owner 决定;
  • Python 只消费 typed effective action,不复制第二套状态机;
  • scheduler 复用现有 stateful transition owner,按 10/20/30/60 分钟回退;
  • peer/config/reassignment/local frontier 变化会重置 cadence;
  • goal_stoppedquota_paused、validated terminal no-followup 仍是停止边界。

deferredPlan 把“同一个 Todo 共享 readiness identity”和“两个 projection 共享同一个 payload”混成了一件事。前者应该共享,后者不应该共享。

具体改动

我确认以下意图已实现并通过验证:candidate-only 的 ready Todo 不再被有界展示 lane 擦掉;peer_coordination_blocked 会保持 automation active、禁止 pause、且不消费 quota;scheduler backoff 复用现有 typed transition 和 ack/reset identity。

有一个阻塞问题:

  1. [P1] compact display payload 被 lossless candidate 替换。 deferredPlan 先做 unique([...projectedCandidates, ...deferredItems]),candidate 在前,因此同一 todo_id 时 lossless candidate 成为唯一 winner。后面的 displayIdentities 只检查 identity,又把这个 winner 当成 deferredItems 返回。结果是 deferred_lanes.deferred_itemscapacity_fields.deferred_items 都带上 lossless payload,破坏了有界 compact projection,可能放大 quota/status/heartbeat 热路径 JSON,也可能暴露原本刻意从展示 row 省略的字段。

我用 production projectTodoResumePlanning 做了一个反例:同一 Todo 的展示 row 文本为 Compact display copy、candidate row 文本为 Lossless candidate copy;当前 head 的 candidate 输出正确保留 lossless 文本,但两个 display 输出也错误变成了 lossless 文本。现有新测试只断言 candidate lane,因此没有捕获 display lane 的语义漂移。

最小修复建议:仍然对 canonical Todo id 做一次去重并计算 readiness/condition,然后按 Todo id 把这个决定分别应用回原始 compact deferredItems 和原始 lossless projectedCandidates;不要让一个 Item 实例同时拥有两个 projection。把现有 same-Todo 回归补成配对断言:candidate 必须保持 Lossless candidate copy,而 deferred_lanes.deferred_itemscapacity_fields.deferred_items 必须保持 Compact display copy;最好再放一个 oversized candidate 覆盖 packet budget。

对主干的风险

本地验证结果:

  • resume_planning.test.ts:6 passed;
  • scheduler arbitration/execution-context/backoff/transition/runtime/ack matrix:275 passed;
  • task-orchestration-smoke.py:passed;
  • automation-liveness-state-machine-smoke.py:passed;
  • changed Python Ruff:passed;
  • git diff --check:passed;
  • 上述 compact-vs-lossless 反例:failed,稳定复现 P1

作者在审阅期间把分支 rebase 到最新 main;我核对了新旧 head,PR 自身的 runtime diff 与上述 finding 没有变化,并在新 exact head 上再次稳定复现。新的 DCO、dependency review、Frontstage Pages、PostgreSQL authority 与 release-artifact checks 在我读取时仍处于 queued,merge state 是 BLOCKED;因此即使修复 finding,仍需等待新 head 的最终 required CI。

我的整体评价

这是一个边界合理、架构方向正确的 liveness 修复,typed authority、no-spend backoff 和终止条件都比旧行为更清楚;我不建议拆掉这些部分。当前不能通过的原因很集中:canonical identity 的复用越过了 projection payload 边界。修复映射并补上双 projection 回归后,再跑 focused TypeScript、orchestration/liveness、quota/status output-budget 与 required CI,我会基于新 exact head 复审。

English verdict: REQUEST_CHANGES

@huangruiteng

Copy link
Copy Markdown
Collaborator Author

CI note for exact head 216fc90cc: Frontstage Pages fails before browser smoke because a clean dashboard build changes packaged loopx/web/chat assets. The same failure is present on default-branch run 35563272840 for main commit 65afc487, before this PR; current main 61b8311b only adds RFC docs and retains that drift. This PR changes no dashboard/frontend source or packaged asset, so generated frontend output is intentionally not mixed into the scheduler fix. The main asset drift has been handed to the engineering lane as a separate repair.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ready deferred work can be hidden and peer waits can stop heartbeats

1 participant