Skip to content

test(replan): execute the command projected to the agent - #4646

Merged
huangruiteng merged 1 commit into
mainfrom
codex/replan-command-20260917
Sep 17, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/replan-command-20260917

Conversation

@huangruiteng

Copy link
Copy Markdown
Collaborator

The existing replan smoke hand-authored its refresh command, so it could pass while the command shown to an agent was unusable. Execute the projected command instead, replacing only declared semantic input slots, and verify that it records the ACK for the exact obligation and clears the guard. Also preserve structured CLI errors from stdout in subprocess mode.

Replaces #4611. Latest main already derives a semantic ACK from an accepted delta; its proposed condition-string field has no execution consumer and is unnecessary. No runtime or protocol change is needed for this outcome.

Validation: the smoke passes through actual CLI subprocesses, including rejection of an equivalent observation; 68 replan gate/ACK/semantic-action/context tests pass. A separate Codex App guard with the same turn identity accepts its projected transition. Ruff, public-boundary checks, exact-scope quality qualification and standard premerge pass.

Refactor pass: reuse and strengthen the existing regression instead of adding another protocol or test suite. Frontend/Lark behavior is unchanged because only the CLI regression is modified.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

Reviewed exact head: 89c34e3df175aded1f9709cd5559afa052c38486 (test(replan): execute the projected semantic transition command).

动机

原来的 replan smoke 自己手写了它期望 agent 执行的 refresh-state 命令,于是它可以一直绿着,而 host 真正投影出来的那条命令可能缺参数或根本不可用。这个 PR 的存在理由是准确的:这是整条 semantic-ACK 路径上唯一会执行投影命令的回归,而它此前绕过了投影本身。

改动思路

复用并加强既有 smoke,而不是再加一套协议或测试:从 quota should-run 的结果里取出 interaction_contract.cli_channel.next_cli_actions 中的 refresh-state 命令,只替换四个已声明的输入槽(结果类、surface、hypothesis、probe、evidence),其余投影参数原样保留后执行;fixture 自己提供隔离的 registry/runtime。顺带让 run_cli_error 同时返回 stderr 与 stdout,以保留 subprocess 模式下打在 stdout 的结构化错误。

具体改动

1 个文件、+20/-22,纯 example:

  • 删除 12 行手写 flag 列表,改为 shlex.split(action) + 槽位替换后 run_cli(*args)。
  • run_cli_error 返回值改为 "\n".join((result.stderr, result.stdout))。
  • from loopx import cli as loopx_cli 补 # noqa: E402。

我复核的关键点(都在这个 head 上自己跑过):

  • python examples/autonomous-replan-obligation-smoke.py → ok,exit 0。
  • pytest -q tests/control_plane/test_autonomous_replan_ack.py tests/control_plane/test_refresh_state_replan_gate.py tests/control_plane/test_replan_semantic_action_behavior.py tests/control_plane/test_replan_host_context_projection.py → 68 passed。
  • 负路径仍在:assert_equivalent_observation_is_rejected_before_write 断言 typed semantic delta 拒绝,并且把 run index 逐字节比对确认「没有写入」。
  • 投影健壮性:替换表只覆盖四个声明槽,任何新出现的 <...> 占位符都会原样传给 CLI 并让 smoke 明确失败,不会静默通过。

遗留问题(非阻塞,P3)

stdout 保留那半改动没有被任何断言覆盖。我在 scratch 副本里把 run_cli_error 还原成只返回 stderr,smoke 仍然 ok/exit 0——assert "typed semantic delta" in error 在当前环境仍能从 stderr 匹配到文本。这行本身是合理的健壮性改进(正文也把它列为交付内容之一),但没有任何用例在它回归时失败。最小修法:补一个错误文本只出现在 stdout 的用例(例如 --format json 的子进程调用)并对其断言,或在正文里注明这行是防御性的、当前无覆盖。

对主干的风险

改动只在 example 内,生产路径零影响;真正变化的是断言的严格度:投影命令现在被逐字执行(除四个声明槽),所以投影一旦发出 smoke 不认识的参数就会红。最强回归场景正是「投影模板改了而 smoke 仍绿」——现在这种沉默被消除。未验证维度:smoke 跑在 fixture 的隔离 registry/runtime 上,证明的是投影可执行且 ACK 绑定到确切 obligation,不是「每个 host 都能触达真实 goal 的投影」;另外上面那条 stdout 改动未被覆盖。回退成本:单文件 revert。

我的整体评价

结论 APPROVE。方向正确且克制:把「打印出来的投影」变成「被执行并被断言的投影」,删掉手写副本而不是再镜像一份,且没有引入新协议(正文说明它取代了此前会加一个无消费方字段的 #4611 方向)。我跑了 smoke 与 68 个 replan 相关测试,确认负路径仍拒绝写入。唯一 P3 是 stdout 保留那行无断言覆盖。

English verdict: APPROVE - exact head 89c34e3; the smoke now executes the projected next_cli_actions command (substituting only the four declared semantic slots, preserving every other projected argument and supplying the fixture registry/runtime) instead of a hand-authored flag list, so a broken projection fails the regression. I ran the smoke (ok, exit 0) and 68 replan gate/ACK/semantic-action/context tests (passed), and the equivalent-observation rejection still asserts no write. One non-blocking P3: reverting the run_cli_error stdout-preservation line keeps the smoke green, so that half of the change has no assertion behind it.

@huangruiteng
huangruiteng merged commit 74f65e1 into main Sep 17, 2026
23 checks passed
@huangruiteng
huangruiteng deleted the codex/replan-command-20260917 branch September 17, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant