Skip to content

fix(smokes): track the dsh runner's reasoning_effort keyword - #4484

Merged
huangruiteng merged 1 commit into
mainfrom
codex/dsh-turn-fixture-reasoning-effort-20260916
Sep 16, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/dsh-turn-fixture-reasoning-effort-20260916

Conversation

@huangruiteng

Copy link
Copy Markdown
Collaborator

What changed

main's full-public-smokes workflow is red, and the failure is not in the
host: two hermetic fake-runner fixtures still declare the run_dsh_turn
keyword set the adapter used before the managed execution profile landed. The
adapter has supplied reasoning_effort since then, so the fake runner raises

TypeError: run_dsh_turn() got an unexpected keyword argument 'reasoning_effort'

which the adapter maps to BuiltInHostError("dsh_execution_failed") and the
Turn reports as an unattributed host_failure. The observable symptom is
examples/loopx-turn-dsh-builtin-host-e2e-smoke.py failing its success leg
(exit 1, result_kind: host_failure), followed by smoke-fleet-health failing
its .ready == true check.

This is pre-existing on main, not fallout from the recent steward/managed
work: the same smoke fails at 001e0c380, before the credential-resolved Turn
default and the chat PRs landed.

Changed surfaces

Public example smokes only; no runtime, CLI, schema, or frontend change.

  • examples/loopx-turn-dsh-e2e-smoke.py: the fake runner source is now emitted
    from one module-level FAKE_DSH_RUNNER_SIGNATURE, carrying the current
    keyword set (... provider, model, reasoning_effort, max_tokens, ...) and
    asserting the resolved provider/model/effort it is handed.
  • examples/dsh-turn-host-adapter-smoke.py: loads that smoke the same way the
    built-in host smoke already does and uses the same emitted source instead of
    keeping a second hand-written copy of the signature.

The built-in host smoke needs no change: it already reuses the generic-cli
smoke's writer, which is why fixing the writer fixes both failing legs.

Why one source

Three smokes load a fake runner and the contract is real (the adapter documents
the keyword set on load_dsh_runner). A fixture that lags it fails as an
unattributed host_failure at the endpoint instead of naming the seam that
moved, so the emitted signature now lives in the module the other host smokes
already import: the next adapter keyword lands once.

Validation

check result
python examples/dsh-turn-host-adapter-smoke.py 11 checks passed
python examples/loopx-turn-dsh-e2e-smoke.py passed
python examples/loopx-turn-dsh-builtin-host-e2e-smoke.py passed (was failing: host_failure, exit 1)
python -m pytest tests/test_dsh_goal_mode.py -q 58 passed
python examples/loopx-turn-managed-step-self-heal-smoke.py passed
python -m ruff check on both changed files passed
python scripts/generate_semantic_inventory.py --check up to date
loopx canary premerge --from-git-diff gate passed, 0 failures, 0 manual holds, 3 selected checks
change-quality receipt cqr_0ea46e303db56e9978c1 at scope 0ea46e303db56e99

Negative path: removing reasoning_effort from the shared signature makes
examples/dsh-turn-host-adapter-smoke.py fail with exactly the CI
TypeError; the mutation was reverted and the smoke passes again. The repaired
fixture is load-bearing rather than merely green.

Boundary and remaining limits

  • Two other hermetic runners in this family accept **kwargs
    (examples/loopx-turn-managed-step-self-heal-smoke.py, the
    tests/*_runner.py helpers), so they cannot catch this drift class by
    construction. They are left as they are: tests/dsh_goal_mode_fake_runner.py
    plus the two smokes repaired here are the exact-keyword guards.
  • No DeepSeek Harness SDK, credential, network, or model call is involved: the
    whole path is hermetic and deterministic.
  • ruff format reports both files as non-conforming on main as well; that
    pre-existing formatting drift is not folded into this fix.

The adapter has supplied reasoning_effort since the managed execution profile
landed, but two hermetic fake-runner fixtures still declared the older keyword
set. The falsifier was main's own public smoke: the built-in dsh host e2e
failed with an unattributed host_failure, whose cause was
`TypeError: run_dsh_turn() got an unexpected keyword argument
'reasoning_effort'` inside the fake runner.

The fake runner now comes from one place -- FAKE_DSH_RUNNER_SIGNATURE in the
generic-cli e2e smoke, which the built-in host smoke and the adapter smoke both
already reuse -- so the next adapter keyword lands once instead of drifting in
each caller. The emitted runner also asserts the resolved provider/model/effort
it is handed, so a dropped profile fails at the seam instead of at the endpoint.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

main 上的全量公开冒烟是红的,但红的不是 host,而是两份跟 host 契约脱节的 hermetic fake runner:adapter 在托管执行档落地后一直传 reasoning_effort,两个 fake runner 却还写着旧的关键字集,于是调用点抛 TypeError: run_dsh_turn() got an unexpected keyword argument 'reasoning_effort',被 adapter 映射成 BuiltInHostError("dsh_execution_failed"),Turn 只能报一个不带归因的 host_failure。结果是 examples/loopx-turn-dsh-builtin-host-e2e-smoke.py 的成功腿挂掉、smoke-fleet-health 跟着 .ready == true 失败。这条修复把"假 runner 必须跟上真实调用契约"这件事重新做成可执行的门,并且顺带消掉一份重复的签名。

改动思路

  1. fake runner 的签名只保留一份:放在 examples/loopx-turn-dsh-e2e-smoke.py 的 FAKE_DSH_RUNNER_SIGNATURE,因为内置 host 冒烟本来就 import 这个模块复用它;examples/dsh-turn-host-adapter-smoke.py 改成同一份来源,不再手写第二份。
  2. 签名按 adapter 在 load_dsh_runner 文档里声明的关键字集写全,并要求 runner 断言拿到已解析的 provider/model/reasoning_effort——profile 被丢掉时在 seam 上失败,而不是到真实端点才失败。
  3. 不改 **kwargs 的 runner(examples/loopx-turn-managed-step-self-heal-smoke.py 与 tests/*_runner.py):它们服务于各自更窄的断言,精确关键字的守卫由本 PR 修的两份冒烟加 tests/dsh_goal_mode_fake_runner.py 承担。

具体改动

  • examples/loopx-turn-dsh-e2e-smoke.py(+14/-5):新增模块级 FAKE_DSH_RUNNER_SIGNATURE(含 reasoning_effort 与 profile 断言),_write_fake_dsh_runner 改用它生成代码。
  • examples/dsh-turn-host-adapter-smoke.py(+14/-4):按内置 host 冒烟同样的 importlib 方式加载通用 e2e 冒烟模块,runner 文本用同一份签名。
  • 未改任何产品代码、CLI、schema、前端;内置 host 冒烟本身无需改动,因为它复用的就是修好的 writer。

关键内容讲解

  1. 失败是被证伪出来的,不是推断出来的:先在干净 worktree 上跑出 host_failure/exit 1,再在 001e0c380(更早的 main)上复现同样失败,确认这不是最近几个管家/managed PR 带来的回归。
  2. 修复方向由"哪份契约为准"决定:adapter 侧 load_dsh_runner 的 docstring 与真实调用点已经声明关键字集,所以改的是 fixture,不是产品。
  3. 负向验证:把 reasoning_effort 从共享签名里拿掉,examples/dsh-turn-host-adapter-smoke.py 立刻以 CI 里那句同样的 TypeError 失败;改回后重新通过。因此这份 fixture 是承重的,不是"碰巧绿"。

对主干的风险

P3(可接受):三份冒烟现在共享一个模块级常量,examples/dsh-turn-host-adapter-smoke.py 因此在导入期多加载一个 example 模块(无副作用,只定义常量与函数)。代价是一次导入,收益是签名不会再各自漂移。

P3(已知残余):接受 **kwargs 的 runner 按构造就无法捕获这一类漂移,本 PR 没有扩大它们的断言范围;精确关键字守卫落在两份被修的冒烟与 tests/dsh_goal_mode_fake_runner.py 上。

其余残余:变更只动公开示例冒烟,loopx canary premerge --from-git-diff 的门通过(0 失败、0 人工挂起、3 项选中检查含 public-boundary 扫描与语义漂移冒烟);本地验证覆盖 adapter 往返、通用 e2e 全链路、内置 host 两条腿、tests/test_dsh_goal_mode.py 58 项、ruff、语义清单检查。未覆盖:真实 DeepSeek Harness SDK/端点的调用(本路径本来就是 hermetic,不涉及凭据与网络)。

我的整体评价

APPROVE。这是一个归因清楚、边界最小、并且带负向证据的 CI 修复:fixture 跟上真实契约、签名单一来源、失败信息可读。它同时让 main 的公开冒烟舰队恢复绿色,属于可以直接自合并的类型。变更范围仅两个公开示例文件,change-quality receipt cqr_0ea46e303db56e9978c1 覆盖同一 scope fingerprint。

English verdict: APPROVE at 5efef15. main's full-public-smokes is red because two hermetic fake-runner fixtures still declare the dsh keyword set as it was before the managed execution profile landed; the adapter supplies reasoning_effort, the fake runner raises TypeError, and the Turn degrades to an unattributed host_failure. Reproduced on the worktree and again at 001e0c380, so it is not fallout from the recent steward/managed PRs. The fix emits the fake runner signature from one place (the generic-cli e2e smoke the built-in host smoke already imports) and has the emitted runner assert the resolved provider/model/effort, so a dropped profile fails at the seam. Verified: adapter smoke 11 checks, generic e2e, built-in host e2e (previously exit 1 / host_failure), 58 goal-mode tests, ruff, semantic inventory check, canary premerge gate passed with 0 failures, and a mutation that removes reasoning_effort reproduces the exact CI TypeError before being reverted. Scope is two public example smokes; no runtime, CLI, schema, or frontend change. Receipt cqr_0ea46e303db56e9978c1.

@huangruiteng
huangruiteng merged commit 1d8a725 into main Sep 16, 2026
16 of 17 checks passed
@huangruiteng
huangruiteng deleted the codex/dsh-turn-fixture-reasoning-effort-20260916 branch September 16, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant