fix(smokes): track the dsh runner's reasoning_effort keyword - #4484
Conversation
The adapter has supplied reasoning_effort since the managed execution profile landed, but two hermetic fake-runner fixtures still declared the older keyword set. The falsifier was main's own public smoke: the built-in dsh host e2e failed with an unattributed host_failure, whose cause was `TypeError: run_dsh_turn() got an unexpected keyword argument 'reasoning_effort'` inside the fake runner. The fake runner now comes from one place -- FAKE_DSH_RUNNER_SIGNATURE in the generic-cli e2e smoke, which the built-in host smoke and the adapter smoke both already reuse -- so the next adapter keyword lands once instead of drifting in each caller. The emitted runner also asserts the resolved provider/model/effort it is handed, so a dropped profile fails at the seam instead of at the endpoint. Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
huangruiteng
left a comment
There was a problem hiding this comment.
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机
main 上的全量公开冒烟是红的,但红的不是 host,而是两份跟 host 契约脱节的 hermetic fake runner:adapter 在托管执行档落地后一直传 reasoning_effort,两个 fake runner 却还写着旧的关键字集,于是调用点抛 TypeError: run_dsh_turn() got an unexpected keyword argument 'reasoning_effort',被 adapter 映射成 BuiltInHostError("dsh_execution_failed"),Turn 只能报一个不带归因的 host_failure。结果是 examples/loopx-turn-dsh-builtin-host-e2e-smoke.py 的成功腿挂掉、smoke-fleet-health 跟着 .ready == true 失败。这条修复把"假 runner 必须跟上真实调用契约"这件事重新做成可执行的门,并且顺带消掉一份重复的签名。
改动思路
- fake runner 的签名只保留一份:放在
examples/loopx-turn-dsh-e2e-smoke.py的FAKE_DSH_RUNNER_SIGNATURE,因为内置 host 冒烟本来就 import 这个模块复用它;examples/dsh-turn-host-adapter-smoke.py改成同一份来源,不再手写第二份。 - 签名按 adapter 在
load_dsh_runner文档里声明的关键字集写全,并要求 runner 断言拿到已解析的provider/model/reasoning_effort——profile 被丢掉时在 seam 上失败,而不是到真实端点才失败。 - 不改
**kwargs的 runner(examples/loopx-turn-managed-step-self-heal-smoke.py与tests/*_runner.py):它们服务于各自更窄的断言,精确关键字的守卫由本 PR 修的两份冒烟加tests/dsh_goal_mode_fake_runner.py承担。
具体改动
examples/loopx-turn-dsh-e2e-smoke.py(+14/-5):新增模块级FAKE_DSH_RUNNER_SIGNATURE(含reasoning_effort与 profile 断言),_write_fake_dsh_runner改用它生成代码。examples/dsh-turn-host-adapter-smoke.py(+14/-4):按内置 host 冒烟同样的 importlib 方式加载通用 e2e 冒烟模块,runner 文本用同一份签名。- 未改任何产品代码、CLI、schema、前端;内置 host 冒烟本身无需改动,因为它复用的就是修好的 writer。
关键内容讲解
- 失败是被证伪出来的,不是推断出来的:先在干净 worktree 上跑出
host_failure/exit 1,再在001e0c380(更早的 main)上复现同样失败,确认这不是最近几个管家/managed PR 带来的回归。 - 修复方向由"哪份契约为准"决定:adapter 侧
load_dsh_runner的 docstring 与真实调用点已经声明关键字集,所以改的是 fixture,不是产品。 - 负向验证:把
reasoning_effort从共享签名里拿掉,examples/dsh-turn-host-adapter-smoke.py立刻以 CI 里那句同样的TypeError失败;改回后重新通过。因此这份 fixture 是承重的,不是"碰巧绿"。
对主干的风险
P3(可接受):三份冒烟现在共享一个模块级常量,examples/dsh-turn-host-adapter-smoke.py 因此在导入期多加载一个 example 模块(无副作用,只定义常量与函数)。代价是一次导入,收益是签名不会再各自漂移。
P3(已知残余):接受 **kwargs 的 runner 按构造就无法捕获这一类漂移,本 PR 没有扩大它们的断言范围;精确关键字守卫落在两份被修的冒烟与 tests/dsh_goal_mode_fake_runner.py 上。
其余残余:变更只动公开示例冒烟,loopx canary premerge --from-git-diff 的门通过(0 失败、0 人工挂起、3 项选中检查含 public-boundary 扫描与语义漂移冒烟);本地验证覆盖 adapter 往返、通用 e2e 全链路、内置 host 两条腿、tests/test_dsh_goal_mode.py 58 项、ruff、语义清单检查。未覆盖:真实 DeepSeek Harness SDK/端点的调用(本路径本来就是 hermetic,不涉及凭据与网络)。
我的整体评价
APPROVE。这是一个归因清楚、边界最小、并且带负向证据的 CI 修复:fixture 跟上真实契约、签名单一来源、失败信息可读。它同时让 main 的公开冒烟舰队恢复绿色,属于可以直接自合并的类型。变更范围仅两个公开示例文件,change-quality receipt cqr_0ea46e303db56e9978c1 覆盖同一 scope fingerprint。
English verdict: APPROVE at 5efef15. main's full-public-smokes is red because two hermetic fake-runner fixtures still declare the dsh keyword set as it was before the managed execution profile landed; the adapter supplies reasoning_effort, the fake runner raises TypeError, and the Turn degrades to an unattributed host_failure. Reproduced on the worktree and again at 001e0c380, so it is not fallout from the recent steward/managed PRs. The fix emits the fake runner signature from one place (the generic-cli e2e smoke the built-in host smoke already imports) and has the emitted runner assert the resolved provider/model/effort, so a dropped profile fails at the seam. Verified: adapter smoke 11 checks, generic e2e, built-in host e2e (previously exit 1 / host_failure), 58 goal-mode tests, ruff, semantic inventory check, canary premerge gate passed with 0 failures, and a mutation that removes reasoning_effort reproduces the exact CI TypeError before being reverted. Scope is two public example smokes; no runtime, CLI, schema, or frontend change. Receipt cqr_0ea46e303db56e9978c1.
What changed
main'sfull-public-smokesworkflow is red, and the failure is not in thehost: two hermetic fake-runner fixtures still declare the
run_dsh_turnkeyword set the adapter used before the managed execution profile landed. The
adapter has supplied
reasoning_effortsince then, so the fake runner raiseswhich the adapter maps to
BuiltInHostError("dsh_execution_failed")and theTurn reports as an unattributed
host_failure. The observable symptom isexamples/loopx-turn-dsh-builtin-host-e2e-smoke.pyfailing its success leg(exit 1,
result_kind: host_failure), followed bysmoke-fleet-healthfailingits
.ready == truecheck.This is pre-existing on
main, not fallout from the recent steward/managedwork: the same smoke fails at
001e0c380, before the credential-resolved Turndefault and the chat PRs landed.
Changed surfaces
Public example smokes only; no runtime, CLI, schema, or frontend change.
examples/loopx-turn-dsh-e2e-smoke.py: the fake runner source is now emittedfrom one module-level
FAKE_DSH_RUNNER_SIGNATURE, carrying the currentkeyword set (
... provider, model, reasoning_effort, max_tokens, ...) andasserting the resolved provider/model/effort it is handed.
examples/dsh-turn-host-adapter-smoke.py: loads that smoke the same way thebuilt-in host smoke already does and uses the same emitted source instead of
keeping a second hand-written copy of the signature.
The built-in host smoke needs no change: it already reuses the generic-cli
smoke's writer, which is why fixing the writer fixes both failing legs.
Why one source
Three smokes load a fake runner and the contract is real (the adapter documents
the keyword set on
load_dsh_runner). A fixture that lags it fails as anunattributed
host_failureat the endpoint instead of naming the seam thatmoved, so the emitted signature now lives in the module the other host smokes
already import: the next adapter keyword lands once.
Validation
python examples/dsh-turn-host-adapter-smoke.pypython examples/loopx-turn-dsh-e2e-smoke.pypython examples/loopx-turn-dsh-builtin-host-e2e-smoke.pyhost_failure, exit 1)python -m pytest tests/test_dsh_goal_mode.py -qpython examples/loopx-turn-managed-step-self-heal-smoke.pypython -m ruff checkon both changed filespython scripts/generate_semantic_inventory.py --checkloopx canary premerge --from-git-diffcqr_0ea46e303db56e9978c1at scope0ea46e303db56e99Negative path: removing
reasoning_effortfrom the shared signature makesexamples/dsh-turn-host-adapter-smoke.pyfail with exactly the CITypeError; the mutation was reverted and the smoke passes again. The repairedfixture is load-bearing rather than merely green.
Boundary and remaining limits
**kwargs(
examples/loopx-turn-managed-step-self-heal-smoke.py, thetests/*_runner.pyhelpers), so they cannot catch this drift class byconstruction. They are left as they are:
tests/dsh_goal_mode_fake_runner.pyplus the two smokes repaired here are the exact-keyword guards.
whole path is hermetic and deterministic.
ruff formatreports both files as non-conforming onmainas well; thatpre-existing formatting drift is not folded into this fix.