Skip to content

fix(testing): align CLI budget fixture paths across review runners - #5178

Merged
huangruiteng merged 1 commit into
mainfrom
codex/cli-budget-fixture-parity-20260927
Sep 27, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/cli-budget-fixture-parity-20260927

Conversation

@huangruiteng

Copy link
Copy Markdown
Collaborator

Summary / 摘要

The real-CLI review runner aliases the whole fixture parent, while pytest aliases each default scenario. The /crowded suffix adds 32 characters across four emitted command paths: the unchanged Turn fixture measures 14,482 in pytest but 14,514 in review, exceeding the 14,500-character budget.

This aligns the runner with the existing per-scenario alias. It does not raise a budget, rewrite stdout, shrink fixtures, omit executable vision-authoring fields, or shorten production commands. Base and head still use the same runner and semantic checks. This removes a validation-layout discrepancy affecting #5171, not its runtime/monitor acceptance.

独立评审脚本与 pytest 的场景别名层级不同,导致相同负载的四处命令路径多出 32 字符。本 PR 复用既有每场景固定宽度别名,保留原预算、负载和完整 stdout;补上短/长测试目录及真实增长拒绝回归,文档中明确测量与生产长路径的边界。

Boundary and delivery

  • Owner: existing CLI-output qualification fixture/runner, roadmap S10/S12. No new capability, configuration owner, authority decision or production logic.
  • Placement: Python remains the existing test-harness language; no parallel Python control-plane decision source is introduced.
  • CLI user output, frontend and Lark remain unchanged. Only developer/reviewer measurement setup changes; there are no settings or schema-driven UI fields to mount. The separate real-long-path command-integrity check remains exercised.
  • Future-facing refinement: reuse the existing alias helper at the two measurement entry points; do not introduce another normalization framework or transform captured output.
  • Private state, diagnostic logs and personal context are excluded.

Validation

  • Before the fix: the two new layout regressions fail; the growth-rejection negative case passes.
  • After the fix: test_cli_output_probe_runner.py + test_cli_output_differential.py: 71 passed. Final focused rerun: 3 passed.
  • examples/control_plane/cli-output-budget-regression-smoke.py: passed, including the real base/head matrix, default/explicit mode budgets, scale checks and long-path command integrity.
  • Ruff lint and git diff --check: passed.

The earlier failure remains a failure in its original record; these are new validation results on the corrected measurement layout. Monitor admission, default installation and original consumer observation remain separate acceptance work.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确评审 head:fac0548f3f99ea2617c6cc98e420a8c05022fb59。结论:APPROVE;这是完整的测试测量修复,不是 #5171 已安装或真实监控已恢复的证明。

动机

当前诉求是解除真实 CLI 评审中的路径布局假失败,而不是让超预算输出被放行。相同 crowded 负载在 pytest 场景别名下为 14,482 字符,在独立 runner 的父目录别名加 /crowded 后为 14,514;四处完整命令路径各多 8 字符,超过原 14,500 限额。保持现状会反复阻塞无关交付;提高限额会掩盖测量入口不一致。对照测试与质量指南的 CLI 输出预算规则,统一既有场景别名是更小且有实际维护价值的修复。持续推进能力改善,产品 CLI、前端及 Lark 体验保持不变。

改动思路

唯一修改的执行入口是开发者使用的 cli-output-probe-runner.py::_default_rows,复用现有 _stable_budget_fixture_root,不新增规范化框架。流程为:每场景根目录别名 → 原 fixture 生成 → 真实 CLI → 原始 stdout 测量 → 原限额和语义检查 → 原 receipt。失败仍由已有断言和 runner 返回;context manager 负责别名清理。base/head 继续使用同一 runner,没有新产品配置或状态权威。

正向验证在短与很长的测试工作目录中保留 36 Todo、1 Agent、12 Run 的负载,固定场景别名宽度并保留完整命令路径及 vision-authoring 字段。反向验证把真实 CLI 的合法 JSON stdout 加上 15,000 个空格,解析仍合法但尺寸增长仍被 14,500 上限拒绝。没有改写 stdout 来制造通过。

具体改动

关键代码讲解

  1. _default_rows 把既有场景处理块包进每场景 alias context;命令集合、测量、预算断言及 receipt 构建均不变。runner 的大量行变动是缩进,不是新增策略。
  2. test_runner_uses_the_pytest_scenario_alias_without_rewriting_stdout 的两种布局经过真实 CLI,检查别名宽度、清理、原始字符数、完整路径、row identity 与必要 JSON shape;没有用 mock 给出“通过”结果。
  3. test_runner_still_rejects_actual_stdout_growth 仅注入尺寸增长,确认既有预算继续生效。聚焦测试缩小执行 surface 以控制成本,完整 smoke 另外覆盖全部既有场景。
  4. 双语指南明确短路径测量不是生产长路径达标;独立真实长路径命令完整性检查继续运行。

整份 diff 为 3 文件 +157/-50:runner +53/-50 主要是缩进,聚焦回归 +90,指南 +14;生产代码改动为零。既有 differential 测试覆盖 receipt 和变体对照,但没有检查 runner 与 pytest 场景路径层级,新增回归在修复前确实出现 2 失败、增长拒绝 1 通过。最近 12 条同作者 PR 扫描未发现另一份相同修复;#5171 的监控语义、#5176 的 delegation 诊断、#5177 的断言同步均不替代本问题。

对主干的风险

最强风险是短路径别名可能掩盖真实输出增长或长路径命令损坏。新负向用例证明增长仍被拒绝;完整 CLI-budget smoke 保留 base/head、默认/显式模式、规模、语义差异和真实长路径命令完整性检查。alias helper 原有并行目录身份与清理语义未变;没有持久数据或混合版本协议迁移。

本 head 的局部结果:runner/differential 共 71 项通过,最终聚焦 3 项通过,完整 CLI-budget smoke、Ruff lint、diff hygiene 通过。premerge 的 4 个直接检查通过;选出的 12 项首次 11 通过、1 因工作树未安装 TypeScript 开发依赖而失败。执行仓库规定的 npm ci --ignore-scripts 后,仅重跑失败的 semantic-vocabulary smoke,成功退出;初始失败记录保留,没有把它改成通过或放宽检查。远端 CI 按该 Goal 的 wait_for_ci=false 不作查询或等待。

我的整体评价

APPROVE:有真实测量差异、最小既有 owner 修复、修复前失败和修复后正反向证据,没有多余版本分支、并行权威或重复 smoke。长期推进改善,现有用户入口和输出语义保留。Python 仅留在既有测试 harness;不涉及通用能力的 TS 实现迁移。

本 PR 的独立可交付边界是测试入口一致性;后续仍需维护者合入,并让 #5171 以修正后的共同 runner 重新验证,然后单独验收安装与真实到期监控。未据此宣称投研最小闭环完成,也未继承其他 head 的批准。

English verdict: APPROVE - fac0548. Reuses the existing per-scenario fixture alias without changing budgets, workloads, raw stdout or production output. Real-CLI layout and growth-rejection regressions, 71 targeted tests and full budget smoke passed; the initial missing TypeScript dependency check passed after setup recovery. Runtime/monitor adoption remains separate.

@huangruiteng
huangruiteng merged commit 39225c7 into main Sep 27, 2026
22 of 27 checks passed
@huangruiteng
huangruiteng deleted the codex/cli-budget-fixture-parity-20260927 branch September 27, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant