fix(benchmark): isolate native profile runtime temporary paths - #5226
Conversation
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
huangruiteng
left a comment
There was a problem hiding this comment.
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
Reviewed exact head 5226@e8ee52003a7554bd2f5ab2c26d4f67c87335f747.
动机
1.2.2 的并行全量公开验证仍遇到正式安装 profile 清理失败。此前 #5224 增加停止 callback,但 HOME/CODEX_HOME 隔离并不能证明 runtime 独占:现有内容寻址 runtime locator 使用系统临时目录,相同源码 profile 因而能共享 writer;其中一个退出后另一个仍可重新写入已清理目录。原始失败记录保留,维护者已授权本次发行的必要修复和合并。
改动思路
由既有 benchmark-toolkit 原生 Codex profile 安装/环境适配器拥有这项边界。两个现有环境 builder 都将 TMPDIR/TMP/TEMP 指向已存在的私有 HOME,安装和后续 CLI/app-server 采用相同 scope。复用原 TypeScript runtime locator、doctor 停止结果和标准临时目录解析;没有新增状态键、全局环境设置、进程 supervisor 或第二个决策 owner。Python 留在已有专用 host/安装 transport,TypeScript 权威逻辑不变。
具体改动
三个文件、57 行增加和 1 行删除:生产适配器 8 行;一项真实双安装 profile 测试 41 行;现有 provider 文档更新 8 行、替换 1 行。实际相同源码的两个正式安装 profile 分别启动 runtime,停止 first 返回 stopped 且 locator 位于 first HOME,second 仍 running;最后只停止各自 owner。15 项相关测试通过,生产代码在已测试功能提交与最终文档提交之间完全相同;最终干净 head 的正式安装/app-server 技能及 Goal 读回再次通过,无 model turn。对同一个实际已安装 fixture,immutable base 的导出环境使 Python tempdir 位于共享系统目录,最终 head 则位于私有 HOME。Ruff、mypy(19 源文件)、TypeScript typecheck、文档治理、公开边界、精确质量以及全部 19 项 native premerge 通过。原始 benchmark-sensitive 人工门禁不被改写;本评审核对没有评分、task、runner 提交、benchmark 作业或生产 Goal 操作,结合维护者授权满足此人工要求。
对主干的风险
变更仅对显式创建的资格验证 profile 生效;通用 runtime、正常 CLI/App 环境、profile schema、技能内容和权限 owner 均保持原字节。主要风险是安装与后续调用 scope 不一致,双 builder 同步和真实安装测试覆盖了这个路径;停止 pending/失败仍会拒绝,不能伪装清理成功。HOME 由已有安装路径预先创建;临时目录不携带上游凭证。实际模型工作仍需原 OS/provider isolation,环境隔离本身不授予凭证、网络沙箱或执行权限。没有 frontend 配置项或应用 consumer:这是已有显式 provider 适配器派生的环境边界,实际安装 CLI 和 app-server 已验证。此局部通过不代替合入后的全量 CI、真实模型、公开下载及签名验收。
我的整体评价
当前精确 head 可以批准。相关未来改进检查选择直接复用已有 HOME 和两个 builder,没有新增字段或抽象;回归测试保护真实安装后的并行 shutdown ownership。#5224 的串行清理结论需补充这项并行边界修正,最终发行仍在当前任务中继续完成。
English verdict: APPROVE
|
Qualification readback update for the existing native-profile lifecycle owner: The private HOME and runtime-locator scope verified by this PR still holds, but a later Linux full-public attempt on The complete five-shard rerun of the same immutable source passes all 514 public smokes with the original budgets, and eight additional concurrent local runs of the real installed-profile smoke pass. Same-source complete rerun. No cleanup error was ignored, assertion removed, timeout raised, or production Goal changed. This successful rerun qualifies the current source; it does not prove elimination of the intermittent Linux fixture-quiescence race. The remaining maintenance boundary stays with this existing profile lifecycle owner: identify the exact late writer/process under Linux, then prove it is quiescent before removing the owned fixture. The final bilingual release validation preserves the first failure and this limit. |
The formal native Codex profile isolated HOME and CODEX_HOME but left the installer, CLI and app-server on the shared process temporary directory. Equal-source profiles consequently reused one Effect runtime, so stopping one fixture could affect another and its directory could still be written during parallel full-public cleanup.
Bind TMPDIR, TMP and TEMP to the existing profile-owned HOME in both established environment builders. Add a real two-profile install/CLI test that stops one runtime and reads the other as still running; retain the existing explicit runtime teardown and provider/privacy assertions. No scoring, task semantics, model turn or benchmark job changes.
Placement: built-in benchmark-toolkit provider, existing native Codex transport/profile owner. Python continues to adapt the installed host environment; the typed runtime authority and global defaults remain unchanged.
Validation passed on the exact head before authorized merge: immutable base/head temp-scope readback, two real isolated profiles, the formal installed-profile/app-server smoke, focused release qualification tests, static/boundary checks and native premerge.
15 focused tests and actual two-profile runtime/shutdown pass. Final clean-head formal install and Codex app-server skills/Goal readback pass with no model turn. Ruff, configured mypy19, TypeScript typecheck, docs governance, boundary and exact quality pass; all19 native premerge checks pass. The conservative benchmark-path manual review is covered by the published exact-head review and maintainer authorization; original classification remains recorded. No frontend consumer or setting changes: this is derived environment in the existing explicit host/provider adapter. Final merged-source full release qualification remains required.