build: keep two progress pushes from clobbering each other - #244
Merged
Conversation
2026-08-12 那轮 stable 从建置到发布全部成功，257 个包已经上镜像机，服务却报 失败，日志只有一句 `mv: cannot stat '/srv/mirrors/.build-status.json.new'`。 两个缺陷叠在一起。 一是临时档名固定。看守进程每 60 秒推一次进度，`on_exit` 里的 `kill` 只打倒看守 进程本身，它正在执行的 ssh 被 init 收养后继续跑；这时 `finish` 又开一条 ssh， 两条都往同一个 `.build-status.json.new` 写。先改名的那条把档案搬走， 后改名的那条 拿到 ENOENT，而落地的那份是两次写入交叠的结果——`build-status.json` 从那一刻起 就不是合法 JSON，首页 stable 的构建状态区块因此空了 14 小时。改成每次推送各自 `mktemp`，并在推送前 `chmod 644`。 二是 `set -e` 之下 EXIT trap 里的失败会变成整个脚本的退出码。一次进度回报失败 不该把已经建好、已经发布的一轮说成失败——真出问题时站点会停在 running， `ops/status.sh` 本来就盯着这个。 看守进程另外改用自己的行程组，`kill` 一并收掉它在途的 ssh，否则那条 ssh 会把 running 的快照盖在最终状态上面。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2026-08-12 那轮 stable 从建置到发布全部成功，257 个包已经上镜像机，服务却报失败，日志只有一句
mv: cannot stat '/srv/mirrors/.build-status.json.new'。两个缺陷
临时档名固定。 看守进程每 60 秒推一次进度，
on_exit的kill只打倒看守进程本身，它在途的 ssh 被 init 收养后继续跑；这时finish又开一条 ssh，两条都写同一个.build-status.json.new。先改名的把档案搬走，后改名的拿到 ENOENT，而落地的那份是两次写入交叠的结果——build-status.json从那一刻起就不是合法 JSON，首页 stable 的构建状态区块空了 14 小时（线上档案已单独修复）。set -e之下 EXIT trap 里的失败会变成脚本退出码。 一次进度回报失败把已经建好、已经发布的一轮说成失败。真出问题时站点会停在 running，ops/status.sh本来就盯着这个。改动
build-progress.sh：每次推送各自mktemp，推送前chmod 644，失败时清掉临时档。cycle.sh：finish的推送失败不再改变退出码；看守进程改用自己的行程组，kill一并收掉在途的 ssh。测试
新增
tests/test-progress-push.sh（7 项，已接入 CI）：ssh stub 真的执行远端那串命令，mv包一层延迟，把原本百分之一的碰撞变成每次必现。tests/test-cycle.sh从 9 项加到 16 项。变异反证 5 项全部转红：换回固定临时档名、拿掉
chmod、finish失败重新致命、看守进程丢掉频道输出档名、换回只杀看守进程本身。