Repository navigation
chore(ci): bound workflow job runtimes - #662
Conversation
vladimirrott
left a comment
There was a problem hiding this comment.
Thanks, @dannyward630. The contract test is the part I'd have asked for: it walks every workflow file, skips uses: jobs, and proves on its own fixture that it names a missing job before it checks the real ones. Reviewed at edaf46a. The PR touches workflow files, so I ran nothing on the host; everything below ran in an offline container against a copy of the tree.
One blocking item: release.yml build needs more than 20 minutes. You sized the release limits from the latest release run, which is a fair sample to start from. I pulled job durations for the last 40 release runs (80 build jobs) and one of them ran past 20:
32266031526 Build x86_64-unknown-linux-gnu success 2026-08-19T14:46:51Z 20.4 min
35707143799 Build aarch64-unknown-linux-gnu success 2026-09-22T08:52:28Z 6.9 min
That was the v0.9.0 release. Its step timings were 16.1min Install system dependencies and 3.7min Build release binaries: a slow apt mirror, then a normal build. The apt-get update / apt-get install -y cmake step is still in the job, so with timeout-minutes: 20 the same mirror stall would cancel a tagged release. Nothing gets half-published (the publish jobs wait on build), but someone has to notice and rerun it. Setting build to 30, the same as publish-crates, covers that run with room to spare.
The other limits hold up against the same history. Longest observed job per workflow over the last 30 to 40 runs, against the cap you set:
ci.yml rust 9.7 / 20 docs-and-hygiene 4.8 / 15 postgres-contract 2.1 / 10
codeql.yml Analyze 11.2 / 20
secret-scan trufflehog 10.2 / 15
release.yml crates.io 8.4 / 30 npm 2.1 / 15 MCP publish 5.5 / 15
rehearsal 8.1 / 45
What I ran on the test itself, each mutation on a fresh copy:
pristine rc=0 workflow-timeouts: checked all jobs in 12 workflow files
drop rust timeout (ci.yml) rc=1 FAIL: workflow jobs missing timeout-minutes: ci.yml/rust
drop release build timeout rc=1 FAIL: workflow jobs missing timeout-minutes: release.yml/build
drop pre-existing npm-audit timeout rc=1 FAIL: workflow jobs missing timeout-minutes: npm-audit.yml/npm-audit
check ignores missing key (always pass) rc=1 RuntimeError: fixture expected ['fixture.yml/fast'], got []
check ignores uses-skip rc=1 RuntimeError: fixture expected ['fixture.yml/fast'], got ['fixture.yml/fast', 'fixture.yml/reusable']
no workflow files rc=1 RuntimeError: no workflow files found under /r/.github/workflows
unparseable workflow rc=1 RuntimeError: cannot read /r/.github/workflows/zz.yml: while parsing a flow node
With your test kept and main's workflows put back, it fails and lists 21 jobs, the same 21 your summary counts. scripts/check_test_reachability.sh passes on the branch, and shellcheck at --severity=warning is clean on the new script.
One optional note: a key with no value (timeout-minutes:) still passes, because the check tests for the key and never reads the value. Checking isinstance(job["timeout-minutes"], int) would close it. I'd merge without it.
I read the workflow diff in full: 21 timeout-minutes lines and one new run: step in docs-and-hygiene, no trigger, permission or pin change. Because it touches .github/workflows/, I'll do the merge myself once the build limit moves.
|
Addressed the blocking timeout feedback in Local checks passed: |
|
Update: all required GitHub Actions passed on head |
vladimirrott
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround on the build cap. The release build job at 30 minutes covers the 20.4-minute run with room, and the contract test now rejects blank, string, boolean, zero and negative values, which covers the values I was worried about.
One more cap needs the same treatment, and I should have caught it in my first pass: ci.yml rust at 20 minutes. I pulled the rust job times from the last 87 completed ci.yml runs: median 3.7 min, p95 7.5 min, and three runs above 18 minutes, the slowest at 18.8. Those are cold-cache builds, and they would have passed with 72 seconds to spare. One slightly slower runner and every open PR gets a cancelled rust check, which reads like a test failure to a contributor.
Please raise rust to 30, the same reasoning as build. Everything else checks out against the history:
| job | cap | slowest observed |
|---|---|---|
codeql Analyze (rust) |
20 | 10.1 |
release Publish to crates.io |
30 | 8.4 |
docs-and-hygiene |
15 | 4.8 |
postgres-contract |
10 | 3.0 |
Rehearse x86_64 |
45 | 8.1 |
Once rust moves I'll approve and merge.
…fix/629-workflow-timeouts
Across 87 completed ci.yml runs the rust job took a median 3.7 min and p95 7.5 min, but three cold-cache runs exceeded 18 minutes (slowest 18.8). A 20-minute cap left 72 seconds of headroom and would cancel the rust check on every open PR after one slow runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU
vladimirrott
left a comment
There was a problem hiding this comment.
Approving at 1bf2b6ef. I pushed the one remaining item myself so this could land today: rust gets 30 minutes, the same reasoning as build (three of 87 runs took more than 18 minutes). I also merged current main into the branch. Your contract test passes with it across all 12 workflow files.
I read the final workflow diff in full: timeout-minutes lines and the one new step that runs your test, nothing else. Thanks for this one, Danny; a hung job holding a runner for six hours is the kind of failure nobody notices until every other job queues behind it.
Minor release. #627 keys request_hash with an HMAC under the audit key (chain_version 4; previews stored by an older release fail closed after upgrading) and #639 changes the history request a newer CLI sends, so a caller relying on the old behaviour notices: the middle digit moves. Also: the planner sees only its host's distribution (#626, #656, #658, #684), checkpoint and the MCP audit tools share the audit-store resolver (#636), every workflow job has a time limit (#662), a weekly cargo audit (#640), release hardening (#635, #637, #660), and every published figure is rendered from its source (#685). Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Closes #629
Summary
timeout-minutesto all 24 timeout-capable jobs across the current workflows. The reusablepublish-registrycaller is skipped because GitHub does not allow a job-level timeout there; the calledpublishjob inpublish-mcp.ymlhas one..ymland.yamlworkflow, skips jobs withuses:, and rejects missing, blank, non-integer, boolean, or non-positive timeout values. The test checks its own fixtures and runs indocs-and-hygiene.cargo-audit.ymljob merged in ci: add scheduled Cargo audit workflow (#622) #640. The release build timeout is now 30 minutes: review found a historical 20.4-minute build, including a 16.1-minute system-dependency install.Validation
bash tests/release/workflow-timeouts.test.sh— passed; checked all 12 workflow files and the missing/invalid-value fixtures.bash scripts/check_test_reachability.sh— passed.yamllint .github/workflows/release.yml— passed with yamllint 1.37.1. The local package index does not provide the repository-pinned 1.38.0.git diff --check— passed.6da4a33; the manualcontainer-smokejob was skipped by its workflow conditions.Local validation limits
The full
scripts/ci-local.shand Rust pre-commit gate were not run on this Mac. The repository YAML wrapper usesmapfile, which the installed Bash 3.2 lacks; I linted the changed workflow directly. CI is running on the updated head.AI assistance: Codex (GPT-6) contributed the test and workflow edits.