Skip to content

chore(ci): bound workflow job runtimes - #662

Merged
vladimirrott merged 6 commits into
lacs-project:mainfrom
dannyward630:fix/629-workflow-timeouts
Oct 27, 2026
Merged

vladimirrott merged 6 commits into
lacs-project:mainfrom
dannyward630:fix/629-workflow-timeouts

Conversation

@dannyward630

@dannyward630 dannyward630 commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Closes #629

Summary

  • Add timeout-minutes to all 24 timeout-capable jobs across the current workflows. The reusable publish-registry caller is skipped because GitHub does not allow a job-level timeout there; the called publish job in publish-mcp.yml has one.
  • Add a PyYAML contract test that scans every .yml and .yaml workflow, skips jobs with uses:, and rejects missing, blank, non-integer, boolean, or non-positive timeout values. The test checks its own fixtures and runs in docs-and-hygiene.
  • Include the cargo-audit.yml job merged in ci: add scheduled Cargo audit workflow (#622) #640. The release build timeout is now 30 minutes: review found a historical 20.4-minute build, including a 16.1-minute system-dependency install.

Validation

  • bash tests/release/workflow-timeouts.test.sh — passed; checked all 12 workflow files and the missing/invalid-value fixtures.
  • bash scripts/check_test_reachability.sh — passed.
  • yamllint .github/workflows/release.yml — passed with yamllint 1.37.1. The local package index does not provide the repository-pinned 1.38.0.
  • git diff --check — passed.
  • All required GitHub Actions passed at head 6da4a33; the manual container-smoke job was skipped by its workflow conditions.

Local validation limits

The full scripts/ci-local.sh and Rust pre-commit gate were not run on this Mac. The repository YAML wrapper uses mapfile, which the installed Bash 3.2 lacks; I linted the changed workflow directly. CI is running on the updated head.

AI assistance: Codex (GPT-6) contributed the test and workflow edits.

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @dannyward630. The contract test is the part I'd have asked for: it walks every workflow file, skips uses: jobs, and proves on its own fixture that it names a missing job before it checks the real ones. Reviewed at edaf46a. The PR touches workflow files, so I ran nothing on the host; everything below ran in an offline container against a copy of the tree.

One blocking item: release.yml build needs more than 20 minutes. You sized the release limits from the latest release run, which is a fair sample to start from. I pulled job durations for the last 40 release runs (80 build jobs) and one of them ran past 20:

32266031526  Build x86_64-unknown-linux-gnu  success  2026-08-19T14:46:51Z  20.4 min
35707143799  Build aarch64-unknown-linux-gnu success  2026-09-22T08:52:28Z   6.9 min

That was the v0.9.0 release. Its step timings were 16.1min Install system dependencies and 3.7min Build release binaries: a slow apt mirror, then a normal build. The apt-get update / apt-get install -y cmake step is still in the job, so with timeout-minutes: 20 the same mirror stall would cancel a tagged release. Nothing gets half-published (the publish jobs wait on build), but someone has to notice and rerun it. Setting build to 30, the same as publish-crates, covers that run with room to spare.

The other limits hold up against the same history. Longest observed job per workflow over the last 30 to 40 runs, against the cap you set:

ci.yml       rust 9.7 / 20   docs-and-hygiene 4.8 / 15   postgres-contract 2.1 / 10
codeql.yml   Analyze 11.2 / 20
secret-scan  trufflehog 10.2 / 15
release.yml  crates.io 8.4 / 30   npm 2.1 / 15   MCP publish 5.5 / 15
rehearsal    8.1 / 45

What I ran on the test itself, each mutation on a fresh copy:

pristine                                             rc=0  workflow-timeouts: checked all jobs in 12 workflow files
drop rust timeout (ci.yml)                           rc=1  FAIL: workflow jobs missing timeout-minutes:   ci.yml/rust
drop release build timeout                           rc=1  FAIL: workflow jobs missing timeout-minutes:   release.yml/build
drop pre-existing npm-audit timeout                  rc=1  FAIL: workflow jobs missing timeout-minutes:   npm-audit.yml/npm-audit
check ignores missing key (always pass)              rc=1  RuntimeError: fixture expected ['fixture.yml/fast'], got []
check ignores uses-skip                              rc=1  RuntimeError: fixture expected ['fixture.yml/fast'], got ['fixture.yml/fast', 'fixture.yml/reusable']
no workflow files                                    rc=1  RuntimeError: no workflow files found under /r/.github/workflows
unparseable workflow                                 rc=1  RuntimeError: cannot read /r/.github/workflows/zz.yml: while parsing a flow node

With your test kept and main's workflows put back, it fails and lists 21 jobs, the same 21 your summary counts. scripts/check_test_reachability.sh passes on the branch, and shellcheck at --severity=warning is clean on the new script.

One optional note: a key with no value (timeout-minutes:) still passes, because the check tests for the key and never reads the value. Checking isinstance(job["timeout-minutes"], int) would close it. I'd merge without it.

I read the workflow diff in full: 21 timeout-minutes lines and one new run: step in docs-and-hygiene, no trigger, permission or pin change. Because it touches .github/workflows/, I'll do the merge myself once the build limit moves.

@dannyward630

Copy link
Copy Markdown
Contributor Author

Addressed the blocking timeout feedback in 6da4a33, which also brings the branch up to current main. The release build job now has a 30-minute limit to cover the observed 20.4-minute run. I also strengthened the contract test so blank, string, boolean, zero, and negative values fail while reusable-workflow callers remain skipped.

Local checks passed: bash tests/release/workflow-timeouts.test.sh, bash scripts/check_test_reachability.sh, yamllint .github/workflows/release.yml (1.37.1), and git diff --check. The current-head GitHub Actions are running now; I updated the PR description with these results and the local Bash limitation.

@dannyward630

Copy link
Copy Markdown
Contributor Author

Update: all required GitHub Actions passed on head 6da4a33; the manual container-smoke job was skipped by its workflow conditions. The review changes are addressed and the PR is ready for maintainer re-review.

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on the build cap. The release build job at 30 minutes covers the 20.4-minute run with room, and the contract test now rejects blank, string, boolean, zero and negative values, which covers the values I was worried about.

One more cap needs the same treatment, and I should have caught it in my first pass: ci.yml rust at 20 minutes. I pulled the rust job times from the last 87 completed ci.yml runs: median 3.7 min, p95 7.5 min, and three runs above 18 minutes, the slowest at 18.8. Those are cold-cache builds, and they would have passed with 72 seconds to spare. One slightly slower runner and every open PR gets a cancelled rust check, which reads like a test failure to a contributor.

Please raise rust to 30, the same reasoning as build. Everything else checks out against the history:

job cap slowest observed
codeql Analyze (rust) 20 10.1
release Publish to crates.io 30 8.4
docs-and-hygiene 15 4.8
postgres-contract 10 3.0
Rehearse x86_64 45 8.1

Once rust moves I'll approve and merge.

Vladimir Rotariu and others added 3 commits October 27, 2026 07:49
Across 87 completed ci.yml runs the rust job took a median 3.7 min and
p95 7.5 min, but three cold-cache runs exceeded 18 minutes (slowest
18.8). A 20-minute cap left 72 seconds of headroom and would cancel the
rust check on every open PR after one slow runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 1bf2b6ef. I pushed the one remaining item myself so this could land today: rust gets 30 minutes, the same reasoning as build (three of 87 runs took more than 18 minutes). I also merged current main into the branch. Your contract test passes with it across all 12 workflow files.

I read the final workflow diff in full: timeout-minutes lines and the one new step that runs your test, nothing else. Thanks for this one, Danny; a hung job holding a runner for six hours is the kind of failure nobody notices until every other job queues behind it.

@vladimirrott
vladimirrott merged commit 4bca841 into lacs-project:main Oct 27, 2026
13 checks passed
vladimirrott added a commit that referenced this pull request Oct 27, 2026
Minor release. #627 keys request_hash with an HMAC under the audit key
(chain_version 4; previews stored by an older release fail closed after
upgrading) and #639 changes the history request a newer CLI sends, so a
caller relying on the old behaviour notices: the middle digit moves.

Also: the planner sees only its host's distribution (#626, #656, #658,
#684), checkpoint and the MCP audit tools share the audit-store resolver
(#636), every workflow job has a time limit (#662), a weekly cargo audit
(#640), release hardening (#635, #637, #660), and every published figure
is rendered from its source (#685).


Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

21 of 24 workflow jobs have no timeout-minutes, so a hanging test holds CI for six hours

2 participants