ci: fix PR-check bison download failures — deps cache version mismatch, GNU mirror, fail-fast wget - #18
Open
rjd15372 wants to merge 3 commits into
Open
ci: fix PR-check bison download failures — deps cache version mismatch, GNU mirror, fail-fast wget#18rjd15372 wants to merge 3 commits into
rjd15372 wants to merge 3 commits into
Conversation
…heck actions/cache hashes the 'path' list into the cache *version* and only matches restore-keys against entries with the same version. sync-main (b43d0f2) added .cache/sync_state to its deps cache while obs-pr-check kept three paths, so the two workflows have been writing incompatible versions ever since: PR runs reported 'Cache not found for input keys: ..., percona-obs-deps-Linux-' even though main's deps cache existed and its scm cache (identical path in both) restored fine. The practical effect was that every PR without a deps cache of its own re-downloaded every download_url source on each run, and could only ever build one after a fully successful run -- actions/cache does not save after a failed job. That turned an unreachable ftp.gnu.org into a loop of 90-minute failures. Add .cache/sync_state to obs-pr-check's list (the manifest is an optimisation keyed by apiurl/rootprj and PR runs never pass --skip-unchanged, so restoring it is inert) and leave a note in both files that the lists must stay identical. download_url entries remain content-addressed by their service params, so a PR that bumps a version still misses and re-downloads exactly that source. Also give the PR-check sync job a timeout-minutes: 120 backstop; a healthy run finishes in well under 30 minutes, and until now a stalled service could hold the job for the 6-hour default. Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
ftp.gnu.org throttles/refuses connections from cloud IP ranges; GitHub Actions runners currently get 'Connection timed out' and then 'Connection refused' on 209.51.188.20:443 while the same URL answers instantly from elsewhere. bison is the only package in the tree that downloads from ftp.gnu.org. Use the GNU redirector instead: https://ftpmirror.gnu.org/bison/... 302s to a nearby mirror and wget follows the redirect (verified with both plain wget and the obs download_url service). The service params change, so the download_url cache key changes and the tarball is fetched once more, then cached. Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
The obs download_url service execs wget with its defaults -- 20 tries and a multi-minute connect timeout each -- and exposes no retry or timeout knobs. A host that refuses the runner's IP (ftp.gnu.org this week) therefore costs ~45 minutes per attempt, doubled by the content check's retry-once, before the job even fails. Export WGETRC from the shared obs-setup action with tries=3, connect_timeout=30 and read_timeout=120. Every workflow that runs services goes through obs-setup, and the venv inherits the job env, so each service-spawned wget picks it up. Measured with the real service against a black-hole address: gives up after 2 min 13 s. Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-check runs on several branches (Q3release, Q3PG15, tarballs-simpleimage, next-train-bumps) have been failing after ~90 minutes in the
--branch-fromcontent check forppg:common:deps/bison:The same URL answers in 0.3 s from outside GitHub; ftp.gnu.org is refusing the runners' IP ranges (later runs got
Connection refused). Three independent fixes:1. Deps cache was invisible to PR runs (cache version mismatch)
sync-main.ymlcaches four paths (download_url,cargo_vendor,services,sync_state— the last added in b43d0f2);obs-pr-check.ymlcached three. actions/cache hashes the path list into the cache version and restore-keys only match the same version, so PR runs loggedCache not found for input keys: …, percona-obs-deps-Linux-while restoring main's scm cache (identical path) just fine. Confirmed via the caches API: main's deps entry has version04cde50b…, PR #12'seefd799a….Consequence: a PR without its own deps cache re-downloaded every
download_urlsource on every run, and could only build a cache after a fully successful run (actions/cache does not save on failure) — i.e. never, while ftp.gnu.org blocks runners.Fix: add
.cache/sync_stateto obs-pr-check's list and note in both files that the lists must stay identical. Restoring the manifest in PR runs is inert (keyed by apiurl/rootprj, and PRs don't pass--skip-unchanged).download_urlentries stay content-addressed by their service params, so a PR that bumps bison still misses and downloads exactly that tarball. Also addstimeout-minutes: 120to the sync job (healthy runs finish in <30 min; default was 6 h).2. bison: fetch via
ftpmirror.gnu.orgThe GNU redirector 302s to a nearby mirror;
wgetfollows it. Verified end-to-end withwgetand thedownload_urlservice locally. bison is the only package pulling from ftp.gnu.org. (New service params ⇒ new cache key ⇒ one fresh download, then cached.)3. Fail fast on unreachable download hosts
obs-setupnow exportsWGETRCwithtries = 3,connect_timeout = 30,read_timeout = 120. Thedownload_urlservice execswgetwith defaults (20 tries, multi-minute connect timeout) and exposes no knobs. Measured with the real service against a black-hole address: 2 min 13 s to give up instead of ~45 min (×2 with the content-check retry).No Python changes.