Repository navigation
👷 fix: Build the arm64 Image on a Native Runner So Publishing Stops Stalling Under QEMU - #206
Merged
Merged
Conversation
…talling Under QEMU The publish workflow built linux/amd64 and linux/arm64 in one job on an amd64 runner, emulating arm64 with QEMU. bun install stalls there: one run sat in "Build and push" for over two hours, and the next one's arm64 `bun install --frozen-lockfile` went silent for 13 minutes (848.7s, against 26.5s the run before, on the same BuildKit and binfmt images). Each architecture now builds on a runner of that architecture (ubuntu-24.04-arm for arm64, free for public repositories) and pushes an untagged digest; a merge job then joins the two digests into the tagged multi-platform image with `docker buildx imagetools create`, as LibreChat's own publish workflow does. The merge refuses to publish unless both architectures built (exactly two digests), and a failing leg no longer cancels the other one, so "Re-run failed jobs" has the healthy leg's digest to reuse (kept for 7 days). `latest` is now decided in the merge job, right before it is pushed. The check is the same (this commit is on main and no later commit changed the build context), but every re-run of the merge repeats it: GitHub reuses a successful job's outputs when failed jobs are re-run, so a decision taken at the start of the run could move `latest` back after newer images shipped. Releases no longer tag `latest` either: metadata-action's default `flavor: latest=auto` added it to every semver release, skipping the check; `latest=false` leaves the merge as its only source. A publishing run that computes no tags now fails instead of pushing nothing (a release tag that isn't semver, say), and a manual run on a branch other than main builds both architectures without pushing, as a dry run. The image's `org.opencontainers.image.version` label is now the commit sha on main (it was `latest`, a tag that moves) and the version on releases, as before.
Contributor
Author
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Summary
The publish workflow built
linux/amd64andlinux/arm64in one job on an amd64 runner, emulating arm64 with QEMU.bun installstalls under that emulation:lateston Main's Newest Image #205's merge published, but its arm64bun install --frozen-lockfilewent silent for 13 minutes: 848.7s, against 26.5s in the run before, on the same BuildKit and binfmt images. Its arm64bun installin the next stage took 29.8s.#205's 30-minute timeout stops a stall from holding the publish queue for 6 hours, but a stalled run still publishes nothing.
Native runners. Each architecture now builds on a runner of that architecture:
ubuntu-latestfor amd64 andubuntu-24.04-armfor arm64, which is free for public repositories. This is the pattern LibreChat's owndocker-publish.ymlmoved to for the same reason; it measured QEMU 5–6x slower on its images.docker/metadata-action, so both architectures carry the same labels.push-by-digest=true,name-canonical=true), then upload the digest as an artifact.docker buildx imagetools create. The image has the same shape as today'slatest: an index with the amd64 and arm64 images and their two provenance attestations. One of the reviewers compared it with LibreChat's merged images.No partial images.
fail-fast: false: a failing leg no longer cancels the healthy one, so "Re-run failed jobs" can reuse the healthy leg's digest. The digest artifacts are kept for 7 days (each is one empty file).latestis decided right before it is pushed. The rule from #205 is unchanged:latestis tagged only when this commit is onmainand no later commit onmainchanged the build context. But the check now runs in the merge job, just beforeimagetools create, not at the start of the run.prepareoutput, re-running a failed arm64 leg after newer images shipped would have reused the oldlatest=trueand movedlatestback.mainmoved on publishes only the commit'sshatag.queue: maxfrom ⬆️ chore: Move the Workflows to the Current Major Actions, Publish on Any Build Input and Keeplateston Main's Newest Image #205), and the group is held for all three jobs, so no other run onmaincan publish between the check and the push.Releases no longer tag
latest. This was already happening onmain.docker/metadata-actiondefaults toflavor: latest=auto, which addslatestto every non-prerelease semver tag, skipping the head check. A release cut from an older commit, such as a backport, would movelatestback. A release run also isn't queued behindmain's runs, since its ref is the tag. Withflavor: latest=false, the head check in the merge is the only waylatestis tagged, andmain's push run for the released commit still tags it.Runs that can't publish.
v1.1, say) gave no tags.push: truewithout tags made buildx fail, but only after building both platforms.mainhas no tags (theshatag is for the default branch). It now builds both architectures without logging in or pushing, and skips the merge: a dry run for testing Dockerfile or workflow changes.One label changes.
org.opencontainers.image.versionis now the commit sha onmain. It waslatest, the first tag, which stops describing the image as soon aslatestmoves on. Releases still get their version.Change Type
Testing
I ran the merge step's exact script under
bash -eo pipefailwith a stubdockerthat prints its arguments:latest=trueimagetools createwith theshatag and:latestlatest=falseshatag only1.2.0,1.2,1)latestThe head check is ⬆️ chore: Move the Workflows to the Current Major Actions, Publish on Any Build Input and Keep
lateston Main's Newest Image #205's script, moved into the merge job unchanged. ⬆️ chore: Move the Workflows to the Current Major Actions, Publish on Any Build Input and Keeplateston Main's Newest Image #205 has its 19-case table.The export step fails when the build reports no digest, and writes the digest file otherwise.
The workflow parses as YAML, and the parsed jobs,
needs,ifconditions, outputs andfail-fastare as intended.Two independent reviews read the workflow against GitHub's and the actions' documentation and source: one for Actions and buildx semantics, one for failure modes, re-runs and security. Their findings are all in this version: the re-run
latestproblem, the releaselatest=autoproblem, failing closed on missing tags,fail-fastand artifact retention. They also checked that everyrun:block reads only environment variables (no${{ }}interpolation), thatpackages: writeis granted only to the two jobs that push, and that no job output carries a secret.I ran the workflow by hand on this PR's branch (run 40678304438). On a branch other than
mainthat is a dry run, and it passed:push=false; registry login, digest export and upload skippedbun installandbun run buildran natively; the build stage took about 40s, against about 140s under QEMUpublishfalse)Nothing was pushed to GHCR.
On merge, I'll watch the first
mainrun and confirm that it pushes theshatag andlatestas a two-platform image.Checklist