Skip to content

ci: publish Helm charts in a dedicated job gated on image builds - #7046

Merged
Warashi merged 2 commits into
pipe-cd:masterfrom
TheRealShek:fix/publish-image-chart-race
Sep 6, 2026
Merged

ci: publish Helm charts in a dedicated job gated on image builds#7046
Warashi merged 2 commits into
pipe-cd:masterfrom
TheRealShek:fix/publish-image-chart-race

Conversation

@TheRealShek

Copy link
Copy Markdown
Contributor

What this does

Moves the Helm chart publish out of the artifacts matrix job into a dedicated charts job gated on needs: artifacts, mirroring the structure publish_pipedv1_exp.yaml already uses (container_image_buildimage_chart).

Fixes #6999.

Problem

In publish_image_chart.yaml, the Install helm / Login to OCI using Helm / Publish helm charts steps live inside the artifacts matrix job. That matrix runs ~10 parallel cells, so:

  • Every cell runs Publish helm charts, and each one pushes all three charts (pipecd, piped, helloworld) to the same oci://ghcr.io/pipe-cd/chart ref — up to 10 redundant, concurrent pushes per run (a source of flaky "failed, passed on re-run" CI).
  • A fast cell (e.g. helloworld) publishes the pipecd/piped charts while those images are still building in slower cells → a window where the registry serves a chart whose image tag doesn't exist yet (ImagePullBackOff).
  • If a slow cell then fails, the charts stay published while their images were never pushed — a broken release the failed run can't undo.

Evidence

From run #29642136136 (all on ghcr.io, which the default charts reference):

Matrix cell Publish helm charts done Image push done
ghcr.io/pipe-cd, launcher 11:15:45Z (pushes all 3 charts) 11:15:32Z
ghcr.io/pipe-cd, piped 11:16:27Z 11:16:14Z (+29s after launcher's chart push)
ghcr.io/pipe-cd, pipecd 11:16:33Z 11:16:19Z (+34s after launcher's chart push)

The launcher cell published the piped/pipecd charts ~30s before those images finished pushing to ghcr. (On the gcr mirror the piped image lagged by ~7m44s, affecting installs that override image.repository to gcr.)

Fix

  • New charts job with needs: artifacts — GitHub starts it only after every matrix cell succeeds, so charts publish once, only after all images exist, and never if a build fails.
  • Repoint trigger-event-watcher and release-quickstart-manifests to needs: charts (the latter runs helm template against the just-published chart, so it must wait for the chart push).

Testing

  • actionlint — clean, aside from the pre-existing oracle-vm-8cpu-32gb-x86-64 custom self-hosted runner-label warning on the unchanged artifacts job.
  • YAML validated; job graph confirmed: artifactschartstrigger-event-watcher / release-quickstart-manifests.

Opened as a draft for maintainer feedback.

Move the Helm chart publish out of the artifacts matrix into a separate
`charts` job with `needs: artifacts`, mirroring publish_pipedv1_exp.yaml.
Charts now publish exactly once, only after every image has been pushed,
and a failed image build blocks the publish. Repoint the downstream
trigger-event-watcher and release-quickstart-manifests jobs to needs: charts.

Fixes pipe-cd#6999

Signed-off-by: TheRealShek <shadow.abhi100@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi @TheRealShek, welcome to PipeCD and thanks for opening your first pull request!

We’re really happy to have you here

Before your PR gets merged, please check a few important things below.


Helpful resources


DCO Sign-off

All commits must include a Signed-off-by line to comply with the Developer Certificate of Origin (DCO).

In case you forget to sign-off your commit(s), follow these steps:

For the last commit:

git commit --amend --signoff
git push --force-with-lease

For multiple commits:

git rebase --signoff origin/master
git push --force-with-lease

Run checks locally

Before pushing updates, please run:

make check

This runs the same checks as CI and helps catch issues early.


💬 Need help?

If anything is unclear, feel free to ask in this PR or join us on the CNCF Slack in the #pipecd channel.
You can get your Slack invite from: https://communityinviter.com/apps/cloud-native/cncf

Thanks for contributing to PipeCD! ❤️

@Ayushmore1214

Ayushmore1214 commented Jul 20, 2026

Copy link
Copy Markdown
Member

@TheRealShek Please mind getting assigned before opening a PR, this is okay for now as it is your first contribution here but please from next time be careful!

@github-actions

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale label Aug 30, 2026
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

This PR was closed because it has been stalled for 7 days with no activity. Feel free to reopen if still applicable.

@github-actions github-actions Bot closed this Sep 6, 2026
@Warashi Warashi reopened this Sep 6, 2026
@Warashi Warashi removed the Stale label Sep 6, 2026
@netlify

netlify Bot commented Sep 6, 2026

Copy link
Copy Markdown

Deploy Preview for pipecd-site canceled.

Name Link
🔨 Latest commit 5f83744
🔍 Latest deploy log https://app.netlify.com/projects/pipecd-site/deploys/6a9cbe2237ce0800087aae7b

@Warashi Warashi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for being late, LGTM
thank you!

@Warashi
Warashi enabled auto-merge (squash) September 6, 2026 01:13
@codecov

codecov Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.33%. Comparing base (cc5dd8e) to head (5f83744).

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #7046       +/-   ##
===========================================
+ Coverage   29.75%   43.33%   +13.58%     
===========================================
  Files         601      139      -462     
  Lines       64442    12664    -51778     
===========================================
- Hits        19172     5488    -13684     
+ Misses      43778     6817    -36961     
+ Partials     1492      359     -1133     
Flag Coverage Δ
. ?
.-pkg-app-pipedv1-plugin-analysis 32.16% <ø> (ø)
.-pkg-app-pipedv1-plugin-ecs 33.64% <ø> (ø)
.-pkg-app-pipedv1-plugin-kubernetes 58.39% <ø> (ø)
.-pkg-app-pipedv1-plugin-kubernetes_multicluster ?
.-pkg-app-pipedv1-plugin-scriptrun 54.83% <ø> (ø)
.-pkg-app-pipedv1-plugin-terraform 38.46% <ø> (ø)
.-pkg-app-pipedv1-plugin-wait 33.04% <ø> (ø)
.-pkg-app-pipedv1-plugin-waitapproval 52.71% <ø> (ø)
.-pkg-plugin-sdk 50.12% <ø> (ø)
.-tool-actions-gh-release 19.23% <ø> (ø)
.-tool-actions-plan-preview 25.51% <ø> (ø)
.-tool-codegen-protoc-gen-auth 0.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Warashi
Warashi merged commit 3321879 into pipe-cd:master Sep 6, 2026
66 checks passed
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Thank you for contributing to PipeCD, @TheRealShek! The changes in this pull request will be part of the upcoming release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: publish_image_chart pushes Helm charts from every matrix cell, before all container images exist (race condition)

3 participants