Skip to content

ci(renovate): pilot platform automerge - #58

Merged
edbfi merged 1 commit into
mainfrom
ci/platform-automerge-pilot
Sep 24, 2026
Merged

edbfi merged 1 commit into
mainfrom
ci/platform-automerge-pilot

Conversation

@edbfi

@edbfi edbfi commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Pilot for the automation v4 merge design: Renovate turns on GitHub's native auto-merge for its PRs instead of merging them itself on its next run. GitHub then merges as soon as the required checks pass.

Change

  • renovate.json: platformAutomerge: false → true.
  • Repository setting (already applied): allow_auto_merge is on, which GitHub needs before Renovate can request auto-merge.

Nothing else changes. The required checks (guard, quality / …, smoke / …, ci / required, policy / ci / policy), strict up-to-date branches, the rebase merge strategy and ignoreTests: false all stay as they are.

Pilot plan

This PR records the results. Each scenario must not merge until its condition clears:

  • A required job fails
  • A gated job is skipped (the gate fails)
  • The CI run is cancelled
  • A required job is still pending (merges only after it passes)
  • A required context is missing (the gate's emitted name: is changed)
  • The head changes after checks go green (fresh checks are needed)
  • The branch falls behind main (waits until it's updated and green)
  • A do-not-merge hold is added after policy is green while CI is still pending
  • A review is requested and then removed (Renovate PR, chore(deps): lock file maintenance #67)
  • Changes are requested, and that review is later dismissed (Renovate PR, chore(deps): lock file maintenance #67; merging needed a rerun of the failed pull_request-event policy run, see comment)
  • At least 3 real Renovate PRs merge through auto-merge (5 total: 2 here plus 3 in edbfi/v4-canary; here: chore(deps): lock file maintenance #67, 76 s; chore(deps): lock file maintenance #69, 5 s from green to merge; post-merge CI and deploy passed for both); green-to-merge time is compared with the baseline, and post-merge CI and deploy both run

Human test PRs use ordinary gh pr merge --auto --rebase (no administrator bypass) and are closed afterwards.

@edbfi

edbfi commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Pilot results, part 1: human test PRs (2026-09-24)

Each test PR got auto-merge armed with ordinary gh pr merge --auto --rebase, with no administrator bypass (protection has enforce_admins: false, and the merge was never forced). Each one stayed unmerged until its observation finished, then I disarmed auto-merge and closed it without merging. Nothing reached main.

Scenario PR Evidence Result
Failed required job #61 probe failure → ci / required failure Blocked ✅
Skipped required job #62 probe skipped → gate failure Blocked ✅
Cancelled run #63 run cancelled mid-probe; the gate still ran (if: always()) and failed Blocked ✅
Missing required context #64 gate renamed to ci / required-probe (success); ci / required never reported Blocked ✅
Branch behind main #65 all checks green, based on main~1 → BEHIND Blocked ✅
Head changed after green #66 first head 9b37831 green with auto-merge off; auto-merge then armed with --match-head-commit on the new head, which had to earn its own checks Blocked until fresh checks ✅
Pending required job #66 armed while an 8-minute probe ran → BLOCKED Blocked ✅
Hold after green while CI pending #66 do-not-merge added at 12:19:57 → policy re-ran at 12:19:58 and failed; CI later finished green at 12:27:56 and the PR stayed BLOCKED Blocked ✅

Every required check is emitted by GitHub Actions (app_id 15368).

Still to do, once this PR is merged: the review scenarios on real Renovate PRs (review requested/removed, changes requested, dismissed), plus at least 3 Renovate PRs merging through auto-merge.

Baseline under the current mechanism, from the last 15 Renovate merges here: the median time from green to merge is 219 min; 5 merged within 11 minutes, and 7 waited 4–10 hours for Renovate's next run.

Signed-off-by: edbfi <326875205+edbfi@users.noreply.github.com>
@edbfi
edbfi force-pushed the ci/platform-automerge-pilot branch from 84d9856 to 4f4bb49 Compare September 24, 2026 13:17
@edbfi
edbfi enabled auto-merge (rebase) September 24, 2026 13:17
@edbfi
edbfi merged commit f9d0578 into main Sep 24, 2026
7 checks passed
@edbfi
edbfi deleted the ci/platform-automerge-pilot branch September 24, 2026 13:19
@edbfi

edbfi commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Pilot results, part 2: review scenarios on a real Renovate PR (2026-09-24)

I created Renovate's lock file maintenance PR early from the Dependency Dashboard (it was only waiting for its Monday schedule), which gave #67. Renovate opened it at 13:36:36 and armed GitHub auto-merge itself (enabledBy: renovate, rebase), so platformAutomerge: true works as intended.

Step Event Policy #67
Changes requested (review submitted right after opening) pull_request_review failure BLOCKED, and still blocked after ci / required passed at 13:38:01
Review requested pull_request failure BLOCKED
Review request removed (changes still requested) pull_request failure (correct: the objection was still open) BLOCKED
Review dismissed pull_request_review success still BLOCKED ⚠️
Failed pull_request-event policy run re-run pull_request (re-run) success at 13:51:07 merged at 13:52:23 by renovate

Finding: policy verdicts don't carry across trigger events

GitHub evaluates policy / ci / policy separately for each check suite, and the pull_request and pull_request_review triggers create separate suites. The success from the dismissal didn't replace the earlier failure from the pull_request event, so the combined status stayed FAILURE until the failed pull_request run was re-run.

This fails closed: nothing merges wrongly. It only matters when a pull_request-event policy failure (a label, a review request or a push) happens while an objection is open, and the objection is then withdrawn through a review event. docs/pr-policy.md already advises a rerun in this situation. The automation v4 plan records it as an open design decision.

Timing

  • All required checks green → merged: 76 s (baseline median: 219 min).
  • Real Renovate auto-merges so far: 1 of the ≥3 needed. The rest will come from ordinary Renovate PRs over the next days.

edbfi added a commit to edbfi/automation that referenced this pull request Sep 25, 2026
automerge.json now enables automerge with platformAutomerge, rebase merges and
no wildcard package rule, so the base preset's manual renovate-config rule and
repository opt-outs still win. default.json no longer sets the merge
mechanism. The guides pilot (edbfi/guides#58, #67) verified that every
required-check and policy condition blocks GitHub auto-merge, and the median
wait from green to merge dropped from 219 minutes to seconds.

Add scripts/renovate_effective.mjs, a conservative resolver that reports
automerge as potentially enabled whenever any automerge: true remains and
treats an omitted platformAutomerge as Renovate's default, true.

BREAKING CHANGE: automerge.json no longer opts renovate-config updates in and
now uses platform automerge; consumers need Allow auto-merge enabled.

Signed-off-by: edbfi <326875205+edbfi@users.noreply.github.com>
@edbfi

edbfi commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

Pilot results, part 3: Renovate auto-merges (2026-09-25)

PR Armed by ci / required green Merged Green → merge Post-merge ci (push) Pages deploy
#67 lock file maintenance renovate — 2026-09-24 13:52:23Z 76 s success success
#69 lock file maintenance renovate 2026-09-25 08:54:28Z 2026-09-25 08:54:33Z 5 s success success

#69 had no review activity: Renovate opened it, armed auto-merge, and GitHub merged 5 s after the last required check (ci / required, app 15368) passed. policy / ci / policy (app 15368) passed at 08:53:27Z. Nobody touched it.

Count: 2 of 3. The third should be renovate/all-minor-patch (astro, Starlight, MDX). It is waiting for minimumReleaseAge, and its oldest eligible versions pass three days at about 19:00Z today. It isn't being forced.

Not counted: #26 (major-typescript, auto-merge armed but CI fails, so it's correctly blocked) and #59 (shared-policy update, manual by design).

edbfi added a commit to edbfi/automation that referenced this pull request Sep 25, 2026
automerge.json now enables automerge with platformAutomerge, rebase merges and
no wildcard package rule, so the base preset's manual renovate-config rule and
repository opt-outs still win. default.json no longer sets the merge
mechanism. The guides pilot (edbfi/guides#58, #67) verified that every
required-check and policy condition blocks GitHub auto-merge, and the median
wait from green to merge dropped from 219 minutes to seconds.

Add scripts/renovate_effective.mjs, a conservative resolver that reports
automerge as potentially enabled whenever any automerge: true remains and
treats an omitted platformAutomerge as Renovate's default, true.

BREAKING CHANGE: automerge.json no longer opts renovate-config updates in and
now uses platform automerge; consumers need Allow auto-merge enabled.

Signed-off-by: edbfi <326875205+edbfi@users.noreply.github.com>
edbfi added a commit to edbfi/automation that referenced this pull request Sep 25, 2026
automerge.json now enables automerge with platformAutomerge, rebase merges and
no wildcard package rule, so the base preset's manual renovate-config rule and
repository opt-outs still win. default.json no longer sets the merge
mechanism. The guides pilot (edbfi/guides#58, #67) verified that every
required-check and policy condition blocks GitHub auto-merge, and the median
wait from green to merge dropped from 219 minutes to seconds.

Add scripts/renovate_effective.mjs, a conservative resolver that reports
automerge as potentially enabled whenever any automerge: true remains and
treats an omitted platformAutomerge as Renovate's default, true.

BREAKING CHANGE: automerge.json no longer opts renovate-config updates in and
now uses platform automerge; consumers need Allow auto-merge enabled.

Signed-off-by: edbfi <326875205+edbfi@users.noreply.github.com>
edbfi added a commit to edbfi/automation that referenced this pull request Sep 25, 2026
automerge.json now enables automerge with platformAutomerge, rebase merges and
no wildcard package rule, so the base preset's manual renovate-config rule and
repository opt-outs still win. default.json no longer sets the merge
mechanism. The guides pilot (edbfi/guides#58, #67) verified that every
required-check and policy condition blocks GitHub auto-merge, and the median
wait from green to merge dropped from 219 minutes to seconds.

Add scripts/renovate_effective.mjs, a conservative resolver that reports
automerge as potentially enabled whenever any automerge: true remains and
treats an omitted platformAutomerge as Renovate's default, true.

BREAKING CHANGE: automerge.json no longer opts renovate-config updates in and
now uses platform automerge; consumers need Allow auto-merge enabled.

Signed-off-by: edbfi <326875205+edbfi@users.noreply.github.com>
@edbfi

edbfi commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

Pilot results, part 4: decision and release (2026-09-25)

Rather than wait days for natural updates, the owner chose to create the remaining conditions in a disposable consumer, edbfi/v4-canary, running the v4 code. There, three more real Renovate PRs auto-merged 7–28 s after going green, with post-merge push CI green each time (details in edbfi/automation#62). Together with #67 and #69 here, that is 5 real Renovate auto-merges in total.

Decision: platformAutomerge: true (GitHub auto-merge, rebase) is the single v4 mechanism, released as automation v4.0.0.

@edbfi

edbfi commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

Canary result under v4 (2026-09-25)

After the v4 migration (#70), the natural astro/Starlight/MDX update #72 was armed by Renovate and merged 3 s after ci / required went green. Post-merge CI and the Pages deploy both passed. Both guides canary gates are met: a human PR (#70) and a Renovate PR (#72) each auto-merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant