USHIFT-6810: Add C2CC upgrade test for RHEL 9.8 to RHEL 10.2

[vimauro]

Summary by CodeRabbit

• Tests
  • Added BOOTC in-place upgrade Robot suite for three RHEL clusters with full connectivity checks and booted image digest change validation.
  • Enhanced remote cluster health verification (state checks + healthy waits) and improved teardown to restore local SSH connection status handling.
  • Streamlined probe health checks to reuse shared “all remote clusters healthy” logic.
• Build/CI & Presubmit
  • Added RHEL98-src → RHEL102-src upgrade presubmit with fixed execution settings and upgrade orchestration.
  • Test runner now forwards optional target and registry variables from the environment.
• Container Images
  • Enabled GA repository selection and added sshd host key permission fix during builds; minor cleanup in one containerfile.

[openshift-ci-robot]

@vimauro: This pull request references USHIFT-6810 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2f29046c-ff0e-41dc-a49f-55717315d4ef

📥 Commits

Reviewing files that changed from the base of the PR and between 89c47c2 and d5bc734.

📒 Files selected for processing (1)

• test/image-blueprints-bootc/el10/layer1-base/group1/rhel102-test-agent.containerfile

✅ Files skipped from review due to trivial changes (1)

• test/image-blueprints-bootc/el10/layer1-base/group1/rhel102-test-agent.containerfile

---

Walkthrough

Adds RHEL9-to-RHEL10 C2CC upgrade presubmit test with new Robot Framework suite that sequentially upgrades three clusters and verifies health and connectivity. Supporting changes include GA repository enablement in container templates, SSH host key permission fixes via sshd drop-in, TARGET_REF and BOOTC_REGISTRY variable forwarding in c2cc_common.sh, and remote cluster state verification helpers moved to shared resources.

Changes

C2CC RHEL9→RHEL10 Upgrade Test Implementation

Layer / File(s)	Summary
Container image preparation: GA repos and SSH host key fix test/image-blueprints-bootc/templates/rpm-repo-config.sh.template, test/image-blueprints-bootc/el10/layer2-presubmit/group1/rhel102-bootc-source.containerfile, test/image-blueprints-bootc/el10/layer1-base/group1/rhel102-test-agent.containerfile	Template adds enable_ga_repositories() function and --enable-ga argument parsing to enable baseos/appstream repos by OS version and architecture. rhel102-bootc-source adds sshd systemd drop-in to chmod 600 SSH host keys before service start.
Test variable forwarding in c2cc_common.sh test/bin/c2cc_common.sh	c2cc_run_tests conditionally declares and appends --variable TARGET_REF and --variable BOOTC_REGISTRY arguments when C2CC_TARGET_REF and MIRROR_REGISTRY_URL environment variables are set.
Shared remote cluster state verification test/resources/c2cc.resource, test/suites/c2cc/probe.robot	c2cc.resource adds Verify RemoteCluster State and Verify All RemoteClusters Healthy keywords; probe.robot removes its local Verify RemoteCluster State implementation and delegates to shared resource; Teardown safely wraps SSH connection switch with status capture.
Presubmit scenario script test/scenarios-bootc/el10/presubmits/el98-src@el102-src@c2cc-upgrade-ok.sh	New script sources c2cc_common.sh, sets test environment (no randomization, 60m timeout, C2CC_TARGET_REF=rhel102-bootc-source), implements VM lifecycle hooks, and invokes upgrade suite.
Upgrade suite foundation: configuration, setup, and teardown test/suites/upgrade/upgrade-c2cc.robot (lines 1–75)	Suite configuration imports resources and libraries, defines per-cluster variable parameterization (BOOTC refs, service CIDRs, domains, kubeconfig paths); Setup validates TARGET_REF, logs into MicroShift host, registers all remote clusters; Teardown closes connections and removes kubeconfig.
Upgrade suite verification helpers test/suites/upgrade/upgrade-c2cc.robot (lines 76–94)	Adds Verify All Clusters Healthy (readiness checks via /readyz) and Verify Full C2CC Connectivity (pod-to-pod and pod-to-service across all distinct cluster pairs).
Upgrade suite orchestration and core logic test/suites/upgrade/upgrade-c2cc.robot (lines 35–170)	Main test deploys workloads, verifies initial health and connectivity, then sequentially upgrades each cluster by capturing booted digest, switching BOOTC ref with temporary insecure registry config, rebooting with greenboot health wait, refreshing SSH context, and re-validating health and connectivity. Helpers extract deployment digest, manage registry config injection/removal, track kernel boot id through reboot, and map cluster aliases to connection info.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• openshift/microshift#6827: Introduces baseline c2cc_run_tests behavior in c2cc_common.sh that this PR extends with TARGET_REF and BOOTC_REGISTRY variable forwarding.

Suggested reviewers

• jerpeter1

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding a C2CC upgrade test for RHEL 9.8 to RHEL 10.2, which is the primary purpose of the changeset.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR uses Robot Framework tests, not Ginkgo. All test cases and keywords have static, deterministic names with no dynamic content. Check is not applicable to this codebase.
Test Structure And Quality	✅ Passed	No Ginkgo tests exist in this repository. The codebase uses Robot Framework (88 test files) and Bash scripts (198 test files) instead. The custom check does not apply.
Microshift Test Compatibility	✅ Passed	No Ginkgo e2e tests added. PR contains only Robot Framework tests, shell scripts, and containerfiles for test infrastructure.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	This PR does not add Ginkgo e2e tests. The new tests are Robot Framework tests (.robot) for MicroShift C2CC upgrades, not Ginkgo tests. The custom check only applies to Ginkgo tests.
Topology-Aware Scheduling Compatibility	✅ Passed	PR adds only test infrastructure and test cases in test/ directory. No deployment manifests, operator code, or controllers with topology-unsafe scheduling constraints are introduced.
Ote Binary Stdout Contract	✅ Passed	PR contains no OTE binaries or Go code; only shell scripts, containerfiles, and Robot Framework tests. Check is not applicable.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	This PR adds Robot Framework tests, shell scripts, and containerfiles, not Ginkgo e2e tests. The custom check only applies to new Ginkgo tests, so it is not applicable.
No-Weak-Crypto	✅ Passed	No weak cryptographic algorithms, custom crypto, or non-constant-time secret comparisons found in PR changes. PSK generation uses openssl rand, which is cryptographically sound.
Container-Privileges	✅ Passed	No container privilege escalation risks found. PR adds test infrastructure for RHEL 9→10 upgrade validation with standard containerfile operations, no privileged/hostPID/hostNetwork/hostIPC/SYS_ADM...
No-Sensitive-Data-In-Logs	✅ Passed	No sensitive data (passwords, tokens, API keys, PII, session IDs) is exposed in logging. PSK and registry credentials are written to files with redirected output, not logged.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch c2cc-upgrade-tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vimauro
Once this PR has been reviewed and has the lgtm label, please assign jerpeter1 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/suites/upgrade/upgrade-c2cc.robot`:
- Around line 135-153: The Cluster Rebooted And Healthy keyword re-registers all
clusters using Register Remote Cluster (which opens a new SSH connection), but
the initial setup uses Register Local Cluster cluster-a (which reuses the
existing SSH connection from Login MicroShift Host). Add conditional logic
within Cluster Rebooted And Healthy to detect if the alias is cluster-a and call
Register Local Cluster instead of Register Remote Cluster for that specific
cluster, ensuring consistent SSH connection handling and registration approach
between the initial setup and post-reboot re-registration.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 70d633a2-942c-4ea0-bb3c-2a03a88f9a1a

📥 Commits

Reviewing files that changed from the base of the PR and between 08c4d7b and bc9201e.

📒 Files selected for processing (6)

• test/bin/c2cc_common.sh
• test/image-blueprints-bootc/el10/layer1-base/group1/rhel102-test-agent.containerfile
• test/image-blueprints-bootc/el10/layer2-presubmit/group1/rhel102-bootc-source.containerfile
• test/image-blueprints-bootc/templates/rpm-repo-config.sh.template
• test/scenarios-bootc/el10/presubmits/el98-src@el102-src@c2cc-upgrade-ok.sh
• test/suites/upgrade/upgrade-c2cc.robot

[vimauro]

/retest

[vimauro]

/label tide/merge-method-squash

[openshift-ci]

@vimauro: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-tests-bootc-periodic-arm-el9	d5bc734	link	true	/test e2e-aws-tests-bootc-periodic-arm-el9
ci/prow/e2e-aws-tests	d5bc734	link	true	/test e2e-aws-tests
ci/prow/e2e-aws-tests-periodic	d5bc734	link	true	/test e2e-aws-tests-periodic
ci/prow/e2e-aws-tests-bootc-periodic-arm-el10	d5bc734	link	true	/test e2e-aws-tests-bootc-periodic-arm-el10
ci/prow/verify	d5bc734	link	true	/test verify
ci/prow/e2e-aws-tests-periodic-arm	d5bc734	link	true	/test e2e-aws-tests-periodic-arm
ci/prow/e2e-aws-tests-bootc-periodic-el10	d5bc734	link	true	/test e2e-aws-tests-bootc-periodic-el10
ci/prow/e2e-aws-tests-arm	d5bc734	link	true	/test e2e-aws-tests-arm
ci/prow/e2e-aws-tests-bootc-periodic-el9	d5bc734	link	true	/test e2e-aws-tests-bootc-periodic-el9

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.