OTA-2024, OTA-2070: Add eval test cases for cluster-update skills

evals/skills/cluster-update-advisor/README.md

@@ -0,0 +1,62 @@
+# cluster-update-advisor eval
+
+Tests that the agent correctly analyzes CVO readiness data and produces the right upgrade decision (`recommend`, `warn`, `block`, or `escalate`) following the classification rules in `cluster-update/cluster-update-advisor/SKILL.md`.
+
+## Prerequisites
+
+### Cluster
+
+A live OpenShift cluster is not required to run these evals. The test cases embed readiness JSON directly in the query, simulating the exact format CVO proposals produce.
+
+The readiness JSON format was validated against a real OCP 4.21.5 cluster on GCP (6 nodes: 3 master + 3 worker) with these OLM operators installed:
+
+| Operator | Package | Version | Channel |
+|---|---|---|---|
+| Red Hat OpenShift Logging | `cluster-logging` | 6.5.1 | stable-6.5 |
+| Compliance Operator | `compliance-operator` | 1.9.0 | stable |
+| Red Hat OpenShift Pipelines | `openshift-pipelines-operator-rh` | 1.22.0 | latest |
+| Web Terminal | `web-terminal` | 1.16.0 | fast |
+| DevWorkspace Operator | `devworkspace-operator` | 0.41.0 | fast |
+
+### CVO readiness data source
+
+The readiness JSON was generated by `pkg/readiness` from [openshift/cluster-version-operator#1395](https://github.com/openshift/cluster-version-operator/pull/1395), which adds 9 parallel cluster health checks:
+
+| Check | What it assesses |
+|---|---|
+| `cluster_conditions` | CVO conditions, update history, channel |
+| `operator_health` | ClusterOperator status, MachineConfigPool state |
+| `api_deprecations` | Deprecated/removed API usage |
+| `node_capacity` | Node readiness, schedulability |
+| `pdb_drain` | PDB drain blockers |
+| `etcd_health` | etcd member health |
+| `network` | Network plugin, TLS, proxy |
+| `crd_compat` | CRD version mismatches |
+| `olm_operator_lifecycle` | OLM operator versions, OCP compatibility |
+
+## Test scenarios
+
+| Test | Scenario | Expected decision |
+|---|---|---|
+| `advisor_healthy_cluster_recommend` | All 9 checks pass, 0 issues | `recommend` |
+| `advisor_degraded_operator_warn` | 1 degraded operator (authentication) | `warn` |
+| `advisor_api_deprecation_block` | Removed API with 1250 active requests | `block` |
+| `advisor_etcd_unhealthy_block` | 1 of 3 etcd members not ready | `block` |
+| `advisor_errored_checks_escalate` | 7 of 9 checks errored (API unreachable) | `escalate` |
+
+## Running
+
+```bash
+bash evals/run.sh -k "cluster-update-advisor"
+```
+
+## Decision matrix reference
+
+From `cluster-update/cluster-update-advisor/SKILL.md`:
+
+| Blockers | Warnings | Decision |
+|---|---|---|
+| Unable to assess | any | `escalate` |
+| 1+ | any | `block` |
+| 0 | 1+ | `warn` |
+| 0 | 0 | `recommend` |

evals/skills/cluster-update-advisor/system_prompt.md

@@ -0,0 +1,9 @@
+You have access to the cluster-update-advisor skill. Use it to evaluate
+cluster upgrade readiness data and produce an upgrade decision.
+
+The request contains a "Cluster Readiness Data" section with a JSON
+block. Parse the JSON, evaluate each check's results, and apply the
+cluster-update-advisor skill's decision framework to classify findings as
+escalate, block, warn, or recommend.
+
+Do not guess or assume cluster state. Do not execute upgrade commands.

evals/skills/product-lifecycle/README.md

@@ -0,0 +1,51 @@
+# product-lifecycle eval
+
+Tests that the agent can query the [Red Hat Product Life Cycle API](https://access.redhat.com/product-life-cycles/api/v2/products) for product support status, EOL dates, and OCP version compatibility using the product-lifecycle skill.
+
+## Prerequisites
+
+### Cluster
+
+A live OpenShift cluster is not required to run these evals. The test cases embed operator metadata in the query, and the agent queries the public [Red Hat Product Life Cycle API](https://access.redhat.com/product-life-cycles/api/v2/products) directly. Internet access is required.
+
+The operator metadata in each test case was collected from a real OCP 4.21.5 cluster on GCP (6 nodes: 3 master + 3 worker) with these OLM operators installed:
+
+| Operator | Package | Version | Channel |
+|---|---|---|---|
+| Red Hat OpenShift Logging | `cluster-logging` | 6.3.1 | stable-6.3 |
+| Compliance Operator | `compliance-operator` | 1.9.0 | stable |
+| Red Hat OpenShift Pipelines | `openshift-pipelines-operator-rh` | 1.22.0 | latest |
+| Web Terminal | `web-terminal` | 1.16.0 | fast |
+| DevWorkspace Operator | `devworkspace-operator` | 0.41.0 | fast |
+
+> **Note:** Some operator versions were adjusted from the real cluster values to better exercise test conditions (e.g. `cluster-logging` uses 6.3.1 instead of 6.5.1 to simulate OCP 4.21 incompatibility).
+
+## Ground truth
+
+Expected values come from the live PLC API. Verify using `plc_lookup.py` — the same CLI the agent uses (see [SKILL.md](../../../cluster-update/product-lifecycle/SKILL.md)):
+
+```bash
+# OCP 4.21 — should show "Full Support"
+python3 cluster-update/product-lifecycle/scripts/plc_lookup.py products "Red Hat OpenShift Container Platform" \
+  | jq '.results[] | select(.version == "4.21") | {version, status}'
+
+# OCP 4.14 — should show "Extended Support"
+python3 cluster-update/product-lifecycle/scripts/plc_lookup.py products "Red Hat OpenShift Container Platform" \
+  | jq '.results[] | select(.version == "4.14") | {version, status}'
+
+# cluster-logging — check compatibility with OCP 4.21
+python3 cluster-update/product-lifecycle/scripts/plc_lookup.py products "logging for Red Hat OpenShift" --ocp 4.21
+
+# compliance-operator v1.9 — should show "Full Support"
+python3 cluster-update/product-lifecycle/scripts/plc_lookup.py products "compliance operator"
+
+# Batch check — all 5 operators against OCP 4.21 using olm-check
+python3 cluster-update/product-lifecycle/scripts/plc_lookup.py olm-check --ocp 4.21 \
+  --operators '[{"package":"cluster-logging","version":"6.3.1"},{"package":"compliance-operator","version":"1.9.0"},{"package":"openshift-pipelines-operator-rh","version":"1.22.0"},{"package":"web-terminal","version":"1.16.0"},{"package":"devworkspace-operator","version":"0.41.0"}]'
+```
+
+## Running
+
+```bash
+bash evals/run.sh -k "product-lifecycle"
+```

evals/skills/product-lifecycle/system_prompt.md

@@ -0,0 +1,10 @@
+You have access to the product-lifecycle skill. Use it to query the
+Red Hat Product Life Cycle API for product support phases, EOL dates,
+and OCP version compatibility.
+
+When given operator package names or Red Hat product names, look up
+their lifecycle status using the product-lifecycle skill. Report the
+support phase and whether the product is compatible with the requested
+OCP version.
+
+Do not guess lifecycle data. Always use the product-lifecycle skill.

evals/skills/product-lifecycle/test_cases.yaml

@@ -0,0 +1,191 @@
+# product-lifecycle skill eval test cases
+#
+# Each test case exercises the product-lifecycle skill against the
+# Red Hat Product Life Cycle API. Ground truth comes from the live API.
+#
+
+# Test 1: Simulate proposal request with 5 OLM operators.
+# Batch check — 5 operators, count which have lifecycle data, which are compatible.
+# devworkspace-operator is not found, web-terminal version 1.16 is not present, cluster-logging 6.3 is not compatible with OCP 4.21
+- name: plc_proposal_olm_batch_check
+  query: |
+    Use the product-lifecycle skill to look up lifecycle data for
+    each of these 5 operators against OCP 4.21:
+
+    - compliance-operator v1.9.0 (channel: stable, package: compliance-operator)
+    - cluster-logging v6.3.1 (channel: stable-6.3, package: cluster-logging)
+    - devworkspace-operator v0.41.0 (channel: fast, package: devworkspace-operator)
+    - openshift-pipelines-operator-rh v1.22.0 (channel: latest, package: openshift-pipelines-operator-rh)
+    - web-terminal v1.16.0 (channel: fast, package: web-terminal)
+
+    For each operator, check if the Product Life Cycle API has lifecycle data for the installed version.
+    Report the total checked, the total found in the API, how many whose version is present, how many compatible with OCP 4.21
+  schema:
+    type: object
+    properties:
+      olm_check_ran:
+        type: boolean
+        description: "Whether OLM operator lifecycle checks were performed using the 'product-lifecycle' skill."
+      operators_checked:
+        type: integer
+        description: "Total number of operators checked against the Product Life Cycle API"
+      operators_api_tracked_count:
+        type: integer
+        description: "Number of operators whose package was found in the PLC API, even if the specific version is not tracked (i.e. package exists but version may be missing)"
+      operators_version_tracked_count:
+        type: integer
+        description: "Number of operators whose specific version is tracked in the PLC API (package found and version matched)"
+      operators_ocp_compatible_count:
+        type: integer
+        description: "Number of operators with ocp_compatible=true for the target OCP version"
+    required: ["olm_check_ran", "operators_checked", "operators_api_tracked_count", "operators_version_tracked_count", "operators_ocp_compatible_count"]
+  expected:
+    _fn: verify_olm_batch_check
+
+# Test 2: Verify cluster-logging v6.5 lifecycle status and OCP 4.21 compat
+# against live API data.
+- name: plc_cluster_logging_supported
+  query: >
+    A cluster upgrade proposal from OCP 4.21.5 to 4.21.16 shows
+    cluster-logging v6.5.1 is installed (package: cluster-logging).
+
+    Use the product-lifecycle skill to lookup cluster-logging
+    (package: cluster-logging) and report whether it was found,
+    its lifecycle status, and whether it is compatible with OCP 4.21.
+
+  schema:
+    type: object
+    properties:
+      product_found:
+        type: boolean
+        description: "Whether cluster-logging was found in the PLC API. Use the 'product-lifecycle' skill."
+      status:
+        type: string
+        enum: ["supported", "maintenance", "extended", "end-of-maintenance", "eol", "unknown"]
+        description: "Normalized lifecycle status of the version compatible with OCP 4.21"
+      ocp_compatible:
+        type: boolean
+        description: "Whether the product is compatible with OCP 4.21"
+    required: ["product_found", "status", "ocp_compatible"]
+  expected:
+    _fn: verify_cluster_logging_status
+
+# Test 3: Verify web-terminal OCP 4.21 compatibility against live API.
+# Uses _fn to query the PLC API at test time so the expectation stays
+# correct as new web-terminal versions are published.
+- name: plc_web_terminal_compat_check
+  query: >
+    A cluster running OCP 4.21 has web-terminal v1.16.0 installed
+    (package: web-terminal).
+
+    Use the product-lifecycle skill to check if web-terminal has a
+    version that is compatible with OCP 4.21 and currently in full support.
+  schema:
+    type: object
+    properties:
+      product_found:
+        type: boolean
+        description: "Whether web-terminal was found in the PLC API. Use the 'product-lifecycle' skill."
+      has_supported_version_for_421:
+        type: boolean
+        description: "Whether any web-terminal version is both 'supported' and compatible with OCP 4.21"
+    required: ["product_found", "has_supported_version_for_421"]
+  expected:
+    _fn: verify_web_terminal_compat
+
+# Test 4: compliance-operator — check v1.9 lifecycle status against live API.
+# The PLC API indexes this operator by package name ("compliance-operator")
+# and by product name ("compliance operator"). The agent may use either approach.
+- name: plc_compliance_operator_status
+  query: >
+    The compliance-operator v1.9.0 was found installed (package: compliance-operator)
+    in the cluster.
+
+    Use the product-lifecycle skill to check the lifecycle status of
+    compliance-operator v1.9. Report whether it was found and its current lifecycle
+    status.
+  schema:
+    type: object
+    properties:
+      product_found:
+        type: boolean
+        description: "Whether the PLC API has lifecycle data for this operator. Use the 'product-lifecycle' skill."
+      status:
+        type: string
+        enum: ["supported", "maintenance", "extended", "end-of-maintenance", "eol", "unknown"]
+        description: "Normalized lifecycle status of compliance-operator v1.9"
+    required: ["product_found", "status"]
+  expected:
+    _fn: verify_compliance_operator_status
+
+# Test 5: OCP platform lifecycle check — verify OCP 4.21 status against live API.
+- name: plc_ocp_platform_status
+  query: >
+    Before upgrading from OCP 4.21.5 to 4.21.16, verify that the
+    target OCP version 4.21 is still in full support.
+
+    Use the product-lifecycle skill to look up "Red Hat OpenShift
+    Container Platform" and check version 4.21's lifecycle status.
+    Report whether it was found and its current lifecycle status.
+  schema:
+    type: object
+    properties:
+      product_found:
+        type: boolean
+        description: "Whether OCP was found in the PLC API. Use the 'product-lifecycle' skill."
+      status:
+        type: string
+        enum: ["supported", "maintenance", "extended", "end-of-maintenance", "eol", "unknown"]
+        description: "Normalized lifecycle status of OCP 4.21"
+    required: ["product_found", "status"]
+  expected:
+    _fn: verify_ocp_platform_status
+
+# Test 6: Older OCP version lifecycle — check OCP 4.14 status against live API.
+- name: plc_ocp_old_version_extended
+  query: >
+    Use the product-lifecycle skill to check the lifecycle status
+    of Red Hat OpenShift Container Platform version 4.14.
+    Report whether it was found and its current lifecycle status.
+  schema:
+    type: object
+    properties:
+      product_found:
+        type: boolean
+        description: "Whether OCP was found in the PLC API. Use the 'product-lifecycle' skill."
+      status:
+        type: string
+        enum: ["supported", "maintenance", "extended", "end-of-maintenance", "eol", "unknown"]
+        description: "Normalized lifecycle status of OCP 4.14"
+    required: ["product_found", "status"]
+  expected:
+    _fn: verify_ocp_old_version_status
+
+# Test 7: Multiple operators batch check from proposal context.
+# Test with 2 operators known to have lifecycle data (cluster-logging, web-terminal).
+- name: plc_batch_known_operators_only
+  query: |
+    Use the product-lifecycle skill to check lifecycle status for these
+    2 operators against OCP 4.21:
+
+    1. cluster-logging (package: cluster-logging)
+    2. web-terminal (package: web-terminal)
+
+    For each operator, look up its lifecycle data. Report whether both
+    were found in the API and whether any of their OCP 4.21-compatible
+    versions are end-of-life.
+  schema:
+    type: object
+    properties:
+      check_completed:
+        type: boolean
+        description: "Whether OLM operator lifecycle checks completed successfully using the 'product-lifecycle' skill."
+      both_found:
+        type: boolean
+        description: "Whether lifecycle data was found for both operators"
+      any_eol:
+        type: boolean
+        description: "Whether any version of either operator that is compatible with OCP 4.21 is end-of-life"
+    required: ["check_completed", "both_found", "any_eol"]
+  expected:
+    _fn: verify_batch_known_operators