[Aakash-Dantre]

Same change as #389, opened from a branch in this repository so the EKS integration tests can run with the repository credentials; CI on fork PRs cannot reach them. Rebased onto main. Review history is on #389.

Summary

Adds otelContainerInsights.solutions.kserve.controlPlane: health metrics of the KServe operator (kserve-controller-manager), collected by the cluster-scraper and sent to CloudWatch over OTLP. On by default; on a cluster without KServe it discovers no target and does nothing.

Part of the LLM model-serving solutions, alongside vLLM model-server metrics (#353) and Knative Serving metrics (#390). End-to-end tests: aws/amazon-cloudwatch-agent-test#779.

Architecture

┌ namespace kserve ──────────────────────────────────┐
│ kserve-controller-manager pod                      │
│  ├ manager          metrics on 127.0.0.1:8080       │
│  └ kube-rbac-proxy  :8443/metrics (HTTPS) ◄────┐    │
└────────────────────────────────────────────────│────┘
                                                 │ bearer token (service account)
 cloudwatch-agent-cluster-scraper (Deployment) ──┘
              │
              └── OTLP/HTTP + SigV4 ──► CloudWatch

The controller is a single cluster-wide component, so it is scraped once, by the cluster-scraper, like Karpenter and KEDA.

What is scraped

• Discovery: pods in namespace (default kserve) with the label control-plane: kserve-controller-manager, selected by the API server. Only the container port 8443 of running pods is kept.
• Endpoint: the manager binds its plain metrics to 127.0.0.1:8080, so the scrape goes through its kube-rbac-proxy on :8443:
  • over HTTPS with the agent's service-account token
  • the chart's agent ClusterRole already grants get on the non-resource URL /metrics, which the proxy checks with a SubjectAccessReview
  • the proxy's certificate is self-signed, so verification is skipped
• Kept: controller_runtime_* (reconcile totals, errors and latency per controller, active workers, webhook request totals and latency) and leader_election_*.
• Dropped:
  • client-go's workqueue_* and rest_client_*, because the apiserver pipeline publishes those names under its own scope (as in filter/cw_k8s_ci_v0_keda_drop_non_keda)
  • go_*, process_* and other runtime families

Pipeline

metrics/cw_k8s_ci_v0_kserve_controlplane on the cluster-scraper, the same shape as Karpenter and KEDA:

1. prometheus/cw_k8s_ci_v0_kserve_controlplane: discovery as above; target labels pod, namespace, node.
2. filter/cw_k8s_ci_v0_scrape_metadata, filter/cw_k8s_ci_v0_kserve_controlplane_keep: keep list above.
3. Unit, start time, scope (github.com/kserve/kserve, cloudwatch.pipeline = kserve-controlplane) and cluster name.
4. groupbyattrs + transform/cw_k8s_ci_v0_kserve_controlplane_promote: one resource per controller pod, with k8s.pod.name, k8s.namespace.name and k8s.node.name.
5. k8sattributes/cw_k8s_ci_v0_pod, transform/cw_k8s_ci_v0_set_workload: Deployment kserve-controller-manager.
6. resourcedetection/cw_k8s_ci_v0_kserve_controlplane: cloud provider, platform, region and account only. AZ and host attributes are disabled because they would describe the node the cluster-scraper runs on, not the controller's.
7. cloud.resource_id, attribute limit, batch, then otlphttp/cw_k8s_ci_v0_cwotel (SigV4) to the CloudWatch OTLP endpoint.

Configuration

otelContainerInsights:
  enabled: true
  solutions:
    enabled: true
    kserve:
      controlPlane:
        enabled: true      # default
        namespace: kserve  # where kserve-controller-manager runs

KServe is a control plane only and is never in the request path. Request-level metrics for an InferenceService come from the model server (#353) and, in Serverless mode, from the Knative queue-proxy (#390).

Testing

• Template test: tests/kserve_pipeline_shape.sh (make helm-kserve-shape, part of make all) renders the chart and checks, on the parsed otelConfig:
  • label-based discovery, with no fixed target
  • HTTPS with the token
  • the keep list, evaluated against sample metric names
  • attribution order
  • the detector without AZ or host
  • the disabled toggle
• make helm-lint helm-flag-matrix helm-yaml-injection helm-neuron-shape pass.
• End-to-end: test/otel/solutions/kserve in test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779 (KServe v0.20 on EKS). All 5 tests (32 subtests) pass. They cover:
  • 8 expected metrics with scope, pipeline and labels
  • controller pod and Deployment attribution, and cloud identity
  • no AZ or host attributes
  • no workqueue_*, rest_client_*, go_* or process_* from this pipeline
• Resource cost: on a 4-node cluster, the cluster-scraper's CPU and memory with KServe and Knative scraping on vs. off are within run-to-run noise (39.2→34.5 millicores, 224→241 MB).

PR Checklist

• Commits are squashed into a logical, reviewable set
• Commits and PR description comply with Amazon internal guidelines
• make checks pass locally
• All GitHub Actions checks on the PR are passing
• Integration test evidence: end-to-end suite run above
• New or updated integration test coverage: test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779 (test/otel/solutions/kserve)
• New functionality has tests (tests/kserve_pipeline_shape.sh)
• Customer-visible changes called out: new otelContainerInsights.solutions.kserve.controlPlane value, on by default