[Aakash-Dantre]

Same change as #399 and #390, which were reviewed and approved. #399 was stacked on #398's branch and was merged into that branch rather than into main, so this PR brings the Knative change to main. Rebased onto main after #398 merged, so the diff is Knative-only.

Summary

Adds otelContainerInsights.solutions.knative: Knative Serving's request metrics (data plane) and its autoscaling and reconcile metrics (control plane), sent to CloudWatch over OTLP. Both are on by default and discover nothing on clusters without Knative.

Knative Serving is what KServe runs on in Serverless mode, so for an InferenceService this is where request concurrency, request latency at the queue-proxy, and autoscaler decisions (desired, ready and pending pods, panic mode) come from. It works the same for plain Knative Services.

Part of the LLM model-serving solutions, alongside vLLM model-server metrics (#353) and KServe control-plane metrics (#389). End-to-end tests: aws/amazon-cloudwatch-agent-test#779.

Architecture

 node A                                       namespace knative-serving
┌───────────────────────────────────────┐    ┌──────────────────────────────────┐
│ revision pod (Knative Service / ISVC) │    │ activator   :9090/metrics ◄──┐   │
│  ├ user-container / kserve-container  │    │ autoscaler  :9090/metrics ◄──┤   │
│  └ queue-proxy :9091/metrics ◄──┐     │    │ controller  :9090/metrics ◄──┤   │
│                                 │     │    │ webhook     :9090/metrics ◄──┤   │
│ cloudwatch-agent (DaemonSet) ───┘     │    └──────────────────────────────│───┘
└──────────────┬────────────────────────┘                                   │
               │                    cloudwatch-agent-cluster-scraper ───────┘
               └──────── OTLP/HTTP + SigV4 ────────┴──► CloudWatch

• Data plane: every revision pod has a queue-proxy sidecar that sits in the request path. The DaemonSet agent on each node scrapes the queue-proxies of the revision pods on that node.
• Control plane: the four Serving components are cluster-wide, so the cluster-scraper scrapes them once, like Karpenter and KEDA.

What is scraped

dataPlane: node agent, pipeline metrics/cw_k8s_ci_v0_knative_dataplane

• Discovery: pods with a serving.knative.dev/revision label on the agent's own node. Both filters run in the API server (field selector spec.nodeName plus a label selector), so no agent watches pods on other nodes or non-Knative pods. An optional namespace restricts it further.
• Endpoint: the queue-proxy's http-usermetric port (9091), path /metrics, plain HTTP.
• Kept: kn_serving_* (Serving ≥ 1.19) and revision_*request* (≤ 1.18), e.g. kn_serving_invocation_duration_seconds.
• Attribution: pod and namespace are promoted to the resource, so k8sattributes resolves the revision's Deployment (k8s.workload.name) and adds the pod's labels, including KServe's k8s.pod.label.serving.kserve.io/inferenceservice. No custom attribute key is added. Node and cloud attributes are the node's.
• aws.service.type = ai_inference on predictor revisions of a KServe InferenceService (pod labels serving.kserve.io/inferenceservice and component: predictor, read after k8sattributes), matching the vLLM solution, so the CloudWatch (Omni) experience identifies them as AI inference. Transformer and explainer revisions carry the same InferenceService label but don't run the model, and plain Knative Services can be any workload, so neither gets aws.service.type.

controlPlane: cluster-scraper, pipeline metrics/cw_k8s_ci_v0_knative_controlplane

• Discovery: pods in namespace (default knative-serving) whose app label is activator, autoscaler, controller or webhook. The pattern is fully anchored, so net-istio-* and other add-ons are excluded. The metrics port (9090), plain HTTP.
• Kept:
  • kn_revision_* (desired, requested, ready, pending and terminating pods, concurrency, panic mode, capacity)
  • kn_activator_*, kn_autoscaler_*, kn_workqueue_*, kn_webhook_*, kn_k8s_client_*
  • the components' http_server_* / http_client_*
• Dropped: go_*, *_go_*, process_*, promhttp_*.
• Attribution: the same steps as Karpenter and KEDA (the component name arrives as k8s.pod.label.app; no custom key). groupbyattrs and a promote give one resource per component pod, then k8sattributes and set_workload add its Deployment. A dedicated resourcedetection sets cloud provider, platform, region and account, but not AZ or host, which would describe the cluster-scraper's node.

Both pipelines set scope knative.dev/serving with cloudwatch.pipeline = knative-dataplane or knative-controlplane, and export through the existing SigV4 OTLP exporters.

Knative requirements

Serving ≤ 1.18 exports Prometheus metrics by default (OpenCensus names: revision_*, autoscaler_*). Serving ≥ 1.19 uses the OpenTelemetry SDK (kn_* names) and exports nothing until told to. metrics-protocol covers the control plane and request-metrics-protocol the queue-proxy:

kubectl -n knative-serving patch cm config-observability --type merge \
  -p '{"data":{"metrics-protocol":"prometheus","request-metrics-protocol":"prometheus"}}'
kubectl -n knative-serving rollout restart deployment

The components read metrics-protocol only at startup, hence the restart. The queue-proxy setting applies to revision pods started after the change. Both name sets are kept by the filters, and the README documents this.

Istio sidecars with STRICT mTLS. KServe and Knative do not inject Istio sidecars into revision pods by default. If a namespace turns injection on and enforces STRICT mTLS, the sidecar rejects the agent's plain-HTTP scrape, because the agent is not in the mesh. The README shows a PeerAuthentication that allows plain text on port 9091 for those pods (for KServe, selected by component: predictor).

Configuration

otelContainerInsights:
  enabled: true
  solutions:
    enabled: true
    knative:
      dataPlane:
        enabled: true             # default
        namespace: ""             # restrict to one namespace; empty = all
      controlPlane:
        enabled: true             # default
        namespace: knative-serving

Testing

• Template test: tests/knative_pipeline_shape.sh (make helm-knative-shape, part of make all) renders the chart and checks, on the parsed otelConfig:
  • API-server node and label selection for the data plane, with no relabel node filter
  • the data-plane promote before k8sattributes, and no custom inferenceservice key
  • aws.service.type = ai_inference only for InferenceService predictor revisions, set after k8sattributes
  • no custom knative_component key on the control plane
  • control-plane attribution order
  • the detector without AZ or host
  • both disabled toggles
• make helm-lint helm-flag-matrix helm-yaml-injection helm-neuron-shape pass.
• End-to-end: test/otel/solutions/knative in test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779 (Knative 1.21 under KServe Serverless on EKS, with load through the queue-proxy). The 5 tests cover:
  • 8 control-plane metrics attributed to their component pod and Deployment, with cloud identity
  • no AZ or host attributes
  • runtime families dropped
  • the queue-proxy metric attributed to the revision Deployment and InferenceService (KServe's pod label, no custom key), with aws.service.type = ai_inference
• Resource cost: on a 4-node cluster, agent CPU and memory with the solutions on vs. off are within run-to-run noise. Figures are in feat(otel-ci): add vLLM model-server metrics solution #353.

PR Checklist

• Commits are squashed into a logical, reviewable set
• Commits and PR description comply with Amazon internal guidelines
• make checks pass locally
• All GitHub Actions checks on the PR are passing
• Integration test evidence: end-to-end suite run above
• New or updated integration test coverage: test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779 (test/otel/solutions/knative)
• New functionality has tests (tests/knative_pipeline_shape.sh)
• Customer-visible changes called out: new otelContainerInsights.solutions.knative values, on by default; Knative ≥ 1.19 needs the config-observability change above