[Aakash-Dantre]

Same change as #353 (approved), opened from a branch in this repository so the EKS integration tests can run with the repository credentials; CI on fork PRs cannot reach them. Rebased onto main (includes #385). Review history is on #353.

Summary

Adds otelContainerInsights.solutions.vllm: the vLLM model server's own Prometheus metrics (vllm:* engine metrics and the API server's http_*), collected by the CloudWatch agent DaemonSet and sent to CloudWatch over OTLP. On by default; it finds vLLM servers without configuration and does nothing on clusters that have none.

Part of the LLM model-serving solutions, alongside KServe control-plane metrics (#389) and Knative Serving metrics (#390). End-to-end tests: aws/amazon-cloudwatch-agent-test#779.

Background: how LLMs are served on Kubernetes

Three separate open-source projects are involved. This PR scrapes the first; #389 and #390 cover the other two.

	What it is	What it does	Control plane?	Handles requests?
vLLM	a model server: one Python process	loads the model onto the GPUs and answers an OpenAI-compatible API (/v1/chat/completions); exposes /metrics on the same port	no: every vLLM pod is independent	yes: it produces the answer
KServe	the model-serving layer	turns an InferenceService into running resources: picks the runtime image (vLLM for Hugging Face LLMs), the model arguments, routing, and model download	yes: kserve-controller-manager	no
Knative Serving	the run-and-scale layer (used by KServe's default Serverless mode)	runs request-driven services: creates revisions and Deployments, adds a queue-proxy sidecar, scales on request concurrency, including to zero	yes: activator, autoscaler, controller, webhook	only through the queue-proxy and, when scaled to zero, the activator

An InferenceService is a KServe custom resource the customer applies to say "serve this model":

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: qwen, namespace: llm }
spec:
  predictor:
    model:
      modelFormat: { name: huggingface }        # → KServe's Hugging Face runtime, which runs vLLM
      args: ["--model_id=Qwen/Qwen2.5-0.5B-Instruct"]
      resources: { limits: { nvidia.com/gpu: 1 } }

KServe and Knative are installed once per cluster by the platform team. The InferenceService does not create them; it is a request their (already running) controllers fulfil.

Creation path vs. request path

creation (once per model):
  customer ──► InferenceService ──► KServe controller ──► Knative Service ──► Knative controller ──► Deployment ──► pod
                                     (translates)                              (deploys, scales)             ├ queue-proxy
                                                                                                             └ kserve-container = vLLM
requests (every call):
  client ──► Istio gateway ──► (Knative activator, only when scaled to zero) ──► queue-proxy ──► vLLM ──► GPU

• The KServe controller is never in the request path; it only sets things up and keeps them matching the InferenceService.
• In KServe's RawDeployment mode there is no Knative: KServe creates a Deployment, Service and HPA directly, and requests go Service → vLLM.
• Without KServe, the customer runs vLLM as their own Deployment (vllm serve <model>, image vllm/vllm-openai), and requests go Service → vLLM.

Either way the vLLM pod is the customer's workload in the customer's namespace. The chart never deploys it; it only scrapes it.

Where it runs, and which agent scrapes what

 ┌─────────────── control planes (cluster-wide, one copy each) ───────────────┐
 │ ns kserve: kserve-controller-manager      ns knative-serving: activator,   │
 │            (reconciles InferenceServices)     autoscaler, controller, webhook│ ◄── cluster-scraper (#389, #390)
 └─────────────────────────────────────────────────────────────────────────────┘
 ┌──────────── node A (GPU) ─────────────────────────────┐   ┌──────── node B (GPU) ─────────────┐
 │ InferenceService pod "qwen-predictor-…" (ns llm)      │   │ standalone vLLM pod (ns llm)      │
 │  queue-proxy ──► kserve-container (vLLM) :8080 ◄──┐   │   │  vllm (vllm/vllm-openai) :8000 ◄─┐│
 │      ▲ :9091 (#390)                     #353 ─────┤   │   │                     #353 ────────┤│
 │      └──── cloudwatch-agent (DaemonSet) ──────────┘   │   │  cloudwatch-agent (DaemonSet) ───┘│
 └───────────────────────┬───────────────────────────────┘   └───────────────┬───────────────────┘
                         └──────────── OTLP/HTTP + SigV4 ──► CloudWatch ◄────┘

 requests: client ─► ingress ─► (activator if scaled to zero) ─► queue-proxy ─► vLLM ─► GPU

This PR scrapes only the vLLM process (marked #353 in the diagram): the kserve-container in KServe pods, and vLLM containers elsewhere. The DaemonSet agent on each node scrapes the vLLM pods on its own node. KServe's controller and Knative's components and queue-proxy are covered by #389 and #390.

Questions this answers

• Is this scraping random customer application containers? No. It scrapes only customer model-server containers that can be identified:

  • the container KServe itself names kserve-container, in the predictor pods it creates for an InferenceService (labels serving.kserve.io/inferenceservice and component: predictor);
  • a container whose image name contains vllm.

  No other containers are touched, and the chart reads no scrape labels or annotations of its own from customer pods. A vLLM server built from a custom-named image has no reliable signal, so it is not scraped automatically; a PodMonitor (feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #376) is the general way to opt such workloads in.

• Does vLLM have a control plane? No. Each vLLM process serves on its own. The control planes in this picture belong to KServe and Knative.

• Why the node agent and not the cluster-scraper? vLLM pods run on many (GPU) nodes. Each node agent watches only its own node's pods, using an API-server field selector, so the load is spread out and grows with pods per node, not per cluster.

What is scraped

Which containers. A container is a target if either applies:

Match	How it is found	Port	Path
KServe model server	kserve-container in a pod with the labels serving.kserve.io/inferenceservice and component: predictor, any image	prometheus.kserve.io/port (set by the serving runtime), else the declared port, else 8080	prometheus.kserve.io/path, else /metrics
vLLM image	container image name contains vllm (vllm/vllm-openai, mirrors, CPU builds)	the declared port, else 8000	/metrics

• KServe gives the container of every InferenceService component the name kserve-container. The component: predictor label restricts the match to the model server, excluding transformers and explainers.
• On KServe pods only the kserve-container is scraped, never the queue-proxy sidecar, which proxies the model server's /metrics.
• The only annotations read are KServe's own prometheus.kserve.io/*, and only on pods already matched. They pick the port and path; they don't select pods.
• Only running, non-init containers are scraped, every metricResolution (default 30s).

Which metrics. Kept: vllm:* (scheduler state, KV cache, prompt and generation tokens, request outcomes, TTFT, inter-token, queue and end-to-end latency histograms) and http_* (request counts and latency by handler and status). Dropped: python_*, process_*, *_created, and summaries, which the CloudWatch OTLP endpoint does not accept.

Pipeline

metrics/cw_k8s_ci_v0_vllm on the node agent:

1. prometheus/cw_k8s_ci_v0_vllm: discovery and relabelling as above; target labels pod and namespace.
2. filter/cw_k8s_ci_v0_scrape_metadata, filter/cw_k8s_ci_v0_vllm_keep: drop scrape metadata and the families above.
3. transform/cw_k8s_ci_v0_set_unit, metricstarttime/cw_k8s_ci_v0, transform/cw_k8s_ci_v0_set_cluster_name: shared Container Insights steps.
4. transform/cw_k8s_ci_v0_vllm_promote: copies pod and namespace to resource attributes, so the next step can associate the pod, and sets aws.service.type.
5. k8sattributes/cw_k8s_ci_v0_pod: pod, ReplicaSet and Deployment metadata.
6. Node name, resourcedetection, cloud.resource_id, node metadata, scope, workload: shared steps. set_workload resolves k8s.workload.name to the Deployment (or StatefulSet, DaemonSet, Job), not the ReplicaSet.
7. transform/cw_k8s_ci_v0_vllm_service_name: sets service.name and service.namespace.
8. awsattributelimit, batch, then otlphttp/cw_k8s_ci_v0_metrics_dest (SigV4) to the CloudWatch OTLP endpoint.

Resulting series

• Scope github.com/vllm-project/vllm, with cloudwatch.pipeline = vllm.
• service.name: the InferenceService name (KServe's own pod label, read as k8s.pod.label.serving.kserve.io/inferenceservice), else the workload, else the pod. The InferenceService comes first because a Serverless revision's Deployment name changes on every redeploy (<isvc>-predictor-00004-deployment). No custom attribute key is added for it.
• aws.service.type = ai_inference, set by the chart on every series. This is the CloudWatch resource attribute that classifies the source, which the CloudWatch (Omni) experience uses to identify AI inference telemetry. vLLM only serves inference, including when it generates rollouts for a training loop, so the value is fixed and nothing is read from the pod.
• Kubernetes (k8s.pod.name, k8s.namespace.name, k8s.workload.*, k8s.node.name, and pod labels such as k8s.pod.label.serving.kserve.io/inferenceservice) and cloud (cloud.region, cloud.availability_zone, cloud.resource_id, host.* of the node) attributes, plus vLLM's own labels (model_name, finished_reason, handler, ...).

Configuration

otelContainerInsights:
  enabled: true
  solutions:
    enabled: true
    vllm:
      enabled: true   # default

Testing

• Template test: tests/vllm_pipeline_shape.sh (make helm-vllm-shape, part of make all) renders the chart and checks, on the parsed otelConfig:
  • node-scoped discovery
  • target selection: KServe predictor container or vLLM image only, with no rule reading prometheus.io/*, cloudwatch.aws/* or aws.service.type from pods
  • port and path precedence, by relabel rule order
  • the fixed aws.service.type, and service.name from KServe's pod label with no custom inferenceservice key
  • processor order
  • the disabled toggle
• make helm-lint helm-flag-matrix helm-yaml-injection helm-neuron-shape pass.
• End-to-end: test/otel/solutions/vllm in test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779, on an EKS cluster with KServe Serverless, a CPU vLLM InferenceService, a plain vLLM Deployment, two annotated pods that must not be scraped (a vLLM mock from a custom-named image and a non-vLLM exporter) and load. The 9 tests cover:
  • expected metrics from both real servers, with scope, pipeline and labels
  • InferenceService, workload and cloud attribution, and aws.service.type
  • one series per pod and no queue-proxy target
  • nothing from either annotated pod
  • the dropped families
• GPU cluster: an EKS cluster with KServe Serverless (Knative 1.21) and GPU vLLM: a KServe InferenceService, a vllm/vllm-openai StatefulSet and an annotated pod. Verified in CloudWatch: both servers are scraped with one series per pod, the annotated pod is not scraped, a pod label aws.service.type: ai_training is ignored, and there are no agent errors.

Resource cost

A/B on a 4-node cluster (3 vLLM servers), same nodes and load, fresh agent pods, 15–20 min averages. Solutions off → on:

Pod	CPU (millicores)	Memory (MB)
node agents	22.8→16.4, 29.0→18.3, 14.5→9.8, 28.0→29.6	119→121, 175→153, 104→108, 130→146

No measurable difference: the deltas go both ways and are within run-to-run noise. Relabel rules run once per discovered target when discovery refreshes, not per sample.

PR Checklist

• Commits are squashed into a logical, reviewable set
• Commits and PR description comply with Amazon internal guidelines
• make checks pass locally (helm-lint, flag matrix, YAML injection, shape tests)
• All GitHub Actions checks on the PR are passing
• Integration test evidence: end-to-end suite run and EKS validation above
• New or updated integration test coverage: test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779 (test/otel/solutions/vllm)
• New functionality has tests (tests/vllm_pipeline_shape.sh)
• Customer-visible changes called out: new otelContainerInsights.solutions.vllm value, on by default

[Aakash-Dantre]

Reset this branch to the vLLM-only commit: #399 had been merged into it by mistake (it was stacked on this branch). The Knative change now goes to main separately in #401, which should be merged after this one.