feat(otel-ci): add vLLM model-server metrics solution by Aakash-Dantre · Pull Request #353 · aws-observability/helm-charts

Aakash-Dantre · 2026-08-07T16:51:39Z

Summary

Adds otelContainerInsights.solutions.vllm: the vLLM model server's own Prometheus metrics (vllm:* engine metrics and the API server's http_*), collected by the CloudWatch agent DaemonSet and sent to CloudWatch over OTLP. On by default; it finds vLLM servers without configuration and does nothing on clusters that have none.

Part of the LLM model-serving solutions, alongside KServe control-plane metrics (#389) and Knative Serving metrics (#390). End-to-end tests: aws/amazon-cloudwatch-agent-test#779.

Background: how LLMs are served on Kubernetes

Three separate open-source projects are involved. This PR scrapes the first; #389 and #390 cover the other two.

	What it is	What it does	Control plane?	Handles requests?
vLLM	a model server: one Python process	loads the model onto the GPUs and answers an OpenAI-compatible API (`/v1/chat/completions`); exposes `/metrics` on the same port	no: every vLLM pod is independent	yes: it produces the answer
KServe	the model-serving layer	turns an InferenceService into running resources: picks the runtime image (vLLM for Hugging Face LLMs), the model arguments, routing, and model download	yes: `kserve-controller-manager`	no
Knative Serving	the run-and-scale layer (used by KServe's default Serverless mode)	runs request-driven services: creates revisions and Deployments, adds a `queue-proxy` sidecar, scales on request concurrency, including to zero	yes: activator, autoscaler, controller, webhook	only through the `queue-proxy` and, when scaled to zero, the activator

An InferenceService is a KServe custom resource the customer applies to say "serve this model":

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: qwen, namespace: llm }
spec:
  predictor:
    model:
      modelFormat: { name: huggingface }        # → KServe's Hugging Face runtime, which runs vLLM
      args: ["--model_id=Qwen/Qwen2.5-0.5B-Instruct"]
      resources: { limits: { nvidia.com/gpu: 1 } }

KServe and Knative are installed once per cluster by the platform team. The InferenceService does not create them; it is a request their (already running) controllers fulfil.

Creation path vs. request path

creation (once per model):
  customer ──► InferenceService ──► KServe controller ──► Knative Service ──► Knative controller ──► Deployment ──► pod
                                     (translates)                              (deploys, scales)             ├ queue-proxy
                                                                                                             └ kserve-container = vLLM
requests (every call):
  client ──► Istio gateway ──► (Knative activator, only when scaled to zero) ──► queue-proxy ──► vLLM ──► GPU

The KServe controller is never in the request path; it only sets things up and keeps them matching the InferenceService.
In KServe's RawDeployment mode there is no Knative: KServe creates a Deployment, Service and HPA directly, and requests go Service → vLLM.
Without KServe, the customer runs vLLM as their own Deployment (vllm serve <model>, image vllm/vllm-openai), and requests go Service → vLLM.

Either way the vLLM pod is the customer's workload in the customer's namespace. The chart never deploys it; it only scrapes it.

Where it runs, and which agent scrapes what

 ┌─────────────── control planes (cluster-wide, one copy each) ───────────────┐
 │ ns kserve: kserve-controller-manager      ns knative-serving: activator,   │
 │            (reconciles InferenceServices)     autoscaler, controller, webhook│ ◄── cluster-scraper (#389, #390)
 └─────────────────────────────────────────────────────────────────────────────┘
 ┌──────────── node A (GPU) ─────────────────────────────┐   ┌──────── node B (GPU) ─────────────┐
 │ InferenceService pod "qwen-predictor-…" (ns llm)      │   │ standalone vLLM pod (ns llm)      │
 │  queue-proxy ──► kserve-container (vLLM) :8080 ◄──┐   │   │  vllm (vllm/vllm-openai) :8000 ◄─┐│
 │      ▲ :9091 (#390)                     #353 ─────┤   │   │                     #353 ────────┤│
 │      └──── cloudwatch-agent (DaemonSet) ──────────┘   │   │  cloudwatch-agent (DaemonSet) ───┘│
 └───────────────────────┬───────────────────────────────┘   └───────────────┬───────────────────┘
                         └──────────── OTLP/HTTP + SigV4 ──► CloudWatch ◄────┘

 requests: client ─► ingress ─► (activator if scaled to zero) ─► queue-proxy ─► vLLM ─► GPU

This PR scrapes only the vLLM process (marked #353): the kserve-container in KServe pods, and vLLM containers elsewhere. The DaemonSet agent on each node scrapes the vLLM pods on its own node. KServe's controller and Knative's components and queue-proxy are covered by #389 and #390.

Questions this answers

Is this scraping random customer application containers? No. It scrapes only customer model-server containers that can be identified:
- the container KServe itself names kserve-container, in the predictor pods it creates for an InferenceService (labels serving.kserve.io/inferenceservice and component: predictor);
- a container whose image name contains vllm.
No other containers are touched, and the chart reads no scrape labels or annotations of its own from customer pods. A vLLM server built from a custom-named image has no reliable signal, so it is not scraped automatically; a PodMonitor (feat(amazon-cloudwatch-observability): ServiceMonitor/PodMonitor scraping via the Target Allocator #376) is the general way to opt such workloads in.
Does vLLM have a control plane? No. Each vLLM process serves on its own. The control planes in this picture belong to KServe and Knative.
Why the node agent and not the cluster-scraper? vLLM pods run on many (GPU) nodes. Each node agent watches only its own node's pods, using an API-server field selector, so the load is spread out and grows with pods per node, not per cluster.

What is scraped

Which containers. A container is a target if either applies:

Match	How it is found	Port	Path
KServe model server	`kserve-container` in a pod with the labels `serving.kserve.io/inferenceservice` and `component: predictor`, any image	`prometheus.kserve.io/port` (set by the serving runtime), else the declared port, else 8080	`prometheus.kserve.io/path`, else `/metrics`
vLLM image	container image name contains `vllm` (`vllm/vllm-openai`, mirrors, CPU builds)	the declared port, else 8000	`/metrics`

KServe gives the container of every InferenceService component the name kserve-container. The component: predictor label restricts the match to the model server, excluding transformers and explainers.
On KServe pods only the kserve-container is scraped, never the queue-proxy sidecar, which proxies the model server's /metrics.
The only annotations read are KServe's own prometheus.kserve.io/*, and only on pods already matched. They pick the port and path; they don't select pods.
Only running, non-init containers are scraped, every metricResolution (default 30s).

Which metrics. Kept: vllm:* (scheduler state, KV cache, prompt and generation tokens, request outcomes, TTFT, inter-token, queue and end-to-end latency histograms) and http_* (request counts and latency by handler and status). Dropped: python_*, process_*, *_created, and summaries, which the CloudWatch OTLP endpoint does not accept.

Pipeline

metrics/cw_k8s_ci_v0_vllm on the node agent:

prometheus/cw_k8s_ci_v0_vllm: discovery and relabelling as above; target labels pod and namespace.
filter/cw_k8s_ci_v0_scrape_metadata, filter/cw_k8s_ci_v0_vllm_keep: drop scrape metadata and the families above.
transform/cw_k8s_ci_v0_set_unit, metricstarttime/cw_k8s_ci_v0, transform/cw_k8s_ci_v0_set_cluster_name: shared Container Insights steps.
transform/cw_k8s_ci_v0_vllm_promote: copies pod and namespace to resource attributes, so the next step can associate the pod, and sets aws.service.type.
k8sattributes/cw_k8s_ci_v0_pod: pod, ReplicaSet and Deployment metadata.
Node name, resourcedetection, cloud.resource_id, node metadata, scope, workload: shared steps. set_workload resolves k8s.workload.name to the Deployment (or StatefulSet, DaemonSet, Job), not the ReplicaSet.
transform/cw_k8s_ci_v0_vllm_service_name: sets service.name and service.namespace.
awsattributelimit, batch, then otlphttp/cw_k8s_ci_v0_metrics_dest (SigV4) to the CloudWatch OTLP endpoint.

Resulting series

Scope github.com/vllm-project/vllm, with cloudwatch.pipeline = vllm.
service.name: the InferenceService name (KServe's own pod label, read as k8s.pod.label.serving.kserve.io/inferenceservice), else the workload, else the pod. The InferenceService comes first because a Serverless revision's Deployment name changes on every redeploy (<isvc>-predictor-00004-deployment). No custom attribute key is added for it.
aws.service.type = ai_inference, set by the chart on every series. This is the CloudWatch resource attribute that classifies the source, which the CloudWatch (Omni) experience uses to identify AI inference telemetry. vLLM only serves inference, including when it generates rollouts for a training loop, so the value is fixed and nothing is read from the pod.
Kubernetes (k8s.pod.name, k8s.namespace.name, k8s.workload.*, k8s.node.name, and pod labels such as k8s.pod.label.serving.kserve.io/inferenceservice) and cloud (cloud.region, cloud.availability_zone, cloud.resource_id, host.* of the node) attributes, plus vLLM's own labels (model_name, finished_reason, handler, ...).

Configuration

otelContainerInsights:
  enabled: true
  solutions:
    enabled: true
    vllm:
      enabled: true   # default

Testing

Template test: tests/vllm_pipeline_shape.sh (make helm-vllm-shape, part of make all) renders the chart and checks, on the parsed otelConfig:
- node-scoped discovery
- target selection: KServe predictor container or vLLM image only, with no rule reading prometheus.io/*, cloudwatch.aws/* or aws.service.type from pods
- port and path precedence, by relabel rule order
- the fixed aws.service.type, and service.name from KServe's pod label with no custom inferenceservice key
- processor order
- the disabled toggle
make helm-lint helm-flag-matrix helm-yaml-injection helm-neuron-shape pass.
End-to-end: test/otel/solutions/vllm in test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779, on an EKS cluster with KServe Serverless, a CPU vLLM InferenceService, a plain vLLM Deployment, two annotated pods that must not be scraped (a vLLM mock from a custom-named image and a non-vLLM exporter) and load. The 9 tests cover:
- expected metrics from both real servers, with scope, pipeline and labels
- InferenceService, workload and cloud attribution, and aws.service.type
- one series per pod and no queue-proxy target
- nothing from either annotated pod
- the dropped families
GPU cluster: an EKS cluster with KServe Serverless (Knative 1.21) and GPU vLLM: a KServe InferenceService, a vllm/vllm-openai StatefulSet and an annotated pod. Verified in CloudWatch: both servers are scraped with one series per pod, the annotated pod is not scraped, a pod label aws.service.type: ai_training is ignored, and there are no agent errors.

Resource cost

A/B on a 4-node cluster (3 vLLM servers), same nodes and load, fresh agent pods, 15–20 min averages. Solutions off → on:

Pod	CPU (millicores)	Memory (MB)
node agents	22.8→16.4, 29.0→18.3, 14.5→9.8, 28.0→29.6	119→121, 175→153, 104→108, 130→146

No measurable difference: the deltas go both ways and are within run-to-run noise. Relabel rules run once per discovered target when discovery refreshes, not per sample.

PR Checklist

Commits are squashed into a logical, reviewable set
Commits and PR description comply with Amazon internal guidelines
make checks pass locally (helm-lint, flag matrix, YAML injection, shape tests)
All GitHub Actions checks on the PR are passing
Integration test evidence: end-to-end suite run and EKS validation above
New or updated integration test coverage: test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779 (test/otel/solutions/vllm)
New functionality has tests (tests/vllm_pipeline_shape.sh)
Customer-visible changes called out: new otelContainerInsights.solutions.vllm value, on by default

oberkem · 2026-08-26T20:03:58Z

-      ## Namespace where Karpenter is deployed.
-      namespace: "kube-system"
+  ## Optional Prometheus scraping of vLLM model-server metrics (vllm:* series)
+  ## from KServe InferenceService pods, published through the OTEL CI pipeline.


why only kserve?

Answered on the vLLM scrape job: #353 (comment) — short version, you're right, and I'd like to add a prometheus.io/scrape fallback so this isn't KServe-only.

oberkem · 2026-09-22T19:36:03Z

+                names:
+                  - {{ .Values.otelContainerInsights.solutions.vllm.namespace }}
+              {{- end }}
+          relabel_configs:


don't we need

- source_labels: [__meta_kubernetes_pod_node_name] regex: ${env:K8S_NODE_NAME} action: keep

like csi-node job?

Good catch. Fixed in 32c4649 on both jobs.

It was a real bug, not just inconsistency: without it every agent scraped every matching pod cluster-wide. One vLLM pod was emitting three copies of every vllm:* series under three k8s.node.name values — 3x cost, and sum() 3x high.

oberkem · 2026-09-22T20:08:45Z

+  prometheus/cw_k8s_ci_v0_knative_dataplane:
+    config:
+      scrape_configs:
+        - job_name: knative-dataplane


same here https://github.com/aws-observability/helm-charts/pull/353/changes#r4075661295

Fixed in 32c4649 along with the vLLM job.

oberkem · 2026-09-22T20:38:53Z

+- **Errors are not observable.** vLLM never sets span status, and only successfully
+  finished requests are traced at all — aborts, timeouts and errors emit no span. So
+  Application Signals `Error` and `Fault` stay at zero regardless of what the engine is
+  doing. Upstream fix: [vllm#32162](https://github.com/vllm-project/vllm/pull/32162).


this PR has been open for 8 months. I asked the author about the status. Can you get that CR to the finish line?

I'll pick it up — blocked on conflicts and pre-commit, not review. It fixes span status and aborted requests (so Application Signals Error/Fault become real), but not gen_ai.response.model, which is vllm-project/vllm#51186.

oberkem · 2026-09-22T20:39:26Z

+On vLLM 0.11 also set `OTEL_EXPORTER_OTLP_TRACES_INSECURE=true`; 0.26 needs nothing.
+`--collect-detailed-traces` is not needed — it does nothing for traces on the V1 engine.
+
+Spans are billed per trace and vLLM emits one per request with no sampling of its own, so on


instead of disabling the whole thing, should we enable it with 10% sampling rate

Agreed, but the chart can't set it — OTEL_TRACES_SAMPLER is an env var on the engine container. I'll put traceidratio / 0.1 in the README next to --otlp-traces-endpoint.

Did "the whole thing" mean traces.enabled: false? That's off by default because without Transaction Search the endpoint 400s and the agent drops every span silently.

oberkem · 2026-09-22T20:41:05Z

 {{- end }}

+{{- if and .Values.otelContainerInsights.solutions.enabled .Values.otelContainerInsights.solutions.kserve.controlPlane.enabled }}
+  # Keep only the KServe operator health families; drop go_*/process_* runtime noise.


you can do the same to vllm if needed.

Already there — filter/cw_k8s_ci_v0_vllm_keep at _otel-container-insights-config.tpl:470, wired in at :1220.

oberkem · 2026-09-22T20:53:06Z

+  default Istio/Knative path propagates one, so there is no end-to-end trace and no
+  visibility into gateway queueing.
+
+### Enabling tracing on the engine


it's okay if you want to submit another PR, but we can and should also get kserve traces https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#enabling-metric-collection

Yes, as a follow-up. The valuable half is queue-proxy propagating W3C context into the engine — vLLM honours an inbound traceparent but nothing sends one today, so every trace is a single orphan span. I'll open an issue and link it.

oberkem · 2026-09-22T20:56:53Z

+    error_mode: ignore
+    metrics:
+      metric:
+        - 'not IsMatch(name, "^(controller_runtime_|workqueue_|rest_client_|leader_election_).*")'


doesn't this drop a lot of other metrics?
https://knative.dev/docs/serving/observability/metrics/serving-metrics/#workload-metrics
https://github.com/kserve/kserve/blob/master/python/kserve/kserve/metrics.py#L20

Request-level telemetry is instead sourced from the Knative queue-proxy via knative.dataPlane, but that doesn't exist in RawDeployment mode afaik, so those users lose serving-layer request metrics entirely.

Two things; the second is a real gap.

Neither link is emitted by the kserve-controller-manager — metrics.py is the KServe Python model server (predictor pod), and the Knative page is queue-proxy. I scraped the controller-manager directly: 158 families in, 21 kept, the rest go_*/process_*/promhttp_*.

Your follow-up is the real issue though: RawDeployment has no queue-proxy, so those users get no request telemetry from this PR. For vLLM the http_* series cover it (already kept by vllm_keep); for other runtimes we'd need a predictor scrape. Here or follow-up?

The KServe filter keeps metric names that KEDA deliberately drops. It keeps workqueue_* and rest_client_*. KEDA's filter/cw_k8s_ci_v0_keda_drop_non_keda drops exactly those because "These conflict with the apiserver pipeline". Either

We should probably decide in general why we drop these metrics or why we keep them. I'm not sure I fully understand the reason to drop them.... these are apiserver client metrics, they existing across a whole range of sources... the metric name is not the only denotation, its the labels

Went with the KEDA behaviour for consistency: KServe now keeps only controller_runtime_* and leader_election_* (#389). I agree we should decide this once for all solutions. The labels do tell sources apart, so keeping them could be fine. Happy to follow up in a separate PR if you'd rather keep them everywhere.

oberkem · 2026-09-22T21:03:32Z

+          relabel_configs:
+            # keep only KServe InferenceService pods (match the label VALUE, like
+            # the ebs-csi job keys on pod_label_app)
+            - source_labels: [__meta_kubernetes_pod_label_serving_kserve_io_inferenceservice]


iiuc this job discovers targets by the serving.kserve.io/inferenceservice label and container name kserve-container, both of which only exist when the pod is reconciled from a KServe InferenceService. a plain vLLM Deployment (container vllm, no such label) gets zero targets and the vllm solution silently collects nothing.
if the intention is to vLLM-on-KServe specifically note in values.yaml that solutions.vllm requires a KServe InferenceService.
if the intention is vLLM generally, could we add a prometheus.io/scrape annotation fallback or any other fallback mechanism?
I prefer we do VLLM generally but not sure if there would be specific things would be required for rayserve

Correct, and same as your Aug 26 comment — let me answer it.

A plain vLLM Deployment matches neither the label nor kserve-container, so it gets zero targets silently.

I'd rather do it your way. Proposal: scrape a pod if it's a KServe predictor or has prometheus.io/scrape: "true" + prometheus.io/port. Zero-config stays for KServe, plain Deployments get a standard opt-in.

RayServe also exposes ray_serve_* on the dashboard-agent port, so it needs its own solution rather than riding on this. OK to leave it out?

miconeilaws · 2026-10-07T15:58:02Z

no automated suite covers otelContainerInsights.solutions.*.

Thats not a good reason not to add tests. We cant accept this PR without a suite of integration tests. I'm also a bit concerned about the complexity of some of the prometheus scrape configs in here, how can we be sure they are matching only what we want them to.

miconeilaws · 2026-10-07T16:00:34Z

The PR description doesn't give me enough context to understand the architecture of kserve and knative so I can meaningfully understand exactly what we're trying to scrape, this seems pretty complex.

Also idelally we would split kserve and knative into separate PRs to reduce the noise

miconeilaws · 2026-10-07T15:44:37Z

+          static_configs:
+            - targets:
+                - kserve-controller-manager-service.{{ .Values.otelContainerInsights.solutions.kserve.controlPlane.namespace }}.svc:8443
+{{- end }}


Karpenter and KEDA find their pods by label, so a cluster without them scrapes nothing, but the KServe scrape points at a hard-coded Service address, so every cluster without KServe gets a scrape that fails every 30s and polluting the logs. Can't we use service discovery here too?

Fixed (now in #389). The controller is found by its control-plane: kserve-controller-manager label, like Karpenter and KEDA. No KServe means no target and no errors.

miconeilaws · 2026-10-07T15:46:21Z

+          tls_config:
+            insecure_skip_verify: true
+          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token


I think we probably didn't do it like this for karpenter and keda... But I think we need to find a solution to let customers adopt our TLS cert for secure communications as an optional feature... Not released yet but this is what I was thinking for OTLP cases: #317

Basicaly OTEL has env vars you can set and the OTLP exporter in whatever the application is will use your certs. We probably need to find the equivalent solution for Prometheus endpoints.

Agree this is worth having, but I'd keep it out of these PRs. Proposal for a follow-up:

an opt-in tls block per solution (or one under solutions), using the chart's existing CA and client cert by default, or a customer CA bundle;

a pod annotation to scrape a target over HTTPS;

for servers that need a certificate, cert-manager issuing from the chart's CA.

vLLM supports TLS natively (--ssl-certfile/--ssl-keyfile). KServe's proxy uses a self-signed cert, so verification stays skipped there until a CA is configured. I can write this up as an issue if that works for you.

miconeilaws · 2026-10-07T15:48:56Z

+        - metricstarttime/cw_k8s_ci_v0
+        - transform/cw_k8s_ci_v0_set_scope_kserve_controlplane
+        - transform/cw_k8s_ci_v0_set_cluster_name
+        - resourcedetection/cw_k8s_ci_v0


I think this sets the availability zone to that of where the cluster scraper is not the kserve endpoint or knative one. Karpenter and KEDA each define their own detector with the AZ turned off I believe.

Correct, fixed. KServe and Knative control plane each have their own detector with AZ and host turned off, like Karpenter and KEDA (#389, #390). The e2e tests check that those attributes are absent.

miconeilaws · 2026-10-07T15:49:17Z

        - otlphttp/cw_k8s_ci_v0_cwotel
 {{- end }}
+{{- if and .Values.otelContainerInsights.solutions.enabled .Values.otelContainerInsights.solutions.kserve.controlPlane.enabled }}
+    metrics/cw_k8s_ci_v0_kserve_controlplane:


These two pipelines also skip the steps Karpenter/KEDA use to attach pod/namespace/node and workload details (groupbyattrs + *_promote + k8sattributes/cw_k8s_ci_v0_pod + set_workload). Is that on purpose?

Not on purpose, fixed. Both now use the same steps as Karpenter/KEDA (groupbyattrs → promote → k8sattributes → set_workload), so series carry the controller's pod and Deployment. The Knative data plane also gets the revision's Deployment.

miconeilaws · 2026-10-07T15:51:26Z

+          scrape_interval: {{ .Values.otelContainerInsights.metricResolution }}
+          scrape_timeout: {{ include "otel-container-insights.scrapeTimeout" . }}
+          metrics_path: /metrics
+          kubernetes_sd_configs:


My agent claims: The Knative data-plane scrape watches every pod in the cluster from every node. Can you look into this?

Right, fixed (now in #390). Discovery uses a field selector for the node and a label selector for serving.knative.dev/revision. The API server applies both, so each agent only watches the revision pods on its own node.

miconeilaws · 2026-10-07T15:59:15Z

+            # never renames it.
+            - source_labels: [__meta_kubernetes_pod_label_serving_kserve_io_inferenceservice, __meta_kubernetes_pod_container_name, __meta_kubernetes_pod_container_image]
+              separator: ";"
+              regex: '.+;kserve-container;.*|(?i)[^;]*;[^;]*;.*vllm.*'


Did we measure the cpu cost of all these regexes? We should evaluate this against the new resource consumption test harness - I think its in a non-blocking mode right now

Measured. I compared solutions on vs. off on a 4-node cluster with the same load. The difference in agent CPU and memory is within noise: CPU even came out lower on 4 of 5 pods, and the agents use 10–40 millicores anyway (numbers in the #353 description). The regexes run once per discovered target when discovery refreshes, not per sample. The repo's resource harness compares against a fixed baseline calibrated on t3.medium nodes, so its pass/fail doesn't mean anything on this cluster; I can run it in its own setup if you'd like.

miconeilaws · 2026-10-07T16:02:29Z

+    error_mode: ignore
+    metrics:
+      metric:
+        - 'not IsMatch(name, "^(controller_runtime_|workqueue_|rest_client_|leader_election_).*")'


The KServe filter keeps metric names that KEDA deliberately drops. It keeps workqueue_* and rest_client_*. KEDA's filter/cw_k8s_ci_v0_keda_drop_non_keda drops exactly those because "These conflict with the apiserver pipeline". Either

We should probably decide in general why we drop these metrics or why we keep them. I'm not sure I fully understand the reason to drop them.... these are apiserver client metrics, they existing across a whole range of sources... the metric name is not the only denotation, its the labels

miconeilaws · 2026-10-07T16:04:31Z

+A vLLM server built from an image without `vllm` in its name opts in with a pod label:
+
+```yaml
+metadata:
+  labels:
+    cloudwatch.aws/vllm: "true"
+```


Is this really necessary? My main concern is we're establishing a new API/pattern to mark containers for scraping via CWA which we currently don't have. That has broader implications, we'll need to adopt this pattern in other places too and we're married to whatever nomenclature we decide here. Is it strictly necessary?

And honestly, this sounds like a another version of the pod/service monitor interface. I'm not sure we want to go down this route

Agreed, removed. Opt-in now uses the standard prometheus.io/scrape, prometheus.io/port and prometheus.io/path annotations, so we add no new API. Since those annotations aren't vLLM-specific, we only keep vllm:* from pods found this way.

Agreed, we're not adding one. With the standard annotations, the only rule we own is "find vLLM", and that needs no configuration in the common cases: KServe pods, and images named vllm.

miconeilaws · 2026-10-07T16:07:30Z

+  # vLLM model-server metrics, scraped per node like the ebs-csi-node job.
+  prometheus/cw_k8s_ci_v0_vllm:
+    config:
+      scrape_configs:


An agent suggests this, check it out:

KServe's runtimes already state where their metrics are, so the chart doesn't need to guess from names. The stock ClusterServingRuntimes carry spec.annotations saying so:

# kserve-huggingfaceserver prometheus.kserve.io/port: '8080' prometheus.kserve.io/path: "/metrics" # kserve-tritonserver prometheus.kserve.io/port: '8002'

I read these from kserve/kserve master config/runtimes/. A runtime's spec.annotations get applied to the predictor pod. I believe that, but haven't confirmed it on a live pod; it's one kubectl get pod -o yaml for the author to check. That gives three options:

KServe pods: scrape whatever port and path the runtime annotations give. Every runtime gets the right endpoint with no configuration, including Triton on 8002, which a guessed default gets wrong. There's no port defaulting and no multi-port ambiguity, and the "is this KServe?" check comes down to "has the annotation". Whether the metrics belong to vLLM is then up to the keep filter, which only keeps vllm:* and http_*, so it no longer matters which image a runtime uses. Caveat: if a cluster sets enable-prometheus-scraping in KServe's metrics aggregator, KServe rewrites the pod to standard prometheus.io/* annotations pointing at the queue-proxy aggregation port. The relabel should prefer prometheus.kserve.io/* and fall back to that. I haven't checked how the aggregator interacts with this. Everything else: the standard prometheus.io/scrape, prometheus.io/port and prometheus.io/path annotations. This is oberkem's suggestion and the usual Kubernetes convention. The user names exactly one port, so it replaces both the invented cloudwatch.aws/vllm label and the "every declared port" rule. The image-name check can stay as a convenience for vllm/vllm-openai with no annotations. Prometheus Operator PodMonitor or ServiceMonitor via the target allocator. This is the most flexible option, but I'd advise against it here. The chart only wires the target allocator into the custom-agent path, not OTel Container Insights. It would also need the Prometheus Operator CRDs and a cluster-wide allocator, which is a much bigger change than this PR.

Done. On KServe pods, prometheus.kserve.io/port and path now win, then the declared port, then 8080. We ignore prometheus.io/* on KServe pods because KServe sets them itself. They would also pick up the queue-proxy, which serves the model's metrics a second time. Testing caught exactly this, and #779 now checks it.

Aakash-Dantre · 2026-10-08T15:22:09Z

We cant accept this PR without a suite of integration tests.

Agreed. Added end-to-end tests in aws/amazon-cloudwatch-agent-test#779, following the KEDA/Karpenter suites. A terraform setup builds an EKS cluster with KServe, Knative and real vLLM (on CPU), plus a few pods that each test one scrape rule. The Go tests then check in CloudWatch what was collected and how it is labelled. They also check what must not be collected: no duplicate series, nothing from a non-vLLM pod, no AZ on cluster-scraper series. All pass. Each chart PR also has a helm template test in make all that pins the scrape config.

Aakash-Dantre · 2026-10-08T15:22:11Z

The PR description doesn't give me enough context to understand the architecture … ideally we would split kserve and knative into separate PRs

Split into three PRs: #353 (vLLM), #389 (KServe), #390 (Knative). Each description now has a diagram of which agent scrapes what, and how targets are found. It also lists which metrics are kept and the pipeline steps.

miconeilaws

Some clarifications requested on slack.

miconeilaws · 2026-10-09T11:41:17Z

+          - set(resource.attributes["inferenceservice"], attributes["inferenceservice"]) where attributes["inferenceservice"] != nil
+          - set(resource.attributes["aws.service.type"], attributes["aws_service_type"]) where attributes["aws_service_type"] != nil


What are these attributes? They don't seem part of any semconv.

miconeilaws · 2026-10-09T11:52:36Z

+            # Temporary label, moved to the aws.service.type resource attribute by
+            # transform/cw_k8s_ci_v0_vllm_promote. ai_inference unless the pod sets
+            # its own aws.service.type label (e.g. ai_training).
+            - target_label: aws_service_type


why is temporary

It was only a scrape-time label to carry the value to the promote step. It's gone: aws.service.type is now set directly in the pipeline.

miconeilaws · 2026-10-09T11:53:05Z

+            - source_labels: [__meta_kubernetes_pod_label_aws_service_type]
+              regex: "(.+)"
+              target_label: aws_service_type
+              action: replace


Why? Is this a supported and documented contract for Omni already?

Removed. The chart no longer reads an aws.service.type (or any other) label from the pod; the value is always ai_inference, which the Omni experience needs to identify AI inference telemetry. vLLM only serves inference, even inside training loops, so there's nothing for customers to override.

Scrape vLLM's own Prometheus metrics (vllm:* and http_*) from the agent on each node. Servers are found without configuration: the kserve-container of a KServe InferenceService's predictor pods, or any container running a vLLM image. The chart reads no scrape labels or annotations of its own from pods; on KServe pods the serving runtime's prometheus.kserve.io/port and path are honoured. Series are attributed to the pod and workload. service.name is the InferenceService (from KServe's own pod label, as extracted by k8sattributes), else the workload, else the pod. Every series carries aws.service.type = ai_inference.

Aakash-Dantre · 2026-10-09T12:57:27Z

Some clarifications requested on slack.

From slack.

Changes in #353, in simple words

What gets scraped: only vLLM model servers. Either the model container (kserve-container) of a KServe InferenceService's predictor pods, or any container whose image name contains vllm.
Nothing customers set on pods is read. The custom cloudwatch.aws/vllm label and the prometheus.io/scrape|port|path annotations are removed. Custom-named vLLM images can be added with a PodMonitor instead.
KServe's own annotations only: prometheus.kserve.io/port/path, which KServe sets, choose the port and path. Otherwise the declared port is used, else 8080 (KServe) or 8000 (vLLM).
aws.service.type = ai_inference is always set by the chart, never taken from a pod label. The ai_training override is gone.
No custom inferenceservice attribute. service.name comes from KServe's own pod label (k8s.pod.label.serving.kserve.io/inferenceservice), else the workload, else the pod.
Each node's agent watches only its own node's pods, filtered by the API server.
Tests: a template test in make all, plus the end-to-end suite in #779.

Aakash-Dantre · 2026-10-13T11:32:07Z

Opened #398 with the same change (rebased onto main) from a branch in this repository, so the EKS integration tests can run with the repository credentials. CI on fork PRs cannot reach them.

oberkem reviewed Aug 26, 2026

View reviewed changes

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch from 3ba49bb to f93c7cd Compare September 1, 2026 13:53

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch 2 times, most recently from e50d1ae to 1e005ec Compare September 14, 2026 09:20

Aakash-Dantre changed the title ~~feat(otel-ci): add optional vLLM model-server metrics scraping~~ feat(otel-ci): add LLM model-serving telemetry (vLLM, KServe, Knative) as Container Insights solutions Sep 14, 2026

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch from 2574ee0 to ba94333 Compare September 17, 2026 14:11

oberkem reviewed Sep 22, 2026

View reviewed changes

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch 4 times, most recently from be5a78c to 96e901d Compare September 29, 2026 16:28

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch 3 times, most recently from c4df29c to aa15319 Compare October 5, 2026 14:39

Aakash-Dantre changed the title ~~feat(otel-ci): add LLM model-serving telemetry (vLLM, KServe, Knative) as Container Insights solutions~~ feat(otel-ci): add vLLM, KServe and Knative metrics solutions Oct 5, 2026

Aakash-Dantre mentioned this pull request Oct 5, 2026

feat(otel-ci): add vLLM and Knative request traces via the agent OTLP endpoint #380

Open

8 of 9 tasks

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch from aa15319 to 30cd213 Compare October 7, 2026 08:48

miconeilaws reviewed Oct 7, 2026

View reviewed changes

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch from 30cd213 to f6e3b11 Compare October 8, 2026 12:57

This was referenced Oct 8, 2026

feat(otel-ci): add KServe control-plane metrics solution #389

Closed

feat(otel-ci): add Knative Serving metrics solution #390

Closed

Aakash-Dantre mentioned this pull request Oct 8, 2026

test(otel): add vLLM, KServe and Knative solution integration tests aws/amazon-cloudwatch-agent-test#779

Merged

7 of 8 tasks

Aakash-Dantre changed the title ~~feat(otel-ci): add vLLM, KServe and Knative metrics solutions~~ feat(otel-ci): add vLLM model-server metrics solution Oct 8, 2026

miconeilaws reviewed Oct 9, 2026

View reviewed changes

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch from f6e3b11 to 669e8c6 Compare October 9, 2026 12:05

miconeilaws reviewed Oct 9, 2026

View reviewed changes

Aakash-Dantre force-pushed the feat/otel-ci-vllm-metrics branch from 669e8c6 to 6c08e25 Compare October 9, 2026 12:53

miconeilaws approved these changes Oct 13, 2026

View reviewed changes

Aakash-Dantre mentioned this pull request Oct 13, 2026

feat(otel-ci): add vLLM model-server metrics solution #398

Merged

7 of 8 tasks

This was referenced Oct 13, 2026

feat(otel-ci): add Knative Serving metrics solution #399

Merged

feat(otel-ci): add KServe control-plane metrics solution #400

Merged

Aakash-Dantre closed this Oct 13, 2026

Aakash-Dantre mentioned this pull request Oct 13, 2026

feat(otel-ci): add Knative Serving metrics solution #401

Merged

7 of 8 tasks

		- set(resource.attributes["inferenceservice"], attributes["inferenceservice"]) where attributes["inferenceservice"] != nil
		- set(resource.attributes["aws.service.type"], attributes["aws_service_type"]) where attributes["aws_service_type"] != nil

Conversation

Aakash-Dantre commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background: how LLMs are served on Kubernetes

Creation path vs. request path

Where it runs, and which agent scrapes what

Questions this answers

What is scraped

Pipeline

Resulting series

Configuration

Testing

Resource cost

PR Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miconeilaws commented Oct 7, 2026

Uh oh!

miconeilaws commented Oct 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aakash-Dantre commented Aug 7, 2026 •

edited

Loading