[Aakash-Dantre]

Summary

End-to-end tests for the SGLang model-server solution and the shared KServe model-server discovery in aws-observability/helm-charts#418, with vLLM and SGLang running side by side under KServe and as plain Deployments.

Builds on the otel-llm-serving root and the vLLM, KServe and Knative suites from #779 (merged). The root requires the SGLang solution in the cloned chart, so CI passes once a helm-charts release branch includes aws-observability/helm-charts#418.

What the chart does that this tests

KServe labels a vLLM predictor and an SGLang predictor identically, so #418 scrapes every predictor's model container once with a shared kserve-model-server job and lets each engine pipeline keep its own series by metric name (vllm:*/http_*, sglang:*). Servers outside KServe are found by image (vllm, sglang). Both engines get aws.service.type = ai_inference and the same service.name order: the resource.opentelemetry.io/service.name annotation, then the InferenceService, then the workload, then the pod.

Changes to terraform/eks/daemon/otel-llm-serving

Workload	Exercises
sglang-isvc: InferenceService, SGLang CPU image, --enable-metrics	SGLang found through the shared KServe job and routed to the sglang pipeline; service.name = InferenceService; the Knative data plane
sglang-standalone: Deployment, SGLang CPU image, no declared port, annotation resource.opentelemetry.io/service.name: sglang-chat	SGLang found by image; the 30000 default port; the annotation wins over the workload name
loadgen	now also sends requests to both SGLang servers

• SGLang uses its CPU build, lmsysorg/sglang:v0.5.23-xeon (sglang_cpu_image), with Qwen2.5-0.5B-Instruct like the vLLM servers. The build needs AVX-512, which m5 has; on AMD EPYC Zen 2 it exits with an illegal instruction. On CPU, --mem-fraction-static applies to host memory, so it is raised to 0.85.
• SGLang's /health takes about a second, the default probe timeout, so the InferenceService's readiness probe sets timeoutSeconds: 5; otherwise the queue-proxy never becomes ready.
• The chart guard also requires the sglang solution, and the validator runs ./test/otel/solutions/sglang after test_dir.

Suite: test/otel/solutions/sglang (9 tests)

• 12 SGLang metrics from both servers, with scope github.com/sgl-project/sglang, pipeline sglang and datapoint labels
• TestSGLangKServeAttribution: service.name = InferenceService, revision Deployment workload, KServe pod labels, kserve-container, ai_inference, cloud identity, no temporary attributes
• TestSGLangImageDetected: service.name from the annotation, scraped on port 30000, no InferenceService label
• one series per pod, never the queue-proxy
• TestModelServerRouting: each of the four servers (vLLM and SGLang, predictor and plain) reaches only its own engine pipeline, with a positive control for each
• the sglang pipeline publishes only sglang:*
• request and prompt-token counters increase on both servers

Testing

• Both suites run against an existing EKS cluster (KServe 0.20 Serverless, Knative 1.21) with a vLLM and an SGLang InferenceService on GPUs, CPU vLLM and CPU SGLang Deployments (SGLang on an Ice Lake node), the chart from feat(otel-ci): add SGLang model-server metrics and share KServe model-server discovery aws-observability/helm-charts#418, and load on all four. Workload names were pointed at that cluster.
  • SGLang suite: 9/9.
  • vLLM suite: 9/11; the transformer and generic-exporter tests need workloads that only this root deploys, and their positive controls reported them missing.
• go vet -tags integration and gofmt are clean; terraform fmt and terraform validate pass.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit)
• Commits and PR description comply with Amazon internal guidelines
• go vet / gofmt / terraform validate pass locally
• All GitHub Actions checks on the PR are passing
• Integration test evidence: suite runs above
• New or updated integration test coverage: this PR
• Related PRs linked: feat(otel-ci): add SGLang model-server metrics and share KServe model-server discovery aws-observability/helm-charts#418 (the chart change under test); builds on test(otel): add vLLM, KServe and Knative solution integration tests #779 (merged)
• New functionality has tests
• No breaking changes: new suite and workloads in an existing wip root