From 63c20a7fd1e8130d75bb3822243c6c1c3a5d6c4a Mon Sep 17 00:00:00 2001
From: Scott Chen <2170chm@gmail.com>
Date: Tue, 20 Oct 2026 17:59:32 -0700
Subject: [PATCH] Introduce scale-to-zero autoscaler feature in readme

---
 README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index a9d8513..d6a98ca 100644
--- a/README.md
+++ b/README.md
@@ -14,7 +14,7 @@ This project evaluates a production inference stack built on top of existing OSS
 - **[llm-d-inference-scheduler (EPP)](https://github.com/llm-d/llm-d-inference-scheduler)** — Per-model Endpoint Picker (image `mcr.microsoft.com/oss/v2/llm-d/llm-d-inference-scheduler`). Performs KV-cache aware routing by injecting the `x-gateway-destination-endpoint` header, directing requests to the optimal inference pod.
 - **[Kaito InferenceSet](https://github.com/kaito-project/kaito)** — Manages groups of vLLM inference pods. Multiple InferenceSets (e.g., Model-A, Model-B) can run different models simultaneously.
 - **[vLLM Inference Pods(llm-d-inference-sim)](https://github.com/llm-d/llm-d-inference-sim)** — Serve model inference requests. On CPU-only E2E clusters, the real vLLM container is replaced by a **shadow pod** running llm-d-inference-sim (image `ghcr.io/llm-d/llm-d-inference-sim`), a lightweight vLLM-compatible simulator that exposes the same OpenAI API and `vllm:*` Prometheus metrics. See [`pkg/gpu-node-mocker/README.md`](pkg/gpu-node-mocker/README.md) for the original-pod ↔ shadow-pod mechanism.
-- **[keda-kaito-scaler](https://github.com/kaito-project/keda-kaito-scaler)** — Metric-based autoscaler built on [KEDA](https://keda.sh/) that scales vLLM inference pods up and down based on workload metrics.
+- **[keda-kaito-scaler](https://github.com/kaito-project/keda-kaito-scaler)** — Metric-based autoscaler built on [KEDA](https://keda.sh/) that scales vLLM inference pods up and down based on workload metrics. Supports scale to zero (`minReplicas=0`), releasing GPU capacity while a model is idle and waking it on the next request.
 - **[productionstack-status-reporter](charts/productionstack/charts/productionstack-status-reporter)** — Control-plane status reporter bundled with the `productionstack` umbrella chart. A single, leader-elected, read-only controller that evaluates the end-to-end error-handling reason catalogue across the cluster / modelharness / modeldeployment layers on every resync and surfaces the aggregated result as Kubernetes `Event`s in `kube-system`. It is the sole producer of these reasons, giving operators one place to observe why the stack (gateway, auth, BBR, EPP, InferenceSets, or nodes) is unhealthy. Runs HA (≥ 2 replicas); only the leader emits events. It additionally serves the OpenAI-compatible **model discovery** endpoints (`GET /v1/models`, `GET /v1/models/{id}`) behind a cluster-singleton Service — that surface is read-only and runs on *every* replica, not just the leader.
 - **[Mocked GPU Nodes](https://github.com/kaito-project/production-stack/blob/main/pkg/gpu-node-mocker/README.md) / CPU Nodes** — Infrastructure layer providing compute resources for inference workloads. The `gpu-node-mocker` controller (E2E-only) fakes GPU nodes on CPU-only clusters and runs the `llm-d-inference-sim` shadow pods on real CPU nodes.
 
@@ -42,6 +42,7 @@ The routes must precede the catch-all routes: a bodyless `GET` gives BBR nothing
 1. **vLLM pods → metrics.** Each pod exposes `vllm:*` Prometheus metrics (queue depth, KV-cache utilisation, request rate).
 2. **keda-kaito-scaler → KEDA.** The external scaler aggregates per-`InferenceSet` pod metrics and returns a single summed metric value.
 3. **KEDA → HPA → InferenceSet.** KEDA exposes that value through the external metrics API; the HPA computes the desired replica count from it and patches the `InferenceSet`, and the KAITO controller adds or removes vLLM pods.
+4. **Scale to zero (optional).** With `minReplicas=0`, an idle deployment parks at zero replicas; while parked, the EPP queues incoming requests and its queue-size metric wakes the deployment (0 → 1). See [Scale to zero](charts/modeldeployment/README.md#scale-to-zero).
 
 ## Installation