Skip to content

fix(BREAKING): keep EPP names and labels within 63 characters - #246

Open
rambohe-ch wants to merge 2 commits into
kaito-project:mainfrom
rambohe-ch:fix-epp-name
Open

rambohe-ch wants to merge 2 commits into
kaito-project:mainfrom
rambohe-ch:fix-epp-name

Conversation

@rambohe-ch

@rambohe-ch rambohe-ch commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Reason for Change:

The modeldeployment chart named the EPP Service <name>-inferencepool-epp and used the same value for the EPP pods' inferencepool label. Both are limited to 63 characters, so any deployment name over 45 characters failed to install. The AIManager smoke test with nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 hit this. The InferencePool name <name>-inferencepool had the same problem: Istio copies it into the istio.io/inferencepool-name label on the pool's shadow Service.

This PR names the InferencePool and the EPP Service after the deployment, and uses the deployment name as the inferencepool label value. The chart's name value is now capped at 63 characters.

 InferencePool                          
-  <name>-inferencepool
+  <name>
 EPP Service + EPP pod label `inferencepool`
-  <name>-inferencepool-epp
+  <name>
 EPP Deployment / ServiceAccount / Role / RoleBinding / ConfigMap
   <name>-inferencepool-epp              # unchanged
 values.schema.json: name
+  maxLength: 63
Name length Before After
≤ 45 works works
46 – 63 EPP Service / label value > 63 characters, install fails works
≥ 64 fails at the API server rejected by the chart schema

E2E:

  • Split EPPServiceName into EPPDeploymentName and EPPSelector, following the new naming.
  • The Helm backend shortens release names over 53 characters with a hash suffix, so it can install deployment names of up to 63 characters.
  • Added chart unit tests checking that every generated name and label value fits its Kubernetes limit for a 63-character name, and that a 64-character name is rejected. The assertion accepts both Helm 3 and Helm 4 error text.
  • When a readiness wait times out, the error now shows the last real failure instead of a client rate-limiter error.
  • The end-to-end 63-character e2e case is deferred: it needs fix(BREAKING): bound resource name lengths to fit Kubernetes limits kaito#2450, which is not yet in KAITO's nightly-latest image. Older KAITO creates a Workspace StatefulSet whose pod label exceeds 63 characters, so the inference pod is never created.

⚠️ Breaking changes

  1. InferencePool renamed from <name>-inferencepool to <name>. Anything that refers to the pool by name must be updated: external HTTPRoutes, scripts, kubectl commands.
  2. EPP Service renamed from <name>-inferencepool-epp to <name>.
  3. EPP pod label inferencepool changes from <name>-inferencepool-epp to <name>. This label is in the EPP Deployment's spec.selector, which Kubernetes does not allow to change, so helm upgrade of an existing release fails with spec.selector: ... field is immutable. Before upgrading, delete the EPP Deployment and let Helm recreate it:
    kubectl delete deployment <name>-inferencepool-epp -n <namespace>
    helm upgrade <release> oci://ghcr.io/kaito-project/helm/modeldeployment -n <namespace> ...
    The model does not route traffic until the new EPP pod is Ready. Alternatively, run helm uninstall followed by helm install.
  4. Metric label values change. The modelharness Azure PodMonitor copies the inferencepool pod label onto EPP metrics, and EPP labels its pool metrics with the InferencePool name. Update any dashboard, alert or recording rule that filters on inferencepool="<name>-inferencepool-epp" or on the pool name <name>-inferencepool.
  5. name is limited to 63 characters. The chart schema now rejects longer names at helm install time. Such names never deployed successfully anyway.

Unchanged: the EPP Deployment, ServiceAccount, Role, RoleBinding and ConfigMap keep <name>-inferencepool-epp, so the status reporter and the benchmark workflow need no changes.

Requirements

  • added unit tests and e2e tests (if applicable).

Issue Fixed:

Notes for Reviewers:

The EPP Service name and the EPP pods' inferencepool selector label value
were "<name>-inferencepool-epp", which exceeds the 63-character Service
name and label value limits for long model names (e.g. the AIManager smoke
test with nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16). The InferencePool name
"<name>-inferencepool" has the same problem, because Istio copies it into
the istio.io/inferencepool-name label on the pool's shadow Service.

Name the InferencePool and the EPP Service after the deployment, use that
name as the inferencepool label value, and cap the chart's name value at 63
characters. The EPP Deployment, ServiceAccount, Role, RoleBinding and
ConfigMap keep "<name>-inferencepool-epp".

E2E:
- Add a long-name case: a 63-character deployment name serves inference
  and a 64-character name is rejected.
- Shorten Helm release names over 53 characters with a hash suffix so the
  Helm backend can install 63-character deployment names.
- Split EPPServiceName into EPPDeploymentName and EPPSelector.
- Add chart and Helm backend unit tests for the name limits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>

@zhuangqh zhuangqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. please add "BREAKING" to pr title.
backward compatibility: Delete existing deployment will result as object leaking.

Remove the long-name e2e case until KAITO's nightly image includes
kaito-project/kaito#2450. Older KAITO names a long InferenceSet's child
Workspace "<inferenceset>-<5 chars>", so the StatefulSet's
controller-revision-hash pod label exceeds 63 characters and the inference
pod is never created.

TestModelDeploymentNameLength matched Helm 3's schema error text only; CI
installs Helm 4, which reports "at '/name': maxLength: got 64, want 63".
Accept both formats.

pollUntilReady now keeps the previous check's error when the final check is
cut short by the deadline, so a timeout reports what was not ready instead
of a client rate limiter error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
@rambohe-ch rambohe-ch changed the title fix(modeldeployment): keep EPP names and labels within 63 characters fix(BREAKING): keep EPP names and labels within 63 characters Oct 21, 2026

This branch was successfully deployed

1 active deployment
e2e-test — 34a3a411 Deployed Oct 21, 2026 by rambohe-ch via e2e-tests #653
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants