Repository navigation
fix(BREAKING): keep EPP names and labels within 63 characters - #246
Open
rambohe-ch wants to merge 2 commits into
Open
rambohe-ch wants to merge 2 commits into
rambohe-ch wants to merge 2 commits into
Conversation
The EPP Service name and the EPP pods' inferencepool selector label value were "<name>-inferencepool-epp", which exceeds the 63-character Service name and label value limits for long model names (e.g. the AIManager smoke test with nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16). The InferencePool name "<name>-inferencepool" has the same problem, because Istio copies it into the istio.io/inferencepool-name label on the pool's shadow Service. Name the InferencePool and the EPP Service after the deployment, use that name as the inferencepool label value, and cap the chart's name value at 63 characters. The EPP Deployment, ServiceAccount, Role, RoleBinding and ConfigMap keep "<name>-inferencepool-epp". E2E: - Add a long-name case: a 63-character deployment name serves inference and a 64-character name is rejected. - Shorten Helm release names over 53 characters with a hash suffix so the Helm backend can install 63-character deployment names. - Split EPPServiceName into EPPDeploymentName and EPPSelector. - Add chart and Helm backend unit tests for the name limits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
rambohe-ch
requested review from
Fei-Guo,
techworldhello and
zhuangqh
as code owners
October 21, 2026 04:21
zhuangqh
approved these changes
Oct 21, 2026
zhuangqh
left a comment
Contributor
There was a problem hiding this comment.
LGTM. please add "BREAKING" to pr title.
backward compatibility: Delete existing deployment will result as object leaking.
Remove the long-name e2e case until KAITO's nightly image includes kaito-project/kaito#2450. Older KAITO names a long InferenceSet's child Workspace "<inferenceset>-<5 chars>", so the StatefulSet's controller-revision-hash pod label exceeds 63 characters and the inference pod is never created. TestModelDeploymentNameLength matched Helm 3's schema error text only; CI installs Helm 4, which reports "at '/name': maxLength: got 64, want 63". Accept both formats. pollUntilReady now keeps the previous check's error when the final check is cut short by the deadline, so a timeout reports what was not ready instead of a client rate limiter error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for Change:
The modeldeployment chart named the EPP Service
<name>-inferencepool-eppand used the same value for the EPP pods'inferencepoollabel. Both are limited to 63 characters, so any deployment name over 45 characters failed to install. The AIManager smoke test withnvidia/NVIDIA-Nemotron-3-Nano-4B-BF16hit this. The InferencePool name<name>-inferencepoolhad the same problem: Istio copies it into theistio.io/inferencepool-namelabel on the pool's shadow Service.This PR names the InferencePool and the EPP Service after the deployment, and uses the deployment name as the
inferencepoollabel value. The chart'snamevalue is now capped at 63 characters.E2E:
EPPServiceNameintoEPPDeploymentNameandEPPSelector, following the new naming.nightly-latestimage. Older KAITO creates a Workspace StatefulSet whose pod label exceeds 63 characters, so the inference pod is never created.<name>-inferencepoolto<name>. Anything that refers to the pool by name must be updated: external HTTPRoutes, scripts,kubectlcommands.<name>-inferencepool-eppto<name>.inferencepoolchanges from<name>-inferencepool-eppto<name>. This label is in the EPP Deployment'sspec.selector, which Kubernetes does not allow to change, sohelm upgradeof an existing release fails withspec.selector: ... field is immutable. Before upgrading, delete the EPP Deployment and let Helm recreate it:helm uninstallfollowed byhelm install.inferencepoolpod label onto EPP metrics, and EPP labels its pool metrics with the InferencePool name. Update any dashboard, alert or recording rule that filters oninferencepool="<name>-inferencepool-epp"or on the pool name<name>-inferencepool.nameis limited to 63 characters. The chart schema now rejects longer names athelm installtime. Such names never deployed successfully anyway.Unchanged: the EPP Deployment, ServiceAccount, Role, RoleBinding and ConfigMap keep
<name>-inferencepool-epp, so the status reporter and the benchmark workflow need no changes.Requirements
Issue Fixed:
Notes for Reviewers: