docs: Document runpodctl model commands, model-remove endpoint safety, and serverless update model flags - #710
Conversation
Document the previously-undocumented runpodctl model command family (add, list, remove), including the new --wait-for-hash/-v flags on model add and the --hash/--version version-removal flags on model remove from runpodctl PR #302. Add model to the command groups table and register the new reference page in navigation.
Add the --hash-timeout flag to the model add reference and document the RUNPOD_API_KEY, RUNPOD_GRAPHQL_URL, and RUNPOD_API_URL environment variables (which override config file values) introduced in runpodctl PR #302.
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
| Directory containing the model files to upload. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="--wait-for-hash" type="bool" default="false"> |
There was a problem hiding this comment.
Documented the new --wait-for-hash and -v, --verbose flags on runpodctl model add from PR #302's cmd/model/addModelToRepo.go (flag registration and the --wait-for-hash requires --model-path constraint), plus the progress bar and total-model-size output added in the same file.
| Model name. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="--owner" type="string"> |
There was a problem hiding this comment.
Documented the new --hash and --version flags on runpodctl model remove from PR #302's cmd/model/removeModel.go, including the mutual-exclusivity check (only one of --hash or --version can be provided) and that a matching version is marked removed rather than deleting the whole model.
| Wait for uploaded model files to finish hashing before the command returns. Requires `--model-path`. When hashing completes, `runpodctl` prints a message confirming the model is ready to deploy along with its model URL. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="--hash-timeout" type="duration" default="30m0s"> |
There was a problem hiding this comment.
Documented the new --hash-timeout duration flag on model add; its default of 30m0s and the "0 disables the timeout" behavior come from the flag registration and modelHashWaitTimeout = 30 * time.Minute in cmd/model/addModelToRepo.go.
|
|
||
| ## Environment variables | ||
|
|
||
| You can also configure the CLI using environment variables. When set, these take precedence over the corresponding values in your config file. |
There was a problem hiding this comment.
Documented the RUNPOD_API_KEY, RUNPOD_GRAPHQL_URL, and RUNPOD_API_URL environment variables and their precedence over config values, based on the new internal/configenv/configenv.go package: APIKeyEnv→apiKey, GraphQLURLEnv→apiUrl, RESTURLEnv→restApiUrl, with envOrConfig giving a non-empty env var priority over the config file.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Add coverage for the --huggingface-model flag (runpodctl PR #320, STO-330) that mirrors a Hugging Face model into the Runpod model repository server-side, including source-vs-destination semantics and mutual exclusivity with the local-upload flags.
promptless
Bot
changed the title
jebenexer
left a comment
jebenexer
left a comment
There was a problem hiding this comment.
Local review: checked out this branch against current main (post-#320 merge) — merges cleanly, no conflicts despite being a bit behind. Cross-verified every documented flag and env var against the actual source:
model add/model list/model removeflags matchcmd/model/*.goexactly, including the--huggingface-modelmirroring semantics from the now-merged runpodctl#320.RUNPOD_API_KEY/RUNPOD_GRAPHQL_URL/RUNPOD_API_URLprecedence verified againstinternal/configenv/configenv.go— all correct.- Links to
runpodctl-serverless.mdx/runpodctl-config.mdxresolve. CI (Mintlify deploy/validation, tooltip imports) is clean.
No blockers. Left 2 minor comments (missing required annotations on --name/--owner, and an optional callout for the undocumented RUNPOD_INVOKE_URL). Approving — this is the accurate, current reference and should be the canonical runpodctl model doc (see #644, recommended for closure as a stale duplicate).
| To mirror an existing Hugging Face model into the Runpod model repository, use `--huggingface-model` with the source model in `owner/repo` form. `--name` sets the destination model name in the Runpod repository, and `--owner` is optional when mirroring. | ||
|
|
||
| The transfer runs server-side, so there is no local upload session, file upload, or progress bar. Runpod creates the new model version right away and transfers the files from Hugging Face in the background. | ||
|
|
There was a problem hiding this comment.
Nit: --name isn't marked required here, but the CLI enforces it via cobra.MarkFlagRequired("name") on model add. Similarly, in the "Remove a model" section below, --owner and --name are both mandatory at runtime (runRemoveModel errors with "both --owner and --name must be provided" if either is missing) but neither ResponseField has a required attribute. Worth adding required to these three so users don't have to discover it from a CLI error.
| The Runpod GraphQL API endpoint URL. Overrides the `apiUrl` value in your config file. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="RUNPOD_API_URL" type="string"> |
There was a problem hiding this comment.
Verified against internal/configenv/configenv.go on current main — RUNPOD_API_KEY→apiKey, RUNPOD_GRAPHQL_URL→apiUrl, RUNPOD_API_URL→restApiUrl all check out exactly as documented. One optional addition: there's also a fourth env var, RUNPOD_INVOKE_URL (overrides invokeUrl, used for invoking Serverless endpoints specifically rather than the control-plane API). Not a correctness issue since it's arguably out of scope here, just flagging in case you want full coverage on this page.
|
Just a reminder: If you'd like me to act on any feedback you have via Github comments, just type @Promptless in your suggestion and I'll get right on it! (I won't show up in the user dropdown, but I'll process any request that has @Promptless in the comment body.) |
STO-360 (runpodctl PR #334): runpodctl model remove now refuses to remove a model or version still referenced by a Serverless endpoint, with no override flag. Document the refusal behavior, the detach or replace remediation via serverless update, and the current GPU-endpoint limitation on --model-reference/--clear-models. Add both flags to the serverless update reference. STO-446 (runpodctl PR #333): document the new --delete-my-model-files-after-upload flag on runpodctl model add.
| Directory containing the model files to upload. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="--huggingface-model" type="string"> |
There was a problem hiding this comment.
Documented the new --huggingface-model flag on runpodctl model add (PR #320, STO-330) for mirroring a Hugging Face model into the Runpod model repository in owner/repo form. Source: flag registration with help text "hugging face model to mirror (owner/repo)" at addModelToRepo.go#L186; --name required via MarkFlagRequired at L179-180 and --owner registered with no required-flag call at L184, confirming --owner is optional when mirroring; isHuggingFaceMirror()/wantsUploadSession() at L202-213 show the mirror path skips the local upload session (no file upload/progress bar) since metadata's "server-side Hugging Face mirror" exception is called out in the code comment; the mirror success path returns immediately via printModelAddOutput at L302-304 with no progress bar. Covers prose at lines 18, 28, and 30 of the doc (server-side transfer, no local upload session/progress bar) but NOT the doc's specific "pending transfer state" wording, which is not backed by any status literal in this file (only ModelVersionStatusReady="READY" and ModelVersionStatusPodRemoved="POD_REMOVED" are defined in api/model.go) — flagged separately as needing softening.
|
|
||
| The transfer runs server-side, so there is no local upload session, file upload, or progress bar. Runpod creates the new model version right away and transfers the files from Hugging Face in the background. | ||
|
|
||
| Mirroring cannot be combined with the local-upload flags (`--model-path`, `--create-upload`, `--file-name`, `--file-size`, `--part-size`, `--content-type`, and `--wait-for-hash`). Uploading and mirroring are two separate ways to add a model. |
There was a problem hiding this comment.
Documented that --huggingface-model cannot be combined with the local-upload flags (--model-path, --create-upload, --file-name, --file-size, --part-size, --content-type, --wait-for-hash). Source: validateAddModelFlags() at addModelToRepo.go#L379-389, which returns the exact error "--huggingface-model cannot be combined with local upload flags" (L385) when any of those seven flags is set alongside isHuggingFaceMirror(). Also anchors the verbatim example at line 35 of the doc (runpodctl model add --name tiny-llm --huggingface-model arnir0/Tiny-LLM), matching the cobra Example field at addModelToRepo.go#L162-163 verbatim, including the destination/source comment.
|
|
||
| When you upload from a directory in a terminal, `runpodctl` shows a progress bar with upload progress and estimated time remaining, then prints the total model size once the upload finishes. | ||
|
|
||
| If you reuse one local directory across model versions without this flag, a later upload can include a mix of old and new files. Add `--delete-my-model-files-after-upload` to remove exactly the files this run uploaded from `--model-path` once the model version hash is confirmed server-side. It never touches unrelated files, and it runs only on confirmed success, so a failed or timed-out upload leaves your files in place. This flag requires both `--model-path` and `--wait-for-hash`. |
There was a problem hiding this comment.
Documented the new --delete-my-model-files-after-upload flag intro/example (PR #333, STO-446) on runpodctl model add: it removes exactly the files this run uploaded once the hash is confirmed server-side, never touches unrelated files, and only runs on confirmed success (a failed/timed-out upload leaves files in place). Source: runAddModel's post-hash-wait block at addModelToRepo.go#L354-365 only calls deleteVerifiedModelFiles after waitForUploadedModelHash returns without error; deleteVerifiedModelFiles (L504-525) iterates only over the modelFiles collected for this run.
| Maximum time to wait when `--wait-for-hash` is set. Set to `0` to wait indefinitely. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="--delete-my-model-files-after-upload" type="bool" default="false"> |
There was a problem hiding this comment.
Documented the --delete-my-model-files-after-upload flag definition (bool, default false; requires both --model-path and --wait-for-hash; deletes only this run's uploaded files including nested subdirectories after hash confirmation; no deletion on hash failure/timeout; non-zero exit naming the failing file on partial delete failure; deletedModelFiles/deletedModelFilesBytes added to JSON output only under --verbose/-v). Source: PR #333 (STO-446), validateAddModelFlags at addModelToRepo.go#L402-413 (validation before any upload), deleteVerifiedModelFiles at L504-525 (exact-files-only deletion, failure naming), and the verbose-gated output path at L367-371 (printCompactModelAddOutput when !addModelVerbose vs full modelAddOutput with DeletedModelFiles/DeletedModelFilesBytes fields, addModelAddOutput struct L111-120).
| runpodctl model remove --name "my-model" --owner "my-team" --version "<version-uuid>" | ||
| ``` | ||
|
|
||
| <Warning> |
There was a problem hiding this comment.
Documented that runpodctl model remove refuses to remove a model/version referenced by a Serverless endpoint (whole-model remove blocked if ANY version is referenced; --hash/--version remove blocked only for that exact version); reports the blocking endpoint(s); no override/force flag exists; a failed dependency check (e.g. API unreachable) also blocks removal. Source: PR #334 (STO-360), checkDependentEndpoints/dependentEndpointsError at removeModel.go#L38-52, L168-184; whole-model call site with hash="" at L242 vs version-specific call with target.Hash at L281; no --force flag registered in bindRemoveModelFlags (L219-224).
|
|
||
| </Warning> | ||
|
|
||
| To clear a blocked removal, detach or replace the model on the referencing endpoint, then retry the removal. Detach the model with `runpodctl serverless update <endpoint-id> --clear-models`, or replace it with `runpodctl serverless update <endpoint-id> --model-reference <model-reference-url>`. See the [Update an endpoint](/runpodctl/reference/runpodctl-serverless#update-an-endpoint) section for details. If the referencing endpoint is a GPU endpoint, this update currently fails until the pending fix ships; see the [limitation on `--model-reference` and `--clear-models`](/runpodctl/reference/runpodctl-serverless#update-flags). |
There was a problem hiding this comment.
Documented the remediation path for a blocked removal: detach with runpodctl serverless update <endpoint-id> --clear-models or replace with --model-reference <new-url>, then retry. Source: PR #334 (STO-360), dependentEndpointsError.Error() at removeModel.go#L47-52, which emits this exact remediation text.
There was a problem hiding this comment.
(Line 208)
Documented the new --model-reference (repeatable string, replaces existing model references) and --clear-models (bool, mutually exclusive with --model-reference) flags on runpodctl serverless update (PR #334, STO-360). Source: update.go#L43-45 flag vars, L55-56 registration, L62-64 mutual-exclusivity check (--clear-models and --model-reference are mutually exclusive); replace semantics confirmed by UpdateEndpointModels in internal/api/endpoints.go#L253-337 (full-replace saveEndpoint call with the new modelRefs).
|
|
||
| Attaching model references caches the referenced models on the endpoint so workers start faster and cost less. See [Model caching](/serverless/endpoints/model-caching) for when and why to cache models. | ||
|
|
||
| <ResponseField name="--model-reference" type="string"> |
There was a problem hiding this comment.
Documented the known limitation that --model-reference/--clear-models currently work only for CPU (non-GPU) endpoints, and that using them on a GPU endpoint drops GPU config and fails (fix pending). This is called out explicitly in the PR #334 description as a known, separate, unfixed issue ("UpdateEndpointModels also drops GPU config ... saveEndpoint then fails with "gpuId(s) is required for a gpu endpoint" on any real GPU endpoint... still broken today for GPU endpoints created via serverless create --gpu-id"), consistent with internal/api/endpoints.go's Endpoint.GpuIDs being a single string field.
Source: runpod/runpodctl#334
promptless
Bot
changed the title
1 participant
Summary
Documents the
runpodctlmodel repository commands and related Serverless CLI behavior across several merged runpodctl PRs.runpodctl modelreference (PR #302, #320)runpodctl modelreference page covering the model repository commands (add,list,remove), including the--wait-for-hash,--hash-timeout, and-v/--verboseflags onmodel addand the mutually exclusive--hash/--versionversion-removal flags onmodel remove.--huggingface-modelflag onrunpodctl model add(PR Add timeout configuration guidance for large multipart uploads #320, STO-330) for mirroring a Hugging Face model into the Runpod model repository server-side, including source-versus-destination semantics, the optional--ownerflag, and mutual exclusivity with the local-upload flags.modelto the CLI overview command groups table and registers the page in navigation.RUNPOD_API_KEY,RUNPOD_GRAPHQL_URL, andRUNPOD_API_URLenvironment variables on theconfigreference page.Model remove dependency safety (PR #334, STO-360)
runpodctl model removenow refuses to remove a model or version while a Serverless endpoint still references it, that there is no override or force flag, and that a dependency check that cannot complete also blocks the removal.runpodctl serverless update --clear-models/--model-reference, and adds those two flags to theserverless updatereference.--model-reference/--clear-modelsreject the update on GPU endpoints (a fix is pending), while noting GPU endpoints can still attach model references at creation or via the console.Model file cleanup flag (PR #333, STO-446)
--delete-my-model-files-after-uploadflag onrunpodctl model add, which removes exactly the files a run uploaded once the model version hash is confirmed server-side, its requirement of both--model-pathand--wait-for-hash, its failure behavior, and the verbose-onlydeletedModelFiles/deletedModelFilesBytesJSON output fields.Trigger events