From d945e75fc695ef74d98408258442873ffa559166 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 7 Jul 2026 08:37:10 -0400 Subject: [PATCH 01/21] Create modelrepotest.mdx --- serverless/modelrepotest.mdx | 121 +++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 serverless/modelrepotest.mdx diff --git a/serverless/modelrepotest.mdx b/serverless/modelrepotest.mdx new file mode 100644 index 00000000..99777a29 --- /dev/null +++ b/serverless/modelrepotest.mdx @@ -0,0 +1,121 @@ +--- +title: "Model Repo testing" +sidebarTitle: "Model Repo testing" +description: "Upload a model to Model Repo and deploy it to a Serverless endpoint." +--- + +## Scripted testing + +A shell script is available that runs all of the steps below automatically using mock model files. Download `model_repo_testing.sh` from the internal Notion page and run it to validate end-to-end without manually executing each step. + +## Manual testing + +### Prerequisites + +- Your email is [feature-flagged](https://us.posthog.com/project/105711/feature_flags/299352) for Model Repo access. +- The following environment variables are exported: + +```bash +export RUNPOD_API_URL="https://rest.runpod.io/v1" +export RUNPOD_GRAPHQL_URL="https://api.runpod.io/graphql" +export RUNPOD_API_KEY="your-api-key" + +export MODEL_NAME="$(whoami)-test-$(date +%s)" # must be unique per test run +export MODEL_PATH="/path/to/model" +``` + +- `MODEL_NAME` should be unique for each test run. Reusing the same name uploads a new version of the existing model rather than creating a new one. +- `jq` is installed for parsing JSON output. + +### Step 1: Build runpodctl + +```bash +git clone git@github.com:runpod/runpodctl.git +cd runpodctl +make +``` + +### Step 2: Upload the model + +```bash +./bin/runpodctl model add \ + --name "$MODEL_NAME" \ + --model-path "$MODEL_PATH" \ + --create-upload +``` + +This outputs a JSON string listing all uploaded files. + +### Step 3: Wait for the model to be hashed + +After upload, the model must be hashed by an asynchronous background process. This typically completes in a few minutes but can take up to 10–15 minutes. + +Poll until the hash field is non-null: + +```bash +./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +``` + +While hashing is in progress, the command returns `null`: + +```bash +% ./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +null +``` + +Once hashing is complete, it returns the hash value: + +```bash +% ./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +71a311bdf0ca44119ed74dbef8cf573bc89b58cbc48a10fe508f756ebb1922dc +``` + +### Step 4: Build the model URL + +```bash +export USER_ID="$(./bin/runpodctl user | jq -r '.id')" +export MODEL_HASH="$(./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash')" +export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}" +``` + +### Step 5: Deploy a Serverless endpoint with the model attached + +```bash +./bin/runpodctl serverless create \ + --name "$(whoami)_ctl_test" \ + --template-id "mockworker" \ + --gpu-id "AMPERE_24" \ + --workers-max 3 \ + --workers-min 1 \ + --model-reference "$MODEL_URL" +``` + +Notes: +- `--model-reference` is only supported with `--template-id` and GPU endpoints. +- `--gpu-id` accepts a single GPU ID — do not pass a comma-separated list. +- `--model-reference` is repeatable if multiple models need to be attached. + +### Step 6: Verify the model is attached to the worker + +1. Go to **Serverless** in the left navigation bar under **Resources**. +2. Select the endpoint you created (`ctl_test` if you used the commands above). +3. Click the **Workers** tab. +4. Select a worker showing a **Running** status. +5. Click **Connect**, then use the `ssh` command or the **Web Terminal**. +6. Run the following to confirm your model files are present: + +```bash +find /runpod-volume/huggingface-cache/hub/models--$(echo $MODEL_NAME | sed 's@/@--@g')/snapshots/${MODEL_REVISION} -type f +``` + + +There is currently no way to retrieve SSH connection details for a running Serverless worker via `runpodctl`. Use the web UI to connect. + + +### Step 7: Clean up + +Delete the endpoint after testing to stop spend. Use the web UI or: + +```bash +./bin/runpodctl serverless delete +``` From f0f0a00f04c0799c95855085d84b2a834a3f04cf Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 7 Jul 2026 09:10:47 -0400 Subject: [PATCH 02/21] Update docs.json --- docs.json | 1 + 1 file changed, 1 insertion(+) diff --git a/docs.json b/docs.json index 474ced9a..d5c205a5 100644 --- a/docs.json +++ b/docs.json @@ -120,6 +120,7 @@ "serverless/development/dual-mode-worker" ] }, + "serverless/modelrepotest", { "group": "Manage endpoints", "pages": [ From a3d92158c1c6d92235d5dff482a0ec0a69eb0c47 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Wed, 8 Jul 2026 15:29:08 +0000 Subject: [PATCH 03/21] Add Install Go as step one in Model Repo testing guide --- serverless/modelrepotest.mdx | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/serverless/modelrepotest.mdx b/serverless/modelrepotest.mdx index 99777a29..c81dc363 100644 --- a/serverless/modelrepotest.mdx +++ b/serverless/modelrepotest.mdx @@ -27,7 +27,15 @@ export MODEL_PATH="/path/to/model" - `MODEL_NAME` should be unique for each test run. Reusing the same name uploads a new version of the existing model rather than creating a new one. - `jq` is installed for parsing JSON output. -### Step 1: Build runpodctl +### Step 1: Install Go + +Building runpodctl requires Go: + +```bash +brew install go +``` + +### Step 2: Build runpodctl ```bash git clone git@github.com:runpod/runpodctl.git @@ -35,7 +43,7 @@ cd runpodctl make ``` -### Step 2: Upload the model +### Step 3: Upload the model ```bash ./bin/runpodctl model add \ @@ -46,7 +54,7 @@ make This outputs a JSON string listing all uploaded files. -### Step 3: Wait for the model to be hashed +### Step 4: Wait for the model to be hashed After upload, the model must be hashed by an asynchronous background process. This typically completes in a few minutes but can take up to 10–15 minutes. @@ -70,7 +78,7 @@ Once hashing is complete, it returns the hash value: 71a311bdf0ca44119ed74dbef8cf573bc89b58cbc48a10fe508f756ebb1922dc ``` -### Step 4: Build the model URL +### Step 5: Build the model URL ```bash export USER_ID="$(./bin/runpodctl user | jq -r '.id')" @@ -78,7 +86,7 @@ export MODEL_HASH="$(./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[ export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}" ``` -### Step 5: Deploy a Serverless endpoint with the model attached +### Step 6: Deploy a Serverless endpoint with the model attached ```bash ./bin/runpodctl serverless create \ @@ -95,7 +103,7 @@ Notes: - `--gpu-id` accepts a single GPU ID — do not pass a comma-separated list. - `--model-reference` is repeatable if multiple models need to be attached. -### Step 6: Verify the model is attached to the worker +### Step 7: Verify the model is attached to the worker 1. Go to **Serverless** in the left navigation bar under **Resources**. 2. Select the endpoint you created (`ctl_test` if you used the commands above). @@ -112,7 +120,7 @@ find /runpod-volume/huggingface-cache/hub/models--$(echo $MODEL_NAME | sed 's@/@ There is currently no way to retrieve SSH connection details for a running Serverless worker via `runpodctl`. Use the web UI to connect. -### Step 7: Clean up +### Step 8: Clean up Delete the endpoint after testing to stop spend. Use the web UI or: From 6ff93f52e97fda67fb028f119a77d35b43498665 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Mon, 13 Jul 2026 11:04:53 -0400 Subject: [PATCH 04/21] Update modelrepotest.mdx --- serverless/modelrepotest.mdx | 150 ++++++++++++++++++++++------------- 1 file changed, 95 insertions(+), 55 deletions(-) diff --git a/serverless/modelrepotest.mdx b/serverless/modelrepotest.mdx index c81dc363..efe70099 100644 --- a/serverless/modelrepotest.mdx +++ b/serverless/modelrepotest.mdx @@ -1,129 +1,169 @@ --- title: "Model Repo testing" -sidebarTitle: "Model Repo testing" description: "Upload a model to Model Repo and deploy it to a Serverless endpoint." --- + +Model Repo is currently in alpha and is available on Mac and Linux only. Windows support is coming soon. + + +## Why use Model Repo + +Model Repo lets you upload your own models to private storage on Runpod and attach them directly to Serverless endpoints. Key benefits: + +- **Faster cold starts**: Models are pre-cached on the worker host rather than downloaded at runtime. +- **No HuggingFace dependency**: Your models are stored in Runpod's infrastructure, so endpoints don't require an outbound download on every cold start. +- **Private storage**: Models are stored in your account and are not accessible to other users. + +--- + ## Scripted testing A shell script is available that runs all of the steps below automatically using mock model files. Download `model_repo_testing.sh` from the internal Notion page and run it to validate end-to-end without manually executing each step. +--- + ## Manual testing ### Prerequisites -- Your email is [feature-flagged](https://us.posthog.com/project/105711/feature_flags/299352) for Model Repo access. -- The following environment variables are exported: +- Your email is feature-flagged for Model Repo access. +- `jq` is installed for parsing JSON output. + +### Set environment variables + +Export the following before running any commands. **Make sure to set your actual API key — missing this is the most common source of auth errors later.** ```bash export RUNPOD_API_URL="https://rest.runpod.io/v1" export RUNPOD_GRAPHQL_URL="https://api.runpod.io/graphql" -export RUNPOD_API_KEY="your-api-key" +export RUNPOD_API_KEY="your-api-key" # replace with your actual API key -export MODEL_NAME="$(whoami)-test-$(date +%s)" # must be unique per test run -export MODEL_PATH="/path/to/model" +export MODEL_NAME="$(whoami)-test-$(date +%s)" # unique name per test run — reusing the same name uploads a new version, not a new model +export MODEL_PATH="/path/to/model" # local path to the model files you want to upload ``` -- `MODEL_NAME` should be unique for each test run. Reusing the same name uploads a new version of the existing model rather than creating a new one. -- `jq` is installed for parsing JSON output. + +`MODEL_NAME` must be unique for each test run. If you reuse the same name, the upload creates a new version of the existing model rather than a new model. + -### Step 1: Install Go +--- + +### Step 1: Install runpodctl -Building runpodctl requires Go: +**Option A: Install via Homebrew (recommended)** ```bash -brew install go +brew install runpod/runpodctl/runpodctl ``` -### Step 2: Build runpodctl +**Option B: Build from source** ```bash +brew install go # install Go, required to build runpodctl git clone git@github.com:runpod/runpodctl.git cd runpodctl -make +make # builds the binary to ./bin/runpodctl ``` -### Step 3: Upload the model + +If you build from source, the binary is at `./bin/runpodctl`. Either run it with that path, or add `./bin` to your `PATH`. The steps below use `runpodctl` — adjust accordingly. + + +--- + +### Step 2: Upload the model ```bash -./bin/runpodctl model add \ - --name "$MODEL_NAME" \ - --model-path "$MODEL_PATH" \ - --create-upload +runpodctl model add \ + --name "$MODEL_NAME" \ # the name to register the model under in your repo + --model-path "$MODEL_PATH" \ # local path to the model files + --create-upload # creates the upload session and transfers files ``` This outputs a JSON string listing all uploaded files. -### Step 4: Wait for the model to be hashed +--- + +### Step 3: Wait for the model to be hashed After upload, the model must be hashed by an asynchronous background process. This typically completes in a few minutes but can take up to 10–15 minutes. -Poll until the hash field is non-null: +Poll until the `hash` field is non-null: ```bash -./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' ``` While hashing is in progress, the command returns `null`: -```bash -% ./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +``` +% runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' null ``` Once hashing is complete, it returns the hash value: -```bash -% ./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +``` +% runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' 71a311bdf0ca44119ed74dbef8cf573bc89b58cbc48a10fe508f756ebb1922dc ``` -### Step 5: Build the model URL +--- + +### Step 4: Build the model URL ```bash -export USER_ID="$(./bin/runpodctl user | jq -r '.id')" -export MODEL_HASH="$(./bin/runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash')" -export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}" +export USER_ID="$(runpodctl user | jq -r '.id')" # your Runpod user ID +export MODEL_HASH="$(runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash')" # the hash from step 3 +export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}" # the full model reference URL ``` -### Step 6: Deploy a Serverless endpoint with the model attached +The resulting `MODEL_URL` will look like: -```bash -./bin/runpodctl serverless create \ - --name "$(whoami)_ctl_test" \ - --template-id "mockworker" \ - --gpu-id "AMPERE_24" \ - --workers-max 3 \ - --workers-min 1 \ - --model-reference "$MODEL_URL" +``` +https://local/user1a2b3c4d/myusername-test-1720540800:71a311bdf0ca44119ed74dbef8cf573bc89b58cbc48a10fe508f756ebb1922dc ``` -Notes: -- `--model-reference` is only supported with `--template-id` and GPU endpoints. -- `--gpu-id` accepts a single GPU ID — do not pass a comma-separated list. -- `--model-reference` is repeatable if multiple models need to be attached. - -### Step 7: Verify the model is attached to the worker +--- -1. Go to **Serverless** in the left navigation bar under **Resources**. -2. Select the endpoint you created (`ctl_test` if you used the commands above). -3. Click the **Workers** tab. -4. Select a worker showing a **Running** status. -5. Click **Connect**, then use the `ssh` command or the **Web Terminal**. -6. Run the following to confirm your model files are present: +### Step 5: Deploy a Serverless endpoint with the model attached ```bash -find /runpod-volume/huggingface-cache/hub/models--$(echo $MODEL_NAME | sed 's@/@--@g')/snapshots/${MODEL_REVISION} -type f +runpodctl serverless create \ + --name "$(whoami)_ctl_test" \ # name for the endpoint + --template-id "mockworker" \ # worker template to use + --gpu-id "AMPERE_24" \ # GPU type + --workers-max 3 \ # maximum number of active workers + --workers-min 1 \ # minimum number of workers kept warm + --model-reference "$MODEL_URL" # attaches your model to the endpoint ``` -There is currently no way to retrieve SSH connection details for a running Serverless worker via `runpodctl`. Use the web UI to connect. +`--model-reference` is only supported with `--template-id` and GPU endpoints. It is repeatable if you need to attach multiple models to the same endpoint. -### Step 8: Clean up +--- + +### Step 6: Verify the model is working + +Send a test request to confirm the endpoint is live and the model is accessible. Replace `ENDPOINT_ID` with the ID returned in the previous step: + +```bash +curl -s -X POST "https://api.runpod.ai/v2/${ENDPOINT_ID}/runsync" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "hello"}}' | jq +``` + +A successful response confirms the endpoint is running and the model is attached. If the request fails with an auth error, verify that `RUNPOD_API_KEY` is set correctly. + +--- + +### Step 7: Clean up -Delete the endpoint after testing to stop spend. Use the web UI or: +Delete the endpoint after testing to stop accruing spend. Use the web UI or: ```bash -./bin/runpodctl serverless delete +runpodctl serverless delete ``` From db399f30f0f2af830b56ca9a07b9ea85631db408 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Tue, 14 Jul 2026 14:52:59 +0000 Subject: [PATCH 05/21] Address review comments on modelrepotest.mdx - Simplify MODEL_NAME example to a placeholder - Remove internal-only scripted testing section - Move inline comments off line-continuation (\) lines in shell blocks - Drop unused MODEL_URL variable and its example - Use generic --name, real --hub-id, and split model reference into --model-reference and --env, plus --min-cuda-version workaround - Mention the web UI as an alternative for sending test requests --- serverless/modelrepotest.mdx | 52 +++++++++++++++++++----------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/serverless/modelrepotest.mdx b/serverless/modelrepotest.mdx index efe70099..eb46499e 100644 --- a/serverless/modelrepotest.mdx +++ b/serverless/modelrepotest.mdx @@ -17,12 +17,6 @@ Model Repo lets you upload your own models to private storage on Runpod and atta --- -## Scripted testing - -A shell script is available that runs all of the steps below automatically using mock model files. Download `model_repo_testing.sh` from the internal Notion page and run it to validate end-to-end without manually executing each step. - ---- - ## Manual testing ### Prerequisites @@ -39,7 +33,7 @@ export RUNPOD_API_URL="https://rest.runpod.io/v1" export RUNPOD_GRAPHQL_URL="https://api.runpod.io/graphql" export RUNPOD_API_KEY="your-api-key" # replace with your actual API key -export MODEL_NAME="$(whoami)-test-$(date +%s)" # unique name per test run — reusing the same name uploads a new version, not a new model +export MODEL_NAME="model_name" # unique name per test run — reusing the same name uploads a new version, not a new model export MODEL_PATH="/path/to/model" # local path to the model files you want to upload ``` @@ -75,10 +69,13 @@ If you build from source, the binary is at `./bin/runpodctl`. Either run it with ### Step 2: Upload the model ```bash +# --name: the name to register the model under in your repo +# --model-path: local path to the model files +# --create-upload: creates the upload session and transfers files runpodctl model add \ - --name "$MODEL_NAME" \ # the name to register the model under in your repo - --model-path "$MODEL_PATH" \ # local path to the model files - --create-upload # creates the upload session and transfers files + --name "$MODEL_NAME" \ + --model-path "$MODEL_PATH" \ + --create-upload ``` This outputs a JSON string listing all uploaded files. @@ -111,18 +108,11 @@ Once hashing is complete, it returns the hash value: --- -### Step 4: Build the model URL +### Step 4: Get your user ID and model hash ```bash export USER_ID="$(runpodctl user | jq -r '.id')" # your Runpod user ID export MODEL_HASH="$(runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash')" # the hash from step 3 -export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}" # the full model reference URL -``` - -The resulting `MODEL_URL` will look like: - -``` -https://local/user1a2b3c4d/myusername-test-1720540800:71a311bdf0ca44119ed74dbef8cf573bc89b58cbc48a10fe508f756ebb1922dc ``` --- @@ -130,17 +120,27 @@ https://local/user1a2b3c4d/myusername-test-1720540800:71a311bdf0ca44119ed74dbef8 ### Step 5: Deploy a Serverless endpoint with the model attached ```bash +# --name: name for the endpoint +# --hub-id: the Hub template to deploy +# --gpu-id: GPU type +# --workers-max: maximum number of active workers +# --workers-min: minimum number of workers kept warm +# --model-reference: attaches your model to the endpoint +# --env: sets the model path on the worker +# --min-cuda-version: works around a bug in runpodctl runpodctl serverless create \ - --name "$(whoami)_ctl_test" \ # name for the endpoint - --template-id "mockworker" \ # worker template to use - --gpu-id "AMPERE_24" \ # GPU type - --workers-max 3 \ # maximum number of active workers - --workers-min 1 \ # minimum number of workers kept warm - --model-reference "$MODEL_URL" # attaches your model to the endpoint + --name "my_worker" \ + --hub-id "cm8h09d9n000008jvh2rqdsmb" \ + --gpu-id "AMPERE_24" \ + --workers-max 3 \ + --workers-min 1 \ + --model-reference "https://local/$USER_ID/$MODEL_NAME:$MODEL_HASH" \ + --env MODEL_NAME="/runpod/model-store/modelrepo-local/models/$USER_ID/$MODEL_NAME/$MODEL_HASH" \ + --min-cuda-version "13.0" ``` -`--model-reference` is only supported with `--template-id` and GPU endpoints. It is repeatable if you need to attach multiple models to the same endpoint. +`--model-reference` is only supported with `--hub-id` and GPU endpoints. It is repeatable if you need to attach multiple models to the same endpoint. --- @@ -158,6 +158,8 @@ curl -s -X POST "https://api.runpod.ai/v2/${ENDPOINT_ID}/runsync" \ A successful response confirms the endpoint is running and the model is attached. If the request fails with an auth error, verify that `RUNPOD_API_KEY` is set correctly. +If you prefer a graphical interface to curl, you can also send requests to the worker from the web UI. + --- ### Step 7: Clean up From b25c3c92919f5c3b5597d514cf9dfa07eb896b34 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Mon, 20 Jul 2026 16:17:49 -0400 Subject: [PATCH 06/21] Create overview.mdx --- serverless/storage/modelrepo/overview.mdx | 110 ++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 serverless/storage/modelrepo/overview.mdx diff --git a/serverless/storage/modelrepo/overview.mdx b/serverless/storage/modelrepo/overview.mdx new file mode 100644 index 00000000..d21186c4 --- /dev/null +++ b/serverless/storage/modelrepo/overview.mdx @@ -0,0 +1,110 @@ +--- +title: "Overview" +sidebarTitle: "Overview" +description: "Store, version, and pre-cache your model files on Runpod infrastructure." +--- + + +Model Repo is currently in alpha and is available on Mac and Linux only. Windows support is coming soon. + + +Model Repo is a private model storage service built into Runpod. Upload your model files once, and Runpod caches them directly on your Serverless worker hosts so they are ready before the worker starts — no external download required at cold start time. + +## How it works + +When you create a Serverless endpoint with a model attached, Runpod fetches the model files from storage and caches them on the host machine before the worker container starts. Once the worker boots, the files are already on disk. This eliminates the download step that normally adds seconds or minutes to a cold start. + +Model files are stored in your private Runpod account on Cloudflare R2. Each uploaded version is content-addressed: after the upload, Runpod computes a hash of the files, and you use that hash to pin a specific version when configuring an endpoint. + +The model reference URL format is: + +``` +https://local/{user-id}/{model-name}:{hash} +``` + +You pass this URL as the `--model-reference` flag when creating or updating an endpoint. Runpod uses it to know exactly which version of which model to cache on the host. + +## What you can upload + + + + + + +Model Repo accepts model files in any format. There is no restriction on file type — you can upload PyTorch checkpoints, GGUF files, safetensors, ONNX models, or any other format your worker needs. + +## How the model is available inside the worker + +When the worker starts, your model files are available at this path inside the container: + +``` +/runpod/model-store/modelrepo-local/models/{user-id}/{model-name}/{model-hash} +``` + +Pass this path to the worker as an environment variable using the `--env` flag when creating the endpoint: + +``` +--env MODEL_NAME="/runpod/model-store/modelrepo-local/models/$USER_ID/$MODEL_NAME/$MODEL_HASH" +``` + +Your handler function reads `MODEL_NAME` to know where to load the model from. Model Repo only handles delivering the files to the host — your handler is fully responsible for loading the model and processing requests. + + + +For queue-based endpoints, you write a handler function that receives requests, loads the model from `MODEL_NAME`, runs inference, and returns results. For load balancing endpoints (FastAPI, Flask, etc.), your server process loads the model from that path on startup. + +## Model Repo vs network volumes + +Both Model Repo and [network volumes](/storage/network-volumes) can make model files available to Serverless workers. The key difference is when and how the files are delivered. + +| | Model Repo | Network volume | +|---|---|---| +| **Delivery** | Cached on host before worker starts | Mounted as a network filesystem at runtime | +| **Cold start impact** | Lower — files already on disk | Higher — network mount adds latency | +| **Best for** | Fixed, versioned model weights | Frequently changing files, shared state | +| **Versioning** | Built-in, content-addressed by hash | Manual — you manage file paths yourself | +| **Multi-worker sharing** | Not shared across workers | Shared across all workers on the same volume | +| **Storage billing** | | $0.07/GB/month | + +Use Model Repo when: +- You have a fixed set of model weights you want to distribute reliably to workers. +- You want shorter cold starts without depending on HuggingFace or any external download source. +- You need version control over exactly which model checkpoint is deployed. + +Use a network volume when: +- Workers need to read and write shared files during a run (e.g., checkpoints, intermediate outputs). +- Your files change frequently and you want all workers to see the latest version immediately. +- You need the same set of files available to many concurrent workers simultaneously. + +## Switching from your current setup + +The effort to adopt Model Repo depends on how you currently deliver your model files to workers. + +**If you download from HuggingFace or a URL at cold start:** + +1. Upload your model files to Model Repo using `runpodctl`. +2. Record the model URL (contains the user ID, name, and hash). +3. Add `--model-reference` to your endpoint configuration. +4. Remove the download logic from your handler or startup script. + +The files will be available at a local path when the worker starts, so your loading code changes from a `hf_hub_download()` call (or equivalent) to a local file open. Everything downstream — your handler function structure, request format, response format — stays the same. + +**If your model weights are baked into your Docker image:** + +Baking weights into a Docker image inflates image size and increases cold start times (Runpod must pull the full image on each new host). Moving weights out into Model Repo and shrinking your Docker image typically improves cold start performance. + +To migrate: +1. Remove the model weights from your Dockerfile and rebuild without them. +2. Upload the weights separately to Model Repo. +3. Update your endpoint to reference the model URL. +4. Update your worker code to load the model from the local path instead of from within the image. + + + +**If you cache models on a network volume:** + +You can switch to Model Repo if your model weights are stable and you do not need workers to write to the same storage location. The primary benefit is faster host-level caching and simpler version management. If your workers currently write to the network volume (for checkpoints, fine-tuning outputs, etc.), keep the network volume for that purpose. + +## Getting started + +See [Model Repo testing](/serverless/model-repo/testing) for a step-by-step guide to uploading a model and deploying it to a Serverless endpoint. From 54a576e046b2b0272da7a1344733e69e25dc455d Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Mon, 20 Jul 2026 16:18:50 -0400 Subject: [PATCH 07/21] Create security --- serverless/storage/modelrepo/security | 40 +++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 serverless/storage/modelrepo/security diff --git a/serverless/storage/modelrepo/security b/serverless/storage/modelrepo/security new file mode 100644 index 00000000..61cecf6a --- /dev/null +++ b/serverless/storage/modelrepo/security @@ -0,0 +1,40 @@ +--- +title: "Security" +sidebarTitle: "Security" +description: "How Model Repo stores and protects your model data." +--- + +## Storage + +Model Repo stores your model files in Runpod's infrastructure, backed by [Cloudflare R2](https://developers.cloudflare.com/r2/). Your data is isolated to your Runpod account and is not accessible to other users. + +## Encryption + +All model data is encrypted at every stage: + +- **At rest**: AES-256-GCM encryption, applied by default to all objects stored in Cloudflare R2. +- **In transit**: TLS encryption for all transfers — between your machine and Runpod when uploading, and between Runpod's systems when caching models on worker hosts. + +## Your data is yours + +Runpod does not access, analyze, or use your model files for any purpose. Models stored in Model Repo are not used for training, evaluation, or any other internal purpose. Only you can access your models through your account credentials. + +## Compliance + + + +Runpod maintains industry-standard security certifications. For a full overview of Runpod's security posture, certifications, and policies, see [runpod.io/security](https://runpod.io/security). + +## Frequently asked questions + +**Does Runpod use my models to train other models?** + +No. Your model files are stored privately and are never used by Runpod for any purpose. + +**Can other Runpod users access my models?** + +No. Models are scoped to your Runpod account. Other users cannot access or list your models. + +**Who manages the storage infrastructure?** + +Cloudflare R2 is the underlying object storage backend. Cloudflare applies encryption-at-rest using AES-256-GCM by default. Runpod manages the access layer and authentication — only your Runpod API key can retrieve your models. From 47c84f76283e23fb10e08385b962561aa0bbfc65 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Mon, 20 Jul 2026 16:19:46 -0400 Subject: [PATCH 08/21] Create testing.mdx --- serverless/storage/modelrepo/testing.mdx | 166 +++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 serverless/storage/modelrepo/testing.mdx diff --git a/serverless/storage/modelrepo/testing.mdx b/serverless/storage/modelrepo/testing.mdx new file mode 100644 index 00000000..eee95e97 --- /dev/null +++ b/serverless/storage/modelrepo/testing.mdx @@ -0,0 +1,166 @@ +--- +title: "Model Repo testing" +description: "Upload a model to Model Repo and deploy it to a Serverless endpoint." +--- + + +Model Repo is currently in alpha and is available on Mac and Linux only. Windows support is coming soon. + + +## Why use Model Repo + +Model Repo lets you upload your own models to private storage on Runpod and attach them directly to Serverless endpoints. Key benefits: + +- **Faster cold starts**: Models are pre-cached on the worker host rather than downloaded at runtime. +- **No HuggingFace dependency**: Your models are stored in Runpod's infrastructure, so endpoints don't require an outbound download on every cold start. +- **Private storage**: Models are stored in your account and are not accessible to other users. + +--- + +## Scripted testing + +A shell script is available that runs all of the steps below automatically using mock model files. Download `model_repo_testing.sh` from the internal Notion page and run it to validate end-to-end without manually executing each step. + +--- + +## Manual testing + +### Prerequisites + +- Your email is feature-flagged for Model Repo access. +- `jq` is installed for parsing JSON output. + +### Set environment variables + +Export the following before running any commands. **Make sure to set your actual API key — missing this is the most common source of auth errors later.** + +```bash +export RUNPOD_API_URL="https://rest.runpod.io/v1" +export RUNPOD_GRAPHQL_URL="https://api.runpod.io/graphql" +export RUNPOD_API_KEY="your-api-key" # replace with your actual API key + +export MODEL_NAME="$(whoami)-test-$(date +%s)" # unique name per test run — reusing the same name uploads a new version, not a new model +export MODEL_PATH="/path/to/model" # local path to the model files you want to upload +``` + + +`MODEL_NAME` must be unique for each test run. If you reuse the same name, the upload creates a new version of the existing model rather than a new model. + + +--- + +### Step 1: Install runpodctl + +**Option A: Install via Homebrew (recommended)** + +```bash +brew install runpod/runpodctl/runpodctl +``` + +**Option B: Build from source** + +```bash +brew install go # install Go, required to build runpodctl +git clone git@github.com:runpod/runpodctl.git +cd runpodctl +make # builds the binary to ./bin/runpodctl +``` + + +If you build from source, the binary is at `./bin/runpodctl`. Either run it with that path, or add `./bin` to your `PATH`. The steps below use `runpodctl` — adjust accordingly. + + +--- + +### Step 2: Upload the model + +```bash +runpodctl model add \ + --name "$MODEL_NAME" \ # the name to register the model under in your repo + --model-path "$MODEL_PATH" \ # local path to the model files + --create-upload # creates the upload session and transfers files +``` + +This outputs a JSON string listing all uploaded files. + +--- + +### Step 3: Wait for the model to be hashed + +After upload, the model must be hashed by an asynchronous background process. This typically completes in a few minutes but can take up to 10–15 minutes. + +Poll until the `hash` field is non-null: + +```bash +runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +``` + +While hashing is in progress, the command returns `null`: + +``` +% runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +null +``` + +Once hashing is complete, it returns the hash value: + +``` +% runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash' +71a311bdf0ca44119ed74dbef8cf573bc89b58cbc48a10fe508f756ebb1922dc +``` + +--- + +### Step 4: Get your user ID and model hash + +```bash +export USER_ID="$(runpodctl user | jq -r '.id')" # your Runpod user ID +export MODEL_HASH="$(runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash')" # the hash from step 3 +``` + +--- + +### Step 5: Deploy a Serverless endpoint with the model attached + +```bash +runpodctl serverless create \ + --name "my_worker" \ # name for the endpoint + --hub-id "cm8h09d9n000008jvh2rqdsmb" \ # Hub template to deploy + --gpu-id "AMPERE_24" \ # GPU type + --workers-max 3 \ # maximum number of active workers + --workers-min 1 \ # minimum number of workers kept warm + --model-reference "https://local/$USER_ID/$MODEL_NAME:$MODEL_HASH" \ # attaches your model to the endpoint + --env MODEL_NAME="/runpod/model-store/modelrepo-local/models/$USER_ID/$MODEL_NAME/$MODEL_HASH" \ # sets the model path on the worker + --min-cuda-version "13.0" # works around a bug in runpodctl +``` + + +`--model-reference` is only supported with `--hub-id` and GPU endpoints. It is repeatable if you need to attach multiple models to the same endpoint. The `--env MODEL_NAME` flag passes the full local path to the worker so your handler knows where to find the model files. + + +--- + +### Step 6: Verify the model is working + +Send a test request to confirm the endpoint is live and the model is accessible. Replace `ENDPOINT_ID` with the ID returned in the previous step: + +```bash +curl -s -X POST "https://api.runpod.ai/v2/${ENDPOINT_ID}/runsync" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "hello"}}' | jq +``` + +A successful response confirms the endpoint is running and the model is attached. If the request fails with an auth error, verify that `RUNPOD_API_KEY` is set correctly. + +If you prefer a graphical interface, you can also send requests to the worker from the web UI. + +--- + +### Step 7: Clean up + +Delete the endpoint after testing to stop accruing spend. Use the web UI or: + +```bash +runpodctl serverless delete +``` From b2e0472a00b85d7dc5c505107b1d0f528ad8410d Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Mon, 20 Jul 2026 19:59:35 -0400 Subject: [PATCH 09/21] Update docs.json --- docs.json | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs.json b/docs.json index d63a14ac..221fe3c6 100644 --- a/docs.json +++ b/docs.json @@ -126,6 +126,14 @@ "serverless/development/dual-mode-worker" ] }, + { + "group": "Storage", + "pages": [ + "serverless/storage/overview", + "serverless/storage/security", + "serverless/storage/testing" + ] + }, "serverless/modelrepotest", { "group": "Manage endpoints", @@ -1095,4 +1103,4 @@ "destination": "/serverless/troubleshooting" } ] -} \ No newline at end of file +} From 2757798014ea529f2c6b60b319c494699b30d8f3 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 06:47:48 -0400 Subject: [PATCH 10/21] Update docs.json --- docs.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs.json b/docs.json index 221fe3c6..6ab56ae3 100644 --- a/docs.json +++ b/docs.json @@ -129,9 +129,9 @@ { "group": "Storage", "pages": [ - "serverless/storage/overview", - "serverless/storage/security", - "serverless/storage/testing" + "serverless/storage/modelrepo/overview", + "serverless/storage/modelrepo/security", + "serverless/storage/modelrepo/testing" ] }, "serverless/modelrepotest", From f3e686bab794f7b6007d1edf905c785939f9ab8e Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 07:28:33 -0400 Subject: [PATCH 11/21] Rename security to security.mdx --- serverless/storage/modelrepo/{security => security.mdx} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename serverless/storage/modelrepo/{security => security.mdx} (100%) diff --git a/serverless/storage/modelrepo/security b/serverless/storage/modelrepo/security.mdx similarity index 100% rename from serverless/storage/modelrepo/security rename to serverless/storage/modelrepo/security.mdx From 029322ffcf9e1510dfba22819ad99ab50378b483 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 07:32:37 -0400 Subject: [PATCH 12/21] Update docs.json --- docs.json | 1 + 1 file changed, 1 insertion(+) diff --git a/docs.json b/docs.json index 6ab56ae3..3a616183 100644 --- a/docs.json +++ b/docs.json @@ -129,6 +129,7 @@ { "group": "Storage", "pages": [ + "serverless/storage/overview", "serverless/storage/modelrepo/overview", "serverless/storage/modelrepo/security", "serverless/storage/modelrepo/testing" From 5d17d4e70cc718b5daec96b3e5d80ba6a60050e5 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 09:07:08 -0400 Subject: [PATCH 13/21] Update overview.mdx --- serverless/storage/modelrepo/overview.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/serverless/storage/modelrepo/overview.mdx b/serverless/storage/modelrepo/overview.mdx index d21186c4..0ceeee18 100644 --- a/serverless/storage/modelrepo/overview.mdx +++ b/serverless/storage/modelrepo/overview.mdx @@ -26,10 +26,10 @@ You pass this URL as the `--model-reference` flag when creating or updating an e ## What you can upload - - - - +/* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */ +/* [ENGINEERING: Is there a per-file size limit?] */ +/* [ENGINEERING: Is there a total storage limit per account or per model?] */ +/* [ENGINEERING: How long does Runpod retain uploaded models? Is there a retention/expiry policy?] */ Model Repo accepts model files in any format. There is no restriction on file type — you can upload PyTorch checkpoints, GGUF files, safetensors, ONNX models, or any other format your worker needs. @@ -49,7 +49,7 @@ Pass this path to the worker as an environment variable using the `--env` flag w Your handler function reads `MODEL_NAME` to know where to load the model from. Model Repo only handles delivering the files to the host — your handler is fully responsible for loading the model and processing requests. - +/* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */ For queue-based endpoints, you write a handler function that receives requests, loads the model from `MODEL_NAME`, runs inference, and returns results. For load balancing endpoints (FastAPI, Flask, etc.), your server process loads the model from that path on startup. @@ -64,7 +64,7 @@ Both Model Repo and [network volumes](/storage/network-volumes) can make model f | **Best for** | Fixed, versioned model weights | Frequently changing files, shared state | | **Versioning** | Built-in, content-addressed by hash | Manual — you manage file paths yourself | | **Multi-worker sharing** | Not shared across workers | Shared across all workers on the same volume | -| **Storage billing** | | $0.07/GB/month | +| **Storage billing** | /* [ENGINEERING: how is Model Repo storage billed? $/GB/month?] */ | $0.07/GB/month | Use Model Repo when: - You have a fixed set of model weights you want to distribute reliably to workers. @@ -99,7 +99,7 @@ To migrate: 3. Update your endpoint to reference the model URL. 4. Update your worker code to load the model from the local path instead of from within the image. - +/* [ENGINEERING: Anything specific users need to do if their model has baked images (e.g., embedded image assets in weights)? Any known edge cases?] */ **If you cache models on a network volume:** From 60a453f145b5d684221e6d0be1fa7a8be9b7c662 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 09:08:07 -0400 Subject: [PATCH 14/21] Update security.mdx --- serverless/storage/modelrepo/security.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverless/storage/modelrepo/security.mdx b/serverless/storage/modelrepo/security.mdx index 61cecf6a..0f5c0977 100644 --- a/serverless/storage/modelrepo/security.mdx +++ b/serverless/storage/modelrepo/security.mdx @@ -21,7 +21,7 @@ Runpod does not access, analyze, or use your model files for any purpose. Models ## Compliance - +/* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] --> Runpod maintains industry-standard security certifications. For a full overview of Runpod's security posture, certifications, and policies, see [runpod.io/security](https://runpod.io/security). From 3b180d7386a8396ceb9e70090be6de5b2e4ab197 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 10:12:46 -0400 Subject: [PATCH 15/21] Update docs.json --- docs.json | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs.json b/docs.json index 3a616183..1136bf15 100644 --- a/docs.json +++ b/docs.json @@ -128,8 +128,7 @@ }, { "group": "Storage", - "pages": [ - "serverless/storage/overview", + "pages": [ "serverless/storage/modelrepo/overview", "serverless/storage/modelrepo/security", "serverless/storage/modelrepo/testing" From ab7a48fdd239b0b37bc3b853220147abf65693ff Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 10:21:29 -0400 Subject: [PATCH 16/21] Update overview.mdx --- serverless/storage/modelrepo/overview.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/serverless/storage/modelrepo/overview.mdx b/serverless/storage/modelrepo/overview.mdx index 0ceeee18..db6c7637 100644 --- a/serverless/storage/modelrepo/overview.mdx +++ b/serverless/storage/modelrepo/overview.mdx @@ -25,12 +25,12 @@ https://local/{user-id}/{model-name}:{hash} You pass this URL as the `--model-reference` flag when creating or updating an endpoint. Runpod uses it to know exactly which version of which model to cache on the host. ## What you can upload - + /* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */ /* [ENGINEERING: Is there a per-file size limit?] */ /* [ENGINEERING: Is there a total storage limit per account or per model?] */ /* [ENGINEERING: How long does Runpod retain uploaded models? Is there a retention/expiry policy?] */ - + Model Repo accepts model files in any format. There is no restriction on file type — you can upload PyTorch checkpoints, GGUF files, safetensors, ONNX models, or any other format your worker needs. ## How the model is available inside the worker @@ -49,7 +49,7 @@ Pass this path to the worker as an environment variable using the `--env` flag w Your handler function reads `MODEL_NAME` to know where to load the model from. Model Repo only handles delivering the files to the host — your handler is fully responsible for loading the model and processing requests. -/* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */ + /* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */ For queue-based endpoints, you write a handler function that receives requests, loads the model from `MODEL_NAME`, runs inference, and returns results. For load balancing endpoints (FastAPI, Flask, etc.), your server process loads the model from that path on startup. @@ -64,7 +64,7 @@ Both Model Repo and [network volumes](/storage/network-volumes) can make model f | **Best for** | Fixed, versioned model weights | Frequently changing files, shared state | | **Versioning** | Built-in, content-addressed by hash | Manual — you manage file paths yourself | | **Multi-worker sharing** | Not shared across workers | Shared across all workers on the same volume | -| **Storage billing** | /* [ENGINEERING: how is Model Repo storage billed? $/GB/month?] */ | $0.07/GB/month | +| **Storage billing** | /* [ENGINEERING: how is Model Repo storage billed? $/GB/month?] */ | $0.07/GB/month | Use Model Repo when: - You have a fixed set of model weights you want to distribute reliably to workers. @@ -99,7 +99,7 @@ To migrate: 3. Update your endpoint to reference the model URL. 4. Update your worker code to load the model from the local path instead of from within the image. -/* [ENGINEERING: Anything specific users need to do if their model has baked images (e.g., embedded image assets in weights)? Any known edge cases?] */ + /* [ENGINEERING: Anything specific users need to do if their model has baked images (e.g., embedded image assets in weights)? Any known edge cases?] */ **If you cache models on a network volume:** From 77f1859b2af54c2ebb06ed0c4c40f6794a2ab351 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 10:22:07 -0400 Subject: [PATCH 17/21] Update security.mdx --- serverless/storage/modelrepo/security.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverless/storage/modelrepo/security.mdx b/serverless/storage/modelrepo/security.mdx index 0f5c0977..8b97d993 100644 --- a/serverless/storage/modelrepo/security.mdx +++ b/serverless/storage/modelrepo/security.mdx @@ -21,7 +21,7 @@ Runpod does not access, analyze, or use your model files for any purpose. Models ## Compliance -/* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] --> + /* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] --> Runpod maintains industry-standard security certifications. For a full overview of Runpod's security posture, certifications, and policies, see [runpod.io/security](https://runpod.io/security). From 91c9f6db2fbb211ae8b4caef7ca7065000cdeeff Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 14:54:03 -0400 Subject: [PATCH 18/21] Update overview.mdx From 83af1e8ef347591fa77315656904057ed9cc71d5 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 15:15:34 -0400 Subject: [PATCH 19/21] Update overview.mdx --- serverless/storage/modelrepo/overview.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/serverless/storage/modelrepo/overview.mdx b/serverless/storage/modelrepo/overview.mdx index db6c7637..fbb1bfaf 100644 --- a/serverless/storage/modelrepo/overview.mdx +++ b/serverless/storage/modelrepo/overview.mdx @@ -25,12 +25,10 @@ https://local/{user-id}/{model-name}:{hash} You pass this URL as the `--model-reference` flag when creating or updating an endpoint. Runpod uses it to know exactly which version of which model to cache on the host. ## What you can upload - /* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */ /* [ENGINEERING: Is there a per-file size limit?] */ /* [ENGINEERING: Is there a total storage limit per account or per model?] */ /* [ENGINEERING: How long does Runpod retain uploaded models? Is there a retention/expiry policy?] */ - Model Repo accepts model files in any format. There is no restriction on file type — you can upload PyTorch checkpoints, GGUF files, safetensors, ONNX models, or any other format your worker needs. ## How the model is available inside the worker @@ -49,7 +47,7 @@ Pass this path to the worker as an environment variable using the `--env` flag w Your handler function reads `MODEL_NAME` to know where to load the model from. Model Repo only handles delivering the files to the host — your handler is fully responsible for loading the model and processing requests. - /* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */ +/* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */ For queue-based endpoints, you write a handler function that receives requests, loads the model from `MODEL_NAME`, runs inference, and returns results. For load balancing endpoints (FastAPI, Flask, etc.), your server process loads the model from that path on startup. @@ -64,7 +62,7 @@ Both Model Repo and [network volumes](/storage/network-volumes) can make model f | **Best for** | Fixed, versioned model weights | Frequently changing files, shared state | | **Versioning** | Built-in, content-addressed by hash | Manual — you manage file paths yourself | | **Multi-worker sharing** | Not shared across workers | Shared across all workers on the same volume | -| **Storage billing** | /* [ENGINEERING: how is Model Repo storage billed? $/GB/month?] */ | $0.07/GB/month | +| **Storage billing** | /* [ENGINEERING: how is Model Repo storage billed? $/GB/month?] */ | $0.07/GB/month | Use Model Repo when: - You have a fixed set of model weights you want to distribute reliably to workers. @@ -99,7 +97,7 @@ To migrate: 3. Update your endpoint to reference the model URL. 4. Update your worker code to load the model from the local path instead of from within the image. - /* [ENGINEERING: Anything specific users need to do if their model has baked images (e.g., embedded image assets in weights)? Any known edge cases?] */ +/* [ENGINEERING: Anything specific users need to do if their model has baked images (e.g., embedded image assets in weights)? Any known edge cases?] */ **If you cache models on a network volume:** From 397da03212d3b106d61542ea9924b66b8cf3cd03 Mon Sep 17 00:00:00 2001 From: lgunreddi Date: Tue, 21 Jul 2026 15:16:06 -0400 Subject: [PATCH 20/21] Update security.mdx --- serverless/storage/modelrepo/security.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverless/storage/modelrepo/security.mdx b/serverless/storage/modelrepo/security.mdx index 8b97d993..2f7d0ef8 100644 --- a/serverless/storage/modelrepo/security.mdx +++ b/serverless/storage/modelrepo/security.mdx @@ -21,7 +21,7 @@ Runpod does not access, analyze, or use your model files for any purpose. Models ## Compliance - /* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] --> +/* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] --> Runpod maintains industry-standard security certifications. For a full overview of Runpod's security posture, certifications, and policies, see [runpod.io/security](https://runpod.io/security). From e72b838e433418fc3932dacba3205d40f32bcf02 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" Date: Thu, 23 Jul 2026 19:29:15 +0000 Subject: [PATCH 21/21] Apply review feedback to Model Repo overview Addresses @brosenpod's review on PR #696: - Use provider-neutral storage wording (drop specific R2 mention) - Document all file types supported with no type checking - Add 5TB per-file maximum size - Note total per-account/per-model storage limit is still being finalized - Document indefinite retention of uploaded model files - Remove resolved internal engineering placeholders --- serverless/storage/modelrepo/overview.mdx | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/serverless/storage/modelrepo/overview.mdx b/serverless/storage/modelrepo/overview.mdx index fbb1bfaf..b09828c6 100644 --- a/serverless/storage/modelrepo/overview.mdx +++ b/serverless/storage/modelrepo/overview.mdx @@ -14,7 +14,7 @@ Model Repo is a private model storage service built into Runpod. Upload your mod When you create a Serverless endpoint with a model attached, Runpod fetches the model files from storage and caches them on the host machine before the worker container starts. Once the worker boots, the files are already on disk. This eliminates the download step that normally adds seconds or minutes to a cold start. -Model files are stored in your private Runpod account on Cloudflare R2. Each uploaded version is content-addressed: after the upload, Runpod computes a hash of the files, and you use that hash to pin a specific version when configuring an endpoint. +Model files are stored in the Runpod ecosystem on private, secure storage. Each uploaded version is content-addressed: after the upload, Runpod computes a hash of the files, and you use that hash to pin a specific version when configuring an endpoint. The model reference URL format is: @@ -25,11 +25,12 @@ https://local/{user-id}/{model-name}:{hash} You pass this URL as the `--model-reference` flag when creating or updating an endpoint. Runpod uses it to know exactly which version of which model to cache on the host. ## What you can upload -/* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */ -/* [ENGINEERING: Is there a per-file size limit?] */ -/* [ENGINEERING: Is there a total storage limit per account or per model?] */ -/* [ENGINEERING: How long does Runpod retain uploaded models? Is there a retention/expiry policy?] */ -Model Repo accepts model files in any format. There is no restriction on file type — you can upload PyTorch checkpoints, GGUF files, safetensors, ONNX models, or any other format your worker needs. + +Model Repo accepts model files in any format. There is no restriction on file type — you can upload PyTorch checkpoints, GGUF files, safetensors, ONNX models, or any other format your worker needs. Runpod performs no type checking on uploads. + +The maximum size for an individual file is 5TB, and uploaded model files are retained indefinitely. + +The total storage limit per account and per model is still being finalized. ## How the model is available inside the worker