twk3 · Oct 24, 2026 · Oct 22, 2026 · Oct 23, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,7 +16,14 @@ Images `2026-10-14-001`, the first Currents on-prem feature release since `2026-
 
 ### Upgrade Notes
 - **Create the app content bucket first** and set `currents.appContentStorage.bucket`. Without it, context sharing and logo editing are off and session file uploads answer `503`. Its lifecycle rules must be scoped to the share prefixes, because the bucket also holds logos and session files. See [App content bucket](docs/eks/dependencies.md#app-content-bucket).
-- **The scheduler runs database migrations when it starts.** With 100,000 runs they took under a minute in testing; they scale with the number of runs. This is not downtime. The scheduler pod reports ready before they finish, so wait for `Startup tasks complete` in its log, and restart it if a migration logs an error (see [Database Migrations and Startup Tasks](docs/eks/upgrading.md#database-migrations-and-startup-tasks)).
+- **The scheduler builds new MongoDB indexes, then runs migrations, when it starts.**
+  - On `instances` it first drops the indexes `0.8.x` reads a run's spec files with, so those reads have no index until the build finishes.
+  - The builds load every replica set member, and on a large database they take hours.
+  - Estimate the time for your install, and on a large database build the indexes before upgrading, in a maintenance window. See [MongoDB Index Builds](docs/eks/upgrade-1.0.md#mongodb-index-builds).
+  - The migrations take about a minute per 100,000 runs.
+  - The scheduler pod reports ready before all this finishes, so wait for `Startup tasks complete` in its log, and restart it if a migration logs an error (see [Database Migrations and Startup Tasks](docs/eks/upgrading.md#database-migrations-and-startup-tasks)).
+- **SAML SSO requires a signed assertion.** A response whose `Assertion` isn't signed is refused with `SAML_ASSERTION_SIGNATURE_REQUIRED`, even when the response itself is signed. In Okta, set Assertion Signature to Signed. See [SAML SSO](docs/eks/upgrade-1.0.md#saml-sso).
+- **Upgrade to `1.0.1` or later.** On `1.0.0`, users who signed in before the upgrade can't sign in. See [Sign-In After the Upgrade](docs/eks/upgrade-1.0.md#sign-in-after-the-upgrade).
 - **Back up MongoDB and ClickHouse before upgrading.** `helm rollback` returns the chart and images to `0.8.x` but doesn't undo the migrations.
 
 ### Added

diff --git a/docs/eks/upgrade-1.0.md b/docs/eks/upgrade-1.0.md
@@ -1,14 +1,22 @@
 # Upgrading to Chart 1.0
 
-This guide covers upgrading from chart `0.8.x` (images `2026-07-26-00x`) to `1.0.0` (images `2026-10-14-001`). Read it before you upgrade: one change breaks SAML SSO sign-in until you update your identity provider, and the app content bucket has to exist first.
+This guide covers upgrading from chart `0.8.x` (images `2026-07-26-00x`) to `1.0.x` (images `2026-10-14-00x`). Read it before you upgrade:
+
+- The upgrade builds new MongoDB indexes. On a large database, build them yourself first, in a maintenance window.
+- One change breaks SAML SSO sign-in until you update your identity provider.
+- The app content bucket has to exist first.
+
+Upgrade to `1.0.1` or later. On `1.0.0` (images `2026-10-14-001`), users who signed in before the upgrade can't sign in again. See [Sign-In After the Upgrade](#sign-in-after-the-upgrade).
 
 For the general upgrade flow, see the [EKS Upgrade Guide](./upgrading.md). The docs for the chart you are upgrading from are at [chart 0.8.x docs](https://currents-dev.github.io/helm-charts/0.8/docs/).
 
 ## What Changes
 
 | Change | Action |
 |--------|--------|
+| [New MongoDB indexes](#mongodb-index-builds) | Size them, and on a large database build them before you upgrade |
 | [SAML SSO ACS URL moves](#saml-sso) | Required if you use SSO: update your IdP when you upgrade |
+| [SAML assertions must be signed](#saml-sso) | Required if you use SSO: check that your IdP signs the assertion |
 | [SAML sign-ins started from the IdP are refused](#saml-sso) | Set `currents.sso.saml.allowIdpInitiated: true` if your users sign in from an IdP tile |
 | [App content bucket](#app-content-bucket) | Create a second bucket and set `currents.appContentStorage.bucket` |
 | [Remote MCP and OAuth](#remote-mcp-and-oauth) | None: on by default. Set the values to `false` to keep them off |
@@ -19,9 +27,10 @@ For the general upgrade flow, see the [EKS Upgrade Guide](./upgrading.md). The d
 ## Before You Upgrade
 
 1. Back up MongoDB and ClickHouse. The upgrade runs database migrations, and `helm rollback` doesn't undo them.
-2. If you use SAML SSO, schedule the IdP change for the upgrade window. See [SAML SSO](#saml-sso).
-3. Create the app content bucket. See [App content bucket](#app-content-bucket).
-4. Update your values file:
+2. Estimate how long the index builds and migrations take, and plan the upgrade window from that. See [Estimate the Upgrade Time](#estimate-the-upgrade-time).
+3. If you use SAML SSO, check that your IdP signs the assertion, and schedule the ACS URL change for the upgrade window. See [SAML SSO](#saml-sso).
+4. Create the app content bucket. See [App content bucket](#app-content-bucket).
+5. Update your values file:
 
    ```yaml
    currents:
@@ -40,14 +49,104 @@ For the general upgrade flow, see the [EKS Upgrade Guide](./upgrading.md). The d
 
 ## Run the Upgrade
 
-```bash
-helm upgrade --install currents currents \
-  --repo https://currents-dev.github.io/helm-charts/ \
-  --version 1.0.0 \
-  -f currents-helm-config.yaml
+In the upgrade window, with `0.8.x` still running:
+
+1. Build the new MongoDB indexes, unless the estimate says they take only a few minutes. See [Build the Indexes Before You Upgrade](#build-the-indexes-before-you-upgrade).
+2. Upgrade:
+
+   ```bash
+   helm upgrade --install currents currents \
+     --repo https://currents-dev.github.io/helm-charts/ \
+     --version 1.0.1 \
+     -f currents-helm-config.yaml
+   ```
+
+3. Follow the scheduler until its startup tasks finish. See [Database Migrations](#database-migrations).
+
+## MongoDB Index Builds
+
+`1.0` adds indexes to collections that grow with your history: `instances`, `runs` and `testRecord`. The scheduler builds any that are missing when it starts, before it runs the migrations:
+
+- **On `instances`, it first drops the two indexes `0.8.x` uses to read a run's spec files** (`runId_1_createdAt_1__id_1` and `instanceId_1_projectId_1_orgId_1__id_-1`), then builds `runId_1_createdAt_1__id_-1` and `createdAt_1`. Until the first build finishes, reading a run's spec files has no index.
+- **Each build reads the whole collection on every replica set member**, including the secondaries the API reads from.
+
+On a large database the builds can take hours and keep MongoDB's CPU busy, and sign-in can fail while they run. Plan for them, as below.
+
+### Estimate the Upgrade Time
+
+Run this in `mongosh`, connected to the Currents database:
+
+```js
+const gb = (c) => db.getCollection(c).stats().size / 1e9;
+const runs = db.runs.estimatedDocumentCount();
+const indexGB = 2 * gb('instances') + gb('runs') + gb('testRecord') + gb('instanceStats');
+print(`runs: ${runs}`);
+print(`data the index builds read: ${indexGB.toFixed(1)} GB`);
+print(`index builds: ${Math.ceil(indexGB / 60)} to ${Math.ceil(indexGB * 0.6)} minutes`);
+print(`migrations: about ${Math.ceil(runs / 100000)} minutes`);
 ```
 
-Then follow the scheduler until its migrations finish. See [Database migrations](#database-migrations).
+How the numbers are worked out:
+
+- **Index builds** take time in proportion to the data they read. Each build reads its whole collection, and `instances` is read twice. The range is:
+  - **Low end, about 1 second per GB:** measured on a test install with 100,000 runs, 500,000 `instances` (6.6 GB) and 600,000 `testRecord` documents (5.9 GB), on a fast local disk with nothing else running. All the builds took 13 seconds.
+  - **High end, about 0.6 minutes per GB:** measured on a large production database, with the API reading from the secondaries while the indexes built.
+
+  Use the high end to plan unless your MongoDB has fast disks, enough memory to hold the data, and no other load during the window.
+- **Migrations** took about a minute per 100,000 runs in testing: about 30 seconds for the run status backfill, 10 to 25 seconds for `070`, and 10 seconds to create the ClickHouse tables. See [Database Migrations](#database-migrations).
+
+If the index builds come to more than a few minutes, build the indexes yourself before you upgrade.
+
+### Build the Indexes Before You Upgrade
+
+With the indexes already built, the upgrade only drops the two old `instances` indexes, and run reads keep an index throughout. `0.8.x` runs normally while they build: they are extra indexes it doesn't use.
+
+1. Start at the beginning of the upgrade window, and allow the time the [estimate](#estimate-the-upgrade-time) gives. Every replica set member, secondaries included, carries the load of the builds while they run.
+2. Connect `mongosh` to the Currents database as a user that can create indexes. The user in `currents.mongoConnection` can.
+3. Run:
+
+   ```js
+   db.runCommand({
+     createIndexes: 'instances',
+     indexes: [{ key: { runId: 1, createdAt: 1, _id: -1 }, name: 'runId_1_createdAt_1__id_-1' }],
+   });
+   db.runCommand({
+     createIndexes: 'instances',
+     indexes: [{ key: { createdAt: 1 }, name: 'createdAt_1' }],
+   });
+   const prRuns = { 'meta.pr.id': { $type: 'string' }, deletedAt: null };
+   db.runCommand({
+     createIndexes: 'runs',
+     indexes: [
+       { key: { orgId: 1, projectId: 1, status: 1, _id: -1 }, name: 'runs_status_recent_by_project' },
+       { key: { orgId: 1, projectId: 1, _id: -1, 'meta.pr.id': 1 }, name: 'runs_pr_list_recent_by_project', partialFilterExpression: prRuns },
+       {
+         key: { orgId: 1, projectId: 1, _id: -1, 'meta.pr.id': 1, status: 1, completionState: 1, 'meta.pr.title': 1, 'meta.pr.source': 1 },
+         name: 'runs_pr_list_search_recent_by_project',
+         partialFilterExpression: prRuns,
+       },
+       { key: { orgId: 1, projectId: 1, 'meta.pr.source': 1, _id: -1 }, name: 'runs_pr_source_recent_by_project', partialFilterExpression: prRuns },
+     ],
+   });
+   db.runCommand({
+     createIndexes: 'testRecord',
+     indexes: [
+       { key: { runId: 1 }, name: 'runId_1' },
+       { key: { recordId: 1, _t: 1 }, name: 'recordId_1__t_1' },
+     ],
+   });
+   db.runCommand({
+     createIndexes: 'instanceStats',
+     indexes: [{ key: { 'props.projectId': 1, 'props.spec': 1, date: -1 }, name: 'props.projectId_1_props.spec_1_date_-1' }],
+   });
+   ```
+
+   Each command returns when its build is done, with `ok: 1`. To follow a build in progress from a second session, run `db.currentOp({ 'command.createIndexes': { $exists: true } })`.
+4. Run `helm upgrade` straight after the last build.
+
+**Don't let the `0.8.x` scheduler restart between the builds and `helm upgrade`.** When it starts, it drops the two `instances` indexes above and rebuilds its own, so run reads lose their index again. Before upgrading, check that `db.instances.getIndexes()` lists `runId_1_createdAt_1__id_-1` and `createdAt_1`; if it doesn't, run the two `instances` commands again.
+
+The scheduler builds the remaining new indexes itself, on `usage`, `runTimeouts`, `apiKeys`, `notifications` and `user`. Those collections stay small, and their builds take seconds.
 
 ## SAML SSO
 
@@ -69,6 +168,21 @@ A sign-in started from the IdP's app dashboard (an Okta tile, the Entra My Apps
 
 Currents also serves its SP metadata at `https://<appHost>/api/auth/sso/saml2/sp/metadata?providerId=<providerId>`, for IdPs that import it.
 
+**The assertion must be signed, not just the response.** `1.0` refuses a SAML response whose `Assertion` element carries no signature, with `SAML_ASSERTION_SIGNATURE_REQUIRED`, even when the response around it is signed. There is no setting to turn this off. Check your IdP before the upgrade: in Okta, set **Assertion Signature** to **Signed**, under the app's Sign On → Advanced Settings.
+
+## Sign-In After the Upgrade
+
+Chart `1.0.0` (images `2026-10-14-001`) looks accounts up by a field that accounts created on `0.8.x` don't have. After upgrading to it, every user who had signed in before could not sign in again:
+
+- SAML users returned to the sign-in page with `account_not_linked`.
+- The root admin's password sign-in answered `INVALID_EMAIL_OR_PASSWORD`.
+
+`1.0.1` (images `2026-10-14-002`) finds those accounts again, with no change to the database:
+
+- If you are on `1.0.0`, upgrade to `1.0.1`.
+- When the scheduler starts, migration `20261021-drop-account-issuer-index` drops the `account_issuer_accountId_uidx` index that `1.0.0` created. You don't need to drop it by hand.
+- If you added an `issuer` field to `account` documents by hand to work around this, leave it: nothing reads it.
+
 ## App Content Bucket
 
 Context shares, organization logos and session files go to a second bucket. Without `currents.appContentStorage.bucket`, context sharing and logo editing are off and session file uploads answer `503`.
@@ -102,20 +216,26 @@ curl -s https://<appHost>/.well-known/oauth-authorization-server
 
 ## Database Migrations
 
-The scheduler runs this release's migrations when it starts:
+When the scheduler starts, it first builds any missing indexes (it logs `migrations: syncing indexes`; see [MongoDB Index Builds](#mongodb-index-builds)), then runs this release's migrations:
 
 | Migration | What it does |
 |-----------|--------------|
 | `20260918-backfill-run-status` | Stores each run's status on its MongoDB document |
 | `20260928-drop-non-unique-plans-price-id-index` | Drops a duplicate MongoDB index |
+| `20261009-backfill-github-pr-comment-mode` | Stores the pull request comment mode on each GitHub integration |
+| `20261021-drop-account-issuer-index` | Drops the index `1.0.0` created on `account` (see [Sign-In After the Upgrade](#sign-in-after-the-upgrade)) |
 | `050`, `060` | Add columns to `run_metric_v2` in ClickHouse |
 | `070-backfill-run-metric-search-columns` | Re-inserts every run into `run_metric_v2` to fill those columns |
 
-With 100,000 runs, the scheduler's startup took under a minute in testing: about 25 seconds for the run status backfill and 10 seconds for `070`. They scale with the number of runs, so expect several minutes for a million.
+The migrations take about a minute per 100,000 runs: the [estimate](#estimate-the-upgrade-time) prints the figure for your install.
+
+The dashboard, API and run recording keep running while the scheduler builds indexes and migrates, but more slowly:
 
-This is not downtime: the dashboard, API and run recording keep working while the scheduler migrates. Until the migrations finish, filtering runs by status can leave out runs recorded before the upgrade. The change-streams pod also copies every updated run to ClickHouse, during the migrations and for a few minutes after, so new results can take longer to process.
+- The index builds and the run status backfill load MongoDB, including the secondaries the API reads from. On a large database, upgrade in a maintenance window and build the indexes first.
+- Until the migrations finish, filtering runs by status can leave out runs recorded before the upgrade.
+- The change-streams pod copies every updated run to ClickHouse, during the migrations and for a few minutes after, so new results can take longer to process.
 
-The scheduler pod reports ready while the migrations run, so a finished rollout doesn't mean they are done. Follow the scheduler's log:
+The scheduler pod reports ready while it builds indexes and runs migrations, so a finished rollout doesn't mean they are done. Follow the scheduler's log:
 
 ```bash
 kubectl logs -n currents deploy/currents-scheduler -f
@@ -134,3 +254,5 @@ Currents does not delete artifacts. To limit how long they are kept, add a lifec
 ## Rollback
 
 `helm rollback` returns the templates and images to `0.8.x`, but not the database: the migrations stay applied. To return the data to its state before the upgrade, restore the backup you took. Point your IdP back at the old ACS URL as well.
+
+When the `0.8.x` scheduler starts, it drops the two `instances` indexes `1.0` builds and rebuilds its own. That takes as long as the `instances` builds in the [estimate](#estimate-the-upgrade-time), and reading a run's spec files has no index until the first build finishes, so roll back in a maintenance window too.