[twk3]

Owner: @twk3
Linear: ENG-1850

The 1.0 upgrade guide told operators the scheduler's startup took "under a minute with 100,000 runs" and was "not downtime". It said nothing about the MongoDB index builds, which on a large database take hours and load the secondaries the API reads from (ENG-1849). This rewrites the guide around those builds, and adds the SAML and sign-in changes the upgrade needs.

Merge after the 1.0.1 release. The guide tells readers to upgrade to 1.0.1 (images 2026-10-14-002, #66), and its helm upgrade command uses --version 1.0.1.

Changes to docs/eks/upgrade-1.0.md

• MongoDB Index Builds (new):
  • What the scheduler does: on instances it drops runId_1_createdAt_1__id_1 and instanceId_1_projectId_1_orgId_1__id_-1 before building runId_1_createdAt_1__id_-1 and createdAt_1, and every build loads the secondaries the API reads from.
  • Estimate the Upgrade Time: a mongosh snippet that prints the data the builds read (2 × instances + runs + testRecord, from stats().size), an index-build range, and minutes of migrations per 100,000 runs, with where each rate comes from.
  • Build the Indexes Before You Upgrade: the createIndexes commands for every new index on instances, runs (four), testRecord (two) and instanceStats, run with 0.8.x still serving. Then helm upgrade straight after.
  • A warning, found while testing: a restart of the 0.8.x scheduler drops the pre-built instances indexes and rebuilds its own. The guide says to check getIndexes() before upgrading.
• Database Migrations:
  • The index builds come first, and the scheduler reports ready during both.
  • "This is not downtime" is replaced with what slows down and why.
  • The table gains 20261009 and 20261021.
• SAML SSO: 1.7 refuses an unsigned Assertion (SAML_ASSERTION_SIGNATURE_REQUIRED, wantAssertionsSigned: true in code). In Okta, set Assertion Signature to Signed.
• Sign-In After the Upgrade (new):
  • 1.0.0 locked out existing users, and 1.0.1 signs them in.
  • Its migration drops account_issuer_accountId_uidx, so the dropIndex the issue mentions isn't needed by hand.
  • An issuer added by hand is harmless.
• Rollback: 0.8.x's scheduler rebuilds the instances indexes, with the same unindexed window.
• CHANGELOG.md: the 1.0.0 upgrade notes get the same corrections.

How the numbers were measured

An upgrade rehearsal on Docker Compose: 2026-07-26-006 upgraded to the 2026-10-14-002 scheduler, with Mongo 8.2.3 on a laptop.

Dataset, cloned from real dev documents and shaped as 006 writes them:

Collection	Documents	stats().size
runs	100,000	1.74 GB
instances	500,000	6.59 GB
testRecord	600,000	5.85 GB

Index builds (from mongod's log):

Build	Scheduler builds them	mongosh pre-build
instances runId_1_createdAt_1__id_-1	2.0–5.5 s	4.4 s
instances createdAt_1	1.8–2.9 s	1.9 s
runs, all four	0.9–1.75 s	1.3 s
testRecord, both	3.0–6.1 s	4.9 s

That is about 1 s per GB read, the low end. Limiting mongod to a 0.25 GB cache and 1.5 GB of memory changed nothing, because the laptop's disk reads the compressed data in seconds. The high end, about 0.6 minutes per GB, comes from a large production database building the same indexes under load, and the rate matches that database's total build time.

Migrations:

• 20260918 took 28–29 s per 100,000 runs (ENG-1741: 25–27 s).
• 070 took 22 s (ENG-1741: 9–12 s, with smaller runs).
• The ClickHouse tables took about 10 s.

The pre-build, as published: I extracted the estimate snippet and the commands from this guide's markdown and ran them on a fresh 006-state database. All the builds finished in 11 s, and the upgrade's index sync then only dropped the two old instances indexes.

Every new index the upgrade builds:

• On the large collections: the ones listed above.
• On small collections, left to the scheduler: usage_totals_by_org_day, timeoutAfter_1, two on apiKeys, two on notifications, and auth_user_cognito_user_id_unique.

The testRecord indexes weren't in the issue. That collection only shrinks when runs are deleted, and on a dev install it was larger than instances.

Not covered here

• ENG-1852: index sync drops before it builds.
• ENG-1877: index sync never drops.
• ENG-1880: ship the pre-build as a Job.

Once those land, the warning about the old scheduler restarting and the hand-run commands can go.

🤖 Generated with Claude Code

[baz-reviewer]

Baz Summary

Rewrite the 1.0 upgrade guide and changelog to size and pre-build MongoDB indexes before migration, minimizing performance impact during upgrades. Document required SAML assertion signing and the 1.0.1 sign-in/account migration fixes for existing users.

Topics

Topic	Details
MongoDB Upgrade Flow	Document how to estimate MongoDB index-build duration, pre-build indexes before upgrading, sequence scheduler migrations, monitor readiness, and plan maintenance or rollback windows. Modified files (2) CHANGELOG.md docs/eks/upgrade-1.0.md Latest Contributors(1) User Commit Date dj@currents.dev docs: count the instan... October 23, 2026
SSO and Sign-In	Explain required signed SAML assertions and direct operators to 1.0.1 for restored legacy-account sign-in and automatic removal of account_issuer_accountId_uidx. Modified files (2) CHANGELOG.md docs/eks/upgrade-1.0.md Latest Contributors(1) User Commit Date dj@currents.dev docs: count the instan... October 23, 2026

---

  _{Activate to get a short verdict whether this PR is good to go or not}

  _{Activate Skill Maintainer to keep your skills up to date}

  _{This PR would have been improved with Baz Planner - Try it now}