feat(client-proxy): read the orchestrator routing record by default

GitOrigin-RevId: c1cc40eda00f98dfa9da7131880e5faf35486d3d

Author: sitole

docs/ARCHITECTURE.md

@@ -302,9 +302,9 @@ looks the sandbox up in the Redis routing record to find the owning node, and re
 that node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a
 different downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls
 the API's `ResumeSandbox` gRPC and retries — paused sandboxes wake transparently on traffic.
-By default the record is the API-owned `sandbox:catalog:{id}`; the `orchestrator-routing-prioritized`
-flag switches the read to the orchestrator-owned `sandbox:routing:{id}` (see "Sandbox routing
-records" below).
+By default the record is the orchestrator-owned `sandbox:routing:{id}`; turning the
+`orchestrator-routing-prioritized` flag off switches the read back to the API-owned
+`sandbox:catalog:{id}` (see "Sandbox routing records" below).
 
 ### Dashboard API (`packages/dashboard-api`)
 
@@ -461,12 +461,12 @@ today. Both have the same JSON shape (`sandbox_catalog.SandboxInfo` in
 
 | Record | Key | Writer | Written | Deleted |
 |---|---|---|---|---|
-| API-owned (default) | `sandbox:catalog:{sandboxID}` | API (cloud) or the cluster edge from gRPC metadata (BYOC) | after `Create` returns | before `Pause`/`Kill` is sent to the node |
-| Orchestrator-owned (v1, flag-gated) | `sandbox:routing:{sandboxID}` | orchestrator, `packages/orchestrator/pkg/routing` | on `MarkRunning` (sandbox enters the live map, envd is ready) | on `MarkStopping` (kill, pause, checkpoint, crash) |
+| API-owned (fallback) | `sandbox:catalog:{sandboxID}` | API (cloud) or the cluster edge from gRPC metadata (BYOC) | after `Create` returns | before `Pause`/`Kill` is sent to the node |
+| Orchestrator-owned (default) | `sandbox:routing:{sandboxID}` | orchestrator, `packages/orchestrator/pkg/routing` | on `MarkRunning` (sandbox enters the live map, envd is ready) | on `MarkStopping` (kill, pause, checkpoint, crash) |
 
-**The API-owned record is still the source of truth.** client-proxy reads `sandbox:catalog:{id}`
-unless the `orchestrator-routing-prioritized` flag is on. The orchestrator-owned record is a v1
-test path. It runs next to the API path and does not replace it yet.
+**client-proxy reads the orchestrator-owned record by default.** It reads `sandbox:catalog:{id}`
+only when the `orchestrator-routing-prioritized` flag is off. The API still writes its record next
+to the orchestrator one, so the flag can be turned off without a gap.
 
 Two feature flags in `packages/shared/pkg/featureflags` control the new path:
 
@@ -475,14 +475,16 @@ Two feature flags in `packages/shared/pkg/featureflags` control the new path:
   (`orchestrator.routing.publish.total{result=error}`); the sandbox keeps running. Build sandboxes
   are skipped. The delete is guarded by `execution_id` in a Lua script, so a stale lifecycle never
   removes the record of a newer execution.
-- `orchestrator-routing-prioritized` (client-proxy, **default off**): resolve the node from `sandbox:routing:{id}`
+- `orchestrator-routing-prioritized` (client-proxy, **default on**): resolve the node from `sandbox:routing:{id}`
   instead of `sandbox:catalog:{id}`. There is no fallback to the API-owned record on a miss. A miss
   goes to the auto-resume path (`ResumeSandbox` gRPC to the API), same as today.
 
-Rollout order: `orchestrator-routing-publish` is on by default, so every live sandbox has a record
-one maximum sandbox length after the orchestrator deploy. Then turn on
-`orchestrator-routing-prioritized`. To roll back, turn off `orchestrator-routing-prioritized`; the
-API path is untouched. Turn off `orchestrator-routing-publish` only to stop the extra Redis write.
+Deploy order: every orchestrator on the cluster must run a build with `orchestrator-routing-publish`
+on for one maximum sandbox length before client-proxy is upgraded to a build with
+`orchestrator-routing-prioritized` on by default. Otherwise requests for sandboxes started before
+the orchestrator upgrade miss the record and go to the resume path. To roll back, turn off
+`orchestrator-routing-prioritized`; the API path is untouched. Turn off
+`orchestrator-routing-publish` only to stop the extra Redis write.
 
 The TTL of both records is `sandbox_max_length_in_hours` from the write time. The record is
 deleted earlier in every normal stop path.

packages/shared/pkg/featureflags/flags.go

@@ -478,9 +478,12 @@ var (
 	OrchestratorRoutingPublishFlag = NewBoolFlag("orchestrator-routing-publish", true)
 	// OrchestratorRoutingPrioritizedFlag makes client-proxy resolve the node
 	// from the orchestrator-owned sandbox:routing:{id} record instead of the
-	// API-owned sandbox:catalog:{id} record. Turn on only after
-	// OrchestratorRoutingPublishFlag has been on for one max sandbox length.
-	OrchestratorRoutingPrioritizedFlag = NewBoolFlag("orchestrator-routing-prioritized", false)
+	// API-owned sandbox:catalog:{id} record. On by default: every orchestrator
+	// must run a build with OrchestratorRoutingPublishFlag on for at least one
+	// max sandbox length before client-proxy picks this up, or requests for
+	// older sandboxes miss and fall to the resume path. Turn off in
+	// LaunchDarkly to read the API-owned record again.
+	OrchestratorRoutingPrioritizedFlag = NewBoolFlag("orchestrator-routing-prioritized", true)
 	MaxCacheWriterConcurrencyFlag      = NewIntFlag("max-cache-writer-concurrency", 10)
 
 	// BuildCacheMaxUsagePercentage the maximum percentage of the cache disk storage