[iris] Complete the controller-backend phase boundary - #8762
Merged
Conversation
Return neutral worker-fleet or direct-task observations from TaskBackend.reconcile. The controller now reloads current transition state, fences direct-provider reports by exact Attempt UID, applies task policy, folds worker liveness, commits effects, and requests teardown after commit. This removes controller DB transition readers and state-machine policy from the Kubernetes and worker backend implementations. A late Kubernetes observation from a deleted and recreated job can no longer update the replacement Attempt.
Make DbTransitionReader the only production source for reconciliation snapshots. BackendWorkerStore now contains worker lifecycle reads only, which prevents transition policy from drifting back into backend implementations.
rjpower
added
agent-generated
rjpower
commented
Aug 28, 2026
rjpower
left a comment
rjpower
left a comment
Collaborator
Author
There was a problem hiding this comment.
i don't understand this "folding", where did that come from, and why are you leaking details about the backend type (isinstance) to the reconcile coordination layer?
Normalize worker RPC and Kubernetes results into one exact task-observation shape with orthogonal worker-health events. Apply every observation through the same controller-owned lifecycle path, without backend-type dispatch or a tagged result union.
Collaborator
Author
|
🤖 Reworked the reconciliation boundary in 901007e. The tagged observation union and coordinator are gone; both worker RPC and Kubernetes adapters now return the same exact task-observation records plus optional worker-health facts. One controller operation validates Attempt identity, applies lifecycle policy, and accounts for health without inspecting backend kind. The focused controller/Kubernetes/worker-lifecycle suite passes 1,178 tests, and lint/Pyrefly pass. |
Move worker liveness, controller snapshots, status and capacity publication, autoscaler inputs, recovery, pruning, and teardown sequencing under Controller ownership. TaskBackend now receives complete phase requests and returns bounded decisions, observations, or provider effects without reading Iris persistence. Delete BackendRuntime and DbBackendWorkerStore. Worker and Kubernetes adapters retain provider I/O behind one registered backend. Physical capacity removal is a separate post-commit phase, so autoscale no longer multiplexes provisioning and teardown.
Split control-tick input assembly into phase-specific builders and carry submit-time capacity validation in one JobFeasibilityRequest. Dashboard tests now reuse the production observation-input builder, while transition tests call the production liveness fold instead of reproducing it. Update the reconcile and benchmark documentation to describe controller-built worker targets, fresh transition reads, and post-commit capacity removal.
rjpower
changed the title
…r-neutral-reconciliation # Conflicts: # lib/iris/tests/cluster/controller/test_dashboard.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make
Controllerthe sole owner of Iris persistence, worker liveness, lifecycle policy, and phase sequencing.TaskBackendnow receives complete request records for initialization, scheduling, reconciliation, observation, autoscaling, and capacity removal. It returns decisions, exact runtime observations, or provider effects without receiving a controller database, transaction, transition reader, or liveness tracker.Worker and Kubernetes reconciliation now produce the same exact task-attempt facts. The controller reloads state after provider I/O, fences each observation by Attempt UID, applies one lifecycle path, and commits before requesting physical capacity removal. Status and capacity are published on the control thread from controller-supplied facts; autoscaler recovery and updates return state for the controller to persist.
Delete
BackendRuntimeandDbBackendWorkerStore, and separate provisioning from worker teardown. The branch removes 1,824 lines and adds 1,536. RPC schemas and client behavior are unchanged. The controller, backend, and lifecycle journey suites pass 1,240 tests; repository formatting, Ruff, and Pyrefly checks pass.