Add Valkey operations handbook#357
Open
Grant McCloskey (MushuEE) wants to merge 2 commits into
Open
Conversation
Grant McCloskey (MushuEE)
requested review from
Dmitry Berkovich (dberkov) and
Julian Gutierrez Oschmann (juli4n)
Operational reference for the Valkey persistence tier and the
in-process worker cache that sits in front of it. Written for
operators, on-call engineers, and contributors who need to reason
about the storage tier without first reading the code.
README.md — entry point, scope, conventions
topology.md — what's deployed, sizing math, configuration knobs
lifecycle.md — actor state machine, four workflows, worker cache
and eligibility model
operations.md — failure modes with inline recovery, common admin
operations, open risks
Verified every handbook claim against the code and fixed what had drifted: - lifecycle: add the CRASHED state and crash-detection paths; correct the boot=true semantics (it only skips the golden snapshot — it cannot bypass an actor's own local checkpoint) and the syncer behavior for finalized PAUSED actors; rewrite eligibility around per-worker isWorkerEligibleForActor matching; document the atespace gate on create and the non-atespace-scoped lock key. - operations: add the coordinated-failover and backup/restore procedures; fix the deployment name, kubectl-ate invocations, and multi-part AOF repair path; correct the cache-not-ready window; note the unversioned-delete pub/sub caveat and pool-label fan-out; add risks rows (paused-actor dead-node trap, stale local-snapshot node list, leaked snapshots). - topology: fix the actor key format and add the atespace key family; document IAM-vs-mTLS client auth; slim the page to deployment description. - scaling_to_10m.md (new): tiered growth path from pilot to the supported 10M-actor envelope — capacity math with stated bases, graduation triggers with validation gates, the dual-profile durability contract, the deployed-vs-target configuration ladder, and the exit ramp beyond the envelope. Now owns all sizing content.
Grant McCloskey (MushuEE)
force-pushed
the
design/valkey-operations-eval
branch
from
c9913ee to
f8a4694
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Operational reference for the Valkey persistence tier and the in-process worker cache that sits in front of it. Written for operators, on-call engineers, and contributors who need to reason about the storage tier without first reading the code. This will be used as ref for further investigation on if Valkey will really stand-up.
NOTE: We chose Valkey due to it's speed for the critical scheduling requirements. With WorkerCache now being in memory, we get the speed for scheduling and still rely on eventual consistency. Do we now want more durability? Also still need to answer: what data can we 100% NOT lose in these failure situations.
README.md — entry point, scope, conventions
topology.md — what's deployed, sizing math, configuration knobs
lifecycle.md — actor state machine, four workflows, worker cache
and eligibility model
operations.md — failure modes with inline recovery, common admin
operations, open risks
#12