Repository navigation
fix(eks-addons): replace Loki's pod when moving it to a volume; roll Alloy faster #216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mabadir
wants to merge
3
commits into
main
Choose a base branch
from
ma/fix-statefulset-volume-migration
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 2 additions & 2 deletions
4
compute/eks/addons/charts/statefulset-volume-migration/Chart.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| apiVersion: v2 | ||
| name: statefulset-volume-migration | ||
| description: Delete one StatefulSet with --cascade=orphan, leaving its pods running, so the release that owns it can recreate it with a changed volume layout. | ||
| description: Move one StatefulSet to a changed volume layout by deleting it with its pods so the owning release recreates both, and replace pods an earlier orphaning migration left without their claims. | ||
| type: application | ||
| version: 0.1.0 | ||
| version: 0.2.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
79 changes: 79 additions & 0 deletions
79
compute/eks/addons/charts/statefulset-volume-migration/files/migrate.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| #!/bin/sh | ||
| # Moves one StatefulSet to the volume layout its owning release is about to | ||
| # apply, and recovers pods left behind by an earlier, orphaning migration. | ||
| # | ||
| # A StatefulSet's volumeClaimTemplates are immutable, so the owner cannot switch | ||
| # between an emptyDir and a PersistentVolumeClaim in place. When the layout | ||
| # differs, the StatefulSet is deleted together with its pods, and the owner, | ||
| # upgraded right after this Job, recreates it with fresh pods on the new | ||
| # volumes. Orphaning the pods instead does not work: the recreated StatefulSet | ||
| # adopts them, tries to add the claim to each running pod, which Kubernetes | ||
| # forbids, and never gets as far as replacing them. | ||
| # | ||
| # When the layout already matches, only pods that do not mount their claims are | ||
| # deleted: the ones chart 0.1.0 orphaned. Their StatefulSet recreates them. A | ||
| # StatefulSet in the right shape with healthy pods is left alone. | ||
| # | ||
| # Before any pod goes, FLUSH_URL (when set) is POSTed so the store writes what | ||
| # it holds in memory to object storage. | ||
| set -eu | ||
|
|
||
| : "${STATEFULSET:?}" "${NAMESPACE:?}" "${PERSISTENT:?}" | ||
| FLUSH_URL="${FLUSH_URL:-}" | ||
| TIMEOUT="${DELETE_TIMEOUT:-10m}" | ||
|
|
||
| log() { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*"; } | ||
|
|
||
| flush() { | ||
| [ -n "${FLUSH_URL}" ] || return 0 | ||
| log "flushing ${FLUSH_URL}" | ||
| # Best effort: a store that does not answer usually has nothing left in | ||
| # memory, and refusing would leave the layout stuck. | ||
| wget -q -T 120 -O /dev/null --post-data= "${FLUSH_URL}" || log "flush failed; continuing" | ||
| } | ||
|
|
||
| # "<replicas> <template>,<template>," or nothing when there is no StatefulSet. | ||
| current="$(kubectl get statefulset "${STATEFULSET}" --namespace="${NAMESPACE}" --ignore-not-found \ | ||
| -o go-template='{{.spec.replicas}} {{range .spec.volumeClaimTemplates}}{{.metadata.name}},{{end}}')" | ||
| if [ -z "${current}" ]; then | ||
| log "no StatefulSet ${NAMESPACE}/${STATEFULSET}; nothing to migrate" | ||
| exit 0 | ||
| fi | ||
| replicas="${current%% *}" | ||
| templates="$(echo "${current#* }" | tr ',' ' ')" | ||
|
|
||
| has_claims=false | ||
| [ -n "$(echo "${templates}" | tr -d ' ')" ] && has_claims=true | ||
|
|
||
| if [ "${has_claims}" != "${PERSISTENT}" ]; then | ||
| log "volume layout changes (persistent: ${has_claims} -> ${PERSISTENT}); deleting ${NAMESPACE}/${STATEFULSET} and its pods" | ||
| flush | ||
| kubectl delete statefulset "${STATEFULSET}" --namespace="${NAMESPACE}" \ | ||
| --cascade=foreground --ignore-not-found --wait --timeout="${TIMEOUT}" | ||
| log "deleted; the owning release recreates it" | ||
| exit 0 | ||
| fi | ||
|
|
||
| i=0 | ||
| while [ "${i}" -lt "${replicas}" ]; do | ||
| pod="${STATEFULSET}-${i}" | ||
| i=$((i + 1)) | ||
| # "present <claim> <claim> ", or nothing when the pod does not exist. | ||
| mounted="$(kubectl get pod "${pod}" --namespace="${NAMESPACE}" --ignore-not-found \ | ||
| -o go-template='present {{range .spec.volumes}}{{with .persistentVolumeClaim}}{{.claimName}} {{end}}{{end}}')" | ||
| [ -n "${mounted}" ] || continue | ||
|
|
||
| missing="" | ||
| for template in ${templates}; do | ||
| case " ${mounted} " in | ||
| *" ${template}-${pod} "*) ;; | ||
| *) missing="${missing} ${template}-${pod}" ;; | ||
| esac | ||
| done | ||
| [ -n "${missing}" ] || continue | ||
|
|
||
| log "${NAMESPACE}/${pod} does not mount${missing}; replacing it" | ||
| flush | ||
| kubectl delete pod "${pod}" --namespace="${NAMESPACE}" --ignore-not-found --wait --timeout="${TIMEOUT}" | ||
| done | ||
| log "volume layout up to date" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 32 additions & 9 deletions
41
compute/eks/addons/charts/statefulset-volume-migration/values.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,48 @@ | ||
| # A StatefulSet's volumeClaimTemplates are immutable, so the release that owns | ||
| # it cannot switch between an emptyDir and a PersistentVolumeClaim in place. | ||
| # This chart's Job deletes the StatefulSet with --cascade=orphan after every | ||
| # install or upgrade of this release; its pods keep running until the owning | ||
| # release recreates the StatefulSet and replaces them. The caller upgrades this | ||
| # release only when the owner's volume layout changes, and upgrades the owner | ||
| # right after it. | ||
| # This chart's Job runs after every install or upgrade of this release (see | ||
| # files/migrate.sh): | ||
| # - when the StatefulSet's layout differs from `persistent`, it deletes the | ||
| # StatefulSet together with its pods, and the owning release, upgraded | ||
| # right after this one, recreates both on the new volumes; | ||
| # - when the layout already matches, it replaces only pods that do not mount | ||
| # their claims (left by chart 0.1.0, which orphaned them); | ||
| # - otherwise it changes nothing. | ||
| # Pods are never orphaned: a recreated StatefulSet that adopts a pod without | ||
| # its claim cannot add the claim in place and never replaces the pod. | ||
|
|
||
| # -- The StatefulSet to delete. Missing is fine (a first install). | ||
| # -- The StatefulSet to migrate. Missing is fine (a first install). | ||
| statefulSet: | ||
| name: "" | ||
| namespace: "" | ||
| # -- Highest replica count the Job may need to handle. The Role names each | ||
| # pod it may delete, <name>-0 to <name>-(maxReplicas - 1). | ||
| maxReplicas: 3 | ||
|
|
||
| # -- The volume layout the owner is moving to. Informational: it names the Job. | ||
| # -- The volume layout the owner is moving to: true for PersistentVolumeClaims, | ||
| # false for none. | ||
| persistent: false | ||
|
|
||
| # -- kubectl image the Job runs. Pinned; override for clusters that must pull | ||
| # from a private mirror. | ||
| # -- POSTed before any pod is removed, so the store writes what it holds in | ||
| # memory to object storage (Loki: http://<service>:3100/flush). Empty skips it. | ||
| flushUrl: "" | ||
|
|
||
| # -- How long each delete may wait for the pods to go. | ||
| deleteTimeout: 4m | ||
|
|
||
| # -- kubectl image the Job runs. It has no shell, so busybox supplies one. | ||
| # Pinned; override for clusters that must pull from a private mirror. | ||
| image: | ||
| repository: registry.k8s.io/kubectl | ||
| tag: v1.36.4 | ||
| pullPolicy: IfNotPresent | ||
|
|
||
| # -- Statically linked busybox, copied into the Job's pod by an init | ||
| # container to provide sh and wget. | ||
| busyboxImage: | ||
| repository: public.ecr.aws/docker/library/busybox | ||
| tag: 1.37.0-musl | ||
| pullPolicy: IfNotPresent | ||
|
|
||
| # -- Give up after this many failed attempts and fail the release. | ||
| backoffLimit: 3 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.