[mabadir]

• In merge queue / Conflicts with main
• Prioritize

Why

The prod eks-addons apply (prun_3Kmskfkhw8FYBFveYLSwThrVZBL) failed with helm_release.loki: context deadline exceeded, and Alloy's release hit the same deadline.

Loki. The statefulset-volume-migration Job deleted the Loki StatefulSet with --cascade=orphan. The recreated StatefulSet then:

• adopted the running pod;
• tried to add the new claim to it in place, which Kubernetes rejects (Pod "ravion-loki-0" is invalid: spec: Forbidden: pod updates may not change fields other than ...);
• never reached the step that replaces the pod.

As a result, storage-ravion-loki-0 stayed Pending, Loki kept running on its emptyDir, and Helm timed out. Prod was fixed by hand: flush Loki, delete the pod.

Alloy. The DaemonSet rolled one node at a time, about 30s per node. On 12 nodes that took longer than Helm's 5-minute wait.

What

statefulset-volume-migration 0.2.0: the Job now runs files/migrate.sh (kubectl, with a shell from busybox, as ebs-storage does). The script handles three cases:

State	Action
Layout differs from persistent	Flush Loki (POST /flush), then delete the StatefulSet with its pods (--cascade=foreground). The Loki release recreates both. About one minute of Loki downtime.
Layout matches, but a pod doesn't mount its claim	Flush, then delete that pod. Clusters that 0.1.0 left stuck recover on the next apply.
Layout matches and the pods are healthy	Nothing.

Other changes:

• RBAC: get/delete on the one StatefulSet and on its pods named by ordinal (<name>-0 to <name>-(maxReplicas-1)). No list or watch.
• Alloy: updateStrategy.rollingUpdate.maxUnavailable: 25%, so an upgrade takes about four rollout waves whatever the cluster size.
• Version: rvn-eks-addons 0.22.1.

Upgrade impact: the chart version is part of loki_volume_layout, so on upgrade the Job runs once everywhere. It's a no-op where the layout already matches, but Loki's pod annotation changes, so Loki restarts once.

Tests

• tests/test_statefulset_volume_migration.py, which is new and wired into the Helm Charts workflow:
  • runs the script offline against fake kubectl and wget;
  • 10 cases: missing StatefulSet, layout change in both directions, matching layout, the wedged pod, multiple replicas, missing pod, no flush URL, failed flush, failed delete.
• charts/test.sh statefulset-volume-migration: 9 checks on the rendered chart (images and command, ConfigMap equals the file, env, hook, RBAC scope).
• tofu test for addon_resilience and logs: 34 passed, including new assertions for the flush URL, the busybox image and Alloy's update strategy.
• shellcheck is clean. The definition guardrails, compile and readme --check pass. The tooling tests pass (144).

Confidence Score: 4/5

Fix the pod permission bound before merging so larger Loki installations can complete the upgrade.

Findings

1.  Larger Loki upgrades fail ▶

Fix with agent prompt

### Issue 1
compute/eks/addons/charts/statefulset-volume-migration/files/migrate.sh:58-63
If a caller sets `singleBinary.replicas` to four or more through `loki_helm_values`, this loop reads pod names that the Job cannot access. The module never sets `statefulSet.maxReplicas`, so the Role permits only pods numbered 0 through 2. Reading `ravion-loki-3` fails with Forbidden and stops the Job, even when every pod mounts the right claim. This blocks the Loki upgrade on the chart-version change.

Pass a permission bound that covers the caller's replica count, or expose a way to configure it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Changes Loki volume moves to flush first and delete the StatefulSet with its pods. It also replaces pods left without their claims by the earlier migration.

• Loki's volume migration flushes and replaces pods that lack their claims.
• Alloy updates a quarter of its nodes in each rollout wave.

Acknowledged behavior: mabadir states that this upgrade changes Loki's pod annotation and causes one restart even when its volume layout already matches.

Acknowledged behavior: mabadir describes flushing as best effort; a failed flush does not stop the migration.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Migration Job] --> B{StatefulSet exists?}
  B -->|No| C[Finish]
  B -->|Yes| D{Volume layout differs?}
  D -->|Yes| E[Request Loki flush]
  E --> F[Delete StatefulSet and pods]
  F --> G[Loki release recreates them]
  D -->|No| H[Read each pod]
  H --> I{Required claim missing?}
  I -->|Yes| J[Request flush and replace pod]
  I -->|No| C

Loading

Reviews (1) · Last reviewed commit: "fix(eks-addons): replace Loki's pod when..." · Reviewed by Greptile

[github-actions]

Ravion Module Publish Plan

Dry run only. No Ravion API mutations were made.

Module	Current Version	New Version	Description
rvn-eks-addons	0.23.0	0.23.1	Fix applies that time out on Loki and Alloy. Moving Loki onto a persistent volume now flushes it and replaces its pod, instead of leaving the old pod running on its emptyDir while the apply waits; clusters left that way recover on the next apply. Alloy now updates a quarter of the nodes at a time instead of one, so its rollout finishes within Helm's wait on any cluster size.

Diffs

rvn-eks-addons 0.23.0 -> 0.23.1

--- remote
+++ compiled
   - Each load balancer's security group is automatically allowed to reach pods through the EKS cluster security group (all TCP ports).
   - Because `TargetGroupBinding` depends on the controller, enabling any load balancer automatically installs the **load balancer controller** - there is nothing extra to configure.
 
-  Terraform source: [ravionhq/modules/compute/eks/addons](https://github.com/ravionhq/modules/tree/rvn-eks-addons@0.23.0/compute/eks/addons)
+  Terraform source: [ravionhq/modules/compute/eks/addons](https://github.com/ravionhq/modules/tree/rvn-eks-addons@0.23.1/compute/eks/addons)
 
   ## Prerequisites
 
@@
         execution_environment_overrides:
           security_group: << module.input.ravion_runner_security_group_id >>
         plan_permissions: '<< module.input.terraform_plan_default_policies_enabled == false ? {"replace": module.input.terraform_plan_iam_policy_arns} : (module.input.terraform_plan_iam_policy_arns ? {"attach": module.input.terraform_plan_iam_policy_arns} : nil) >>'
-        ref: rvn-eks-addons@0.23.0
+        ref: rvn-eks-addons@0.23.1
         repo: https://github.com/ravionhq/modules
         stack_id: <<stack.id>>
         terraform_variables:

[ravion-ops]

@mabadir @flybayer This PR could not be combined with the queued merge tree.

Failed operation: merge #216 at be817f6 into the queue's accumulated changes. Base branch commit: 4dd9251.

How to resolve

• Compare #216 at be817f6 with the conflicting head be817f6: exact commit diff.
• On the conflicting PR branch, merge or rebase the latest base branch. If the earlier PRs have not merged yet, bring their listed changes into your branch in queue order.
• Resolve the linked files while preserving both changes, run the required checks, and push the resolved commit. The queue will reassess the new head.

No conflict diff was captured, and assembly stopped before a candidate commit was created. Use the exact commit comparisons and conflicting files above; there is no candidate-commit diff to link.