Skip to content

fix(eks-addons): let Loki return query results larger than 4MiB - #220

Open
mabadir wants to merge 1 commit into
mainfrom
ma/loki-grpc-message-size
Open

mabadir wants to merge 1 commit into
mainfrom
ma/loki-grpc-message-size

Conversation

@mabadir

@mabadir mabadir commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

In prod, Loki's logs show log searches failing with:

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (31010467 vs. 4194304)

In single-binary mode the querier returns each result to the query frontend over gRPC, and Loki's server keeps the 4MiB message default. A filter over a busy workload's logs (e.g. {namespace="ravion-prod", workload="api"} |= "rate_limit" over a few hours) produces results well past that, so the search fails.

What

  • loki.server.grpc_server_max_recv_msg_size / grpc_server_max_send_msg_size = 100MiB (104857600), matching the querier's own client limit.
  • Test in tests/logs.tftest.hcl. Verified with helm template against grafana/loki 7.3.0 that both keys render into Loki's server config.
  • Release 0.23.2. fix(eks-addons): replace Loki's pod when moving it to a volume; roll Alloy faster #216 holds 0.23.1; whichever merges second keeps its own number (bump this one if it lands first).

Tests

  • tofu test in compute/eks/addons: 263 passed
  • charts/test.sh: 390 passed

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

Raises Loki's gRPC send and receive limits to 100MiB so larger log-query results can pass.

  • Loki log searches can return results larger than 4MiB.

Reviews (1) · Last reviewed commit: "fix(eks-addons): let Loki return query r..." · Reviewed by Greptile

In single-binary mode the querier hands each result back to the query
frontend over gRPC, and Loki's server kept the 4MiB message default. A
filter query over a busy workload's logs failed with "received message
larger than max (31010467 vs. 4194304)". Raise the server's send and
receive limits to 100MiB, the querier's own client limit.
@github-actions

Copy link
Copy Markdown

Ravion Module Publish Plan

Dry run only. No Ravion API mutations were made.

Module Current Version New Version Description
rvn-eks-addons 0.23.0 0.23.2 Loki answers log searches whose results pass 4MiB. Its gRPC server kept the 4MiB message default, so a filter over a busy workload's logs failed with "received message larger than max"; it now accepts up to 100MiB, the querier's own limit.

Diffs

rvn-eks-addons 0.23.0 -> 0.23.2

--- remote
+++ compiled
   - Each load balancer's security group is automatically allowed to reach pods through the EKS cluster security group (all TCP ports).
   - Because `TargetGroupBinding` depends on the controller, enabling any load balancer automatically installs the **load balancer controller** - there is nothing extra to configure.
 
-  Terraform source: [ravionhq/modules/compute/eks/addons](https://github.com/ravionhq/modules/tree/rvn-eks-addons@0.23.0/compute/eks/addons)
+  Terraform source: [ravionhq/modules/compute/eks/addons](https://github.com/ravionhq/modules/tree/rvn-eks-addons@0.23.2/compute/eks/addons)
 
   ## Prerequisites
 
@@
         execution_environment_overrides:
           security_group: << module.input.ravion_runner_security_group_id >>
         plan_permissions: '<< module.input.terraform_plan_default_policies_enabled == false ? {"replace": module.input.terraform_plan_iam_policy_arns} : (module.input.terraform_plan_iam_policy_arns ? {"attach": module.input.terraform_plan_iam_policy_arns} : nil) >>'
-        ref: rvn-eks-addons@0.23.0
+        ref: rvn-eks-addons@0.23.2
         repo: https://github.com/ravionhq/modules
         stack_id: <<stack.id>>
         terraform_variables:

@mabadir
mabadir requested a review from flybayer October 17, 2026 14:46
@ravion-ops

ravion-ops Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

@mabadir @flybayer This PR could not be combined with the queued merge tree.

Failed operation: merge #220 at dab1fc9 into the queue's accumulated changes. Base branch commit: 4dd9251.

How to resolve

  • Compare #220 at dab1fc9 with the conflicting head dab1fc9: exact commit diff.
  • On the conflicting PR branch, merge or rebase the latest base branch. If the earlier PRs have not merged yet, bring their listed changes into your branch in queue order.
  • Resolve the linked files while preserving both changes, run the required checks, and push the resolved commit. The queue will reassess the new head.

No conflict diff was captured, and assembly stopped before a candidate commit was created. Use the exact commit comparisons and conflicting files above; there is no candidate-commit diff to link.

@ravion-ops

ravion-ops Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Removed from the queue. Re-add the label to queue it again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants