Repository navigation
indexer health server hardening + gRPC API / nginx HA fixes (#764, #765, #766) - #882
Open
GoldenTuxbee wants to merge 4 commits into
Open
GoldenTuxbee wants to merge 4 commits into
GoldenTuxbee wants to merge 4 commits into
Conversation
…eadiness to poll loop (Telocel-Labs#764) Hardens crates/indexer/src/health.rs: - Request-line/header reads are bounded by a 5s timeout and an 8KiB budget, so a slowloris-style client can no longer hold a task/socket open indefinitely or force unbounded buffering. - /admin/dead-letter now requires a shared-secret Authorization: Bearer <ADMIN_TOKEN> header; with ADMIN_TOKEN unset the endpoint 404s instead of staying open. - /readyz fails once the poll loop's heartbeat (crates/indexer/src/liveness.rs, shared with the streamer) is older than READINESS_HEARTBEAT_STALE_SECS (default 120s), so a wedged-but-still-running poll loop is taken out of rotation instead of reporting ready forever. New env vars (both optional): ADMIN_TOKEN, READINESS_HEARTBEAT_STALE_SECS.
…d, HPA (Telocel-Labs#765) helm/trident/values.yaml: grpcApi.replicaCount default raised 1 -> 2 (still overridable per-environment, e.g. values-staging.yaml already pins it back to 1 for its smoke-test environment). Added grpcApi.podDisruptionBudget, topologySpreadConstraints, and hpa blocks mirroring the existing goApi config. helm/trident/templates/grpc-api-deployment.yaml: added the matching topologySpreadConstraints on the pod spec, a PodDisruptionBudget, and an HPA, following the same structure as the Go API's deployment/HPA templates. Not addressed here: the Go API's gRPC client (services/api/grpc/client.go) dials a single ClusterIP-backed address with no round_robin service config, so it keeps one long-lived HTTP/2 connection to one gRPC API pod regardless of replica count. Scaling replicas and adding a PDB still improves availability (a killed/drained pod forces a fast reconnect via the client's existing backoff/keepalive config instead of every REST read failing outright), but true load spreading across replicas needs a client-side change that's out of scope for this Helm-only fix.
…ge (Telocel-Labs#766) helm/trident/values.yaml: nginx.replicaCount default raised 1 -> 2 (staging still overrides to 1, PDB disabled there). Switched the PDB from minAvailable: 1 to maxUnavailable: 1 in both values.yaml and values-prod.yaml, and dropped the template's "replicaCount > 1" gate on rendering it: maxUnavailable never forbids every voluntary disruption the way minAvailable: 1 did at a single replica, so it no longer needs that special case. Added topologySpreadConstraints for nginx, mirroring goApi/grpcApi. helm/trident/templates/nginx-deployment.yaml: image reference now appends the digest when nginx.image.digest is set; pinned to nginx:1.27-alpine (the current stable mainline, replacing the EOL 1.25-alpine) by both tag and digest, satisfying Telocel-Labs#303's digest-pinning requirement for this image.
…#764) helm/trident/templates/indexer-deployment.yaml: adds the optional ADMIN_TOKEN env var from the existing secret, completing Telocel-Labs#764's health server auth change. A secret missing this key leaves the env var unset, which disables /admin/dead-letter (404) rather than leaving it open, same as running the indexer outside Helm with ADMIN_TOKEN unset.
GoldenTuxbee
force-pushed
the
fix/indexer-health-grpc-pdb-nginx-eol
branch
from
October 26, 2026 06:23
65ca1be to
45de506
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the indexer's health server against an unbounded/unauthenticated probe port and a wedged poll loop, and fixes two Helm availability gaps: the gRPC API running as a single replica with no PDB/HPA, and nginx running one replica with a PDB that blocked node drains on an EOL floating image tag.
closes #764
closes #765
closes #766
Changes
/admin/dead-letterunauthenticated, and readiness ignores the poll loop #764 - indexer health server hardening:crates/indexer/src/health.rsnow bounds request-line/header reads with a 5s timeout and an 8KiB budget, so a slowloris-style client can't hold a socket open forever or force unbounded buffering./admin/dead-letterrequires a shared-secretAuthorization: Bearer <ADMIN_TOKEN>header; withADMIN_TOKENunset the endpoint 404s instead of staying open./readyznow also fails once the poll loop's heartbeat (newcrates/indexer/src/liveness.rs, shared with the streamer viaheartbeat.record()each iteration) is older thanREADINESS_HEARTBEAT_STALE_SECS(default 120s). Two new optional env vars:ADMIN_TOKEN,READINESS_HEARTBEAT_STALE_SECS.helm/trident/values.yamlraisesgrpcApi.replicaCountdefault from 1 to 2 and addspodDisruptionBudget(minAvailable: 1),topologySpreadConstraints, andhpablocks mirroring the existinggoApiconfiguration.helm/trident/templates/grpc-api-deployment.yamlgets the matching topology spread on the pod spec plus a PDB and HPA template, same structure as the Go API's.minAvailable: 1, blocking node drains, on an EOL floating image tag #766 - nginx availability and image:nginx.replicaCountdefault raised from 1 to 2. The PDB switched fromminAvailable: 1tomaxUnavailable: 1(in bothvalues.yamlandvalues-prod.yaml), and the template'sreplicaCount > 1render gate was dropped sincemaxUnavailablenever forbids every voluntary disruption the wayminAvailable: 1did at a single replica. AddedtopologySpreadConstraintsfor nginx. The image is now pinned by tag and digest,nginx:1.27-alpine(current stable mainline, replacing the EOL1.25-alpine) atsha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10, satisfying the existing digest-pinning convention from Infra: pin all remaining floating base images and GitHub Action versions to digests #303.Not included
services/api/grpc/client.go: it dials a single address withgrpc.NewClientand noWithDefaultServiceConfiground_robin policy, and the Service isClusterIP(not headless), so it keeps one long-lived HTTP/2 connection to a single gRPC API pod regardless of replica count. Scaling replicas and adding a PDB still improves availability (a killed/drained pod now forces a fast reconnect through the client's existing backoff/keepalive config, instead of every REST read failing outright), but genuine load spreading across replicas needs a client-side change (headless Service + round_robin policy, or an L7 proxy) that touches Go application code and felt too large to make correctly and safely as part of this Helm-only batch. Flagging it rather than guessing at a fix.Test plan
cargo build -p trident-indexer: compiles cleanly with these changes. Note:upstream/devcurrently fails to build/test as-is due to two pre-existing, unrelated issues incrates/indexer/src/redis_stream/relay.rs(a borrow-checker error) and a staleRelayConfigtest fixture incrates/indexer/src/streamer/mod.rsmissing itsnetworkfield; verified by stashing this branch's changes and reproducing the same failures against a cleanupstream/devcheckout. Neither touches any file this PR changes.cargo test -p trident-indexer --bin trident-indexer health::/liveness::/config::: all pass, including new unit tests for the bearer-token check and the heartbeat-staleness readiness path, plus the existing/admin/dead-letterschema regression tests.helm lint helm/tridentandhelm template helm/trident(default,-f values.yaml -f values-prod.yaml, and-f values.yaml -f values-staging.yaml): all render without error. Manually inspected the rendered output for the gRPC API's PDB/HPA, nginx's PDB (maxUnavailable: 1), and the nginx image reference (nginx:1.27-alpine@sha256:...).Staging and the indexer's own single-replica design are intentionally left alone (staging already scales these services down with its own documented reasoning; the indexer's replicaCount: 1 is a deliberate single-poll-loop design, not the gap these issues describe).
ADMIN_TOKENis wired intohelm/trident/templates/indexer-deployment.yamlas an optional key on the chart's existing secret (global.existingSecret); a deployment that doesn't set it simply runs with/admin/dead-letterdisabled, same as running the indexer outside Helm.