Skip to content

indexer health server hardening + gRPC API / nginx HA fixes (#764, #765, #766) - #882

Open
GoldenTuxbee wants to merge 4 commits into
Telocel-Labs:devfrom
GoldenTuxbee:fix/indexer-health-grpc-pdb-nginx-eol
Open

GoldenTuxbee wants to merge 4 commits into
Telocel-Labs:devfrom
GoldenTuxbee:fix/indexer-health-grpc-pdb-nginx-eol

Conversation

@GoldenTuxbee

Copy link
Copy Markdown
Contributor

Summary

Hardens the indexer's health server against an unbounded/unauthenticated probe port and a wedged poll loop, and fixes two Helm availability gaps: the gRPC API running as a single replica with no PDB/HPA, and nginx running one replica with a PDB that blocked node drains on an EOL floating image tag.

closes #764
closes #765
closes #766

Changes

Not included

  • infrastructure: Helm deploys the gRPC API as a single replica with no PDB or autoscaling #765's Go client load-balancing note: the issue's scope asks to "confirm the Go client load-balances across endpoints." I checked services/api/grpc/client.go: it dials a single address with grpc.NewClient and no WithDefaultServiceConfig round_robin policy, and the Service is ClusterIP (not headless), so it keeps one long-lived HTTP/2 connection to a single gRPC API pod regardless of replica count. Scaling replicas and adding a PDB still improves availability (a killed/drained pod now forces a fast reconnect through the client's existing backoff/keepalive config, instead of every REST read failing outright), but genuine load spreading across replicas needs a client-side change (headless Service + round_robin policy, or an L7 proxy) that touches Go application code and felt too large to make correctly and safely as part of this Helm-only batch. Flagging it rather than guessing at a fix.

Test plan

  • cargo build -p trident-indexer: compiles cleanly with these changes. Note: upstream/dev currently fails to build/test as-is due to two pre-existing, unrelated issues in crates/indexer/src/redis_stream/relay.rs (a borrow-checker error) and a stale RelayConfig test fixture in crates/indexer/src/streamer/mod.rs missing its network field; verified by stashing this branch's changes and reproducing the same failures against a clean upstream/dev checkout. Neither touches any file this PR changes.
  • cargo test -p trident-indexer --bin trident-indexer health:: / liveness:: / config::: all pass, including new unit tests for the bearer-token check and the heartbeat-staleness readiness path, plus the existing /admin/dead-letter schema regression tests.
  • helm lint helm/trident and helm template helm/trident (default, -f values.yaml -f values-prod.yaml, and -f values.yaml -f values-staging.yaml): all render without error. Manually inspected the rendered output for the gRPC API's PDB/HPA, nginx's PDB (maxUnavailable: 1), and the nginx image reference (nginx:1.27-alpine@sha256:...).

Staging and the indexer's own single-replica design are intentionally left alone (staging already scales these services down with its own documented reasoning; the indexer's replicaCount: 1 is a deliberate single-poll-loop design, not the gap these issues describe). ADMIN_TOKEN is wired into helm/trident/templates/indexer-deployment.yaml as an optional key on the chart's existing secret (global.existingSecret); a deployment that doesn't set it simply runs with /admin/dead-letter disabled, same as running the indexer outside Helm.

…eadiness to poll loop (Telocel-Labs#764)

Hardens crates/indexer/src/health.rs:
- Request-line/header reads are bounded by a 5s timeout and an 8KiB budget,
  so a slowloris-style client can no longer hold a task/socket open
  indefinitely or force unbounded buffering.
- /admin/dead-letter now requires a shared-secret Authorization: Bearer
  <ADMIN_TOKEN> header; with ADMIN_TOKEN unset the endpoint 404s instead
  of staying open.
- /readyz fails once the poll loop's heartbeat (crates/indexer/src/liveness.rs,
  shared with the streamer) is older than READINESS_HEARTBEAT_STALE_SECS
  (default 120s), so a wedged-but-still-running poll loop is taken out of
  rotation instead of reporting ready forever.

New env vars (both optional): ADMIN_TOKEN, READINESS_HEARTBEAT_STALE_SECS.
…d, HPA (Telocel-Labs#765)

helm/trident/values.yaml: grpcApi.replicaCount default raised 1 -> 2 (still
overridable per-environment, e.g. values-staging.yaml already pins it back
to 1 for its smoke-test environment). Added grpcApi.podDisruptionBudget,
topologySpreadConstraints, and hpa blocks mirroring the existing goApi
config.

helm/trident/templates/grpc-api-deployment.yaml: added the matching
topologySpreadConstraints on the pod spec, a PodDisruptionBudget, and an
HPA, following the same structure as the Go API's deployment/HPA templates.

Not addressed here: the Go API's gRPC client (services/api/grpc/client.go)
dials a single ClusterIP-backed address with no round_robin service config,
so it keeps one long-lived HTTP/2 connection to one gRPC API pod regardless
of replica count. Scaling replicas and adding a PDB still improves
availability (a killed/drained pod forces a fast reconnect via the
client's existing backoff/keepalive config instead of every REST read
failing outright), but true load spreading across replicas needs a
client-side change that's out of scope for this Helm-only fix.
…ge (Telocel-Labs#766)

helm/trident/values.yaml: nginx.replicaCount default raised 1 -> 2 (staging
still overrides to 1, PDB disabled there). Switched the PDB from
minAvailable: 1 to maxUnavailable: 1 in both values.yaml and
values-prod.yaml, and dropped the template's "replicaCount > 1" gate on
rendering it: maxUnavailable never forbids every voluntary disruption the
way minAvailable: 1 did at a single replica, so it no longer needs that
special case. Added topologySpreadConstraints for nginx, mirroring
goApi/grpcApi.

helm/trident/templates/nginx-deployment.yaml: image reference now appends
the digest when nginx.image.digest is set; pinned to nginx:1.27-alpine (the
current stable mainline, replacing the EOL 1.25-alpine) by both tag and
digest, satisfying Telocel-Labs#303's digest-pinning requirement for this image.
…#764)

helm/trident/templates/indexer-deployment.yaml: adds the optional
ADMIN_TOKEN env var from the existing secret, completing Telocel-Labs#764's health
server auth change. A secret missing this key leaves the env var unset,
which disables /admin/dead-letter (404) rather than leaving it open, same
as running the indexer outside Helm with ADMIN_TOKEN unset.
@GoldenTuxbee
GoldenTuxbee force-pushed the fix/indexer-health-grpc-pdb-nginx-eol branch from 65ca1be to 45de506 Compare October 26, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant