Skip to content

fix: indexer Fly config, per-IP rate limiting, and release image signing/SBOM - #879

Open
Ejere wants to merge 3 commits into
Telocel-Labs:devfrom
Ejere:fix/indexer-retention-poll-interval-rate-limiter-sbom
Open

Ejere wants to merge 3 commits into
Telocel-Labs:devfrom
Ejere:fix/indexer-retention-poll-interval-rate-limiter-sbom

Conversation

@Ejere

@Ejere Ejere commented Oct 21, 2026

Copy link
Copy Markdown

Summary

Fixes three infra/backend bugs in a single batch: the broken fly/indexer.toml config that crash-loops every indexer deploy, a per-IP rate limiter that collapses to one global bucket behind Helm/Fly's proxies, and unsigned/unattested release images. The fourth assigned issue (historical ingestion beyond RPC retention) is left out; see "Not included" below.

closes #777
closes #779
closes #780

Changes

  • infrastructure: fly/indexer.toml sets POLL_INTERVAL_MS, which now aborts indexer startup #777 - fly/indexer.toml sets POLL_INTERVAL_MS, which now aborts indexer startup: Config::from_env (crates/indexer/src/config.rs:276-281) hard-fails startup if POLL_INTERVAL_MS is set at all, but fly/indexer.toml still set it, crash-looping every Fly deploy. Replaced with POLL_INTERVAL_FLOOR_MS/POLL_INTERVAL_CEILING_MS (both 5000, reproducing the old fixed cadence), set NETWORK explicitly instead of relying on the testnet default, and added scripts/check-fly-env-rejected.sh (wired into the env-reference CI job) which fails if any fly/*.toml sets an env var the indexer's config loader rejects. Verified the new check catches the exact regression it's meant to (added POLL_INTERVAL_MS back temporarily and confirmed it fails, then reverted).
  • api: per-IP rate limiter collapses to a single global bucket behind the shipped proxies #779 - per-IP rate limiter collapses to a single global bucket behind the shipped proxies: trustedClientIP (services/api/middleware/abuse.go) previously only trusted the last X-Forwarded-For hop unconditionally, and neither Helm nor Fly set TRUSTED_PROXY_ENABLED at all, so every request was keyed on the proxy's own IP. Added CIDR-aware trusted-proxy logic: Fly-Client-IP is used directly on Fly (unforgeable, set by Fly's edge), and the XFF chain is walked from the right skipping only hops inside a new TRUSTED_PROXY_CIDRS list, with the immediate TCP peer itself required to be in that CIDR set before XFF is trusted at all (so a client bypassing the proxy can't forge a trusted-looking chain). Wired TRUSTED_PROXY_ENABLED=true + TRUSTED_PROXY_CIDRS (RFC1918 ranges) into the Helm chart's goApi.env/trustedProxyCIDRs values, and TRUSTED_PROXY_ENABLED=true (no CIDRs needed) into fly/api.toml. Updated docs/ENVIRONMENT.md, docs/deployment.md, docs/threat-model.md, docs/site/self-hosting/configuration.mdx. Added unit tests covering Fly-Client-IP priority, CIDR-walk, untrusted-peer rejection, and the legacy last-hop-only fallback when no CIDRs are configured.
  • security: release images are unsigned and carry no provenance or SBOM attestation #780 - release images are unsigned and carry no provenance or SBOM attestation: .github/workflows/release.yml built and pushed four images to GHCR with no way to verify a digest came from this repo's release workflow. Added, per image (indexer, gRPC API, Go API, migrate): keyless cosign signing via GitHub OIDC (sigstore/cosign-installer + cosign sign, no stored signing key), an SPDX SBOM (anchore/sbom-action) attested with cosign attest, and a SLSA build-provenance attestation via actions/attest-build-provenance pushed to the registry. Documented the cosign verify / gh attestation verify / SBOM-inspection commands in docs/deployment.md, plus a note on wiring an admission policy against the same OIDC identity.

Not included

  • indexer: no historical ingestion source beyond the RPC retention window #774 - no historical ingestion source beyond the RPC retention window: scoped as "add a historical source (Galexie ledger-metadata data lake output or captive-core replay) feeding the same event path, and route backfill jobs older than RPC retention to it automatically." This is a genuinely new ingestion architecture, not a bug fix: it needs a new data-source integration (object-storage reader or captive-core replay), new XDR/ledger-metadata parsing wired into the existing commit path, and a new routing decision in the backfill job scheduler, plus infra (an archive source actually needs to be reachable: a Galexie bucket or captive-core binary/dataset). None of that exists in the repo today and I don't have a safe way to stand up or point at a real archive data source in this environment. Implementing a stub or partial version would be worse than not shipping it, so this is left for a dedicated PR with proper access to an archive source.

Test plan

  • go build ./... and go test ./middleware/... in services/api pass, including new tests: TestTrustedClientIP_FlyClientIPTakesPriority, TestTrustedClientIP_FlyClientIPIgnoredWhenNotTrusting, TestTrustedClientIP_WalksPastTrustedCIDRs, TestTrustedClientIP_IgnoresXFFWhenPeerNotTrusted, TestTrustedClientIP_TrustsLastHopOnly_NoCIDRsConfigured (renamed from the old last-hop-only test, still passing to confirm the no-CIDRs fallback).
  • bash scripts/check-fly-env-rejected.sh passes against the fixed fly/indexer.toml, and was manually verified to fail when POLL_INTERVAL_MS is reintroduced.
  • bash scripts/check-env-reference.sh still passes (new TRUSTED_PROXY_CIDRS var documented).
  • helm lint helm/trident passes; helm template trident helm/trident confirmed TRUSTED_PROXY_ENABLED/TRUSTED_PROXY_CIDRS render correctly onto the go-api Deployment.
  • .github/workflows/release.yml validated with actionlint (clean) and python3 -c "import yaml; yaml.safe_load(...)" (valid YAML); the cosign/SBOM/attestation steps themselves can only be exercised by an actual tag-triggered release run, which this PR doesn't trigger.

No Rust source was changed (only fly/indexer.toml, a new bash script, and docs), so no cargo build was required for #777. The Go API changes were built and tested directly. I did not have a live Fly/Helm cluster to deploy against, so the fly deploy / helm install "done when" criteria in #777/#779 are verified by config inspection and the automated CI check rather than an end-to-end deploy.

@drips-wave

drips-wave Bot commented Oct 21, 2026

Copy link
Copy Markdown

@Ejere Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Ejere added 3 commits October 26, 2026 06:12
…INTERVAL_MS (Telocel-Labs#777)

POLL_INTERVAL_MS was removed from the indexer's effective config, and
Config::from_env now fails startup if it is set at all, but
fly/indexer.toml still set it, crash-looping every Fly deploy. Replace it
with POLL_INTERVAL_FLOOR_MS/POLL_INTERVAL_CEILING_MS (5000/5000,
reproducing the old fixed cadence), set NETWORK explicitly instead of
relying on the testnet default, and add a CI check
(scripts/check-fly-env-rejected.sh) that fails if any fly/*.toml sets an
env var the indexer's config loader now hard-rejects.
… for per-IP rate limiting (Telocel-Labs#779)

Neither Helm nor Fly set TRUSTED_PROXY_ENABLED, so the per-IP limiter
keyed every request on the proxy's own address, collapsing all clients
behind it into one shared bucket. Behind Helm's LoadBalancer -> nginx ->
Go API chain, trusting only the last XFF hop (the prior behaviour) also
attributed every request to the load balancer rather than the real
client.

trustedClientIP now:
  - uses Fly-Client-IP directly when present (set by Fly's edge, not
    forgeable by the client)
  - otherwise walks X-Forwarded-For from the right, skipping hops inside
    the configured TRUSTED_PROXY_CIDRS, requiring the immediate TCP peer
    itself be inside one of those CIDRs before trusting XFF at all so a
    client connecting directly cannot forge a trusted-looking chain
  - falls back to the prior last-hop-only behaviour when
    TRUSTED_PROXY_CIDRS is unset, so existing single-proxy-hop
    deployments are unaffected

Wires TRUSTED_PROXY_ENABLED=true + TRUSTED_PROXY_CIDRS (RFC1918 ranges,
covering nginx's pod IP on any standard cluster) into the Helm chart, and
TRUSTED_PROXY_ENABLED=true (no CIDRs needed) into fly/api.toml. Updates
docs/ENVIRONMENT.md, docs/deployment.md, docs/threat-model.md and
docs/site/self-hosting/configuration.mdx.
…l-Labs#780)

.github/workflows/release.yml pushed the indexer, gRPC API, Go API and
migration images to GHCR with no way to verify a given digest actually
came from this repository's release workflow.

Add, per image:
  - keyless cosign signing (sigstore/cosign-installer + cosign sign),
    using the workflow run's own GitHub OIDC identity; no signing key is
    stored in the repo or in secrets
  - an SPDX SBOM (anchore/sbom-action) attested onto the image with
    cosign attest
  - a SLSA build-provenance attestation via
    actions/attest-build-provenance, pushed to the registry

Document the cosign verify / gh attestation verify commands (and an SBOM
inspection one-liner) in docs/deployment.md, with a note on wiring an
admission policy against the same identity.
@Ejere
Ejere force-pushed the fix/indexer-retention-poll-interval-rate-limiter-sbom branch from 454c0b6 to 57e5e2b Compare October 26, 2026 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant