registry: wire the per-IP rate limiter, fix audit-export retries - #113
Merged
Conversation
Two independent bugs where working code was never reached.
1. The per-IP rate limiter was dead code.
NewRateLimiter(100, time.Second, 50_000) is constructed at accept.go:585 and
fully implemented — token buckets, whitelist-aware elevated rates, stale
bucket pruning, a maxBuckets cap, metrics hooks. Allow() had ZERO production
call sites. The only methods ever invoked on a.rateLimiter were Cleanup,
IsWhitelisted, SetWhitelist and WhitelistSize.
So the entire per-IP tier never ran. A single client IP was bounded only by
globalBucket, which is PROCESS-WIDE and shared with every other client, so
one IP could consume the whole registry budget and starve the fleet. The only
other guard was a per-connection >500 req/s check, trivially sidestepped by
opening more connections.
It also made pilot_ratelimit_denied_total{kind="ip"} structurally always
zero: the dashboard reported "no IP-level abuse" because nothing could
increment that counter, not because none was happening.
Allow is now called on both the text and binary accept paths, inside the
existing !IsWhitelisted branch so operator-trusted infrastructure keeps its
bypass. Allow is additionally whitelist-aware internally (elevated per-bucket
rate, exempt from the maxBuckets cap), and honours the existing
PILOT_REGISTRY_NORATELIMIT=1 escape hatch, so enforcement can be disabled
without a redeploy if it misbehaves.
2. Audit export retries were guaranteed to fail.
http.NewRequest was built once ABOVE the retry loop with
bytes.NewReader(body). The reader is drained by the first client.Do, so
attempts 2 and 3 always POSTed an empty body. The backoff loop looked
correct and could never succeed — audit batches were silently lost whenever
the first attempt failed, which is exactly when an audit trail matters. The
request is now constructed per attempt.
DEPLOY NOTE: merging does not deploy. Before rolling this to the production
rendezvous, confirm rate-limit-whitelist.json covers the operator node-farm
source IPs. Those hosts sustain a high connection count that is normal
baseline traffic, and they must be whitelisted or they will now be throttled
by a limiter that has never once been enforced against them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent bugs where fully-working code was simply never reached.
1. The per-IP rate limiter was dead code
NewRateLimiter(100, time.Second, 50_000)is constructed ataccept.go:585and fully implemented — token buckets, whitelist-aware elevated rates, stale-bucket pruning, amaxBucketscap, metrics hooks.Allow()had zero production call sites. The only methods ever invoked ona.rateLimiterwereCleanup,IsWhitelisted,SetWhitelist,WhitelistSize.So the entire per-IP tier never ran. A single client IP was bounded only by
globalBucket— which is process-wide and shared with every other client — so one IP could consume the whole registry budget and starve the fleet. The only other guard was a per-connection>500 req/scheck, trivially sidestepped by opening more connections.It also meant
pilot_ratelimit_denied_total{kind="ip"}was structurally always zero. The dashboard reported "no IP-level abuse" because nothing could increment that counter — not because none was happening. A monitoring signal that cannot fire is worse than no signal.Allowis now called on both the text and binary accept paths, inside the existing!IsWhitelistedbranch so operator-trusted infrastructure keeps its bypass.2. Audit-export retries were guaranteed to fail
http.NewRequestwas built once, above the retry loop withbytes.NewReader(body). That reader is drained by the firstclient.Do, so attempts 2 and 3 always POSTed an empty body.The backoff loop looked correct and could never succeed. Audit batches were silently lost whenever the first attempt failed — which is precisely when an audit trail matters. The request is now constructed per attempt.
Merging does not deploy. Before this reaches the production rendezvous, confirm
rate-limit-whitelist.jsoncovers the operator node-farm source IPs.Those hosts sustain a very high connection count that is normal baseline traffic. They have never once been evaluated by this limiter, because it has never run. Turning it on without whitelisting them would throttle legitimate fleet traffic.
Mitigations already in place: the whitelist bypass is honoured, and
PILOT_REGISTRY_NORATELIMIT=1disables enforcement without a redeploy if anything misfires.Tests
go build ./...,go vet, andgo test ./accept/ ./audit/all pass.Found during a codebase-wide sweep for safety mechanisms that silently fail to engage.
🤖 Generated with Claude Code