Skip to content

registry: wire the per-IP rate limiter, fix audit-export retries - #113

Merged
matthew-pilot merged 1 commit into
mainfrom
fix/ratelimit-and-audit-retry
Jul 28, 2026
Merged

registry: wire the per-IP rate limiter, fix audit-export retries#113
matthew-pilot merged 1 commit into
mainfrom
fix/ratelimit-and-audit-retry

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Two independent bugs where fully-working code was simply never reached.

1. The per-IP rate limiter was dead code

NewRateLimiter(100, time.Second, 50_000) is constructed at accept.go:585 and fully implemented — token buckets, whitelist-aware elevated rates, stale-bucket pruning, a maxBuckets cap, metrics hooks.

Allow() had zero production call sites. The only methods ever invoked on a.rateLimiter were Cleanup, IsWhitelisted, SetWhitelist, WhitelistSize.

So the entire per-IP tier never ran. A single client IP was bounded only by globalBucket — which is process-wide and shared with every other client — so one IP could consume the whole registry budget and starve the fleet. The only other guard was a per-connection >500 req/s check, trivially sidestepped by opening more connections.

It also meant pilot_ratelimit_denied_total{kind="ip"} was structurally always zero. The dashboard reported "no IP-level abuse" because nothing could increment that counter — not because none was happening. A monitoring signal that cannot fire is worse than no signal.

Allow is now called on both the text and binary accept paths, inside the existing !IsWhitelisted branch so operator-trusted infrastructure keeps its bypass.

2. Audit-export retries were guaranteed to fail

http.NewRequest was built once, above the retry loop with bytes.NewReader(body). That reader is drained by the first client.Do, so attempts 2 and 3 always POSTed an empty body.

The backoff loop looked correct and could never succeed. Audit batches were silently lost whenever the first attempt failed — which is precisely when an audit trail matters. The request is now constructed per attempt.

⚠️ Deploy prerequisite — please read before rolling this out

Merging does not deploy. Before this reaches the production rendezvous, confirm rate-limit-whitelist.json covers the operator node-farm source IPs.

Those hosts sustain a very high connection count that is normal baseline traffic. They have never once been evaluated by this limiter, because it has never run. Turning it on without whitelisting them would throttle legitimate fleet traffic.

Mitigations already in place: the whitelist bypass is honoured, and PILOT_REGISTRY_NORATELIMIT=1 disables enforcement without a redeploy if anything misfires.

Tests

go build ./..., go vet, and go test ./accept/ ./audit/ all pass.

Found during a codebase-wide sweep for safety mechanisms that silently fail to engage.

🤖 Generated with Claude Code

Two independent bugs where working code was never reached.

1. The per-IP rate limiter was dead code.

NewRateLimiter(100, time.Second, 50_000) is constructed at accept.go:585 and
fully implemented — token buckets, whitelist-aware elevated rates, stale
bucket pruning, a maxBuckets cap, metrics hooks. Allow() had ZERO production
call sites. The only methods ever invoked on a.rateLimiter were Cleanup,
IsWhitelisted, SetWhitelist and WhitelistSize.

So the entire per-IP tier never ran. A single client IP was bounded only by
globalBucket, which is PROCESS-WIDE and shared with every other client, so
one IP could consume the whole registry budget and starve the fleet. The only
other guard was a per-connection >500 req/s check, trivially sidestepped by
opening more connections.

It also made pilot_ratelimit_denied_total{kind="ip"} structurally always
zero: the dashboard reported "no IP-level abuse" because nothing could
increment that counter, not because none was happening.

Allow is now called on both the text and binary accept paths, inside the
existing !IsWhitelisted branch so operator-trusted infrastructure keeps its
bypass. Allow is additionally whitelist-aware internally (elevated per-bucket
rate, exempt from the maxBuckets cap), and honours the existing
PILOT_REGISTRY_NORATELIMIT=1 escape hatch, so enforcement can be disabled
without a redeploy if it misbehaves.

2. Audit export retries were guaranteed to fail.

http.NewRequest was built once ABOVE the retry loop with
bytes.NewReader(body). The reader is drained by the first client.Do, so
attempts 2 and 3 always POSTed an empty body. The backoff loop looked
correct and could never succeed — audit batches were silently lost whenever
the first attempt failed, which is exactly when an audit trail matters. The
request is now constructed per attempt.

DEPLOY NOTE: merging does not deploy. Before rolling this to the production
rendezvous, confirm rate-limit-whitelist.json covers the operator node-farm
source IPs. Those hosts sustain a high connection count that is normal
baseline traffic, and they must be whitelisted or they will now be throttled
by a limiter that has never once been enforced against them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@matthew-pilot
matthew-pilot merged commit 111d86c into main Jul 28, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants