beacon: relay budget keyed on transport source, WSS peer-map ownership, bounded WSS writes, gossip peer-set gate - #49
Closed
TeoSlayer wants to merge 1 commit into
Closed
Conversation
…p, bounded WSS writes, gossip peer-set gate L6 — dispatchRelay charged the per-source relay budget against the senderID field of the frame body, which the sender writes. One origin could therefore open an unbounded number of independent budgets by varying that field per packet. The budget is now keyed on the observed datagram endpoint, or, for frames handed over by the compat WSS bridge (which have no datagram source of their own), on the peer id the bridge verified against the node's registered key at connect time. The key is a comparable value type built without allocating, so the relay hot path keeps its per-packet cost. Keying on the full source endpoint rather than the address alone keeps co-located fleets — hosts running hundreds of daemons behind one address — on per-daemon budgets instead of collapsing them onto one. H4 — wss dropPeer removed whatever connection was registered for a node id. A node that reconnects installs a fresh connection under the same key while the previous connection's read loop is still unwinding, and that loop's cleanup then deregistered the live replacement. dropPeer now takes the *wssPeer and compare-and-deletes. M11 — wss WriteFrame bounded the write with IdleTimeout (90s). It is called from the beacon's relay workers, so a peer that stopped draining its socket could hold a worker for that whole window. Added Config.WriteTimeout (default 2s) for the per-frame deadline; the idle window keeps governing connection lifetime. A peer whose write expires is dropped, so later calls return immediately instead of re-entering the same stall. M10 — handleSync rewrites the node-id → beacon routing table the relay workers read, and accepted a sync from any source. Added an opt-in peer-set gate (SetStrictGossip / -strict-gossip / BEACON_STRICT_GOSSIP=1) restricting syncs to addresses in the peer set learned from -peers and the registry. Default off, because a beacon behind DNAT egresses from an address other than the one it advertises and enabling the check there would stop the mesh converging. Route expiry (gossipPeerTTL, swept from reapStaleNodes) is unconditional: routes published by a beacon that stops gossiping used to stand forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
perf/shard-ratelimiters(#48), which is where the sharded limiters live.Four findings from the review register: L6, H4, M11, M10.
L6 — relay budget keyed on a field of the frame body
dispatchRelaycharged the per-source relay budget againstsenderID, which is a field the sender writes into the relay header. One origin could open an unbounded number of independent budgets by varying it per packet, so the cap did not bound anything.The budget is now keyed on the transport-established origin:
relaySourceKeyis a comparable value type built without allocating (net.IP.To16allocates for a 4-byte address, so the IPv4-mapped form is written by hand) — the relay hot path keeps its per-packet cost.Choice worth reviewing: the key is the full source endpoint (address and port), not the address alone as the punch limiter uses. Address-only would put a host running hundreds of co-located daemons on a single 1000/s budget. Endpoint keying preserves per-daemon budgets while removing the in-band forgery axis, which is what the finding is about.
H4 — a replaced WSS connection's cleanup deregistered its replacement
dropPeerdeleted whatever connection was registered for a node id. When a node reconnects,handleUpgradeinstalls a fresh*wssPeerunder the same key and closes the old connection; the old read loop then unwinds and its deferred cleanup removed the live replacement. The node reads as connected for a moment and is then silently deregistered, and relays for a peer that only has a WSS path drop as unroutable.dropPeernow takes the*wssPeerand compare-and-deletes.M11 — WriteFrame bounded by the idle window
WriteFrameusedIdleTimeout(90s) as its write deadline. It is called from the beacon's relay workers, so a peer that stopped draining its socket could hold a worker for that whole window.Added
Config.WriteTimeout(default 2s) for the per-frame deadline;IdleTimeoutkeeps governing connection lifetime. A peer whose write expires is dropped, so subsequent calls return immediately rather than re-entering the same stall.M10 — unauthenticated gossip (opt-in gate + unconditional expiry)
handleSyncrewrites the node-id → beacon routing table the relay workers read, and accepted a sync from any UDP source.Two parts, deliberately split:
SetStrictGossip/-strict-gossip/BEACON_STRICT_GOSSIP=1restricts syncs to addresses in the peer set (from-peersplus registry discovery). Default off, mirroring-require-punch-token. A beacon behind DNAT egresses from an address other than the one it advertises, so enabling this by default could stop the production mesh converging. It matches on address, not port, to tolerate port rewriting.gossipPeerTTL(90s, three gossip intervals) is now swept fromreapStaleNodes.HMAC-authenticating the gossip frame was the other option in the register; it changes the frame format, so it is not on the table while deployed beacons must stay interoperable.
No wire-format changes
Nothing here alters a frame layout or a signed payload. The only behaviour changes that are on by default are the relay budget's key and gossip route expiry.
Tests
New tests, each verified failing before the corresponding fix and passing after:
TestRelayBudgetIsPerDatagramSource— one source rotatingsenderIDper packet stays inside one budget (3000 enqueued before, capped at 1000 after).TestRelayBudgetIsIndependentPerSource/TestRelayBudgetForBridgedFramesUsesPeerID— distinct origins do not share a budget.TestRelaySourceKeyDistinguishesEndpoints— key construction: IPv4/IPv6 spellings equal, port and address distinct, bridged form distinct, nil-safe.TestServer_ReplacedPeerDoesNotEvictItsReplacement— reconnect keeps the node registered and deliverable.TestServer_WriteFrameIsBoundedByWriteTimeout— a stalled peer releases the caller nearWriteTimeout, notIdleTimeout, and is dropped.TestStrictGossipRejectsSourceOutsidePeerSet,TestStrictGossipDefaultsOff,TestStrictGossipMatchesPeerOnAddressNotPort,TestStaleGossipRoutesExpire.The new WSS tests authenticate over a raw
coder/websocketconnection rather than the daemon transport — that transport reconnects on its own, so a replaced connection immediately dials back in and replaces its replacement, which makes the H4 scenario unobservable.go build ./... && go test -race -count=1 ./...green.🤖 Generated with Claude Code