Skip to content

beacon: relay budget keyed on transport source, WSS peer-map ownership, bounded WSS writes, gossip peer-set gate - #49

Closed
TeoSlayer wants to merge 1 commit into
perf/shard-ratelimitersfrom
sec/findings-beacon
Closed

beacon: relay budget keyed on transport source, WSS peer-map ownership, bounded WSS writes, gossip peer-set gate#49
TeoSlayer wants to merge 1 commit into
perf/shard-ratelimitersfrom
sec/findings-beacon

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Stacked on perf/shard-ratelimiters (#48), which is where the sharded limiters live.

Four findings from the review register: L6, H4, M11, M10.

L6 — relay budget keyed on a field of the frame body

dispatchRelay charged the per-source relay budget against senderID, which is a field the sender writes into the relay header. One origin could open an unbounded number of independent budgets by varying it per packet, so the cap did not bound anything.

The budget is now keyed on the transport-established origin:

  • UDP datagrams → the observed source endpoint.
  • Frames handed over by the compat WSS bridge → the peer id the bridge verified against the node's registered Ed25519 key during the connect challenge. Bridged frames all share one synthetic address, so that address carries no identity; the authenticated peer id is the right unit there.

relaySourceKey is a comparable value type built without allocating (net.IP.To16 allocates for a 4-byte address, so the IPv4-mapped form is written by hand) — the relay hot path keeps its per-packet cost.

Choice worth reviewing: the key is the full source endpoint (address and port), not the address alone as the punch limiter uses. Address-only would put a host running hundreds of co-located daemons on a single 1000/s budget. Endpoint keying preserves per-daemon budgets while removing the in-band forgery axis, which is what the finding is about.

H4 — a replaced WSS connection's cleanup deregistered its replacement

dropPeer deleted whatever connection was registered for a node id. When a node reconnects, handleUpgrade installs a fresh *wssPeer under the same key and closes the old connection; the old read loop then unwinds and its deferred cleanup removed the live replacement. The node reads as connected for a moment and is then silently deregistered, and relays for a peer that only has a WSS path drop as unroutable.

dropPeer now takes the *wssPeer and compare-and-deletes.

M11 — WriteFrame bounded by the idle window

WriteFrame used IdleTimeout (90s) as its write deadline. It is called from the beacon's relay workers, so a peer that stopped draining its socket could hold a worker for that whole window.

Added Config.WriteTimeout (default 2s) for the per-frame deadline; IdleTimeout keeps governing connection lifetime. A peer whose write expires is dropped, so subsequent calls return immediately rather than re-entering the same stall.

M10 — unauthenticated gossip (opt-in gate + unconditional expiry)

handleSync rewrites the node-id → beacon routing table the relay workers read, and accepted a sync from any UDP source.

Two parts, deliberately split:

  • Opt-in: SetStrictGossip / -strict-gossip / BEACON_STRICT_GOSSIP=1 restricts syncs to addresses in the peer set (from -peers plus registry discovery). Default off, mirroring -require-punch-token. A beacon behind DNAT egresses from an address other than the one it advertises, so enabling this by default could stop the production mesh converging. It matches on address, not port, to tolerate port rewriting.
  • Unconditional: routes published by a beacon that then stops gossiping used to stand forever, pointing relay traffic at a beacon that is gone. gossipPeerTTL (90s, three gossip intervals) is now swept from reapStaleNodes.

HMAC-authenticating the gossip frame was the other option in the register; it changes the frame format, so it is not on the table while deployed beacons must stay interoperable.

No wire-format changes

Nothing here alters a frame layout or a signed payload. The only behaviour changes that are on by default are the relay budget's key and gossip route expiry.

Tests

New tests, each verified failing before the corresponding fix and passing after:

  • TestRelayBudgetIsPerDatagramSource — one source rotating senderID per packet stays inside one budget (3000 enqueued before, capped at 1000 after).
  • TestRelayBudgetIsIndependentPerSource / TestRelayBudgetForBridgedFramesUsesPeerID — distinct origins do not share a budget.
  • TestRelaySourceKeyDistinguishesEndpoints — key construction: IPv4/IPv6 spellings equal, port and address distinct, bridged form distinct, nil-safe.
  • TestServer_ReplacedPeerDoesNotEvictItsReplacement — reconnect keeps the node registered and deliverable.
  • TestServer_WriteFrameIsBoundedByWriteTimeout — a stalled peer releases the caller near WriteTimeout, not IdleTimeout, and is dropped.
  • TestStrictGossipRejectsSourceOutsidePeerSet, TestStrictGossipDefaultsOff, TestStrictGossipMatchesPeerOnAddressNotPort, TestStaleGossipRoutesExpire.

The new WSS tests authenticate over a raw coder/websocket connection rather than the daemon transport — that transport reconnects on its own, so a replaced connection immediately dials back in and replaces its replacement, which makes the H4 scenario unobservable.

go build ./... && go test -race -count=1 ./... green.

🤖 Generated with Claude Code

…p, bounded WSS writes, gossip peer-set gate

L6 — dispatchRelay charged the per-source relay budget against the
senderID field of the frame body, which the sender writes. One origin
could therefore open an unbounded number of independent budgets by
varying that field per packet. The budget is now keyed on the observed
datagram endpoint, or, for frames handed over by the compat WSS bridge
(which have no datagram source of their own), on the peer id the bridge
verified against the node's registered key at connect time. The key is a
comparable value type built without allocating, so the relay hot path
keeps its per-packet cost.

Keying on the full source endpoint rather than the address alone keeps
co-located fleets — hosts running hundreds of daemons behind one address
— on per-daemon budgets instead of collapsing them onto one.

H4 — wss dropPeer removed whatever connection was registered for a node
id. A node that reconnects installs a fresh connection under the same
key while the previous connection's read loop is still unwinding, and
that loop's cleanup then deregistered the live replacement. dropPeer now
takes the *wssPeer and compare-and-deletes.

M11 — wss WriteFrame bounded the write with IdleTimeout (90s). It is
called from the beacon's relay workers, so a peer that stopped draining
its socket could hold a worker for that whole window. Added
Config.WriteTimeout (default 2s) for the per-frame deadline; the idle
window keeps governing connection lifetime. A peer whose write expires
is dropped, so later calls return immediately instead of re-entering the
same stall.

M10 — handleSync rewrites the node-id → beacon routing table the relay
workers read, and accepted a sync from any source. Added an opt-in
peer-set gate (SetStrictGossip / -strict-gossip /
BEACON_STRICT_GOSSIP=1) restricting syncs to addresses in the peer set
learned from -peers and the registry. Default off, because a beacon
behind DNAT egresses from an address other than the one it advertises
and enabling the check there would stop the mesh converging. Route
expiry (gossipPeerTTL, swept from reapStaleNodes) is unconditional:
routes published by a beacon that stops gossiping used to stand forever.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer deleted the branch perf/shard-ratelimiters July 26, 2026 14:09
@TeoSlayer TeoSlayer closed this Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants