ws: port WS transport + tree manager to Go (#135 phases 1-6) - #136
Merged
Conversation
Ports repram-mcp/src/node/ws-transport.ts to Go in internal/transport/ws/. Substrate/transient WebSocket attachments share the AttachmentMessage envelope; gossip-typed payloads serialize to the same JSON shape as the HTTP gossip endpoint, so handlers process WS and HTTP frames identically. - Connection: heartbeat, HMAC sign/verify (shares gossip.SignBody/VerifyBody), handler dispatch for messages / attachments / close / error, write serialization through writeMu. - Client: ConnectToSubstrate dials /v1/ws with a configurable timeout. - Server: Handler upgrades incoming requests and hands the wrapped Connection to an onAccept callback. - 25 unit tests at parity with ws-transport.test.ts (gossip round-trip, hello/welcome/goodbye, HMAC accept/reject paths, heartbeat send + pong-reset + missed-pong termination, dial-timeout, post-close behavior). Race-clean. Tree manager, relay, and --mcp wiring are deferred to subsequent PRs as specified in the phase plan. // ticktockbent
Ports the lifecycle half of repram-mcp/src/node/tree.ts to Go in internal/tree/. Adds the substrate/transient role model on top of the WS transport from phase 1. Substrate side: HandleHello validates capacity and registers the transient as a child, sends welcome with the current peer topology snapshot, and installs a close handler that auto-removes the child on disconnect. Capacity rejections and MaxChildren=0 both yield a goodbye-with-alternatives followed by a short delayed close. Transient side: Attach drives the hello/welcome handshake with the substrate, populates lastKnownAlts from welcome.topology (excluding self), and installs long-lived goodbye + close handlers with identity guards that prevent a stale handler from clobbering an active reattach. Reattach: three-layer loop runs as a single-flight goroutine — goodbye-supplied alts → cached welcome topology (per-attempt 5s, total 30s deadline) → seed list (per-attempt 10s) — with exponential backoff between full cycles capped at 60s. Self-skip is enforced inside tryAlternatives by address+http_port literal match (regression for #120). Stop() wakes the backoff sleep via stopCh so shutdown is prompt. Multi-subscriber refactor: ws.Connection now offers AddAttachmentHandler / AddCloseHandler returning a remove function. The TS reference uses EventEmitter add/remove semantics for the attach handshake; the prior single-slot setter API couldn't express the temporary-welcome-listener pattern. OnMessage / OnError remain single-slot since there's one application-level consumer for those. 17 tree tests at parity with the attach/detach portion of tree.test.ts (role detection, accept + welcome, child registration + auto-removal on close, capacity rejection, MaxChildren=0, attach handshake + timeout, alternatives ordering, goodbye to children, parent clear after substrate goodbye, topology cache with self exclusion, parseSeedAddress edge cases, stale-connection close guard, #120 self-skip timing assertion, ungraceful close → seed fallback, Stop() unblocks sleep). Race-clean. Relay forwarding (substrate → enclave peers) and ACK reverse-routing land in phases 3-4. // ticktockbent
Adds the data structures and helpers the cluster integration will use to wire WS frames into the existing gossip dispatch path: - RecordAckRoute / LookupAckRoute / ClearAckRoute — substrate-side table that maps a relayed PUT's messageId to the originating child connection so an enclave peer's HTTP ACK can be routed back through WS. Entries auto-evict after the configured TTL (matches REPRAM_WRITE_TIMEOUT). Stop() now cancels in-flight timers and clears the table. - BroadcastToChildren — fans a gossip message out to every attached transient whose enclave matches. Substrate calls this from the cluster's PUT handler so transients see other agents' writes in their local store. Cross-enclave traffic is dropped. - Lifecycle hardening: a stop-aware context is threaded into the reattach loop's dialer, the reattach goroutine is tracked in a WaitGroup, and Stop() waits for it. Eliminates a goroutine leak that surfaced as test pollution under -race + -count=N. - Race fix in attach(): the temporary welcome/goodbye/close handlers must be installed BEFORE hello is sent. A fast substrate could answer welcome inside SendAttachment's return path, the handlers weren't registered yet, the event was dropped, and the select below waited the full AttachTimeout. The TS reference's EventEmitter pattern hid this by allocating the listener synchronously around the same event loop tick; the Go port needs to be explicit. Found by 10x race iteration of the suite. 5 new tests: - TestRecordAndLookupAckRoute - TestAckRouteAutoEvicts (TTL eviction) - TestClearAckRoute - TestBroadcastToChildren (delivers to all attached children) - TestBroadcastToChildrenSkipsOtherEnclave (cross-enclave gate) Cluster wiring (phase 3 fan-out + phase 4 ACK reverse + phase 5 child broadcast hookup) lands in the next commit. Tests pass under go test -race -count=10. // ticktockbent
Wires the WS transport and tree manager into the running binary so
substrate nodes accept inbound attachments and transient nodes attach
outbound after HTTP bootstrap.
cluster.ClusterNode
- AckRouter / ChildBroadcaster interfaces — substrate's tree manager
satisfies both. SetAckRouter / SetChildBroadcaster installs them;
transient nodes leave them nil.
- handlePutMessage now consults ackRouter when msg.From is not in
the HTTP peer list. Routes the substrate's own immediate ACK back
over the originating WS pipe (phase 4 — substrate local store is
the first quorum vote).
- handlePutMessage also calls childBroadcaster.BroadcastToChildren
after storing, fanning replicas out to attached enclave-matched
transients (phase 5).
- handleAckMessage now falls through to ackRouter when the message
isn't ours — forwards enclave-peer ACKs back through the WS pipe
so the transient's quorum tally advances beyond the substrate's
own vote.
- WriteTimeout accessor for the WS dispatch's ACK-route eviction TTL.
tree.Manager
- RouteAck implements cluster.AckRouter — looks up the route by
messageId, writes the ACK to the child connection.
cmd/repram/main.go
- Always constructs a tree.Manager. Substrate (REPRAM_INBOUND=true)
accepts /v1/ws; transient (default) attempts outbound attach after
HTTP bootstrap (skipped when REPRAM_PEERS is empty — preserves
"single-agent local scratchpad" mode).
- /v1/ws is bound on an outer http.ServeMux that bypasses the
gorilla router's TimeoutHandler + MaxRequestSize wrappers (both
fight WebSocket Hijack / long-lived connections).
- wsHandler 404s on transient nodes — avoids leaking role to scanners.
- bindWSConnection installs a one-shot hello gate; on accept it
wires conn.OnMessage to dispatch incoming gossip into the cluster
handler and records an ACK route per relayed PUT. 30s no-hello
timeout closes silent connections.
- SetReattachCallback re-binds the parent dispatch on the new
connection after a successful reattach.
- SetSeedProvider feeds the bootstrap list into the tree-side
reattach loop.
- /v1/topology now exposes role, attached children, and parent_id
(acceptance: "substrate's HTTP topology endpoint shows transient
as attached child").
- clusterPeerer adapter so *cluster.ClusterNode satisfies tree.Peerer
via its Topology method.
cmd/repram/ws_integration_test.go (new)
- TestWSAttachHandshake — hello/welcome round-trip through the real
HTTP server stack
- TestWSPutStoresLocallyAndAcks — relay round-trip: PUT over WS,
substrate stores, ACK back over WS
- TestWSReceivePathFanout — HTTP-gossip arrival fans out to attached
transients (phase 5)
- TestWSRejectIfTransient — transients return 404 on /v1/ws
Full repo passes go test ./... and go test -race ./... — no
regressions in existing 118+ tests.
Phase 7 (24h burn-in 2.2 on real cluster infra) is deferred to a
dedicated follow-up: needs a multi-substrate + multi-transient
docker-compose setup and 24h+ of k6 workload, which doesn't belong
in a unit-test PR.
// ticktockbent
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Addresses the four "important" items from the cold sonnet review: 1. handleAckMessage closed-channel panic (internal/cluster/node.go:399-426). Splitting writesMutex to avoid deadlocking on RouteAck's WS write opened a window where two ACKs could both observe exists=true, one closes Complete, the second tries to send on a closed channel, panic. Fix: WriteOperation grows a sync.Once-guarded markComplete() helper; both the local-quorum-met path and the gossip-ACK path call markComplete() instead of close()/buffered-send. The receive side already tolerated either signal style. 2. Post-welcome dispatch race (internal/tree/manager.go, cmd/repram/main.go). The reattach callback wired OnMessage AFTER Attach returned, so a substrate's first PUT after welcome could land on a not-yet-wired onMessage and be silently dropped. Same class of race the earlier attach()-handler-install fix closed, just one level up. Fix: tree.Manager grows SetParentDispatch which is installed inside attach() BEFORE SendAttachment(hello), so the WS readLoop's serial processing guarantees OnMessage is ready for any post-welcome gossip frame. The SetReattachCallback hook stays around for non-dispatch wiring (heartbeat start, metrics scopes); main.go drops the redundant bindParentDispatch closure. 3. Enclave bypass via empty hello.Enclave (internal/tree/manager.go). BroadcastToChildren's filter skipped a child only when its enclave was non-empty AND mismatched. A hello with Enclave="" slipped through and received cross-enclave traffic. Fix: HandleHello normalizes empty to "default" (matching the gossip layer's normalization of peer enclaves), and the BroadcastToChildren filter tightens to strict inequality. New TestEmptyEnclaveNormalizedOnHello regression test exercises a non-default substrate + empty-enclave hello and asserts the broadcast is dropped. 4. Missing CHANGELOG entry (spec DoD requirement). Adds a "Restored" section under [Unreleased] describing the recovery from #125 and the four review fixes folded into this PR. Full repo passes go test -race ./... -count=5 with no new flakes. // ticktockbent
Owner
Author
|
Addressed all four important findings from the cold review in 80181ff:
Tests: |
This was referenced May 12, 2026
tree: buildAltsFromTopology drops gossipPort — cached alts can't bootstrap HTTP gossip directly
#146
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores the substrate/transient tree topology in Go so
repram --mcpnodes are real cluster participants — closing the gap from #133 that #125 introduced.Implements phases 1-6 from #135 in 4 commits:
internal/transport/ws/):Connectionwith heartbeat + HMAC,ConnectToSubstratedialer,Handlerupgrader. Wire format identical to HTTP gossip payload. 25 unit tests at parity withws-transport.test.ts.internal/tree/): substrate-sideHandleHello+ child registration + welcome with topology, transient-sideAttach, three-layer reattach loop (goodbye alts → cached topology → seed list) with self-skip + exponential backoff + identity guards. 17 tests at parity withtree.test.ts.BroadcastToChildrenon the tree manager, stop-context threading, race fix inattach()(handlers must install before hello is sent). 5 tests.cluster.ClusterNodegetsAckRouter+ChildBroadcasterinterfaces;handlePutMessageroutes ACKs back through WS and broadcasts replicas to attached transients;handleAckMessageforwards enclave-peer ACKs upstream;/v1/wsroute in main with hello-gate dispatch;--mcpmode auto-attaches outbound after HTTP bootstrap;/v1/topologyexposes role + children + parent_id. 4 integration tests.Phase status
--mcpintegration (outbound attach after bootstrap, fallback to HTTP-only on failure)Closes #133 once Phase 7 lands.
Library choice
github.com/gorilla/websocketv1.5.3. The issue recommendedcoder/websocket, but that requires Go ≥ 1.23 and this repo is pinned to 1.22; gorilla is the conservative alternative the issue explicitly allowed.Test plan
go build ./...cleango vet ./...cleango test ./...— 50+ new tests, no regressions in existing 118+go test -race ./...— entire repo race-cleango test -race ./internal/tree/... -count=10— race-stable under 10 iterations (caught + fixed a real race inattach()where temporary handlers were installed after hello was sent)--mcptransient and exercise PUT/GET round-tripOut of scope
// ticktockbent