relay: merge multiple upstream publishers per track (§9.5)#20
Merged
Conversation
Implements full §9.5 multiple-publishers-per-track. The relay can now
ingest a track from N upstreams — redundant origins, a switchover
overlap, or several discovered relays — and fan them into a single
clean outbound subgroup stream per subscriber, deduplicating objects so
each is delivered exactly once and continuing seamlessly when one origin
drops out.
Core change: outbound subgroup writers move from per-inbound-stream
(local to runFanout) to a per-(GroupID, SubgroupID) shared, refcounted
set on TrackEntry (SharedSubgroup / Acquire/ReleaseSubgroup). N inbound
streams for the same Subgroup share ONE writer per subscriber, so the
relay never splits a Subgroup across streams (§2.2).
Deduplication uses a persistent, group-windowed ledger on the
TrackEntry (ClaimDelivered), keyed by {GroupID, ObjectID} per §2.1. It
lives on the entry rather than on a per-Subgroup structure so redundant
streams that do not temporally overlap (one origin's subgroup FINs
before the peer's arrives) still dedup, and it is independent of the
size-bounded Object Cache so a peer lagging by more than the cache
capacity dedups correctly. Memory is bounded to the most recent
deliveredGroupWindow groups.
Teardown aggregates FIN-vs-reset across contributors: the merged stream
FINs if any upstream completed cleanly and resets only if all reset, so
one origin failing does not disturb subscribers.
handler_subscribe now subscribes to every matching publisher (all local
publishers + all remotes via upstreamPool.resolveAll) with source dedup,
instead of stopping at the first.
Tests: redundant-origin dedup, mid-stream failover continuity, disjoint
fan-in union, dedup surviving cache eviction, two-remote cross-relay
fan-in, and a ClaimDelivered unit test. Full suite and -race are green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements full §9.5 multiple-publishers-per-track in the relay. A track can now be ingested from N upstreams — redundant origins, a graceful switchover overlap, or several discovered relays — and merged into a single clean outbound subgroup stream per subscriber, with each object deduplicated and seamless continuity when one origin drops out.
Previously the relay assumed one usable upstream per track; two publishers of the same
(Namespace, Name)produced two outbound streams per subscriber (a §2.2 violation) and double-delivered every object.How
runFanout) to a per-(GroupID, SubgroupID)shared, refcounted set onTrackEntry(SharedSubgroup+Acquire/ReleaseSubgroup). All inbound streams for one Subgroup share one writer per subscriber, so a Subgroup is never split across streams (§2.2).TrackEntry.ClaimDeliveredkeyed by{GroupID, ObjectID}(§2.1 — SubgroupID is not part of object identity). It lives on the entry (not a per-Subgroup structure) so redundant streams that don't temporally overlap still dedup, and it's independent of the size-bounded Object Cache so a peer lagging by more than the cache capacity dedups correctly. Memory is bounded to the most recentdeliveredGroupWindow(32) groups.handler_subscribenow subscribes to every matching publisher (all local + all remotes viaupstreamPool.resolveAll) with source dedup, instead of stopping at the first.Spec conformance
Verified against
draft-ietf-moq-transport-18: object identity{GroupID, ObjectID}(§2.1, Subgroup ID excluded); one-stream-per-Subgroup (§2.2) with its reset/out-of-order carve-out covering failover.Tests
ClaimDeliveredunit test (dedup + window pruning)go build ./...,go test ./..., andgo test -race ./pkg/relay/...are green;gofmt/go vet/modernizeclean.Notes / residual scope
subscribeUpstreamnow does a Discovery lookup + sequential remote dials on every SUBSCRIBE even when a local publisher already succeeded — a latency cost on cross-relay deployments, left as a follow-up (could be parallelized / short-circuited).🤖 Generated with Claude Code