[ecmp] Deduplicate redundant multicast next hops via per-target ECMP selection#1012
Open
zeeshanlakhani wants to merge 3 commits into
Open
[ecmp] Deduplicate redundant multicast next hops via per-target ECMP selection#1012zeeshanlakhani wants to merge 3 commits into
zeeshanlakhani wants to merge 3 commits into
Conversation
The delivery script tracked multicast events only in coarse rollups (by event, vni, port, group), which confirm that traffic moved but cannot attribute a specific group's fanout or loss to a port, next hop, or underlay address. End-to-end emulation testing profers this resolution to assert which sleds and switch ports a flow reached.
…selection The multicast forwarding loop emitted a copy to every programmed next hop. A next hop is a switch endpoint that fans out the packet to every sled (the underlay) or out the front panel (the external) behind its port groups. Redundant next hops sharing a replication target therefore reach the same destinations and deliver the flow twice, and a receiver cannot tell the duplicates apart, so we handle deduplication at Tx. This PR adds `select_nexthops`, which resolves one next hop per replication target in two passes over the forwarding table. The loop composes the two selections into each hop's effective replication because a `Both` replication hop is narrowed when chosen for one target but not the other, and skipped when chosen for neither. All next hops stay programmed for failover. Also included: Tests: add an Rx-path isolation test that drives handle_mcast_rx directly by injecting a raw Geneve-over-IPv6 multicast frame over DLPI, with no guest Tx, so a delivered packet can only have arrived via the underlay receive path. (Local) builds: we thread in TGT_BASE through the buildomat jobs so artifacts can be written to a user-owned directory, and guard the cleanup chown against re-rooting the tree when run elevated locally.
zeeshanlakhani
changed the title
FelixMcFelix
reviewed
Jul 6, 2026
FelixMcFelix
left a comment
FelixMcFelix
left a comment
Collaborator
There was a problem hiding this comment.
Thanks Zeeshan, I think this looks correct for Underlay replication. I've not looked at the added tests yet, I wanted to make sure I'm on the right page with how the external/underlay split is handled in the datapath first.
…ting
Here's the list of review follow-ups:
*Selection reworking*:
- Rewrite `select_nexthops` as a single scan that partitions eligible
hops by replication mode onto stack-held heapless vecs.
- Coalesce first approach: when any eligible hop replicates to `Both`,
hash-select among those and pin both targets to that hop, so one packet
covers the pair and the switch performs the replication. The flow splits
only when no single hop covers both targets, with each copy carrying its
target's disjoint replication instruction(s).
- Evaluate source filters once per hop during the scan, firing the
`mcast_tx_fwd_source_filtered` stat and probe at the point of
exclusion.
- Build the emit path's emissions (at most two) directly from the
selection, replacing the forwarding-map rescan and membership match.
*Reserved replication*:
- Remove the emit path's `Reserved` pass-through, which cloned the
packet only to discard it without a stat or probe.
- Reject `Reserved` at the set ioctl with EINVAL instead, as an accepted
entry serves no replication target.
*Upper Bound*:
- We add a `MAX_MULTICAST_NEXT_HOPS` (@ 4: two switches plus reconciler
headroom) and enforce it at the set ioctl via the cardinality of the
merged key set, so no per-mode partition can overextend its capacity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The multicast forwarding loop emitted a copy to every programmed next hop. A next hop is a switch endpoint that fans out the packet to every sled (the underlay) or out the front panel (the external) behind its port groups. Redundant next hops sharing a replication target therefore reach the same destinations and deliver the flow twice, and a receiver cannot tell the duplicates apart, so we handle deduplication at Tx.
This PR adds a
select_nexthopsfunction, which resolves one next hop per replication target in a single scan over the forwarding table, partitioning eligible hops by replication mode and indexing each partition with the flow's L4 hash. Selection prefers aBothreplication hop that satisfies the pair with one packet, splitting the flow into per-target copies only when no single hop covers both. The emit loop builds its emissions (at most two) directly from that selection, so the emitted replication reflects which targets chose the hop. All next hops stay programmed for (warm) failover.Also included:
Tests: added an Rx-path isolation test that drives
handle_mcast_rxdirectly by injecting a raw Geneve-over-IPv6 multicast frame over DLPI, with no guest Tx, so a delivered packet can only have arrived via the underlay receive path.(Local) builds: we thread in TGT_BASE through the buildomat jobs so artifacts can be written to a user-owned directory, and guard the cleanup chown against re-rooting the tree when run elevated locally.
On dtrace, we added per-key mcast aggregations for E2E delivery verification. The delivery script tracked multicast events only in coarse rollups (by event, vni, port, group), which confirm that traffic moved but cannot attribute a specific group's fanout or loss to a port, next hop, or underlay address. End-to-end emulation testing provides this resolution to assert which sleds and switch ports a flow reached.