Add SONiC containerlab (docker-sonic-vs) support#3680
Conversation
07be6d2 to
281a6ae
Compare
| SSH plays no part in configuration deployment. | ||
| * Most FRR daemons ship disabled in `/etc/frr/daemons` by default (to save resources); the | ||
| initial configuration enables the ones the configured modules need and restarts FRR once. | ||
| * MPLS/SR-MPLS labs need a one-time host prerequisite before `netlab up`: `sudo modprobe |
There was a problem hiding this comment.
We have a mechanism for this, see kmods:
Line 42 in 9acc322
Since Sonic runs FRR inside, it may be better to inherit the FRR settings where possible, rather than redefine them
| # Configure interfaces | ||
| # | ||
| config hostname {{ inventory_hostname.replace("_","-") }} | ||
| {% for l in netlab_interfaces|default([]) if l.type == 'lag' %} |
There was a problem hiding this comment.
There is a recently added mechanism for running module specific initialization during init, see VyOS templates for an example
| config portchannel add {{ l.ifname }} | ||
| fi | ||
| {% endfor %} | ||
| {% if vlans is defined %} |
There was a problem hiding this comment.
This belongs in the vlan module, you could use vlan/sonic.initial.j2 similar to vyos
| # | ||
| # And now let's configure the interfaces | ||
| # | ||
| cat >/tmp/netlab-initial.frr <<CONFIG |
There was a problem hiding this comment.
This should probably reuse FRR scripting
| @@ -0,0 +1,33 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
should be named sonic-clab.j2 I believe (i.e. dash)
| @@ -0,0 +1,75 @@ | |||
| #!/bin/bash | |||
| @@ -0,0 +1,91 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
If possible reuse FRR scripts
| vrf: [ bgpd ] | ||
| gateway: [ vrrpd ] | ||
| mpls: [ ldpd ] | ||
| clab: |
There was a problem hiding this comment.
add empty kmods: key to enable kernel module loading support
| circuit_type: true | ||
| unnumbered: { ipv4: true, ipv6: true, network: true } | ||
| import: [ static ] | ||
| mpls: |
There was a problem hiding this comment.
It's up to Ivan, but I imagine it might be better to break this up into smaller PRs: First add initial support, then add OSPF, BGP, etc.
| dellos10: | ||
| features.gateway.vrrp.version: True | ||
| # the meta device used by tests/check-integration-tests.sh has to accept the plugin | ||
| none: |
There was a problem hiding this comment.
This belongs in a separate PR
| interface {{ intf.ifname }} | ||
| vrrp {{ intf.gateway.vrrp.group }} shutdown | ||
| vrrp {{ intf.gateway.vrrp.group }} version {{ gateway.vrrp.version|default(3) }} | ||
| no vrrp {{ intf.gateway.vrrp.group }} shutdown |
There was a problem hiding this comment.
The user could have requested that the interface be shutdown, the logic should check that
| #!/bin/bash | ||
| # | ||
| # vrrp.version plugin — FRR vrrpd supports both VRRP v2 and v3. netlab | ||
| # renders this as a separate per-node config step; FRR requires the vrrp instance |
There was a problem hiding this comment.
afaik none of the existing devices supports changing VRRP versions mid-flight, it's a nice detail but also a bit overkill?
jbemmel
left a comment
jbemmel
left a comment
There was a problem hiding this comment.
Thanks a lot for your sizable contribution, it's appreciated!
Overall, I think our preference would be to reuse FRR logic and templates as much as possible, rather than reimplement the same thing in parallel. You already do that in some places like validation scripts, since Sonic is running standard FRR (I presume) the configuration should be fairly similar (if not identical)
I also think a 50 file PR is perhaps a bit too big of a chunk to accept in one go, but I'll let Ivan comment
281a6ae to
1d885ff
Compare
Add a sonic_clab device (parent: sonic) for the community docker-sonic-vs image running under containerlab, alongside the existing libvirt SONiC VM device. Since SONiC runs standard FRR, the device reuses FRR wherever the config is FRR: every routing module -- including bgp -- is delegated to the in-tree FRR templates via ansible_network_os: frr (no parallel routing templates), the single validation plugin re-exports the FRR checks (from netsim.validate.frr import *), and a clab.kmods declaration (mirroring frr.yml) lets netlab auto-load the mpls/vrf/vxlan kernel modules. What stays SONiC-specific is only what genuinely is not FRR: interface, VLAN and PortChannel objects are created through SONiC config_db (the config CLI), FRR daemons that ship disabled in docker-sonic-vs are enabled per configured module, configuration is deployed over the Ansible docker connection (docker exec), and a deterministic CONFIG_DB->kernel sync backs the modules the VS orchestration agents do not program. VLAN and PortChannel creation lives in the vlan/lag module-init hooks (vlan|lag/sonic_clab.initial.j2), pulled into initial config by extra_module_initial(). Module coverage: OSPF/OSPFv3, IS-IS, BGP (+ bgp.session/policy/originate/domain, ebgp.multihop), RIPv2, BFD, VRF, VLAN, LAG, VXLAN/EVPN (incl. symmetric IRB and multihoming), MPLS L3VPN, SR-MPLS, SRv6, gateway/VRRP, routing, tunnel.gre. No netlab-core changes.
Device bring-up and per-module smoke tests for the sonic_clab device, each verified live against a docker-sonic-vs container over the docker connection plugin. Covers initial, OSPF/OSPFv3, IS-IS (v4/v6), BGP and its plugins, RIPv2, BFD, VRF and route-leaking (v4/v6), VLAN, LAG, VXLAN/EVPN, symmetric IRB, EVPN multihoming, MPLS/SR/L3VPN, SRv6, GRE tunnels, and a files-plugin worked example. Several topologies carry native validate: sections.
Add the sonic_clab device to the platform tables (device list, configuration deployment, initial config, routing/dataplane/EVPN/layer-2/IPv6 module support), a caveats section covering the image/connection model, automatic kernel-module loading for MPLS/SR, and the SRv6 control-plane limit, a containerlab usage section in the SONiC lab notes, and a 26.07 release note.
1d885ff to
bf087fc
Compare
|
Thanks for the thorough review — the "reuse FRR logic and templates rather than reimplement in parallel" principle is exactly right, and I've taken it as the primary lens for this revision, not just the individually-flagged lines. Summary of what collapsed into FRR reuse vs what stays genuinely SONiC-specific, then the per-comment detail. Reused from FRR (no parallel implementation):
Genuinely SONiC-specific (kept, because it isn't FRR):
Per comment: kmods (initial.j2 modprobe / Module init hooks (LAG PortChannel loop, VLAN creation): Moved out of the monolithic initial template into the per-module init hooks Reuse FRR scripting (initial.j2 interface block) and vxlan template: Assessed against the FRR templates. The routing and EVPN control planes already reuse FRR entirely ( vrrp.version (separate PR / buggy shutdown logic / overkill): Agreed on all three — dropped entirely from this PR. Removed the template, the test, and the Rename Break into smaller PRs: Agreed it's a sizable change. I'll leave the split call to you/Ivan — happy to restructure into staged PRs (e.g. device+initial first, then per-module) if that's easier to review. |
2 participants
Add SONiC containerlab (docker-sonic-vs) support
Summary
netlab already ships a
sonicdevice, but only as a libvirt Vagrant box (the Azure/VMSONiC image) with
minimalsupport declared — initial configuration and BGP. This PR addsSONiC support under containerlab using the community
docker-sonic-vsimage, and expands thedeclared/validated SONiC module coverage well beyond initial + BGP.
The new module coverage is attached to a child device
sonic_clab(parent:sonic) ratherthan to the existing libvirt device, because that coverage was developed and validated against
the
docker-sonic-vscontainerlab image and its access model specifically — a single monolithiccontainer with FRR
vtyshrunning directly in it, configured over Ansible'sdockerconnectionplugin. The libvirt SONiC VM has a different access model (FRR runs in a nested
bgpsub-container reached via
docker exec bgp vtysh), so the templates and config-push path hereare not the ones exercised on the VM. Scoping the new support to the child device keeps each
device advertising only what has actually been validated on its image; it is not a claim that
the VM is incapable of these features.
sonic_clabinherits interface/loopback naming and the base BGP feature set fromsonic,adds a
clab:block, and provides its owninitial/bgp/vlan/lag/vxlan/vrrp.versiontemplates where the monolithic image requires it. Every other module is delegated to the in-tree
FRR templates via the inherited
ansible_network_os: frr.No netlab-core changes.
netsim/cli/,netsim/augment/,netsim/attributes.yml,netsim/providers/, andnetsim/modules/are byte-identical todev.mypy -p netsimreportsSuccess: no issues found in 339 source files and the transformation test suite passes
284 passed, 1 skipped, 0 failed (run in a clean venv from
requirements.txt+requirements-dev.txt).Design note
Added as a child device
sonic_clab(parent:sonic), following the existingiol/ioll2,csr/cat8000v,vjunos-router/vjunos-switchprecedent — same NOS, adifferent deployment image/shape and validated feature set, expressed as a separate device so the
new coverage attaches to the containerlab image without changing the existing libvirt
sonicdevice. Happy to rename or restructure if you'd prefer a different approach (e.g. a different
child-device name, or modelling it as a provider/image variant of
sonic) — just let me know.Connection / config-deployment model
Configuration is deployed over Ansible's built-in
dockerconnection plugin (docker exec),not
network_cli— the same pattern netlab already uses for the in-treefrrcontainerlabdevice.
docker-sonic-vsships no runningsshdand no cliconf-compatible CLI (the only SONiCcliconf Ansible ships,
dellemc.enterprise_sonic, targets Dell's licensed Management-FrameworkCLI, which the community image does not contain). Routing config is rendered as FRR
vtyshscripts;
initial/vlan/laguse SONiC's nativeconfigCLI plus a deterministicCONFIG_DB→kernel sync where the VS orchestration agents don't program the datapath.
FRR daemons that ship disabled in
/etc/frr/daemonsare enabled per configured module duringinitial configuration.
Supported modules
ospf(incl. OSPFv3,ospf.areas),isis(v4/v6),bgp(+bgp.session,bgp.policy,bgp.originate,bgp.domain,ebgp.multihop),ripv2,bfd,vrf(incl. route-leaking,per-VRF OSPFv2/OSPFv3/IS-IS/BGP),
vlan(L3 switch / SVIs),lag,vxlan,evpn(VXLAN andMPLS transport, symmetric IRB / L3VNI,
evpn.multihoming),mpls(LDP + L3VPN),sr(SR-MPLSvia IS-IS),
srv6,gateway/vrrp.version,routing(static/prefix/aspath/community/policy),and
tunnel.gre.Support level: best-effort — broad module coverage, each module carrying a device
integration topology under
tests/integration/platform/sonic_clab/; the image is user-suppliedand the device is not (yet) wired into the automated per-release integration matrix.
Verification and provenance
Live-verified for this submission against a real
docker-sonic-vs:latestcontainer oncontainerlab, config + validation over
docker exec:netlab validate(native docker-exec)netlab validate(native docker-exec)netlab validate(native docker-exec)redmaster Bridge state forwarding(CONFIG_DB→kernel bridge sync confirmed)Supported, with integration topologies included but not re-run for this submission: the
remaining modules (EVPN / symmetric-IRB /
evpn.multihoming, MPLS L3VPN, SR-MPLS, SRv6, RIPv2,BFD,
tunnel.gre, the BGP plugins, and the IPv4/IPv6 dual-stack variants) each ship a topologyunder
tests/integration/platform/sonic_clab/(several with nativevalidate:sections) that wasexercised during the device's development. They are presented as supported, not as re-verified in
this PR.
Caveats / limits
Documented in
docs/caveats.md(caveats-sonic-clab) and flagged in the platform supporttables:
docker-sonic-vs:latestimage; netlab does not ship it.sudo modprobe mpls_router mpls_iptunnel(loadsnet.mplsinto every container netns → real kernel label FIB). Flaggedwith a caveat marker on the MPLS/SR-MPLS cells in the support tables.
IS-IS LSDB and installed as real kernel
seg6localroutes, but end-to-end SRv6-OAM datapath(ping to a
uNEnd SID) is not resolved — the same open item seen with FRR/IS-IS SRv6elsewhere. Marked with a caveat in the support table.
dhcrelay/dhcp6relaybinary), real STP port-blocking (nostpddaemon), and themlag.vtepactive-active EVPN datapath (nomclagd). These are shown as ✗/absent in thetables, not claimed.
Files
netsim/devices/sonic_clab.yml— device definition (parentsonic,clab:block, features).netsim/ansible/templates/{initial,bgp,vlan,lag,vxlan}/sonic_clab.j2,netsim/extra/vrrp/version/sonic_clab.j2— device-specific templates.netsim/ansible/tasks/{deploy-config,readiness-check}/sonic_clab.yml— docker-exec deploy.netsim/validate/{,ospf,bgp,isis,route}/sonic_clab.py— FRR validation re-exports.tests/integration/platform/sonic_clab/— integration topologies + README.docs/platforms.md,docs/caveats.md,docs/labs/sonic.md,docs/release/26.07.md— docs.netsim/extra/vrrp/version/defaults.yml— registers thenonemeta-device for thevrrp.versionplugin sotests/check-integration-tests.shcan transform the10-vrrp-versiontopology under
-d none, mirroring the existingevpn.multihomingplugin. This is the onlychange outside the
sonic_clabdevice + docs.