OPNsense 26.4.1 Business Edition: Bridge interface blocks L2-forward for IP (while ARP is working)

[Sumpfdotter]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guidelines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I am migrating an existing bridge-based VLAN setup from an older OPNsense installation.

The setup follows the documentation for a layer-2-bridge as described here: https://docs.opnsense.org/manual/how-tos/lan_bridge.html

Given a bridge consisting of two VLAN interfaces:

• vlan0.12.30 (parent: lagg0(igc1,igc2))
• vlan1.0.30 (parent: ax0)

The bridge itself has an IPv4 address for routing.

The devices behind both vlans can communicate with all other subnets and vice versa (Layer 3, routing involved). But the devices behind vlan0.12.30 cannot communicate with the devices in the same subnet behind vlan1.0.30 and vice vesa (Layer 2, bridging use case).

The surprising part is that the bridge behaves like a transparent Layer-2 bridge for ARP, but requires an explicit Layer-3 pass rule for forwarding IP traffic between bridge members. This is unexpected because no routing is involved.

To Reproduce

Bridge filtering sysctls are set:

net.link.bridge.pfil_bridge=1
net.link.bridge.pfil_member=0
net.link.bridge.pfil_local_phys=0

Steps to reproduce the behavior:

1. Setup a bridge as described
2. Connect one client device to vlan0.12.30
3. Connect another client device to vlan1.0.30
4. arping from device 1 to device 2 --> passed
5. ping from device 1 to device 2 --> blocked (default deny / state violation)

Expected behavior

Following the given documentation, I expected a layer-2-bridging, not only for ARP but also for IP. IIRC this was the behavior in the older OPNSense setup, what I unfortunately cannot test anymore.

Describe alternatives you considered

Workaround

Adding the following firewall rule on the BRIDGE interface immediately restores forwarding:

• Action: Pass
• Interface: bridge30
• Source: bridge30 net
• Destination: bridge30 net
• Resulting pf rule: pass in log quick on bridge30 inet from (bridge30:network) to (bridge30:network) keep state

Relevant log files

Packet captures show:

• lagg0: ICMP Echo Request visible
• vlan0.12.30: ICMP Echo Request visible
• bridge30: ICMP Echo Request visible
• vlan1.0.30: NO ICMP packet
• ax0: NO ICMP packet

Additional context

Has bridge filtering semantics changed recently?

If this is intentional, I think the documentation (e.g. https://docs.opnsense.org/manual/how-tos/lan_bridge.html) should explicitly mention that bridge-local traffic must be permitted by firewall rules on the bridge interface when pfil_bridge=1.

If it is not intentional, this may be a regression.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 26.4.1 Business Edition
FreeBSD 14.3-RELEASE-p15
DEC740

[AdSchellevis]

Although I'm not using bridges very often, I wouldn't expect routing to make the difference to be honest, if the traffic passes the bridge, it needs a rule, the tunables merely choose if the legs and/or the bridge are passed through the bridge.

Our documentation is a bit limited on this subject, the supplied book with the license contains more context.

[Sumpfdotter]

Thank you, @AdSchellevis. Routing is possible, because for routing the paths are "otherIf -> FW -> bridge" and "bridge -> FW -> otherIf". For these paths one obviously has to create rules of type "allow from sth in net1 to sth in net2" and/or vice versa. But here we need to add the rule "allow from net1 to net1" to make the bridge bridging IP. This is quite a weird rule, semantically. :-)

The book you are referring provides indead at least a very little bit more context. Thank you! So it states that "in" rules are to be used and it at least mentions the option "net.link.bridge.pfil_onlyip" which might explain the "ARP pass through while IP is blocked".

My question is: Is it really intended to have the rule "allow from net1 to net1" on the bridge to make the bridge bridging IP? This rule sounds very weird, to be honest. :-) If yes, my kind request would be to add a corresponding note to the How-to. Second: Can sb rule out the possibility that such a rule was created automatically by OPNSense in older versions?

Noteworthy: One has to create such rules on each bridge device individually and some write you may want to have separate rules for IPv4 and IPv6 for robustness purposes. So you cannot solve this with any generic approach like using firewall groups. It is error prone to create such rules for each bridge individually. The user ends up in a rule setup like this:

Not optimal, but working in case it is intended like this (see above: Weird rule and although required currently not explicitly documented in the How-To). Thank you much!

[AdSchellevis]

My question is: Is it really intended to have the rule "allow from net1 to net1" on the bridge to make the bridge bridging IP? This rule sounds very weird, to be honest. :-) If yes, my kind request would be to add a corresponding note to the How-to. Second: Can sb rule out the possibility that such a rule was created automatically by OPNSense in older versions?

To me it sounds quite logical because traffic is traveling interfaces, when not planning to filter (at all), you likely just have to set pfil_bridge to 0. From the filter's perspective, "routing" isn't really a thing, there's just traffic or there isn't.

I'm open for additions to the documentation, but it doesn't have a very high priority on our end at the moment, bigger fish to fry and the documentation itself doesn't look wrong, just not very extensive.

[Sumpfdotter]

Thank you much @AdSchellevis, unfortunately your proposal will not work out. Setting pfil_bridge to 0 is not an option in case you want to have a logical interface on a bridge device, that's why your HowTo also states to turn it on.

Is it expected to have port isolation by default or not to have it? Both would be consistent and somehow "natural" as you mentioned. But both is not the case here.

When you approach from a user's perspective, a bridge per default usually does not perform port isolation. This is also what you get from that HowTo: no port isolation. No port isolation? There is one exception: IP protocol is isolated. So I cannot find a natural logic in this behavior. For me this is something that it's impossible to anticipate and needs to be explained to me. Not knowing this inconsistency drove me into hours of dumps and analysis, and even the hour I spend for this report is not for me - it's for you to improve your product and for others running into the same issue.

I can see a lot of cool new features like the new rules view, so for sure it's your decision where to invest.

[fichtner]

Just a question because it’s not clear from the first post: are you assigning addresses to the VLAN interfaces as well?

[Sumpfdotter]

Hi @fichtner, thanks for asking.

Correction: Sorry, I was not reading correctly: NO, the VLAN interfaces are active in oder to make them a bridge member, but they do not have IP adresses etc.

In this use case, the bridge wants to be configured to act like a "software switch" and the firewall itself is connected to it as router (with IP address, typically acting as gateway, DHCP server, DNS resolver etc.) for the other participants:

So with the tunables set according to the How-To plus the firewall rule "allow from MyNet to MyNet" it looks like being complete (ARP + IP).

[Sumpfdotter]

Hi @fichtner.

Correction: Sorry, I was not reading correctly: NO, the VLAN interfaces are active in oder to make them a bridge member, but they do not have IP addresses etc.

[fichtner]

Apparently net.link.bridge.pfil_onlyip controls the non-layer 2 behaviour. Bridge to bridge traffic is a bit of a hit and miss as it aligns with "input interface is bridge" and "output interface is bridge" but still requires explicit pass policies. The only way around this is disabling net.link.bridge.pfil_bridge.

So, conceptually, you cannot emulate a separate switch behaviour that only filters on traffic escaping the bridge and automatically allow all cross talk.

The cleanest way would still be to use net.link.bridge.pfil_member but since both VLANs appear to be the same in subnetting that's also a problem due to spoofing concerns.

Long story short: I don't think this will work and a switch is much better suited to deal with such basic network requirements.

Cheers,
Franco

[Sumpfdotter]

I agree that an external hardware switch is often the preferred choice for high-performance networking. However, a software bridge can be a highly efficient alternative in situations with lower throughput requirements, or when saving physical cabling is a priority.

In my environment, this setup is now running flawlessly. From what I observed, the only real "stumbling block" is how PF processes local L2 IP traffic when member filtering is disabled (pfil_member=0, pfil_bridge=1). Because of the default-deny policy, local traffic between the ports is intercepted by PF on the logical bridge0 interface and dropped, even though the hosts are in the same subnet.

Once I added a firewall rule from <bridge_net> to <bridge_net> directly to the bridge interface, everything started working exactly as expected.

I found indicators in the ecosystem that support this observation:

• The official FreeBSD Filtering Bridges Article showcases this behavior by using a broad pass in via LAN to any rule, which naturally covers the local bridge subnet.
• The official OPNsense VXLAN Bridge Documentation (Step 4) explicitly highlights this necessity: "These rules must allow LAN to LAN traffic, e.g., source 192.168.10.0/24 to destination 192.168.10.0/24."

Since automatic rules are out of the question, would you consider updating the standard OPNsense LAN Bridge How-To to mention this behavior? Just adding a small note or an example rule about this requirement would raise awareness and help other users successfully deploy a standard bridge without hitting this exact same wall.

[Monviech]

These two are guides I wrote:

https://docs.opnsense.org/manual/how-tos/transparent_bridge.html#add-firewall-rules
https://docs.opnsense.org/manual/how-tos/vxlan_bridge.html#bridging-vxlan-and-lan

I think what you need is comparable to the transparent filtering bridge, which is also unnumbered but still needs firewall rules.

The LAN bridge documentation looks a bit outdated, feel free to offer a PR to add a firewall rules hint to it:
https://docs.opnsense.org/manual/how-tos/lan_bridge.html
https://github.com/opnsense/docs

[Sumpfdotter]

Thanx for permission. Done: opnsense/docs#890