CARP vs. if_bridge

[Boris Kochergin]

Boris Kochergin wrote:
> Max Laier wrote:
>> On Thursday 18 February 2010 18:02:55 Boris Kochergin wrote:
>>  
>>> Ahoy. I'm seeing what appears to be erroneous interaction between CARP
>>> and if_bridge on multiple machines with a variety of Ethernet
>>> controllers and architectures. I've observed it on 7.2-R and 8.0-R. The
>>> test setup is simple enough:
>>>
>>> CARP master:
>>>
>>> FreeBSD t30 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 #5: Sun Feb 14
>>> 20:22:41 EST 2010     root at t30:/usr/obj/usr/src/sys/T30  i386
>>>
>>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>>         options=3<RXCSUM,TXCSUM>
>>>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
>>>         inet6 ::1 prefixlen 128
>>>         inet 127.0.0.1 netmask 0xff000000
>>> dc0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
>>> metric 0
>>> mtu 1500
>>>         options=8<VLAN_MTU>
>>>         ether 00:04:5a:a8:e0:bf
>>>         inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
>>>         media: Ethernet autoselect (100baseTX <full-duplex>)
>>>         status: active
>>> carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
>>>         inet 192.168.0.1 netmask 0xffffff00
>>>         carp: MASTER vhid 1 advbase 1 advskew 0
>>>
>>> CARP backup:
>>>
>>> FreeBSD ultra5 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Thu Feb 18 15:19:39
>>> UTC 2010     boris at ultra5:/usr/obj/usr/src/sys/GENERIC.carp  sparc64
>>>
>>> hme0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>>         options=b<RXCSUM,TXCSUM,VLAN_MTU>
>>>         ether 08:00:20:f5:65:d4
>>>         media: Ethernet autoselect
>>> xl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
>>> metric 0
>>> mtu 1500
>>>         options=9<RXCSUM,VLAN_MTU>
>>>         ether 00:01:03:2c:06:6d
>>>         inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
>>>         media: Ethernet autoselect (100baseTX <full-duplex>)
>>>         status: active
>>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>>         options=3<RXCSUM,TXCSUM>
>>>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
>>>         inet6 ::1 prefixlen 128
>>>         inet 127.0.0.1 netmask 0xff000000
>>> carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
>>>         inet 192.168.0.1 netmask 0xffffff00
>>>         carp: MASTER vhid 1 advbase 1 advskew 100
>>> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 
>>> mtu
>>> 1500
>>>         ether 3a:e6:09:2d:da:bc
>>>         id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>>>         maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
>>>         root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>>>         member: xl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>>>                 ifmaxaddr 0 port 2 priority 128 path cost 200000
>>>         member: hme0 flags=8<SPAN>
>>>                 ifmaxaddr 0 port 1 priority 128 path cost 200000
>>>
>>> In summary, I have a basic CARP configuration and, on the backup CARP
>>> machine, a bridge with the CARP device's physical interface in it. The
>>> purpose of this setup is the ability to monitor traffic passing through
>>> that interface using another machine. If the master CARP machine is
>>> disconnected from the network, the CARP interface on the backup machine
>>> correctly changes to the MASTER state, but does not act on traffic 
>>> bound
>>> for the shared IP address--192.168.0.1. tcpdump shows the traffic 
>>> coming
>>> in on the correct physical interface, but it is never replied to, 
>>> or, in
>>> the case of routing, forwarded. Removing xl0 from the bridge on the
>>> backup machine instantly fixes this, and the shared IP address behaves
>>> as expected. Adding xl0 back to the bridge while the backup CARP
>>> interface is in the MASTER state keeps things running correctly, so the
>>> problem is only observed when xl0 is part of the bridge during the CARP
>>> transition from BACKUP to MASTER. Thoughts?
>>>     
>>
>> I assume the bridge filters out the traffic as it thinks the 
>> destination is elsewhere (it has previously seen ARPs from the other 
>> MASTER entering via xl0).  It shouldn't do that, but that's a 
>> different story.  You can try to force edge or ptp status on xl0, not 
>> sure if this does the trick, but it's worth a try.
>>
>> Regards,
>>   Max
>>   
> Sure. No go, though, I'm afraid. It's not an operational show-stopper 
> for me, at least. In the worst case, I can always hack up a PCAP 
> program to copy the frames around in user space.
>
> -Boris
For the archives, in the off chance that someone else encounters this:

http://acm.poly.edu/wiki/Userspace_SPAN_Port

-Boris