Understanding the interplay of ipfw, vlan, and carp

[Max Laier]

Am Mi, 5.03.2008, 20:39, schrieb Freddie Cash:
> On March 4, 2008 03:25 pm Freddie Cash wrote:
>> On March 4, 2008 02:20 pm Max Laier wrote:
>> > Am Di, 4.03.2008, 22:51, schrieb Freddie Cash:
>> > ...
>> >
>> > > The lack of a "carpdev" option to directly link a carp device to an
>> > > interface (similar to "vlandev" for vlan(4)) is what's really
>> > > tripping me up.  It appears the carp(4) driver looks at all the
>> > > interfaces in the box to find one with an IP in the same subnet as
>> > > the carp IP and then uses that as the physical device.
>> >
>> > You could try the attached patch.  It adds carpdev support.  You'll
>> > have to recompile ifconfig to make use of it.
>> >
>> > This patch has some shortcomings that I wanted to address for a long
>> > time now, but never found the time to do so.  Mostly that IPv6 over
>> > CARP is broken with this patch.  Everything else is supposed to work
>> > and I'd like to hear if you experience otherwise (success stories
>> > welcome, too).  This is from back in early January, but should apply
>> > to RELENG_7 and HEAD w/o too much trouble.
>
> Patch applied cleanly to RELENG_7.0.  However, there are a few strange
> things happening now.
>
> If there are IPs on the physical devices (em0|em1) things only seem to
> work if my ipfw rules allow traffic over em0|em1.  If there are no IPs on
> em0|em1, then the ipfw rules work fine using carp0|carp1.  But it's not
> consistent.  Sometimes the counters for the em rules increment and
> sometimes the counters for the carp rules increment.

I'll look into this ... it would help if you could qualify "it's not
consistent" a bit, so that I can reproduce.

> If there are no IPs on the physical devices, and I configure rc.conf to
> put two IPs onto carp0 (one with /24, one with /32) it loses the route
> for the /24, can't find the default router, and traffic doesn't go
> through.  Manually adding the route via "route add -net
> 192.168.0.0/24 -iface carp0" allows traffic to flow again.

I see where the error is and will try to fix it.

> The rc.conf entries are:
>   cloned_interfaces="carp0 carp2"
>   ifconfig_em0="up"
>   ifconfig_em2="up"
>   ifconfig_carp0="carpdev em0 vhid 100 pass whatever  192.168.0.11/24"
>   ifconfig_carp0_alias0="192.168.0.10/32"
>   ifconfig_carp2="carpdev em2 vhid 102 pass whatever2 172.20.0/1/24"
>
> I only upgraded one of my test boxes to RELENG_7_0.  The other is still
> RELENG_6_3.  They no longer stay in sync.  Even though
> net.inet.carp.preempt=1 is set on both boxes, only the interface that I
> pull the plug on or manually down will fail-over to the other box.
>
> The ifconfig ouput on the 6.3 box will show (unplug em2 on the 6.3 box):
> carp0: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
>         inet 192.168.0.11 netmask 0xffffff00
>         inet 192.168.0.10 netmask 0xffffffff
>         carp: MASTER vhid 100 advbase 1 advskew 150
> carp2: flags=49<UP,LOOPBACK,RUNNING> mtu 1500
>         inet 172.20.0.1 netmask 0xffffff00
>         carp: BACKUP vhid 102 advbase 1 advskew 150
>
> And the ifconfig output on the 7.0 box will show:
> carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
>         ether 00:00:5e:00:01:64
>         inet 192.168.0.10 netmask 0xffffffff
>         inet 192.168.0.11 netmask 0xffffff00
>         carp: MASTER carpdev em0 vhid 100 advbase 1 advskew 0
> carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
>         ether 00:00:5e:00:01:66
>         inet 172.20.0.1 netmask 0xffffff00
>         carp: MASTER carpdev em2 vhid 102 advbase 1 advskew 0

What does "netstat -ssp carp" say?  It seems that vhid 100 doesn't sync at
all.  Might be a problem with the order of the address list.

> And, finally, if I try to create two carp devices using the same physical
> device, with IPs in the same subnet, the box crashes.  The first time, it
> locked up with the kernel panic.  Every other time it just locks the box.
>
> The commands to do this are reproducable:
>   ifconfig em0 up
>   ifconfig carp0 create
>   ifconfig carp0 carpdev em0 vhid 1 192.168.0.1/24
>   ifconfig carp1 create
>   ifconfig carp1 carpdev em0 vhid 2 192.168.0.2/24
>
> It will complain once that it can't assign the requested address.  If you
> try the ifconfig command again, the box locks up.  Might take two or
> three tries if you're lucky.  :)

This is bad - I'll look at it.

-- 
/"\  Best regards,                      | mlaier at freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier at EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News