randomized CARP ip alias order -> breaks CARP (incorrect hash)

[Wouter de Jong]

Hi,

(Sorry for double posting this to freebsd-questions before, but I think this 
place might be more appropriate).

We have 2 FreeBSD machines running as a firewall in a CARP+pf+pfsync setup.
Worked great, however ..... last Friday I noticed something weird.

I had to reboot the master machine, and when it came back ...
one of the CARP addresses no longer worked.

Looking in the logs, I got carp4: incorrect hash

And looking at the carp interface .... both machines were running MASTER for
this interface.

Looking closer, I noticed my primary machine had this configuration :

carp4: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 213.206.xx.62 netmask 0xfffffff0
        inet 213.206.xx.49 netmask 0xfffffff0
        carp: MASTER vhid 4 advbase 1 advskew 100

and my secondary :

carp4: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 213.206.xx.49 netmask 0xfffffff0
        inet 213.206.xx.62 netmask 0xfffffff0
        carp: MASTER vhid 4 advbase 1 advskew 100

It swapped the carp alias alias (213.206.xx.62) to be the first address on
the interface.
This was the only interface it happened.

The config :

primary:
##############################
defaultrouter="213.206.yy.193"
hostname="fw01.xxx.yyy"

cloned_interfaces="carp0 carp1 carp2 carp3 carp4 carp5 carp6 carp7 carp8
carp9 carp10 carp11 carp12 carp13 carp14 carp15 carp16 carp17 carp18 carp19
carp20 carp21 carp22 carp23 carp24 carp25 carp26 carp27 carp28"

ifconfig_bge0="inet 213.206.yy.194  netmask 255.255.255.240"
ifconfig_bge1="inet 213.206.xx.2  netmask 255.255.255.240"
ifconfig_bge1_alias0="inet 213.206.xx.18  netmask 255.255.255.240"
ifconfig_bge1_alias1="inet 213.206.xx.34  netmask 255.255.255.240"
ifconfig_bge1_alias2="inet 213.206.xx.50  netmask 255.255.255.240"
ifconfig_bge1_alias3="inet 213.206.xx.66  netmask 255.255.255.240"
ifconfig_bge1_alias4="inet 213.206.xx.82  netmask 255.255.255.240"
<etc,etc>
ifconfig_carp0="vhid 255 pass blubVIP0255 213.206.yy.206/28"
ifconfig_carp1="vhid 1 pass blubVIP0001 213.206.xx.1/28"
ifconfig_carp2="vhid 2 pass blubVIP0002 213.206.xx.17/28"
ifconfig_carp2_alias0="vhid 2 pass blubVIP0002 213.206.xx.30/28"
ifconfig_carp3="vhid 3 pass blubVIP0003 213.206.xx.33/28"
ifconfig_carp4="vhid 4 pass blubVIP0004 213.206.xx.49/28"
ifconfig_carp4_alias0="vhid 4 pass blubVIP0004 213.206.xx.62/28"
ifconfig_carp5="vhid 5 pass blubVIP0005 213.206.xx.65/28"
ifconfig_carp6="vhid 6 pass blubVIP0006 213.206.xx.81/28"
##############################

secondary:
##############################
defaultrouter="213.206.yy.193"
hostname="fw02.xxx.yyy"

cloned_interfaces="carp0 carp1 carp2 carp3 carp4 carp5 carp6 carp7 carp8
carp9 carp10 carp11 carp12 carp13 carp14 carp15 carp16 carp17 carp18 carp19
carp20 carp21 carp22 carp23 carp24 carp25 carp26 carp27 carp28"

ifconfig_bge0="inet 213.206.yy.195  netmask 255.255.255.240"
ifconfig_bge1="inet 213.206.xx.3  netmask 255.255.255.240"
ifconfig_bge1_alias0="inet 213.206.xx.19  netmask 255.255.255.240"
ifconfig_bge1_alias1="inet 213.206.xx.35  netmask 255.255.255.240"
ifconfig_bge1_alias2="inet 213.206.xx.51  netmask 255.255.255.240"
ifconfig_bge1_alias3="inet 213.206.xx.67  netmask 255.255.255.240"
ifconfig_bge1_alias4="inet 213.206.xx.83  netmask 255.255.255.240"
<etc,etc>
ifconfig_carp0="vhid 255 advskew 100 pass blubVIP0255 213.206.yy.206/28"
ifconfig_carp1="vhid 1 advskew 100 pass blubVIP0001 213.206.xx.1/28"
ifconfig_carp2="vhid 2 advskew 100 pass blubVIP0002 213.206.xx.17/28"
ifconfig_carp2_alias0="vhid 2 advskew 100 pass blubVIP0002 213.206.xx.30/28"
ifconfig_carp3="vhid 3 advskew 100 pass blubVIP0003 213.206.xx.33/28"
ifconfig_carp4="vhid 4 advskew 100 pass blubVIP0004 213.206.xx.49/28"
ifconfig_carp4_alias0="vhid 4 advskew 100 pass blubVIP0004 213.206.xx.62/28"
ifconfig_carp5="vhid 5 advskew 100 pass blubVIP0005 213.206.xx.65/28"
ifconfig_carp6="vhid 6 advskew 100 pass blubVIP0006 213.206.xx.81/28"
##############################

After rebooting the secondary, it still gave me incorrect hash.
But, it gave me the same thing on carp2 now. ... however, here the secondary
had the carp2_alias0 listed as first,
where as the primary had the carp2 as first, and the carp2_alias0 as second
address.

How can this ever happen ? And what can I do to (manually) prevent this ?

Now I'm redundant .... but I must pray that the addresses will come up in
the same order.

Never had this issue on FreeBSD 6.x, but now I'm running FreeBSD 
7.0-RELEASE.

Help ! :)

At lease one person on freebsd-questions observes the same thing...(both 
with 6.x and 7.x)


Kind regards,

Wouter de Jong
The Netherlands