Production use of carp?

[Steve Polyack]

On 6/2/2011 8:14 PM, John De Boskey wrote:
> ----- Patrick Lamaiziere's Original Message -----
>> Le Thu, 2 Jun 2011 16:39:40 -0400,
>> John<jwd at SlowBlink.Com>  a ?crit :
>>
>>>     Instead of running carp on the external interfaces as below:
>>>
>>> ifconfig_cxgb0="inet 10.24.99.11 netmask 255.255.0.0"  # System 1
>>> physical ip ifconfig_cxgb0="inet 10.24.99.12 netmask 255.255.0.0"  #
>>> System 2 physcial ip ifconfig_carp1="vhid 1 pass zfscarp1 advbase 1
>>> advskew 100 10.24.99.13 netmask 255.255.0.0" # HA ip used by clients
>>>     ... we instead connect a direct cross-over cable between the two
>>> systems providing HA/Failover and use a private (backside) network:
>> I've missed this...
>>
>> As the purpose of carp is to provide a shared ip on a network, I don't
>> see why you are trying to use it on a cross-over network, between
>> only two machines? It seems useless to me.
>>
>> Regards.
> I have separate scripts which monitor the external interfaces on
> the two systems. If, for instance, one of the public ip addresses,
> 10.24.99.11 were to go down, the monitor script issues a ifconfig
> carp1 down, causing the service to shift over to partner system.
> These are actually hooked up with devd. These scripts also shift
> the virtual ip from one system to the other (as an ifconfig alias).
>
> I'm trying to avoid the controlling interface being on the external
> interfaces. As I said previously, exposing the vhid values to the
> external net (company internal net) means we have to be careful
> not to allow the same vhid twice, and to avoid the problem of the
> switch going down and both systems thinking they should be the
> master.
Just to clarify, the reason using the same VHID twice on the same 
network is bad news is because it is what is used to generate the 
virtual MAC address which the master system will respond to ARP requests 
with.  Unfortunately, the password doesn't come into play here.

Why worry about when the switch goes down?  Shouldn't that eliminate any 
traffic which could cause changes to the exported filesystems?  What if 
somebody knocks out your crossover cable and you have both systems 
thinking they are master AND are accessible (in some sense, since the 
MAC is going to flap between switchports).
> In general, I attach exported filesystem services to different
> carp interfaces. I load balance them between the two HA servers.
> For instance, /vol/data1 is on carp1, /vol/data2 is on carp2.
> Under normal circumstance, /vol/data1 is "owned" by system A,
> and /vol/data2 is owned by sysem B.  Issuing 'ifconfig carp1 down'
> on system A causes the export of /vol/data1 to shift over to
> system B at which point maintanence can be done on system A.
> The only problem is taking down system A causes the carp interfaces
> on system B to go down/up a few times.
>
> Does that help a bit?
I think your problem lies with the crossover cable.  When the crossover 
interface goes down (as in link state), the CARP interface on the system 
which is still online is going to flap.  Even if you do not specify 
'carpdev cxgb0', the system will locate the proper physical interface to 
associate your CARP interface with.  If this is really what you want, 
you may have to hack the kernel to have CARP ignore link state changes 
on the associated physical interface.

I would suggest just using CARP on the interface which connect to the 
switches.  Just as each set of two storage units will likely share a 
Virtual IP which is unique to them, they should also share a unique 
VHID.  Yes, you only have 1-255, but since the VHID only controls the 
virtual MAC, reusing a VHID in a different VLAN should not matter 
(provided you are using a different password or the VLANs are also in 
different multicast domains).  Perhaps you can create a VLAN local to 
the switch just for this purpose?  Setup a vlan(4) interface on each 
system just for this purpose and attach the CARP device to it.

Overall, I think you're trying to use CARP to accomplish something other 
than what it was designed for.  Why not just run a ping on the crossover 
interface every second and use that to trigger your switchovers?  You 
could get away with a pretty low timeout.  Add in some kind of way to 
force a maintenance-mode on a box and I think you'd have a solution that 
does what you want without the pitfalls you are seeing.

- Steve