FreeBSD + carp on VMWare ESX

[Ermal Luçi]

On Mon, Jul 20, 2009 at 1:18 PM, Max Laier<max at love2party.net> wrote:
> On Monday 20 July 2009 01:25:03 Matthew Grooms wrote:
>> The other options you mention are enabled by default. This problem
>> also effects pre vSphere versions but only under certain
>> circumstances. Others claim this is only an issue when NIC teaming is
>> in use. However, I disabled this feature and still experience the
>> problem without the patch applied.
>>
>> I'll be happy to post the config when I'm back in front of the
>> computer. It's not a configuration problem. I'm not new to FreeBSD or
>> carp and have quite a bit of time invested trying to get this to work.
>>
>> Have a look at the vmware forums. A lot of folks have reported the
>> problem in great detail and, as far as I can tell, have yet to get
>> this working.
>
> There is clearly something very wrong with how the vswitch works and it's not
> really FreeBSD's job to work around these issues.  The patch you posted is
> rather intrusive and certainly not something we want in the tree.  You should
> talk to VMWare's support to fix the obvious short-comings in the vswitch
> design.

I think this is a problem that has been there since carp merge.
By looking at in_arpinput() in if_ether.c around this comment
        /*
	 * For a bridge, we want to check the address irrespective
	 * of the receive interface. (This will change slightly
	 * when we have clusters of interfaces).
	 * If the interface does not match, but the recieving interface
	 * is part of carp, we call carp_iamatch to see if this is a
	 * request for the virtual host ip.
	 * XXX: This is really ugly!
         */
Following the code the check for the bridge case is done for the
destination ip(protocol) address and for the source one while for the
carp this is done only for the destination ip(protocol) address. Which
seems wrong since the same check should apply for carp.
This would eliminate a looping back packet.
This is prevented as it is now for  target ip address but not by
source ip address since the check is missing.
The following patch against head should fix this IMO. It is untested
but seems the right solution for this.

Technically this would make code around the 'match' label discard the packet.


Index: if_ether.c
===================================================================
--- if_ether.c	(revision 195741)
+++ if_ether.c	(working copy)
@@ -522,7 +522,7 @@
 		}
 #endif
 	}
-	LIST_FOREACH(ia, INADDR_HASH(isaddr.s_addr), ia_hash)
+	LIST_FOREACH(ia, INADDR_HASH(isaddr.s_addr), ia_hash) {
 		if (((bridged && ia->ia_ifp->if_bridge != NULL) ||
 		    ia->ia_ifp == ifp) &&
 		    isaddr.s_addr == ia->ia_addr.sin_addr.s_addr) {
@@ -530,6 +530,17 @@
 			IN_IFADDR_RUNLOCK();
 			goto match;
 		}
+#ifdef DEV_CARP
+                if (ifp->if_carp != NULL &&
+                    carp_iamatch(ifp->if_carp, ia, &isaddr, &enaddr) &&
+                    isaddr.s_addr == ia->ia_addr.sin_addr.s_addr) {
+                        carp_match = 1;
+                        ifa_ref(&ia->ia_ifa);
+                        IN_IFADDR_RUNLOCK();
+                        goto match;
+                }
+#endif
+	}

 #define BDG_MEMBER_MATCHES_ARP(addr, ifp, ia)				\
   (ia->ia_ifp->if_bridge == ifp->if_softc &&				\


>
> As for your patch - you want "IF_ADDR_[UN]LOCK(ifp);" around walking the
> address list.  Don't forget to unlock before the return.
>
>> -Matthew
>>
>> On Jul 19, 2009, at 5:56 PM, Andrew Snow <andrew at modulus.org> wrote:
>> > Matthew Grooms wrote:
>> >> I was having problems running carp on VMWare ESX 4 and did a little
>> >> investigative work to determine the cause of the problem.
>> >
>> > If have tested CARP on ESX 3.5u4 successfully with a 32-bit FreeBSD
>> > guest with e1000 vNICs.
>> >
>> > As well as turning on promiscuous mode on the vSwitch, you have to
>> > enable "MAC Address changes" and "Forged transmits" as CARP requires
>> > these to work properly.
>> >
>> > Unless this is a vSphere-specific problem I must suspect your
>> > configuration as the problem.  Do you want to post your CARP config?
>> >
>> >
>> > - Andrew
>>
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>>
>> !DSPAM:4a63ab81519501271912576!
>
> --
> /"\  Best regards,                      | mlaier at freebsd.org
> \ /  Max Laier                          | ICQ #67774661
>  X   http://pf4freebsd.love2party.net/  | mlaier at EFnet
> / \  ASCII Ribbon Campaign              | Against HTML Mail and News
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>



-- 
Ermal
-------------- next part --------------
A non-text attachment was scrubbed...
Name: if_ether.c.diff
Type: application/octet-stream
Size: 1015 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20090720/77011907/if_ether.c.obj