Create bond on Infiniband ports

[Garrett Cooper]

On Tue, May 28, 2013 at 4:35 AM, Alex Liptsin <alexl at mellanox.com> wrote:

> Hi.
>
> I use FreeBSD 9.1 with OFED compiled on it.
>
> There is a Mellanox adapter:
>
> [root at qa-h-vrt-030-006 ~]# pciconf -lv |grep mlx4 -A 3
> mlx4_core0 at pci0:0:5:0:  class=0x028000 card=0x005015b3 chip=0x100315b3
> rev=0x00 hdr=0x00
>     vendor     = 'Mellanox Technologies'
>     device     = 'MT27500 Family [ConnectX-3]'
>     class      = network
>
> I want to create Bond on the two ports (ib0 and ib1) of this device:
>
> [root at qa-h-vrt-030-006 ~]# ifconfig
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>
> options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
>         ether 00:50:56:23:1e:06
>         inet6 fe80::250:56ff:fe23:1e06%em0 prefixlen 64 scopeid 0x2
>         inet 10.195.30.6 netmask 0xffff0000 broadcast 10.195.255.255
>         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
>         media: Ethernet autoselect (1000baseT <full-duplex>)
>         status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>         options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>         inet6 ::1 prefixlen 128
>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
>         inet 127.0.0.1 netmask 0xff000000
>         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
> ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
>         options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
>         lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.51
>         inet 11.195.30.1 netmask 0xffff0000 broadcast 11.195.255.255
>         inet6 fe80::250:56ff:fe23:1e06%ib0 prefixlen 64 scopeid 0x4
>         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
> ib1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
>         options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
>         lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.0.2.c9.0.1.0.d0.52
>         inet 12.195.30.1 netmask 0xffff0000 broadcast 12.195.255.255
>         inet6 fe80::250:56ff:fe23:1e06%ib1 prefixlen 64 scopeid 0x5
>         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
>
>
> 1.      Is there "bond" in FreeBSD or its "Lagg"?
>

It's lagg. Setting up bonded interfaces is different (Anthony C. or someone
else can provide more details about this). failover mode via lagg is
completely different from failover mode with IB IIRC.


> 2.     I tried to create this way, but failed:
>
> [root at qa-h-vrt-030-006 ~]# ifconfig lagg0 create
>
> root at qa-h-vrt-031-005 conf]# ifconfig lagg0 laggproto failover laggport
> ib0 laggport ib1
>
> ifconfig: SIOCSLAGGPORT: Protocol not supported
>
>             Any ideas?
> Is it supported on Infiniband ports?
>

The media setting pieces are look to only be supported with mlx4 in en mode:

1076 static int mlx4_en_ioctl(struct ifnet *dev, u_long command, caddr_t
data)
1077 {
...
1115         case SIOCGIFMEDIA:
1116                 error = ifmedia_ioctl(dev, ifr, &priv->media, command);
1117                 break;

If I remember correctly, IB ports with mlx4 default to ib mode, not en mode
(it also helps to have the right drivers loaded for this).

Cheers,
-Garrett