Fwd: using ConnectX card as Ethernet (mlxen)

[John Nielsen]

No takers on -current, anyone on -net know how to do this?

Begin forwarded message:

> From: John Nielsen <lists at jnielsen.net>
> Subject: using ConnectX card as Ethernet (mlxen)
> Date: September 24, 2012 10:37:30 AM MDT
> To: freebsd-current at freebsd.org
> 
> I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with two ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv below. The cards are connected directly to a 10GB Ethernet switch so I need to run them in "eth" mode rather than "ib". Unfortunately they come up in "ib" mode and I don't know how to change it.
> 
> The same hardware works fine under CentOS 6.3, though I need to manually set the cards to 'eth' there as well (which I do using a 'connectx_port_config script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs). Under FreeBSD I see these sysctls but I can't set them to 'eth' either via /boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib loaded:
> sys.device.mlx4_core0.mlx4_port1: ib
> sys.device.mlx4_core1.mlx4_port1: ib
> 
> Assuming mlxen is actually supported, how do I configure the card so it will attach?
> 
> 
> mlx4_core0: <mlx4_core> mem 0xdfa00000-0xdfafffff,0xdd800000-0xddffffff irq 32 at device 0.0 on pci4
> mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.2 (August 4, 2010)
> mlx4_core: Initializing mlx4_core
> mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.2 (July 2010)
> mlx4_en mlx4_core0: UDP RSS is not supported on this device.
> mlx4_core1: <mlx4_core> mem 0xdf900000-0xdf9fffff,0xdd000000-0xdd7fffff irq 42 at device 0.0 on pci7
> mlx4_core: Initializing mlx4_core
> 
> mlx4_core0 at pci0:4:0:0:	class=0x0c0600 card=0x002215b3 chip=0x673c15b3 rev=0xb0 hdr=0x00
>    vendor     = 'Mellanox Technologies'
>    device     = 'MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]'
>    class      = serial bus
> mlx4_core1 at pci0:7:0:0:	class=0x028000 card=0x001715b3 chip=0x100315b3 rev=0x00 hdr=0x00
>    vendor     = 'Mellanox Technologies'
>    device     = 'MT27500 Family [ConnectX-3]'
>    class      = network
> 
> Thanks,
> 
> JN