Mellanox MT25418 performance IPOIB?

[Jason Bacon]

Hi Karl,

What type of network are you running?  DDR?  QDR?  FDR?  Switch and HCA 
models?

We're purchasing a new cluster with Mellanox ConnectX-3 FDR cards.  I'll 
report back some results with and without IPOIB once I have my hands on it.

FYI, with some tuning effort, I was able to get up to 600 megabytes/sec 
both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a 
PERC H710, ext4, RAID 6.  Might be able to do a little better with 
FreeBSD and ZFS where IB is providing 7.5gb/sec.

Regards,

     JB

On 4/17/14 4:09 AM, Karl Pielorz wrote:
>
> --On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bjørnsen 
> <stiginge at pvv.org> wrote:
>
>> You could try to use Infiniband in datagram mode, recompile without the
>> IPOIB_CM option.
>
> Ok, same boxes - I just removed IPOIB_CM out of the kernel config and 
> recompiled / installed it.
>
> iperf now consistently returns:
>
> [  3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  8.71 GBytes  7.46 Gbits/sec
>
>> The maximum MTU will then be 4092, but it might default to 2044. I run
>> OpenSM on both machines and set the MTU explicitly:
>
> Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss 
> of .5-1Gbit with iperf (obviously not 'real world') - but at leas I 
> can now try and use the stuff with real world (now it's returning more 
> than a few hundred kilobytes a second!).
>
>> The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions.
>
> Same as here then.
>
> Thanks for your email!
>
> Regards,
>
> -Karl
> _______________________________________________
> freebsd-infiniband at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to 
> "freebsd-infiniband-unsubscribe at freebsd.org"


-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Jason W. Bacon
   jwbacon at tds.net

   Circumstances don't make a man:
   They reveal him.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~