FreeBSD/mthca (Mellanox MT25218 InfiniHostEx ) [poor] Performance, Firmware Upgrade, and SDP

[David P. Discher]

On the performance side for Oded’s reply might be the answer to this :

 - https://lists.freebsd.org/pipermail/freebsd-infiniband/2014-December/000124.html

This hardware, especially some of the lower end cards like the MT25218 - are trickling into the used marked, and are really attractively priced for some small clusters (or in my case storage + Xen/Hypervisor) setups.  So, I’d love to see this code updated, even if its not upstreamed right away, I’d be happy to do some testing with some patches to -head or 10-stable.

Using both 10-stable and -head, with IPoIB - I can’t get more than 1.3 Gbps (in one direction).  Even bringing up the second port, doesn’t increase bandwidth. Even though the IPoIB is old, what I’ve notice is that I’m maxing the CPU for one of the three MSI-X IRQs (IRQ 260, the first one, which I think is MTHCA_EQ_COMP, if they are allocated in order ).

dmesg + boot verbose:

	ib_mthca0: HCA FW version 5.2.916 is old (5.3.000 is current).
	ib_mthca0: If you have problems, try updating your HCA FW.
	ib_mthca0: attempting to allocate 3 MSI-X vectors (32 supported)
	msi: routing MSI-X IRQ 260 to local APIC 0 vector 62
	msi: routing MSI-X IRQ 261 to local APIC 0 vector 63
	msi: routing MSI-X IRQ 262 to local APIC 0 vector 64
	ib_mthca0: using IRQs 260-262 for MSI-X


I assume this has nothing to really with the IPoIB part, as much as interrupt handling in the hardware driver.  Looking at the code, it looks like the MSI-X vectors are hard-coded and can’t be increased, event thought it seems like the card can support up to 32.   So, I think I know the answer, which is no, but can the number of MSI-X vectors be increased, so the interrupt load can be spread over more CPUs ?

2) Firmware upgrade for the HCA.   I got all the tools from Mellanox, however the syntax/directions for using flint says to point it to a non-exisiting /dev path in FreeBSD.  Can one use FreeBSD to flash the firmware ?  I could use some help getting the tools to talk to the card.


and 3) SDP - Sockets Direct Protocol - while I’m waiting for other things, I’d like to play with SDP.   Quick googling :
 - http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html

	> env LD_PRELOAD=/usr/lib/libibsdp.so iperf -s
	/usr/lib/libibsdp.so: Undefined symbol “program_invocation_short_name”

However, doing this in FreeBSD, gives me a missing symbol.  From searching around, program_invocation_short_name, is a gcc-libc thing. I’ve searched around, can’t seem to find an easy way to work around this.   Any suggestions ?


Thanks !


-
David P. Discher
http://davidpdischer.com/
AIM: DavidDPD | Y!M: daviddpdz


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-infiniband/attachments/20141227/8028fbed/attachment.sig>