Kernel modules
Jason Bacon
bacon4000 at gmail.com
Fri Apr 12 12:52:54 UTC 2019
On 2019-04-12 07:39, Jason Bacon wrote:
> On 2019-03-18 10:54, Jason Bacon wrote:
>> On 3/18/19 10:08 AM, Jason Bacon wrote:
>>> On 3/18/19 9:10 AM, Hans Petter Selasky wrote:
>>>> On 3/18/19 2:56 PM, Jason Bacon wrote:
>>>>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote:
>>>>>> On 3/18/19 1:32 PM, Jason Bacon wrote:
>>>>>>>
>>>>>>> Anyone using the kernel modules lately?
>>>>>>>
>>>>>>> After building per the instructions on the wiki
>>>>>>> (https://wiki.freebsd.org/InfiniBand):
>>>>>>>
>>>>>>> <<<ROOT at unixdev.ceas>>> /home/bacon 1005 # kldload ibcore
>>>>>>> kldload: an error occurred while loading module ibcore. Please
>>>>>>> check dmesg(8) for more details.
>>>>>>>
>>>>>>> dmesg:
>>>>>>>
>>>>>>> link_elf_obj: symbol if_index undefined
>>>>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> You built the KMODs out of the kernel? Try adding
>>>>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build.
>>>>>>
>>>>>> --HPS
>>>>>
>>>>> I used the script for building kernel modules (which I wrote and
>>>>> posted at https://wiki.freebsd.org/InfiniBand a few years ago,
>>>>> based on advice from a Mellanox developer at the time). This will
>>>>> show exactly what I did.
>>>>>
>>>>
>>>> Can you try:
>>>>
>>>> make DEBUG_FLAGS="-DVIMAGE=1"
>>>>
>>>> instead of
>>>>
>>>> make
>>>>
>>>> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike
>>>> FreeBSD-11.
>>>>
>>>> --HPS
>>>>
>>>
>>> That worked, thanks. I'll update the wiki after some testing.
>>>
>>> ...
>>> objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug
>>> objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug
>>> ipoib.ko.full ipoib.ko
>>> install -T release -o root -g wheel -m 555 ipoib.ko /boot/modules/
>>> kldxref /boot/modules
>>> Id Refs Address Size Name
>>> 1 72 0xffffffff80200000 243cd00 kernel
>>> 2 1 0xffffffff8263e000 8b50 ng_ubt.ko
>>> 3 2 0xffffffff82647000 12dd0 ng_hci.ko
>>> 4 2 0xffffffff8265a000 2d00 ng_bluetooth.ko
>>> 5 3 0xffffffff8265d000 175d8 netgraph.ko
>>> 7 5 0xffffffff82688000 279b0 linuxkpi.ko
>>> 9 1 0xffffffff8274d000 3a9a10 zfs.ko
>>> 10 2 0xffffffff82af7000 a4f0 opensolaris.ko
>>> 11 1 0xffffffff82b02000 20af0 firewire.ko
>>> 12 1 0xffffffff82b23000 bf70 cuse.ko
>>> 13 2 0xffffffff82b2f000 665e0 mlx4.ko
>>> 15 1 0xffffffff82bd0000 5fb8 uplcom.ko
>>> 16 2 0xffffffff82bd6000 ab88 ucom.ko
>>> 17 1 0xffffffff82e21000 1a20 fdescfs.ko
>>> 18 1 0xffffffff82e23000 490c linprocfs.ko
>>> 19 3 0xffffffff82e28000 2e28 linux_common.ko
>>> 20 1 0xffffffff82e2b000 81f0 tmpfs.ko
>>> 21 1 0xffffffff82e34000 1800 uhid.ko
>>> 22 1 0xffffffff82e36000 23a8 ums.ko
>>> 23 1 0xffffffff82e39000 39960 linux.ko
>>> 24 1 0xffffffff82e73000 33c50 linux64.ko
>>> 25 1 0xffffffff82ea7000 acf mac_ntpd.ko
>>> 26 1 0xffffffff82ea8000 27b0 nullfs.ko
>>> 33 3 0xffffffff82eab000 4f2d8 ibcore.ko
>>> 34 1 0xffffffff82efb000 1fd39 mlx4ib.ko
>>> 35 1 0xffffffff82f1b000 85a0 ipoib.ko
>>>
>>
>> So ib0 came up immediately after configuring in rc.conf:
>>
>> ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0"
>>
>> Next issue is poor performance shown by iperf:
>>
>> root at zfs-01:~ # iperf -c raid-01-hpc
>> ------------------------------------------------------------
>> Client connecting to raid-01-hpc, TCP port 5001
>> TCP window size: 33.3 KByte (default)
>> ------------------------------------------------------------
>> [ 3] local 192.168.129.17 port 28021 connected with 192.168.129.1
>> port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0-10.1 sec 127 MBytes 106 Mbits/sec
>>
>> raid-01 is a CentOS 7 server.
>>
>> Performance between two CentOS servers is more reasonable:
>>
>> [root at raid-02.mortimer ~] 133: iperf -c raid-01-hpc
>> ------------------------------------------------------------
>> Client connecting to raid-01-hpc, TCP port 5001
>> TCP window size: 2.50 MByte (default)
>> ------------------------------------------------------------
>> [ 3] local 192.168.129.2 port 57952 connected with 192.168.129.1
>> port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0-10.0 sec 24.1 GBytes 20.7 Gbits/sec
>>
>> Ultimately I aim to use NFS over IB here, but I think iperf is
>> telling me that ipoib needs to be tuned.
>>
>> Might this be connected vs datagram issue? If so, how to change it
>> when using loadable modules? The wiki only mentions building IPOIB_CM
>> into the kernel.
>>
>> If we can figure this out, I'll fully document everything on the
>> wiki, of course.
>>
>> Thanks,
>>
>> JB
>>
>
> Finally found a moment to look at this again.
>
> I added
>
> CFLAGS+=-DIPOIB_CM
>
> to
>
> /etc/make.conf
>
> and rebuilt the modules.
>
> This seems to have worked, because ifconfig now allows me to set an
> MTU over 4096, which was not possible without IPOIB_CM.
>
> I'm still seeing abysmal performance with iperf, though:
>
> [root at raid-01.mortimer bacon] 221: ip link show ib0
> 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast
> state UP mode DEFAULT group default qlen 256
> link/infiniband
> 80:00:02:08:fe:80:00:00:00:00:00:00:f4:52:14:03:00:98:37:81 brd
> 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
>
> [root at raid-01.mortimer bacon] 222: iperf -s -B raid-01-hpc
> ------------------------------------------------------------
> Server listening on TCP port 5001
> Binding to local address raid-01-hpc
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
>
> root at zfs-01:~ # ifconfig ib0
> ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
> options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
> lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> root at zfs-01:~ # iperf -c raid-01-hpc
> ------------------------------------------------------------
> Client connecting to raid-01-hpc, TCP port 5001
> TCP window size: 40.8 KByte (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.17 port 39939 connected with 192.168.129.1 port
> 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-15.1 sec 106 KBytes 57.5 Kbits/sec
>
> Any suggestions?
>
> Thanks,
>
> JB
>
Hold on...
I just noticed that ifconfig wasn't showing an IP address for ib0, so
how was it even connecting to the Linux server????
After restarting netif, performance is close to what I'm seeing CentOS
to CentOS:
root at zfs-01:~ # ifconfig ib0
ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1
inet 192.168.129.17 netmask 0xffff8000 broadcast 192.168.255.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root at zfs-01:~ # iperf -c raid-01-hpc
------------------------------------------------------------
Client connecting to raid-01-hpc, TCP port 5001
TCP window size: 968 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.129.17 port 57686 connected with 192.168.129.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 18.8 GBytes 16.2 Gbits/sec
Pending stability and NFS testing, it appears that a stock FreeBSD
server may be viable in an IB HPC cluster at this point. At least
enabling and maintaining the IB modules can be done with minimal down
time now that world is IB-enabled and the stock kernel modules appear to
work reasonably well.
Regards,
JB
--
Earth is a beta site.
More information about the freebsd-infiniband
mailing list