Kernel modules
Jason Bacon
bacon4000 at gmail.com
Fri Apr 12 12:39:33 UTC 2019
On 2019-03-18 10:54, Jason Bacon wrote:
> On 3/18/19 10:08 AM, Jason Bacon wrote:
>> On 3/18/19 9:10 AM, Hans Petter Selasky wrote:
>>> On 3/18/19 2:56 PM, Jason Bacon wrote:
>>>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote:
>>>>> On 3/18/19 1:32 PM, Jason Bacon wrote:
>>>>>>
>>>>>> Anyone using the kernel modules lately?
>>>>>>
>>>>>> After building per the instructions on the wiki
>>>>>> (https://wiki.freebsd.org/InfiniBand):
>>>>>>
>>>>>> <<<ROOT at unixdev.ceas>>> /home/bacon 1005 # kldload ibcore
>>>>>> kldload: an error occurred while loading module ibcore. Please
>>>>>> check dmesg(8) for more details.
>>>>>>
>>>>>> dmesg:
>>>>>>
>>>>>> link_elf_obj: symbol if_index undefined
>>>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type
>>>>>
>>>>> Hi,
>>>>>
>>>>> You built the KMODs out of the kernel? Try adding
>>>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build.
>>>>>
>>>>> --HPS
>>>>
>>>> I used the script for building kernel modules (which I wrote and
>>>> posted at https://wiki.freebsd.org/InfiniBand a few years ago,
>>>> based on advice from a Mellanox developer at the time). This will
>>>> show exactly what I did.
>>>>
>>>
>>> Can you try:
>>>
>>> make DEBUG_FLAGS="-DVIMAGE=1"
>>>
>>> instead of
>>>
>>> make
>>>
>>> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike
>>> FreeBSD-11.
>>>
>>> --HPS
>>>
>>
>> That worked, thanks. I'll update the wiki after some testing.
>>
>> ...
>> objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug
>> objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug
>> ipoib.ko.full ipoib.ko
>> install -T release -o root -g wheel -m 555 ipoib.ko /boot/modules/
>> kldxref /boot/modules
>> Id Refs Address Size Name
>> 1 72 0xffffffff80200000 243cd00 kernel
>> 2 1 0xffffffff8263e000 8b50 ng_ubt.ko
>> 3 2 0xffffffff82647000 12dd0 ng_hci.ko
>> 4 2 0xffffffff8265a000 2d00 ng_bluetooth.ko
>> 5 3 0xffffffff8265d000 175d8 netgraph.ko
>> 7 5 0xffffffff82688000 279b0 linuxkpi.ko
>> 9 1 0xffffffff8274d000 3a9a10 zfs.ko
>> 10 2 0xffffffff82af7000 a4f0 opensolaris.ko
>> 11 1 0xffffffff82b02000 20af0 firewire.ko
>> 12 1 0xffffffff82b23000 bf70 cuse.ko
>> 13 2 0xffffffff82b2f000 665e0 mlx4.ko
>> 15 1 0xffffffff82bd0000 5fb8 uplcom.ko
>> 16 2 0xffffffff82bd6000 ab88 ucom.ko
>> 17 1 0xffffffff82e21000 1a20 fdescfs.ko
>> 18 1 0xffffffff82e23000 490c linprocfs.ko
>> 19 3 0xffffffff82e28000 2e28 linux_common.ko
>> 20 1 0xffffffff82e2b000 81f0 tmpfs.ko
>> 21 1 0xffffffff82e34000 1800 uhid.ko
>> 22 1 0xffffffff82e36000 23a8 ums.ko
>> 23 1 0xffffffff82e39000 39960 linux.ko
>> 24 1 0xffffffff82e73000 33c50 linux64.ko
>> 25 1 0xffffffff82ea7000 acf mac_ntpd.ko
>> 26 1 0xffffffff82ea8000 27b0 nullfs.ko
>> 33 3 0xffffffff82eab000 4f2d8 ibcore.ko
>> 34 1 0xffffffff82efb000 1fd39 mlx4ib.ko
>> 35 1 0xffffffff82f1b000 85a0 ipoib.ko
>>
>
> So ib0 came up immediately after configuring in rc.conf:
>
> ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0"
>
> Next issue is poor performance shown by iperf:
>
> root at zfs-01:~ # iperf -c raid-01-hpc
> ------------------------------------------------------------
> Client connecting to raid-01-hpc, TCP port 5001
> TCP window size: 33.3 KByte (default)
> ------------------------------------------------------------
> [ 3] local 192.168.129.17 port 28021 connected with 192.168.129.1
> port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.1 sec 127 MBytes 106 Mbits/sec
>
> raid-01 is a CentOS 7 server.
>
> Performance between two CentOS servers is more reasonable:
>
> [root at raid-02.mortimer ~] 133: iperf -c raid-01-hpc
> ------------------------------------------------------------
> Client connecting to raid-01-hpc, TCP port 5001
> TCP window size: 2.50 MByte (default)
> ------------------------------------------------------------
> [ 3] local 192.168.129.2 port 57952 connected with 192.168.129.1 port
> 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 24.1 GBytes 20.7 Gbits/sec
>
> Ultimately I aim to use NFS over IB here, but I think iperf is telling
> me that ipoib needs to be tuned.
>
> Might this be connected vs datagram issue? If so, how to change it
> when using loadable modules? The wiki only mentions building IPOIB_CM
> into the kernel.
>
> If we can figure this out, I'll fully document everything on the wiki,
> of course.
>
> Thanks,
>
> JB
>
Finally found a moment to look at this again.
I added
CFLAGS+=-DIPOIB_CM
to
/etc/make.conf
and rebuilt the modules.
This seems to have worked, because ifconfig now allows me to set an MTU
over 4096, which was not possible without IPOIB_CM.
I'm still seeing abysmal performance with iperf, though:
[root at raid-01.mortimer bacon] 221: ip link show ib0
6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast
state UP mode DEFAULT group default qlen 256
link/infiniband
80:00:02:08:fe:80:00:00:00:00:00:00:f4:52:14:03:00:98:37:81 brd
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
[root at raid-01.mortimer bacon] 222: iperf -s -B raid-01-hpc
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address raid-01-hpc
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
root at zfs-01:~ # ifconfig ib0
ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root at zfs-01:~ # iperf -c raid-01-hpc
------------------------------------------------------------
Client connecting to raid-01-hpc, TCP port 5001
TCP window size: 40.8 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.17 port 39939 connected with 192.168.129.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-15.1 sec 106 KBytes 57.5 Kbits/sec
Any suggestions?
Thanks,
JB
--
Earth is a beta site.
More information about the freebsd-infiniband
mailing list