Kernel modules

[Jason Bacon]

On 3/18/19 10:08 AM, Jason Bacon wrote:
> On 3/18/19 9:10 AM, Hans Petter Selasky wrote:
>> On 3/18/19 2:56 PM, Jason Bacon wrote:
>>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote:
>>>> On 3/18/19 1:32 PM, Jason Bacon wrote:
>>>>>
>>>>> Anyone using the kernel modules lately?
>>>>>
>>>>> After building per the instructions on the wiki 
>>>>> (https://wiki.freebsd.org/InfiniBand):
>>>>>
>>>>> <<<ROOT at unixdev.ceas>>> /home/bacon 1005 # kldload ibcore
>>>>> kldload: an error occurred while loading module ibcore. Please 
>>>>> check dmesg(8) for more details.
>>>>>
>>>>> dmesg:
>>>>>
>>>>> link_elf_obj: symbol if_index undefined
>>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type
>>>>
>>>> Hi,
>>>>
>>>> You built the KMODs out of the kernel? Try adding 
>>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build.
>>>>
>>>> --HPS
>>>
>>> I used the script for building kernel modules (which I wrote and 
>>> posted at https://wiki.freebsd.org/InfiniBand a few years ago, based 
>>> on advice from a Mellanox developer at the time). This will show 
>>> exactly what I did.
>>>
>>
>> Can you try:
>>
>> make DEBUG_FLAGS="-DVIMAGE=1"
>>
>> instead of
>>
>> make
>>
>> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike 
>> FreeBSD-11.
>>
>> --HPS
>>
>
> That worked, thanks.  I'll update the wiki after some testing.
>
> ...
> objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug
> objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug ipoib.ko.full 
> ipoib.ko
> install -T release -o root -g wheel -m 555   ipoib.ko /boot/modules/
> kldxref /boot/modules
> Id Refs Address                Size Name
>  1   72 0xffffffff80200000  243cd00 kernel
>  2    1 0xffffffff8263e000     8b50 ng_ubt.ko
>  3    2 0xffffffff82647000    12dd0 ng_hci.ko
>  4    2 0xffffffff8265a000     2d00 ng_bluetooth.ko
>  5    3 0xffffffff8265d000    175d8 netgraph.ko
>  7    5 0xffffffff82688000    279b0 linuxkpi.ko
>  9    1 0xffffffff8274d000   3a9a10 zfs.ko
> 10    2 0xffffffff82af7000     a4f0 opensolaris.ko
> 11    1 0xffffffff82b02000    20af0 firewire.ko
> 12    1 0xffffffff82b23000     bf70 cuse.ko
> 13    2 0xffffffff82b2f000    665e0 mlx4.ko
> 15    1 0xffffffff82bd0000     5fb8 uplcom.ko
> 16    2 0xffffffff82bd6000     ab88 ucom.ko
> 17    1 0xffffffff82e21000     1a20 fdescfs.ko
> 18    1 0xffffffff82e23000     490c linprocfs.ko
> 19    3 0xffffffff82e28000     2e28 linux_common.ko
> 20    1 0xffffffff82e2b000     81f0 tmpfs.ko
> 21    1 0xffffffff82e34000     1800 uhid.ko
> 22    1 0xffffffff82e36000     23a8 ums.ko
> 23    1 0xffffffff82e39000    39960 linux.ko
> 24    1 0xffffffff82e73000    33c50 linux64.ko
> 25    1 0xffffffff82ea7000      acf mac_ntpd.ko
> 26    1 0xffffffff82ea8000     27b0 nullfs.ko
> 33    3 0xffffffff82eab000    4f2d8 ibcore.ko
> 34    1 0xffffffff82efb000    1fd39 mlx4ib.ko
> 35    1 0xffffffff82f1b000     85a0 ipoib.ko
>

So ib0 came up immediately after configuring in rc.conf:

ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0"

Next issue is poor performance shown by iperf:

root at zfs-01:~ # iperf -c raid-01-hpc
------------------------------------------------------------
Client connecting to raid-01-hpc, TCP port 5001
TCP window size: 33.3 KByte (default)
------------------------------------------------------------
[  3] local 192.168.129.17 port 28021 connected with 192.168.129.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec   127 MBytes   106 Mbits/sec

raid-01 is a CentOS 7 server.

Performance between two CentOS servers is more reasonable:

[root at raid-02.mortimer ~] 133: iperf -c raid-01-hpc
------------------------------------------------------------
Client connecting to raid-01-hpc, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 192.168.129.2 port 57952 connected with 192.168.129.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  24.1 GBytes  20.7 Gbits/sec

Ultimately I aim to use NFS over IB here, but I think iperf is telling 
me that ipoib needs to be tuned.

Might this be connected vs datagram issue?  If so, how to change it when 
using loadable modules? The wiki only mentions building IPOIB_CM into 
the kernel.

If we can figure this out, I'll fully document everything on the wiki, 
of course.

Thanks,

     JB

-- 
Earth is a beta site.