Supported NICs
Jason Bacon
jwbacon at tds.net
Wed Mar 19 20:22:03 UTC 2014
Also, ibping is very slow in both directions:
root at raid1:~ # ibping 28
Pong from compute-1-15.(none) (Lid 28): time 7.481 ms
Pong from compute-1-15.(none) (Lid 28): time 4.465 ms
Pong from compute-1-15.(none) (Lid 28): time 11.933 ms
Pong from compute-1-15.(none) (Lid 28): time 21.255 ms
Pong from compute-1-15.(none) (Lid 28): time 8.672 ms
Pong from compute-1-15.(none) (Lid 28): time 13.633 ms
[root at compute-1-15 ~]# /usr/sbin/ibping 163
Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 4.035 ms
Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.690 ms
Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.721 ms
Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.776 ms
Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 24.457 ms
Pong from raid1.avi.hpc.uwm.edu (Lid 163): time 10.723 ms
On 03/19/14 12:26, Jason Bacon wrote:
>
> We finally got a new server in with a ConnectX-3 card. It seems to
> work reliably with our SilverStorm DDR network using a CX4 - QSFP cable.
>
> Below are specs and iperf benchmarks between the FreeBSD server and a
> CentOS 6.5 compute node with a Qlogic DDR card. Connected mode with
> MTU 65520. FreeBSD 10.0-RELEASE on a Dell R720xd.
>
> Wondering if anyone has any suggestions about what might cause the
> poor output performance. Any sysctls or loader.conf settings I should
> look into? None of the kern.conftext flags mentioned below are present.
>
> Any pointers on how to diagnose the issue would be appreciated as well.
>
> Thanks,
>
> Jason
>
> Input throughput is excellent - over 11 gb/sec. Linux to Linux on the
> same network does about 10 gb/sec.
>
> [root at compute-5-33 ~]# iperf -c 10.1.2.250
> ------------------------------------------------------------
> Client connecting to 10.1.2.250, TCP port 5001
> TCP window size: 645 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.1.2.140 port 54223 connected with 10.1.2.250 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 13.0 GBytes 11.2 Gbits/sec
>
>
> bacon at raid1:~ % netstat 1
> input (Total) output
> packets errs idrops bytes packets errs bytes colls
> 10631 0 0 685149597 5390 0 409806 0
> 22281 0 0 1452318244 11136 0 846730 0
> 23776 0 0 1557833160 11884 0 903654 0
> 23891 0 0 1565368420 11952 0 907986 0
> 23328 0 0 1528478408 11663 0 886554 0
>
> Output throughput is terrible:
>
> [root at compute-5-33 ~]# iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [ 4] local 10.1.2.140 port 5001 connected with 10.1.2.250 port 50207
> [ ID] Interval Transfer Bandwidth
> [ 4] 0.0-10.9 sec 1.50 MBytes 1.16 Mbits/sec
>
>
> bacon at raid1:~ % netstat 1
> input (Total) output
> packets errs idrops bytes packets errs bytes colls
> 20 0 0 1444 27 1 124684 0
> 5 0 0 284 1 4 393718 0
> 5 0 0 284 1 4 393718 0
> 6 0 0 344 1 4 393718 0
> 6 0 0 344 18 4 426562 0
> 5 0 0 284 1 4 393718 0
>
> root at raid1:/home/bacon # ibstat
> CA 'mlx4_0'
> CA type: MT4099
> Number of ports: 1
> Firmware version: 2.11.500
> Hardware version: 0
> Node GUID: 0xf4521403007d2c10
> System image GUID: 0xf4521403007d2c13
> Port 1:
> State: Active
> Physical state: LinkUp
> Rate: 20
> Base lid: 163
> LMC: 0
> SM lid: 1
> Capability mask: 0x02514868
> Port GUID: 0xf4521403007d2c11
>
> root at raid1:/home/bacon # ibportstate -G 0xf4521403007d2c11 query
> PortInfo:
> # Port info: Lid 163 port 0
> LinkState:.......................Active
> PhysLinkState:...................LinkUp
> LinkWidthSupported:..............4X (IBA extension)
> LinkWidthEnabled:................4X
> LinkWidthActive:.................4X
> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
> LinkSpeedActive:.................5.0 Gbps
>
>
>
>
> On 10/16/13 10:23, Oded Shanoon wrote:
>> Please run sysctl kern.conftxt and grep for the following options:
>>
>> OFED_DEBUG_INIT
>> IPOIB_DEBUG
>> DEADLKRES
>> INVARIANTS
>> INVARIANT_SUPPORT
>> WITNESS
>> WITNESS_SKIPSPIN
>>
>> These could result in performance degredation
>>
>> Regards,
>>
>> Oded Shanoon
>> OFED-FreeBSD Team Leader
>> Mellanox Technologies, Raanana
>>
>>
>> -----Original Message-----
>> From: Jason Bacon [mailto:bacon at uwm.edu]
>> Sent: Monday, October 14, 2013 9:37 PM
>> To: Oded Shanoon
>> Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband at freebsd.org
>> Subject: Re: Supported NICs
>>
>>
>> Some initial test results...
>>
>> I installed an old Infinihost DDR HCA in one of our compute nodes
>> running FreeBSD 9.1.
>>
>> RHEL nodes are using qlogic IB HCAs.
>>
>> 10.1.1 is gigabit Ethernet, 10.1.2 is IB.
>>
>> Running iperf server on FreeBSD and client on one of our RHEL nodes
>> shows OK performance:
>>
>> bacon at infinibsd:/home/bacon % iperf -s
>> ------------------------------------------------------------
>> Server listening on TCP port 5001
>> TCP window size: 64.0 KByte (default)
>> ------------------------------------------------------------
>> [ 4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947
>> [ ID] Interval Transfer Bandwidth
>> [ 4] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec
>>
>> [ 5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090
>> [ 5] 0.0-10.0 sec 7.20 GBytes 6.18 Gbits/sec
>>
>> RHEL to RHEL gives us about 8 Gbits/sec.
>>
>> Iperf server on RHEL and client on FreeBSD shows very poor
>> performance for IB, while GigE is fine:
>>
>> bacon at infinibsd:/home/bacon % iperf -c 10.1.1.39
>> ------------------------------------------------------------
>> Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8
>> KByte (default)
>> ------------------------------------------------------------
>> [ 3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
>>
>> bacon at infinibsd:/home/bacon % iperf -c 10.1.2.39
>> ------------------------------------------------------------
>> Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0
>> KByte (default)
>> ------------------------------------------------------------
>> [ 3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0-32.7 sec 768 KBytes 192 Kbits/sec
>>
>> Any ideas what might cause this?
>>
>> I'm going to install 9.2-RELEASE and retest in any case, plus explore
>> the ib config tools, but any feedback in the meantime would be
>> appreciated.
>>
>> Thanks,
>>
>> JB
>>
>> On 10/07/13 01:51, Oded Shanoon wrote:
>>> Hi Jason,
>>>
>>> IB support in 9.2 is supposed to be stable enough.
>>> Please note that Mellanox started supporting FreeBSD only recently.
>>> The driver in 9.2 was ported by someone from Isilon (Jeff Roberson)
>>> from OFA-1.5.3.
>>> Since we started our involvement we entered some fixes to that
>>> driver (that were submitted into 9.2).
>>> We also mapped various issues which needs to be fixed in the future.
>>> We are now working on a major "face lift" to the driver - making it
>>> much more stable and with improved performance and features.
>>>
>>> Regards,
>>>
>>> Oded Shanoon
>>> OFED-FreeBSD Team Leader
>>> Mellanox Technologies, Raanana
>>>
>>>
>>> -----Original Message-----
>>> From: owner-freebsd-infiniband at freebsd.org
>>> [mailto:owner-freebsd-infiniband at freebsd.org] On Behalf Of Jason Bacon
>>> Sent: Monday, September 16, 2013 4:32 PM
>>> To: Anthony Cornehl
>>> Cc: freebsd-infiniband at freebsd.org
>>> Subject: Re: Supported NICs
>>>
>>>
>>> Thanks Anthony& Oded!
>>>
>>> Would you say that IB support in 9.2 is stable enough for a
>>> production HPC file server?
>>>
>>> Will do plenty of my own testing first, of course.
>>>
>>> Regards,
>>>
>>> Jason
>>>
>>> On 09/15/13 22:38, Anthony Cornehl wrote:
>>>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"<accornehl at gmail.com
>>>> <mailto:accornehl at gmail.com>> wrote:
>>>>> On Sep 15, 2013 1:20 PM, "Jason bacon"<jwbacon at tds.net
>>>> <mailto:jwbacon at tds.net>> wrote:
>>>>>> Is there a list of supported IB NICs out there somewhere?
>>>>>>
>>>>>> I followed the wiki instructions for rebuilding with IB support
>>>> and now have mlx4ib, mlxen, etc.
>>>>>> Was hoping there would be man pages for the drivers that list
>>>> known working cards, but there don't seem to be. I'm hoping to test
>>>> a file server using IPOIB and possible roll a FreeNAS ISO with IB
>>>> support if it works out.
>>>>>> Thanks,
>>>>>>
>>>>>> --
>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>> Jason W. Bacon
>>>>>> jwbacon at tds.net<mailto:jwbacon at tds.net>
>>>>>>
>>>>>> Circumstances don't make a man:
>>>>>> They reveal him.
>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>
>>>>>> _______________________________________________
>>>>>> freebsd-infiniband at freebsd.org
>>>> <mailto:freebsd-infiniband at freebsd.org> mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>>>>>> To unsubscribe, send any mail to
>>>> "freebsd-infiniband-unsubscribe at freebsd.org
>>>> <mailto:freebsd-infiniband-unsubscribe at freebsd.org>"
>>>>> Hey Jason,
>>>>>
>>>>> Mellanox ConnectX series cards are the only ones supported
>>>>> currently...
>>>>>
>>>>> http://www.mellanox.com/page/infiniband_cards_overview
>>>>>
>>>>> Don't forget to compile with IPoIB as well, since the IPoIB support
>>>> is compiled into the driver, unlike in Linux.
>>>>> Just be aware that IPoIB performance is also improved by enabling
>>>> connected mode when you compile the kernel module. The IB code in
>>>> FreeBSD is a few years older the what is in Linux, but, the following
>>>> forum thread is probably relevant...
>>>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo
>>>> r
>>>> mance-problems.html
>>>>> Cheers!
>>>> It also appears that Jeff fixed SDP a few weeks ago, which is more
>>>> capable of reaching line-speed for IB-connected devices.
>>>>
>>>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html
>>>>
>>>> Cheers!
>>>>
>>> --
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> Jason W. Bacon
>>> jwbacon at tds.net
>>>
>>> Circumstances don't make a man:
>>> They reveal him.
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> _______________________________________________
>>> freebsd-infiniband at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>>> To unsubscribe, send any mail to
>>> "freebsd-infiniband-unsubscribe at freebsd.org"
>>
>
More information about the freebsd-infiniband
mailing list