Supported NICs

Jason Bacon bacon at uwm.edu
Wed Oct 16 16:47:13 UTC 2013 

Good timing: just finished a new round of testing on 9.2-RELEASE.

No discernible difference from 9.1.  Throughput was a bit slower both 
BSD-Linux and Linux-Linux, probably due to the high load on the cluster 
today.

FreeBSD infinibsd bacon ~ 36: sysctl kern.conftxt | egrep 
'OFED|IPOIB|DEAD|INVARIANT|WITNESS'
options    IPOIB_CM
options    OFED

Thanks,

     JB

On 10/16/13 10:23, Oded Shanoon wrote:
> Please run sysctl kern.conftxt and grep for the following options:
>
> OFED_DEBUG_INIT
> IPOIB_DEBUG
> DEADLKRES
> INVARIANTS
> INVARIANT_SUPPORT
> WITNESS
> WITNESS_SKIPSPIN
>
> These could result in performance degredation
>
> Regards,
>
> Oded Shanoon
> OFED-FreeBSD Team Leader
> Mellanox Technologies, Raanana
>
>
> -----Original Message-----
> From: Jason Bacon [mailto:bacon at uwm.edu]
> Sent: Monday, October 14, 2013 9:37 PM
> To: Oded Shanoon
> Cc: Jason Bacon; Anthony Cornehl; freebsd-infiniband at freebsd.org
> Subject: Re: Supported NICs
>
>
> Some initial test results...
>
> I installed an old Infinihost DDR HCA in one of our compute nodes running FreeBSD 9.1.
>
> RHEL nodes are using qlogic IB HCAs.
>
> 10.1.1 is gigabit Ethernet, 10.1.2 is IB.
>
> Running iperf server on FreeBSD and client on one of our RHEL nodes shows OK performance:
>
> bacon at infinibsd:/home/bacon % iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 64.0 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.1.1.140 port 5001 connected with 10.1.1.39 port 35947
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.0 sec  1.10 GBytes   947 Mbits/sec
>
> [  5] local 10.1.2.140 port 5001 connected with 10.1.2.39 port 60090 [  5]  0.0-10.0 sec  7.20 GBytes  6.18 Gbits/sec
>
> RHEL to RHEL gives us about 8 Gbits/sec.
>
> Iperf server on RHEL and client on FreeBSD shows very poor performance for IB, while GigE is fine:
>
> bacon at infinibsd:/home/bacon % iperf -c 10.1.1.39
> ------------------------------------------------------------
> Client connecting to 10.1.1.39, TCP port 5001 TCP window size: 32.8 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.1.1.140 port 60066 connected with 10.1.1.39 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec
>
> bacon at infinibsd:/home/bacon % iperf -c 10.1.2.39
> ------------------------------------------------------------
> Client connecting to 10.1.2.39, TCP port 5001 TCP window size: 32.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.1.2.140 port 14608 connected with 10.1.2.39 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-32.7 sec   768 KBytes   192 Kbits/sec
>
> Any ideas what might cause this?
>
> I'm going to install 9.2-RELEASE and retest in any case, plus explore the ib config tools, but any feedback in the meantime would be appreciated.
>
> Thanks,
>
>       JB
>
> On 10/07/13 01:51, Oded Shanoon wrote:
>> Hi Jason,
>>
>> IB support in 9.2 is supposed to be stable enough.
>> Please note that Mellanox started supporting FreeBSD only recently. The driver in 9.2 was ported by someone from Isilon (Jeff Roberson) from OFA-1.5.3.
>> Since we started our involvement we entered some fixes to that driver (that were submitted into 9.2).
>> We also mapped various issues which needs to be fixed in the future.
>> We are now working on a major "face lift" to the driver - making it much more stable and with improved performance and features.
>>
>> Regards,
>>
>> Oded Shanoon
>> OFED-FreeBSD Team Leader
>> Mellanox Technologies, Raanana
>>
>>
>> -----Original Message-----
>> From: owner-freebsd-infiniband at freebsd.org
>> [mailto:owner-freebsd-infiniband at freebsd.org] On Behalf Of Jason Bacon
>> Sent: Monday, September 16, 2013 4:32 PM
>> To: Anthony Cornehl
>> Cc: freebsd-infiniband at freebsd.org
>> Subject: Re: Supported NICs
>>
>>
>> Thanks Anthony&   Oded!
>>
>> Would you say that IB support in 9.2 is stable enough for a production HPC file server?
>>
>> Will do plenty of my own testing first, of course.
>>
>> Regards,
>>
>>        Jason
>>
>> On 09/15/13 22:38, Anthony Cornehl wrote:
>>> On Sep 15, 2013 8:17 PM, "Anthony Cornehl"<accornehl at gmail.com
>>> <mailto:accornehl at gmail.com>>   wrote:
>>>> On Sep 15, 2013 1:20 PM, "Jason bacon"<jwbacon at tds.net
>>> <mailto:jwbacon at tds.net>>   wrote:
>>>>> Is there a list of supported IB NICs out there somewhere?
>>>>>
>>>>> I followed the wiki instructions for rebuilding with IB support
>>> and now have mlx4ib, mlxen, etc.
>>>>> Was hoping there would be man pages for the drivers that list
>>> known working cards, but there don't seem to be.  I'm hoping to test
>>> a file server using IPOIB and possible roll a FreeNAS ISO with IB
>>> support if it works out.
>>>>> Thanks,
>>>>>
>>>>> --
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>     Jason W. Bacon
>>>>> jwbacon at tds.net<mailto:jwbacon at tds.net>
>>>>>
>>>>>     Circumstances don't make a man:
>>>>>     They reveal him.
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-infiniband at freebsd.org
>>> <mailto:freebsd-infiniband at freebsd.org>   mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>>>>> To unsubscribe, send any mail to
>>> "freebsd-infiniband-unsubscribe at freebsd.org
>>> <mailto:freebsd-infiniband-unsubscribe at freebsd.org>"
>>>> Hey Jason,
>>>>
>>>> Mellanox ConnectX series cards are the only ones supported currently...
>>>>
>>>> http://www.mellanox.com/page/infiniband_cards_overview
>>>>
>>>> Don't forget to compile with IPoIB as well, since the IPoIB support
>>> is compiled into the driver, unlike in Linux.
>>>> Just be aware that IPoIB performance is also improved by enabling
>>> connected mode when you compile the kernel module. The IB code in
>>> FreeBSD is a few years older the what is in Linux, but, the following
>>> forum thread is probably relevant...
>>> http://forums.servethehome.com/networking/1554-infiniband-ipoib-perfo
>>> r
>>> mance-problems.html
>>>> Cheers!
>>> It also appears that Jeff fixed SDP a few weeks ago, which is more
>>> capable of reaching line-speed for IB-connected devices.
>>>
>>> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html
>>>
>>> Cheers!
>>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>      Jason W. Bacon
>>      jwbacon at tds.net
>>
>>      Circumstances don't make a man:
>>      They reveal him.
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> _______________________________________________
>> freebsd-infiniband at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
>> To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe at freebsd.org"
>


-- 

-------------------------------------
   Jason W. Bacon
   Systems Programmer
   Research Computing Support
   University of Wisconsin Milwaukee
   bacon at uwm.edu
-------------------------------------

More information about the freebsd-infiniband mailing list