Mellanox MT25418 performance IPOIB?
jwbacon at tds.net
Thu Apr 17 13:01:51 UTC 2014
What type of network are you running? DDR? QDR? FDR? Switch and HCA
We're purchasing a new cluster with Mellanox ConnectX-3 FDR cards. I'll
report back some results with and without IPOIB once I have my hands on it.
FYI, with some tuning effort, I was able to get up to 600 megabytes/sec
both ways over DDR IB from a CentOS NFS server with 12 SATA disks on a
PERC H710, ext4, RAID 6. Might be able to do a little better with
FreeBSD and ZFS where IB is providing 7.5gb/sec.
On 4/17/14 4:09 AM, Karl Pielorz wrote:
> --On 16 April 2014 14:28:37 +0200 Stig Inge Lea Bjørnsen
> <stiginge at pvv.org> wrote:
>> You could try to use Infiniband in datagram mode, recompile without the
>> IPOIB_CM option.
> Ok, same boxes - I just removed IPOIB_CM out of the kernel config and
> recompiled / installed it.
> iperf now consistently returns:
> [ 3] local 10.10.10.2 port 37715 connected with 10.10.10.1 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0-10.0 sec 8.71 GBytes 7.46 Gbits/sec
>> The maximum MTU will then be 4092, but it might default to 2044. I run
>> OpenSM on both machines and set the MTU explicitly:
> Yes, mtu defaulted to 2044 - raising it to 4092 actually causes a loss
> of .5-1Gbit with iperf (obviously not 'real world') - but at leas I
> can now try and use the stuff with real world (now it's returning more
> than a few hundred kilobytes a second!).
>> The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions.
> Same as here then.
> Thanks for your email!
> freebsd-infiniband at freebsd.org mailing list
> To unsubscribe, send any mail to
> "freebsd-infiniband-unsubscribe at freebsd.org"
Jason W. Bacon
jwbacon at tds.net
Circumstances don't make a man:
They reveal him.
More information about the freebsd-infiniband