Mellanox MT25418 performance IPOIB?

[Stig Inge Lea Bjørnsen]

Hi,

I'm experimenting with a similar setup for a direct link using two HP 
MT25418 cards. One machine runs FreeBSD 10.0-RELEASE-p1 and the other 
runs Debian Unstable jessie/sid.

You could try to use Infiniband in datagram mode, recompile without the 
IPOIB_CM option.

The maximum MTU will then be 4092, but it might default to 2044. I run 
OpenSM on both machines and set the MTU explicitly:

$ cat /etc/opensm/partitions.conf
# mtu=5 for 4092, 4 for 2044
Default=0x7fff, ipoib, mtu=5 : ALL=full ;

For iperf, I found that the performance when using FreeBSD as the client 
improved dramatically when I set the TCP window size equal to what the 
iperf client on Linux use:

$ iperf -c linux-host -w 165K

The bandwidth reported by iperf is ~7.5 Gbits/sec in both directions.

Regards,
Stig Inge Lea Bjørnsen


On 2014-03-23 00:15, Karl Pielorz wrote:
>
> Hi All,
>
> I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 /
> firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping to
> use them as a cheap alternative to 10Gbps LAN cards for a point to point
> link to run HAST over.
>
> I've remade/installed the world with OFED support - and compiled OFED,
> SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel.
>
> I've setup/run opensm - and the cards appear to be active OK - ibstat
> output seems OK, and ibportstate shows the cabled port is 'Active' /
> 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to
> each other).
>
> I've put an IP on both ends - and can ping between the cards, but TCP/IP
> performance seems very poor - e.g. a test 'ftp' between the cards only
> nets around 381Kbyte/s? [iperf shows similarly poor results]
>
> I can't see any errors logged anywhere - and 'netstat -b -i' doesn't
> show any errors either.
>
> Anyone else running these cards successfully?
>
> My opensm config consists of just 'PORTS="ALL"' - I was unable to find a
> lot of example opensm file configs.
>
> Also I get some chatter dumped on the console (presumably from the
> drivers) most of it looks just like some kind of debug output [can't see
> any obvious errors].
>
> fwiw both machines are 1230v3 Xeon based.
>
> Can anyone suggest what might be wrong? / What to look at?
>
> Thanks,
>
> -Karl