Mellanox MT25418 performance IPOIB?

[Jason Bacon]

Coincidentally, I had just run similar tests on our cluster using a 
FreeBSD file server and CentOS compute nodes.

I reported the results in to this list.  If you didn't see the thread 
already, you should be able to find it in the archives at:

     http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband

Basically, iperf reported excellent incoming speed from the CentOS 
nodes, but outgoing speed was terrible.

In response, I was informed by a Mellanox engineer that the ipoib layer 
is currently being overhauled and we should see improvement later this year.

Regards,

     JB

On 03/22/14 18:15, Karl Pielorz wrote:
>
> Hi All,
>
> I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 / 
> firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping 
> to use them as a cheap alternative to 10Gbps LAN cards for a point to 
> point link to run HAST over.
>
> I've remade/installed the world with OFED support - and compiled OFED, 
> SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel.
>
> I've setup/run opensm - and the cards appear to be active OK - ibstat 
> output seems OK, and ibportstate shows the cabled port is 'Active' / 
> 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to 
> each other).
>
> I've put an IP on both ends - and can ping between the cards, but 
> TCP/IP performance seems very poor - e.g. a test 'ftp' between the 
> cards only nets around 381Kbyte/s? [iperf shows similarly poor results]
>
> I can't see any errors logged anywhere - and 'netstat -b -i' doesn't 
> show any errors either.
>
> Anyone else running these cards successfully?
>
> My opensm config consists of just 'PORTS="ALL"' - I was unable to find 
> a lot of example opensm file configs.
>
> Also I get some chatter dumped on the console (presumably from the 
> drivers) most of it looks just like some kind of debug output [can't 
> see any obvious errors].
>
> fwiw both machines are 1230v3 Xeon based.
>
> Can anyone suggest what might be wrong? / What to look at?
>
> Thanks,
>
> -Karl
> _______________________________________________
> freebsd-infiniband at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
> To unsubscribe, send any mail to 
> "freebsd-infiniband-unsubscribe at freebsd.org"