Mellanox MT25418 performance IPOIB?
jwbacon at tds.net
Sun Mar 23 13:54:59 UTC 2014
Coincidentally, I had just run similar tests on our cluster using a
FreeBSD file server and CentOS compute nodes.
I reported the results in to this list. If you didn't see the thread
already, you should be able to find it in the archives at:
Basically, iperf reported excellent incoming speed from the CentOS
nodes, but outgoing speed was terrible.
In response, I was informed by a Mellanox engineer that the ipoib layer
is currently being overhauled and we should see improvement later this year.
On 03/22/14 18:15, Karl Pielorz wrote:
> Hi All,
> I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 /
> firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping
> to use them as a cheap alternative to 10Gbps LAN cards for a point to
> point link to run HAST over.
> I've remade/installed the world with OFED support - and compiled OFED,
> SDP, IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel.
> I've setup/run opensm - and the cards appear to be active OK - ibstat
> output seems OK, and ibportstate shows the cabled port is 'Active' /
> 'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to
> each other).
> I've put an IP on both ends - and can ping between the cards, but
> TCP/IP performance seems very poor - e.g. a test 'ftp' between the
> cards only nets around 381Kbyte/s? [iperf shows similarly poor results]
> I can't see any errors logged anywhere - and 'netstat -b -i' doesn't
> show any errors either.
> Anyone else running these cards successfully?
> My opensm config consists of just 'PORTS="ALL"' - I was unable to find
> a lot of example opensm file configs.
> Also I get some chatter dumped on the console (presumably from the
> drivers) most of it looks just like some kind of debug output [can't
> see any obvious errors].
> fwiw both machines are 1230v3 Xeon based.
> Can anyone suggest what might be wrong? / What to look at?
> freebsd-infiniband at freebsd.org mailing list
> To unsubscribe, send any mail to
> "freebsd-infiniband-unsubscribe at freebsd.org"
More information about the freebsd-infiniband