Mellanox MT25418 performance IPOIB?
kpielorz_lst at tdx.co.uk
Sat Mar 22 23:15:50 UTC 2014
I've added a couple of dual port Mellanox MT25418's (Hardware v. a0 /
firmware 2.6.0) to two FreeBSD 10.0-Stable amd64 boxes. I was hoping to use
them as a cheap alternative to 10Gbps LAN cards for a point to point link
to run HAST over.
I've remade/installed the world with OFED support - and compiled OFED, SDP,
IPOIB_CM, ipoib, mlx4ib, mlxen and mthca in the kernel.
I've setup/run opensm - and the cards appear to be active OK - ibstat
output seems OK, and ibportstate shows the cabled port is 'Active' /
'LinkUp' and running at 5.0 Gbps using 4X (cards are only connected to each
I've put an IP on both ends - and can ping between the cards, but TCP/IP
performance seems very poor - e.g. a test 'ftp' between the cards only nets
around 381Kbyte/s? [iperf shows similarly poor results]
I can't see any errors logged anywhere - and 'netstat -b -i' doesn't show
any errors either.
Anyone else running these cards successfully?
My opensm config consists of just 'PORTS="ALL"' - I was unable to find a
lot of example opensm file configs.
Also I get some chatter dumped on the console (presumably from the drivers)
most of it looks just like some kind of debug output [can't see any obvious
fwiw both machines are 1230v3 Xeon based.
Can anyone suggest what might be wrong? / What to look at?
More information about the freebsd-infiniband