Slow speed with Mellanox IS5023 Switch and Connect-X3 cards

[Vinícius Ferrão]

Hello guys,

I’m testing a server running FreeBSD 10.3-RELEASE and a Connect-X3 card. When communicating with other hosts in a point-to-point connections everything works as expected. But when I put the FreeBSD server on tem switching fabric the performances drops to a unusable state. In this setup we are using IPoIB with NFS on top of it.

Initial benchmarks with dd returned this numbers:

[root at node005 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=100
100+0 records in
100+0 records out
13107200 bytes (13 MB) copied, 3.96714 s, 3.3 MB/s

This happens only on the FreeBSD machine. If we test Linux to Linux speeds through the same switch we got a very reasonable performance:

[root at node005 home]# dd if=/dev/zero of=./dd1.dd bs=128k count=100
100+0 records in
100+0 records out
13107200 bytes (13 MB) copied, 0.0463456 s, 283 MB/s

And finally if we remove the switch from the setup, and link the FreeBSD machine directly with a CentOS 7.2 client the throughput is exactly what we expect:

[root at node010 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=1000
1000+0 records in
1000+0 records out
131072000 bytes (131 MB) copied, 0.640333 s, 205 MB/s

This Mellanox Switch is an unmanaged box and isn’t really anything to consider. This message is a shot on the dark to see if someone have an ideia of what to, or at least start a debugging process of the issue.

Thanks,
V.