Slow speed with Mellanox IS5023 Switch and Connect-X3 cards

[Hans Petter Selasky]

On 11/30/16 20:18, Vinícius Ferrão wrote:
> Hello guys,
>
> I’m testing a server running FreeBSD 10.3-RELEASE and a Connect-X3 card. When communicating with other hosts in a point-to-point connections everything works as expected. But when I put the FreeBSD server on tem switching fabric the performances drops to a unusable state. In this setup we are using IPoIB with NFS on top of it.
>
> Initial benchmarks with dd returned this numbers:
>
> [root at node005 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=100
> 100+0 records in
> 100+0 records out
> 13107200 bytes (13 MB) copied, 3.96714 s, 3.3 MB/s
>
> This happens only on the FreeBSD machine. If we test Linux to Linux speeds through the same switch we got a very reasonable performance:
>
> [root at node005 home]# dd if=/dev/zero of=./dd1.dd bs=128k count=100
> 100+0 records in
> 100+0 records out
> 13107200 bytes (13 MB) copied, 0.0463456 s, 283 MB/s
>
> And finally if we remove the switch from the setup, and link the FreeBSD machine directly with a CentOS 7.2 client the throughput is exactly what we expect:
>
> [root at node010 scratch]# dd if=/dev/zero of=./dd1.dd bs=128k count=1000
> 1000+0 records in
> 1000+0 records out
> 131072000 bytes (131 MB) copied, 0.640333 s, 205 MB/s
>
> This Mellanox Switch is an unmanaged box and isn’t really anything to consider. This message is a shot on the dark to see if someone have an ideia of what to, or at least start a debugging process of the issue.
>
> Thanks,
> V.

Hi,

What are the link speeds negoitiated towards the switch?

--HPS