How to change between Connected Mode and Datagram Mode with IPoIB?
ferrao at vmio.com.br
Mon Dec 5 17:08:13 UTC 2016
Continuing my journey on Infiniband on FreeBSD I’ve discovered that some bugs and performance issues appears to be a mismatch on CM and UD modes of IPoIB.
I was able to raise the performance of NFS to 750MB/s changing the Linux connection mode to CM. After that the MTU changed from 2044 to 65520 and I saw some good values. But another annoyances appeared on Linux side. A sluggish performance when running simple commands like “ls” on NFS over IB mounts rendered the solution inviable.
So I’m considering running FreeBSD in UD (Datagram) mode to see if I finally can get rid of the problems. But, how can I change between CM and UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set from 0 to 1 and change modes.
Some old forum posts mentions a recompile needed, disabling "options IPOIB_CM” on kernel configuration just to change modes, which is pretty inviable, and it doesn’t seems right.
So how can we achieve changing between CM and UD modes?
And one last question: how it was even working (badly) but working when a side is talking in CM and the other side on UD mode?
Thanks in advance,
More information about the freebsd-infiniband