How to change between Connected Mode and Datagram Mode with IPoIB?

[Vinícius Ferrão]

Hello guys,

Continuing my journey on Infiniband on FreeBSD I’ve discovered that some bugs and performance issues appears to be a mismatch on CM and UD modes of IPoIB.

I was able to raise the performance of NFS to 750MB/s changing the Linux connection mode to CM. After that the MTU changed from 2044 to 65520 and I saw some good values. But another annoyances appeared on Linux side. A sluggish performance when running simple commands like “ls” on NFS over IB mounts rendered the solution inviable.

So I’m considering running FreeBSD in UD (Datagram) mode to see if I finally can get rid of the problems. But, how can I change between CM and UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set from 0 to 1 and change modes.

Some old forum posts mentions a recompile needed, disabling "options IPOIB_CM” on kernel configuration just to change modes, which is pretty inviable, and it doesn’t seems right.

So how can we achieve changing between CM and UD modes?

And one last question: how it was even working (badly) but working when a side is talking in CM and the other side on UD mode?

Thanks in advance,
V.