NFS + Infiniband problem

[Andrew Vylegzhanin]

Hello everyone,

I have a several FreeBSD machines connected via Infiniband netwok ( FDR
switch Mellanox SW3036 + ConnectX-3 VPI cards ).
One of them is a NAS-server with multiply ZFS pools.

All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on
server) are with infiniband connected mode (option IPOIB_CM, option SDM)
and world with OFED stack support. (WITH_OFED='yes').

File transfers via FTP or SSH between server and clients works almost
flawless ( ~ 12 Gbit/s ).

But when I try to copy in/out some significant data via NFS share mounted
on clients, NFS i/o hangs at all or got extremely slow (couple kB/s)
transfer speed after uncertain amount of copied data. For example, on the
one node I can copy 1GB file, and after NFS hang on file with size 30 kb.

Some details:
[root at node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt
[root at node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024

Ctrl-T for "hang" dd
load: 0.01  cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k
load: 0.01  cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k

for "slow" dd
load: 0.00  cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k

load: 0.00  cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k

I've tried mount with different wsize option with same result.

Any help would be greatly appreciated.

--
Andrew