NFS + Infiniband problem
avv314 at gmail.com
Mon Oct 29 03:14:09 UTC 2018
I have a several FreeBSD machines connected via Infiniband netwok ( FDR
switch Mellanox SW3036 + ConnectX-3 VPI cards ).
One of them is a NAS-server with multiply ZFS pools.
All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on
server) are with infiniband connected mode (option IPOIB_CM, option SDM)
and world with OFED stack support. (WITH_OFED='yes').
File transfers via FTP or SSH between server and clients works almost
flawless ( ~ 12 Gbit/s ).
But when I try to copy in/out some significant data via NFS share mounted
on clients, NFS i/o hangs at all or got extremely slow (couple kB/s)
transfer speed after uncertain amount of copied data. For example, on the
one node I can copy 1GB file, and after NFS hang on file with size 30 kb.
[root at node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt
[root at node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024
Ctrl-T for "hang" dd
load: 0.01 cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k
load: 0.01 cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k
for "slow" dd
load: 0.00 cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k
load: 0.00 cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k
I've tried mount with different wsize option with same result.
Any help would be greatly appreciated.
More information about the freebsd-infiniband