NFS + Infiniband problem
avv314 at gmail.com
Thu Dec 6 02:21:21 UTC 2018
Back to the thread after month.
Some prehistory of this case:
>>>> I have a several FreeBSD machines connected via Infiniband netwok (
FDR switch Mellanox SW3036 + ConnectX-3 VPI cards ).
>>>> One of them is a NAS-server with multiply ZFS pools.
>>>>All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried)
on server) are with infiniband connected mode (option IPOIB_CM, option SDM)
and world with >>>> OFED stack support. (WITH_OFED='yes').
>>>>File transfers via FTP or SSH between server and clients works almost
flawless ( ~ 12 Gbit/s ).
>>>>But when I try to copy in/out some significant data via NFS share
mounted on clients, NFS i/o hangs at all or got extremely slow (couple
kB/s) transfer speed after uncertain amount of copied data. For example, on
the one node I can copy 1GB file, and after NFS hang on file with size 30 k
>чт, 1 нояб. 2018 г. в 04:46, Andrew Vylegzhanin <avv314 at gmail.com>:
>> чт, 1 нояб. 2018 г. в 3:27, Rick Macklem <rmacklem at uoguelph.ca>:
>>Also need to test setup with infniband set from connected mode to
Tests for datagram mode IPoIB give disappointing results for NFS: just 20 -
40 MB/s for in average for NFS seq reads and even less for writes.
And inifniband interface report significant errors on:
# netstat -nI ib0 1
input ib0 output
packets errs idrops bytes packets errs bytes colls
11267 0 0 1172852 20471 652 42046744 0
15628 0 0 1626860 28387 994 58257440 0
16920 0 0 1761896 30832 1065 63196256 0
13566 0 0 1410424 24882 722 51205964 0
17942 0 0 1867312 32652 1114 67164484 0
9443 0 0 982104 17340 525 35610908 0
Similar transfer speed and errors were got for other protocols: ftp, scp .
I've tried to mount NFS share via UDP also, but no success.
With UDP, NFS operations are hangs with MTU errors on IB interface: kernel:
ib0: packet len 4096 (> 2044) too long to send, dropping.
So, in short summary, IPoIB actually inoperable in datagram mode.
More information about the freebsd-infiniband