NFS on 10G interface terribly slow

[Rick Macklem]

Gerrit Kuhn wrote:
> Hi all,
> 
> We have a recent FreeBSD 10.1 installation here that is supposed to act as
> nfs (v3) client to an Oracle x4-2l server running Soalris 11.2.
> We have Intel 10-Gigabit X540-AT2 NICs on both ends, iperf is showing
> plenty of bandwidth (9.xGB/s) in both directions.
> However, nfs appears to be terribly slow, especially for writing:
> 
> root at crest:~ # dd if=/dev/zero of=/net/hellpool/Z bs=1024k count=1000
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes transferred in 20.263190 secs (51747824 bytes/sec)
> 
Recent commits to stable/10 (not in 10.1) done by Alexander Motin (mav@)
might help w.r.t. write performance (it avoids large writes doing synchronous
writes when the wcommitsize is exceeded). If you can try stable/10, that
might be worth it.

Otherwise, the main mount option you can try is "wcommitsize", which you
probably want to make larger.
(It sounds like you already tried most of what I could suggest.)

> 
> Reading appears to be faster, but still far away from full bandwidth:
> 
> root at crest:~ # dd of=/dev/null if=/net/hellpool/Z bs=1024k
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes transferred in 5.129869 secs (204406000 bytes/sec)
> 
You could try increasing readahead. Look for the mount option and try
cranking it up to 8 or 16.

Good luck with it, rick

> 
> We have already tried to tune rsize/wsize parameters, but they appear to
> have little (if any) impact on these results. Also, neither stripping down
> rxsum, txsum, tso etc. from the interface nor increasing MTU to 9000 for
> jumbo frames did improve anything.
> It is quite embarrassing to achieve way less than 1GBE performance with
> 10GBE equipment. Are there any hints what else might be causing this (and
> how to fix it)?
> 
> 
> cu
>   Gerrit
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>