NFS on 10G interface terribly slow

[Gerrit Kühn]

Hi all,

We have a recent FreeBSD 10.1 installation here that is supposed to act as
nfs (v3) client to an Oracle x4-2l server running Soalris 11.2.
We have Intel 10-Gigabit X540-AT2 NICs on both ends, iperf is showing
plenty of bandwidth (9.xGB/s) in both directions.
However, nfs appears to be terribly slow, especially for writing:

root at crest:~ # dd if=/dev/zero of=/net/hellpool/Z bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 20.263190 secs (51747824 bytes/sec)


Reading appears to be faster, but still far away from full bandwidth:

root at crest:~ # dd of=/dev/null if=/net/hellpool/Z bs=1024k 
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.129869 secs (204406000 bytes/sec)


We have already tried to tune rsize/wsize parameters, but they appear to
have little (if any) impact on these results. Also, neither stripping down
rxsum, txsum, tso etc. from the interface nor increasing MTU to 9000 for
jumbo frames did improve anything.
It is quite embarrassing to achieve way less than 1GBE performance with
10GBE equipment. Are there any hints what else might be causing this (and
how to fix it)?


cu
  Gerrit