NFS on 10G interfaces still painfully slow

[Gerrit Kühn]

On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers <asomers at freebsd.org> wrote
about Re: NFS on 10G interfaces still painfully slow:

> > Is there anyone around here who can confirm that nfs can go faster
> > over 10G links?
> > Any hints for further tuning/debugging are greatly appreciated.

AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow.
AS> There's probably something about your setup that's slowing it down.
AS> What is your NFS client?

This time, FreeBSD 10.3 on both client and server (to make debugging
easier).

AS>  What benchmark are you using to measure that speed?

Right now only very simple things like using dd from /dev/zero or copying
large files. In my experience it is useless to go for more sophisticated
benchmarks, if these simple things already don't work as expected.

AS>  Did you remember to start lockd and statd? 

Yes.

AS> If you post your /etc/exports and the client's /etc/fstab, that might
AS> reveal something.

exports on the server side:

V4: /mt-rear    -sec=sys 192.168.1.11
/mt-right       192.168.1.11 -maproot=root 
/mt-rear        -maproot=root 192.168.1.11 
/mt-left        192.168.1.11 -maproot=root 


fstab on the client does not tell you anything, I still use commandline
mounts during testing. This is what nfsstat -m will tell (V4 is not
mounted right now):

tom:/mt-rear on /net/mt-rear
nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2
tom:/mt-right on /net/mt-right
nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2
tom:/mt-left on /net/mt-left
nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2


This is what, e.g., dd gives:

root at crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec)

root at crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k
count=1000 1000+0 records in
1000+0 records out
1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec)


And yes (before that question pops up :-), I'm using zfs on the server
side, but I disabled syncing for testing purposes.


cu
  Gerrit