Why is NFSv4 so slow?

[Jeremy Chadwick]

On Mon, Jun 28, 2010 at 10:18:35AM -0500, Rick C. Petty wrote:
> > 8) Contents of /etc/sysctl.conf
> 
> server and client:
> 
> # for NFSv4
> kern.ipc.maxsockbuf=524288

You might want to discuss this one with Rick a bit (I'm not sure of the
implications).  Regarding heavy network I/O (I don't use NFS but Samba),
I've found that the following tunables do in fact make a performance
difference -- you might try and see if these have some impact (or, try
forcing a specific protocol type for NFS, e.g. TCP-only; I'm not
familiar with NFSv4 though).  These are adjustable in sysctl.conf, thus
adjustable in real-time.

# Increase send/receive buffer maximums from 256KB to 16MB.
# FreeBSD 7.x and later will auto-tune the size, but only up to the max.
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216

# Double send/receive TCP datagram memory allocation.  This defines the
# amount of memory taken up by default *per socket*.
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=131072

That's about all I can comment on -- if NFSv3 works OK for you
(performance-wise), then I'm not sure where the bottleneck could be.

> > > Jun 27 18:04:44 rpcbind: cannot get information for udp6
> > > Jun 27 18:04:44 rpcbind: cannot get information for tcp6
> > 
> > These two usually indicate you removed IPv6 support from the kernel,
> > except your ifconfig output (I've remove it) on the server shows you do
> > have IPv6 support.  I've been trying to get these warnings removed for
> > quite some time (PR kern/96242).  They're harmless, but the
> > inconsistency here is a little weird -- are you explicitly disabling
> > IPv6 on nfe0?
> 
> I have WITHOUT_IPV6= in my make.conf on all my machines (or I have
> problems with jdk1.6) and WITHOUT_INET6= in my src.conf.  I'm not sure
> why the rpcbind/ifconfig binaries have a different concept than the
> kernel since I always "make buildworld kernel" and keep things in sync
> with mergemaster when I reboot.  I'm building new worlds/kernels now
> to see if that makes any difference.

make.conf WITHOUT_IPV6 would affect ports, src.conf WITHOUT_INET6 would
affect the base system (thus rpcbind).  The src.conf entry is what's
causing rpcbind to spit out the above "cannot get information" messages,
even though IPv6 is available in your kernel (see below).

However: your kernel configuration file must contain "options INET6" or
else you wouldn't have IPv6 addresses on lo0.  So even though your
kernel and world are synchronised, IPv6 capability-wise they probably
aren't.  This may be your intended desire though, and if so, no biggie.

If you wanted to work around the problem, you can supposedly comment out
the udp6 and tcp6 lines in /etc/netconfig.  I choose not to do this (put
up with the warning messages) since I'm not sure of the repercussions of
adjusting this file (e.g. will something else down the road break).

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |