Why is NFSv4 so slow?

[Rick C. Petty]

On Mon, Jun 28, 2010 at 07:56:00AM -0700, Jeremy Chadwick wrote:
> 
> Three other things to provide output from if you could (you can X out IPs
> and MACs too), from both client and server:
> 
> 6) netstat -idn

server:

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
nfe0   1500 <Link#1>      00:22:15:b4:2d:XX 1767890778     0     0 872169302     0     0    0 
nfe0   1500 172.XX.XX.0/2 172.XX.XX.4       1767882158     -     - 1964274616     -     -    - 
lo0   16384 <Link#2>                            3728     0     0     3728     0     0    0 
lo0   16384 (28)00:00:00:00:00:00:fe:80:00:02:00:00:00:00:00:00:00:00:00:00:00:01     3728     0     0     3728     0     0    0 
lo0   16384 (28)00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:01     3728     0     0     3728     0     0    0 
lo0   16384 127.0.0.0/8   127.0.0.1             3648     -     -     3664     -     -    - 

client:

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
re0    1500 <Link#1>      e0:cb:4e:cd:d3:XX 955288523     0     0 696819089     0     0    0 
re0    1500 172.XX.XX.0/2 172.XX.XX.2       955279721     -     - 696814499     -     -    - 
lo0   16384 <Link#2>                            3148     0     0     3148     0     0    0 
lo0   16384 (28)00:00:00:00:00:00:fe:80:00:02:00:00:00:00:00:00:00:00:00:00:00:01     3148     0     0     3148     0     0    0 
lo0   16384 (28)00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:01     3148     0     0     3148     0     0    0 
lo0   16384 127.0.0.0/8   127.0.0.1             3112     -     -     3112     -     -    - 

> 7) sysctl hw.pci | grep msi

both server and client:

hw.pci.honor_msi_blacklist: 1
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1

> 8) Contents of /etc/sysctl.conf

server and client:

# 4 virtual channels
dev.pcm.0.play.vchans=4
# Read modules from /usr/local/modules
kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules
# Remove those annoying ARP moved messages:
net.link.ether.inet.log_arp_movements=0
# 32MB write cache on disk controllers system-wide
vfs.hirunningspace=33554432
# Allow users to mount file systems
vfs.usermount=1
# misc
net.link.tap.user_open=1
net.inet.ip.forwarding=1
compat.linux.osrelease=2.6.16
debug.ddb.textdump.pending=1
# for NFSv4
kern.ipc.maxsockbuf=524288

> > server, immediately after restarting all of nfs scripts (rpcbind
> > nfsclient nfsuserd nfsserver mountd nfsd statd lockd nfscbd):
> >
> > Jun 27 18:04:44 rpcbind: cannot get information for udp6
> > Jun 27 18:04:44 rpcbind: cannot get information for tcp6
> 
> These two usually indicate you removed IPv6 support from the kernel,
> except your ifconfig output (I've remove it) on the server shows you do
> have IPv6 support.  I've been trying to get these warnings removed for
> quite some time (PR kern/96242).  They're harmless, but the
> inconsistency here is a little weird -- are you explicitly disabling
> IPv6 on nfe0?

I have WITHOUT_IPV6= in my make.conf on all my machines (or I have
problems with jdk1.6) and WITHOUT_INET6= in my src.conf.  I'm not sure
why the rpcbind/ifconfig binaries have a different concept than the
kernel since I always "make buildworld kernel" and keep things in sync
with mergemaster when I reboot.  I'm building new worlds/kernels now
to see if that makes any difference.

-- Rick C. Petty