Terrible NFS performance under 9.2-RELEASE?

[Rick Macklem]

J David wrote:
> A few questions as I try to parse the various patches floating around
> for this.
> 
> What's the difference between src/sys/nfsserver and
> src/sys/fs/nfsserver?  It looks like maybe the former is for NFSv3
> and
> the latter is for NFSv4?
> 
> If so, these patches appear to be for the NFSv4 server.  Since we are
> using the NFSv3 server exclusively, does that mean we would need to
> do
> something similar somewhere in the neighborhood of line 930 of
> src/sys/nfsserver/nfs_serv.c?
> 
> Also, these patches appear server-side.  To make sure things flow
> smoothly, will additional client-side changes be necessary?  There is
> some MGET/MCLGET in src/sys/nfsclient/nfs_subs.c.  (The equivalent in
> src/sys/fs/nfsclient/nfs_clcomsubs.c appear to be using the
> NFSMGET/NFSMCLGET macros, so presumably those are handled?)
> 
> In any case, the switch from 2k to 4k mbufs and m_getm2 seems well
> worthwhile regardless of whether it addresses this specific issue.
>  It
> should reduce a lot of overhead in many common cases.
> 
> If my understanding isn't too far off, I can take a whack at testing
> the result, but only on NFSv3.
> 
> Thanks!
> 
I think Garrett clarified which sources are which. The attached simple
patch makes both the new/default client and new/default server use
MJUMPAGESIZE clusters. (It is the one I already mentioned, called 4kmcl.patch.)

Garrett's patch using m_getm2() would only affect the server side read,
but not client write or server side readdir. (It can probably be combined
with my simple one, but I haven't tested that.)
4kmcl.patch is not ready for head (as John Mark-Gurney pointed out, it does
4K clusters for readlink and it also does 4K clusters for all the small RPC
messages), but it works ok for testing to see if it gets rid of the drops
and calls to m_collapse().

Since you are using 9.2-release, you have the DRC changes. At some point,
you can try setting these in the server (they reduce CPU overheads by allowing
the DRC to grow, holding onto more mbufs). Btw, head (and I think stable/9,10)
have been significantly changed by Alexander Motin's recent commits, although
these sysctls still exist.
vfs.nfsd.tcphighwater=100000
vfs.nfsd.tcpcachetimeout=600

rick