nfsd server cache flooded, try to increase nfsrc_floodlevel

[Rick Macklem]

It's me again:
> I ran into an issue today of our server thinking that it was being
> flooded and locking our nfs users out. Got a LOT of these messages:
> 
> Jul 12 16:08:22 xxxxx kernel: nfsd server cache flooded, try to
> increase
> nfsrc_floodlevel
> 
> Our server(`uname -a`): FreeBSD xxxxx 8.2-RELEASE-p2 FreeBSD
> 8.2-RELEASE-p2 #0: Tue Jun 21 16:52:27 MDT 2011
> yyy at xxxxx:/usr/obj/usr/src/sys/XXXXX amd64
> 
> I could find no information on nfsrc_floodlevel other than source code
> which didn't explain too much about it. I don't know if it's a kernel
> config var, or what.
> 
> `nfsstat -e` did show this:
> 
> CacheSize TCPPeak
> 16385 16385
> 
> So I'm guessing that that is the current cache limit.
> 
> The source code and this output suggest that we're just running into
> the
> limit. However, a comment in that source does suggest that "The cache
> will still function over flood level" but that doesn't seem to be the
> case. I ended up having to revoke the clients and restarting nfsd to
> get
> it operational again.
> 
I've created a patch that gets rid of open_owners (and their associated
cached replies) agressively when the count hits about 90% of nfsrc_floodlevel.
A quick test here indicates it allows the server to "recover" without hitting
the flood gate.

Note that this is allowed by the NFSv4.0 RFC, so the change doesn't break the
protocol.

Please try the patch, which is at:
   http://people.freebsd.org/~rmacklem/noopen.patch
(This patch is against the file in -current, so patch may not like it, but
 it should be easy to do by hand, if patch fails.)

Again, good luck with it and please let me know how it goes, rick