nfsd server cache flooded, try to increase nfsrc_floodlevel

[Rick Macklem]

replying to my own message:
> Mike Shultz wrote:
> > I ran into an issue today of our server thinking that it was being
> > flooded and locking our nfs users out. Got a LOT of these messages:
> >
> > Jul 12 16:08:22 xxxxx kernel: nfsd server cache flooded, try to
> > increase
> > nfsrc_floodlevel
> >
> > Our server(`uname -a`): FreeBSD xxxxx 8.2-RELEASE-p2 FreeBSD
> > 8.2-RELEASE-p2 #0: Tue Jun 21 16:52:27 MDT 2011
> > yyy at xxxxx:/usr/obj/usr/src/sys/XXXXX amd64
> >
> > I could find no information on nfsrc_floodlevel other than source
> > code
> > which didn't explain too much about it. I don't know if it's a
> > kernel
> > config var, or what.
> >
> > `nfsstat -e` did show this:
> >
> > CacheSize TCPPeak
> > 16385 16385
> >
> > So I'm guessing that that is the current cache limit.
> >
> > The source code and this output suggest that we're just running into
> > the
> > limit. However, a comment in that source does suggest that "The
> > cache
> > will still function over flood level" but that doesn't seem to be
> > the
> > case. I ended up having to revoke the clients and restarting nfsd to
> > get
> > it operational again.
> >
> Since you were seeing the messages "...try increasing flood level" it
> means
> that at least some of your client(s) are using NFSv4. For NFSv4, the
> client
> will get NFS4ERR_RESOURCE back as a reply at this point. My guess is
> that
> the client(s) just kept sending retries of the RPCs and, since the
> cache size
> didn't decrease, just kept getting NFS4ERR_RESOURCE.
> 
> The real question becomes "how did it hit the flood level?".
> 
> Hmm, there was a recent SMP related cache problem that is fixed by
> this patch:
> http://people.freebsd.org/~rmacklem/cache.patch
> 
> I'd suggest you try this patch and see if the problem occurs again.
> 
> It seems unlikely you would hit the flood level unless there is a bug
> (or very
> weird client behaviour), but it's conceivable.
> 
> You can increase it by editting sys/fs/nfs/nfs.h and increasing the
> value, then
> rebuilding a kernel/modules.
> 
> Also, what client(s) are mounting the server and how many/how busy are
> they?
> 
> Hopefully the SMP patch will fix this for you, although it's hard to
> predict
> what behaviour could be observed without the patch. (Btw, the patch is
> in head
> and stable/8, but not releng 8.2.) I only have single core hardware
> for testing,
> so I'd never see these kinds of bugs myself.
> 
> > I would appreciate anyone that could clarify what nfsrc_floodlevel
> > is
> > and how to go about changing it.
> >
> This is mostly a "sanity check" and it's hard to imagine hitting the
> limit
> of 16K. To hit this without some sort of bug or client/server
> interoperability
> issue would take something like 4000 TCP mounts against the server.

Oops, my error here. For NFSv4 (I was thinking NFSv3 over TCP mounts), the
cache can grow much larger per mount, because it needs to hold onto the
last reply for each open_owner (this depends upon the client, but if you
think of an open_owner as a process that opened at least one file on the
server, that would be the FreeBSD NFSv4 client).

Do an "nfsstat -e -s" and compare the "Cachesize" with the "OpenOwner"
count. If they are about the same, I suspect you may be hitting the limit
because of this and you'll need to bump it up. The OpenOwners don't go away
for a while (that's a whole other topic), so the cached last replies are stuck
for a while, too.

In summary:
Since you are using NFSv4 mounts, you could hit the 16K flood level with far
fewer mounts than the 4000 mentioned above. I can see 200+ cached entries for
each NFSv4 mount for some situations. (For NFSv3 over TCP, I guessed 4, so you
can see how big the difference is.)

A patch to add a sysctl to change the flood level would be easy to add, if
it becomes apparent that bumping this up is something you need to do routinely.
(Changing it "on the fly" should be safe, but you'd need to use a debugger to
 do that now.)

rick
ps: If it just needs increasing then "Congratulations, you are the first site
    I know of to use NFSv4 a significant amount".:-)