nfsd server cache flooded, try to increase nfsrc_floodlevel

[Rick Macklem]

Zack Kirsch wrote:
> Just wanted to add a bit of Isilon color. We've hit this limit before,
> but I believe it was mostly due to strange client behavior of 1) Using
> a new lockowner for each lock and 2) Using a new TCP connection for
> each 'test run'.
When I saw this before, I remarked that this shouldn't be relevant. I
realize now that you were referring to a test environment (not a real
NFS client) where it keeps creating new TCP connections, even if the
previous connection wasn't broken due to a network partitioning or similar.

Sorry about that.

> As far as I know, we haven't hit this in the field.
> 
It appears that this case was a result of use of an old Linux NFSv4 client
and was resolved via a kernel upgrade. (ie. I suspect there are others out
there that will run into the same thing sooner or later.)

> We've done a few things to combat this problem:
> 1) We increased the floodlevel to 65536.
> 2) We made the floodlevel configurable via sysctl.
> 3) We made significant changes to the replay cache itself. Specific
> gains were drastic performance improvements and freeing of cache
> entries from stale TCP connections.
> 
It is important to note that the request cache holds onto replies for
inactive TCP connections because it assumes that the client might be
network partitioned for long enough that it is forced to reconnect using
a fresh TCP connection and will then retry all outstanding RPCs. This
could take a looonnngggg time to happen, so these replies can't be free'd
quickly, or the whole purpose of the cache (avoiding redoing non-idempotent
operations when an RPC is retried) is defeated.

The fact that some artificial test program (pynfs maybe?) chooses to do
fresh TCP connections isn't relevant imho, since it isn't a real client
and, as far as I know, real clients only reconnect when the old TCP connection
no longer works.

I thought I'd try and clarify this for anyone interested, rick