NFS DRC size

[Rick Macklem]

Garrett Wollman wrote:
> <<On Sat, 9 Mar 2013 11:27:32 -0500 (EST), Rick Macklem
> <rmacklem at uoguelph.ca> said:
> 
> > around the highwater mark basically indicates this is working. If it
> > wasn't
> > throwing away replies where the receipt has been ack'd at the TCP
> > level, the cache would grow very large, since they would only be
> > discarded after a loonnngg timeout (12hours unless you've changes
> > NFSRVCACHE_TCPTIMEOUT in sys/fs/nfs/nfs.h).
> 
> That seems unreasonably large.
> 
I suppose. How long a network partitioning do you want the cache to
deal with? (My original design was trying to achieve a high level of
correctness by default.)

The only time cache entries normally hang around this long is when a
client has dismounted the volume(s) using the TCP connection. The
cached replies for the last few replies will then hang around until
the timeout. For a few clients this isn't an issue. For 2,000 clients,
I can see that it might be, if the clients choose to dismount volumes
(using something like amd).

Feel free to make it smaller, based on the longest network partitioning
that you anticipate might occur.

> > Well, the DRC will try to cache replies until the client's TCP layer
> > acknowledges receipt of the reply. It is hard to say how many
> > replies
> > that is for a given TCP connection, since it is a function of the
> > level
> > of concurrently (# of nfsiod threads in the FreeBSD client)
> > in the client. I'd guess it's somewhere between 1<->20?
> 
> Nearly all our clients are Linux, so it's likely to be whatever Debian
> does by default.
> 
> > Multiply that by the number of TCP connections from all clients and
> > you have about how big the server's DRC will be. (Some clients use
> > a single TCP connection for the client whereas others use a separate
> > TCP connection for each mount point.)
> 
> The Debian client appears to use a single TCP connection for
> everything.
> 
> So if I want to support 2,000 clients each with 20 requests in flight,
> that would suggest that I need a DRC size of 40,000, which my
> experience shows is not sufficient with even a much smaller number of
> clients.
> 
Well, especially since Debian is using one TCP connection for everything
from a client, the guess of 20 could be way low.

rick

> -GAWollman
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"