NFS on ZFS pure SSD pool

[Garrett Wollman]

<<On Wed, 28 Aug 2013 16:49:43 -0400 (EDT), Rick Macklem <rmacklem at uoguelph.ca> said:

> Eric Browning wrote:
>> Sam and I applied the patch (kernel now at r254983M) and set
>> vfs.nfsd.tcphighwater=5000 in sysctl.conf and my CPU is still
>> slammed. SHould I up it to 10000?
>> 
> You can try. I have no insight into where this goes, since I can't
> produce the kind of server/load where it makes any difference. (I have
> single core i386 (P4 or similar) to test with and I don't use ZFS at all.)
> I've cc'd Garrett Wollman, since he runs rather large servers and may
> have some insight into appropriate tuning, etc.

10,000 is probably way too small.  We run high-peformance servers with
vfs.nfsd.tcphighwater set between 100k and 150k, and we crank
vfs.nfsd.tcpcachetimeo down to five minutes or less.

Just to give you an idea of how rarely this cache is actually hit: my
two main production file servers have both been up for about three
months now, and have answered billions of requests (enough for the
32-bit signed statistics counters to wrap).  One server shows 63 hits,
with a peak TCP cache size of 150k and the other shows zero, with a
peak cache size of 64k.  Another server, which serves scratch space,
has been up for a little more than a month, and in nearly two billion
accesses has yet to see a single cache hit (peak cache size 131k,
which was actually hitting the configured limit, which I've since
raised).

-GAWollman