Understanding the vfs.nfsd.request_space* sysctls

[Alan Somers]

Could somebody help me understand the meaning of the various
vfs.nfsd.request_space* sysctls, and what implication they have for tuning
NFS?

I have several NFS servers.  Most are performing well, but one with a
metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAttr
and Lookup ops) is not.  Total throughput is a not-unreasonable several
hundred MB/s, but some clients are limited to very slow speeds, sometimes
writing at < 10 MB/s.

The request_space sysctls show some pretty stark divergence between the
well-performing and poor-performing servers:

sysctl                       well-performing poor-performing
request_space_used                  < 6 M                  varies, but
currently 40 M
request_space_used_highest     < 39 M                24 G
request_space_throttle_count  0                         35

So what does request_space_used measure, anyway?  And how can I either
increase the available space or decrease the stuff that's using it?

-Alan