Understanding the vfs.nfsd.request_space* sysctls

[Alan Somers]

On Tue, Mar 16, 2021 at 6:06 PM Rick Macklem <rmacklem at uoguelph.ca> wrote:

> Alan Somers wrote:
> >Could somebody help me understand the meaning of the various
> >vfs.nfsd.request_space* sysctls, and what implication they have for tuning
> >NFS?
> >
> >I have several NFS servers.  Most are performing well, but one with a
> >metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence,
> GetAttr
> >and Lookup ops) is not.  Total throughput is a not-unreasonable several
> >hundred MB/s, but some clients are limited to very slow speeds, sometimes
> >writing at < 10 MB/s.
> >
> >The request_space sysctls show some pretty stark divergence between the
> >well-performing and poor-performing servers:
> >
> >sysctl                       well-performing poor-performing
> >request_space_used                  < 6 M                  varies, but
> >currently 40 M
> >request_space_used_highest     < 39 M                24 G
> >request_space_throttle_count  0                         35
> >
> >So what does request_space_used measure, anyway?
> It's the number of bytes of request message(s) received,
> but not yet being processed by nfsd threads.
> (They're actually in sys/rpc/svc.c, although named for the
>  nfs server.)
>
> Anytime used > high, it throttles, which means it leaves
> the RPC messages on the socket receive queue.
> This indicates the server is not keeping up with requests.
> (ie Overloaded or ???)
>

That all makes sense.  So having a high request_space_used basically just
means that my storage is too slow.


>
> >  And how can I either
> >increase the available space or decrease the stuff that's using it?
> Increasing vfs.nfsd.request_space_high avoids the throttling,
> but it is hard to say that is a good idea, since there won't be
> backpressure applied to the clients via TCP windows, etc.
>
> Fixing the server so that it isn't overloaded would be better,
> I think?
> --> I'm the last guy to take ZFS advice from, but I think there
>       is a tunable w.r.t. how much arc is used for metadata.
>       Getattrs will need cached (metadata) to reply quickly.
>       Lookups also depend on cached attributes for good
>       perf. as well.
>

I've added an L2ARC since I sent that original email.  We'll see how well
it works.  I expect it to take 48 hours before results are apparent.


> --> Make sure you have lots of nfsd threads. They are
>       just kernel threads, so they don't use a lot of resources
>       and having too many is much better than not enough.
>       -->I can't remember what the upper limit is these days,
>            but I'd set it to that for a busy nfs server.
>            For NFSv4.1, each client can send a maximum of
>            session_slot_table_size concurrent RPCs. FreeBSD
>            uses a single 64slot table for each mount. I think
>            Linux uses a 32slot table, but supports trunking.
>            I don't know if the Linux client uses a separate
>            session table for each trunk or not?
>            --> Something like #clients * 32 (or 64) nfsd
>                  threads running on the server.
>

Experimentally, I determined that 768 threads is sufficient.  But your
formula would suggest > 2500, which is a lot of threads!  I'll keep it in
mind if I ever reach 768, though.


>
> rick
>
> -Alan
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>
>