Understanding the vfs.nfsd.request_space* sysctls

[Rick Macklem]

Alan Somers wrote:
>Could somebody help me understand the meaning of the various
>vfs.nfsd.request_space* sysctls, and what implication they have for tuning
>NFS?
>
>I have several NFS servers.  Most are performing well, but one with a
>metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAttr
>and Lookup ops) is not.  Total throughput is a not-unreasonable several
>hundred MB/s, but some clients are limited to very slow speeds, sometimes
>writing at < 10 MB/s.
>
>The request_space sysctls show some pretty stark divergence between the
>well-performing and poor-performing servers:
>
>sysctl                       well-performing poor-performing
>request_space_used                  < 6 M                  varies, but
>currently 40 M
>request_space_used_highest     < 39 M                24 G
>request_space_throttle_count  0                         35
>
>So what does request_space_used measure, anyway?
It's the number of bytes of request message(s) received,
but not yet being processed by nfsd threads.
(They're actually in sys/rpc/svc.c, although named for the
 nfs server.)

Anytime used > high, it throttles, which means it leaves
the RPC messages on the socket receive queue.
This indicates the server is not keeping up with requests.
(ie Overloaded or ???)

>  And how can I either
>increase the available space or decrease the stuff that's using it?
Increasing vfs.nfsd.request_space_high avoids the throttling,
but it is hard to say that is a good idea, since there won't be
backpressure applied to the clients via TCP windows, etc.

Fixing the server so that it isn't overloaded would be better,
I think?
--> I'm the last guy to take ZFS advice from, but I think there
      is a tunable w.r.t. how much arc is used for metadata.
      Getattrs will need cached (metadata) to reply quickly.
      Lookups also depend on cached attributes for good
      perf. as well.
--> Make sure you have lots of nfsd threads. They are
      just kernel threads, so they don't use a lot of resources
      and having too many is much better than not enough.
      -->I can't remember what the upper limit is these days,
           but I'd set it to that for a busy nfs server.
           For NFSv4.1, each client can send a maximum of
           session_slot_table_size concurrent RPCs. FreeBSD
           uses a single 64slot table for each mount. I think
           Linux uses a 32slot table, but supports trunking.
           I don't know if the Linux client uses a separate
           session table for each trunk or not?
           --> Something like #clients * 32 (or 64) nfsd
                 threads running on the server.

rick

-Alan
_______________________________________________
freebsd-fs at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"