Understanding the vfs.nfsd.request_space* sysctls

[Rick Macklem]

Alan Somers wrote:
>Rick Macklem wrote:
[stuff snipped]
>>Anytime used > high, it throttles, which means it leaves
>>the RPC messages on the socket receive queue.
>>This indicates the server is not keeping up with requests.
>>(ie Overloaded or ???)
>
>That all makes sense.  So having a high request_space_used basically just means that >my storage is too slow.
One other thing you might want to look at is rcv interrupt moderation
in the net device driver.
--> This *might* result in received RPC requests being "batched", resulting in
      a burst of RPCs that overloads the server. (Except for write requests, each
      RPC is a small message typically in a single small packet.)
      --> Disabling rcv interrupt moderation will result in more interrupts, but
             then each RPC request will be handled right away. This might result
             in more evenly spaced RPCs (in a temporal sense).
             Also, delaying processing of RPC requests increases RPC RTT and that
             always has a negative impact on NFS performance.

rick

>  And how can I either
>increase the available space or decrease the stuff that's using it?
Increasing vfs.nfsd.request_space_high avoids the throttling,
but it is hard to say that is a good idea, since there won't be
backpressure applied to the clients via TCP windows, etc.

Fixing the server so that it isn't overloaded would be better,
I think?
--> I'm the last guy to take ZFS advice from, but I think there
      is a tunable w.r.t. how much arc is used for metadata.
      Getattrs will need cached (metadata) to reply quickly.
      Lookups also depend on cached attributes for good
      perf. as well.

I've added an L2ARC since I sent that original email.  We'll see how well it works.  I expect it to take 48 hours before results are apparent.

--> Make sure you have lots of nfsd threads. They are
      just kernel threads, so they don't use a lot of resources
      and having too many is much better than not enough.
      -->I can't remember what the upper limit is these days,
           but I'd set it to that for a busy nfs server.
           For NFSv4.1, each client can send a maximum of
           session_slot_table_size concurrent RPCs. FreeBSD
           uses a single 64slot table for each mount. I think
           Linux uses a 32slot table, but supports trunking.
           I don't know if the Linux client uses a separate
           session table for each trunk or not?
           --> Something like #clients * 32 (or 64) nfsd
                 threads running on the server.

Experimentally, I determined that 768 threads is sufficient.  But your formula would suggest > 2500, which is a lot of threads!  I'll keep it in mind if I ever reach 768, though.


rick

-Alan
_______________________________________________
freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"