Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2

[Rick Macklem]

Berend de Boer wrote:
> Hi All,
> 
> I've just completed a round of NFS testing on FreeBSD 9.1 on AWS. The
> underlying file system is ZFS.  I have a real nfs killer test: doing
> an "svn update" of a directory of 3541 files.
> 
> Performance on the NFS server itself is good: checking this out takes
> 11 seconds (doing the same on a not really comparable Linux NFS
> server
> is 43 seconds).
> 
> Doing this on a client writing to an NFS mounted home directory is
> however terrible. Really terrible. This takes 25 minutes! With
> "sync=disable" it's 16 minutes.
> 
> (doing this against the underpowered Linux NFS server is about 4.5
> minutes).
> 
> The problem might be that the NFS server (nfsd) runs at 70-80% CPU.
> So
> my writing speed is cpu bound (go figure).
> 
See below w.r.t. patch that reduces cpu overheads in the DRC.

> Repeating this with nfs4 + udp: doesn't work at all, get input/output
> error after a few files.
> 
The RFCs for NFSv4 require use of transport protocols that include
congestion control --> no UDP support. What client are you using?
(If it is a FreeBSD one, I need to patch it to make the mount fail.
 Since NFSv4 over UDP was done in early testing, the client might
 still have that bogus code in it.)

> nfs3: 1m55s (nfsd cpu is in the 6%-8% range)
> 
> Varying tcp/udp, lock/nolock, sync/async, or enabling zfs
> sync=standard doesn't make any particular difference. All timing were
> within the 1m49s - 2m3s range.
> 
> So what's up with NFS4 on FreeBSD?
> 
Please try this patch:
  http://people.freebsd.org/rmacklem/~drc4.patch
  - After you apply the patch and boot the rebuilt kernel, the cpu
    overheads should be reduced after you increase the value of
    vfs.nfsd.tcphighwater. The larger you make it, the more space
    (mbuf clusters and other kernel malloc'd data structures) the DRC
    uses, but hopefully with reduced CPU overheads.
The plan is to commit a patch semantically similar to this to head and
then MFC it someday. ivoras@ has a similar patch, but written in cleaner
C. However, I've never gotten around to combining the patches into a
version for head. Someday I'll get to it, but not in time for 9.2.

Good luck with it, rick
ps: There is also ken@'s file handle affinity patch, which is in head
    (and I think stable/9), but it only works for NFSv3 at this point.
    Hopefully we'll come up with a patch for NFSv4 for it someday.

> --
> All the best,
> 
> Berend de Boer
> 
> 
>           ------------------------------------------------------
>           Awesome Drupal hosting: https://www.xplainhosting.com/
>