Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2

[Rick Macklem]

I wrote:
> Berend de Boer wrote:
> > Hi All,
> > 
> > I've just completed a round of NFS testing on FreeBSD 9.1 on AWS.
> > The
> > underlying file system is ZFS.  I have a real nfs killer test:
> > doing
> > an "svn update" of a directory of 3541 files.
> > 
> > Performance on the NFS server itself is good: checking this out
> > takes
> > 11 seconds (doing the same on a not really comparable Linux NFS
> > server
> > is 43 seconds).
> > 
> > Doing this on a client writing to an NFS mounted home directory is
> > however terrible. Really terrible. This takes 25 minutes! With
> > "sync=disable" it's 16 minutes.
> > 
> > (doing this against the underpowered Linux NFS server is about 4.5
> > minutes).
> > 
> > The problem might be that the NFS server (nfsd) runs at 70-80% CPU.
> > So
> > my writing speed is cpu bound (go figure).
> > 
> See below w.r.t. patch that reduces cpu overheads in the DRC.
> 
> > Repeating this with nfs4 + udp: doesn't work at all, get
> > input/output
> > error after a few files.
> > 
> The RFCs for NFSv4 require use of transport protocols that include
> congestion control --> no UDP support. What client are you using?
> (If it is a FreeBSD one, I need to patch it to make the mount fail.
>  Since NFSv4 over UDP was done in early testing, the client might
>  still have that bogus code in it.)
> 
> > nfs3: 1m55s (nfsd cpu is in the 6%-8% range)
> > 
> > Varying tcp/udp, lock/nolock, sync/async, or enabling zfs
> > sync=standard doesn't make any particular difference. All timing
> > were
> > within the 1m49s - 2m3s range.
> > 
> > So what's up with NFS4 on FreeBSD?
> > 
> Please try this patch:
>   http://people.freebsd.org/rmacklem/~drc4.patch
Oops, it actually is:
    http://people.freebsd.org/~rmacklem/drc4.patch

>   - After you apply the patch and boot the rebuilt kernel, the cpu
>     overheads should be reduced after you increase the value of
>     vfs.nfsd.tcphighwater. The larger you make it, the more space
>     (mbuf clusters and other kernel malloc'd data structures) the DRC
>     uses, but hopefully with reduced CPU overheads.
> The plan is to commit a patch semantically similar to this to head
> and
> then MFC it someday. ivoras@ has a similar patch, but written in
> cleaner
> C. However, I've never gotten around to combining the patches into a
> version for head. Someday I'll get to it, but not in time for 9.2.
> 
> Good luck with it, rick
> ps: There is also ken@'s file handle affinity patch, which is in head
>     (and I think stable/9), but it only works for NFSv3 at this
>     point.
>     Hopefully we'll come up with a patch for NFSv4 for it someday.
> 
> > --
> > All the best,
> > 
> > Berend de Boer
> > 
> > 
> >           ------------------------------------------------------
> >           Awesome Drupal hosting: https://www.xplainhosting.com/
> > 
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>