Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2

[Rick Macklem]

Berend de Boer wrote:
> >>>>> "Berend" == Berend de Boer <berend at pobox.com> writes:
> 
>     Berend> 1. Does not effect nfs3.
> 
> One update: with zfs sync=standard, I can get nfs3 to perform as bad
> as the nfs4+patch.
> 
Hmm, this is interesting. ken@'s file handle affinity patch works for
NFSv3, but not NFSv4. If I understood his posts correctly, the fh affinity
patch was needed, so ZFS's heuristic for recognizing sequential reading would
function correctly. (A file handle affinity patch for NFSv4 will take some time,
since all RPCs in NFSv4 are compounds, with reads/writes imbedded in them, along
with other ops.)

If you could do some testing where you export a UFS volume, the results might
help to isolate the issue to ZFS vs nfsd.

> 1. without nolock, without async, sync=standard: 8m21s
> 
> 2. with nolock, without async, sync=standard: 5m37s
> 
> 3. without nolock, with async, sync=standard: 4m56s.
> 
> 4. with nolock, with async, sync=standard: 4m23s.
> 
> 5. without nolock, without async, sync=disabled: 1m57.
> 
> 
> PS: the nfs4 test was done with sync=disabled.
> 
W.r.t. CPU overheads and the size of vfs.nfsd.tcphighwater.
The size of the NFSRVCACHE_HASHSIZE was increased to 500 by
the patch. 500 seems about right for a vfs.nfsd.tcphighwater
of a few thousand.

If you are going to use a very large value (100000), I'd suggest
you increase NFSRVCACHE_HASHSIZE to something like 10000. (You have
to edit sys/fs/nfs/nfsrvcache.h and recompile to change this.)

I'd suggset you try something like:
vfs.nfsd.tcphighwater=5000
vfs.nfsd.tcpcachetimeo=300 (5 minutes instead of 12hrs)

Also, I can't remember if you've bumped up the # of nfsd threads,
but I'd go for 256.
nfs_server_flags="-u -t -n 256"
- in your /etc/rc.conf

I never use ZFS, so I can't help w.r.t. ZFS tuning, rick

> 
> --
> All the best,
> 
> Berend de Boer
> 
> 
>           ------------------------------------------------------
>           Awesome Drupal hosting: https://www.xplainhosting.com/
>