Slow nfsd write performance, tweaks needed

[Rick Macklem]

Mark Schouten wrote:
> Hi,
> 
> 
> I am in the process of switching from a ZFS On Linux-based NFS-server
> to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is
> way superiour over ZoL, and the box serves as storage for a
> virtualizationplatform, so stability is welcome. :)
> 
> 
> The box is stable, but performs terribly. Surely, I'm doing something
> wrong, but I would like some tips and tricks to speed things up.
> 
> 
> 
> 
> Here's my setup:
> CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is
> enabled)
> RAM: 64GB
> NIC: 2x igb in lagg0 (loadbalancing)
Oops again. I don't think igb had the 64K TSO problem (I get the
Intel nics confused;-), but I would definitely suggest trying without
lagg and then disabling jumbo packets, if possible.

Please let us know if you fix the problem, rick

> Disks:
> 
> export1     1.81T   914G   942G    49%  1.00x  ONLINE  -
>   mirror     928G   457G   471G         -
>     da0         -      -      -         -
>     da1         -      -      -         -
>   mirror     928G   457G   471G         -
>     da2         -      -      -         -
>     da3         -      -      -         -
>   mirror    9.94G   173M  9.77G         -
>     da4p1       -      -      -         -
>     da5p1       -      -      -         -
> cache           -      -      -      -      -      -
>   da4p2      223G   223G     8M         -
>   da5p2      223G   223G     8M         -
> 
> 
> da0-3 are 1TB WDs
> da4-5 are 240GB Samsung SSD 840s
> 
> 
> Here's (related) info from rc.conf.
> 
> nfs_server_enable="YES"
> nfs_server_flags="-u -t -n 128"
> rpcbind_enable="YES"
> mountd_enable="YES"
> rpc_lockd_enable="YES"
> rpc_statd_enable="YES"
> 
> 
> 
> 
> I have compression enabled on all the ZFS-filesystems, and
> jumboframes are enabled on the nics.
> 
> 
> As soon as one of the (Linux) clients start to do some IO, NFS
> responsetimes go up bigtime (yesterday up to 13 seconds), while the
> hardware is pretty much idle, I must be doing something very wrong.
> I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is
> appreciated.
> 
> 
> Regards,
> 
> 
> --
> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
> Mark Schouten  | Tuxis Internet Engineering
> KvK: 61527076 | http://www.tuxis.nl/
> T: 0318 200208 | info at tuxis.nl