Slow nfsd write performance, tweaks needed

[Mark Schouten]

Hi,


I am in the process of switching from a ZFS On Linux-based NFS-server to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is way superiour over ZoL, and the box serves as storage for a virtualizationplatform, so stability is welcome. :)


The box is stable, but performs terribly. Surely, I'm doing something wrong, but I would like some tips and tricks to speed things up.




Here's my setup:
CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is enabled)
RAM: 64GB
NIC: 2x igb in lagg0 (loadbalancing)
Disks: 

export1     1.81T   914G   942G    49%  1.00x  ONLINE  -
  mirror     928G   457G   471G         -
    da0         -      -      -         -
    da1         -      -      -         -
  mirror     928G   457G   471G         -
    da2         -      -      -         -
    da3         -      -      -         -
  mirror    9.94G   173M  9.77G         -
    da4p1       -      -      -         -
    da5p1       -      -      -         -
cache           -      -      -      -      -      -
  da4p2      223G   223G     8M         -
  da5p2      223G   223G     8M         -


da0-3 are 1TB WDs
da4-5 are 240GB Samsung SSD 840s


Here's (related) info from rc.conf.

nfs_server_enable="YES"
nfs_server_flags="-u -t -n 128"
rpcbind_enable="YES"
mountd_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"




I have compression enabled on all the ZFS-filesystems, and jumboframes are enabled on the nics.


As soon as one of the (Linux) clients start to do some IO, NFS responsetimes go up bigtime (yesterday up to 13 seconds), while the hardware is pretty much idle, I must be doing something very wrong. I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is appreciated.


Regards,


-- 
Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
Mark Schouten  | Tuxis Internet Engineering
KvK: 61527076 | http://www.tuxis.nl/
T: 0318 200208 | info at tuxis.nl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4549 bytes
Desc: Electronic Signature S/MIME
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20141113/f4e9bc98/attachment.bin>