NFS on ZFS pure SSD pool

Tue Aug 27 17:24:18 UTC 2013

Hello, first time posting to this list.  I have a new server that is not
living up to the promise of SSD speeds and NFS is maxing out the CPU.  I'm
new to FreeBSD but I've been reading up on it as much as I can.  I have
obscured my IP addresses and hostname with x's so just ignore that.
Server has about 200 users on it each draing under 50Mb/s peak sustained
around 1-2Mb/s

I've followed some network tuning guides for our I350t4 nic and that has
helped with network performance somewhat but the server is still
experiencing heavy load with pegging the CPU at 1250% on average with only
50Mb/s of traffin in/out of the machine. All of the network tuning came
from https://calomel.org/freebsd_network_tuning.html since it was relevant
to the same nic that I have.

Server Specs:
FreeBSD 9.1
16 cores AMDx64
64GB of ram
ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe
    Intel DC S3500 for ZIL and enabling/disabling has made no difference
    Used a spare DC S3700 for the ZIL and that made no difference either.
NFS v3 & v4 for Mac home folders whose Cache fodler is redirected.

I've tried:
Compression on/of <-- no appreciable difference
Deduplication on/off <-- no appreciable difference
sync=disabled and sync=standard <-- no appreciable difference
setting arc cache to 56GB and also to 32GB <-- no difference in performance
in terms of kern.

I've tried to follow the freebsd tuning guide:
https://wiki.freebsd.org/ZFSTuningGuide to no avail either.  I've read
everything I can find on NFS on ZFS and nothing has helped.  WHere am I
going wrong?

Here's /boot/loader:
[quote]
# ZFS tuning tweaks
aio_load="YES"   # Async IO system calls
autoboot_delay="10"   # reduce boot menu delay time from 10 to 3 seconds
vfs.zfs.arc_max="56868864000"   # Reserves 10GB or ram for system, leaves
56GB for ZFS
vfs.zfs.cache_flush_disable="1"
#vfs.zfs.prefetch_disble="1"
vfs.zfs.write_limit_override="429496728"

kern.ipc.nmbclusters="264144"      # increase the number of network mbufs
kern.maxfiles="65535"
net.inet.tcp.syncache.hashsize="1024"   # Size of the syncache hash table
net.inet.tcp.syncache.bucketlimit="100" # Limit the number of entries
permitted in each bucket of the hash table.
net.inet.tcp.tcbhashsize="32768"

# Link Aggregation loader tweaks. see:
https://calomel.org/freebsd_network_tuning.html
hw.igb.enable_msix="1"
hw.igb.num_queues="0"
hw.igb.enable_aim="1"
hw.igb.max_interrupt_rate="32000"
hw.igb.rxd="2048"
hw.igb.txd="2048"
hw.igb.rx_process_limit="4096"
if_lagg_load="YES"
[/quote]

Here's etc/sysctl.conf:
[quote]
# $FreeBSD: release/9.1.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes
that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.ipc.somaxconn=1024
kern.maxusers=272
#kern.maxvnodes=1096848 #increase this if necessary
kern.ipc.maxsockbuf=8388608
net.inet.tcp.mssdflt=1460
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
dev.igb.2.fc=0
dev.igb.3.fc=0
dev.igb.4.fc=0
dev.igb.5.fc=0
dev.igb.2.rx_procesing_limit=10000
dev.igb.3.rx_procesing_limit=10000
dev.igb.4.rx_procesing_limit=10000
dev.igb.5.rx_procesing_limit=10000
net.inet.ip.redirect=0
net.inet.icmp.bmcastecho=0      # do not respond to ICMP packets sent to IP
.255
net.inet.icmp.maskfake=0        # do not fake reply to ICMP Address Mask
Request packets
net.inet.icmp.maskrepl=0        # replies are not sent for ICMP address mask
net.inet.icmp.log_redirect=0    # do not log redirected ICMP packet attempts
net.inet.icmp.drop_redirect=1   # no redirected ICMP packets
net.inet.tcp.drop_synfin=1      # SYN/FIN packets get dropped on initial
connection
net.inet.tcp.ecn.enable=1       # explicit congestion notification (ecn)
warning: some ISP routers abuse it
net.inet.tcp.icmp_may_rst=0     # icmp may not send RST to avoid spoofed
icmp/udp floods
net.inet.tcp.maxtcptw=15000     # max number of tcp time_wait states for
closing connections
net.inet.tcp.msl=5000           # 5 second maximum segment life waiting for
an ACK in reply to a SYN-ACK or FIN-ACK
net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since most ICMP
packets are dropped by others
net.inet.tcp.rfc3042=0          # disable the limited transmit mechanism
which can slow burst transmissions
net.inet.ip.rtexpire=60         # 3600 secs
net.inet.ip.rtminexpire=2       # 10 secs
net.inet.ip.rtmaxcache=1024     # 128 entries
[/quote]

Here's /etc/rc.conf
[quote]
#ifconfig_igb2=" inet xxx.xx.x.xx netmask 255.255.248.0"
hostname="xxxxxxxxxxxxxxxxxxx"
#
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
#
### LACP config
ifconfig_igb2="up"
ifconfig_igb3="up"
ifconfig_igb4="up"
ifconfig_igb5="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport igb4
laggport igb5 xxx.xx.x.xx netmask 255.255.248.0"
ipvr_addrs_lagg0="xxx.xx.x.xx"
defaultrouter="xxx.xx.x.xx"
#
### Defaults for SSH, NTP, ZFS
sshd_enable="YES"
ntpd_enable="YES"
zfs_enable="YES"
#
## NFS Server
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r -l"
nfsd_enable="YES"
mountd_enable="YES"
rpc_lockd_enable="NO"
rpc_statd_enable="NO"
nfs_server_flags="-u -t -n 128"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
[/quote]

Thanks in advance,
-- 
Eric Browning