NFS on ZFS pure SSD pool

Wed Aug 28 20:49:51 UTC 2013

Eric Browning wrote:
> 
> Rick,
> 
> 
> Sam and I applied the patch (kernel now at r254983M) and set
> vfs.nfsd.tcphighwater=5000 in sysctl.conf and my CPU is still
> slammed. SHould I up it to 10000?
> 
You can try. I have no insight into where this goes, since I can't
produce the kind of server/load where it makes any difference. (I have
single core i386 (P4 or similar) to test with and I don't use ZFS at all.)
I've cc'd Garrett Wollman, since he runs rather large servers and may
have some insight into appropriate tuning, etc.

rick

> 
> 
> On Tue, Aug 27, 2013 at 1:29 PM, Rick Macklem < rmacklem at uoguelph.ca
> > wrote:
> 
> 
> 
> 
> Eric Browning wrote:
> > Hello, first time posting to this list. I have a new server that is
> > not
> > living up to the promise of SSD speeds and NFS is maxing out the
> > CPU.
> > I'm
> > new to FreeBSD but I've been reading up on it as much as I can. I
> > have
> > obscured my IP addresses and hostname with x's so just ignore that.
> > Server has about 200 users on it each draing under 50Mb/s peak
> > sustained
> > around 1-2Mb/s
> > 
> > I've followed some network tuning guides for our I350t4 nic and
> > that
> > has
> > helped with network performance somewhat but the server is still
> > experiencing heavy load with pegging the CPU at 1250% on average
> > with
> > only
> > 50Mb/s of traffin in/out of the machine. All of the network tuning
> > came
> > from https://calomel.org/freebsd_network_tuning.html since it was
> > relevant
> > to the same nic that I have.
> > 
> > Server Specs:
> > FreeBSD 9.1
> > 16 cores AMDx64
> > 64GB of ram
> > ZFS v28 with four Intel DC S3700 drives (800GB) as a zfs stripe
> > Intel DC S3500 for ZIL and enabling/disabling has made no
> > difference
> > Used a spare DC S3700 for the ZIL and that made no difference
> > either.
> > NFS v3 & v4 for Mac home folders whose Cache fodler is redirected.
> > 
> > I've tried:
> > Compression on/of <-- no appreciable difference
> > Deduplication on/off <-- no appreciable difference
> > sync=disabled and sync=standard <-- no appreciable difference
> > setting arc cache to 56GB and also to 32GB <-- no difference in
> > performance
> > in terms of kern.
> > 
> > I've tried to follow the freebsd tuning guide:
> > https://wiki.freebsd.org/ZFSTuningGuide to no avail either. I've
> > read
> > everything I can find on NFS on ZFS and nothing has helped. WHere
> > am
> > I
> > going wrong?
> > 
> You could try this patch:
> http://people.freebsd.org/~rmacklem/drc4-stable9.patch
> - After applying the patch and booting a kernel built from the
> patched
> sources, you need to increase the value of vfs.nfsd.tcphighwater.
> (Try something like 5000 for it as a starting point.)
> 
> Although this patch is somewhat different code, it should be
> semantically
> the same as r254337 in head, that is scheduled to be MFC'd to
> stable/9 in
> a couple of weeks.
> 
> rick
> 
> 
> 
> > Here's /boot/loader:
> > [quote]
> > # ZFS tuning tweaks
> > aio_load="YES" # Async IO system calls
> > autoboot_delay="10" # reduce boot menu delay time from 10 to 3
> > seconds
> > vfs.zfs.arc_max="56868864000" # Reserves 10GB or ram for system,
> > leaves
> > 56GB for ZFS
> > vfs.zfs.cache_flush_disable="1"
> > #vfs.zfs.prefetch_disble="1"
> > vfs.zfs.write_limit_override="429496728"
> > 
> > kern.ipc.nmbclusters="264144" # increase the number of network
> > mbufs
> > kern.maxfiles="65535"
> > net.inet.tcp.syncache.hashsize="1024" # Size of the syncache hash
> > table
> > net.inet.tcp.syncache.bucketlimit="100" # Limit the number of
> > entries
> > permitted in each bucket of the hash table.
> > net.inet.tcp.tcbhashsize="32768"
> > 
> > # Link Aggregation loader tweaks. see:
> > https://calomel.org/freebsd_network_tuning.html
> > hw.igb.enable_msix="1"
> > hw.igb.num_queues="0"
> > hw.igb.enable_aim="1"
> > hw.igb.max_interrupt_rate="32000"
> > hw.igb.rxd="2048"
> > hw.igb.txd="2048"
> > hw.igb.rx_process_limit="4096"
> > if_lagg_load="YES"
> > [/quote]
> > 
> > Here's etc/sysctl.conf:
> > [quote]
> > # $FreeBSD: release/9.1.0/etc/sysctl.conf 112200 2003-03-13
> > 18:43:50Z
> > mux $
> > #
> > # This file is read when going to multi-user and its contents piped
> > thru
> > # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for
> > details.
> > #
> > 
> > # Uncomment this to prevent users from seeing information about
> > processes
> > that
> > # are being run under another UID.
> > #security.bsd.see_other_uids=0
> > kern.ipc.somaxconn=1024
> > kern.maxusers=272
> > #kern.maxvnodes=1096848 #increase this if necessary
> > kern.ipc.maxsockbuf=8388608
> > net.inet.tcp.mssdflt=1460
> > net.inet.ip.forwarding=1
> > net.inet.ip.fastforwarding=1
> > dev.igb.2.fc=0
> > dev.igb.3.fc=0
> > dev.igb.4.fc=0
> > dev.igb.5.fc=0
> > dev.igb.2.rx_procesing_limit=10000
> > dev.igb.3.rx_procesing_limit=10000
> > dev.igb.4.rx_procesing_limit=10000
> > dev.igb.5.rx_procesing_limit=10000
> > net.inet.ip.redirect=0
> > net.inet.icmp.bmcastecho=0 # do not respond to ICMP packets sent
> > to IP
> > .255
> > net.inet.icmp.maskfake=0 # do not fake reply to ICMP Address
> > Mask
> > Request packets
> > net.inet.icmp.maskrepl=0 # replies are not sent for ICMP
> > address mask
> > net.inet.icmp.log_redirect=0 # do not log redirected ICMP packet
> > attempts
> > net.inet.icmp.drop_redirect=1 # no redirected ICMP packets
> > net.inet.tcp.drop_synfin=1 # SYN/FIN packets get dropped on
> > initial
> > connection
> > net.inet.tcp.ecn.enable=1 # explicit congestion notification
> > (ecn)
> > warning: some ISP routers abuse it
> > net.inet.tcp.icmp_may_rst=0 # icmp may not send RST to avoid
> > spoofed
> > icmp/udp floods
> > net.inet.tcp.maxtcptw=15000 # max number of tcp time_wait states
> > for
> > closing connections
> > net.inet.tcp.msl=5000 # 5 second maximum segment life
> > waiting for
> > an ACK in reply to a SYN-ACK or FIN-ACK
> > net.inet.tcp.path_mtu_discovery=0 # disable MTU discovery since
> > most
> > ICMP
> > packets are dropped by others
> > net.inet.tcp.rfc3042=0 # disable the limited transmit
> > mechanism
> > which can slow burst transmissions
> > net.inet.ip.rtexpire=60 # 3600 secs
> > net.inet.ip.rtminexpire=2 # 10 secs
> > net.inet.ip.rtmaxcache=1024 # 128 entries
> > [/quote]
> > 
> > Here's /etc/rc.conf
> > [quote]
> > #ifconfig_igb2=" inet xxx.xx.x.xx netmask 255.255.248.0"
> > hostname="xxxxxxxxxxxxxxxxxxx"
> > #
> > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
> > dumpdev="NO"
> > #
> > ### LACP config
> > ifconfig_igb2="up"
> > ifconfig_igb3="up"
> > ifconfig_igb4="up"
> > ifconfig_igb5="up"
> > cloned_interfaces="lagg0"
> > ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 laggport
> > igb4
> > laggport igb5 xxx.xx.x.xx netmask 255.255.248.0"
> > ipvr_addrs_lagg0="xxx.xx.x.xx"
> > defaultrouter="xxx.xx.x.xx"
> > #
> > ### Defaults for SSH, NTP, ZFS
> > sshd_enable="YES"
> > ntpd_enable="YES"
> > zfs_enable="YES"
> > #
> > ## NFS Server
> > rpcbind_enable="YES"
> > nfs_server_enable="YES"
> > mountd_flags="-r -l"
> > nfsd_enable="YES"
> > mountd_enable="YES"
> > rpc_lockd_enable="NO"
> > rpc_statd_enable="NO"
> > nfs_server_flags="-u -t -n 128"
> > nfsv4_server_enable="YES"
> > nfsuserd_enable="YES"
> > [/quote]
> > 
> > Thanks in advance,
> > --
> > Eric Browning
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "
> > freebsd-fs-unsubscribe at freebsd.org "
> > 
> 
> 
> 
> 
> --
> 
> Eric Browning
> Systems Administrator
> 801-984-7623
> 
> Skaggs Catholic Center
> Juan Diego Catholic High School
> Saint John the Baptist Middle
> Saint John the Baptist Elementary
>