slow ZFS on FreeBSD 8.1

[Jeremy Chadwick]

On Wed, Dec 29, 2010 at 12:56:57AM +0100, Freek van Hemert wrote:
> Hello everyone,
> 
> This is my first mail on the mailinglist and I very much appreciate this
> option of getting some help.
> 
> I have a question regarding zfs on freebsd.
> (I'm making a home server)
> This afternoon I did a "zpool create data mirror ad4 ad6" Now I'm copying
> things from my ufs2 disk into the 2TB zpool, it is very slow. I'm on freebsd
> 8.1 amd64 on an atom n330 with 2 sata disks, gstat tells me I'm going at
> around 2 mbps at near 100  %busy while the ufs2 drives are near 0. Also,
> ufs2 to ufs2 was much faster (I estimate about 10 times faster). How do I
> tune? The wiki is not helpful for amd64 users stating that the defaults
> should be optimal. I'm using the 8.1-stable version which has just been
> installed this afternoon from a minimal install iso.
> 
> I'm copying from a single ufs2 pata drive into a sata zpool mirror. I have
> 2GB of ram.
> 
> Top tells me: 761 MB Active, 790 Inactive and there is hardly any cpu usage
> (96-98% idle). vfs.numvnodes: around 12500 now (after several hours of
> copying) and stil slowly rising.

For your system.  /boot/loader.conf:


# Increase vm.kmem_size to allow for ZFS ARC to utilise more memory.
vm.kmem_size="2048M"
vfs.zfs.arc_max="1024M"

# Disable ZFS prefetching
# http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
# Increases overall speed of ZFS, but when disk flushing/writes occur,
# system is less responsive (due to extreme disk I/O).
# NOTE: Systems with 8GB of RAM or more have prefetch enabled by
# default.
vfs.zfs.prefetch_disable="1"

# Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout="5"


/etc/sysctl.conf:

# Increase number of vnodes; we've seen vfs.numvnodes reach 115,000
# at times.  Default max is a little over 200,000.  Playing it safe...
kern.maxvnodes=250000

# Set TXG write limit to a lower threshold.  This helps "level out"
# the throughput rate (see "zpool iostat").  A value of 256MB works well
# for systems with 4GB of RAM, while 1GB works well for us w/ 8GB.
# We're using 1GB here because of the ada3 disk which can really handle
# a massive amount of write speed.
vfs.zfs.txg.write_limit_override=1073741824


Note that the last entry in sysctl.conf may need to be adjusted, or
commented out entirely.  It's up to you.  You can mess with this in
real-time using sysctl.  Google for vfs.zfs.txg.write_limit_override to
get an idea of how to adjust it, if you want to.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |