ZFS system unresponsive

[Jeremy Chadwick]

I'm snipping the statistics that instead people can read via the mailing
list archive URL:

http://lists.freebsd.org/pipermail/freebsd-fs/2011-February/010796.html

On Sun, Mar 13, 2011 at 04:44:30PM +0000, Tim Bishop wrote:
> I did the upgrade to 8-STABLE and it didn't help. I'm still seeing the
> same issue.
> 
> Someone else in another thread mentioned about graphing the zfs-stats
> output with munin, so I thought I'd give that a whirl. Here are the
> results:
> 
> (static snapshot of data so it doesn't change between when I write this
> and when somebody reads it)
> 
> http://www.bishnet.net/tim/tmp/munin/carrick/carrick/index.html
> 
> This slow down is triggered by the tarsnap backups I run each day. They
> start about 01:15 (in the same timezone as the graphs) and finished at
> around 11:15 after I poked it. It slows down when processing my Maildir
> folders, but doesn't actually stop - it's processing a single file every
> couple of seconds.

The below may be informational to you, or at least give you something to
ponder.  It may not be the problem you're experiencing, so I don't want
to divert you from your efforts.  Please keep that in mind when reading
the below.

I've ranted and raved in the past about how badly Maildir performs under
ZFS (or NFS for that matter).  This is one of the reasons I switched
back to using classic mail spools.  I'm well-aware of the problems with
them, but Maildir is absurdly painful (I often describe it as "rude").
It performs awfully at my workplace when used by an NFS-backed
filesystem (using a very high-end NetApp filer) with Solaris 10 as a
client, and also performs horribly on a smaller (10-15 user) Solaris 10
server (backed by ZFS filesystems) using Dovecot for IMAP and Maildir.

Focusing on FreeBSD, I've written about the abysmal performance here:

http://koitsu.wordpress.com/2009/06/01/freebsd-and-zfs-horrible-raidz1-read-speed/
http://koitsu.wordpress.com/2009/06/01/freebsd-and-zfs-horrible-raidz1-speed-part-2/
http://koitsu.wordpress.com/2009/06/14/freebsd-and-zfs-horrible-raidz1-speed-finale/
http://koitsu.wordpress.com/2009/06/17/freebsd-and-zfs-more-performance-quirks/

Be aware the last blog post may have been addressed since its creation;
that is to say, things in ZFS on FreeBSD have changed.  However, the
Maildir performance problem almost certainly hasn't, and to be honest I
don't think a filesystem should be catering to a mail format.

The only real solution is a new format called MIX.  After reading the
below you'll probably get get quite excited, but become depressed when
you hear that basically nobody has bothered to implement support for it
(especially mutt, which saddens me greatly):

http://en.wikipedia.org/wiki/MIX_%28Email%29

All this said, if you can get rid of Maildir (convert things back to
classic spools) as a form of testing and see how things perform after
that, I would strongly recommend it.

I imagine there is probably something that can help with the Maildir
performance issue under ZFS, such as making use of a "cache" device
(which makes use of L2ARC; don't confuse this with ARC).  Note that
"cache" devices on pools need to be very fast for reading; SSDs tend to
work beautifully for this.  But IMHO, the cost of an SSD just to
"attempt to get good performance with Maildir" isn't worth it.  Gotta
think about it in the long run...

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |