slow ZFS on FreeBSD 8.1

[Jeremy Chadwick]

On Sat, Jan 01, 2011 at 10:33:43AM +1100, Peter Jeremy wrote:
> On 2010-Dec-30 07:20:57 -0500, Dan Langille <dan at langille.org> wrote:
> >The reason I've not installed ZFS on root is because of the added 
> >complications.  I run the OS on ufs (with gmirror) and my data is on 
> >ZFS.   We must be hanging out with different groups.  Most of the people 
> >I know don't have ZFS on root.
> 
> My primary system at home is setup this way - primarily because at the
> time I built it (Nov 2008), I felt ZFS was a bit immature and wanted
> to have src and obj on UFS so I could do a rebuild if I lost access to
> ZFS for some reason.  My experience has been that the UFS root has
> caused me far more headaches than the ZFS parts.  I've since done some
> reconfiguration and plan to switch to ZFS root soon.
> 
> Based on my experiences at home, I converted my desktop at work to
> pure ZFS.  The only issues I've run into have been programs that
> extensively use mmap(2) - which is a known issue with ZFS.

Is your ZFS root filesystem associated with a pool that's mirrored or
using raidzX?  What about mismatched /boot content (ZFS vs. UFS)?  What
about booting into single-user mode?

http://wiki.freebsd.org/ZFSOnRoot indirectly hints at these problems but
doesn't outright admit them (yet should), so I'm curious to know how
people have solved them.  Remembering manual "one-offs" for a system
configured this way is not acceptable (read: highly prone to
error/mistake).  Is it worth the risk?  Most administrators don't have
the tolerance for stuff like that in the middle of a system upgrade or
what not; they should be able to follow exactly what's in the handbook,
to a tee.

There's a link to www.dan.me.uk at the bottom of the above Wiki page
that outlines "the madness" that's required to configure the setup, all
of which has to be done by hand.  I don't know many administrators who
are going to tolerate this when deploying numerous machines, especially
when compounded by the complexities mentioned above.

The mmap(2) and sendfile(2) complexities will bite an junior or
mid-level SA in the butt too -- they won't know why software starts
failing or behaving oddly (FreeBSD ftpd is a good example).  It just so
happens that Apache, out-of-the-box, comes with mmap and sendfile use
disabled.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |