Filesystem snapshots dog slow

[Jeremy Chadwick]

On Tue, Oct 16, 2007 at 06:54:11AM -0500, Eric Anderson wrote:
> The problem is the way the snapshots work in UFS2.

[...]

> ... the docs that say 'incredibly fast' are actually referring to
> small filesystems, that are not busy (with writes).  Maybe the docs
> should be clarified for now.  You can submit patches to the docs you
> found that say that if you'd like to help out.

I'll work on that this week.  I'm not sure if I read it in the FreeBSD
Handbook or the FAQ or someplace on the web.  If it's in the lesser two,
I'll send-pr changes.

> I recommend reading up a little bit on how the snapshots for UFS2
> work.  It will give you a good understanding of what the issue is.
> Essentially, your disk is hammered making copies of all the cylinder
> groups, skipping those that are 'busy', and coming back to them later.
> On a 200Gb disk, you could have 1000 cylinder groups, each having to
> be locked, copied, unlocked, and then checked again for any subsequent
> changes.  The stalls you see are when there are lock contentions, or
> disk IO issues.  On a single disk (like your setup above), your
> snapshots will take forever since there is very little random IO
> performance available to you.

I understand the majority of what you've written, and this definitely
gives me a better insight to what the problem is.  I hope that others
will eventually find this thread and find their answer here.  Thank you
for providing this.

That said, for such setups, would you recommend *not* using snapshots?
If so, possibly we should consider removing the following code from
src/sbin/dump/main.c:

334                 } else if (snapdump == 0) {
335                         msg("WARNING: %s\n",
336                             "should use -L when dumping live read-write "
337                             "filesystems!");
338                 } else {

> FreeBSD 7 includes ZFS.  Have you thought about using it?

I haven't.  For starters, I keep seeing mails from users reporting data
corruption or kernel panics when using it.  This doesn't mean ZFS is
"bad" (it's likely ZFS is exposing data corruption for them that's
occuring at a lower level (controller, disk, or RAM)), but it does keep
me from considering it a worthy alternative to UFS2 on production
systems at this time.  And then there's this:

WARNING: ZFS is considered to be an experimental feature in FreeBSD.

This doesn't give admins "warm fuzzies".  :-) I'm considering trying ZFS
on my home system (a ZFS filesystem atop a gstripe(8) pair), where I
also perform nightly backups using dump, but there's a part of me which
is asking "just how important is your data?  What if ZFS breaks in the
middle of you using dump(8) on it?  What then?"

> The problem isn't that developers don't know the problem exists, or
> that they don't have hardware, or a serial console access to a system.
> The problem is that there are only so many developers, and so much
> time, and this is a big mountain to climb.  It's hard to find an
> experienced person to do the work (for free), when they could be doing
> anything else they wish. I think, that in the end, for some of these
> aging issues to get resolved, there needs to be another bounty put out
> on it.  I think rsync.net might even have one started for this issue
> already - you might think about adding to the bounty, or officially
> offering hardware through there.

I'll take a peek at the site.  I have no problem donating some money
and/or hardware to the cause, but as I'm one person, I don't have an
infinite supply of cash to donate to projects like this.  About the
most I could donate would be US$1000 (and that's a bit extreme), which
wouldn't provide for very much by itself.

-- 
| Jeremy Chadwick                                    jdc at parodius.com |
| Parodius Networking                           http://www.parodius.com/ |
| UNIX Systems Administrator                      Mountain View, CA, USA |
| Making life hard for others since 1977.                  PGP: 4BD6C0CB |