UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY

[Jeremy Chadwick]

On Tue, Sep 30, 2008 at 11:40:46AM +1000, Andrew Snow wrote:
>
> Matthew Dillon wrote:
>> It can take 6 hours to fsck a full 1TB HD.  It can
>>     take over a day to fsck larger setups.  Putting in a few sleeps here
>>     and there just makes the run time even longer and perpetuates the pain.
>
> We have a box with millions of files spread over 2TB, on a 16 disk RAID.
> Foreground fsck takes almost 8 hours, so background fsck, which takes
> almost 24 hours or more, is my only option when I want to bring the box
> back online quickly.   And UFS Snapshots are so slow as to be completely
> useless.
>
> I've now converted the volume to ZFS, and am now enjoying instant boot
> time and higher speed I/O under heavy load, at the expense of memory
> consumption.
>
>>     My recommendation?  Default UFS back to a synchronous fsck and stop
>>     treating ZFS (your only real alternative) as being so ultra-alpha that
>>     it shouldn't be used.
>
> Completely agree.  ZFS is the way of the future for FreeBSD.  In my
> latest testing, the memory problems are now under control, there is just
> stability problems with random lockups after days of heavy load unless I
> turn off ZIL.  So its nearly there.

It just now occurred to me that this entire conversation should've been
moved to freebsd-fs weeks ago.  *laugh*  Oh well.  :-)

You're the first person I've encountered who has had to disable the ZIL
to get stability in ZFS; ouch, that must hurt.

ZFS stability has been discussed on freebsd-fs numerous times, but the
answers provided are always penultimate; no one (AFAIK) has examined how
to solve this from the start (specifically new FreeBSD installations).

Yes, I know sysinstall/sade doesn't support ZFS (though the PC-BSD folks
have apparently implemented this), but that's not what I'm talking
about.  I'm talking about the most commonly-encountered problem: kmem
exhaustion.  People want to be able to install FreeBSD then say "Okay!
Time to give ZFS a try!" on some separate disks, and have it work.  They
don't want to encounter kmem exhaustion half way through the migration
process; that's just going to dishearten them.

I'll be starting up a new topic on freebsd-fs later tonight with an idea
I came up with for solving this out-of-the-box.  I have a feeling I'm
going to get told "so who's going to do all the work?" or downright
flamed, but I hope it induces a discussion of ideas, specifically with
regards to new FreeBSD installations.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |