consistent file system inconsistencies (tried replacing drive)

[Jeremy Chadwick]

On Fri, Oct 15, 2010 at 05:19:22PM -0400, Scott Charron wrote:
> >  > Also I was under the impression soft-updates would actually require
> >  > a little more disk access time and thus make the problem slightly
> >  > worse.
> >
> > No, soft-updates doesn't require more disk access time in
> > general.  It caches and re-orders meta data updates, so it
> > can even save disk access time.  But the important thing
> > is that soft-updates re-orders the meta data updates in a
> > way that guarantees that it is in a consistent state at
> > any time (provided that the disk's firware cooperates
> > correctly).  This means that there won't be _unexpected_
> > inconsistencies after a crash, and fsck will be able to
> > run without user-intervention.  (NB:  If you want to avoid
> > fsck completely, you will have to use journalling, or go
> > to a ZFS-only system without any UFS file systems.)
> 
> Maybe I should just go to ZFS... it's fully supported now even for
> root right? Will that be more robust against power outages??

IMHO, no.  Some would argue that a ZFS mirror or raidz pool would be
sufficient, and I disagree.  People need to remember the situation:
*power is being completely lost to the entire system abruptly*.  Not
just one device (e.g. yanking a disk off the bus without warning), but
literally everything.

A ZFS mirror would require 2 or more disks (or devices; that could mean
slice or partition), while ZFS raidz would require 3 or more.  A single
disk or device using ZFS gets you the ability to detect when there's
read/write errors, but no way to fix them.

I don't think it matters anyway because......

> > If you still get unexpected inconsistencies even though
> > you use soft-updates everywhere, then something else must
> > be wrong.  Maybe your hard disk doesn't play along nicely.
> > The usual recommendation is to disable the write-cache
> > on hard disks.  This will make your system slower, though.
> 
> Remember I'm using a USB stick here :)

Which makes this entire thread even more crazy.

The only thing I think that might help you "more" (but again, absolutely
no guarantee of filesystem integrity) would be to mount all of your
filesystems with the "sync" option.  Please see mount(8).

Be aware this will make I/O very, very, very slow.  It will induce a
flush-to-disk on every single write operation as I understand it.

You should really just solve the actual problem of power being lost
abruptly.

> > If you see filesystem problems on your non-root filesystem as well, e.g.
> > ones with SU (soft-updates) applied, I would recommend setting
> > background_fsck="no" in your /etc/rc.conf.  There are some old threads
> > documenting how background filesystem checks don't always fix all
> > problems before the system starts actually using the filesystem.  There
> > were reports of people finding that manual fsck would detect issues that
> > background fsck wouldn't fix.  YMMV.
> 
> Is this recommended? Should I schedule regular manual fscks?

The thread is long, and you should take some time to read it in full
(please do not skim it, and please read what Kirk wrote).  The original
thread ("panic: handle_written_inodeblock: bad size") thread gets split
off into a sub-thread ("background fsck considered harmful?").  This is
the best I can find on short notice:

http://lists.freebsd.org/pipermail/freebsd-stable/2010-July/thread.html#57849
http://lists.freebsd.org/pipermail/freebsd-stable/2010-July/thread.html#57876
http://lists.freebsd.org/pipermail/freebsd-fs/2010-July/thread.html#8927

It's recommended (by myself and some others) because it's been proven
background fsck doesn't always repair errors.  Some feel the opinion is
too harsh, and that's a legitimate opinion.  The "solution" is to "use
journalled soft-updates when they become available in FreeBSD 9.0".

But again, I'll take a moment to recommend that you solve the power loss
problem altogether by investing in a UPS.  Sudden power loss is going to
blow/fry your hardware eventually (especially if you experience
repetitive brown outs like we do here -- literally "flickering power"
outages where power is rapidly being toggled on/off at 0.5s intervals
over the course of 15-20 seconds.  I *have* had consumer home appliances
die permanently because of this.  Let's hear it for Silicon Valley power
grids...)

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |