Odd file system corruption in ZFS pool

[Andrew Reilly]

On Tue, Apr 24, 2012 at 04:37:45PM +0200, Peter Maloney wrote:
> On 04/24/2012 04:30 PM, Andrew Reilly wrote:
> >ZFS filesystem version 5
> >ZFS storage pool version 28
> Is there anything special about your pool? Was it created in an old 
> FreeBSD and upgraded? Was it send/recv'd from OpenSolaris? etc.

I don't know enough about zfs to know whether there's anything
special about it, I'm afraid.  The pool "tank" is a raidz across
four 1T Seagate NS series drives.  The first incarnation died
from corruption (boot panic loop after a zpool scrub) a year or
so ago, so the current system is new since then.  The first had
been upgraded at least once, not sure about the current.  Has
only ever been attached to this (regularly upgraded) _STABLE
system.  It isn't protected by a UPS, and the power has been
going out without warning fairly regularly, so IMO that is
sufficient to explain the cause of the corruption.  Setting up a
UPS is my next project.

Interesting update to last night's message: the corruption is
rubust under send/receive of snapshots: the last version of my
backup exhibits exactly the same problem.  (That is: a directory
that shows up in glob expansion but can't be removed or touched,
and a directory full of files that find returns: Unknown
error: 122.)

> So far the only corruption I had was the result of installing FreeBSD on 
> a 4 GB USB flash stick. It had no redundancy, and within a few months, 
> some files were spontaneously broken.
> 
> And in that one instance I found that move, copy, etc. on broken files 
> reported by zpool status -v will always fail. Only "rm" worked for me. 
> So I suggest you try rmdir or rm -r.

Rm and rm -r doesn't work.  Even as root, rm -rf Maildir.bad
returns a lot of messages of the form: foo/bar: no such file
or directory.  The result is that I now have a directory that
contains no "good" files, but a concentrated collection of
breakage.

I have another zpool scrub running at the moment.  We'll see if
that is able to clean it up, but it hasn't had much luck in the
past.

Note that none of these broken files or directories show up in
the zpool status -v error list.  That just contains the one
entry for the zfs root directory: tank/home:<0x0>

Cheers,

-- 
Andrew