Odd file system corruption in ZFS pool

Andrew Reilly areilly at bigpond.net.au
Tue Apr 24 14:30:21 UTC 2012


Hi there,

Had a power outage followed by an electrician visit, and the
power supply to my server room yoyo'd a few times before I
pulled the plugs.  When I powered up again this afternoon I had
ZFS pool corruption, according to zpool status.

Several zpool scrub/reboot cycles later, I have the odd
situation of a directory entry that shows up in glob expansion,
but which claims not to be there when ls does a stat, or when I
try to move or delete it.  Any suggestions?

Ls shows:

$ ls
ls: .Suppliers.2010: No such file or directory

(Yes, this is in a Maildir managed jointly by qmail and
dovecot.)  There are also a bunch of message files in a
different directory that find reports "unknown error 112" for,
which doesn't look good.

Zpool status -v tank says:

  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
  scan: scrub in progress since Wed Apr 25 00:12:48 2012
        50.9G scanned out of 1.03T at 106M/s, 2h42m to go
        0 repaired, 4.81% done
config:

	NAME                                            STATE     READ WRITE CKSUM
	tank                                            ONLINE       0     0   853
	  raidz1-0                                      ONLINE       0     0 3.33K
	    gptid/b06b6337-e511-11e0-9d62-00270e0fb8e9  ONLINE       0     0     0
	    gptid/b6c7d5b0-e511-11e0-9d62-00270e0fb8e9  ONLINE       0     0     0
	    gptid/bbf6c485-e511-11e0-9d62-00270e0fb8e9  ONLINE       0     0     0
	    gptid/bf86a966-e511-11e0-9d62-00270e0fb8e9  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        tank/home:<0x0>

Every time I run zpool scrub and reboot the CKSUM column goes to
zero, and the errors go away and the status says that the
previous scrub found no errors.  Every time I look at the
missing directory the CKSUM column gets larger...

I do have a backup (I hope): on another zfs filesystem on a usb
drive, constructed by sending and receiving snapshots in an
incremental fashion.  What I don't know is whether I actually
need to use it or not.  Everything seems to be working fine
apart from this mysteriously untouchable Maildir directory
and a few mail files, all of which I could recover from the
backup if only I could remove the original.  Should I just zfs
destroy tank/home then create it again and send the last backup
snapshots back?  Would that remove the corruption?

Oh, I'm running 9-stable, and my Zpool and zfs versions are :

FreeBSD johnny.reilly.home 9.0-STABLE FreeBSD 9.0-STABLE #15: Sun Apr 22 11:37:17 EST 2012     root at johnny.reilly.home:/usr/obj/usr/src/sys/GENERIC  amd64

ZFS filesystem version 5
ZFS storage pool version 28

Cheers,

-- 
Andrew



More information about the freebsd-fs mailing list