Odd file system corruption in ZFS pool

[Peter Maloney]

On 04/25/2012 01:21 AM, Andrew Reilly wrote:
> On Tue, Apr 24, 2012 at 04:37:45PM +0200, Peter Maloney wrote:
>> So far the only corruption I had was the result of installing FreeBSD on
>> a 4 GB USB flash stick. It had no redundancy, and within a few months,
>> some files were spontaneously broken.
>>
>> And in that one instance I found that move, copy, etc. on broken files
>> reported by zpool status -v will always fail. Only "rm" worked for me.
>> So I suggest you try rmdir or rm -r.
> Rm and rm -r doesn't work.  Even as root, rm -rf Maildir.bad
> returns a lot of messages of the form: foo/bar: no such file
> or directory.  The result is that I now have a directory that
> contains no "good" files, but a concentrated collection of
> breakage.
That sucks. But there is one thing I forgot... you need to run the "rm" 
command immediately after scrub. (no export, reboot, etc. in between). 
And it probably only applies to the files listed with the -v part of 
"zpool status -v". So since yours aren't listed... that is something 
different.

Is your broken stuff limited to a single dataset, or the whole pool? You 
could try making a second dataset, copying good files to it, and 
destroying the old one (losing all your snapshots on that dataset, of 
course).


Here is another thread about it:
http://lists.freebsd.org/pipermail/freebsd-current/2011-October/027902.html

And this message looks interesting: "but if you search on the lists for 
up to a year or so, you'll find some useful commands to inspect and 
destroy corrupted objects."
http://lists.freebsd.org/pipermail/freebsd-current/2011-October/027926.html

And
"I tried your suggestion and ran the command "zdb -ccv backups" to try 
and check the consistency of the troublesome "backups" pool.  This is 
what I ended up with:"

But they don't say what the solution is (other than destroy the pool, 
and I would think the dataset could be enough since the filesystem is 
corrupt, but maybe not the pool).


>
> I have another zpool scrub running at the moment.  We'll see if
> that is able to clean it up, but it hasn't had much luck in the
> past.
>
> Note that none of these broken files or directories show up in
> the zpool status -v error list.  That just contains the one
> entry for the zfs root directory: tank/home:<0x0>
>
> Cheers,
>
I doubt scrubbing more than once (repeating the same thing and expecting 
different results) should fix anything. But if you scrubbed on 
OpenIndiana, it would at least be different. And if it worked, you could 
file a PR about it.