ZFS can't mount filesystem

[Paul Schenkeveld]

On Sat, May 07, 2011 at 09:20:57AM +0400, Igor Zabelin wrote:
> Hi,
> 
> I have trouble with ZFS. One of set filesystems can't mount.
> zpool scrub is not doing anything
> ZFS reports an error when geting the properties.
> SMART extended offline test for each disk completed without error.
> It's possible to recover data? Mount ignoring errors?
> 
> FreeBSD 8.2-RELEASE 
> 
> ZFS reports an error when geting the properties.
> 
> # zfs get all tank/var
> 
> [skip normal output]
> internal error: unable to get version property
> internal error: unable to get utf8only property
> internal error: unable to get normalization property
> internal error: unable to get casesensitivity property
> [skip normal output]
> 
> # zpool status -v tank
>   pool: tank
>  state: ONLINE
> status: One or more devices has experienced an error resulting in data
>         corruption.  Applications may be affected.
> action: Restore the file in question if possible.  Otherwise restore the
>         entire pool from backup.
>    see: http://www.sun.com/msg/ZFS-8000-8A
>  scrub: scrub stopped after 0h0m with 0 errors on Sat May  7 08:09:35 2011
> config:
> 
>         NAME           STATE     READ WRITE CKSUM
>         tank           ONLINE       0     0    36
>           raidz1       ONLINE       0     0   144
>             gpt/disk5  ONLINE       0     0     0
>             gpt/disk6  ONLINE       0     0     0
>             gpt/disk7  ONLINE       0     0     0
> 
> errors: Permanent errors have been detected in the following files:
> 
>         tank/var:<0x0>

The 'status:' line indicates that (previous) problems with the pool left
tank/var unusable because zfs encountered checksum errors that could
not be repaired, i.e. because not enough replicas were available.  The
story doesn't tell the cause of this problem, if the problem was with
the drives, cables or controller, the evidence is in the messages file
but that is probably on the affected dataset, unless you have backups or
send syslog to a syslog server.  Disk problems are not likely the cause
because you're using raidz1 so at least two disks must have had problems
to cause this and SMART apparently did not report drive problems too.

Other causes for your problems include:
 - blocks on disk were overwritten for whatever reason
 - powerfailure + write-back cache but no battery-backup
 - problems with the mobo/processor/memory

I think your chances of recovering tank/var are very slim but when you
'zfs destroy tank/var' (using -r if there are snapshots) you probably
can save the rest of your pool.  If you have sub-datasers under tank/var
youd could first *try* to move them to another parent like
'zfs rename tank/var/log tank/rescued_var_log' but probably the damage
to tank/var will prevent that.

As always, sysadmin rule #1 applies here: backups, backups, backups.

HTH

Paul Schenkeveld