zpool export/import on failover - The pool metadata is corrupted

[Outback Dingo]

On Thu, Jun 6, 2013 at 3:24 PM, mxb <mxb at alumni.chalmers.se> wrote:

>
> Hello list,
>
> I have two-head ZFS setup with external disk enclosure over SAS expander.
> This is a failover setup with CARP and devd triggering spool export/import.
> One of two nodes is preferred master.
>
> Then master is rebooted, devd kicks in as of CARP becomes master and the
> second node picks up ZFS-disks from external enclosure.
> Then master comes back, CARP becomes master, devd kicks in and pool gets
> exported from the second node and imported on the first one.
>
> However, I have experienced metadata corruption several times with this
> setup.
> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is both
> local and external - da1,da2, da13s2, da14s2
>
> root at nfs2:/root # zpool import
>    pool: jbod
>      id: 17635654860276652744
>   state: FAULTED
>  status: The pool metadata is corrupted.
>  action: The pool cannot be imported due to damaged devices or data.
>    see: http://illumos.org/msg/ZFS-8000-72
>  config:
>
>         jbod        FAULTED  corrupted data
>           raidz3-0  ONLINE
>             da3     ONLINE
>             da4     ONLINE
>             da5     ONLINE
>             da6     ONLINE
>             da7     ONLINE
>             da8     ONLINE
>             da9     ONLINE
>             da10    ONLINE
>             da11    ONLINE
>             da12    ONLINE
>         cache
>           da1
>           da2
>           da13s2
>           da14s2
>         logs
>           mirror-1  ONLINE
>             da13s1  ONLINE
>             da14s1  ONLINE
>
> Any ideas what is going on?
>

Best case scenerio, both nodes tried to import the disks simultaneously,
split brain condition, or disks appear out of order and no using labels, we
had a similar situation with geom_multipath, theres no quorum disks
knowledge yet for FreeBSD using zfs in this configuration, we ran it
similarly for a while, until we realized through research it was bad karma
to let carp and devd control nodes without a fool proof way to be sure
nodes were ready to export/import. though you did state

"Then master comes back, CARP becomes master, devd kicks in and pool gets
exported from the second node and imported on the first one."  be nice to
see how you managed that with scripts... even if both nodes booted
simultaneously their both going to "fight" for master and try to import
that pool.


> //mxb
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>