zpool export/import on failover - The pool metadata is corrupted

[mxb]

root at nfs2:/root # uname -a
FreeBSD nfs2 9.1-STABLE FreeBSD 9.1-STABLE #1: Tue May 21 21:42:21 CEST 2013     root at nfs2:/usr/obj/usr/src/sys/GENERIC  amd64
root at nfs2:/root #

On 6 jun 2013, at 21:33, Jeremy Chadwick <jdc at koitsu.org> wrote:

> On Thu, Jun 06, 2013 at 09:24:34PM +0200, mxb wrote:
>> 
>> Hello list,
>> 
>> I have two-head ZFS setup with external disk enclosure over SAS expander.
>> This is a failover setup with CARP and devd triggering spool export/import.
>> One of two nodes is preferred master.
>> 
>> Then master is rebooted, devd kicks in as of CARP becomes master and the second node picks up ZFS-disks from external enclosure.
>> Then master comes back, CARP becomes master, devd kicks in and pool gets exported from the second node and imported on the first one.
>> 
>> However, I have experienced metadata corruption several times with this setup.
>> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is both local and external - da1,da2, da13s2, da14s2
>> 
>> root at nfs2:/root # zpool import
>>   pool: jbod
>>     id: 17635654860276652744
>>  state: FAULTED
>> status: The pool metadata is corrupted.
>> action: The pool cannot be imported due to damaged devices or data.
>>   see: http://illumos.org/msg/ZFS-8000-72
>> config:
>> 
>> 	jbod        FAULTED  corrupted data
>> 	  raidz3-0  ONLINE
>> 	    da3     ONLINE
>> 	    da4     ONLINE
>> 	    da5     ONLINE
>> 	    da6     ONLINE
>> 	    da7     ONLINE
>> 	    da8     ONLINE
>> 	    da9     ONLINE
>> 	    da10    ONLINE
>> 	    da11    ONLINE
>> 	    da12    ONLINE
>> 	cache
>> 	  da1
>> 	  da2
>> 	  da13s2
>> 	  da14s2
>> 	logs
>> 	  mirror-1  ONLINE
>> 	    da13s1  ONLINE
>> 	    da14s1  ONLINE
>> 
>> Any ideas what is going on?
> 
> uname -a please.
> 
> -- 
> | Jeremy Chadwick                                   jdc at koitsu.org |
> | UNIX Systems Administrator                http://jdc.koitsu.org/ |
> | Making life hard for others since 1977.             PGP 4BD6C0CB |
>