zpool export/import on failover - The pool metadata is corrupted

[mxb]

Then MASTER goes down, CARP on the second node goes MASTER (devd.conf, and script for lifting):

root at nfs2:/root # cat /etc/devd.conf


notify 30 {
match "system"		"IFNET";
match "subsystem"	"carp0";
match "type"		"LINK_UP";
action "/etc/zfs_switch.sh active";
};

notify 30 {
match "system"          "IFNET";
match "subsystem"       "carp0";
match "type"            "LINK_DOWN";
action "/etc/zfs_switch.sh backup";
};

root at nfs2:/root # cat /etc/zfs_switch.sh
#!/bin/sh

DATE=`date +%Y%m%d`
HOSTNAME=`hostname`

ZFS_POOL="jbod"


case $1 in
	active)
		echo "Switching to ACTIVE and importing ZFS" | mail -s ''$DATE': '$HOSTNAME' switching to ACTIVE' root
		sleep 10
		/sbin/zpool import -f jbod
		/etc/rc.d/mountd restart
		/etc/rc.d/nfsd restart
		;;
	backup)
		echo "Switching to BACKUP and exporting ZFS" | mail -s ''$DATE': '$HOSTNAME' switching to BACKUP' root
		/sbin/zpool export jbod
		/etc/rc.d/mountd restart
                /etc/rc.d/nfsd restart
		;;
	*)
		exit 0
		;;
esac

This works, most of the time, but sometimes I'm forced to re-create pool. Those machines suppose to go into prod.
Loosing pool(and data inside it) stops me from deploy this setup.

//mxb

On 6 jun 2013, at 22:06, Outback Dingo <outbackdingo at gmail.com> wrote:

> 
> 
> 
> On Thu, Jun 6, 2013 at 3:24 PM, mxb <mxb at alumni.chalmers.se> wrote:
> 
> Hello list,
> 
> I have two-head ZFS setup with external disk enclosure over SAS expander.
> This is a failover setup with CARP and devd triggering spool export/import.
> One of two nodes is preferred master.
> 
> Then master is rebooted, devd kicks in as of CARP becomes master and the second node picks up ZFS-disks from external enclosure.
> Then master comes back, CARP becomes master, devd kicks in and pool gets exported from the second node and imported on the first one.
> 
> However, I have experienced metadata corruption several times with this setup.
> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is both local and external - da1,da2, da13s2, da14s2
> 
> root at nfs2:/root # zpool import
>    pool: jbod
>      id: 17635654860276652744
>   state: FAULTED
>  status: The pool metadata is corrupted.
>  action: The pool cannot be imported due to damaged devices or data.
>    see: http://illumos.org/msg/ZFS-8000-72
>  config:
> 
>         jbod        FAULTED  corrupted data
>           raidz3-0  ONLINE
>             da3     ONLINE
>             da4     ONLINE
>             da5     ONLINE
>             da6     ONLINE
>             da7     ONLINE
>             da8     ONLINE
>             da9     ONLINE
>             da10    ONLINE
>             da11    ONLINE
>             da12    ONLINE
>         cache
>           da1
>           da2
>           da13s2
>           da14s2
>         logs
>           mirror-1  ONLINE
>             da13s1  ONLINE
>             da14s1  ONLINE
> 
> Any ideas what is going on?
> 
> Best case scenerio, both nodes tried to import the disks simultaneously, split brain condition, or disks appear out of order and no using labels, we had a similar situation with geom_multipath, theres no quorum disks knowledge yet for FreeBSD using zfs in this configuration, we ran it similarly for a while, until we realized through research it was bad karma to let carp and devd control nodes without a fool proof way to be sure nodes were ready to export/import. though you did state
> 
> "Then master comes back, CARP becomes master, devd kicks in and pool gets exported from the second node and imported on the first one."  be nice to see how you managed that with scripts... even if both nodes booted simultaneously their both going to "fight" for master and try to import that pool.
> 
> 
> //mxb
> 
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>