kern/166566: zfs split renders 2 disk (MBR based) mirror unbootable

[George Hartzell]

>Number:         166566
>Category:       kern
>Synopsis:       zfs split renders 2 disk (MBR based) mirror unbootable
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 02 03:10:11 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     George Hartzell
>Release:        9-STABLE
>Organization:
>Environment:
FreeBSD delicious.alerce.com 9.0-STABLE FreeBSD 9.0-STABLE #0: Mon Mar 26 22:24:31 PDT 2012     root at zfshive.alerce.com:/usr/obj/usr/src/sys/DELICIOUS  amd64
>Description:
I have a mac pro running 9-STABLE.  Two disks are part of a bootable zfs mirror.  They're MBR based.

------------------------------------------------------------------------------------
(delicious)[8:00pm]~>>gpart show ada1
=>        63  1953525105  ada1  MBR  (931G)
          63  1953525105     1  freebsd  [active]  (931G)

(delicious)[8:01pm]~>>gpart show ada1s1
=>         0  1953525105  ada1s1  BSD  (931G)
           0  1941962752       1  freebsd-zfs  (926G)
  1941962752    11562353       2  freebsd-swap  (5.5G)
------------------------------------------------------------------------------------

The mirror is currently resilvering, unrelated to this bug report.

------------------------------------------------------------------------------------
(delicious)[8:01pm]~>>zpool status zroot
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr  1 19:55:59 2012
        12.9G scanned out of 523G at 34.7M/s, 4h11m to go
        12.9G resilvered, 2.47% done
config:

	NAME         STATE     READ WRITE CKSUM
	zroot        ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    ada3s1a  ONLINE       0     0     0
	    ada1s1a  ONLINE       0     0     0  (resilvering)

errors: No known data errors
------------------------------------------------------------------------------------

/boot/loader.conf contains:
vfs.root.mountfrom="zfs:zroot"

and zroot has it's bootfs set to zroot.

This system boots from either disk and runs happily.

I tried a zpool split on it

  zpool split zroot zsplitroot

and it booted up until the kernel tried to mount the root filesystem and it failed.
>How-To-Repeat:
I believe that setting up a bootable zfs mirror and running zpool split on it should repeat the problem.  It does for me.
>Fix:
I was able to repair the situation by booting from a 9.0 DVD, loading the zfs kernel module, doing

  zpool import

which showed both pools

  zpool import -f -o cachefile=/tmp/zpool.cache -o altroot=/mnt zroot
  mount -t zfs zroot /mnt
  cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache

then destroying the splitroot pool and attaching it back to the mirror.


>Release-Note:
>Audit-Trail:
>Unformatted: