Intermittent mirror rebuilds upon reboot.

[Time Catalyst]

Hello,

I'm experiencing and odd problem with gmirror (and possibly gstripe) and I
was recommended to this mailing list by an acquaintance on bsdforums.org.

I have a raid0+1 setup through GEOM consisting of two mirrored, five-disk
stripes (all identical SCSI hard drives).  I've been stress testing a
database on it and everything seems to be working fine.  However, when I try
to reboot the system, roughly 20% of the time gmirror will report as
'DEGRADED' and rebuild one of the five-disk stripes.  Oddly enough, this
happens on the same device (stripe1) every time a degraded mirror occurs.
Here's a dump of the GEOM section of my dmesg and the procedure that I used
to set up the array:

GEOM_STRIPE: Device stripe1 created (id=127607522).
GEOM_STRIPE: Disk da0 attached to stripe1.
GEOM_STRIPE: Disk da1 attached to stripe1.
GEOM_STRIPE: Disk da2 attached to stripe1.
GEOM_STRIPE: Disk da3 attached to stripe1.
GEOM_STRIPE: Disk da4 attached to stripe1.
GEOM_STRIPE: Device stripe1 activated.
GEOM_STRIPE: Device stripe2 created (id=2058765235).
GEOM_STRIPE: Disk da5 attached to stripe2.
GEOM_STRIPE: Disk da6 attached to stripe2.
GEOM_STRIPE: Disk da7 attached to stripe2.
GEOM_STRIPE: Disk da8 attached to stripe2.
GEOM_STRIPE: Disk da9 attached to stripe2.
GEOM_STRIPE: Device stripe2 activated.
GEOM_MIRROR: Device raid1 created (id=702664467).
GEOM_MIRROR: Device raid1: provider stripe/stripe1 detected.
GEOM_MIRROR: Device raid1: provider stripe/stripe2 detected.
GEOM_MIRROR: Device raid1: provider stripe/stripe2 activated.
GEOM_MIRROR: Device raid1: provider mirror/raid1 launched.
GEOM_MIRROR: Device raid1: rebuilding provider stripe/stripe1.

%gstripe label -v -s 1024 stripe1 /dev/da0 /dev/da1 /dev/da2 /dev/da3
/dev/da4
%gstripe label -v -s 1024 stripe2 /dev/da5 /dev/da6 /dev/da7 /dev/da8
/dev/da9
%gmirror label -v -b load raid1 /dev/stripe/stripe1 /dev/stripe/stripe2
%gmirror load
%newfs /dev/mirror/raid1
%mkdir /mirror
%mount /dev/mirror/raid1 /mirror

I added the following to /boot/loader.conf:
geom_stripe_load="YES"
geom_mirror_load="YES"

And the mount to fstab:
/dev/mirror/raid1 /mirror ufs rw 2 2

Since the symptoms were consistent (as in, always appearing on stripe1) I
tried reconfiguring the array a couple different ways.  First I tried to
swap the hard drives on stripe1 and stripe2:

%gstripe label -v -s 1024 stripe2 /dev/da0 /dev/da1 /dev/da2 /dev/da3
/dev/da4
%gstripe label -v -s 1024 stripe1 /dev/da5 /dev/da6 /dev/da7 /dev/da8
/dev/da9

But despite the different drives, stripe1 still caused a degraded mirror.
I'm thinking that this should rule out any hardware failure.  Next I tried
changing the order in which I created the mirror:

%gmirror label -v b load raid1 /dev/stripe/stripe2 /dev/stripe/stripe1

This time stripe2 consistently causes the degraded mirror.  Which leads me
to believe that there could be some sort of race condition occuring here.

Given the intermittent nature of the problem, I'm wondering a few of
things...
1) Could it be the case that gmirror is trying to setup the stripes before
one of them is finished individually setting up?
2) Is it possible that my SCSI drives are not finished settling before
gstripe and gmirror start to do their thing?
3) Am I doing something wrong in my setup procedure that's causing my mirror
to be unstable?

Here's a link to the forum topic in case anyone wants to read the
'back-and-forth':
http://www.bsdforums.org/forums/showthread.php?p=260944#post260944

Thanks in advance (sorry for the 'novel' of text).
-Andy