it's a race between gmirror and UFS labels and gjournal

[Rick C. Petty]

On Fri, Sep 10, 2010 at 12:14:04PM +0200, Miroslav Lachman wrote:
> 
> The same problem is with gjournal on gmirror. If gmirror drops the disk, 
> then gjournal is (sometimes) detected on the first "broken disk" and 
> this disk cannot be re-inserted in to gmirror again. (I have this 
> experience on 7.x). One must reboot in to single-user without gjournal 
> module loaded and re-insert the disk in to gmirror.

I take back my earlier statement about booting into single-user.  That
didn't even work, because my root filesystem is on that mirror.  I had to
physically pull the cable off of one drive so that GEOM probed correctly
and grabbed the mirror.  Thankfully, the controller recognized when I
plugged in the disk again (some controllers still don't, in FreeBSD), or
I wouldn't have been able to get that other disk back into the mirror.
And thankfully I had physical access to the machine.  My condolences to
those of you who run up against this broken behavior with remote
machines.

The point here is that there is some brokenness with GEOM probing.  The
fix that I hope to see is that gmirror doesn't drop the "broken"
providers so that other GEOMs can taste them.  It should instead hold
on to those providers (yet marked as broken) and let the sysadmin decide
how to handle it.  In my case, I would wanted to do a "gmirror rebuild"
after verifying the correct disks were mapped to the correct mirrors.
With the current implementation, this just isn't possible.

I'm hoping some can explain to me why broken providers are released back
into GEOM for re-tasting.  I'm willing to submit patches, but I haven't
figured out why disks marked as broken aren't kept as part of the mirror
in the first place.

-- Rick C. Petty