Booting from a gmirror always leads to a broken gmirror

[Paul Schenkeveld]

Hi Guido,

On Thu, Nov 18, 2010 at 05:33:23PM +0100, Guido van Rooij wrote:
> Hi,
> 
> I am running 8.1-RELEASE with the folloing mirror setup:
> Geom name: gm0
> State: DEGRADED
> Components: 2
> Balance: round-robin
> Slice: 4096
> Flags: NONE
> GenID: 3
> SyncID: 2
> ID: 2610005691
> Providers:
> 1. Name: mirror/gm0
>    Mediasize: 500107861504 (466G)
>    Sectorsize: 512
>    Mode: r3w2e5
> Consumers:
> 1. Name: da1
>    Mediasize: 500107862016 (466G)
>    Sectorsize: 512
>    Mode: r1w1e1
>    State: ACTIVE
>    Priority: 0
>    Flags: DIRTY
>    GenID: 3
>    SyncID: 2
>    ID: 3263791510
> 2. Name: da0
>    Mediasize: 500107862016 (466G)
>    Sectorsize: 512
>    Mode: r1w1e1
>    State: SYNCHRONIZING
>    Priority: 0
>    Flags: DIRTY, SYNCHRONIZING
>    GenID: 3
>    SyncID: 2
>    Synchronized: 15%
>    ID: 4162979509
> 
> I have a dedicated slice to boot from and one GELI slice, both on the mirror.
> Now when I reboot this system (at a time when the mirror is 100% fine, unlike above), the following happens:
> Nov 18 15:36:01 gvr kernel: Root mount waiting for: usbus4
> Nov 18 15:36:01 gvr kernel: ugen4.2: <Western Digital> at usbus4
> Nov 18 15:36:01 gvr kernel: umass0: <Western Digital External HDD, class 0/0, re
> v 2.00/1.75, addr 2> on usbus4
> Nov 18 15:36:01 gvr kernel: ugen1.2: <vendor 0x0b97> at usbus1
> Nov 18 15:36:01 gvr kernel: uhub6: <vendor 0x0b97 product 0x7761, class 9/0, rev
>  1.10/1.10, addr 2> on usbus1
> Nov 18 15:36:01 gvr kernel: uhub5: 4 ports with 1 removable, self powered
> Nov 18 15:36:01 gvr kernel: Root mount waiting for: usbus4
> Nov 18 15:36:01 gvr kernel: uhub6: 3 ports with 2 removable, bus powered
> Nov 18 15:36:01 gvr kernel: da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
> Nov 18 15:36:01 gvr kernel: da0: <WD 5000BEV External 1.75> Fixed Direct Access 
> SCSI-4 device 
> Nov 18 15:36:01 gvr kernel: da0: 40.000MB/s transfers
> Nov 18 15:36:01 gvr kernel: da0: 476940MB (976773168 512 byte sectors: 255H 63S/
> T 60801C)
> Nov 18 15:36:01 gvr kernel: ugen1.3: <O2> at usbus1
> Nov 18 15:36:01 gvr kernel: ugen4.3: <Western Digital> at usbus4
> Nov 18 15:36:01 gvr kernel: umass1: <Western Digital External HDD, class 0/0, re
> v 2.00/1.75, addr 3> on usbus4
> Nov 18 15:36:01 gvr kernel: da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
> Nov 18 15:36:01 gvr kernel: da1: <WD 5000BEV External 1.75> Fixed Direct Access 
> SCSI-4 device 
> Nov 18 15:36:01 gvr kernel: da1: 40.000MB/s transfers
> Nov 18 15:36:01 gvr kernel: da1: 476940MB (976773168 512 byte sectors: 255H 63S/
> T 60801C)
> Nov 18 15:36:01 gvr kernel: GEOM_MIRROR: Component da0 (device gm0) broken, skipping.
> Nov 18 15:36:01 gvr kernel: GEOM_MIRROR: Device mirror/gm0 launched (1/2).
> Nov 18 15:36:01 gvr kernel: Enter passphrase for da0s2: 

Your mirror covers the whole drives da0 amd da1.  Geli discovers a
encrypted provider on da0s2, which sounds odd.  Is geli supposed to find
gm0s2 instead?

GEOM_MIRROR reports component da0 as broken, could it be that its metadata
is not updated correctly at shutdown?  Your disks are connected thru USB,
which is a bit tricky.  I've some experience with a machine that once had
a three disk graid3 connected through USB.  After a normal shutdown (or
reboot) graid3 came up fine but I remember that at some occasions graid3
found one of its providers broken and atarted a rebuild.

Could you try the following:

 1 Shutdown the computer, watch carefully for any errors regarding gm0
   or the physical disks.
 2 Boot a kernel without geom_mirror and geom_eli classes compiled in
   (e.g. a GENERIC kernel) from CDROM or USB stick.
 3 Look at the gmirror metadata with gmirror dump da0 and gmirror dump
   da1 to see if they are in sync.  Did differs, priority for one disk
   should be 0 the other 1, the md5 hash differs but all other fields
   should be equal for both disks.

> Now, the passphrase the geli asks, is for the so-called broken disk. But it is not broken.
> I can always rebuild it fine when I rebooted and re-insert da0. 
> My question: why does gmirror mark da0 as broken?
> 
> -Guido
> 
> _______________________________________________
> freebsd-geom at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-geom
> To unsubscribe, send any mail to "freebsd-geom-unsubscribe at freebsd.org"