gmirror problem on 5.3-R i386 (SOLVED)

Fri Jan 14 15:12:46 PST 2005

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 14 January 2005 20:43, Doug Poland wrote:
> On Thu, Jan 13, 2005 at 09:51:03AM +0100, Christian Hiris wrote:
> > On Thursday 13 January 2005 03:54, Doug Poland wrote:
> > >
> > > I'm having a problem with gmirror.  It would seem that I cannot add my
> > > first disk to the mirror after booting off the second drive.  My
> > > current status is a degraded mirror and I cannot see any of the
> > > partitions on disk one.
> > >
> > > # dd if=/dev/zero of=/dev/ad4 bs=512 count=79
> >
> > You probably destroyed your slice table here. The dd command only makes
> > sense if you insert whole disks (ie. /dev/ad4 and /dev/ad6) as providers.
> > Otherwise you need to create (or already have) a valid slice table on the
> > disk, which enables gmirror to locate and insert your slice.
>
> One of my problems was incorrect partition sizes when I created the
> bsdlabel for /dev/mirror/gm0s1.  I didn't really know what I was doing
> nor did I understand the consequences of my actions ;)

:)

> > (!) Before you start to correct your gmirror setup, please read all of my
> > comments, because to me it looks like you run a mirror of ad6 and not
> > ad6s1.
>
> I'm not sure why gmirror label shows that but it's definately misleading.
> Since this is a brand-new box with nothing but a minimal install, I
> decided to re-install 5.3-R.  This time I got the install right and
> gmirror reads:

This looks like gmirror in fact uses the the whole disk. The difference 
between the providers and the consumers length is 63 sectors, which is the 
first slice's offset (see fdisk below).

> Geom name: gm0s1
> State: COMPLETE
> Components: 2
> Balance: round-robin
> Slice: 4096
> Flags: NONE
> SyncID: 1
> ID: 3524221455
> Providers:
> 1. Name: mirror/gm0s1
>    Mediasize: 80026329088 (75G)
>    Sectorsize: 512
>    Mode: r5w5e1
> Consumers:
> 1. Name: ad4
>    Mediasize: 80026361856 (75G)
>    Sectorsize: 512
>    Mode: r5w5e2
>    State: ACTIVE
>    Priority: 0
>    Flags: DIRTY
>    SyncID: 1
>    ID: 2031344187
> 2. Name: ad6
>    Mediasize: 80026361856 (75G)
>    Sectorsize: 512
>    Mode: r5w5e2
>    State: ACTIVE
>    Priority: 0
>    Flags: DIRTY
>    SyncID: 1
>    ID: 3116345061
>
> Geom name: gm0s1.sync
>
>
> I'm going to include the steps I took, with one deviation from the
> published doc I referenced, and ask a couple more questions.
>
>
> 1 dd if=/dev/zero of=/dev/ad6 bs=512 count=79
> 2 fdisk -v -B -I /dev/ad6
> 3
> 4 gmirror label -v -n -b round-robin gm0s1 /dev/ad6s1
> 5 gmirror load
> 6
> 7 bsdlabel -w -B /dev/mirror/gm0s1
> 8 bsdlabel -e /dev/mirror/gm0s1
> 9
> 10 newfs -U /dev/mirror/gm0s1a
> 11 mount /dev/mirror/gm0s1a /mnt
> 12 dump -L -0 -f- / | ( cd /mnt ; restore -r -v -f- )
> 13 newfs -U /dev/mirror/gm0s1d
> 14 mount /dev/mirror/gm0s1d /mnt/var
> 15 dump -L -0 -f- /var | ( cd /mnt/var ; restore -r -v -f- )
> 16 newfs -U /dev/mirror/gm0s1e
> 17 mount /dev/mirror/gm0s1e /mnt/tmp
> 18 dump -L -0 -f- /tmp | ( cd /mnt/tmp ; restore -r -v -f- )
> 19 newfs -U /dev/mirror/gm0s1f
> 20 mount /dev/mirror/gm0s1f /mnt/usr
> 21 dump -L -0 -f- /usr | ( cd /mnt/usr ; restore -r -v -f- )
> 22
> 23 cp -p /mnt/etc/fstab /mnt/etc/fstab.orig
> 24 sed -e 's/dev\/ad4s1/dev\/mirror\/gm0s1/g' < /mnt/etc/fstab.orig >
> /mnt/etc/fstab 
> 25 echo 'swapoff="YES"' >> /mnt/etc/rc.conf 
> 26 echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf
> 27 echo "1:ad(6,a)/boot/loader" > /boot.config
> 28
> 29 shutdown -r now
> 30
> 31 dd if=/dev/zero of=/dev/ad4 bs=512 count=79
> 32 fdisk -v -B -I /dev/ad4
> 33 gmirror configure -a gm0s1
> 34 gmirror insert gm0s1 /dev/ad4s1
> 35 sh -c 'while [ ".`gmirror list | grep SYNCHRONIZING`" != . ]; do sleep
> 1; done' 36
> 37 shutdown -r now
>
> On line 32, I deviated from the instructions on:
>
>  http://people.freebsd.org/~rse/mirror/
>
> The published command was:
>
> size=`fdisk ad6 | grep ', size ' | head -1 | sed -e 's;^.*size
> \([0-9]*\).*$;\1;'` (echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f-
> -i /dev/ad4
>
> That command returned:  "improperly placed quotes ()".  I set the value
> of $size manually but it was some small number like 50000.  I wrote it
> to ad4 but the command on line 33 puked.  So I simply issued the command
> shown on line 32.

I ever handled this by sysinstall. The above commands simply sets up the first 
slice (1) to a FreeBSD-type slice (165) starting at offset (63) with length 
($size) of the first slice of ad6 and sets the active flag on the first slice 
(1).  

The command 'fdisk ad6' shows the size of the slice(s) on ad6. The same length 
should be used for ad4. I tried to run the above commands under /bin/sh:

# size=`fdisk ad6 | grep ', size ' | head -1 | sed -e 's;^.*size 
\([0-9]*\).*$;\1;'`

# (echo "p 1 165 63 $size"; echo "a 1")
p 1 165 63 240107427
a 1

# (echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -t -B -f -i mirror/mirror0
******* Working on device /dev/mirror/mirror0 *******

So all the commands seem to work properly. Did you run this under /bin/sh?

> Question:  On line 27 we issue the command to...
> # instruct boot stage 2 loader on first disk to boot
> # with the boot stage 3 loader from the second disk
> # (mainly because BIOS might not allow easy booting from second ATA disk
> # or at least requires manual intervention on the console)
>
> So how do I get rid of that boot.config file?  Should I get rid of it?

If you have a modern machine the BIOS (hopefully) can boot from every 
harddisks that has a partition/slice on it with the active flag set. I for 
myself use a bootmanager on every gmirror disk. You can install it with 
boot0cfg(8) or sysinstall (I only would use boot0cfg, if your mirror already 
has been set up). 

> Question: On line 25 we set swapoff=YES.  Is this necessary if I'm going
> to run 5-STABLE?  What are the implications of having swap on a mirror
> like this?

No, if you run stable you don't need this:
http://lists.freebsd.org/pipermail/cvs-all/2005-January/099589.html
http://lists.freebsd.org/pipermail/cvs-all/2005-January/099590.html

On 5-STABLE systems, younger than the cvsdate of the above commits, 
the /etc/rc.d/swap1 script runs on system shutdown and ensures that 
(eventually mirrored) swap devices are stopped before gmirror shuts down. 
Otherwise the mirror will brake. This is the reason why you need to set 
swapoff=YES. The recommend method to reboot your gmirrored system is 
'shutdown -r now' instead of 'reboot'.   

Cheers,
ch

- -- 
Christian Hiris <4711 at chello.at> | OpenPGP KeyID 0x3BCA53BE 
OpenPGP-Key at hkp://wwwkeys.eu.pgp.net and http://pgp.mit.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (FreeBSD)

iD8DBQFB6FHq09WjGjvKU74RAj0cAKCAz3yT6ZPbbsW+h+ob8xoiazfCiwCghDH9
v9U3rfes4F3C/+JOpB+Kn2o=
=Iu3w
-----END PGP SIGNATURE-----