Gmirror - how to do?

Sat Feb 5 18:58:19 PST 2005

On Sat, Feb 05, 2005 at 09:12:21PM -0500, Paul Mather wrote:
> On Sat, 2005-02-05 at 17:33 -0600, Karl Denninger wrote:
> > On Sun, Feb 06, 2005 at 12:08:42AM +0100, Pawel Jakub Dawidek wrote:
> > > On Sat, Feb 05, 2005 at 05:04:15PM -0600, Karl Denninger wrote:
> > > +> > Remember not to boot the main machine with this disk inside, as it can
> > > +> > be tasted before your main 'boot' mirror. Inserting this disk after
> > > +> > boot, when your 'boot' mirror is configured should be safe.
> > > +> 
> > > +> Nope, won't work.
> > > +> 
> > > +> The mirrors potentially have different PHYSICAL slice sizes (remember
> > > +> this debate a while back on this list?) and if I do this, I'm guaranteed to
> > > +> screw the partition table, as the fdisk size of the slice table will be
> > > +> picked up.
> > > 
> > > Sorry, but I don't understand.
> > > How can you touch slices configuration by labeling ad4s1?
> > > 
> > > -- 
> > > Pawel Jakub Dawidek                       http://www.wheel.pl
> > > pjd at FreeBSD.org                           http://www.FreeBSD.org
> > > FreeBSD committer                         Am I Evil? Yes, I Am!
> > 
> > 
> > Won't the gmirror system create the new mirror (on the "backup disk" using 
> > the full size of the slice?
> 
> Gmirror will truncate the mirror's size to that of the smallest mirror
> component at creation (or, if you try and insert a provider that is too
> small for an existing mirror, it will [should] refuse to add it).  So,
> if you create a slice within the mirror component (/dev/mirror/...) then
> it cannot, by definition, be "too big."  The slice may not cover the
> entire drive, but it won't be bigger than the drive.
> 
> The only problem you can run into is if you try and use a drive that is
> smaller than the smallest one used to create the initial mirror.  That
> will not work.
> 
> Cheers,
> 
> Paul.

I think you're missing what I'm trying to accomplish, and where (I think)
it won't work.

Given the following configuration:

1. Two internal SATA drives, each of size "X".
2. An external SATA drive bay, with two disk carriers each of size "X" (or
   larger)
3. An initial RAID-1 boot volume that is comprised of the two internal disks.
4. FreeBSD 5-Stable

The goal is that:

1. I can <leave a carrier in the external enclosure> and it will NOT be
   used, by default, by the mirror, even if the system should reboot for
   some odd reason.  This is important, because if I screw myself in some
   fashion on the running machine, having the backup disk scribbled on
   at the same time defeats the purpose of a backup.

2. I can cause the carrier disk to be recognized, and if it is, the system
   will bring it current.  A physical command to initiate this is
   acceptable (indeed, required, since I do not want that process to start
   unexpectedly.)

3. Once the disk in the carrier IS current, I can detach it from the
   running configuration and return to State #1 above.  I now have a
   "near-line" backup that can be mounted (SEPARATELY!) at any time to 
   recover data if for some reason there is a problem (e.g. I 'fat-finger'
   an "rm -rf" from a directory that I really wanted)  I can also rotate
   these disks in the carriers so that I have an offsite copy, plus a
   near-line copy.

4. If the original machine takes a dump entirely (e.g. the controller 
   or OS goes insane and scribbles on both disks, there is a physical
   catastrophe with the system, etc) I can take the disk in either the
   carrier or the safe deposit box, plug it into an arbitrary machine
   and boot it without drama.  I could also (if I had TWO disks) boot the
   most current one and then bring the other into the mirror, effectively
   recovering both the system and the redundancy all at once.  An FSCK in 
   that instance is acceptable since I don't want to quiesce the system 
   (which I understand is necessary to unmount/etc)  Critical applications 
   can be stopped in (3) before the disk in the carrier is detached (e.g. 
   dbms, etc) so I know THAT data is current.

The problem(s) I see with the possible ways to do this are:

1. A "gmirror remove" removes the metadata on the disk and it will no
   longer boot, since the root filesystem isn't on a mirror anymore (the
   metadata is gone).

2. A "fake out" with a gmirror label leaves me vulnerable to an unexpected
   resync without being prompted if the system reboots for any reason, and
   the possibility of the system activating the <WRONG> root volume.  The
   latter could be catastrophically bad, especially if gmirror then rebuilt
   the STALE disk back onto the other two, immediately destroying the
   current data set!  It <ALSO> leaves me with the possibility that the
   mirror created on that disk (if its one of the "larger" ones) is too big,
   and if that one is booted in recovery mode that the other backup disk
   could not be made part of the mirror due to it being physically smaller
   in block count on the same slice.  In other words, whether the system is
   truly recoverable transparently then depends on which disk is the one
   you boot singly.  That's not good.

3. A "gmirror deactivate" followed by a "gmirror forget" is likely (not
   sure yet, will test) to leave me with an unbootable disk as in (1),
   because while the metadata is still there it is marked "do not use";
   thus, the boot will fail (I think)

4. A "atacontrol detach" leaves me with the situation where a reboot
   finds the disk and can leave me in exactly the same situation as (2).

Something I have not tried is to:

1. Mark the mirror as "no automatic rebuilds."
2. Bring the third disk current.
3. Stop the critical processes, and perform a couple of sync's.
4. Forcibly detach it using "atacontrol detach 2".  This appears to 
   also spin the disk down.
5. Clean up the idea that there is another disk out there with
   "gmirror forget".  I should now have a "complete" mirror set with
   two components, and a detached disk.  Since the mirror is set "no
   automatic rebuilds", if the system reboots it should NOT reattach the
   backup volume.
6. The backup volume SHOULD be bootable on its own, without drama, as the
   mirror, while it will be seen as degraded, should still be operational
   off that volume.

The only problem is that there is no way to insure any buffers for that
disk have been flushed, so its very similar to a "plug pull" for an
unprotected machine.

I think I'll try that one next, although I don't care for the semantics.  
Worst thing I can get is a panic for the unexpected detach, I think :->

I can always use the disk as if it was a great big tape drive, and just
dump each filesystem to it, which avoids all of this tomfoolery (maybe 
with a minimal system on it so I can restore the dumps onto a new set of
mirrors) but while that will work, it kinda evades the purpose of what I'm
trying to do - come up with a way to "hot mirror" the machine in such a
fashion that recovery is painless and possible even for someone who is
completely untrained, being told only to stick the backup disk into a 
carrier on a new CPU and turn on the power.

If this sort of thing is a model that the gmirror authors didn't think of,
its something they might want to in the future.....

--
-- 
Karl Denninger (karl at denninger.net) Internet Consultant & Kids Rights Activist
http://www.denninger.net	My home on the net - links to everything I do!
http://scubaforum.org		Your UNCENSORED place to talk about DIVING!
http://www.spamcuda.net		SPAM FREE mailboxes - FREE FOR A LIMITED TIME!
http://genesis3.blogspot.com	Musings Of A Sentient Mind