backups using rsync

[Polytropon]

On Mon, 04 Mar 2013 12:19:09 -0800, Ronald F. Guilmette wrote:
> 
> In message <20130304125634.8450cfaf.freebsd at edvax.de>, 
> Polytropon <freebsd at edvax.de> wrote:
> 
> >On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote:
> >> Now, unfortunately, I have just been bitten by the evil... and apparently
> >> widely known (except to me)... ``You can't use dump(8) to dump a journaled
> >> filesystem with soft updates'' bug-a-boo.
> >
> >There are other tools you can use, for example tar or cpdup
> >or rsync, as you've mentioned in the subject.
> 
> tar I already knew about, but I think you will agree that it has lots of
> limitations that make it entirely inappropriate for mirroring an entire
> system.

That's true. If your purpose is "backup of data files",
tar is a good tool, especially for cross-platform use.
But if you need to deal with "exceptional" things like
extended permissions, ACL, sparse files and such, you
will quickly see its limits. On the other hand, it can
be used for multi-volume savesets, but this is not your
intention.



> This cpdup thing is entirely new to me.  Thanks for mentioning it!  I really
> never heard of it before, but I just now installed it from ports, and I'm
> perusing the man page. 

It's a little bit comparable to rsync and can also do
things like "only add" (so you won't lose any files:
if they are removed in source, they will be kept in
backup). It also has limitations that rsync will not.



> It looks very promising.  Too bad it doesn't
> properly handle sparse files, but oh well.  That's just a very minor nit.
> (Does it properly handle everything else that rsync claims to be able to
> properly handle, e.g. ACLs, file attributes, etc., etc.?)

That's something you should check with an "example
dataset" you back up, restore, and compare. I've been
using it for "normal files" successfully.



> >The same problems that apply when dumping live systems can
> >bite you using rsync,
> 
> What problems are we talking about, in particular?

The problems I'm refering to is the kind of _possible_
trouble you can get into when backing up files that
keep changing. The ability to make a snapshot prior
to starting the backup is a great help here (if you
don't have the chance to unmount the partitions to
backup). I can't imagine _how_ programs will react
if they start reading a file, prepare sufficient space
in some kind of TOC, then continue reading while the
file grows... or if a file is being read which is
removed during read... If you minimize the writing
activity to the (still) _live_ data you're dealing
with, that could be a benefit.




> I am guessing that if I use rsync, then I *won't* encounter this rather
> annoying issue/problem relating to UFS filesystems that have both soft
> updates and journaling enabled, correct?
> 
> >but support for this "on file system
> >level" seems to be better in rsync than what dump does "on
> >block level".
> 
> What exactly did you mean by "this" ?

As mentioned above: Unexpected and unpredictable results,
strange kinds of inconsistency, may they appear during
backup or later on restore.



> >> If I use all of the following rsync options...  -a,-H,-A, -X, and -S ....
> >> when trying to make my backups, and if I do whatever additional fiddling
> >> is necessary to insure that I separately copy over the MBR and boot loader
> >> also to my backup drive, then is there any reason that, in the event of
> >> a sudden meteor shower that takes out my primary disk drive while leaving
> >> my backup drive intact, I can't just unplug my old primary drive, plug in
> >> my (rsync-created) backup drive, reboot and be back in the sadddle again,
> >> almost immediately, and with -zero- problems?
> >
> >You would have to make sure _many_ things are consistent
> >on the backup disk.
> 
> Well, this is what I am getting at.  This is/was the whole point of my post
> and my question.  I want to know:  What is that set of things, exactly?

The backup disk (or failover disk, as I said) needs to be
initialized properly prior to the first backup run: Make
sure it's bootable. Depending on how you handle identification
of the disk (by device name, by label or UFSID) and how
you're going to boot from it (by selecting the failover
disk in some post-BIOS/POST dialog or by swapping cables
or bays), you should check it actually starts booting.



> >Regarding terminology, that would make the disk a failover disk
> 
> OK.  Thank you.  I will henceforth use that terminology.

Just a suggestion from how you described you will be
using the disk, which isn't what commonly (or mostly)
is expressed by the term "backup" (also cf. "archive"
which is something entirely different).



> >The disk would need to have an initialized file system and
> >a working boot mechanism, both things rsync does not deal with
> 
> Check and check.  I implicitly understood the former, and I explicitly
> mentioned the latter in my original post in this thread.
> 
> But is there anything else, other than those two things (which, just as
> you say, are both clearly outside of the scope of what rsync does)?
> Anything else I need to do or worry about in order to be able to use
> rsync to create & maintain a full-blown fully-working system failover
> drive?

I don't think so, the provided summary seems to be complete
from my understanding.



> If so, I'd much rather learn about it now... you know... as opposed
> to learning about it if and when I actually have to _use_ my failover
> drive.

No. That's the case when you learn about forensic analysis
and disaster recovery. ;-)



-- 
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...