Anyone used rsync scriptology for incremental backup?

[Freddie Cash]

On October 30, 2008 01:25 am Nikolay Denev wrote:
> On 30 Oct, 2008, at 07:00 , Freddie Cash wrote:
> > On Thu, Oct 30, 2008 at 1:50 AM, Andrew Snow <andrew at modulus.org>
> >
> > wrote:
> >> In this way, each day we generate a batch file that lets us step
> >> back one
> >> day.  The diffs themselves, compressed with gzip, and extremely
> >> space efficient.  We can step back potentially hundreds of days,
> >> though it seems
> >> to throw errors sometimes when backing up Windows boxes, which I
> >> haven't
> >> tracked down yet.
> >>
> >> But to be honest, soon you can save yourself a lot of hassle by
> >> simply using
> >> ZFS and taking snapshots.  It'll be faster, and with compression
> >> very space
> >> efficient.
> >
> > That's exactly what we do, use ZFS and RSync.  We have a ZFS
> > /storage/backup filesystem, with directories for each remote site,
> > and sub-directories for each server to be backed up.
> >
> > Each night we snapshot the directory, then run rsync to backup each
> > server.  Snapshots are named with the current date.  For 80 FreeBSD
> > and Linux servers, we average 10 GB of changed data a night.
> >
> > No muss, no fuss.  We've used it to restore entire servers (boot off
> > Knoppix/Frenzy CD, format partitions, rsync back), individual files
> > (no mounting required, just cd into the .zfs/snapshot/snapshotname
> > directory and scp the file), and even once to restore the permissions
> > on a pair of servers where a clueless admin "chmod -R user /home" and
> > "chmod -R 777 /home".
> >
> > Our backup script is pretty much just a double-for loop that scans a
> > set of site-name directories for server config files, and runs rsync
> > in parallel (1 per remote site).
> >
> > We we looking into using variations on rsnapshot, custom
> > squashfs/hardlink stuff, and other solutions, but once we started
> > using ZFS, we stopped looking down those roads.  We were able to do
> > in 3 days of testing and scripting what we hadn't been able to do in
> > almost a month of research and testing.

> Do you experience problems with the snapshots?
> Last time I tried something similiar for backups the bachine
> began to spit errors after a few days of snapshots.
>
> http://lists.freebsd.org/pipermail/freebsd-fs/2008-February/004413.html

We have 72 daily snapshots so far.  Have had up to 30 of them mounted 
read-only while looking for the right version of a file to restore.

These are ZFS snapshots, very different from UFS snapshots.

-- 
Freddie Cash
fjwcash at gmail.com