Storing revisions of large files using ZFS snapshots
Bob Friesenhahn
bfriesen at simple.dallas.tx.us
Tue May 31 22:57:54 UTC 2011
On Tue, 31 May 2011, Per von Zweigbergk wrote:
>
> Basically what would happen is that every night after the backup
> completes, rsync would be run, synchronizing over the differences
> between the synthetic full backup from the previous day. Historic
> copies of the full backup images as synchronized by rsync would be
> kept using ZFS snapshots. After our retention window closes, I'd
> just nuke the oldest snapshots from the server.
This is feasable using rsync's "--inplace --no-whole-file" options,
and is what I use as part of my daily backup strategy so that each
backup cycle only consumes disk space for the changed blocks. It
works fantastically. There is of course some risk associated with
these rsync options since if rsync fails, the backup file may only be
partially updated. Of course you can re-try the rsync a number of
times during the backup interval if rsync fails. If the rsync
previously failed (or perhaps always), it is necessary to also supply
the expensive --checksum option so that rsync will definitely assure
that the file content is correct. This option is expensive since it
will force all blocks to be read on both sides.
> First of all, will ZFS will do copy-on-write on a block level when
> it comes to snapshots, or is copy-on-write on ZFS snapshots done on
> a whole-file level? It would seem that block-level COW would be
> required for this to even have a chance of working. Please note that
> I'm not talking about deduplication in ZFS itself, but rather using
> snapshots as a means to perform a crude kind of deduplication.
ZFS does copy-on-write for file blocks when the file is updated.
There is surely some copy-on-write of zfs metadata when a snapshot is
taken, but surely vastly less than the amount of data consumed by the
file. Remember that filesystem volumes look like files and they
snapshot nicely.
Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
More information about the freebsd-fs
mailing list