Storing revisions of large files using ZFS snapshots

Bob Friesenhahn bfriesen at simple.dallas.tx.us
Tue May 31 22:57:54 UTC 2011


On Tue, 31 May 2011, Per von Zweigbergk wrote:
>
> Basically what would happen is that every night after the backup 
> completes, rsync would be run, synchronizing over the differences 
> between the synthetic full backup from the previous day. Historic 
> copies of the full backup images as synchronized by rsync would be 
> kept using ZFS snapshots. After our retention window closes, I'd 
> just nuke the oldest snapshots from the server.

This is feasable using rsync's "--inplace --no-whole-file" options, 
and is what I use as part of my daily backup strategy so that each 
backup cycle only consumes disk space for the changed blocks.  It 
works fantastically.  There is of course some risk associated with 
these rsync options since if rsync fails, the backup file may only be 
partially updated.  Of course you can re-try the rsync a number of 
times during the backup interval if rsync fails.  If the rsync 
previously failed (or perhaps always), it is necessary to also supply 
the expensive --checksum option so that rsync will definitely assure 
that the file content is correct.  This option is expensive since it 
will force all blocks to be read on both sides.

> First of all, will ZFS will do copy-on-write on a block level when 
> it comes to snapshots, or is copy-on-write on ZFS snapshots done on 
> a whole-file level? It would seem that block-level COW would be 
> required for this to even have a chance of working. Please note that 
> I'm not talking about deduplication in ZFS itself, but rather using 
> snapshots as a means to perform a crude kind of deduplication.

ZFS does copy-on-write for file blocks when the file is updated. 
There is surely some copy-on-write of zfs metadata when a snapshot is 
taken, but surely vastly less than the amount of data consumed by the 
file.  Remember that filesystem volumes look like files and they 
snapshot nicely.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/


More information about the freebsd-fs mailing list