ZFS dedup and replication

[Peter Maloney]

On 12/01/2011 10:11 PM, Zaphod Beeblebrox wrote:
> On Thu, Dec 1, 2011 at 11:42 AM, krad <kraduk at gmail.com> wrote:
>> On 1 December 2011 13:03, Peter Maloney
>> <peter.maloney at brockmann-consult.de>wrote:
>>> On 12/01/2011 11:20 AM, krad wrote:
>>> On my system with 12 TiB used up, what I do in a script is basically:
>>> -generate a snap name
>>> -make a recursive snapshot
>>> -ssh to the remote server and compare snapshots (find the latest common
>>> snapshot, to find an incremental reference point)
>>> -if a usable reference point exists, start the incremental send like
>>> this (which wipes all changes on the remote system without confirmation):
>>>        zfs send -R -I ${destLastSnap} ${srcLastSnap} | ssh ${destHost}
>>> zfs recv -d -F -v ${destPool}
>>> -and if no usable reference point existed, then do a full send,
>>> non-incremental:
>>>        zfs send -R ${srcLastSnap} | ssh ${destHost} zfs recv -F -v
>>> ${destDataSet}
>>> The part about finding the reference snapshot is the most complicated
>>> part of my script, and missing from anything else I found online when I
>>> was looking for a good solution. For example this script:
>>> http://blogs.sun.com/clive/resource/zfs_repl.ksh
>>> found on this page:
>>> http://blogs.oracle.com/clive/entry/replication_using_zfs
> Not that everyone doesn't enjoy inventing their own wheel, but would
> you mind sharing your snapshot parser?

Sure. Here are a bunch of my zfs scripts.  (attached)

Disclaimer/notes:
-provided as is... might destroy your system, furthermore, I am not
responsible for bodily injury nor nuclear war that may result from misuse
-there are no unit tests, and no documentation other than a few comments
that are possibly only coherent when I read them. For example, it says
that it does it recursively and rolls back the destination dataset, but
there are a few undocumented cases I can't remember when I needed to do
something manual like delete a snapshot, or destroy a dataset. Maybe
that is all in the past. I don't know.
-the zfs_repl2.bash is the one that makes snapshots and replicates which
I wrote myself. The other ksh one is the Oracle one I linked above, and
the .sh version of it was just what I was working on to try to make it
work reliably, before redoing it all myself (reinventing the wheel is
indeed fun).
-especially beware of the deleteOldSnapshots.bash which is not well
tested and not used yet (and deleteEmptySnapshots.bash which does not
work and I believe cannot work).
-granted transferable your choice of any present or future version of
the BSD or GPL license

and another note, I meant to study these which might be better versions
of the same thing, or something different, but never got around to it:
    /usr/ports/sysutils/zfs-replicate/
    /usr/ports/sysutils/zfsnap/
    /usr/ports/sysutils/zfs-periodic


Enjoy!

-- 

--------------------------------------------
Peter Maloney
Brockmann Consult
Max-Planck-Str. 2
21502 Geesthacht
Germany
Tel: +49 4152 889 300
Fax: +49 4152 889 333
E-mail: peter.maloney at brockmann-consult.de
Internet: http://www.brockmann-consult.de
--------------------------------------------