ZFS dedup and replication

[Techie]

Hey Peter,

Thanks for your informative response.

This is odd, I have been waiting for a response to this question for a
few days and these messages just came through..I am glad they did.

Anyhow please allow me to explain the whole "tar" thing. I regret it
because no one even addressed the DDT, Dedup Table part of it.

You see I wanted to use ZFS as a deduplication disk target for my
backup applications and use the native replication capabilities of ZFS
to replicate the virtual backup cartridges. All modern backup apps
leverage disk as a backup target but some don't do replication.

My idea was to use ZFS to do this. However after testing I came to the
realization that ZFS deduplication is NOT ideal for "deduping" third
party  backup streams. From what I read this is due to the fact that
backup applications put their own metadata in the streams and throw
off the block alignment. Products like Data Domain and Quantum DXi use
a variable length block and are developed towards deduplicating backup
application streams. ZFS does OK but nothing in comparison to the
dedup ratio seen on these aforementioned appliances. I used tar as an
example and should have been more specific. I understand what you are
saying about replicating every 15 minutes etc.. However since backup
application create huge files, an incremental send would need to send
the newly created huge file..At least that is how I understand, I may
be correctly. In my testing this was the case but perhaps my syntax
was not correct.

In any case deduplication appliances when replicating only send the
changed blocks that don't exist on the target side. To do this they
have to have knowledge of what exists in the target side "block pool",
"dedup hash table",or whatever it may be called.

>From what I understand a ZFS file system on the source side has no
idea of what exists on the target side. I also understand and maybe
incorrectly, that the zfs send -D only eliminates duplicate blocks in
the stream it is sending and does not account for a block that may
already exist at the target.

As an example let's say I am using a backup app like Amanda.. I do a
full backup every day to a ZFS based disk target..Every day after the
backup completes I do a -- "zfs send -D -i {snap_yesterday}
{snap_today} | ssh DESTINATION zfs recv DEST_FS". Now each day's full
backup will only have maybe a 1% change rate and this will be
reflected on the source side file system. So if I had 5 days of 2 GB
full backups, the source file system will show maybe 3GB Alloc in the
zpool list output. However since the source does not know about
duplicate blocks on the target side from yesterday's backup, it sends
the entire 2GB full backup from today only removing any duplicate
blocks that exist in the stream it is sending. The difference with a
dedup appliance is that it is aware of duplicate blocks on the target
side and won't send them.

This is the reason my original question was asking if there were any
plans to implement a "global DDT" or dedup table to make the target
aware of the destination duplicate blocks so that only unique blocks
are transferred.

Am I incorrect in my understanding of the ZFS DDT being unique to each
ZFS file system/pool?


Thanks
Jimmy

On Fri, Dec 2, 2011 at 12:04 PM, Peter Maloney
<peter.maloney at brockmann-consult.de> wrote:
> Am 02.12.2011 15:50, schrieb Jeremy Chadwick:
>> On Fri, Dec 02, 2011 at 03:27:03PM +0100, Michel Le Cocq wrote:
>>> it's just me or there is no attachment ?
>> The mailing list stripped the attachment.  The previous individual will
>> need to put it up on the web somewhere.
>>
> It is possible that I forgot to attach it. I assumed it would be
> stripped off but the ones in the to/cc would get it.
>
> Here it is on the company website:
>
> http://www.brockmann-consult.de/peter2/zfs.tgz
>
>
>
> Disclaimer/notes:
> -provided as is... might destroy your system, furthermore, I am not
> responsible for bodily injury nor nuclear war that may result from misuse
> -there are no unit tests, and no documentation other than a few comments
> that are possibly only coherent when I read them. For example, it says
> that it does it recursively and rolls back the destination dataset, but
> there are a few undocumented cases I can't remember when I needed to do
> something manual like delete a snapshot, or destroy a dataset. Maybe
> that is all in the past. I don't know.
> -the zfs_repl2.bash is the one that makes snapshots and replicates which
> I wrote myself. The other ksh one is the Oracle one I linked above, and
> the .sh version of it was just what I was working on to try to make it
> work reliably, before redoing it all myself (reinventing the wheel is
> indeed fun).
> -especially beware of the deleteOldSnapshots.bash which is not well
> tested and not used yet (and deleteEmptySnapshots.bash which does not
> work and I believe cannot work).
> -granted transferable your choice of any present or future version of
> the BSD or GPL license
>
> and another note, I meant to study these which might be better versions
> of the same thing, or something different, but never got around to it:
>    /usr/ports/sysutils/zfs-replicate/
>    /usr/ports/sysutils/zfsnap/
>    /usr/ports/sysutils/zfs-periodic
>
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"