EBS snapshot backups from a FreeBSD zfs file system: zpool freeze?

[Freddie Cash]

On 2013-07-03 7:16 PM, "Jeremy Chadwick" <jdc at koitsu.org> wrote:
>
> On Thu, Jul 04, 2013 at 01:40:07PM +1200, Berend de Boer wrote:
> > >>>>> "Jeremy" == Jeremy Chadwick <jdc at koitsu.org> writes:
> >
> >
> >     Jeremy>   Also, because nobody seems to warn others of this: if
> >     Jeremy> you go the ZFS route on FreeBSD, please do not use
> >     Jeremy> features like dedup or compression.
> >
> > Exactly the two reasons why I'm experimenting with FreeBSD on AWs.
> >
> > Please tell me more.
>
> dedup has immense and crazy memory requirements; the commonly referenced
> model (which is in no way precise, it's just a general recommendation)
> is that for every 1TB of data you need 1GB of RAM just for the DDT
> (deduplication table)) -- understand that ZFS's ARC also eats lots of
> memory, so when I say 1GB of RAM, I'm talking about that being *purely
> dedicated* to DDT.

Correction: 1 GB of *ARC* space per TB of *unique* data in the pool. Each
unique block in the pool gets an entry in the DDT.

You can use L2ARC to store the DDT, although it takes ARC space to track
data in L2ARC, so you can't go crazy (512 GB L2 with only 16 GB ARC is a
no-no).

However, you do need a lot of RAM to make dedupe work, and your I/O does
drop through the floor.