FreeBSD restartable send/receive over WAN

Shiva Bhanujan Shiva.Bhanujan at Quorum.net
Mon Jun 5 21:06:46 UTC 2017 

Hi Ronald,

Ideally, I'd like to send/receive the compressed blocks, and don't wish to compress/uncompress on the fly.    Is compressed send/receive available on FreeBSD and if so, in what version?

Regards,
Shiva


________________________________________
From: Ronald Klop [ronald-lists at klop.ws]
Sent: Sunday, March 05, 2017 3:01 AM
To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan
Cc: freebsd-fs at freebsd.org; Jeremy Faulkner
Subject: Re: FreeBSD restartable send/receive over WAN

On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan
<Shiva.Bhanujan at quorum.net> wrote:

> I ran the same set of tests between 2 FreeBSD instances, connected on a
> 1G LAN.  The the comms was setup using mbuffer.  Here's the basic
> command.
>
> time zfs send -v <snapshot> | <compress> | mbuffer -O <IP>:8099 -b 1024
> -m 128M -P 10 -q -l /tmp/mbuffer.log
> mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | <uncompress>
> | zfs recv <snapshot>
>
> Here are the results that I got.
>
> no compression:
> real    3m18.591s
> user    0m0.390s
> sys     0m8.177s
>
> xz -0:
> real    7m26.349s
> user    7m6.436s
> sys     0m8.471s
>
> pxz -0:
> real    2m28.470s
> user    6m44.168s
> sys     0m12.002s
>
> gzip:
> real    3m12.482s
> user    3m8.260s
> sys     0m4.974s
>
> lz4:
> real    1m58.363s
> user    0m10.000s
> sys     0m8.708s
>
>
>
> lz4 still comes out best.  Is there anything else that I might be
> missing in my tests?  don't have a real setup at this time to test WAN
> connections, but I'd like to think that these results can be
> extrapolated to a WAN link also.


Uhm, in your previous test no-compression came out best. I thought you
wanted to test if sending on-disk compressed blocks over a network was a
gain. Not a plain test of compression algoritms.

As your current numbers show: the overhead of mbuffer.log is much higher
than the overhead of compression+decompression for the lz4 case. real:
0m29 to 1m58. (All other compressors are actually faster with mbuffer in
between? Probably because of more efficient buffering by mbuffer.)
As the overhead of mbuffer is large it is a question how much gain you get
 from removing the extra compression step by sending the on-disk compressed
blocks.
I guess the time would go from 1m58 to about 1m30.

NB: By splitting your zfs send/receive command in two, the numbers of time
can be affected. Time prints the output as soon as zfs send ends.

Have fun,
Ronald.



> ________________________________________
> From: Ronald Klop [ronald-lists at klop.ws]
> Sent: Tuesday, February 28, 2017 11:44 AM
> To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan
> Cc: freebsd-fs at freebsd.org; Jeremy Faulkner
> Subject: Re: FreeBSD restartable send/receive over WAN
>
> On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan
> <Shiva.Bhanujan at quorum.net> wrote:
>
>> thanks for all the pointers for the compression algorithms.  I ran a few
>> tests to compare compression overhead.  These are local zfs
>> send/receive, and no network is involved.
>>
>> time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs
>> receive -s <dst-snapshot>
>>
>> Here are the performance results that I got.
>>
>> no compression:
>> real    0m20.892s
>> user    0m0.000s
>> sys     0m5.587s
>>
>> xz -0:
>> real    8m38.569s
>> user    10m28.551s
>> sys     0m6.866s
>>
>> pxz -0:
>> real    4m38.448s
>> user    10m55.863s
>> sys     0m13.324s
>>
>> gzip:
>> real    3m51.297s
>> user    4m12.035s
>> sys     0m4.696s
>>
>> lz4:
>> real    0m29.912s
>> user    0m16.543s
>> sys     0m10.810s
>>
>>
>> lz4 has the least overhead in terms of time.   pxz/xz seem to be
>> prohibitive give the above results.  Unless, there is something basic
>> I'm missing?
>>
>> I was really hoping that compressed sends would be available, as that
>> would actively eliminate this overhead, given that we use lz4 as the
>> compression algorithm when writing to disks.
>
>
> Why don't you test this with a network in between? That would give much
> more valuable numbers to compare for your use case.
> The number above say nothing about the gain vs the bottleneck you are
> trying to remove.
>
> Ronald.
>
>
>
>
>>
>>
>> ________________________________
>> From: Karli Sjöberg [karli.sjoberg at slu.se]
>> Sent: Sunday, February 26, 2017 8:41 AM
>> To: Gary Palmer
>> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs at freebsd.org
>> Subject: Re: FreeBSD restartable send/receive over WAN
>>
>>
>> Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer at freebsd.org>:
>>>
>>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:
>>> > The compression that we use on our ZFS filesystems is lz4.  So, if I
>>> have to pipe it through a compression algorithm, that'd be
>>> uncompressing and compressing it 4 times.
>>> >
>>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)
>>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)
>>> >
>>> > isn't this quite expensive?  We have to transfer multi terabyte files
>>> on a WAN link.  I'm also of the understanding that gzip by itself is
>>> single-threaded, so that'd peg one of the CPUs to 100%.  there might be
>>> other compression algorithms that can be used, but sending the ZFS as
>>> it is compressed on the filesystem is something that would be optimal,
>>> and would reduce the overhead of the additional [de]compressions that
>>> are taking place?
>>>
>>> Without going into the efficiency part of your message:
>>>
>>> archivers/pigz: Parallel GZIP
>>> archivers/pbzip2: Parallel BZIP2
>>> archivers/pixz: Parallel, indexing version of XZ
>>> archivers/pxz: Parallel LZMA compressor using liblzma
>>
>> Also worth mentioning is, obviously:
>> archivers/lz4
>>
>> :)
>>
>> /K
>>
>>>
>>> Regards,
>>>
>>> Gary
>>>
>>> >
>>> > ________________________________________
>>> > From: owner-freebsd-fs at freebsd.org [owner-freebsd-fs at freebsd.org] on
>>> behalf of Jeremy Faulkner [gldisater at gmail.com]
>>> > Sent: Saturday, February 25, 2017 4:03 PM
>>> > To: freebsd-fs at freebsd.org
>>> > Subject: Re: FreeBSD restartable send/receive over WAN
>>> >
>>> > Pipe it through a compressor
>>> >
>>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:
>>> > > Hi,
>>> > >
>>> > > I just tried restartable send/receive in 10.3 and it works like a
>>> charm.  I was wondering if compressed send has made its way into
>>> FreeBSD?  I checked 10.3 and 11.0-RELEASE, and I don't see the
>>> -c/--compressed option.  Any pointers?
>>> > >
>>> > > Regards,
>>> > > Shiva
>>> > >
>>> > >
>>> > > ________________________________________
>>> > > From: owner-freebsd-fs at freebsd.org [owner-freebsd-fs at freebsd.org]
>>> on behalf of Adam Nowacki [nowakpl at platinum.linux.pl]
>>> > > Sent: Thursday, February 16, 2017 10:41 AM
>>> > > To: freebsd-fs at freebsd.org
>>> > > Subject: Re: FreeBSD restartable send/receive over WAN
>>> > >
>>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:
>>> > >> Hello,
>>> > >>
>>> > >> I was wondering if restartable send/receive is available in
>>> FreeBSD?  We're running 10.2 and have a requirement of sending and
>>> receiving ZFS snapshots over a WAN link.  The snapshots could be more
>>> than a few terabytes.
>>> > >>
>>> > >> Can somebody please give me pointers, and if this feature is or
>>> isn't available in FreeBSD?
>>> > >
>>> > > FreeBSD 10.3 and later.
>>> > >
>>> > > _______________________________________________
>>> > > freebsd-fs at freebsd.org mailing list
>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> > > To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> > > _______________________________________________
>>> > > freebsd-fs at freebsd.org mailing list
>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> > > To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>>> > >
>>> > _______________________________________________
>>> > freebsd-fs at freebsd.org mailing list
>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>> > _______________________________________________
>>> > freebsd-fs at freebsd.org mailing list
>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>> >
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"

More information about the freebsd-fs mailing list