FreeBSD restartable send/receive over WAN

Tue Jun 13 23:34:17 UTC 2017

Hi Ronald,

thanks for the info.  I'm going to work w/ FreeBSD 11.0, as it has restartable send/receive.  the compressed send is desirable though.

Regards,
Shiva

________________________________
From: Ronald Klop [ronald-lists at klop.ws]
Sent: Monday, June 05, 2017 2:33 PM
To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan
Cc: freebsd-fs at freebsd.org; Jeremy Faulkner
Subject: Re: FreeBSD restartable send/receive over WAN

Hi,

I'm not authoritative about this, but I see the feature in FreeBSD 12:
https://www.freebsd.org/cgi/man.cgi?query=zfs&apropos=0&sektion=8&manpath=FreeBSD+12-current&arch=default&format=html
(option -c to 'zfs send')

But not in FreeBSD 11.
https://www.freebsd.org/cgi/man.cgi?query=zfs&apropos=0&sektion=8&manpath=FreeBSD+11-current&arch=default&format=html

Regards,
Ronald.

On Mon, 05 Jun 2017 23:06:41 +0200, Shiva Bhanujan
<Shiva.Bhanujan at quorum.net><mailto:Shiva.Bhanujan at quorum.net>> wrote:

> Hi Ronald,
>
> Ideally, I'd like to send/receive the compressed blocks, and don't wish
> to compress/uncompress on the fly. Is compressed send/receive
> available on FreeBSD and if so, in what version?
>
> Regards,
> Shiva
>
>
> ________________________________________
> From: Ronald Klop [ronald-lists at klop.ws<mailto:ronald-lists at klop.ws>]
> Sent: Sunday, March 05, 2017 3:01 AM
> To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan
> Cc: freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org>; Jeremy Faulkner
> Subject: Re: FreeBSD restartable send/receive over WAN
>
> On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan
> <Shiva.Bhanujan at quorum.net><mailto:Shiva.Bhanujan at quorum.net>> wrote:
>
>> I ran the same set of tests between 2 FreeBSD instances, connected on a
>> 1G LAN. The the comms was setup using mbuffer. Here's the basic
>> command.
>>
>> time zfs send -v <snapshot> | <compress> | mbuffer -O <IP>:8099 -b 1024
>> -m 128M -P 10 -q -l /tmp/mbuffer.log
>> mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | <uncompress>
>> | zfs recv <snapshot>
>>
>> Here are the results that I got.
>>
>> no compression:
>> real 3m18.591s
>> user 0m0.390s
>> sys 0m8.177s
>>
>> xz -0:
>> real 7m26.349s
>> user 7m6.436s
>> sys 0m8.471s
>>
>> pxz -0:
>> real 2m28.470s
>> user 6m44.168s
>> sys 0m12.002s
>>
>> gzip:
>> real 3m12.482s
>> user 3m8.260s
>> sys 0m4.974s
>>
>> lz4:
>> real 1m58.363s
>> user 0m10.000s
>> sys 0m8.708s
>>
>>
>>
>> lz4 still comes out best. Is there anything else that I might be
>> missing in my tests? don't have a real setup at this time to test WAN
>> connections, but I'd like to think that these results can be
>> extrapolated to a WAN link also.
>
>
> Uhm, in your previous test no-compression came out best. I thought you
> wanted to test if sending on-disk compressed blocks over a network was a
> gain. Not a plain test of compression algoritms.
>
> As your current numbers show: the overhead of mbuffer.log is much higher
> than the overhead of compression+decompression for the lz4 case. real:
> 0m29 to 1m58. (All other compressors are actually faster with mbuffer in
> between? Probably because of more efficient buffering by mbuffer.)
> As the overhead of mbuffer is large it is a question how much gain you
> get
> from removing the extra compression step by sending the on-disk
> compressed
> blocks.
> I guess the time would go from 1m58 to about 1m30.
>
> NB: By splitting your zfs send/receive command in two, the numbers of
> time
> can be affected. Time prints the output as soon as zfs send ends.
>
> Have fun,
> Ronald.
>
>
>
>> ________________________________________
>> From: Ronald Klop [ronald-lists at klop.ws<mailto:ronald-lists at klop.ws>]
>> Sent: Tuesday, February 28, 2017 11:44 AM
>> To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan
>> Cc: freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org>; Jeremy Faulkner
>> Subject: Re: FreeBSD restartable send/receive over WAN
>>
>> On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan
>> <Shiva.Bhanujan at quorum.net><mailto:Shiva.Bhanujan at quorum.net>> wrote:
>>
>>> thanks for all the pointers for the compression algorithms. I ran a
>>> few
>>> tests to compare compression overhead. These are local zfs
>>> send/receive, and no network is involved.
>>>
>>> time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs
>>> receive -s <dst-snapshot>
>>>
>>> Here are the performance results that I got.
>>>
>>> no compression:
>>> real 0m20.892s
>>> user 0m0.000s
>>> sys 0m5.587s
>>>
>>> xz -0:
>>> real 8m38.569s
>>> user 10m28.551s
>>> sys 0m6.866s
>>>
>>> pxz -0:
>>> real 4m38.448s
>>> user 10m55.863s
>>> sys 0m13.324s
>>>
>>> gzip:
>>> real 3m51.297s
>>> user 4m12.035s
>>> sys 0m4.696s
>>>
>>> lz4:
>>> real 0m29.912s
>>> user 0m16.543s
>>> sys 0m10.810s
>>>
>>>
>>> lz4 has the least overhead in terms of time. pxz/xz seem to be
>>> prohibitive give the above results. Unless, there is something basic
>>> I'm missing?
>>>
>>> I was really hoping that compressed sends would be available, as that
>>> would actively eliminate this overhead, given that we use lz4 as the
>>> compression algorithm when writing to disks.
>>
>>
>> Why don't you test this with a network in between? That would give much
>> more valuable numbers to compare for your use case.
>> The number above say nothing about the gain vs the bottleneck you are
>> trying to remove.
>>
>> Ronald.
>>
>>
>>
>>
>>>
>>>
>>> ________________________________
>>> From: Karli Sjöberg [karli.sjoberg at slu.se<mailto:karli.sjoberg at slu.se>]
>>> Sent: Sunday, February 26, 2017 8:41 AM
>>> To: Gary Palmer
>>> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org>
>>> Subject: Re: FreeBSD restartable send/receive over WAN
>>>
>>>
>>> Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer at freebsd.org>:<mailto:gpalmer at freebsd.org>:>
>>>>
>>>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:
>>>> > The compression that we use on our ZFS filesystems is lz4. So, if I
>>>> have to pipe it through a compression algorithm, that'd be
>>>> uncompressing and compressing it 4 times.
>>>> >
>>>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)
>>>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)
>>>> >
>>>> > isn't this quite expensive? We have to transfer multi terabyte
>>>> files
>>>> on a WAN link. I'm also of the understanding that gzip by itself is
>>>> single-threaded, so that'd peg one of the CPUs to 100%. there might
>>>> be
>>>> other compression algorithms that can be used, but sending the ZFS as
>>>> it is compressed on the filesystem is something that would be optimal,
>>>> and would reduce the overhead of the additional [de]compressions that
>>>> are taking place?
>>>>
>>>> Without going into the efficiency part of your message:
>>>>
>>>> archivers/pigz: Parallel GZIP
>>>> archivers/pbzip2: Parallel BZIP2
>>>> archivers/pixz: Parallel, indexing version of XZ
>>>> archivers/pxz: Parallel LZMA compressor using liblzma
>>>
>>> Also worth mentioning is, obviously:
>>> archivers/lz4
>>>
>>> :)
>>>
>>> /K
>>>
>>>>
>>>> Regards,
>>>>
>>>> Gary
>>>>
>>>> >
>>>> > ________________________________________
>>>> > From: owner-freebsd-fs at freebsd.org<mailto:owner-freebsd-fs at freebsd.org> [owner-freebsd-fs at freebsd.org<mailto:owner-freebsd-fs at freebsd.org>] on
>>>> behalf of Jeremy Faulkner [gldisater at gmail.com<mailto:gldisater at gmail.com>]
>>>> > Sent: Saturday, February 25, 2017 4:03 PM
>>>> > To: freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org>
>>>> > Subject: Re: FreeBSD restartable send/receive over WAN
>>>> >
>>>> > Pipe it through a compressor
>>>> >
>>>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:
>>>> > > Hi,
>>>> > >
>>>> > > I just tried restartable send/receive in 10.3 and it works like a
>>>> charm. I was wondering if compressed send has made its way into
>>>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the
>>>> -c/--compressed option. Any pointers?
>>>> > >
>>>> > > Regards,
>>>> > > Shiva
>>>> > >
>>>> > >
>>>> > > ________________________________________
>>>> > > From: owner-freebsd-fs at freebsd.org<mailto:owner-freebsd-fs at freebsd.org> [owner-freebsd-fs at freebsd.org<mailto:owner-freebsd-fs at freebsd.org>]
>>>> on behalf of Adam Nowacki [nowakpl at platinum.linux.pl<mailto:nowakpl at platinum.linux.pl>]
>>>> > > Sent: Thursday, February 16, 2017 10:41 AM
>>>> > > To: freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org>
>>>> > > Subject: Re: FreeBSD restartable send/receive over WAN
>>>> > >
>>>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:
>>>> > >> Hello,
>>>> > >>
>>>> > >> I was wondering if restartable send/receive is available in
>>>> FreeBSD? We're running 10.2 and have a requirement of sending and
>>>> receiving ZFS snapshots over a WAN link. The snapshots could be more
>>>> than a few terabytes.
>>>> > >>
>>>> > >> Can somebody please give me pointers, and if this feature is or
>>>> isn't available in FreeBSD?
>>>> > >
>>>> > > FreeBSD 10.3 and later.
>>>> > >
>>>> > > _______________________________________________
>>>> > > freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
>>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> > > To unsubscribe, send any mail to
>>>> "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"
>>>> > > _______________________________________________
>>>> > > freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
>>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> > > To unsubscribe, send any mail to
>>>> "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"
>>>> > >
>>>> > _______________________________________________
>>>> > freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
>>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> > To unsubscribe, send any mail to
>>>> "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"
>>>> > _______________________________________________
>>>> > freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
>>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> > To unsubscribe, send any mail to
>>>> "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"
>>>> >
>>>> _______________________________________________
>>>> freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"
>>> _______________________________________________
>>> freebsd-fs at freebsd.org<mailto:freebsd-fs at freebsd.org> mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org<mailto:freebsd-fs-unsubscribe at freebsd.org>"
________________________________