FreeBSD restartable send/receive over WAN

Eric A. Borisch eborisch at gmail.com
Fri Mar 3 04:20:38 UTC 2017 

You can rate limit with mbuffer if you want experimental results for
different bandwidths. (If you know what you can expect to get on your WAN,
that would be a key point to test.) There are very likely crossover points
in your "rankings" as a function of BW, depending on the compressibility of
your data stream (and each compressor's ability to exploit it.) (Also with
different machines/CPU.)

Or you can measure the constituent parts to create a model and do the math
across simulated link BW. You'll need throughputs and compression ratios
for each compressor. Mbuffer without the -q should report the transferred
(compressed) size. Decompression rate is likely faster than compression and
therefore of little impact in the grand scheme.

  -Eric

On Thu, Mar 2, 2017 at 6:35 PM Shiva Bhanujan <Shiva.Bhanujan at quorum.net>
wrote:

> I ran the same set of tests between 2 FreeBSD instances, connected on a 1G
> LAN.  The the comms was setup using mbuffer.  Here's the basic command.
>
> time zfs send -v <snapshot> | <compress> | mbuffer -O <IP>:8099 -b 1024 -m
> 128M -P 10 -q -l /tmp/mbuffer.log
> mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | <uncompress> |
> zfs recv <snapshot>
>
> Here are the results that I got.
>
> no compression:
> real    3m18.591s
> user    0m0.390s
> sys     0m8.177s
>
> xz -0:
> real    7m26.349s
> user    7m6.436s
> sys     0m8.471s
>
> pxz -0:
> real    2m28.470s
> user    6m44.168s
> sys     0m12.002s
>
> gzip:
> real    3m12.482s
> user    3m8.260s
> sys     0m4.974s
>
> lz4:
> real    1m58.363s
> user    0m10.000s
> sys     0m8.708s
>
>
>
> lz4 still comes out best.  Is there anything else that I might be missing
> in my tests?  don't have a real setup at this time to test WAN connections,
> but I'd like to think that these results can be extrapolated to a WAN link
> also.
>
>
> ________________________________________
> From: Ronald Klop [ronald-lists at klop.ws]
> Sent: Tuesday, February 28, 2017 11:44 AM
> To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan
> Cc: freebsd-fs at freebsd.org; Jeremy Faulkner
> Subject: Re: FreeBSD restartable send/receive over WAN
>
> On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan
> <Shiva.Bhanujan at quorum.net> wrote:
>
> > thanks for all the pointers for the compression algorithms.  I ran a few
> > tests to compare compression overhead.  These are local zfs
> > send/receive, and no network is involved.
> >
> > time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs
> > receive -s <dst-snapshot>
> >
> > Here are the performance results that I got.
> >
> > no compression:
> > real    0m20.892s
> > user    0m0.000s
> > sys     0m5.587s
> >
> > xz -0:
> > real    8m38.569s
> > user    10m28.551s
> > sys     0m6.866s
> >
> > pxz -0:
> > real    4m38.448s
> > user    10m55.863s
> > sys     0m13.324s
> >
> > gzip:
> > real    3m51.297s
> > user    4m12.035s
> > sys     0m4.696s
> >
> > lz4:
> > real    0m29.912s
> > user    0m16.543s
> > sys     0m10.810s
> >
> >
> > lz4 has the least overhead in terms of time.   pxz/xz seem to be
> > prohibitive give the above results.  Unless, there is something basic
> > I'm missing?
> >
> > I was really hoping that compressed sends would be available, as that
> > would actively eliminate this overhead, given that we use lz4 as the
> > compression algorithm when writing to disks.
>
>
> Why don't you test this with a network in between? That would give much
> more valuable numbers to compare for your use case.
> The number above say nothing about the gain vs the bottleneck you are
> trying to remove.
>
> Ronald.
>
>
>
>
> >
> >
> > ________________________________
> > From: Karli Sjöberg [karli.sjoberg at slu.se]
> > Sent: Sunday, February 26, 2017 8:41 AM
> > To: Gary Palmer
> > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs at freebsd.org
> > Subject: Re: FreeBSD restartable send/receive over WAN
> >
> >
> > Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer at freebsd.org>:
> >>
> >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:
> >> > The compression that we use on our ZFS filesystems is lz4.  So, if I
> >> have to pipe it through a compression algorithm, that'd be
> >> uncompressing and compressing it 4 times.
> >> >
> >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)
> >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)
> >> >
> >> > isn't this quite expensive?  We have to transfer multi terabyte files
> >> on a WAN link.  I'm also of the understanding that gzip by itself is
> >> single-threaded, so that'd peg one of the CPUs to 100%.  there might be
> >> other compression algorithms that can be used, but sending the ZFS as
> >> it is compressed on the filesystem is something that would be optimal,
> >> and would reduce the overhead of the additional [de]compressions that
> >> are taking place?
> >>
> >> Without going into the efficiency part of your message:
> >>
> >> archivers/pigz: Parallel GZIP
> >> archivers/pbzip2: Parallel BZIP2
> >> archivers/pixz: Parallel, indexing version of XZ
> >> archivers/pxz: Parallel LZMA compressor using liblzma
> >
> > Also worth mentioning is, obviously:
> > archivers/lz4
> >
> > :)
> >
> > /K
> >
> >>
> >> Regards,
> >>
> >> Gary
> >>
> >> >
> >> > ________________________________________
> >> > From: owner-freebsd-fs at freebsd.org [owner-freebsd-fs at freebsd.org] on
> >> behalf of Jeremy Faulkner [gldisater at gmail.com]
> >> > Sent: Saturday, February 25, 2017 4:03 PM
> >> > To: freebsd-fs at freebsd.org
> >> > Subject: Re: FreeBSD restartable send/receive over WAN
> >> >
> >> > Pipe it through a compressor
> >> >
> >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:
> >> > > Hi,
> >> > >
> >> > > I just tried restartable send/receive in 10.3 and it works like a
> >> charm.  I was wondering if compressed send has made its way into
> >> FreeBSD?  I checked 10.3 and 11.0-RELEASE, and I don't see the
> >> -c/--compressed option.  Any pointers?
> >> > >
> >> > > Regards,
> >> > > Shiva
> >> > >
> >> > >
> >> > > ________________________________________
> >> > > From: owner-freebsd-fs at freebsd.org [owner-freebsd-fs at freebsd.org]
> >> on behalf of Adam Nowacki [nowakpl at platinum.linux.pl]
> >> > > Sent: Thursday, February 16, 2017 10:41 AM
> >> > > To: freebsd-fs at freebsd.org
> >> > > Subject: Re: FreeBSD restartable send/receive over WAN
> >> > >
> >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:
> >> > >> Hello,
> >> > >>
> >> > >> I was wondering if restartable send/receive is available in
> >> FreeBSD?  We're running 10.2 and have a requirement of sending and
> >> receiving ZFS snapshots over a WAN link.  The snapshots could be more
> >> than a few terabytes.
> >> > >>
> >> > >> Can somebody please give me pointers, and if this feature is or
> >> isn't available in FreeBSD?
> >> > >
> >> > > FreeBSD 10.3 and later.
> >> > >
> >> > > _______________________________________________
> >> > > freebsd-fs at freebsd.org mailing list
> >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> > > To unsubscribe, send any mail to
> >> "freebsd-fs-unsubscribe at freebsd.org"
> >> > > _______________________________________________
> >> > > freebsd-fs at freebsd.org mailing list
> >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> > > To unsubscribe, send any mail to
> >> "freebsd-fs-unsubscribe at freebsd.org"
> >> > >
> >> > _______________________________________________
> >> > freebsd-fs at freebsd.org mailing list
> >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> >> > _______________________________________________
> >> > freebsd-fs at freebsd.org mailing list
> >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> >> >
> >> _______________________________________________
> >> freebsd-fs at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>

More information about the freebsd-fs mailing list