From nobody Thu May 20 10:25:30 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DDD518B2BD9 for ; Thu, 20 May 2021 10:25:58 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fm5VZ0MYGz3s8C for ; Thu, 20 May 2021 10:25:57 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-io1-xd2f.google.com with SMTP id a8so7934942ioa.12 for ; Thu, 20 May 2021 03:25:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yLz7iZxh9r2dq8XYU+qafLJ1Cjord3htyiaJuxFQqMc=; b=A+j3eD0WtNyBxdYy4PrV+cI+qQOyh0JW8wtaiHBMyqTdE8w/XlC9XbtvK0SJZoWavx XF5iAoOpsoy/3J+ba4lOBicOpUO/RIwLD61vz0z32l6PdNQZhZjNJrTq9aSWVE/exDVH RDWLL2cFgfRrod5xMGtAYIz+0EmeAL0xlXb64fHLXFsPIrxkndDHJiJuaNgsZ7rLhb5U f/q5cPu2cFQ74KuBCaZ7o+XEaAA44/MMs8SBhrQj6ZExBttT7doZr+DwvBaFNEc/t+x7 HT+xj5NKck/w5bWsUOfVd0ozxX3wffkUMy7SWx9KzAeKNvoRbv3CgjuQwXsf+Yz1Ky8o sBog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yLz7iZxh9r2dq8XYU+qafLJ1Cjord3htyiaJuxFQqMc=; b=mNOzp3UpB9sGEKdkNY8B0vAlm89ukLWReK3oKfoTGu4NFp3FgO48hp+f41EMAuEdk0 Kgv+2y60QTqn1Hsa+XaLI53dfwjKK6EDC0OR/oKdO8mo/Fb/rQ43I0Eq9IuSDHO4duTx fZStP5+mrGJEnAvr3ntaUlM2wbfnqzMq8fSErOx6deDQH34R1HFqZwun0UYgWRalnOwS JT+JCGjY/H7VkU75ujHZI+8zx3vYrfJe4Q134LYgKKOyX7gPqY75jLzok96xQdJDa1KX jtlbHuujJbOPki9kdBx8uDWBIn8MqYpfy6AS9IXpLBVZtxfTlpw1J6569eB+Gfn7ad4i p3+w== X-Gm-Message-State: AOAM530vf9gfCphEB9UtEHW9xATdzm0oK0+9z4hVAAPYzkdjasDWWpSV WRfcIWbFdSiJaDwGW7oeQJBVHrnJx0x76Q+lnVrq3Q== X-Google-Smtp-Source: ABdhPJzdip7UmespzGGLMs24+fg0c4pCBg2RV/z4D3Jnb8Moc4O0pe7/W3ODGX9AkBjNXX1LJgMt+jX36MJNvLad1Zc= X-Received: by 2002:a6b:6c0b:: with SMTP id a11mr4742150ioh.37.1621506356557; Thu, 20 May 2021 03:25:56 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> In-Reply-To: From: Steven Hartland Date: Thu, 20 May 2021 11:25:30 +0100 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: mike tancsa Cc: Alan Somers , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000624b2305c2c05e06" X-Rspamd-Queue-Id: 4Fm5VZ0MYGz3s8C X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=A+j3eD0W; dmarc=pass (policy=none) header.from=multiplay.co.uk; spf=pass (mx1.freebsd.org: domain of steven@multiplay.co.uk designates 2607:f8b0:4864:20::d2f as permitted sender) smtp.mailfrom=steven@multiplay.co.uk X-Spamd-Result: default: False [-3.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::d2f:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; DMARC_POLICY_ALLOW(-0.50)[multiplay.co.uk,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2f:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[killing@multiplay.co.uk,steven@multiplay.co.uk]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::d2f:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[killing@multiplay.co.uk,steven@multiplay.co.uk]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] --000000000000624b2305c2c05e06 Content-Type: text/plain; charset="UTF-8" What is your pool structure / disk types? On Mon, 17 May 2021 at 16:58, mike tancsa wrote: > On 5/13/2021 11:37 AM, Alan Somers wrote: > > On Thu, May 13, 2021 at 8:45 AM mike tancsa > > wrote: > > > > For offsite storage, I have been doing a zfs send across a 10G > > link and > > noticed something I don't understand with respect to speed. I have a > > > > > > Is this a high latency link? ZFS send streams can be bursty. Piping > > the stream through mbuffer helps with that. Just google "zfs send > > mbuffer" for some examples. And be aware that your speed may be > > limited by the sender. Especially if those small files are randomly > > spread across the platter, your sending server's disks may be the > > limiting factor. Use gstat to check. > > -Alan > > > Just a quick follow up. I was doing some tests with just mbuffer, > mbuffer and ssh, and just ssh (aes128-gcm) with an compressed stream and > non compressed stream (zfs send vs zfs send -c). Generally, didnt find > too much of a difference. I was testing on a production server that is > generally uniformly busy, so it wont be 100% reliable, but I think close > enough as there is not much variance in the back ground load nor in the > results. > > I tried this both with datasets that were backups of mailspools, so LOTS > of little files and big directories as well as with zfs datasets with a > few big files. > > On the mail spool just via mbuffer (no ssh involved at all) > > zfs send > summary: 514 GiByte in 1h 09min 35.9sec - average of 126 MiB/s > zfs send -c > summary: 418 GiByte in 1h 05min 58.5sec - average of 108 MiB/s > > and the same dataset, sending just through OpenSSH took 1h:06m (zfs > send) and 1h:01m (zfs send -c) > > > On the large dataset (large VMDK files), similar pattern. I did find one > interesting thing, when I was testing with a smaller dataset of just > 12G. As the server has 65G of ram, 29 allocated to ARC, sending a zfs > stream with -c made a giant difference. I guess there is some efficiency > with sending something thats already compressed in arc ? Or maybe its > just all cache effect. > > Testing with one with about 1TB of referenced data using mbuffer with > and without ssh and just ssh > > zfs send with mbuffer and ssh > summary: 772 GiByte in 51min 06.2sec - average of 258 MiB/s > zfs send -c > summary: 772 GiByte in 1h 22min 09.3sec - average of 160 MiB/s > > And the same dataset just with ssh -- zfs send 53min and zfs send -c 55min > > and just mbuffer (no ssh) > > summary: 772 GiByte in 56min 45.7sec - average of 232 MiB/s (zfs send -c) > summary: 1224 GiByte in 53min 20.4sec - average of 392 MiB/s (zfs send) > > This seems to imply the disk is the bottleneck. mbuffer doesnt seem to > make much of a difference either way. Straight up ssh looks to be fine > / best. > > Next step is going to allocate a pair of SSDs as special allocation > class vdevs to see if it starts to make a difference for all that > metadata. I guess I will have to send/resend the datasets to make sure > they make full use of the special vdevs > > ----Mike > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > --000000000000624b2305c2c05e06 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
What is your pool structure / disk types?

On Mon, 17 May 20= 21 at 16:58, mike tancsa <mike@sentex= .net> wrote:
On 5/13/2021 11:37 AM, Alan Somers wrote:
> On Thu, May 13, 2021 at 8:45 AM mike tancsa <mike@sentex.net
> <mailto:mike@s= entex.net>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0For offsite storage, I have been doing a zfs send a= cross a 10G
>=C2=A0 =C2=A0 =C2=A0link and
>=C2=A0 =C2=A0 =C2=A0noticed something I don't understand with respe= ct to speed.=C2=A0 I have a
>
>
> Is this a high latency link?=C2=A0 ZFS send streams can be bursty.=C2= =A0 Piping
> the stream through mbuffer helps with that.=C2=A0 Just google "zf= s send
> mbuffer" for some examples.=C2=A0 And be aware that your speed ma= y be
> limited by the sender.=C2=A0 Especially if those small files are rando= mly
> spread across the platter, your sending server's disks may be the<= br> > limiting factor.=C2=A0 Use gstat to check.
> -Alan


Just a quick follow up.=C2=A0 I was doing some tests with just mbuffer,
mbuffer and ssh, and just ssh (aes128-gcm) with an compressed stream and non compressed stream (zfs send vs zfs send -c).=C2=A0 Generally, didnt fin= d
too much of a difference.=C2=A0 I was testing on a production server that i= s
generally uniformly busy, so it wont be 100% reliable, but I think close enough as there is not much variance in the back ground load nor in the
results.

I tried this both with datasets that were backups of mailspools, so LOTS of little files and big directories as well as with zfs datasets with a
few big files.=C2=A0

On the mail spool just via mbuffer (no ssh involved at all)

zfs send
summary:=C2=A0 514 GiByte in 1h 09min 35.9sec - average of=C2=A0 126 MiB/s<= br> zfs send -c
summary:=C2=A0 418 GiByte in 1h 05min 58.5sec - average of=C2=A0 108 MiB/s<= br>
and the same dataset, sending just through OpenSSH took 1h:06m (zfs
send) and 1h:01m (zfs send -c)


On the large dataset (large VMDK files), similar pattern. I did find one interesting thing, when I was testing with a smaller dataset of just
12G.=C2=A0 As the server has 65G of ram, 29 allocated to ARC, sending a zfs=
stream with -c made a giant difference. I guess there is some efficiency with sending something thats already compressed in arc ? Or maybe its
just all cache effect.

Testing with one with about 1TB of referenced data using mbuffer with
and without ssh=C2=A0 and just ssh

zfs send with mbuffer and ssh
summary:=C2=A0 772 GiByte in 51min 06.2sec - average of=C2=A0 258 MiB/s
zfs send -c
summary:=C2=A0 772 GiByte in 1h 22min 09.3sec - average of=C2=A0 160 MiB/s<= br>
And the same dataset just with ssh -- zfs send 53min and zfs send -c 55min<= br>
and just mbuffer (no ssh)

summary:=C2=A0 772 GiByte in 56min 45.7sec - average of=C2=A0 232 MiB/s (zf= s send -c)
summary: 1224 GiByte in 53min 20.4sec - average of=C2=A0 392 MiB/s (zfs sen= d)

This seems to imply the disk is the bottleneck. mbuffer doesnt seem to
make much of a difference either way.=C2=A0 Straight up ssh looks to be fin= e
/ best.

Next step is going to allocate a pair of SSDs as special allocation
class vdevs to see if it starts to make a difference for all that
metadata. I guess I will have to send/resend the datasets to make sure
they make full use of the special vdevs

=C2=A0=C2=A0=C2=A0 ----Mike


_______________________________________________
freebsd-fs@free= bsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/free= bsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org&= quot;
--000000000000624b2305c2c05e06--