From nobody Wed Feb 22 18:06:06 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMPGp2zYKz3tFqs for ; Wed, 22 Feb 2023 18:06:10 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMPGn2W36z4HD5; Wed, 22 Feb 2023 18:06:09 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of "SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz"; dmarc=none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 898B6D788C; Wed, 22 Feb 2023 19:06:07 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 5A2FDD7896; Wed, 22 Feb 2023 19:06:06 +0100 (CET) Message-ID: Date: Wed, 22 Feb 2023 19:06:06 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) Content-Language: cs-Cestina, en-US To: Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-1.79 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[quip.cz]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4PMPGn2W36z4HD5 X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 17/05/2021 17:57, mike tancsa wrote: [...] > On the mail spool just via mbuffer (no ssh involved at all) > > zfs send > summary:  514 GiByte in 1h 09min 35.9sec - average of  126 MiB/s > zfs send -c > summary:  418 GiByte in 1h 05min 58.5sec - average of  108 MiB/s > > and the same dataset, sending just through OpenSSH took 1h:06m (zfs > send) and 1h:01m (zfs send -c) > > > On the large dataset (large VMDK files), similar pattern. I did find one > interesting thing, when I was testing with a smaller dataset of just > 12G.  As the server has 65G of ram, 29 allocated to ARC, sending a zfs > stream with -c made a giant difference. I guess there is some efficiency > with sending something thats already compressed in arc ? Or maybe its > just all cache effect. > > Testing with one with about 1TB of referenced data using mbuffer with > and without ssh  and just ssh > > zfs send with mbuffer and ssh > summary:  772 GiByte in 51min 06.2sec - average of  258 MiB/s > zfs send -c > summary:  772 GiByte in 1h 22min 09.3sec - average of  160 MiB/s > > And the same dataset just with ssh -- zfs send 53min and zfs send -c 55min > > and just mbuffer (no ssh) > > summary:  772 GiByte in 56min 45.7sec - average of  232 MiB/s (zfs send -c) > summary: 1224 GiByte in 53min 20.4sec - average of  392 MiB/s (zfs send) I am facing similar problem with low performance of zfs send over network. I have 2 machines in two different datacenters, both have 1Gbps NIC and I would like to saturate the network but it seems impossible even if "everything" seem to have enough unused resources. The sending side is very old HP ML110 G5 with bge0 NIC, receiving side is VM with enough CPU, RAM, 22TB storage and vtnet0 NIC. Sender has about 25% CPU idle during sending, disks are not saturated according to iostat -w -x but I still cannot see more than 52MiB/s. Everything about zfs snapshot, send, receive etc. is handled by syncoid from sanoid package (ssh + mbuffer + pv, no lzop) I thought it is slow because of ssh ciphers so I tried to change with --sshciper but it was 10MiB/s slower when I changed from default chacha20-poly1305@openssh.com to aes128-ctr The interesting thing was testing with scp with different ciphers (tested all available) and sending 1GB file full of random bytes from dd if=/dev/urandom. With each scp cipher it was able to achieve 65MiB/s. Even more interesting is that "zfs send -c" is slower than "zfs send" (without compressed stream). I would expect the opposite. Why is zfs send -c slower? Some datasets are transferred with much lower speed about 40MiB/s. What are the limiting factors for zfs send if there is 25% idle CPU, RAIDZ of 4x SATA disks should give more than 50MB/s (iostat show disks busy oscillating between 50% - 80% and receiving side is under 50% usage (CPU and disk) Sender stats looks like this: # iostat -x -w 30 ada{0,1,2,3} extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada0 227 18 10961.1 260.5 2 2 56 2 0 17 ada1 276 18 12996.9 260.8 2 2 62 2 0 18 ada2 225 17 10567.9 252.1 3 2 68 3 0 19 ada3 275 17 12217.1 252.4 2 2 77 2 0 19 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada0 185 11 6623.3 126.5 5 4 110 5 5 24 ada1 204 11 7238.8 124.0 3 4 109 4 3 24 ada2 187 11 6474.7 119.5 5 10 106 5 5 26 ada3 205 10 6825.5 115.7 4 8 108 4 2 26 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada0 239 3 7698.6 59.3 5 1 46 4 0 29 ada1 241 3 7507.2 59.3 4 1 61 4 1 29 ada2 240 3 7476.6 59.3 5 1 40 5 0 30 ada3 235 3 7156.9 59.3 4 2 75 4 0 30 # ifstat -i bge0 60 bge0 KB/s in KB/s out 1245.04 49964.32 1321.39 53035.46 1323.03 53108.41 1319.83 52947.85 1248.55 50089.84 1240.00 49706.52 1278.22 51309.49 1274.60 51159.64 1279.98 51366.20 1157.62 46435.98 Network latency between machines round-trip min/avg/max/stddev = 0.326/0.492/0.897/0.156 ms As I need to transfer about 15TB of data I really would like to speed it up to the maximum. In the current state it would take a week to transfer it. Kind regards Miroslav Lachman