From nobody Wed Feb 22 20:13:55 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMS6L3fRQz3s8Pb for ; Wed, 22 Feb 2023 20:14:02 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMS6K5rVgz3GSF; Wed, 22 Feb 2023 20:14:01 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 31MKDsKR093337 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Wed, 22 Feb 2023 15:13:54 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4::29] ([IPv6:2607:f3e0:0:4:0:0:0:29]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 31MKDrlH089627 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 22 Feb 2023 15:13:54 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: Date: Wed, 22 Feb 2023 15:13:55 -0500 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: speeding up zfs send | recv (update) Content-Language: en-US To: Miroslav Lachman <000.fbsd@quip.cz>, Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: mike tancsa In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 X-Rspamd-Queue-Id: 4PMS6K5rVgz3GSF X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 2/22/2023 1:06 PM, Miroslav Lachman wrote: > > I am facing similar problem with low performance of zfs send over > network. I have 2 machines in two different datacenters, both have > 1Gbps NIC and I would like to saturate the network but it seems > impossible even if "everything" seem to have enough unused resources. > The sending side is very old HP ML110 G5 with bge0 NIC, receiving side > is VM with enough CPU, RAM, 22TB storage and vtnet0 NIC. > Sender has about 25% CPU idle during sending, disks are not saturated > according to iostat -w -x but I still cannot see more than 52MiB/s. > Everything about zfs snapshot, send, receive etc. is handled by > syncoid from sanoid package (ssh + mbuffer + pv, no lzop) > > I thought it is slow because of ssh ciphers so I tried to change with > --sshciper but it was 10MiB/s slower when I changed from default > chacha20-poly1305@openssh.com to aes128-ctr aes128-gcm@openssh.com is what I settled on for the cipher.   If you just blast dd if=/dev/zero | ssh are you able to achieve close to wirespeed ?  As (I think) I mentioned in this thread, different zfs datasets transmit at different speeds.  Ones with tens of thousands of small files are much slower than those with a few multi gigabit files.  The disk seems to be the limiting factor for me as graphing "time spent in IO" via telegraf, shows the disks pretty well 100% busy. e.g. dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null"   20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s 20000+0 records in 19982+36 records out 20971520000 bytes transferred in 18.259553 secs (1148523199 bytes/sec) vs  dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null"   20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s 20000+0 records in 19961+78 records out 20971520000 bytes transferred in 43.947239 secs (477197671 bytes/sec) dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr mdtancsa@coldstore "cat - > /dev/null"   20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s 20000+0 records in 19973+54 records out 20971520000 bytes transferred in 29.263111 secs (716653818 bytes/sec)     ---Mike