From nobody Wed Feb 22 21:03:44 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMTCt5DtWz3sCJf for ; Wed, 22 Feb 2023 21:03:54 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMTCt36Ylz3LPh; Wed, 22 Feb 2023 21:03:54 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D8635D788C; Wed, 22 Feb 2023 22:03:45 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 20ABED7886; Wed, 22 Feb 2023 22:03:45 +0100 (CET) Message-ID: <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> Date: Wed, 22 Feb 2023 22:03:44 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) Content-Language: cs-Cestina To: mike tancsa , Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4PMTCt36Ylz3LPh X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 22/02/2023 21:13, mike tancsa wrote: > aes128-gcm@openssh.com is what I settled on for the cipher.   If you > just blast dd if=/dev/zero | ssh are you able to achieve close to > wirespeed ?  As (I think) I mentioned in this thread, different zfs > datasets transmit at different speeds.  Ones with tens of thousands of > small files are much slower than those with a few multi gigabit files. > The disk seems to be the limiting factor for me as graphing "time spent > in IO" via telegraf, shows the disks pretty well 100% busy. Yes, I read about it. Which is also interesting as I thought when zfs sends the whole filesystem it will not walk file by file as will rsync do. > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c > aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null" >   20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s >  dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c > chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null" >   20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr > mdtancsa@coldstore "cat - > /dev/null" >   20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s > Interresting numbers. I think I am the only one who get best speed with chacha20-poly1305@openssh.com # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-gcm@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null" 2074083328 bytes (2074 MB, 1978 MiB) transferred 50.003s, 41 MB/s # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c chacha20-poly1305@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null" 2069889024 bytes (2070 MB, 1974 MiB) transferred 33.006s, 63 MB/s # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-ctr quip@aa.bb.cc.dd "cat - > /dev/null" 2091909120 bytes (2092 MB, 1995 MiB) transferred 45.007s, 46 MB/s It seems the speed of SSH is limited by single core performance which is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). Even if CPU has 50% idle, ssh runs on 99.8% of single core. I know there were some HPN patches to ssh, beside that is there any option I can try to use less CPU? I will play with cpuset to pin ssh on one core and everything else on the other core. Kind regards Miroslav Lachman