From nobody Thu Feb 23 03:51:03 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMfFn2ZSYz3stlZ for ; Thu, 23 Feb 2023 03:51:09 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from mailout-l3b-97.contactoffice.com (mailout-l3b-97.contactoffice.com [212.3.242.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMfFm3wtpz4P3v for ; Thu, 23 Feb 2023 03:51:08 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mailfence.com header.s=20210208-e7xh header.b=k8cyeO7w; spf=pass (mx1.freebsd.org: domain of sysadmin.lists@mailfence.com designates 212.3.242.97 as permitted sender) smtp.mailfrom=sysadmin.lists@mailfence.com; dmarc=pass (policy=quarantine) header.from=mailfence.com Received: from ichabod.co-bxl (ichabod.co-bxl [10.2.0.36]) by mailout-l3b-97.contactoffice.com (Postfix) with ESMTP id D209BB00 for ; Thu, 23 Feb 2023 04:51:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1677124265; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=9370; bh=wGBH19F+W/usqtgXfEv4vA9VwRMK9bNHrOASya4lsSs=; b=k8cyeO7w/EiIwXWvXQuCqS3lgTeFWFidEDEC6gTcRnzdq7IDK4ioULPbWqP0nIBy LnDUy+4QFq53/sLF5VrMFhFC1OY5opbownKUbUUV/6H7EiWSWbcTRPIAm446i1YGsG/ +pKxAG0otNBZhtmnbJ9Ni4e9eDuyS1vvSfziL7dm/Qam/98GEoCGvK3pOq4tTNdOo4h nBFV08aZ1jR/qIPwZWrLm+M0lR8llIrNGkUnynq0QY4EcteCqIASiSoLZEitAfMsou0 YxWLVJfSX8IdoQcNNUGLs52CWxI5vKxMgqy0eLJ45/6m9Upy5t68F5BYm3s0jhGJ9sG yzOUYNK8Bg== Date: Thu, 23 Feb 2023 04:51:03 +0100 (CET) From: Sysadmin Lists To: freebsd-fs Message-ID: <409401259.92260.1677124263649@ichabod.co-bxl> In-Reply-To: <741387429.91447.1677122934622@ichabod.co-bxl> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> Subject: Re: speeding up zfs send | recv (update) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_92257_810470091.1677124263648" X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Spamd-Result: default: False [-3.93 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.84)[-0.838]; DMARC_POLICY_ALLOW(-0.50)[mailfence.com,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:212.3.242.64/26]; R_DKIM_ALLOW(-0.20)[mailfence.com:s=20210208-e7xh]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[212.3.242.97:from]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:10753, ipnet:212.3.242.64/26, country:US]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[mailfence.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4PMfFm3wtpz4P3v X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N ------=_Part_92257_810470091.1677124263648 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Feb 22, 2023 at 7:28 PM, Sysadmin Lists w= rote: On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the sen= d stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get th= e "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on = port X, then pipe the send through netcat to the remote system on port X. T= hat way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrot= e: On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed=20 >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which=20 >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E2160).= =20 >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >=20 > The CPU I have has > aesni0: on motherboard >=20 > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any=20 >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on=20 >> the other core. >=20 > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman You could pipe the stream through an encrypting program before piping to netcat, then decrypt on the recieving end. $ zfs send | crypt | netcat ipaddr 2222 $ netcat -vl 2222 | crypt | zfs recv I don't know if zfs can handle that, but worth a try. $ man crypt =C2=A0 =C2=A0 The enigma utility, also known as crypt is a very simple encr= yption =C2=A0 =C2=A0 =C2=A0program, working on a =E2=80=9Csecret-key=E2=80=9D basi= s.=C2=A0 It operates as a filter, i.e., =C2=A0 =C2=A0 =C2=A0it encrypts or decrypts a stream of data from standard = input, and writes =C2=A0 =C2=A0 =C2=A0the result to standard output.=C2=A0 Since its operatio= n is fully symmetrical, =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it. -- Sent with https://mailfence.com Secure and private email Seems to work: # zfs create zroot/test # mount -t zfs zroot/test /mnt/test # date > /mnt/test/testfile # zfsnap snapshot -p testsend- zroot/test # zfs list -t snap zroot/test NAME ... zroot/test@testsend-2023-02-22_19.46.15--1m ... # nc -l 2222 | crypt | zfs recv zroot/newtest Enter key: # zfs send zroot/test@testsend-2023-02-22_19.46.15--1m | crypt | nc -w 5 lo= calhost 2222 Enter key: # zfs list zroot/newtest NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 USED=C2=A0 AVAIL=C2=A0 =C2=A0= =C2=A0REFER=C2=A0 MOUNTPOINT zroot/newtest=C2=A0 =C2=A0 96K=C2=A0 70.7G=C2=A0 =C2=A0 =C2=A0 =C2=A096K=C2= =A0 none -- Sent with https://mailfence.com Secure and private email ------=_Part_92257_810470091.1677124263648 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=

On Feb 22, 2023 at 7:28 PM, Sysadmin Lists <sysadmin.lists@= mailfence.com> wrote:
">

On Feb 22, 2023 at 1:43 PM, Freddie= Cash <fjwcash@gmail.com> wrote:
">
[Sorry for top part, GMail sucks for replies.]
<= br>
If this is a LAN or private WAN where you trust = the network, piping the send stream through netcat will remove ssh from the= equation.

That's what we swit= ched to using once it became almost impossible to get the "none" cipher wor= king with ssh on FreeBSD.

We use ssh to connect to the remote server and enable a netcat listener o= n port X, then pipe the send through netcat to the remote system on port X.= That way it's logged and uses ssh for authentication.

We easily saturate gigabit links between our= ZFS systems using netcat.



Cheers,=
Freddie

Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:= 31 p.m. Miroslav Lachman, <000.fbsd@= quip.cz> wrote:
On 22/02/2023 22:08, mi= ke tancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could pipe= the stream through an encrypting program before piping to
netcat= , then decrypt on the recieving end.

$ zfs send | = crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but worth a tr= y.

$ man crypt
    = The enigma utility, also known as crypt is a very simple encryption
     program, working on a =E2=80=9Csecret-key=E2=80=9D b= asis.  It operates as a filter, i.e.,
     it= encrypts or decrypts a stream of data from standard input, and writes
     the result to standard output.  Since its op= eration is fully symmetrical,
     feeding the enc= rypted data stream again through the engine (using the
  &nb= sp;  same secret key) will decrypt it.

=
--=20 Sent with https://mailfence.com =20 Secure and private email

Seems to work:

# zfs create zroot/test
# mount -t zfs= zroot/test /mnt/test
# date > /mnt/test/testfile
# = zfsnap snapshot -p testsend- zroot/test
# zfs list -t snap zroot/= test
NAME ...
zroot/test@testsend-2023-02-22_19.46.15--= 1m ...

# nc -l 2222 | crypt | zfs recv zroot/= newtest
Enter key:

# zfs send zroo= t/test@testsend-2023-02-22_19.46.15--1m | crypt | nc -w 5 localhost 2222
Enter key:

# zfs list zroot/newtest
<= div>NAME            USED  AVAIL  &n= bsp;  REFER  MOUNTPOINT
zroot/newtest    96K&= nbsp; 70.7G       96K  none
=
--=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_92257_810470091.1677124263648--