From nobody Wed Feb 22 21:43:24 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMV5j3kkKz3sFS1 for ; Wed, 22 Feb 2023 21:43:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMV5j3J17z3hPm; Wed, 22 Feb 2023 21:43:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-536e10ae021so102351797b3.7; Wed, 22 Feb 2023 13:43:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=2SjvyeMs4Im0k1KmHMh4roJKEFQh8AzRLRta3IB7RWQ=; b=KeSktspGOnCCfL5hUCmz3kdaPcLw0J9D1Ft0KvCpbpVSU0KVYvzWCgETFY3spvRSjQ ptuFOd+gPlerK8/0uRQ/jQVeE0rof/z21oXH54aXdHdzcVpZR/E2SA9x42DCPytXcs5F mfGMfPRcLlKG2LtbO50I9IUgiWbBjFBDY4bMOrtPKoe7OWRDyT5Be4kXGbNGnLWKwAci +LzfmloncGs3oM9pXIgNxMLmtlwY0GwwxdqCjaiVL97CvvqXliFVyqJ2xtUQi+7eMrp9 vrJEYhNKkKJ8l50br9HPwGdkggr616mPugM1w4Tj3qwTHZCd3TslAYT5roq/alRzbYEM wu6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=2SjvyeMs4Im0k1KmHMh4roJKEFQh8AzRLRta3IB7RWQ=; b=GjmIs1HwV0kAdqvxenvBr6voJ3jL1fukfMc7PgAsFde6S1zFOWtPAV15X0p8Ii3IMZ gy0WUo2tGOzgpiLWCVh110RRkHG8Wl92DR8FZ+pecrQWIxQK8O08VAL54ok04Rr2f2CT ivEI6/+FLWJwqEbC0zU9j11+JHRvq3sy06pCTLPKIzP2pG+UaT55D9fh3D3PalDu8I7I 5pcXlReCyt81JnG2BH+U4m2eo66Z3y6PDETZC3eoUJyavoT30MnYc8sDoBUerf0TdBdA Pgs9pLuzVvbODyQRQXBx3b+vbBLxrXli8Rco5kpLBmZlHr7EFJGj9jlXfOd+R6FWiuph UMiQ== X-Gm-Message-State: AO0yUKXbXSttf4zTM3JgeQnXwNM+L+lA2M1sthiMxkg9hOTRQWHvMaJ+ 0mrqCNRvtabDT/MvD1JVXWZxqCrv3ESJc7pggKW6fyeK X-Google-Smtp-Source: AK7set+DIigyQx0zQua+kqfZli45uBWX/TRXG7VJglcYaNMyAkmaCW2makCKJhBfqDKK/KQTtPiwUbzTolQzBDoho0Y= X-Received: by 2002:a05:690c:a97:b0:52e:b74b:1b93 with SMTP id ci23-20020a05690c0a9700b0052eb74b1b93mr1772271ywb.0.1677102216687; Wed, 22 Feb 2023 13:43:36 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> In-Reply-To: <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> From: Freddie Cash Date: Wed, 22 Feb 2023 13:43:24 -0800 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: Miroslav Lachman <000.fbsd@quip.cz> Cc: mike tancsa , Alan Somers , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000e0b62c05f550c911" X-Rspamd-Queue-Id: 4PMV5j3J17z3hPm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000e0b62c05f550c911 Content-Type: text/plain; charset="UTF-8" [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the send stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get the "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on port X, then pipe the send through netcat to the remote system on port X. That way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrote: > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed > >> with chacha20-poly1305@openssh.com > >> > >> > >> It seems the speed of SSH is limited by single core performance which > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > > > The CPU I have has > > aesni0: on motherboard > > > > which probably helps. > > That explains it > aesni0: No AES or SHA support. > > >> I know there were some HPN patches to ssh, beside that is there any > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else on > >> the other core. > > > > It looks like you are running into a CPU bottleneck TBH > > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but > without some tweaks on ssh I will not gain more speed :( > > Thank you for your help! > > Miroslav Lachman > > > --000000000000e0b62c05f550c911 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
[Sorry for top part, GMail sucks for replies.]

If this is a LAN or private WAN where yo= u trust the network, piping the send stream through netcat will remove ssh = from the equation.

That's = what we switched to using once it became almost impossible to get the "= ;none" cipher working with ssh on FreeBSD.

=
We use ssh to connect to the remote server and enab= le a netcat listener on port X, then pipe the send through netcat to the re= mote system on port X. That way it's logged and uses ssh for authentica= tion.

We easily saturate= gigabit links between our ZFS systems using netcat.



Cheers,
Freddie

Typos due to smartphone keyboa= rd.

On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, &l= t;000.fbsd@quip.cz> wrote:
On 22/02/2023 22:08, mike tancsa wrote:<= br> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman


--000000000000e0b62c05f550c911--