From nobody Wed Feb 22 22:22:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMVyf6BjBz3sHZd for ; Wed, 22 Feb 2023 22:22:34 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp1.bway.net (smtp1.bway.net [216.220.96.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMVyf39Nxz3mGP; Wed, 22 Feb 2023 22:22:34 +0000 (UTC) (envelope-from spork@bway.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (pool-173-70-201-95.nwrknj.fios.verizon.net [173.70.201.95]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp1.bway.net (Postfix) with ESMTPSA id D3D7111816; Wed, 22 Feb 2023 17:22:32 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bway.net; s=mail; t=1677104553; bh=yq8Xcd5iGF7txSLvXp21z8+gl+CbX3s7gHc4eCWzITE=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=Oaz+qf98543xUMSa3bFh3ZPGyIQg/dO0MCpd6CChbSbf+vBjLZFkgitA/laB9BXGN ngTUyOcGgYPcKAaenIcT4qnhMuLldAT0UcvlEE2lG8X404u6rM/qayZbciTwbNFMsu GYdNc5Uwlmhtrc9Li1QFOxXBZNaeTVzmqm8Tn/54= From: Charles Sprickman Message-Id: <0171E506-3899-42B2-B7DC-4145BAA595D7@bway.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: speeding up zfs send | recv (update) Date: Wed, 22 Feb 2023 17:22:32 -0500 In-Reply-To: Cc: Miroslav Lachman <000.fbsd@quip.cz>, mike tancsa , Alan Somers , freebsd-fs To: Freddie Cash References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> X-Mailer: Apple Mail (2.3696.120.41.1.2) X-Rspamd-Queue-Id: 4PMVyf39Nxz3mGP X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8059, ipnet:216.220.96.0/19, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Feb 22, 2023, at 4:43 PM, Freddie Cash wrote: >=20 > [Sorry for top part, GMail sucks for replies.] >=20 > If this is a LAN or private WAN where you trust the network, piping = the send stream through netcat will remove ssh from the equation. >=20 > That's what we switched to using once it became almost impossible to = get the "none" cipher working with ssh on FreeBSD. >=20 > We use ssh to connect to the remote server and enable a netcat = listener on port X, then pipe the send through netcat to the remote = system on port X. That way it's logged and uses ssh for authentication. >=20 > We easily saturate gigabit links between our ZFS systems using netcat. This is kind of tangential, but is there any ssh client/server that is = able to make use of multiple CPU cores or is that just not easily = possible? The first set of hosts I worked with that had 10Gb/s internal network = ports kind of showed me how much of a bottleneck trying to encrypt with = a single core is. If using netcat or similar to avoid the ssh overhead, can IPSEC or a VPN = option (wireguard?) be a bit of a workaround? Do any VPN implementations = on FreeBSD put multiple cores to use? Thanks, Charles >=20 >=20 >=20 > Cheers, > Freddie >=20 > Typos due to smartphone keyboard. >=20 > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz = > wrote: > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed=20= > >> with chacha20-poly1305@openssh.com = > >> > >> > >> It seems the speed of SSH is limited by single core performance = which=20 > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160).=20= > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > >=20 > > The CPU I have has > > aesni0: on motherboard > >=20 > > which probably helps. >=20 > That explains it > aesni0: No AES or SHA support. >=20 > >> I know there were some HPN patches to ssh, beside that is there any=20= > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else = on=20 > >> the other core. > >=20 > > It looks like you are running into a CPU bottleneck TBH >=20 > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 > without some tweaks on ssh I will not gain more speed :( >=20 > Thank you for your help! >=20 > Miroslav Lachman >=20 >=20 --Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On = Feb 22, 2023, at 4:43 PM, Freddie Cash <fjwcash@gmail.com> = wrote:

[Sorry for top part, GMail sucks for = replies.]

If this is a LAN or private WAN where you trust = the network, piping the send stream through netcat will remove ssh from = the equation.

That's what we switched to using once it became = almost impossible to get the "none" cipher working with ssh on = FreeBSD.

We use ssh to connect to the remote server and = enable a netcat listener on port X, then pipe the send through netcat to = the remote system on port X. That way it's logged and uses ssh for = authentication.

We easily saturate gigabit = links between our ZFS systems using netcat.

This is kind of tangential, but is there any ssh = client/server that is able to make use of multiple CPU cores or is that = just not easily possible?

The first = set of hosts I worked with that had 10Gb/s internal network ports kind = of showed me how much of a bottleneck trying to encrypt with a single = core is.

If using netcat or similar = to avoid the ssh overhead, can IPSEC or a VPN option (wireguard?) be a = bit of a workaround? Do any VPN implementations on FreeBSD put multiple = cores to use?

Thanks,

Charles




Cheers,
Freddie

Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, = <000.fbsd@quip.cz> wrote:
On 22/02/2023 22:08, mike tancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best = speed
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance = which
>> is very poor on this machine (Intel(R) Pentium(R) Dual  = CPU E2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on = motherboard
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there = any
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything = else on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



= --Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD--