From nobody Wed Aug 09 12:40:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RLV6005HZz4pvQp for ; Wed, 9 Aug 2023 12:40:56 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RLV5z0t1tz3b45 for ; Wed, 9 Aug 2023 12:40:55 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=RWojPRDR; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=KlB3PabC; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.20 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 32A2432003CE for ; Wed, 9 Aug 2023 08:40:53 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Wed, 09 Aug 2023 08:40:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to; s=fm3; t=1691584852; x=1691671252; bh=++1kqpKWgQtIC0xY0bfOUHc8z dehyE8iwdcPa06ZSXg=; b=RWojPRDR9wThKJLy4Ew6hjucU0TYDLu4t6vB4VRXZ AAK5QoM5cjlXWlfoA+n0vi6qSBCgjlNUaNUxcudZfdZMYxG9aquD+N2VxVjPeogk B2cPovUA4fdk1L7UcPDNwJfWq4uhLJfNM+0tdIXTxECfRXNUoJ1DfDI74INEfYeN Q/09/nrJMK+6JhNfLvSbljBR/I+ycYIVSRZ3+fuSSs7Fso+OEDOnPCnaN/g0tvVH 4K52lKR8E8Onu1AlEjlQbbfst45nimSgApQ3RelD7kXNhE0BiD8DUnB/N8lI7F9V pmp+yJAEczR9uqZYEWnOje/0zobossxo4pEFbqbVAyxIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1691584852; x=1691671252; bh=++1kqpKWgQtIC0xY0bfOUHc8zdehyE8iwdc Pa06ZSXg=; b=KlB3PabCV9Y8AVmhPvmFl40IkIFY/ETqBQX41VEoubNRJxE8EBm j3Rq2MGeUz22B33bhWzd8FvDTV1mz/2ADaaZ59qXJ7gCP26iY3EQ8Mu93gUVkLu6 kvCiuM+pBF9LeA0TJwtgv+6QHf2j7v2jBaYd5Y5A50/eI3N2yz3ygYWDKAnFAUgG 7NlWOX5bK3ZIO15q40Yl+ZnbZlPBoi9M1qQSVPco1imsuGZC/nbUXmFjrG5ggDIT SfT3LOtpZzIweymBm87+GX83m3PW64xOJ4iij6gQHM6CZ7ygMF6BhvzAyH5TXl8i QgQFhC31LWJ8sVPltNvv0o4XUpAJ3I7wx3g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrleeggdehfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehttdertddttd dvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrghtthgv rhhnpeevudffiedvffffgffhgeefjeefffdtieetheetkeefhfdvfefgtedtueehgeffue enucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhi ugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 9 Aug 2023 08:40:52 -0400 (EDT) Date: Wed, 9 Aug 2023 13:40:49 +0100 From: void To: freebsd-hackers@freebsd.org Subject: can sftp be made multi-threaded? Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Result: default: False [-3.07 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; MID_RHS_NOT_FQDN(0.50)[]; RWL_MAILSPIKE_EXCELLENT(-0.40)[64.147.123.20:from]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.20:from]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_SHORT(-0.07)[-0.070]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RLV5z0t1tz3b45 Hello hackers@ Can sftp be made multi-threaded? Would rsync-over-ssh accomplish the same thing? The issue I'm having is with large backups taking ages to transfer even with both source and destination being on gigabit fibre. It looks like sftp is single-threaded and the bandwidth per thread is something like 5-50 Mbps and it's nowhere near saturating the bandwidth. Would multi-threaded sftp (if it were possible) or rsync-over-ssh work multi-threaded on a single huge backup file? In my example, the ssh systems are both OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023 ty --