From nobody Wed Aug 09 12:49:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RLVJc6cKXz4pwC0 for ; Wed, 9 Aug 2023 12:50:08 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RLVJc65Tbz3dYq; Wed, 9 Aug 2023 12:50:08 +0000 (UTC) (envelope-from otis@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691585408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H2fVsmoqLwev+y1MmKhqLRIrvoAffyjdoDWO53yrr7s=; b=bV8Xo/8miPY29AmyyHOo6mmuweit2FdekzutXcuBtWdzYFWj+spUDf+Q5xLRrL8LdojDY/ ma/O8eVhuTn7GSULSorYjEylcYFUVOlD7ZFjaP3ZtMNI3G/MioTGok9WBxeh2Fpl8u/3Va Dxa8pOVTKg2KH5jO0fHQSczhARew4wwKgBpTpgln9CB1izaO7e9I3qbk4RLnsbtWAaPtUL I84+Ll++4TK/IaNNaCKheRk9VT/j76vKO9fRotd03TpTBxPAjepYzor6G3HU4WM+rVKelX mRQBvqb/pp47a95iU4djHob76qMgSs696E+csVA2+0paMANYFX53LTMky+eSCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691585408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H2fVsmoqLwev+y1MmKhqLRIrvoAffyjdoDWO53yrr7s=; b=KKAgdQgBUW7LPC2B/Wijctg6ab+YeNRbXOW/HdKszWYLGLLnPjROzlsEqI3BGn10hkSZwe SLB14uZwS6TSF+qU4RkTID/UosCMD36mN6fdNBxhidBoWz0VHLLCJFy97epj7sU0CVnIf5 vEO8FTSQWwYGoeczp+o+1/KidcBpM53tqVjgPL1fkRANawgtZj3awcJVRwz69PNi5YonEg RTmgOO4n1ux7LqLd65r1uDXP/zaOXWASWZY3JxLEPISNUf3mRJsHtnQLVN3g1Wqj185Ycy vtYwRmjmRkZ3ZDaFfZ5qyDlLL+OyNgeWLcTf1t4YpPyXcO1/a2dMX02f7mZ0JQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1691585408; a=rsa-sha256; cv=none; b=AE0X2clCDEF+1RaPn1zzGhiG025SL/PQu3nHug5uGcLCZIZLjADMQZM5kOWKOleJC5ClZR 3sDI7p3ltno8sognJbLlCyZ7GVsOUfDVOOajuSLep3ks251LVzSCbKeDE4X3E9GgsnSLgf i+FuK5tAmHfu3oK4ue6ot5oIFRY8O+xPFcHMbDowRrYffDGpOkDxAVWE4nqWi5iHWJh8a8 wB4qXxMXq8CVsoOGh0lLe/1+O09gW/HLKTsPtUO5jnri8pZwo8ho3uu7JH9pse/bs8R/EG lGSllfnlL/iebxqjngFr19R1gquwcOS6tMRRlHIvLzIa4Uq45eq//whIxXXXjQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from ns2.wilbury.net (ns2.wilbury.net [IPv6:2a01:b200:0:1:f816:3eff:fecd:13e6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) (Authenticated sender: otis) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RLVJc4N6jz13yL; Wed, 9 Aug 2023 12:50:08 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtpclient.apple (unknown [217.73.28.193]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id DD56B61F98; Wed, 9 Aug 2023 14:49:59 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: can sftp be made multi-threaded? From: Juraj Lutter In-Reply-To: Date: Wed, 9 Aug 2023 14:49:48 +0200 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3639ADD0-3715-4B2B-9C25-EBF367FC09A0@FreeBSD.org> References: To: void X-Mailer: Apple Mail (2.3731.700.6) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on ns2.wilbury.net > On 9 Aug 2023, at 14:40, void wrote: >=20 > Hello hackers@ >=20 > Can sftp be made multi-threaded? Would rsync-over-ssh accomplish the = same thing? >=20 > The issue I'm having is with large backups taking ages to transfer > even with both source and destination being on gigabit fibre. >=20 > It looks like sftp is single-threaded and the bandwidth per thread is > something like 5-50 Mbps and it's nowhere near saturating the = bandwidth. Would multi-threaded sftp (if it were possible) > or rsync-over-ssh work multi-threaded on a single huge backup file? Have you played with misc/mbuffer from ports? It might do what you are = looking for (provided that sftp alone isn=E2=80=99t sufficient). =E2=80=94 Juraj Lutter otis@FreeBSD.org