From nobody Mon Aug 28 10:03:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ5jH2GCMz4rMmq for ; Mon, 28 Aug 2023 10:03:15 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from HK2P15301CU002.outbound.protection.outlook.com (mail-eastasiaazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c400::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ5jG0Xsgz3Rpw for ; Mon, 28 Aug 2023 10:03:13 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hymMV0fKPINgcnYrfWw5FDcw9FwYn1F+RGk3Isu5X5/ydBhRGVOLW/Iu05miiwgbvuhzffqrrrxjlxNnQL7/tgnKgr+YTmy2nAimQ/uxkxMChV9EbmQEBrKBuKJX6ye56XJm0HkHZUj1xVt368y7/oHmQLMrXyoKOA3nKJCQKZVfO7vATy+5YAV7QTVV6p2QRIrIymODlJtMcTGlInPoUeGvpY450+yJ0d7aq3U4hwmylOIQUOPxkfJF6pwc2DBkLuuiki9XXz2/qPpDb55yP7MBkS0xWvdUcAcVS8XxgeX72NhR4Y1YsjIILV8VFvDuP1sUaNivfLeEe5JlLYol6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WvSV6+ijK4D/EnQ5ISVvVJbcM9DOnnbCQ2YrGpJ9WIc=; b=R4IqoUWRHwrUAUWPIdyIlZL1NxtoPwXPYxtdfwIOTyqC9+eRcr2qoTczHzOH3Hm71VOLKHVuqar39i6JjPzDJVdI4nDwx0hM8sRKU9FBUlwGu/TizvuYZ/HC/tp5gvK00K9hduoE74E62ZkkvT/3/8/m5XHxh5q08Xxp8TnfSF/eRSKiV8oqk9rcXnOd3lV71vKo7YIjjTysDEf4xmkCYvHbYikpQS+PR2eupKGqbYVpi/ISbofn4gt5QbqUVyAVQv1Frmfelwep0Zfj/ZRifgQzHD5N0ixo/Vrh4qxoK4W4sEAjfBItjkb3O4ztch6+dLDaSS+JkAmK5iIOEzC+/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WvSV6+ijK4D/EnQ5ISVvVJbcM9DOnnbCQ2YrGpJ9WIc=; b=ClEsQIVj+CznPVSgca/ntgMk/DpTKSZ076riLjOjuHPtB6lG1MLu6L7x/VgX+QqZvIo4izzj603cSdSJuv851Jlbp8/yQxCkary/ETuyQEhmQ+XPz6JyDiuSVirQdliBB4j/LQe8aKT/X9UmvaCKB0A89E8RR+hv2sIEhdYB3Q4= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by PSAP153MB0501.APCP153.PROD.OUTLOOK.COM (2603:1096:301:63::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.6; Mon, 28 Aug 2023 10:03:03 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.005; Mon, 28 Aug 2023 10:03:02 +0000 From: Wei Hu To: Mikhail Zakharov , "freebsd-hackers@freebsd.org" Subject: RE: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AdnZfvJbaS4mlPy2QcSWspi9tSFg8QACqxQAAAM0yTA= Date: Mon, 28 Aug 2023 10:03:02 +0000 Message-ID: References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> In-Reply-To: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=121293ab-a406-4fd4-b2ba-d3f7878a6681;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-28T10:00:16Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|PSAP153MB0501:EE_ x-ms-office365-filtering-correlation-id: 11b27bc6-eed1-4af8-d8c7-08dba7adf796 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3FbPRiTjXAUmQTm9+IJsTF0Dno1v30BMiaqPhMg4THcMNDSIat1VyodSC3jN9KjGXe6zl5MM7ysCIAjftnpkFxDKfvhoUNwfLSKnvLl2RvmsVGLoweb3hqP7uJgxVIxt6+snPfopDQozta1QpifdOImA+6vsJepwkBqgJt7ZG5lXVddJN5UWatKoP/DBO4fvmNaH2muQkBoDMyp3wL2+oD0+HhvZNGZHDdkBlIwk9xrlbtzNqsM966HG+DHBkgk3KTOvxSVJeOQ0xYV2kuGXY40U3ck2YOKpDYzW0UgpDkhRWmueThNLxcMVRy0e1GcocyUm6xly076GhuAo2Y/mHpK7kgkRU/ENcxy+L0+izwIlYjiopyVjP72UM/7T8dSxmy8rMTktZ8QYDD66pa+WCvWGkxcOkkN+D3p3OCo3yd5B3eDmgtTNYgNtWOEOWqaJkw94pA77U13KXICtTPcJgt8OOGAaEr48A681ee+ICFCJzzrpNmCBjdlD+EFMEj7uugaml1S+y5J3NJVFPYaso/C99YAKtzeZLWBaX765Dgsjfz/0IkjRlgcUmkBtpLCWxBjjr6HYEqqDqjMSnkYMMGG1k4D8Ln7KX8b8bvME0nyK8TNJeDd3Y54wLvD/DD+j9yQ0cSLmINx7iw85WD3ow5zaPDKbXVnT/iNRtS4ivKXtjM0OVSgZ60R0N6rBbs2g x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(366004)(39860400002)(346002)(376002)(396003)(136003)(1800799009)(186009)(451199024)(41300700001)(82950400001)(122000001)(38100700002)(71200400001)(38070700005)(86362001)(10290500003)(82960400001)(966005)(478600001)(83380400001)(8990500004)(26005)(9686003)(53546011)(7696005)(6506007)(55016003)(66946007)(76116006)(66556008)(110136005)(64756008)(66446008)(316002)(2906002)(33656002)(66476007)(8676002)(8936002)(5660300002)(52536014)(12101799020);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?e8xI0jQaNXQVqKt7S/0UHQwo+K5ke5wGuAh4xCONQPytUTECyhBV00cFUVyU?= =?us-ascii?Q?nzNnpOJmZ4HuaMI08AgkAySg/mLJeZXaKgdxU1oxkXVA42aUFIU6hFRcCKYT?= =?us-ascii?Q?rnD6PVBQSkovkcttxmwEkehG/c8DjZ2amhlKKMTBYpjyvYfX6wjRT7J3ZMVM?= =?us-ascii?Q?WW8y8WMDLE1ARMMKe3w9wsbuXAepeRKq9Mt+BfBdIoZcCvLe8pstpC9f6XQL?= =?us-ascii?Q?DOFAj3SOuDPTEmUZ1Zjuovi3SIovwZu3uG0rB89yh+fmqf9hahHMz0vQG/9w?= =?us-ascii?Q?k7pgYxjSSxdd4hXGV+pMBNKzqbRyh+NSgwdpCsHysq5+MPT9JCM1pQWvcVt9?= =?us-ascii?Q?jMXVuf63MINiolNtft82KqbKDqJ7YGUdtq5zBMEI8CdbNKoJcwIIWNnasfzA?= =?us-ascii?Q?GChBx4vhGSRVjSHnX9K6HQIuYyfyyXX9XGDfGFI18K/po3o2npy+Yxos9JQt?= =?us-ascii?Q?sf8rrOA2dl4+IfoZvNoDJCYAyt7NvV3W94X7q2LYTLe4K6mKlVrObmfYTfG1?= =?us-ascii?Q?ae64hCcPrxs122OOTf5hursDEFbq66zMzO7D3hfHHPhkThPINExa6qo3855v?= =?us-ascii?Q?Ri9eMQaW7QG5uoj/g15DPMDY5yC6UAx74hmYwZKEv2Yfp3SnUJt5SJToon9U?= =?us-ascii?Q?AsV2AlPey4D26GEN9TPA44n1joQIvjqfYAW/TCYYvSRsIu7/NUAz1WXIp5Z9?= =?us-ascii?Q?Iupwz02MmXBK3Beatl5j8OZz1pMVlzVEuQz0Hihu0TzTtXo4oOqIBEa/LAMA?= =?us-ascii?Q?YgTVL3YMJeEtMTE1GDAX4E1Cc22eB1244L/dpJfT9/9g5Q+r975NCf/Ar8QJ?= =?us-ascii?Q?DKj9Z4sGWVAYvJoDT/5FIwYaPes4qGhQs6Q2ows4udFEYHQFjpH2m3wBPGFK?= =?us-ascii?Q?lbXedzCnXeiG8ZNz3nLm66i6REaj7XrdJ8GQ3DrNAT7muGBJ976vaMN2Vp1o?= =?us-ascii?Q?6ZO7gHIra/sQ47kPL54lYu8HEsNB3V00M191wWBNJw93JGXh3caBf9MXk8+5?= =?us-ascii?Q?o4A/9rkwKFbs7u/iCRliOntbuPpekzFB0VR40D+Mydy8cTRvU9xPDCTe7OVc?= =?us-ascii?Q?sjaOtQ97tkTc4I3QOisnrH1cIzVkVnR9ymr6FkEBvO7xjg7glBhtgy+kpWfR?= =?us-ascii?Q?7UYEmx3HiC7M6YtQseZpH5ykLTYWCpLw5c/3oAb4RfpmvDhO/rXCw5u6khQp?= =?us-ascii?Q?N9S9L/z+cDi9eIMOtig2ak/ZmGcr/1D4le5USGyPAjDwIkXadC9o9Mn47SKc?= =?us-ascii?Q?/o8JlV/K3Aajc+6SQV+6y7ANUNuwXbpY2L7LCgR4maMEwausnT76HkSrcsWd?= =?us-ascii?Q?S/wqhs5Wwhp4M2T5mrWfZ5Kh/Y9ixogUUu9Hzm7qxdRETkek8LqrQOTM3tY0?= =?us-ascii?Q?VV4ogbeJLSUDEGm4rMycvDbJ47LqC3y9cyjZpRmbvQztcf7CqznBUfN32mPt?= =?us-ascii?Q?3R1aaCl38lzWX67rXnu0pHOZzfQNS0UwUEOH0Ir7s1t9v087J98ehzVzZ2+g?= =?us-ascii?Q?m1FjGLWrZWn9O/bLN3toFRSbT8btSDgAJMAu2NwPncCsRCXQGGySej0i/Xmb?= =?us-ascii?Q?19n8LMtfJrawgF/ic3g=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 11b27bc6-eed1-4af8-d8c7-08dba7adf796 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Aug 2023 10:03:02.8070 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ob4tlHxI9193V7YF1NlbwZAS3yUEGcmg7h/x8dRzo8O3A+qx2gcDnOVT8hYT3Z4/F9nJFShtmzCGxCKtp6/9Ig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSAP153MB0501 X-Rspamd-Queue-Id: 4RZ5jG0Xsgz3Rpw X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] Hi Mike, All VMs are with same type and they are in Azure. If the copy on Linux is b= eing cached on the RX side, so is FreeBSD? Thanks, Wei > -----Original Message----- > From: owner-freebsd-hackers@freebsd.org hackers@freebsd.org> On Behalf Of Mikhail Zakharov > Sent: Monday, August 28, 2023 4:28 PM > To: freebsd-hackers@freebsd.org > Subject: Re: Very slow scp performance comparing to Linux >=20 > [You don't often get email from zmey20000@yahoo.com. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] >=20 > Hi, >=20 > What is the drive back-end of your machines? I mean, what is seem to be f= ast > copied could still be cached in filesystem/storage buffers but not actual= ly > written on the disk. >=20 > Best, > Mike >=20 >=20 > On 8/28/2023 9:32 AM, Wei Hu wrote: > > Hi, > > > > When I was testing a new NIC, I found the single stream scp performance > was almost 8 time slower than Linux on the RX side. Initially I thought i= t might > be something with the NIC. But when I switched to sending the file on > localhost, the numbers stay the same. > > > > Here I was sending a 2GB file from sender to receiver using scp. FreeB= SD is a > recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. > Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has > 16 vcpus, while Ubuntu VM has 4 vcpu. > > > > Sender Receiver throughput > > Linux FreeBSD 70 MB/s > > Linux Linux 550 MB/s > > FreeBSD FreeBSD 70 MB/s > > FreeBSD Linux 350 MB/s > > FreeBSD localhost 70 MB/s > > Linux localhost 550 MB/s > > > > From theses test, it seems I can rule out the issue on NIC and its dri= ver. > Looks the FreeBSD kernel network stack is much slower than Linux on singl= e > stream TCP, or there are some problem with scp? > > > > I also tried turning on following kernel parameters on FreeBSD kernel. = But it > makes no difference, neither do the other tcp cc algorithms such as htcp = and > newreno. > > > > net.inet.tcp.soreceive_stream=3D"1" > > net.isr.maxthreads=3D"-1" > > net.isr.bindthreads=3D"1" > > > > net.inet.ip.intr_queue_maxlen=3D2048 > > net.inet.tcp.recvbuf_max=3D16777216 > > net.inet.tcp.recvspace=3D419430 > > net.inet.tcp.sendbuf_max=3D16777216 > > net.inet.tcp.sendspace=3D209715 > > kern.ipc.maxsockbuf=3D16777216 > > > > Any ideas? > > > > Thanks, > > Wei > >