From nobody Mon Aug 28 07:32:35 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ2Mc5YM4z4r01V; Mon, 28 Aug 2023 07:32:44 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2105.outbound.protection.outlook.com [40.107.215.105]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ2Mb35C3z3CWY; Mon, 28 Aug 2023 07:32:43 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=microsoft.com header.s=selector2 header.b=PLxewIZe; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of weh@microsoft.com designates 40.107.215.105 as permitted sender) smtp.mailfrom=weh@microsoft.com; dmarc=pass (policy=reject) header.from=microsoft.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SJ9KOURO2/nY1FBfPCOV3FtpHkDkAjbehnOxQMgyK/6VWVPrHigeY0kDDtwak89x6zT5STBRyUApwKvQnjDtLRVAT1RJspBPU4ROdkrEfKwLtG1SfrVQombxa/5F1qRpEh1DtNux7PjmtAHnrBM6seOmewD5IylOtxp3mx4uRr15+Fm9jjUZgJ73QLjEpduVA81PCS1muUaBeJLW/O13JlJ/t/E/mdH4wFVOgsIPbb6EVMMHxXG+z72OQeOndlYioRpRNQUzKnYceNDPW14HIlwyWOD0Hbb+eWOVSlVrT1u580HsXB4IR717dNpVzPvvVGkSoM+n+IwyJQWCRtHTjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P2U6L2iDPUzMhus1Cndi4Bym8K6J4WEs8tq9xnXEo80=; b=Nr8R5e0vx60QxJuLYGQlgM6mZxFF+AF5KR+2n1x/S3/96e1S+QRqK1zEbuJv6cAlikHY48K1gVeJdzxqjNGF1QQXp9ky1M4lDyrsczFlhzBCY2+oVdEP8cAvHSvjdRJV93tXXmvZqdSRG8kj5wjRfayyIRBvQDTk8wPJinNePSRjVp/6QFsE2sbfgTU1ata1NNx5sbqsAAyjDQu/1Gn3oOvAukwZ9RiB00M9W2EyOgiCGVpBMgHAUY5yC/jxiedVh1f+MYkELhWKwGFfjWYMOXtrNSguHGYLaTPHEwAOGwsuki0kVyWurke/rOHrci/jA1mwNUDC+1/ebHz0Z4yKgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2U6L2iDPUzMhus1Cndi4Bym8K6J4WEs8tq9xnXEo80=; b=PLxewIZeRWegUK7zyfhm12yOJ8i6i+C6GMUfC96JZwhHTAr6lwI18Yi3shqLwrzS9VNuCGP1ABFpzgg5tjr7NPXJVOG8h4PO7BTOAmTdGtcgmppWNaSQegApO79hunnPdqDMERVio3270F78+zRqy53No/F2Ohgcq2Ketxo9f3c= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by SEZP153MB0791.APCP153.PROD.OUTLOOK.COM (2603:1096:101:a6::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.6; Mon, 28 Aug 2023 07:32:36 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.005; Mon, 28 Aug 2023 07:32:36 +0000 From: Wei Hu To: "freebsd-hackers@FreeBSD.org" CC: "freebsd-net@FreeBSD.org" Subject: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AdnZfvJbaS4mlPy2QcSWspi9tSFg8Q== Date: Mon, 28 Aug 2023 07:32:35 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=c0bd6cc2-296c-4e45-ab32-f5c2dce362b4;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-28T07:11:28Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|SEZP153MB0791:EE_ x-ms-office365-filtering-correlation-id: f3c5db47-01c9-41fa-5405-08dba798f2df x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: e+NCVxVbVI5oa5VonIW3B1F3HF47QEtE7ZsxQlo+j6CN5Et2d/nn41wWUA0qBYF0a1A5WNMWch+XID8VtDAIexn2zTG7gMyg84JxYqDFYyrhrgQD4IEFGYeVBfZF9D7M0QrwIVCS59Djs/QyNtX1mmaVrOhqflclyWIGgWW00hkyA3poQKSQA0m5gy2pppieCdw1TBwSjawXapTfMoQIlQoCx3YZ4m8HqqIj9/ln5fQO/e+VOJi08ps0txwWz85Mf65/OB8UMxCpt71znOxtrQHjCVCVFI2yRZ0jKab4fbenFOiIq2U01BC2iaFXqqPiyJEh+76r9P2G/TXcqEjc0sYOFl0vwT3j8AVrdjmju6YpJjROXQchVitdQW0C9vriHPI0i4p7gbrA3s9E5FiJugBCy8MYxpts40vuwKspCJTyZm2LCUCROUEkZC6ghSF7QggM0aFbNKfKUjYCErLCqW7kT3m7rP78Bnu5DqhkNwJbbB4VyWwUu1nztGP4d/ArMD0CjunEVA26zk4yMrV1gh/nJypgCLxRWyuctl4yIs6Nl/U9nXC1m7oRqh4nPohXw9RbP3eid9igSsTCrvCCWH6wgyIlk0rj0lTV6QHKqDKAKGMFAxACmR6vOC1YelAIRFndZBiKw7nrqNLiYj6vhYgoyX5eEjTzgXcd8FfngjhCmNqdQaexMzTtFp7EUSAbuKs5uvwH2g0ACg1S8bhtuQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(376002)(396003)(366004)(39860400002)(136003)(346002)(186009)(451199024)(1800799009)(4326008)(8676002)(8936002)(5660300002)(316002)(6916009)(2906002)(66556008)(66446008)(64756008)(33656002)(66476007)(66946007)(76116006)(450100002)(52536014)(12101799020)(41300700001)(6506007)(7696005)(26005)(9686003)(55016003)(38100700002)(38070700005)(122000001)(82950400001)(478600001)(82960400001)(71200400001)(8990500004)(86362001)(10290500003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?I5/G9n3AA2iOceDiVGp6Ue2awkO55hUw5RJUoY50bWLif9W8git5f3JuIgsx?= =?us-ascii?Q?BHkFh9nxFcqNBaxDPprUPLfYcuhG6ZUJtmbItR6ITsIIaK4j09ZY26C2HTRj?= =?us-ascii?Q?Kh2ijgsNSuHsEsmLuSF5VX3EbC/ss7JMd4LCzolTf8Xy9z109Sueb6B7Ff1k?= =?us-ascii?Q?ijpb3e4Hx6H+paD5pRX1UKD/z3370Nxu8fj8X3yy1DZWGdPIEG25OoXpLd6U?= =?us-ascii?Q?bNdillJCy9VYlx891yCeWFuu3p1H68xcBX465jlH48rdr8bn1eGjU/pq1h/c?= =?us-ascii?Q?jpm7oYwgRBvYbFuUDmfY6G151jwrXhct3RFOjPHckr94vikwT6uhglbLp5eM?= =?us-ascii?Q?OiNCiWYZbS7hVUtHzPrByer/0r1fre/tc1CX2MgbnCb6MEvGQ0oYUhkf31Kd?= =?us-ascii?Q?Fj/+9PLZ8RMK67UsSe9Ges1mHMbuSriQb9hSMCHLwhrS0xnuznaIdXQ2GWGA?= =?us-ascii?Q?GvrihEviwvOPiOvVa+nQInpQhGHwMZpDFLEjpGq+zwkGmG8LnL2Vb+3kXs4n?= =?us-ascii?Q?mkI1pp7q10zNTlhO76XWNvRf8G2s9ahdbpdXZjPfJxxWo8Weux6vAyvBkSkd?= =?us-ascii?Q?7c/wmkBHY7wMAljTynq0BUiBdZGM5Il1rGeX0DP5x+wPdCYV+Q6rpAmbOHOR?= =?us-ascii?Q?uHSq5J+ZugEii6Rza7H8mP2hm6HwciBVTG2BFxriBZ90Pn1T4axohC/zbCG/?= =?us-ascii?Q?vsPk7pXGLeXHf0aQqvabTubvLCWpaA8Os94Z3Y8E8MzMA+wIlqYQG9q+bDTf?= =?us-ascii?Q?uhqhLjwKk24/Da07BClgbP1gNENbFAQLHjNo2ndHSwNrsk2u90STlB0fWlOL?= =?us-ascii?Q?MG/qK5+xccTwRBWnTjGJWbcReYHkuZBes5QARfgAHXctoVAS1WYXzDZWxdPv?= =?us-ascii?Q?9oKHpi1PQWvXn6C7fZx6IQBen0Ls1yd868Csmnzr+aEldqRCeuI3cLQNNgpC?= =?us-ascii?Q?PaFVCnZX7UWm/NkGa7p0bTSs7NBYA1tSbo0QCZnUojXbE/sOwaFJ3ff9G5+L?= =?us-ascii?Q?dOG4D7pq5f/bPAQ++zHZwxgLoQrbtKg7BwV90wAyBkCnjTZE4NbJHpf1tLwL?= =?us-ascii?Q?7vj7RvKH0xvHZ7r02NCtZiirHvOx8qCbH2Tkmh12yuDrxravS1bTNYWMKsNs?= =?us-ascii?Q?BjZqL24yv5thefvTTHfq2s6ZRfC0OEEOrp4/OelFB/zAvFOPJ9l4A+oQrU2i?= =?us-ascii?Q?EptOcU1J9rrrkywf3/mBpf+PhH4NeDYfSInfQSsHSYuAarJlpouifjQcl61e?= =?us-ascii?Q?TVY6LnFGWk2YPICfI5GSBZoCzommke8kW2/vHC3+Sn9uKWzhtIld8iLbKV8N?= =?us-ascii?Q?Eg/AMMqtJl1o3SEZy5IN43ciYHD751898Nq9nSl4R829IG1TqmdjOMh7/BY3?= =?us-ascii?Q?R3xshKZu+lrUsNI7WUKQButO4AyUUlO+QS0HDx53SJcO0nYlfDspQXgc8GYo?= =?us-ascii?Q?/WNIUcR4faMqQ+1gBx0q4SuZgyTQye9C2RGUkL5sc9/p/29yjx1FnqvYNzuo?= =?us-ascii?Q?fZAs6Kf2E2L1k0glReDgwb/IRgkChh8pWx4Q5PgPthz4jncosg++t9sG5aew?= =?us-ascii?Q?TRJFd+Je6ybZ+vbTYGs=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: f3c5db47-01c9-41fa-5405-08dba798f2df X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Aug 2023 07:32:35.4400 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4yHkZoylDoawJoZRxZqVC+dRmVDZNOIuwoyCBrtkvXO29a39erZY8TEy1VHPU6/JZZQU7tiqm2IqooTj9GJ4cg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEZP153MB0791 X-Spamd-Result: default: False [-9.00 / 15.00]; WHITELIST_SPF_DKIM(-3.00)[microsoft.com:d:+,microsoft.com:s:+]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[microsoft.com:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[microsoft.com,reject]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[microsoft.com:s=selector2]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.215.105:from]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[microsoft.com:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.215.105:from] X-Spamd-Bar: -------- X-Rspamd-Queue-Id: 4RZ2Mb35C3z3CWY Hi, When I was testing a new NIC, I found the single stream scp performance was= almost 8 time slower than Linux on the RX side. Initially I thought it mig= ht be something with the NIC. But when I switched to sending the file on lo= calhost, the numbers stay the same.=20 Here I was sending a 2GB file from sender to receiver using scp. FreeBSD i= s a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. = Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 = vcpus, while Ubuntu VM has 4 vcpu. Sender Receiver throughput Linux FreeBSD 70 MB/s Linux Linux 550 MB/s FreeBSD FreeBSD 70 MB/s FreeBSD Linux 350 MB/s FreeBSD localhost 70 MB/s Linux localhost 550 MB/s From theses test, it seems I can rule out the issue on NIC and its driver. = Looks the FreeBSD kernel network stack is much slower than Linux on single = stream TCP, or there are some problem with scp? I also tried turning on following kernel parameters on FreeBSD kernel. But = it makes no difference, neither do the other tcp cc algorithms such as htcp= and newreno. net.inet.tcp.soreceive_stream=3D"1" net.isr.maxthreads=3D"-1" net.isr.bindthreads=3D"1" net.inet.ip.intr_queue_maxlen=3D2048 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.recvspace=3D419430 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.sendspace=3D209715 kern.ipc.maxsockbuf=3D16777216 Any ideas? Thanks, Wei