From nobody Mon Aug 28 08:28:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ3c70XQ0z4rHhL for ; Mon, 28 Aug 2023 08:28:39 +0000 (UTC) (envelope-from zmey20000@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ3c61y1vz3J0S for ; Mon, 28 Aug 2023 08:28:37 +0000 (UTC) (envelope-from zmey20000@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=c7zLn5eW; spf=pass (mx1.freebsd.org: domain of zmey20000@yahoo.com designates 98.137.65.32 as permitted sender) smtp.mailfrom=zmey20000@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693211315; bh=AbQFcCMxzQSw7cXhRZ2KcF1w17ak4FhfhPKmzpeIoOU=; h=Date:Subject:To:References:From:In-Reply-To:From:Subject:Reply-To; b=c7zLn5eW1OQq6gi5o/gZx1NWQatwctT4Nq+63nhdKf0ILkEE3pxXf4OWT4H1Cpjej+nuc7XYys5q4cpGgSQCS4cLwRiwrYIKjYT6c8ZnZJlHOFtK97LX9j8E/lwwP1+NNTOdMwggOAq2OHpaevPG2SVFo2aJ4p7fzXyVFyMKkGEHdi+H6hq45kaxoKOaJjlnZoQfdWLhjDAknzaASEQ2vUy8UTBsVHnCeYEhuMWX97tvBpoB9ASJhKODG1NKUUaP5M3LdNQhwq6VHixnSFbYEB12mNBdjfK3MTptJG5PrWybE2qLoSyulc5W7m9t2KypPuHk1Kh3Kk2YHS1N24gUGg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693211315; bh=VZinGwKkTLpehSnPxP1tKJdf+Qdp4VAXnUvccx6h/Kf=; h=X-Sonic-MF:Date:Subject:To:From:From:Subject; b=ZqkE2hgvae2dhdHHsSjWyjKF2khucKYOz11anpAPRf/zP3ZON6GxC31RI+C+kfFS/ZRtr9NAU2gSvFHuqcejBI9hmJ5/HxFk7i9LlncSLO0PWEoqE4gpsC1BNO9MoDsNeYKHSZb/4GyE5qYsNgnPGMdgEbqpXU3cvnvND958cmNsFfrNHIrEDASlljGxqJ/E//0V0y8idPO/kERAiIEv07jB7RErrhGk1D5epQSIhi+Q4HG5dR/yxt7boM2Icwdhd5nsi3+z47GRzuSixR5eB4zttaz3OPjg0gHAv+GbA2wbv63Aj+5WSe2pPjJLlWyP7UfkQQj0rO4PTC2quRfY1w== X-YMail-OSG: 0gXzmzEVM1kJ3rIyehMLwdDNbI.hEdnLHfOJmUf6LX0SGdZbI3VfBSQrh55pw23 fzWxGClsSxvBUvFlY0g5J.4SOIuYU2hKHA7e0CgOA2SXfPyboPJBtqY.3WlCUSIc7eeR3uIRnTRh VFL._2I8IMnc1P1BwB6d6P4jk.6rcJ3QJB9o1ziQnUMQUg8pM.jgm_S9FXKngDShrUfSR2ChVTNm fe_Tw_rX.k5kENxjNUSRv4ClJBUzVhxnIBKBleAei8HkWbxGzgGF7elZJSkf9XWs8lAoVMxKehkJ E9bS9VUbgAooBVCg5OWeoY_pzSJSJkEiLkzw7J31YRlU9czhUn0O7k3qBT9xIxi1QD04gKukGVDc HA7NTDWCDHfMk5hFVMEn2u8ksMWtcDhq5xuypP2fdG3hdpEmjvU_19DNhPL6kYzWOjNV1FrH.CHx MGlrjBk4AyywO47B5RzBkrvRoxlIGdS9noRJyOy37hyJBEBDAjBlyW91zWA3zwLzK9d5fpvGoWZE uYRNLaFZ7A22w07UUeExMolTO7XcsKD_KgOPidn5hRcAI9_1M3pk8UL77QJAiBl0Z4nCZDtpXdtL lXhEG2AYgUsRmkwS.dSgb5Qu7EwLSRzTFEy1cwI8BfK0jTbyubKf2Lsh7Pn8sS_3R554hnjuH6BT a7HbJJKBsQY0QXeXPO9fUp6425vqTr2k24sEHm9ik8dZkBtcMdPS4Z5705AUWbbMKAKTUCNnvZOb LbEsb0yQqFSOtZHFf08iia72PVVvxAVEbGbPFPIzK6ceCgwnAu3gJcLzHKFUcoZ_4REP8402UNrB fsqwxsDJT6kXSGk4X4CNeWCnsLhSyfKyHz9TCeJYQRXnIoFpWjAr3NBlJp2Ew9ywhdw5W_nHmv31 9tgahS0cqgzBhSxAWJ.AEWWG_INXbt6Yw7qeliHxVd7PhhN9pqkLfPSpNUpywxrSRR7aCACoutVa 0zn5d_77VOv0BMLueP.Kw6Zry5uQ8omz86sjGlvo55uNWRDlLXAS4.9_cvyjiutMNJjiN4qCCm96 5Tua8JIG2qLKSIQuJWwDLJsU2ABOuD97835JyRz9WeR7N1u8eC.U5rovn0eksZIscZMrXVoyat1c cT3CYveOrpFBp6NX3.yXnlFQiWuyP0gvwGI0lXXnrBt0zvkTMSm.lUVvRSj06uIn0CUsOVQz2yX3 ZygS_0Bq56XPGuhBJnHWyLNKtzfr9bMsMkvq13mi0YSGjjhFW62YHxqu77dv2qR1NtAb8J4dkngw 02X68Oe6UpzaGfHliuVwId9gVoyqMeAmyv4Z9sq7bNOCxaMcZcEH8jj60MGRmR3Z8JFB9cSGA4m3 yWshniwk_zgohaj2fgqYc.xXe.lZNzcAuFxHkQo_L0BXS58OIwYKHoME24F.Vq7jtw75RTeMGSbm HHLL7JAYtScfaPsSNeMQNui6UUllkvSE3x1r4Ml5P_sxkavHK27CAJq8WFTSOO495m5VWOLf6LXl RcFYu.lny1iTRgCqsOO_ulZbgseBj.brrucjoo7ojDq_qgzBlhWlY8YCcnRMRg_rQdgP5YZevfe2 05vrEQUCahTDqgPydBI6iISPidD5if3yYNWxScpc1_3WDq3Pxpmxqx0ymDJI0hgi3.D838wYEcP6 AILFsHZP2Ei2cUMaRFMssNCUAmmNVDwSVh3ZRR09cFL27sR.pJgQvvCwlW1HgScN0JYYmXSlCRoW SdMcZGYqIilHkS9VQK27HWlkDp7SNlfJmvddmS7v69vtEDl6OuqWC3O1pFf2jw_9IRTvsyqdS_Bz vMfFThoEfWO5cu4gSZJvGvYiyLvtzHbcBbCwpoC0b8xrNGXz92kUPrip0OiKFZJIB1HE121_I95w ut01pXkp64DoHenwdhEsPDzzHxVxkVhQVIHK4..aUbO5DuhXJ_MCDgDnapnYoXYynnBT13g2.nNw U9HgLXhu5K.Tbx6N0rldS5UmCzm8.l0d7icYoZmYD02EY36bDvsl4j7n6UDpK.rwUtBvf34HGzde 5DM6ZVx_8X7KwMLs09CIhpuP6g5GGJkPYwiR1QNc73jMqccjGStgzPEOa3gKmiYVa0U8n6YRRhdy J0uxi2hQUjr1tLIIMmHh64eLXp8PGU0Rj7T1ILwS52O3UBBSvfEiuWws6aSGriTd_IOv5HKm7Fg1 Lhm5t7jgegHeqgLyVZjYhSC0mHNEEFsn1GKVCqaPoUttpH0vXBf_s_3nOGPM8jVEqyDfow95Nv2T 6WaGS7fPewxf4Bw-- X-Sonic-MF: X-Sonic-ID: ba7ebb67-5ef3-4d5f-a3b3-37e3e9de0e41 Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Aug 2023 08:28:35 +0000 Received: by hermes--production-ir2-5cc57b9c45-mv62g (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b233e9053ea0e299500d7dbedd0a9223; Mon, 28 Aug 2023 08:28:29 +0000 (UTC) Message-ID: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> Date: Mon, 28 Aug 2023 10:28:28 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: Very slow scp performance comparing to Linux To: freebsd-hackers@freebsd.org References: Content-Language: en-US From: Mikhail Zakharov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailer: WebService/1.1.21763 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.32:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.32:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RZ3c61y1vz3J0S Hi, What is the drive back-end of your machines? I mean, what is seem to be fast copied could still be cached in filesystem/storage buffers but not actually written on the disk. Best, Mike On 8/28/2023 9:32 AM, Wei Hu wrote: > Hi, > > When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same. > > Here I was sending a 2GB file from sender to receiver using scp. FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu. > > Sender Receiver throughput > Linux FreeBSD 70 MB/s > Linux Linux 550 MB/s > FreeBSD FreeBSD 70 MB/s > FreeBSD Linux 350 MB/s > FreeBSD localhost 70 MB/s > Linux localhost 550 MB/s > > From theses test, it seems I can rule out the issue on NIC and its driver. Looks the FreeBSD kernel network stack is much slower than Linux on single stream TCP, or there are some problem with scp? > > I also tried turning on following kernel parameters on FreeBSD kernel. But it makes no difference, neither do the other tcp cc algorithms such as htcp and newreno. > > net.inet.tcp.soreceive_stream="1" > net.isr.maxthreads="-1" > net.isr.bindthreads="1" > > net.inet.ip.intr_queue_maxlen=2048 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvspace=419430 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.sendspace=209715 > kern.ipc.maxsockbuf=16777216 > > Any ideas? > > Thanks, > Wei >