From nobody Tue Aug 29 11:22:26 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZlQW6WSqz4rhH1 for ; Tue, 29 Aug 2023 11:22:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZlQV50gNz4mSj for ; Tue, 29 Aug 2023 11:22:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=tIV2DMdQ; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693308160; bh=LCvqO5u/ULi/vmytuXwB0BMMVBkavW2kRHUkmRdsS0g=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=tIV2DMdQvPLdL4iqoy4SDnKLbYD07uTI57SbcloYEM5lemE0TjamcvjVp4MSPsC9ATjFiwgpYRcxNRg30kqDZrnKTulQGj9/4WMTAl0STALvZxJUKrJ0vXYPwZgUst9dW+6SF/3uwmJsRnNY4cu4cnP4cdx5VEUe1qgye0cZrrfD6iTuDd+m21y/Qmp147kDywmjCeuKA9iVRw9Wn4YUSiEjqaPmbmZRmfE2dxsxROCWH6r0Bwojdd5bsbAiCcHqVv+e2QPe2CMHHdCKaXnYNW4xZ2/dj3d446iD+4Ws8Rutjhv28G1VJM8hUkppYsjGONWJSq/X7cTweuhuIOkxHw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693308160; bh=RYeSsZzikPg/r5zu1d7qgQm4kAqYq5Vdy2KFxCAEXOl=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=IEkr/K7agbmv8ai3pEIJ8jSjjGVhj7QLk/mC44Y7exenfQeJNw8C5I9AxU4r2VpcafqUhgkB7/nRFkNacYhtCklDeGlxT3+RXjqWxysf/76unn3RuHLyIEje46WbAVAlKw5v5cfCoMsRtL/Hz9VE1jKA6MK2NbPAEFiZ1Xfm/8NMG9XWBU1vzjFr6lreWgm0SYYyfG7CGzGkdDE/Ouyj7zZq9SX6Bsp7lx8Ez/c81+02NZ/v7fIZW6e1vVjsrxVAUeuWa0uOEVtedN71Lo/ad05DtJVAl/Jn3DBstnlLKE+d1LNGuMenw//et0XES+8HrxLgMjRA5UIsETMNBbai2w== X-YMail-OSG: r0uPXokVM1m2Fpo5RB86BiG3FsWjEZOLy3Gl0MjrBrSpq7QOcrLQFZ2xtX9Y.OY HDJtL0p7quJwFQC0hIHuLvVBrkdU7slote9bLA4RplttkD.uhBZTP1XLEp0Qp.1nVr8QFqNRMFXJ uv0ZsIGxVTu6rLluFOhLu_QvAadrZnR3F2Rh9EhfUfPv2bwAn8UrqehxIkXQI7J2MGGEwARjz9Lk P.EgqMBsmHZ62bbelvCHDhDMv6TIZ_jnxnL72XbXmF3arzzbiBgyh5DPWmdwhkZJvTkxoMAvpfw5 z_8IdnDuR2tQ6ndRBmjr.EoJoiZVsx9meWdAz2KxBExBvC84PkHzC2vUyYkV0SbRBr2TroLBWtKE TkcT3oN5ULQvFPpg3M9G4VDrJ9e5JzrEfYM4FsI9z7nDq3hiL7jWvJLx.K0klinT24Btu1WgSABX CUnTPr3o1DiNu9FuuQsN2o9uwqnuOijOIv8JX816G1H5BAzQ8QEINXEXY6xZORUK4SStyQjB9SkF Hn3_M5scEYg4pQVOhTLeQ5aiPgXahsNdkqmYAox8NqFN61k0GTwl6mBpvZ0lqVkZ6Ut7fDlH6Vde 6LeRUsmBfvVtPDWouSRBNZlkqU6d0QhFDUpP3MsXakb6wCLPhU8Ic9zpoeas6VA4AEqmFiwVRE96 KRUzFWSJw0j9KShFVqnR43g.XRvm.LyIlg2MBnfITvEstazebhh9HXfpk43MHEvFKnbsOdRYN70f Lpo7xM6z9lktOu1JOim3KveQZJHloE5tg.3DFGNS39xbNUaSRU9PUSgRRuPzkoUs4B7mnCtn9g1I 2SiivmThTIsEZQ0QGydYpZ0FHuEnOF10gWmk6VddQPnrOyQTzzhekQUMUCmWhwHL541BPvkuXX7G jk2NY6JUwy4LLJ_c0Qf_ydZ4VmNib__V4AnYjJFaoa574Oq4qplrSWegb.P5A8GWgCz28f0cO9Lb lex76D8n6mP7xNUlKAmfw1w3KrnQOvSkXPU6HE_2lkgtfAGXbcwGsFzGB7aD5XV6duCma0tSXrG1 AE0EgMV3zrnWOG2BSOd7glw7MynSzkzMS93viNwcUcGicoIbxa6P8SsMCYau1qip5ZdmG81af8V. tyd0rUPxVTD85FCVm_hjssxYzyZCdwV7Ey1hNmZ0mHGcqG0wn6NUlSo2FpvczMXOdp7Np_6CuZFs omXmtjkBQhRoHNCEeoRCOcVil2fgIy_cSNdK.TIXyFz.KQcYWkixO8g7PXxCRWyzq8.ApGSay84e W9KwCfeuMbYLgwAy0igD8pJjiMFXO4t0xOkkGczH6sSoKaiuK3V2pKV6sWOqr0wTNxmBWD6G9I3Y 2h7aeVr645zFajxor_mGVsjoc6YB9jzdIOsQ1z.Qw_Kqs6lEXNJyA9YEuvWAuYUNTXyibOneq3jJ k7ral_trLAjBlm08sGLPDLIyvqTLxOpzdJ_MgV4OzAQ3diUwKw1lYaP1OaefEphY7Lr4oI045KR. 8DgbtQbDm7v6oaMPyt57JuaXRbiPQKwcJhlHn1kgSvnCYMNZvjmZOTPrQZfmkD9Jf4m0UFEOvP2A Qf_RH9fcyfSR5i3kaKfUqwEnNQkzj4re6w88NkC.L6EZqB5VrjXGPoIciEUvyVMjFqxxBQMIPbsR VbHarW9WX7koiJxgxeoQNg.ZDEk4PMVLzmNmBCiUcTItkEqXfwv5OpO3PXWTQut8cBvCXmxujVpF pqB4D_4OMTybhcT9Xe20jqoVkioTXHKyTwQ6vgivL3dglq3TlZbDyqDRShVrPPc7DY.l4SllMYVq bR9zna43OSfZKCiLucrwCGvlS3bzCrQ28yctbHHDqgUiIzc5GzTxJM21DF.09m6QycQo22RGzWPR VDi.DW7m0q49sVZcpIMcfFW1jmH_fRcshVwElMH8oPl7hOL4LV5NZzOC1dHZxnTR3rEvcu_Nh7oi Jqxjkw14S9aRid.wdL74RbUL6ldEvWME2Y.K0uko5cMy9ULd1i_R0rsyMZTxys4uhwqDlUWMbgPM 1vCCV2GcDsk8RMuisxhmEOmaC7vlR6B5HnPz3jcckwsLocJCEuzsnjYBk_NZyR2zd8sUchZ.7PDc BOLDL449dYtIGHNqfqKne2o.BmSH.2_wRD_0rwnES7t8DC53Bl6HAjrKLP_XPMh5A7Z2vYzEsC5q CAahDNCl3DpKUTz69BTmnji3RYrLeponiHYVjUa6A_unINFxRgsUKyM.tUTYC5nRJFPJD9Tw0xmC wNNVupLCVCQnMUBaxPAJOugXz1DnTi1FwePDtBLKNG6bU.HuGyddGdgcrCJI61EqthIaC9EvSLhI oN52Mpg-- X-Sonic-MF: X-Sonic-ID: ff211f11-9159-485f-ba8d-fa2f2dd35ec5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 29 Aug 2023 11:22:40 +0000 Received: by hermes--production-gq1-6b7c87dcf5-sv5pn (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7e5d4e90102019137dffc6bf2e91f0ba; Tue, 29 Aug 2023 11:22:37 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux From: Mark Millard In-Reply-To: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> Date: Tue, 29 Aug 2023 04:22:26 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <6E433834-E192-44F1-9FF1-3814F13449FF@yahoo.com> References: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> To: weh@microsoft.com X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.30:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.30:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZlQV50gNz4mSj [Adding USB3/U.2 Optane UFS Windows Dev Kit 2023 scp examples, no VM's involved.] On Aug 29, 2023, at 03:27, Mark Millard wrote: > Wei Hu wrote on > Date: Tue, 29 Aug 2023 07:07:39 UTC : >=20 >> Sorry for the top posting. But I don't want to make it look too = messy. Here is the >> Information that I have missed in my original email. >>=20 >> All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz = K8-class CPU). >>=20 >> FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: >> 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 = nodbg-n264692-59e706ffee52-dirty... = /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 >>=20 >> Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: >> 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 = x86_64 x86_64 x86_64 GNU/Linux >>=20 >> I did a couple more tests as suggested by others in this thread. In = recap: >>=20 >> Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s >> Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 = MB/s >>=20 >> Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs = 48.8 Gb/s >>=20 >> Would these numbers suggest that >> 1. ext4 caches a lot more than ufs? >> 2. there is a tcp performance gap in the network stack between = FreeBSD and Ubuntu? >>=20 >> Would you also try run scp on ufs on your bare metal arm host? I am = curious to now how different between ufs and zfs. >=20 >=20 > For this round I'm rebooting between the unxz and the 1st scp. > So I'll also have zfs results again. I'll also do a 2nd scp > (no reboot) to see if it gets notably different results. >=20 > . . . >=20 > Well, I just got FreeBSD main [so: 15] running under > HyperV on the Windows Dev Kit 2023. So reporting for > there first. This was via an ssh session. The context > is ZFS. The VM file size is fixed, as is the RAM size. > 6 cores (of 8) and 24576 MiBytes assigned (of 32 > GiBytes) to the one FreeBSD instance. The VM file is > on the internal NVMe drive in the Windows 11 Pro file > system in the default place. >=20 > (I was having it copy the hardrive media to the VM file > when I started this process. Modern HyperV no longer > seems to support direct use of USB3 physical media. I > first had to produce a copy of the material on smaller > media so that a fixed VM file size from a copy to > create the VM file would fit in the NVMe's free space.) >=20 > # uname -apKU > FreeBSD CA78C-WDK23s-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 >=20 > (The ZFS content is a copy of the USB3 interfaced > ZFS Optane media's content previously reported on. > So the installed system was built with -mcpu=3D based > optimization, as noted before.) >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 193.6MB/s 00:26 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.0MB/s 00:25 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > For reference: >=20 > # gpart show -pl > =3D> 40 468862055 da0 GPT (224G) > 40 32728 - free - (16M) > 32768 102400 da0p1 wdk23sCA78Cefi (50M) > 135168 421703680 da0p2 wdk23sCA78Czfs (201G) > 421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G) > 468860928 1167 - free - (584K) >=20 > # zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT > zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x = ONLINE - >=20 > (UFS would have notably more allocated and less free > for the same size partition.) >=20 >=20 >=20 > The below is be based on the HoneyComb (16 cortex-a72's) > since I've got the HyperV context going on the Windows > Dev Kit 2023 at the moment. >=20 >=20 > UFS first: >=20 > # uname -apKU > FreeBSD HC-CA72-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 129.7MB/s 00:39 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 130.9MB/s 00:39 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > Note: This is via a U.2 Optane 960 GB media and an M.2 adapter > instead of being via a PCIe Optane 960 GB media in the PCIe > slot. >=20 >=20 > ZFS second: >=20 > # uname -apKU > FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT = 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 121.1MB/s 00:42 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > (root@localhost) Password for root@CA72-16Gp-ZFS: > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 124.6MB/s 00:41 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > Note: This is via a PCIe Optane 960 GB media in the > PCIe slot. >=20 >=20 > UFS was slightly faster then ZFS for the HoneyComb > context but there is the M.2 vs. PCIe difference > as well. >=20 # uname -apKU FreeBSD CA78C-WDK23-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 Again, a -mcpu=3D optimized build context for the FreeBSD in operation. (Still rebooting first. Then . . .) # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 199.3MB/s 00:25 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 204.9MB/s 00:24 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. The Windows Dev Kit 2023 figures are generally faster than the HoneyComb figures. =3D=3D=3D Mark Millard marklmi at yahoo.com