From nobody Thu Aug 31 02:17:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RblFN5fsBz4rYPR for ; Thu, 31 Aug 2023 02:18:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RblFN2q6fz3M8g for ; Thu, 31 Aug 2023 02:18:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693448294; bh=FxthQSbbNo8rXjtf9Au29rmhT2jrdRtt4LmJWmm6HYk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=HAIJZqvhQtXKWJlNhMBNjnyq3ENz4OP2MQatq2jg2EBaL1ZLD8kpqUdxrSFFLJZQwMR9/JzuODFMM20jtz54XSi+prPR3ZXFLdle9TCqL0++lUwpS1RA4CBXEg1rrrTwrcasR8idV25G9RY/wKAI9wNCB93TII8V45kzv0OjW6Szp/IStZMnvC3aG2NeAi91PA1otxuo1x++JqJpViPFIv31n0Mgs5NvMyRqeI6VIyspsus3lJSYlzaZxRcyXdmgM4Aro+3rKFJ/FMU8OS6d4UN626i/9mbftVS5nMk7oJAYEvIO15q92D10LV7RbxdN7s2dLU8GKkBEBpenrLr+7w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693448294; bh=XwrErb+EQGS0QTvkAobT4aO6HUPyzmNaNVky8X8CCj/=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=BfYC+PqZPcWOM21N+If7xBBJGxe3d3tphFsloHwtVK9T+ND8jgw90PAdE4vp7LIvoUBrhnjl1V1aWpg8BtFTDZwEqUB38ujtGsnM9XUCPkSZtuqC9Jsx/71vqKD6Z8Zm228AejcT5YvWVkHIs+CtNy7JSRaCmAz9nGKH/s67JyI5ww3pRwjOWbafSw2P2d2NjAPJ/ZcPNH82eHSDz70OWj68GdoC2x0wNjaZaebslOSNmUFzoJvF5iyZ7dgdkKapyj+ibOyCyzJEnzF06LUjrqQwBVOhu6Iimdu5J9Nq1eS2i6Rl2U6PaeVzPbK30jzxzOjDJqep9nrdoXLohS+oBg== X-YMail-OSG: 0BsUKrcVM1mWM9Q3Z5pXcbAUj2Ia6Hb5s0.NF6K_y_N6iU.T0exUFo5JGmh5jgO W_8.m6YJA_Ic9VjDIAAo6NO3jjg4ZYJwEW_diONgkXFkLKQkKOHBhR3xMLutDoXAmzTrdEdY.iGa Z917kHp9PB200DP9rkgCX3sloF04EDe5Ock7vpSM0LyMp8loBTf0PJ9IDNAEbiYoESrsmjP9lAnW JjNeXj2gUmQYof9FgX06whyeoqf6Q4BYWdRwvyotOyvGJ8xMqiMJTNJQ3Z4gIyrRDqtHxkV8A0zz yUPF7i6R.gKMIYCizyruU2tByFoL628oRNpExY9oVYucrKusQeBUNv5mXmAiekpZtBXxSQWGNN8a Fb_x5I9b1uf6uNwm5dy1rS3mblT1giWAXGUF0P14Mcoc0M8KSsvL2PlArGb9PHjsxj.Aa7lqFTw8 bOFgzLe81jjHGAVBuMHx9KGRWsdjEWXC_ZSL5VqVjRhvSHYc8lysxPkPaGqHEFalcj58B7dXFtek UknsXGjxHFg_JGM5TsgyrQ.E9.vjUm9I5cQLFTUBiJuD9X_svW8aswhdmVjQlvIOu.ezbA2G_op. DXXCkVAqYbfhc6jiFyJw5CKfJI_HbQUW2xVIqU91kjb2LjsjFRptkbOXAPUlxfLJ4YThzbTvFDbM rlRraK_.W_q2Oe2tlpezH9_ssh2Ty2p3HS8518m71mejIeT_ZwhTEdbN1Dr63SzDlUghE77QWkWA Se_QO9jd4Oi9y0Wzl_y583Pu0_iKVzuRVRHrW9OpGtWZV5C7UUyaMqzQ8WztYB70GFzNnQOVOiD8 hZvTnzoov8ZqRh0XmGmuDYOpmlyqDUMrNdgUdtrtfpVV_Uw7pBFWTrNRFUy7URkNAkiIQPeE2Snb ZS99J.qVWh5DUobmCh4NyC8c9FaRirnNezbAATtW_RCSCNwNTYJiiLco6SGyY1j72rAWOIhsgjS7 EIRLSm6I.chCZkGHVWOKFltuN51YQKgyqWaRuo00CV26z5iHUJomOJhw_Lti96N_VmlUt8F_T_41 QsPbw9MTLcxKSJwkuecwoMkOegZr94B7DZnXPEQoImCfFB.dN_1wReKCV_RxAmGvYf3ESJG1xqFj DNkJFb0NXqf4tI5dD0YDZrOD.hBlWE5eZG9OgAiJvptMeLbFez3D.acWU1abxmOPnjWu67Y0Ib2f UFKE_rDVo5Pfp12fje7fUDNRJ77TZDM15m0HI3uRPH.EABZoPisw9_FaYFwuIKO8i3J36gJGFFhq 46k7x7OUdqamkUREbeB1tdxrMawC2ZMW0K25gmh3NC5.OUKlEFwwH49xSqytU01QK37UEfYX467. f6r7oYFPjgLnC2fFvfGtGsVnI51Q3F3Af0utuQYM.D_F2Xi9Z9nJO7NplvqND76MLhf7WhePmD2N iIIFxGfQOQ7FZpmmFPSR7WCcTTXlVN4.6T0BKJd6_1ZjtXf53t2.4Knyt6DCzUuHFszi5sxmw3yJ 7QmnLU2gzUgspp56.I0EErWDHQj1etpVqeghy9PZLK7DuZpWYkoC38vxAitNDleIqGEoP6tO0ERX 3uy3vN_igEg6B7yD8vGD8e9AanBZctXmXZwK9ifv30Aygstx34Ns2ux6FSb8K6jZu7rhj2uR0oCH zUDfxDwDtOObnlofXS9.3aNqrzXA1TQ.SUMvB3YN9nQN0BArnLpx4UxTU4TdDtuX4Prfv6gEANlk PYjKyfQi1AFU22tR5C_YNzAae2NYQGBdcGWvSdxRQUh8MpZ74p09lOe9CNr3Ahf_f6aifwb1Z0RZ QxQfOgWIpukphht8mUFhOVQVUI1ZyyMmJykgtt4PKv5s3Pukm1VxT3ZsbndE4ZFROdknid1QOEN7 F70z4yPYFdmF.kTmS4VYkgngMe0Z45S5t.aD5cVf0PFQKCm3w18EUVyZ_UL8EnWRzD8UZgP0t0ld uLHVOPppuWKSwR1Wri.Hm8hQV6iL_O4AXI51mB2hQVETli0ejMUbDw5w8ermCA7igAmcTMc9d6Lt .Oe9hd_OM4E3E_qnw35o498Wt6vLZNnwEceLJlRyjoRPwhp2ZxxUQcINxtW1ID8n8Za9Lx6L7QXc hZiZbfPZkpDBjXdzb3SzfOtFJ5cdGs0C1LuhpLo3sA56zyOSaoYI9WtPQ26fvpd3AR3FOHwAqjby CGSwPxjlXUXrTb6z9QOOlTlkDNgHFstwNIe0H_Ro_aUfugTn8b0DnhquPngUhOq9XvFB9c07EhB4 SpADOW6vmzrl6Anz4J2s1ArFDsOEEblbnNI9Xq85kU_nqVXFQhwkH3wBWkEBk5XFnfKzxWjD3x_Q l X-Sonic-MF: X-Sonic-ID: 275bbaaa-2790-4b63-883e-04de88491482 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 31 Aug 2023 02:18:14 +0000 Received: by hermes--production-bf1-865889d799-k5x9p (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 0854b6417402866361f1300c9d96ae17; Thu, 31 Aug 2023 02:18:10 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] From: Mark Millard In-Reply-To: <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> Date: Wed, 30 Aug 2023 19:17:58 -0700 Cc: weh@microsoft.com, FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> To: Mark Saad X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4RblFN2q6fz3M8g On Aug 30, 2023, at 18:45, Mark Saad wrote: > All > Why not take scp out of the picture and try iperf? Why , we could be = looking at rss by default in Linux . The explorations with ???@localhost:/dev/null and with dd suggest non-networking issues are a significant contributor to the data rate differences being observed on FreeBSD vs. Ubuntu 22.04.3 --including when no networking is involved at all. I also did experiments with cipher selections that I've not reported. As I've no clue why the original note was about specifically about scp performance, I've just been try to gather data that might be of some use, even for other contexts than just scp used over an actual network. Also having iperf figures for just the network contribution would be useful too --if the network had appropriate characteristics for comparison to the original context. The network that I'm using is limited to 1 Gbit/s and may not be a good match for comparison to the original context. I've just not gone that direction so far. > --- > Mark Saad | nonesuch@longcount.org >=20 >> On Aug 30, 2023, at 8:10 PM, Mark Millard wrote: >>=20 >> =EF=BB=BFOn Aug 30, 2023, at 01:49, Mark Millard = wrote: >>=20 >>>> On Aug 30, 2023, at 01:22, Mark Millard wrote: >>>>=20 >>>>> On Aug 30, 2023, at 01:17, Mark Millard wrote: >>>>=20 >>>>> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>>>>=20 >>>>>> Wei Hu wrote on >>>>>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>>>>=20 >>>>>>> Thanks for the update. Seems the numbers are the same on zfs and = ufs. That's=20 >>>>>>> good to know.=20 >>>>>>>=20 >>>>>>> Yes, your numbers on ARM64 are better than mine on Intel. = However, my original >>>>>>> intention was to find out why scp on Linux is performing much = better than FreeBSD >>>>>>> under the same hardware env.=20 >>>>>>>=20 >>>>>>> Is it possible to try Linux in your ARM64 setting? I am using = Ubuntu 22.04 on ext4=20 >>>>>>> file system. >>>>>>=20 >>>>>>=20 >>>>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit = 2023 >>>>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But = the >>>>>> Hyper-V UEFI reports for the disk created: >>>>>>=20 >>>>>> 1. SCSI Disk 0,0 >>>>>> The boot loader did not load an operating system. >>>>>>=20 >>>>>> (It then reports the network adapter attempt found no >>>>>> boot image, but that is expected.) >>>>>>=20 >>>>>> That leaves me wondering if Hyper-V Quick Create >>>>>> established a VM file holding Intel/AMD material >>>>>> despite the aarch64 context. >>>>>>=20 >>>>>> Establishing a Ubuntu more directly is not familiar and >>>>>> will have to be a background activity and, so, likely >>>>>> will not be timely. If I did any experiments outside >>>>>> Hyper-V (native booting), they would be with slower >>>>>> USB3 SSD media than I use for FreeBSD. >>>>>>=20 >>>>>> I did notice that Hyper-V Quick Create did not create >>>>>> a fixed sized disk but a dynamic sized one. That is >>>>>> different than what I did for FreeBSD. >>>>>>=20 >>>>>> Also, it was not obvious if you were after aarch64 >>>>>> Hyper-V testing vs. native-boot testing vs. both. So >>>>>> I may have gone the wrong direction from the start. >>>>>> It is possible that I'd find establishing a native-boot >>>>>> easier and then be able to have a VM file created from >>>>>> the media, more like what I did with FreeBSD. >>>>>>=20 >>>>>> The Ubuntu activity likely would not be analogous to >>>>>> the FreeBSD builds having -mcpu=3D optimization used. >>>>>>=20 >>>>>> Back to $work. >>>>>>=20 >>>>>=20 >>>>> I found a sequence of UI operations that worked for >>>>> installing Ubuntu server 22.04.3 into Hyper-V in >>>>> Windows 11 Pro on the Windows Dev Kit 2023 via >>>>> use of a downloaded *.iso . >>>>>=20 >>>>> The kernel that results predates 6.0: >>>>>=20 >>>>> $ uname -ap >>>>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 = 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux >>>>>=20 >>>>> Using my usual rule of rebooting before the first scp: >>>>>=20 >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 >>>>>=20 >>>>> $ rm FreeBSD-14-TEST.img >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 >>>>>=20 >>>>> Definitely faster than the FreeBSD results that I reported >>>>> earlier, including faster than the ThreadRipper 1950X with >>>>> Optane in a PCIe slot (more like 300 MiBytes/sec). >>>>>=20 >>>>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual = hard >>>>> disk under Hyper-V. >>>>>=20 >>>>> For reference: >>>>>=20 >>>>> $ lsblk -f >>>>> NAME FSTYPE FSVER LABEL UUID = FSAVAIL FSUSE% MOUNTPOINTS >>>>> loop0 squashfs 4.0 = 0 100% /snap/core20/1977 >>>>> loop1 squashfs 4.0 = 0 100% /snap/lxd/24326 >>>>> loop2 squashfs 4.0 = 0 100% /snap/snapd/19459 >>>>> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >>>>> =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / >>>>>=20 >>>>>=20 >>>>>=20 >>>>> An experiment would be to have a small amount if RAM relative >>>>> the file size. That would force it to actually write to media >>>>> for some part of the file copy. >>>>=20 >>>> The wording was poor: "force it" here is just from the >>>> Ubuntu viewpoint. I make no claim to know if Hyper-V >>>> is actually writing the material out to media at the >>>> time vs. later. >>>>=20 >>>>> So using 1024 MiByte of RAM assigned in Hyper-V: >>>>>=20 >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 >>>>>=20 >>>>> $ rm FreeBSD-14-TEST.img >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 >>>>>=20 >>>>> Still definitely faster than the FreeBSD results that I >>>>> reported earlier, including faster than the ThreadRipper >>>>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). >>>=20 >>> One more variation in ubuntu under Hyper-V, still with 1024 MiBytes >>> of assigned RAM: use of localhost:/dev/null >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 >>>=20 >>>=20 >>> The matching FreeBSD examples with 24576 MiBytes of RAM assigned = (ZFS context): >>>=20 >>> # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>>=20 >>> # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 >>>=20 >>>=20 >>> Note: At most one VM running at a time, never both in overlapping = times. >>=20 >> Avoiding having a cipher involved and even localhost >> involved: use dd . . . >>=20 >>=20 >> FreeBSD examples for Windows Dev Kit 2023 Hyper-V context, >> 24576 MiByts of RAM assigned): >>=20 >> # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1m status=3Dprogress >> 2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s >> 5120+0 records in >> 5120+0 records out >> 5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec) >> CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1k status=3Dprogress >> 5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s >> 5242880+0 records in >> 5242880+0 records out >> 5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec) >> CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress >> 5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s >> 10485760+0 records in >> 10485760+0 records out >> 5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec) >>=20 >>=20 >> Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context, >> only 1024 MiBytes of RAM assigned: >>=20 >> $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1M status=3Dprogress >> 4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s >> 5120+0 records in >> 5120+0 records out >> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s >> $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1K status=3Dprogress >> 4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s >> 5242880+0 records in >> 5242880+0 records out >> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s >> markmi@ubwdk23s:~$ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress >> 4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s >> 10485760+0 records in >> 10485760+0 records out >> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com