From nobody Mon May 01 01:57:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8mYk6lJXz48bjp for ; Mon, 1 May 2023 01:57:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8mYk2HKhz3Gmm for ; Mon, 1 May 2023 01:57:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682906249; bh=/cmO4ZCOIvGXCD1Fw2ps4OdrhftESgdb7dAOMUjlo6c=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=p8NJJ8Qn6sildfLBF4+3ZcGc7HY+g1R2EgcciFY+xC+uGLLVOR0KfhdYv0e75xkoxXEKJ25cGxZ47PivLzSWb6FElctekK37qMya1kdmfufaWMoW4O8UfDjolQTl9s3DOyBGbf0hZMU/QC9Y2toxqY6/7OVQUmyC+srPgH8Y1YPzprHL3GUG4zOgmb7by9hdrNzb4wBa6xfMH/KExrhWMXX67NHLrN+WjD0iNXacMPHqxRkJh8SO09Q+sHGZe8XDFBrSjrpGEwiWNvFmvxWULMYkftUdd73zKhmxnxxBatsGSEpGNdq6DjtcISIoZ9HyHb+rspkkPTszCFISQWz7Iw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1682906249; bh=IcuLuPRM6Qx6D0iXZoRNleRklEyw1cQ7kmQHLDZt0mv=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=tStQzEkmf4UUpaAbB6aZRXBLMImmFHFcUdGVMzsJIdlE3fPLF5jGcNNSaRQo3HNr0UykCBbVVE+yP7rpl01ORIGiiM6B/5+WdgTC3nnsJENATkb7Kqr+Mw8mQuGslXHUMTW6qQJF4T1fQIOqVZ/lDf9m/KjQPKxc6s9Xu8mYfVOIZN2YjKHdU3rPq9sJ4fnwyCgfKtxKEhKp4ntvhJGPvuGxIwmleBV1dV/JQDkUIu0f0y9N0eUVbdK54hFwXd/3qGpQf5cbT2eUDHgruXAr0j7x7p+uS7knVQ/Q7/JzxzDBCgWiOjKnH3p3VyxyqX/X2mRxiqBNTkX4JBJMCJjA5A== X-YMail-OSG: .MpGC0EVM1lsBfC768LJ4BBmnh5_Q0CBQQSdx9k.E4NFGr285ZVBbl56PbCpy49 T09EvIc4JuojdTq6Gv.zUKIEsEpJD6a4kxYxSlVS8anIxGND3OnOdQkbgN092wWxYhyhZquzxxS0 M1US_zWdnwlBfQXAPPuLziXo.ykFClsNdVEG5347ul9f.XoJejsC0.KWtWvtUIVqc1XZDPmXyZ8m 9PHDHuftbjg5zDfinBzSSAhpeCibcvc4ehd24ePmwV1HjSeW1cD7b2aQs0ORFf1xUlr5613YsTr8 JfZnCx_1Wiszq5h1tVbu_Lc1.JdBE9mI7L6Lu57c37a4_n3RHQrW4Vzp_04TK2KbXOT8tgZ7.46R dfUIF8ZZjQexmHpt417su9r.U0bgraPVUB4BYC5WPCnc0UGhqg0ZqxsPRCsVwoNXpBp76WS7_plT wQcn9BhwhGlZ0YY0bPKVf4HX_4xlFJWc6hxmYf_M63Vz03iD2KPeqbVBS9lbeIz7c1ym55PE9mKB OhPSG0VN0X2E0UCOtSCMFxqyjppaLgN5EA2WfFwwiFBRxvEvpmsaQivr0htWNsrflHYmLlERLdBT .VrC27x.ZA4_HQp00srAY4Gb1XEn03kiBcKqMMpDYFMtqIbGHf3rvj0mv9Sxo5xDBqnDP1bIFqSk z4irFtqVkYx.QNIFICVTX2mbChpLE17DJjUWlOtm1oyoHqxXlx0KvkfDP635Zu1K7l30gGGTE9gt rSTm4y5AhHSBQqQWfSPpU5Sx_8ljZQzEM2qRwvNg05zjMRN__jQESYWLI7Phc_rcjCKbNwpJChd2 RR.t7kskIa5.cld8z2Fq1XnTHs2kQDNJZEOCAbTmXnJ.1gVmYsrMLuQ0YpY6rR2FT8WsFoFg2CqT wTkhCwSSlj4viYjPRsJPThmlKdvoQc56LAFvq.OZdT8k3ANmht1C1u3IrVqcyvvO6MwPWueCvTwK sqdlKTuMC6oD01TXm7OS2WT1xYjfgoAzi8nUbnF9n2UytD.mEgGQ.X.q76BfAFv2qTzEWWr.VWUY mZNFO5khCRjFyDmlxciD31SqLrJ_1M4gCqmkYPeKraZ7AJvny1tkKQGE0Vu7_xqEL5Se6v3Ceufo 5Hd0YpItb8ljNrwbt0Rx47eUlgjRP6HHHhXXua.EdhvAMLbngbtO5v1MYsIKLO9fXKBxBNgyd6pj AqgUifECsbqdNVf5nWEU0YDSZyMzj2020o6R2RuO5g44.3Dew.pV0oBpXFzaS0qvHqIV7I1rF4Qc jFrEFGaQKYKXTTvgh_MuPYyjTO2cCeqXB9OY.bnOwWZl7mTjxmU6Kb.08hMRv3db2chrGCEmx8aV .yrn8P2o7sXIRut4KXt40kK2rCiMIUgJANj6msTdYmTxGAWyp77pY4ovSK.bHnHOHhVN_fEa5YZC qw0Ujmr..lb2S6FLafibS3F5n_lOb5l4Fxt3lvWTGGR3ecu1TnDPaygyQm31NVoY_LQM8nsBGDT0 v8aTOxPF0yb_6XE7.tAjzsxu21C2KsTFZAzpZtnMptPPIZx5IlzNEMT3QeI_hv.3wJVkcaa0usmH zRr.lz7nwdsbCC1I0STxZ_gG2jn3Ifg2U_y7c8zM7DVqcCp_GDxiogQVR2EbyZ55K525o2moIljC c4JNdAm3YcMd94luW4ohync5KTy0zzZKzAox1ODIXIJy3INd8vf4zEhoEVaA22WDN78W05YfFWcS IKIXGE3fS8qI6LgwCSVnUi4V_Wmwxuavpy40hPS40C7qjAttytk_PtcDtOjFBY7gAQB4ELAgiKzr dj7rj3xeJ6A5vygfSuZsZoJN5Q1O1LxVM4HTU0pKb7X.jjFe66Jf3N2N0MPHG1kMjGgj2WD1yfo5 8na3J9dnv9k_JsI.QHZZS_jl8wMr.g0HQB.9fOIqDXySjnHJnP6B15BmGy0rpOfpPd1TWCPiWYuG 8xmyRX6TF79uKS9BfXMWR5.ZxtT7z8sqRIFF8fqFeYrL5.nYxXg.sS0YY0u407ZsBzgVrl83vwkW Hmiu6PUkwfhRzl_6cL29g5Pc6ExDEq2NgF9YzjEZk8W7ZbdCPSd_IHPFqWoqyndUFEVCdEvAtlvc 95k8CQinpmdjKI5SwX994pCWMGNmUSZd3x8KDLsnAzRfsakbvX0mYayXuy.7cBT2QyuIb4Ugbd4b y86NMG2gRJ5c_ybmyeb39zJID8DLj3MAMASuKjUlotPqIhqXnIsCyM4AlKp7uhexdcK6USg8W4c1 YsKXeubGKC1zNEpdGoyIMzjagzCkr4LZlhCqlVE2wTfve51eHOmss8DinHwXUfwml42UVxhA9OYm 1 X-Sonic-MF: X-Sonic-ID: 73c7f526-a957-41be-ac47-57f7d13d9aad Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Mon, 1 May 2023 01:57:29 +0000 Received: by hermes--production-gq1-546798879c-qx24x (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b5c8fa2a09aca77006c3b1692a533c4b; Mon, 01 May 2023 01:57:27 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes? From: Mark Millard In-Reply-To: Date: Sun, 30 Apr 2023 18:57:16 -0700 Cc: FreeBSD Hackers , freebsd-arm Content-Transfer-Encoding: 7bit Message-Id: References: <7AE28A5B-109E-4C26-9D70-BCA5D49CD79D@yahoo.com> <02DC03AE-E082-4FB5-AA0D-396F64CC23CB@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4Q8mYk2HKhz3Gmm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Apr 30, 2023, at 17:44, Mateusz Guzik wrote: > can you redo zfs test with: > sysctl vfs.zfs.per_txg_dirty_frees_percent=5 Sure. Result summary: Seems to have avoided the sustained periods of low load average activity. Much better for the context. Context: Original ZFS USB3 media. World or Kernel in use had been built (non-debug style) using: -mcpu=cortex-a78C+flagm+nofp16fml Steps for this test . . . # poudriere pkgclean -A -jmain-CA78C . . . # sysctl vfs.zfs.per_txg_dirty_frees_percent=5 vfs.zfs.per_txg_dirty_frees_percent: 30 -> 5 # grep USE_TMPFS= /usr/local/etc/poudriere.conf # EXAMPLE: USE_TMPFS="wrkdir data" USE_TMPFS="data" #USE_TMPFS=all # poudriere bulk -jmain-CA78C -w -f ~/origins/CA78C-origins.txt . . . At 15 minutes into the build: 46 ports in 1st 15 minutes. Load average stayed reasonable for the configuration. At 30 minutes into the build: 102 ports in 1st 30 minutes. Load average still reasonable for the configuration. Looks good compared to before. I've no clue what optimal would be for the context, but vfs.zfs.per_txg_dirty_frees_percent=5 is vastly better for the context than the default 30 was. Thanks. I'm going to stop the test and do the conversion to the U2 960GB Optane media in the USB3 adaptor and then compare USE_TMPFS=data vs. USE_TMPFS=all --but using your vfs.zfs.per_txg_dirty_frees_percent=5 assignment. > On 5/1/23, Mark Millard wrote: >> As the evidence this time is largely independent of the details >> reported previousy, I'm top posting this. >> >> The previous ZFS on USB3 results were based on poudriere using >> "USE_TMPFS=data", meaning that almost all file I/O was via ZFS >> to the USB3 media. >> >> The UFS on U2 960GB Optane via USB3 adapter results did not not >> suffer the reported problems, despite "USE_TMPFS=data". (I let >> it run to completion.) But this had both a media and a file >> system difference. >> >> This time the results are for just changing poudriere to use >> "USE_TMPFS=all" instead but back on the original ZFS on >> USB3 media. The implication is that the vast majority of the >> file I/O is not via ZFS to the USB3 media. >> >> The context has 32 GiBytes of RAM and about 118 GiBytes of >> swap/paging space. It would need to page if pet run to >> completion. >> >> Observing, the load average is behaving normally: Things are >> not stuck waiting. "gstat -spod" indicates the ZFS I/O is >> not sustained (no paging in swap space use yet). >> >> First 1 hr: 262 ports built. >> >> But this had both a media and a file system difference again. >> >> I'm stopping after this, in order to set up the next just- >> ZFS experiments. >> >> >> Next experiments: >> >> I expect to move the ZFS context to U2 960GB Optane media >> used with the USB3 adapter and to test both "USE_TMPFS=data" >> and "USE_TMPSF=all", probably letting USE_TMPFS=all run to >> completion. >> >> If Optane based USE_TMPFS=data context still has the problem, >> even the more performance media would have been not enough to >> avoid what would then appear to be a ZFS problem, two other >> file systems not having the problem. >> >> The Optane based USE_TMPFS=all context should just handle the >> paging and more rare ZFS I/O quicker. I do not expect problems >> for this combination, given the UFS on Optane USB3 results >> and the partial USE_TMPFS=all non-Optane USB3 results. Even >> with ZFS working, this likely is the more performant type of >> context for the Windows Dev Kit 2023, given that I'm leaving >> Windows 11 Pro in place on the internal media. >> >> >> Hypothesis for the original problem: >> >> I wonder if ZFS write activity to the USB3 media was largely >> blocking the ZFS read activity to the same media, causing >> lots of things to have to spend much time waiting for data >> instead of making progress, leading to long periods of low >> load averages. >> >> >> Older material: [I've removed the older material.] === Mark Millard marklmi at yahoo.com