Re: armv8.2-A+ tuned FreeBSD kernels vs. poudriere bulk and USB3 media: tx->tx_quiesce_done_cv related blocking of processes?

On Apr 30, 2023, at 17:44, Mateusz Guzik <mjguzik@gmail.com> wrote:

> can you redo zfs test with:
> sysctl vfs.zfs.per_txg_dirty_frees_percent=5

Sure.

Result summary: Seems to have avoided the sustained periods
of low load average activity. Much better for the context.


Context: Original ZFS USB3 media. World or Kernel in use
had been built (non-debug style) using:

-mcpu=cortex-a78C+flagm+nofp16fml


Steps for this test . . .

# poudriere pkgclean -A -jmain-CA78C
. . .

# sysctl vfs.zfs.per_txg_dirty_frees_percent=5
vfs.zfs.per_txg_dirty_frees_percent: 30 -> 5

# grep USE_TMPFS= /usr/local/etc/poudriere.conf
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS="data"
#USE_TMPFS=all

# poudriere bulk -jmain-CA78C -w -f ~/origins/CA78C-origins.txt
. . .

At 15 minutes into the build:
46 ports in 1st 15 minutes. Load average stayed reasonable
for the configuration.

At 30 minutes into the build:
102 ports in 1st 30 minutes. Load average still reasonable
for the configuration.

Looks good compared to before.


I've no clue what optimal would be for the context, but

vfs.zfs.per_txg_dirty_frees_percent=5

is vastly better for the context than the default 30 was.

Thanks.


I'm going to stop the test and do the conversion to the
U2 960GB Optane media in the USB3 adaptor and then
compare USE_TMPFS=data vs. USE_TMPFS=all --but using your
vfs.zfs.per_txg_dirty_frees_percent=5 assignment.


> On 5/1/23, Mark Millard <marklmi@yahoo.com> wrote:
>> As the evidence this time is largely independent of the details
>> reported previousy, I'm top posting this.
>> 
>> The previous ZFS on USB3 results were based on poudriere using
>> "USE_TMPFS=data", meaning that almost all file I/O was via ZFS
>> to the USB3 media.
>> 
>> The UFS on U2 960GB Optane via USB3 adapter results did not not
>> suffer the reported problems, despite "USE_TMPFS=data". (I let
>> it run to completion.) But this had both a media and a file
>> system difference.
>> 
>> This time the results are for just changing poudriere to use
>> "USE_TMPFS=all" instead but back on the original ZFS on
>> USB3 media. The implication is that the vast majority of the
>> file I/O is not via ZFS to the USB3 media.
>> 
>> The context has 32 GiBytes of RAM and about 118 GiBytes of
>> swap/paging space. It would need to page if pet run to
>> completion.
>> 
>> Observing, the load average is behaving normally: Things are
>> not stuck waiting. "gstat -spod" indicates the ZFS I/O is
>> not sustained (no paging in swap space use yet).
>> 
>> First 1 hr: 262 ports built.
>> 
>> But this had both a media and a file system difference again.
>> 
>> I'm stopping after this, in order to set up the next just-
>> ZFS experiments.
>> 
>> 
>> Next experiments:
>> 
>> I expect to move the ZFS context to U2 960GB Optane media
>> used with the USB3 adapter and to test both "USE_TMPFS=data"
>> and "USE_TMPSF=all", probably letting USE_TMPFS=all run to
>> completion.
>> 
>> If Optane based USE_TMPFS=data context still has the problem,
>> even the more performance media would have been not enough to
>> avoid what would then appear to be a ZFS problem, two other
>> file systems not having the problem.
>> 
>> The Optane based USE_TMPFS=all context should just handle the
>> paging and more rare ZFS I/O quicker. I do not expect problems
>> for this combination, given the UFS on Optane USB3 results
>> and the partial USE_TMPFS=all non-Optane USB3 results. Even
>> with ZFS working, this likely is the more performant type of
>> context for the Windows Dev Kit 2023, given that I'm leaving
>> Windows 11 Pro in place on the internal media.
>> 
>> 
>> Hypothesis for the original problem:
>> 
>> I wonder if ZFS write activity to the USB3 media was largely
>> blocking the ZFS read activity to the same media, causing
>> lots of things to have to spend much time waiting for data
>> instead of making progress, leading to long periods of low
>> load averages.
>> 
>> 
>> Older material:

[I've removed the older material.]

===
Mark Millard
marklmi at yahoo.com