Re: measuring swap partition speed

void <void_at_f-m.fm> wrote on
Date: Thu, 28 Dec 2023 12:24:14 UTC :

> Plugged the disk into usb3 interface on amd64 machine and ran the following [a] tests all at roughly the same time. The write test (2x git clone/pull) goes to a ufs partition. Writing 2x swap at the same time never got below 3mb/s
> 
> tl;dr :
> 
> 1. in the arm64 context, the extreme decline in swap performance happens if swap 
> gets used, by anything (system)
> 
> 2. in the [1] context, even when swap later becomes unused by the system, 
> the performance problem persists.
> 
> 3. the situation is recoverable only via swapoff then on.
> Then the swap partition can be written to in the tests at full speed.
> Writing subsequent swap tests also run at full speed.
> When the system swaps again, the problem returns.
> 
> 4. the slowdown does not arise in an amd64 context beyond what one would expect 
> 
> 5. ufs or zfs makes no real difference
> 
> it wouldn't matter to me if the write speed was 3mb, 16mb or 22mb/s.
> What causes problems is the sub-1mb/s speed (worst I've seen it is 300kb/s) and the
> non-recovery of that swap space when load eases off. I'm probably doing myself no 
> favours using zfs in the arm64 context though.
> 
> thank you everyone who helped, I now have an idea of a workaround
> 
> ########### amd64 tests of the same disk

Could you produce the same sort of report (gstat -spod
and such included) for the arm64 context? The direct
(well matched) comparison/contrast to amd64 could prove
of interest to folks that work on arm64 support.

> # gpart show da4
> => 40 1953525088 da4 GPT (932G)
> 40 532480 1 efi (260M)
> 532520 2008 - free - (1.0M)
> 534528 4194304 2 freebsd-swap (2.0G)
> 4728832 4194304 4 freebsd-swap (2.0G)
> 8923136 4194304 5 freebsd-swap (2.0G)
> 13117440 4194304 6 freebsd-swap (2.0G)
> 17311744 2048 - free - (1.0M)
> 17313792 8384512 7 freebsd-ufs (4.0G)
> 25698304 2048 - free - (1.0M)
> 25700352 1927823360 3 freebsd-zfs (919G)
> 1953523712 1416 - free - (708K)
> 
> # newfs -U /dev/da4p7
> /dev/da4p7: 4094.0MB (8384512 sectors) block size 32768, fragment size 4096
> using 7 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
> with soft updates
> super-block backups (for fsck_ffs -b #) at:
> 192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880
> 
> # mount /dev/da4p7 /mnt
> # cd /mnt
> 
> [a]
> 
> #### all the following tests were run at roughly the same time, in different 
> terminal windows. 2x git (ufs) & 2x dd.
> 
> # time -l git clone https://git.freebsd.org/src.git
> Cloning into 'src'...
> remote: Enumerating objects: 4437205, done.
> remote: Counting objects: 100% (383246/383246), done.
> remote: Compressing objects: 100% (28769/28769), done.
> remote: Total 4437205 (delta 377514), reused 354477 (delta 354477), pack-reused 4053959
> Receiving objects: 100% (4437205/4437205), 1.59 GiB | 6.10 MiB/s, done.
> Resolving deltas: 100% (3526142/3526142), done.
> Updating files: 58% (57132/97280)
> /mnt: write failed, filesystem is full
> error: unable to write file sys/contrib/dev/acpica/components/utilities/utxface.c
> Updating files: 58% (57317/97280)
> /mnt: write failed, filesystem is full
> Updating files: 100% (97280/97280), done.
> fatal: unable to checkout working tree
> warning: Clone succeeded, but checkout failed.
> You can inspect what was checked out with 'git status'
> and retry with 'git restore --source=HEAD :/'
> 
> 639.76 real 566.78 user 41.47 sys
> 1054240 maximum resident set size
> 2566 average shared memory size
> 222 average unshared data size
> 127 average unshared stack size
> 475518 page reclaims
> 0 page faults
> 0 swaps
> 3 block input operations
> 15556 block output operations
> 422759 messages sent
> 1527197 messages received
> 606 signals received
> 3656856 voluntary context switches
> 851907 involuntary context switches
> # 
> 
> # time -l git clone https://git.freebsd.org/ports.git
> Cloning into 'ports'...
> remote: Enumerating objects: 6046014, done.
> remote: Counting objects: 100% (936/936), done.
> remote: Compressing objects: 100% (120/120), done.
> remote: Total 6046014 (delta 923), reused 816 (delta 816), pack-reused 6045078
> Receiving objects: 100% (6046014/6046014), 1.16 GiB | 5.28 MiB/s, done.
> Resolving deltas: 100% (3650283/3650283), done.
> 
> /mnt: write failed, filesystem is full
> fatal: sha1 file '/mnt/ports/.git/objects/pack/tmp_rev_cQdim5' write error. Out of diskspace
> fatal: fetch-pack: invalid index-pack output
> 553.87 real 0.41 user 1.47 sys
> 6092 maximum resident set size
> 2400 average shared memory size
> 208 average unshared data size
> 118 average unshared stack size
> 451 page reclaims
> 0 page faults
> 0 swaps
> 4 block input operations
> 2 block output operations
> 152484 messages sent
> 414280 messages received
> 0 signals received
> 409098 voluntary context switches
> 571 involuntary context switches
> # 
> 
> # time -l dd if=/dev/urandom of=/dev/da4p4 bs=8k conv=sync status=progress
> dd: /dev/da4p4: end of device2040 MiB) transferred 660.001s, 3242 kB/s
> 
> 262145+0 records in
> 262144+0 records out
> 2147483648 bytes transferred in 660.734537 secs (3250146 bytes/sec)
> 660.87 real 0.44 user 10.61 sys
> 1992 maximum resident set size
> 20 average shared memory size
> 8 average unshared data size
> 132 average unshared stack size
> 133 page reclaims
> 0 page faults
> 0 swaps
> 0 block input operations
> 262145 block output operations
> 0 messages sent
> 0 messages received
> 660 signals received
> 262424 voluntary context switches
> 2182 involuntary context switches
> # 
> 
> # time -l dd if=/dev/urandom of=/dev/da4p5 bs=8k conv=sync status=progress
> dd: /dev/da4p5: end of device2047 MiB) transferred 666.010s, 3223 kB/s
> 
> 262145+0 records in
> 262144+0 records out
> 2147483648 bytes transferred in 666.162391 secs (3223664 bytes/sec)
> 666.16 real 0.57 user 10.25 sys
> 1992 maximum resident set size
> 21 average shared memory size
> 8 average unshared data size
> 138 average unshared stack size
> 133 page reclaims
> 0 page faults
> 0 swaps
> 0 block input operations
> 262145 block output operations
> 0 messages sent
> 0 messages received
> 666 signals received
> 262420 voluntary context switches
> 2867 involuntary context switches
> # 
> # gstat -spod
> 
> dT: 1.007s w: 1.000s
> L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o %busy Name
> 
> 1 1096 0 0 0 0.0 1096 19 20876 2.2 0 0 0 0.0 0 0.0 107.2| da4
> [...]
> 2 1252 0 0 0 0.0 1252 8 10020 1.5 0 0 0 0.0 0 0.0 100.0| da4
> 
> When the simultaneous write test completes, swap speed in an amd64 context 
> reverts to normal, without any user intervention:
> 
> # time -l dd if=/dev/urandom of=/dev/da4p5 bs=8k conv=sync status=progress
> dd: /dev/da4p5: end of device2046 MiB) transferred 200.020s, 11 MB/s
> 
> 262145+0 records in
> 262144+0 records out
> 2147483648 bytes transferred in 200.166970 secs (10728462 bytes/sec)
> 200.16 real 0.45 user 9.69 sys
> 2016 maximum resident set size
> 16 average shared memory size
> 6 average unshared data size
> 106 average unshared stack size
> 130 page reclaims
> 0 page faults
> 0 swaps
> 0 block input operations
> 262145 block output operations
> 0 messages sent
> 0 messages received
> 200 signals received
> 262147 voluntary context switches
> 320 involuntary context switches

===
Mark Millard
marklmi at yahoo.com