ZFS on FreeBSD 11.3 slower than 10.4

Sun Jun 7 17:34:02 UTC 2020

On Sun, 07 Jun 2020 19:09:14 +0200, Miroslav Lachman <000.fbsd at quip.cz>  
wrote:

> On 2020-06-06 00:57, Ronald Klop wrote:
>> On Fri, 05 Jun 2020 22:57:15 +0200, Ronald Klop <ronald-lists at klop.ws>  
>> wrote:
>>
>>> On Sat, 30 May 2020 23:29:48 +0200, Miroslav Lachman  
>>> <000.fbsd at quip.cz> wrote:
>>>
>>>> On 2020-05-30 22:10, Ronald Klop wrote:
>>>>> On Sat, 23 May 2020 21:44:03 +0200, Miroslav Lachman  
>>>>> <000.fbsd at quip.cz> wrote:
>>>>>
>>>>>> I upgraded my old desktop computer few month ago from old 10.4  
>>>>>> based PC-BSD to stock FreeBSD 11.3. It uses single 2TB HDD 7200rpm.
>>>>>> My problem is that upgraded version is really slow and some desktop  
>>>>>> applications are very lagging (playing multimedia is interrupted  
>>>>>> for a fraction of seconds) when there is heavy filesystem activity.
>>>>>>
>>>>>> I am using zfsnap2 for taking snapshots periodically and when there  
>>>>>> is enough snapshots zfs destroy is called. In this time the user  
>>>>>> experience is terrible. Starting new application like browser or  
>>>>>> even something much smaller takes minutes. The old version based on  
>>>>>> FreeBSD 10.4 behaves much better. I used the old version for years  
>>>>>> and never have problems with interrupted multimedia playback.
>>>>>>
>>>>>> Are there some sysctls to tune to get better desktop interactivity  
>>>>>> in heavy filesystem operations like zfs destroy, pkg check or other  
>>>>>> "find" periodic scripts?
>>>>
>>>>
>>>>> How full is the disk? ZFS has poor performance if the disk becomes  
>>>>> full.
>>>>> What is in /etc/sysctl.conf and /boot/loader.conf?
>>>>> And did you try to boot 12.1 and did it have the same behavious?
>>>>
>>>> It is currently 77% full. But it is the same pool with the same  
>>>> capacity as with 10.4.
>>>>
>>>> I didn't try 12.1, I need to stay on 11.3 for now.
>>>>
>>>> ## loader.conf
>>>>
>>>> nvidia_load="YES"
>>>> drm_load="YES"
>>>> drm2_load="YES"
>>>> iicbus_load="YES"
>>>> vboxdrv_load="YES"
>>>> crypto_load="YES"
>>>> aesni_load="YES"
>>>> geom_eli_load="YES"
>>>> vfs.zfs.arc_max="1024M"
>>>> zfs_load="YES"
>>>> iicbus_load="YES"
>>>>
>>>> ## sysctl.conf
>>>>
>>>> kern.coredump=0
>>>> kern.maxfiles=49312
>>>> vfs.usermount=1
>>>> security.jail.allow_raw_sockets=1
>>>> security.jail.sysvipc_allowed=1
>>>> security.jail.mount_allowed=1
>>>> security.jail.chflags_allowed=1
>>>> hw.syscons.bell=0
>>>> kern.sched.preempt_thresh=224
>>>> kern.ipc.shm_allow_removed=1
>>>> kern.shutdown.poweroff_delay=500
>>>> kern.bootfile=/boot/kernel/kernel
>>>> hw.usb.no_shutdown_wait=1
>>>> hw.snd.default_unit=3
>>>> kern.sched.interact=10
>>>> vfs.aio.max_aio_per_proc=256
>>>> vfs.aio.max_aio_queue=8192
>>>> vfs.aio.max_aio_queue_per_proc=1024
>>>> vfs.aio.max_buf_aio=64
>>>> net.local.stream.recvspace=65536
>>>> net.local.stream.sendspace=65536
>>>>
>>>>
>>>> loader.conf and sysctl.conf are the same for 10.4 and 11.3 but 11.3  
>>>> is much much slower when it comes to heavy IO like "find" daily  
>>>> periodic scripts, zfs destroy, starting new applications etc.
>>>>
>>>>
>>>> Kind regards
>>>> Miroslav Lachman
>>>
>>>
>>> I don't have anything I see which I'm sure will fix things, but you  
>>> could try to remove/comment some of these sysctls to see if 11.3 has  
>>> better defaults now.
>>> kern.sched.preempt_thresh, kern.maxfiles, kern.sched.interact,  
>>> vfs.aio.*
>>>
>>> What kind of machine is it? CPU, MEM?
>>> What does gstat say about the saturation of the disk?
>>>
>>> Regards,
>>> Ronald.
>>   What might also be interesting is which timer is selected. Sometimes  
>> for some reason another time-source is chosen which can influence a lot  
>> of things (like sound).
>> Please post the output of "sysctl kern.eventtimer" and "sysctl  
>> kern.timecounter" if possible of 10.4 and 11.3.
>>  Or compare the /var/run/dmesg.boot files of 10.4 and 11.3 to see if  
>> some hardware is recognized differently.
>
> sysctls from 11.3 instance
>
> # sysctl kern.eventtimer
> kern.eventtimer.periodic: 0
> kern.eventtimer.timer: LAPIC
> kern.eventtimer.idletick: 0
> kern.eventtimer.singlemul: 2
> kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440)  
> HPET3(440) HPET4(440) i8254(100) RTC(0)
> kern.eventtimer.et.i8254.quality: 100
> kern.eventtimer.et.i8254.frequency: 1193182
> kern.eventtimer.et.i8254.flags: 1
> kern.eventtimer.et.RTC.quality: 0
> kern.eventtimer.et.RTC.frequency: 32768
> kern.eventtimer.et.RTC.flags: 17
> kern.eventtimer.et.HPET4.quality: 440
> kern.eventtimer.et.HPET4.frequency: 14318180
> kern.eventtimer.et.HPET4.flags: 3
> kern.eventtimer.et.HPET3.quality: 440
> kern.eventtimer.et.HPET3.frequency: 14318180
> kern.eventtimer.et.HPET3.flags: 3
> kern.eventtimer.et.HPET2.quality: 440
> kern.eventtimer.et.HPET2.frequency: 14318180
> kern.eventtimer.et.HPET2.flags: 3
> kern.eventtimer.et.HPET1.quality: 440
> kern.eventtimer.et.HPET1.frequency: 14318180
> kern.eventtimer.et.HPET1.flags: 3
> kern.eventtimer.et.HPET.quality: 550
> kern.eventtimer.et.HPET.frequency: 14318180
> kern.eventtimer.et.HPET.flags: 7
> kern.eventtimer.et.LAPIC.quality: 600
> kern.eventtimer.et.LAPIC.frequency: 3192676730
> kern.eventtimer.et.LAPIC.flags: 7
>
>
> # sysctl kern.timecounter
> kern.timecounter.tsc_shift: 1
> kern.timecounter.smp_tsc_adjust: 0
> kern.timecounter.smp_tsc: 1
> kern.timecounter.invariant_tsc: 1
> kern.timecounter.fast_gettime: 1
> kern.timecounter.tick: 1
> kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000)  
> dummy(-1000000)
> kern.timecounter.hardware: TSC-low

Does setting "sysctl kern.timecounter.hardware=HPET" or "sysctl  
kern.timecounter.hardware=ACPI-fast" help?

Ronald.

> kern.timecounter.alloweddeviation: 5
> kern.timecounter.stepwarnings: 0
> kern.timecounter.tc.ACPI-fast.quality: 900
> kern.timecounter.tc.ACPI-fast.frequency: 3579545
> kern.timecounter.tc.ACPI-fast.counter: 16156069
> kern.timecounter.tc.ACPI-fast.mask: 16777215
> kern.timecounter.tc.i8254.quality: 0
> kern.timecounter.tc.i8254.frequency: 1193182
> kern.timecounter.tc.i8254.counter: 49991
> kern.timecounter.tc.i8254.mask: 65535
> kern.timecounter.tc.HPET.quality: 950
> kern.timecounter.tc.HPET.frequency: 14318180
> kern.timecounter.tc.HPET.counter: 3887656701
> kern.timecounter.tc.HPET.mask: 4294967295
> kern.timecounter.tc.TSC-low.quality: 1000
> kern.timecounter.tc.TSC-low.frequency: 1596338365
> kern.timecounter.tc.TSC-low.counter: 1764358755
> kern.timecounter.tc.TSC-low.mask: 4294967295
>
>
> CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.68-MHz K8-class   
> CPU)
> real memory  = 13388218368 (12768 MB)
> avail memory = 12337872896 (11766 MB)
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>
>
> last pid: 15694;  load averages:  0.42,  0.46,  0.38  
>                                                       up 0+00:30:40  
> 19:01:30
> 138 processes: 1 running, 136 sleeping, 1 zombie
> CPU:  3.2% user,  0.0% nice,  0.4% system,  0.5% interrupt, 95.9% idle
> Mem: 1531M Active, 977M Inact, 39M Laundry, 3047M Wired, 6251M Free
> ARC: 1275M Total, 311M MFU, 286M MRU, 2542K Anon, 23M Header, 652M Other
>       124M Compressed, 476M Uncompressed, 3.85:1 Ratio
> Swap: 4096M Total, 4096M Free
>
> As you can see there is a lot of free memory.
>
> I will try to catch gstat output next time and also post some info from  
> 10.4 instance.
>
> Kind regards
> Miroslav Lachman