ZFS on FreeBSD 11.3 slower than 10.4

Sun Jun 7 17:09:27 UTC 2020

On 2020-06-06 00:57, Ronald Klop wrote:
> On Fri, 05 Jun 2020 22:57:15 +0200, Ronald Klop <ronald-lists at klop.ws> 
> wrote:
> 
>> On Sat, 30 May 2020 23:29:48 +0200, Miroslav Lachman 
>> <000.fbsd at quip.cz> wrote:
>>
>>> On 2020-05-30 22:10, Ronald Klop wrote:
>>>> On Sat, 23 May 2020 21:44:03 +0200, Miroslav Lachman 
>>>> <000.fbsd at quip.cz> wrote:
>>>>
>>>>> I upgraded my old desktop computer few month ago from old 10.4 
>>>>> based PC-BSD to stock FreeBSD 11.3. It uses single 2TB HDD 7200rpm.
>>>>> My problem is that upgraded version is really slow and some desktop 
>>>>> applications are very lagging (playing multimedia is interrupted 
>>>>> for a fraction of seconds) when there is heavy filesystem activity.
>>>>>
>>>>> I am using zfsnap2 for taking snapshots periodically and when there 
>>>>> is enough snapshots zfs destroy is called. In this time the user 
>>>>> experience is terrible. Starting new application like browser or 
>>>>> even something much smaller takes minutes. The old version based on 
>>>>> FreeBSD 10.4 behaves much better. I used the old version for years 
>>>>> and never have problems with interrupted multimedia playback.
>>>>>
>>>>> Are there some sysctls to tune to get better desktop interactivity 
>>>>> in heavy filesystem operations like zfs destroy, pkg check or other 
>>>>> "find" periodic scripts?
>>>
>>>
>>>> How full is the disk? ZFS has poor performance if the disk becomes 
>>>> full.
>>>> What is in /etc/sysctl.conf and /boot/loader.conf?
>>>> And did you try to boot 12.1 and did it have the same behavious?
>>>
>>> It is currently 77% full. But it is the same pool with the same 
>>> capacity as with 10.4.
>>>
>>> I didn't try 12.1, I need to stay on 11.3 for now.
>>>
>>> ## loader.conf
>>>
>>> nvidia_load="YES"
>>> drm_load="YES"
>>> drm2_load="YES"
>>> iicbus_load="YES"
>>> vboxdrv_load="YES"
>>> crypto_load="YES"
>>> aesni_load="YES"
>>> geom_eli_load="YES"
>>> vfs.zfs.arc_max="1024M"
>>> zfs_load="YES"
>>> iicbus_load="YES"
>>>
>>> ## sysctl.conf
>>>
>>> kern.coredump=0
>>> kern.maxfiles=49312
>>> vfs.usermount=1
>>> security.jail.allow_raw_sockets=1
>>> security.jail.sysvipc_allowed=1
>>> security.jail.mount_allowed=1
>>> security.jail.chflags_allowed=1
>>> hw.syscons.bell=0
>>> kern.sched.preempt_thresh=224
>>> kern.ipc.shm_allow_removed=1
>>> kern.shutdown.poweroff_delay=500
>>> kern.bootfile=/boot/kernel/kernel
>>> hw.usb.no_shutdown_wait=1
>>> hw.snd.default_unit=3
>>> kern.sched.interact=10
>>> vfs.aio.max_aio_per_proc=256
>>> vfs.aio.max_aio_queue=8192
>>> vfs.aio.max_aio_queue_per_proc=1024
>>> vfs.aio.max_buf_aio=64
>>> net.local.stream.recvspace=65536
>>> net.local.stream.sendspace=65536
>>>
>>>
>>> loader.conf and sysctl.conf are the same for 10.4 and 11.3 but 11.3 
>>> is much much slower when it comes to heavy IO like "find" daily 
>>> periodic scripts, zfs destroy, starting new applications etc.
>>>
>>>
>>> Kind regards
>>> Miroslav Lachman
>>
>>
>> I don't have anything I see which I'm sure will fix things, but you 
>> could try to remove/comment some of these sysctls to see if 11.3 has 
>> better defaults now.
>> kern.sched.preempt_thresh, kern.maxfiles, kern.sched.interact, vfs.aio.*
>>
>> What kind of machine is it? CPU, MEM?
>> What does gstat say about the saturation of the disk?
>>
>> Regards,
>> Ronald.
> 
> 
> What might also be interesting is which timer is selected. Sometimes for 
> some reason another time-source is chosen which can influence a lot of 
> things (like sound).
> Please post the output of "sysctl kern.eventtimer" and "sysctl 
> kern.timecounter" if possible of 10.4 and 11.3.
> 
> Or compare the /var/run/dmesg.boot files of 10.4 and 11.3 to see if some 
> hardware is recognized differently.

sysctls from 11.3 instance

# sysctl kern.eventtimer
kern.eventtimer.periodic: 0
kern.eventtimer.timer: LAPIC
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) 
HPET3(440) HPET4(440) i8254(100) RTC(0)
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.HPET4.quality: 440
kern.eventtimer.et.HPET4.frequency: 14318180
kern.eventtimer.et.HPET4.flags: 3
kern.eventtimer.et.HPET3.quality: 440
kern.eventtimer.et.HPET3.frequency: 14318180
kern.eventtimer.et.HPET3.flags: 3
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 14318180
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 14318180
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 550
kern.eventtimer.et.HPET.frequency: 14318180
kern.eventtimer.et.HPET.flags: 7
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 3192676730
kern.eventtimer.et.LAPIC.flags: 7

# sysctl kern.timecounter
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) 
dummy(-1000000)
kern.timecounter.hardware: TSC-low
kern.timecounter.alloweddeviation: 5
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 16156069
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 49991
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 14318180
kern.timecounter.tc.HPET.counter: 3887656701
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.TSC-low.quality: 1000
kern.timecounter.tc.TSC-low.frequency: 1596338365
kern.timecounter.tc.TSC-low.counter: 1764358755
kern.timecounter.tc.TSC-low.mask: 4294967295

CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.68-MHz K8-class  CPU)
real memory  = 13388218368 (12768 MB)
avail memory = 12337872896 (11766 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

last pid: 15694;  load averages:  0.42,  0.46,  0.38 
                                                      up 0+00:30:40 
19:01:30
138 processes: 1 running, 136 sleeping, 1 zombie
CPU:  3.2% user,  0.0% nice,  0.4% system,  0.5% interrupt, 95.9% idle
Mem: 1531M Active, 977M Inact, 39M Laundry, 3047M Wired, 6251M Free
ARC: 1275M Total, 311M MFU, 286M MRU, 2542K Anon, 23M Header, 652M Other
      124M Compressed, 476M Uncompressed, 3.85:1 Ratio
Swap: 4096M Total, 4096M Free

As you can see there is a lot of free memory.

I will try to catch gstat output next time and also post some info from 
10.4 instance.

Kind regards
Miroslav Lachman