How to speed up slow zpool scrub?

[Miroslav Lachman]

Eric A. Borisch wrote on 04/27/2016 00:26:

>> Is there any tuning to make scrub faster in this idle time?
>> Or is it better to do it other way - slower scrub with even lower priority
>> taking for about one week but not affecting time of normal operations? (is
>> it dangerous to have scrub running this long or reboot machine during the
>> scrub?)
>
> It's likely easier to make scrub slower (but rsync faster) when the
> system is not idle, but hopefully no slower otherwise.

Yes, I've been thinking about this approach yesterday evening.

> I would start with increasing the vfs.zfs.scrub_delay (in ticks;
> typically 1/1000 of a second) settings to reduce scrub's impact to
> other IO this without impacting resilvers. (Expect longer total time
> as it will scrub less during rsyncs.) You can also increase
> vfs.zfs.scan_idle if scrub_delay isn't enough, but do note that
> vfs.zfs.scan_idle impacts resilvers, too -- so make sure you don't
> also increase vfs.zfs.resilver_delay, or you may get very slow
> resilvers during IO.
>
> The logic for this is in dsl_scan.c [1] -- look there for the final
> word on what the different parameters do; there are a number of
> informative comments throughout the file if you are not into the code
> itself.
>
> See also vfs.zfs.vdev.*_max_active and vfs.zfs.vdev.*_min_active, if
> you are really want more knobs to try.

Than you for this pointers, I will look at it and hope find some way to 
make things better for this machine.

> It's might also be worth trying '/usr/local/bin/perl
> /usr/share/dtrace/toolkit/hotkernel' to see what the kernel is up
> to...

Showing results with 0.1% or more.

kernel`atomic_add_long                                    647   0.1%
kernel`uma_zfree_arg                                      666   0.1%
kernel`ata_intel_sata_sidpr_read                          695   0.1%
kernel`uma_zalloc_arg                                     754   0.1%
kernel`free                                               895   0.1%
kernel`ata_tf_write                                      1011   0.1%
kernel`bzero                                             1453   0.1%
kernel`bcopy                                             1551   0.1%
kernel`acpi_timer_get_timecount                          2550   0.2%
kernel`_sx_xunlock                                       2894   0.2%
kernel`spinlock_exit                                     3347   0.3%
kernel`_sx_xlock                                         3379   0.3%
kernel`sched_idletd                                      4053   0.3%
zfs.ko`lz4_decompress                                    5293   0.4%
kernel`cpu_idle                                         65358   5.4%
kernel`acpi_cpu_c1                                    1096734  89.8%


zpool scrub and rsync backup is running at this time

   pool: tank0
  state: ONLINE
   scan: scrub in progress since Sun Apr 24 03:01:35 2016
         8.44T scanned out of 10.6T at 29.7M/s, 21h8m to go
         0 repaired, 79.64% done


>   - Eric
>
> [1] https://svnweb.freebsd.org/base/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c?view=markup
> or
> https://github.com/freebsd/freebsd/blob/releng/10.3/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c

Thank you for your very useful reply!

Miroslav Lachman