How to speed up slow zpool scrub?

[Steven Hartland]

On 27/04/2016 21:20, Adam Nowacki wrote:
> On 2016-04-27 20:11, Miroslav Lachman wrote:
>> Adam Nowacki wrote on 04/27/2016 19:36:
>>> On 2016-04-27 14:05, Miroslav Lachman wrote:
>>>> Thank you for output of your zpool scrub. It is definitely faster than
>>>> mine.
>>>>
>>>> To: Paul pk1048
>>>> Mine scrub does not repair anything. Drives are OK (in SMART).
>>>> CPU is about 70%-90% idle during scrub + rsync backup and drives are
>>>> about 60%-70% busy according to iostat:
>>>>
>>>> root at kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3
>>>> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
>>>> ada0     121.8  16.6   602.1   526.9    3   9.2  52
>>>> ada1     122.2  16.5   606.9   528.5    4   9.8  54
>>>> ada2     117.0  14.6   601.7   524.9    2  11.3  60
>>>> ada3     120.6  13.5   610.1   491.3    0  11.4  61
>>>>
>>>> I really don't know why it cannot go faster if nothing is loaded for
>>>> 100%.
>>> 1) zpool scrub is single threaded with prefetch,
>> Hmm, this can be the cause. Does it mean that ZFS is faster on CPU with
>> higher "per core" power and number of cores (threads) is not so
>> important? (in this case of scrub)
> No. Zpool scrub thread doesn't need much CPU time as disk I/O handling
> (including decompression and checkums) happens in other threads.
>
>>> 2) some data blocks do not span all disks (metadata, small files,
>>> compression)
>>> End result is that zfs can't always read from all disks during scrub so
>>> disk utilization is going to be less than 100% even when going at full
>>> speed.
>> Thank you for the explanation.
> Try increasing vfs.zfs.top_maxinflight to 100 or more.
>
If your on a recent version vfs.zfs.vdev.scrub_max_active is usually the 
most pertinent change to make.

     Regards
     Steve