How to speed up slow zpool scrub?

[Adam Nowacki]

On 2016-04-27 20:11, Miroslav Lachman wrote:
> Adam Nowacki wrote on 04/27/2016 19:36:
>> On 2016-04-27 14:05, Miroslav Lachman wrote:
> 
>>> Thank you for output of your zpool scrub. It is definitely faster than
>>> mine.
>>>
>>> To: Paul pk1048
>>> Mine scrub does not repair anything. Drives are OK (in SMART).
>>> CPU is about 70%-90% idle during scrub + rsync backup and drives are
>>> about 60%-70% busy according to iostat:
>>>
>>> root at kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3
> 
>>> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
>>> ada0     121.8  16.6   602.1   526.9    3   9.2  52
>>> ada1     122.2  16.5   606.9   528.5    4   9.8  54
>>> ada2     117.0  14.6   601.7   524.9    2  11.3  60
>>> ada3     120.6  13.5   610.1   491.3    0  11.4  61
>>>
>>> I really don't know why it cannot go faster if nothing is loaded for
>>> 100%.
>>
>> 1) zpool scrub is single threaded with prefetch,
> 
> Hmm, this can be the cause. Does it mean that ZFS is faster on CPU with
> higher "per core" power and number of cores (threads) is not so
> important? (in this case of scrub)

No. Zpool scrub thread doesn't need much CPU time as disk I/O handling
(including decompression and checkums) happens in other threads.

>> 2) some data blocks do not span all disks (metadata, small files,
>> compression)
>> End result is that zfs can't always read from all disks during scrub so
>> disk utilization is going to be less than 100% even when going at full
>> speed.
> 
> Thank you for the explanation.

Try increasing vfs.zfs.top_maxinflight to 100 or more.