How to speed up slow zpool scrub?

[Adam Nowacki]

On 2016-04-27 14:05, Miroslav Lachman wrote:
> Andy Farkas wrote on 04/27/2016 06:39:
>>> -----Original Message-----
>>> From: PK1048 [mailto:paul at pk1048.com]
>>> Sent: Wednesday, 27 April 2016 12:34 PM
>>> To: Andy Farkas <andyf at andyit.com.au>
>>> Cc: freebsd-fs at freebsd.org
>>> Subject: Re: How to speed up slow zpool scrub?
>>>
>>> ...
>>> Scrubs (and resilver) operations are essentially all random I/O. Those
>>> drives are low end, low performance, desktop drives.
>>
>> Yes, the system is an old low end, low performance desktop. That was
>> my point, that it took 25 hours to scrub 7.52T and not 4 days like the
>> OP is saying.
> 
> Thank you for output of your zpool scrub. It is definitely faster than
> mine.
> 
> To: Paul pk1048
> Mine scrub does not repair anything. Drives are OK (in SMART).
> CPU is about 70%-90% idle during scrub + rsync backup and drives are
> about 60%-70% busy according to iostat:
> 
> root at kiwi ~/# iostat -x -w 10 ada0 ada1 ada2 ada3
>                         extended device statistics
> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
> ada0      70.1  17.9  1747.0   802.4    0   7.0  24
> ada1      70.1  17.9  1747.1   802.4    0   7.0  25
> ada2      66.9  17.1  1686.4   791.3    4   6.5  23
> ada3      66.9  16.9  1686.3   790.1    2   6.6  23
>                         extended device statistics
> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
> ada0      93.6  13.0   576.4   244.0    0  21.6  70
> ada1      98.6  12.9   587.2   246.4    2  20.5  71
> ada2      87.9  15.3   566.0   242.4    3  20.5  67
> ada3      84.9  14.5   549.2   237.2    3  20.4  66
>                         extended device statistics
> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
> ada0      98.7  42.5  1924.7  2536.3    1  26.3  86
> ada1      99.1  45.5  1931.5  2671.5    1  23.8  87
> ada2      94.2  44.9  1840.7  2720.3    0  20.1  76
> ada3      93.6  42.7  1807.9  2607.1    0  18.7  75
>                         extended device statistics
> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
> ada0     108.2  28.2  1092.6  1316.6    2  17.3  68
> ada1     101.6  26.3  1053.8  1183.4    3  15.5  67
> ada2      98.6  26.0  1000.2  1126.2    2  12.2  57
> ada3     104.0  24.0  1015.8  1080.6    3  14.1  60
>                         extended device statistics
> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
> ada0     116.0  18.5   821.8   807.8    0  12.9  62
> ada1     117.2  18.5   822.2   807.0    0  13.5  63
> ada2     110.8  20.9   743.0   803.8    0  11.1  58
> ada3     108.2  20.0   688.2   755.0    2  11.3  55
>                         extended device statistics
> device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
> ada0     121.8  16.6   602.1   526.9    3   9.2  52
> ada1     122.2  16.5   606.9   528.5    4   9.8  54
> ada2     117.0  14.6   601.7   524.9    2  11.3  60
> ada3     120.6  13.5   610.1   491.3    0  11.4  61
> 
> I really don't know why it cannot go faster if nothing is loaded for 100%.

1) zpool scrub is single threaded with prefetch,
2) some data blocks do not span all disks (metadata, small files,
compression)
End result is that zfs can't always read from all disks during scrub so
disk utilization is going to be less than 100% even when going at full
speed.