How to speed up slow zpool scrub?

[Miroslav Lachman]

Brandon J. Wandersee wrote on 04/26/2016 19:11:
>
> Miroslav Lachman writes:
>
>> krad wrote on 04/26/2016 16:02:
>>> Erk, i would try to move your system off those data disks as you have
>>> two pools competing for the disk spindles. This is never ideal. You can
>>> by all means backup your os to those data pools but keep them on
>>> separate physical mediums. A couple of small SSD would do the trick
>>> nicely and could probably be added with no down time. You would probably
>>> want to find a suitable window though to make sure the box reboots
>>> nicely though.
>>
>> The system pool is really small - only 15GB and scrub is done relatively
>> fast. This machine cannot handle additional disks so I cannot move
>> system to other devices anyway. I tried system on USB flashdisk (read
>> only) in the past but it was slow and USB disk broke early.
>>
>
> I see no sign that the partitions are encrypted, so I can't think of a
> reason that the system and data can't reside on the same pool.

Believe it or not there are systems which can't boot from RAIDZ but can 
boot from mirrored drives. Some controllers does not expose all drives 
at boot time, only the first one.

> The
> physical effort of the read arms on the disks isn't the only resource
> to consider---each pool has its own ARC, for example, and if you have
> compression enabled each pool will compete for CPU cycles. Having two
> pools grabbing at such limited resources is going to noticeably hurt
> performance.

Read arms moving cannot be an issue. They must move the same way in one 
or ten pools if the need to read 100 files on different places of the 
disks. Sharing ARC anc CPU cycles can be an issue but as I said before 
the CPU is about 70% idle and I believe ARC is not used for scrub and 
resilver.

> All that said, running even a single relative large pool on a
> low-powered dual-core CPU and 5G of RAM will noticeably limit *every*
> ZFS operation. I would bet that while you might be able to shave some
> time off the scrub, at best it's still going to take at least an entire
> day on that hardware.

I will be fine with entire day. It is expected time for this pool size. 
But I don't know why it takes 3-4 days if CPU nor disks are utilized to 100%

> As a side note, I'm not so sure it's a good idea to use two different
> redundancy methods on the same set of disks, though I can't say exactly
> what the downsides would be (if any). It just seems odd.

Explained above. I don't think there can be any issue with two different 
redundancy methods.

Thank you for your effort.

Miroslav Lachman