ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ...

[Allan Jude]

On 2017-12-14 09:52, O. Hartmann wrote:
> Am Thu, 14 Dec 2017 15:46:17 +0100
> Dimitry Andric <dim at FreeBSD.org> schrieb:
> 
>> On 14 Dec 2017, at 14:43, O. Hartmann <ohartmann at walstatt.org> wrote:
>>>
>>> Am Thu, 14 Dec 2017 14:09:39 +0100
>>> Daniel Nebdal <dnebdal at gmail.com> schrieb:  
>>>> On Thu, Dec 14, 2017 at 12:48 PM, O. Hartmann <ohartmann at walstatt.org> wrote:  
>>>>> I just started the rebuild/resilvering process and watch the pool crwaling at ~ 18
>>>>> MB/s. At the moment, there is no load on the array, the host is a IvyBridge XEON
>>>>> with 4 core/8 threads and 3,4 GHz and 16 GB of RAM. The HDDs are attached to a
>>>>> on-board SATA II (300 MB/s max) Intel chip - this just for the record.
>>>>>
>>>>> Recently, I switch on the "sync" attribute on most of the defined pools's zfs
>>>>> filesystems
>>>>> - I also use a SSD for ZIL/L2ARC caching, but it seems to be unused recently in
>>>>> FreeBSD CURRENT's ZFS - this from a observers perspective only.
>>>>>
>>>>> When scrubbing, I see recently also reduced performance on the pool, so I'm
>>>>> wondering about the low throughput at the very moment when resilvering is in
>>>>> progress.
>>>>>
>>>>> If the "perspective" of "zpool status" is correct, then I have to wait after two
>>>>> hours for another 100 hours - ~ 4 days? Ups ... I think there is something badly
>>>>> misconfigured or missing.  
>> ...
>>>> This is kind of to be expected - for whatever reason, resilvers seem
>>>> to go super slow at first and then speed up significantly. Just don't
>>>> ask me how long "at first" is - I'd give it several (more) hours.  
>>
>> Hopefully this will get better in the future, please read:
>>
>> http://open-zfs.org/wiki/Scrub/Resilver_Performance
>>
>> -Dimitry
>>
> 
> It has already been started to become better ;-)
> 
> After a while now, the throughput is at 128 MBytes/s and the estimated time decreased to
> ~ 8 h now - that is much more appreciable than 4 days ;-)
> 
> Kind regards,
> Oliver
> 
> 

The time estimate is a pure average over the entire length of the scrub
or resilver operation. The very start of the operation is quite slow,
because it involves a lot of random seeks, and the read-ahead is not
very smart (patches are in progress). And yes, after you adjust the
resilver_delay, it will take time for its impact to be visible in 'zpool
status'.

At times, you are better off looking at 'gstat', to see how busy the
disks actually are. You will likely notice the bottleneck is IOPS, you
will be at the limit of at least one the entire duration of the
resilver, unless your resilver_delay is high enough to leave some
available IOPS.

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-current/attachments/20171214/7438776e/attachment.sig>