Speeding up resilvering

[Karl Denninger]

On 7/8/2015 11:20, Matthew Seaman wrote:
> On 07/08/15 15:28, Mike Tancsa wrote:
>>> Look at the IO saturation on the disk channel(s) involved with either
>>>> systat -vm or iostat.  If the channel is saturated then there's nothing
>>>> you can do in terms of tuning; the question then turns to why actual I/O
>>>> performance is so poor and has to be addressed there.
>> I had one server that was taking ages, and it turned out to be the
>> controller, not the disk that was hosed. As Karl suggested, take a look
>> at the throughput on gstat. Is anyone disk or groups of disks lagging
>> far behind on write speeds ?  In my case, it was a very obvious and
>> glaring outlier.
> Thanks for the suggestions.  I've been playing with gstat et al, and as
> far as I can tell, all the drives are behaving reasonably well.  I'm
> certainly getting 90-100% capacity (mostly reads) continually on the
> original drives whilst the new one (mostly writes) seems to go in
> bursts.  Which is pretty much what I'd expect when resilvering a RAIDZ.
>
> So, having exhausted that, I actually sat down and timed what progress
> it was making rather more carefully.  Turns out my impression that
> applying the sysctl tweaks I mentioned previously had little effect was
> wrong.  Current projection is about 50h total to do the resilvering,
> which is much, much better than the approx 12days I was expecting
> previously.  In fact, that's pretty much inline with what I'd expect
> from this hardware.
>
> 	Cheers,
>
> 	Matthew
>
>
>

Be aware that when a resilver /starts /it does a number of very small
and diverse I/O operations.  Throughput during that time _*stinks*_, and
as a result in the first few minutes to half-hour or so (with a very
large vdev) you will see utterly ridiculous projections of completion time.

Once that part of the process completes (usually within a few minutes on
modest to moderate size vdevs, but it will be longer on very large ones)
the performance level of the resilver will go up a lot and the projected
completion time will become far more reasonable.

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2944 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20150708/1c61083f/attachment.bin>