Sluggish performance on head r311648

[Ultima]

> One thing I keep a look out for, is in gstat, if one drive is busier
> than the others, it's a clear sign of a drive dying. Offlining and
> replacing the drive usually makes a huge difference.

> Also, in all of these cases, SMART data shows no sign of problem and no
> errors in 'zpool status'. Just watch and see if any of the drives is
> working harder than the others has been the surest way to troubleshoot
> performance issues, in my experience.

Staring at gstat for about 5 minutes, I'm not really sure if any one stand
out. They seem to all vary in activity. Something that does stand out is
that on occasion a few will spike in the red, sometimes 1 or 2, other times
8ish (24 drives in pool total). I also notice once, instead of all drives
working at the same time, it seems to move like a wave. Red activity hit at
the top of gstat and worked its way down. Not all drives hit red during
this wave, around 16 in this 5ish seconds. Not sure if this is out of the
ordinary tho.

I did look at SMART before porting. One thing I thought about is the
corrected errors amount for each drive. When I get some time I'll create a
graph and try and determine the possible bad drive(s, hopefully without the
s) based on this information.




> Just to eliminate the simple - is the zpool capacity high? When a pool
> gets into the 80-90% capacity, performance drops.
The pool is at 28% capacity atm according to zpool list.

On Wed, Jan 11, 2017 at 9:03 PM, Shane Ambler <FreeBSD at shaneware.biz> wrote:

> On 11/01/2017 15:32, Ultima wrote:
>
>> I'v been noticing lately sluggish performance, maybe zfs? First noticed
>> this a few days ago right after upgrading on Jan 7th to r311648 and the
>> last upgrade before that was around dec 30-jan 1 (not sure of rev).
>> Decided
>> to upgrade again today. I usually build and install head every week or
>> two,
>> but I have been extremely busy the past couple months.
>>
>> FreeBSD U1 12.0-CURRENT FreeBSD 12.0-CURRENT #16 r311903: Tue Jan 10
>> 17:20:11 EST 2017 amd64
>>
>> Normally when one of my services scans a few directories it takes about 15
>> seconds tops, it has been taking several minutes. I want to note that this
>>
>
> Just to eliminate the simple - is the zpool capacity high? When a pool
> gets into the 80-90% capacity, performance drops.
>
>
>
> --
> FreeBSD - the place to B...Storing Data
>
> Shane Ambler
>
>