ZFS (or something) is absurdly slow

[Eitan Adler]

On Sat, Nov 23, 2013 at 12:07 PM, Thomas Steen Rasmussen
<thomas at gibfest.dk> wrote:
> On 23-11-2013 09:26, Eitan Adler wrote:
>>
>> Every so often I see absurdly slow tasks stuck on "zio->io_cv".
>>
>> For example a recent "git checkout file.c" did not complete for many
>> minutes
>> load: 0.65  cmd: git 74577 [zio->io_cv] 435.58r 0.20u 2.54s 0% 71488k
>>
>> I have seen "ls ~" take tends of minutes to complete.  Even "ls
>> /var/empty" can take just as long.  This length of time is variable
>> but is usually much longer than expected.
>>
>> Does anyone have any suggestions for helping to figure out what is
>> taking a long time?
>
> Hello,
>
> If "top -m io -o total" doesn't reveal what is using the disks,

Nothing unexpected here though I will pay attention during the slow times.

> I've had good experiences with the following dtrace script,
> you'd need to build dtrace support in your kernel though:

> vfsstat.d https://forums.freebsd.org/showpost.php?p=182070&postcount=6

I can run this script, what output should I be looking for?

> You can also check systat -iostast 1 and check the TPS count for
> the disks. The regular (spinning) harddisk can manage 200-300 iops
> if it is a regular consumer class disk. Are you "running out" of
> iops for some reason ?

tps stays between 0 and 30 or so.  I rarely if ever seen numbers above
30 (This is a consumer grade laptop HDD).
       tty            ada0             ada1              cd0             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  25  1144  6.88   5  0.03  13.76  35  0.46   0.00   0  0.00   1  4  2  0 93




-- 
Eitan Adler