ZFS + NFS poor performance after restarting from 100 day uptime

[Steven Hartland]

> ----- Original Message ----- 
> From: Josh Beard 
>> A snip of gstat:
>> dT: 1.002s  w: 1.000s
>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
...
>>    4    160    126   1319   31.3     34    100    0.1  100.3| da1
>>    4    146    110   1289   33.6     36     98    0.1   97.8| da2
>>    4    142    107   1370   36.1     35    101    0.2  101.9| da3
>>    4    121     95   1360   35.6     26     19    0.1   95.9| da4
>>    4    151    117   1409   34.0     34    102    0.1  100.1| da5
>>    4    141    109   1366   35.9     32    101    0.1   97.9| da6
>>    4    136    118   1207   24.6     18     13    0.1   87.0| da7
>>    4    118    102   1278   32.2     16     12    0.1   89.8| da8
>>    4    138    116   1240   33.4     22     55    0.1  100.0| da9
>>    4    133    117   1269   27.8     16     13    0.1   86.5| da10
>>    4    121    102   1302   53.1     19     51    0.1  100.0| da11
>>    4    120     99   1242   40.7     21     51    0.1   99.7| da12
>> 
>> Your ops/s are be maxing your disks. You say "only" but the ~190 ops/s
>> is what HD's will peak at, so whatever our machine is doing is causing
>> it to max the available IO for your disks.
>> 
>> If you boot back to your previous kernel does the problem go away?
>> 
>> If so you could look at the changes between the two kernel revisions
>> for possible causes and if needed to a binary chop with kernel builds
>> to narrow down the cause.
>
> Thanks for your response.  I booted with the old kernel (9.1-RC3) and the
> problem disappeared!  We're getting 3x the performance with the previous
> kernel than we do with the 9.1-RELEASE-p1 kernel:
> 
> Output from gstat:
> 
>     1    362      0      0    0.0    345  20894    9.4   52.9| da1
>     1    365      0      0    0.0    348  20893    9.4   54.1| da2
>     1    367      0      0    0.0    350  20920    9.3   52.6| da3
>     1    362      0      0    0.0    345  21275    9.5   54.1| da4
>     1    363      0      0    0.0    346  21250    9.6   54.2| da5
>     1    359      0      0    0.0    342  21352    9.5   53.8| da6
>     1    347      0      0    0.0    330  20486    9.4   52.3| da7
>     1    353      0      0    0.0    336  20689    9.6   52.9| da8
>     1    355      0      0    0.0    338  20669    9.5   53.0| da9
>     1    357      0      0    0.0    340  20770    9.5   52.5| da10
>     1    351      0      0    0.0    334  20641    9.4   53.1| da11
>     1    362      0      0    0.0    345  21155    9.6   54.1| da12
> 
> 
> The kernels were compiled identically using GENERIC with no modification.
> I'm no expert, but none of the stuff I've seen looking at svn commits
> looks like it would have any impact on this.  Any clues?

Your seeing a totally different profile there Josh as in all writes no
reads where as before you where seeing mainly reads and some writes.

So I would ask if your sure your seeing the same work load, or has
something external changed too?

Might be worth rebooting back to the new kernel and seeing if your
still see the issue ;-)

    Regards
    Steve

 Regards
 Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.