ARC size constantly shrinks, then ZFS slows down extremely

[Artem Belevich]

I've tested with Kip's patch -- no lockups so far.

--Artem



On Thu, Oct 8, 2009 at 9:07 AM, Pawel Jakub Dawidek <pjd at freebsd.org> wrote:
> On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote:
>> Attila Nagy wrote:
>> >Hello,
>> >
>> >Pawel Jakub Dawidek wrote:
>> >>On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>> >>
>> >>>Backing out this change from the 8-STABLE kernel:
>> >>>http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>> >>>
>> >>>
>> >>>makes it survive about half and hour of IMAP searching. Of course
>> >>>only time will tell whether this helps in the long run, but so far
>> >>>10/10 tries succeeded to kill the machine with this method...
>> >>>
>> >>
>> >>Could you try this patch:
>> >>
>> >>    http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>> >>
>> >It seems (after running for two days) that this fixes my problem. And
>> >I see that Kip has came out with a similar version (which I couldn't
>> >yet test, but hope that will also do).
>> It seems that I was a little bit quick regarding this.
>> The machine just stopped with this:
>> last pid: 32358;  load averages:  0.01,  0.04,  0.12    up 2+06:33:56
>> 14:36:25
>> 114 processes: 1 running, 112 sleeping, 1 zombie
>> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
>> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
>> Swap: 4096M Total, 15M Used, 4081M Free
>>
>>  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>> 24025 root        1  44    0  3932K   992K vmwait  0   6:06  0.00% zpool
>> 84190 root        1  44    0  4700K  1592K CPU1    1   4:17  0.00% top
>> 99029 root        1  44    0  4132K  1212K nanslp  1   3:53  0.00% gstat
>> 26317 root        1  44    0  1528K   352K piperd  1   3:38  0.00%
>> readproctitl
>> 49143    125      4  45    0 12248K  3788K sigwai  0   2:50  0.00%
>> milter-greyl
>> 39969 root        1  44    0  1536K   516K vmwait  0   2:50  0.00% supervise
>> 40241 root        1  44    0  1536K   516K vmwait  0   2:47  0.00% supervise
>> 44633 root        1  44    0  1536K   512K vmwait  0   2:43  0.00% supervise
>> 43434 root        1  44    0  1536K   516K vmwait  0   2:43  0.00% supervise
>> 50575 root        1  44    0  1536K   516K vmwait  0   2:42  0.00% supervise
>> 45510 root        1  44    0  1536K   512K vmwait  0   2:42  0.00% supervise
>> 58146     60      1  44    0   264M  8828K pfault  0   2:32  0.00% imapd
>> 47526    389      6  44    0 92688K  2296K ucond   1   1:29  0.00% slapd
>> 5417 root        1  44    0  9396K  1680K pfault  1   1:26  0.00% sshd
>> 13147 root        1  44    0  3340K   860K vmwait  1   0:45  0.00% syslogd
>> 92597 root        1  44    0  9396K  1676K pfault  1   0:39  0.00% sshd
>> 26437    125      1  44    0  6924K  1700K vmwait  0   0:33  0.00% qmgr
>>
>> The above top was refreshing, but every other stuff on different ssh
>> consoles (like a running zpool iostat and gstat) was frozen.
>> Even top stopped when I have resized the window.
>
> Please try Kip's patch that was committed, it changes priorities a bit,
> which should help.
>
> --
> Pawel Jakub Dawidek                       http://www.wheel.pl
> pjd at FreeBSD.org                           http://www.FreeBSD.org
> FreeBSD committer                         Am I Evil? Yes, I Am!
>