ARC size constantly shrinks, then ZFS slows down extremely

[Attila Nagy]

Attila Nagy wrote:
> Hello,
>
> Pawel Jakub Dawidek wrote:
>> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>>  
>>> Backing out this change from the 8-STABLE kernel:
>>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 
>>>
>>>
>>> makes it survive about half and hour of IMAP searching. Of course 
>>> only time will tell whether this helps in the long run, but so far 
>>> 10/10 tries succeeded to kill the machine with this method...
>>>     
>>
>> Could you try this patch:
>>
>>     http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>>   
> It seems (after running for two days) that this fixes my problem. And 
> I see that Kip has came out with a similar version (which I couldn't 
> yet test, but hope that will also do).
It seems that I was a little bit quick regarding this.
The machine just stopped with this:
last pid: 32358;  load averages:  0.01,  0.04,  0.12    up 2+06:33:56  
14:36:25
114 processes: 1 running, 112 sleeping, 1 zombie
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
Swap: 4096M Total, 15M Used, 4081M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
24025 root        1  44    0  3932K   992K vmwait  0   6:06  0.00% zpool
84190 root        1  44    0  4700K  1592K CPU1    1   4:17  0.00% top
99029 root        1  44    0  4132K  1212K nanslp  1   3:53  0.00% gstat
26317 root        1  44    0  1528K   352K piperd  1   3:38  0.00% 
readproctitl
49143    125      4  45    0 12248K  3788K sigwai  0   2:50  0.00% 
milter-greyl
39969 root        1  44    0  1536K   516K vmwait  0   2:50  0.00% supervise
40241 root        1  44    0  1536K   516K vmwait  0   2:47  0.00% supervise
44633 root        1  44    0  1536K   512K vmwait  0   2:43  0.00% supervise
43434 root        1  44    0  1536K   516K vmwait  0   2:43  0.00% supervise
50575 root        1  44    0  1536K   516K vmwait  0   2:42  0.00% supervise
45510 root        1  44    0  1536K   512K vmwait  0   2:42  0.00% supervise
58146     60      1  44    0   264M  8828K pfault  0   2:32  0.00% imapd
47526    389      6  44    0 92688K  2296K ucond   1   1:29  0.00% slapd
 5417 root        1  44    0  9396K  1680K pfault  1   1:26  0.00% sshd
13147 root        1  44    0  3340K   860K vmwait  1   0:45  0.00% syslogd
92597 root        1  44    0  9396K  1676K pfault  1   0:39  0.00% sshd
26437    125      1  44    0  6924K  1700K vmwait  0   0:33  0.00% qmgr

The above top was refreshing, but every other stuff on different ssh 
consoles (like a running zpool iostat and gstat) was frozen.
Even top stopped when I have resized the window.