ARC size constantly shrinks, then ZFS slows down extremely

[Pawel Jakub Dawidek]

On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote:
> Attila Nagy wrote:
> >Hello,
> >
> >Pawel Jakub Dawidek wrote:
> >>On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
> >> 
> >>>Backing out this change from the 8-STABLE kernel:
> >>>http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 
> >>>
> >>>
> >>>makes it survive about half and hour of IMAP searching. Of course 
> >>>only time will tell whether this helps in the long run, but so far 
> >>>10/10 tries succeeded to kill the machine with this method...
> >>>    
> >>
> >>Could you try this patch:
> >>
> >>    http://people.freebsd.org/~pjd/patches/arc.c.4.patch
> >>  
> >It seems (after running for two days) that this fixes my problem. And 
> >I see that Kip has came out with a similar version (which I couldn't 
> >yet test, but hope that will also do).
> It seems that I was a little bit quick regarding this.
> The machine just stopped with this:
> last pid: 32358;  load averages:  0.01,  0.04,  0.12    up 2+06:33:56  
> 14:36:25
> 114 processes: 1 running, 112 sleeping, 1 zombie
> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
> Swap: 4096M Total, 15M Used, 4081M Free
> 
>  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
> 24025 root        1  44    0  3932K   992K vmwait  0   6:06  0.00% zpool
> 84190 root        1  44    0  4700K  1592K CPU1    1   4:17  0.00% top
> 99029 root        1  44    0  4132K  1212K nanslp  1   3:53  0.00% gstat
> 26317 root        1  44    0  1528K   352K piperd  1   3:38  0.00% 
> readproctitl
> 49143    125      4  45    0 12248K  3788K sigwai  0   2:50  0.00% 
> milter-greyl
> 39969 root        1  44    0  1536K   516K vmwait  0   2:50  0.00% supervise
> 40241 root        1  44    0  1536K   516K vmwait  0   2:47  0.00% supervise
> 44633 root        1  44    0  1536K   512K vmwait  0   2:43  0.00% supervise
> 43434 root        1  44    0  1536K   516K vmwait  0   2:43  0.00% supervise
> 50575 root        1  44    0  1536K   516K vmwait  0   2:42  0.00% supervise
> 45510 root        1  44    0  1536K   512K vmwait  0   2:42  0.00% supervise
> 58146     60      1  44    0   264M  8828K pfault  0   2:32  0.00% imapd
> 47526    389      6  44    0 92688K  2296K ucond   1   1:29  0.00% slapd
> 5417 root        1  44    0  9396K  1680K pfault  1   1:26  0.00% sshd
> 13147 root        1  44    0  3340K   860K vmwait  1   0:45  0.00% syslogd
> 92597 root        1  44    0  9396K  1676K pfault  1   0:39  0.00% sshd
> 26437    125      1  44    0  6924K  1700K vmwait  0   0:33  0.00% qmgr
> 
> The above top was refreshing, but every other stuff on different ssh 
> consoles (like a running zpool iostat and gstat) was frozen.
> Even top stopped when I have resized the window.

Please try Kip's patch that was committed, it changes priorities a bit,
which should help.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091008/1375ce64/attachment.pgp