ARC size constantly shrinks, then ZFS slows down extremely
Attila Nagy
bra at fsn.hu
Mon Oct 12 07:56:12 UTC 2009
Pawel Jakub Dawidek wrote:
> On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote:
>
>> Attila Nagy wrote:
>>
>>> Hello,
>>>
>>> Pawel Jakub Dawidek wrote:
>>>
>>>> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>>>>
>>>>
>>>>> Backing out this change from the 8-STABLE kernel:
>>>>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>>>>>
>>>>>
>>>>> makes it survive about half and hour of IMAP searching. Of course
>>>>> only time will tell whether this helps in the long run, but so far
>>>>> 10/10 tries succeeded to kill the machine with this method...
>>>>>
>>>>>
>>>> Could you try this patch:
>>>>
>>>> http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>>>>
>>>>
>>> It seems (after running for two days) that this fixes my problem. And
>>> I see that Kip has came out with a similar version (which I couldn't
>>> yet test, but hope that will also do).
>>>
>> It seems that I was a little bit quick regarding this.
>> The machine just stopped with this:
>> last pid: 32358; load averages: 0.01, 0.04, 0.12 up 2+06:33:56
>> 14:36:25
>> 114 processes: 1 running, 112 sleeping, 1 zombie
>> CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
>> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
>> Swap: 4096M Total, 15M Used, 4081M Free
>>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>> 24025 root 1 44 0 3932K 992K vmwait 0 6:06 0.00% zpool
>> 84190 root 1 44 0 4700K 1592K CPU1 1 4:17 0.00% top
>> 99029 root 1 44 0 4132K 1212K nanslp 1 3:53 0.00% gstat
>> 26317 root 1 44 0 1528K 352K piperd 1 3:38 0.00%
>> readproctitl
>> 49143 125 4 45 0 12248K 3788K sigwai 0 2:50 0.00%
>> milter-greyl
>> 39969 root 1 44 0 1536K 516K vmwait 0 2:50 0.00% supervise
>> 40241 root 1 44 0 1536K 516K vmwait 0 2:47 0.00% supervise
>> 44633 root 1 44 0 1536K 512K vmwait 0 2:43 0.00% supervise
>> 43434 root 1 44 0 1536K 516K vmwait 0 2:43 0.00% supervise
>> 50575 root 1 44 0 1536K 516K vmwait 0 2:42 0.00% supervise
>> 45510 root 1 44 0 1536K 512K vmwait 0 2:42 0.00% supervise
>> 58146 60 1 44 0 264M 8828K pfault 0 2:32 0.00% imapd
>> 47526 389 6 44 0 92688K 2296K ucond 1 1:29 0.00% slapd
>> 5417 root 1 44 0 9396K 1680K pfault 1 1:26 0.00% sshd
>> 13147 root 1 44 0 3340K 860K vmwait 1 0:45 0.00% syslogd
>> 92597 root 1 44 0 9396K 1676K pfault 1 0:39 0.00% sshd
>> 26437 125 1 44 0 6924K 1700K vmwait 0 0:33 0.00% qmgr
>>
>> The above top was refreshing, but every other stuff on different ssh
>> consoles (like a running zpool iostat and gstat) was frozen.
>> Even top stopped when I have resized the window.
>>
>
> Please try Kip's patch that was committed, it changes priorities a bit,
> which should help.
>
My i386 machine is still alive after two days of uptime (with your
patch, it lived for about two days, so I can't say -at least now- that
it's OK).
The amd64 machine started to loose ARC memory again. See these:
http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem-week.png
http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png
Your patch was active between 7 and 9. You can see that the ARC size was
somewhat constant.
On october 9, I installed Kip's modification, and ARC size started to
decrease.
BTW, previously (before october 7) I set the arc min size to 10-15GB
(can't remember the exact value), but now it runs with the defaults
(only the max size is set):
vfs.zfs.arc_min: 3623878656
vfs.zfs.arc_max: 28991029248
As you can see, there are plenty of memory. This machine uses UFS as
well (and writes it heavily), maybe that's what affects ZFS size, by
caching a lot of stuff?
More information about the freebsd-fs
mailing list