SpamAssassin/Perl eating enormous amounts of memory?

[Mel]

On Tuesday 03 March 2009 07:26:02 Andrew Moran wrote:
> On Mar 3, 2009, at 8:01 AM, Andrew Moran wrote:
> > In Mar 3, 2009, at 7:40 AM, Mel wrote:
> >> On Monday 02 March 2009 16:21:53 Andrew Moran wrote:
> >>>> What's even weirder is that the process gets that far. Did you play
> >>>> with
> >>>> kern.maxdsiz loader tuneable?
> >>>> If so, set it lower, so you can at least have the machine in a
> >>>> usable state at
> >>>> all times. 4G should be enough for any process and should give
> >>>> enough time
> >>>> for you to spot the leak and get a ktrace.
> >>>
> >>> Nope, I haven't tweaked any kernel settings, just using the generic
> >>> DEFAULT amd64 kernel.   I've been way about tweaking settings
> >>> because
> >>> I don't fully understand what the 'correct' values for my setup are.
> >>
> >> Could you show kenv kern.maxdsiz and if unset limits -H -d? Looks
> >> like it's
> >> 32G on my 6.x amd64, in which case setting it is a good idea.
> >> echo 'kern.maxdsiz="8G"' >> /boot/loader.conf
> >> echo 'kern.defdsiz="4G"' >> /boot/loader.conf
> >
> > Thank you for helping me.     The earlier suggestion of toggling
> > Perl Malloc didn't work, nor did syncing sources and installing new
> > kernel and new world which is what I did lsat night.
> >
> > As for kernel settings, I don't have anything in my loader.conf,
> > and I'm not entirely sure how to show things that aren't sysctls, as
> > this one doesn't seem to be:
> >
> > celebrian# sysctl -a | grep kern.maxdsiz
> > celebrian# sysctl -a | grep kern.defdsiz
> > celebrian#
>
> Ok sadly that didn't seem to do much:
>
> celebrian# cat /boot/loader.conf
> kern.maxdsiz="8G"
> kern.defdsiz="4G"
> celebrian#

Can you show limits -H -d?

> I rebooted, but still see my memory being chewed up.   Almost
> immediately after booting, one of my spamassassin processes spun out
> of control.  Here's the top with it eating 16 gigs of memory:
>
>    PID    UID    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
> COMMAND
>   1611   1030      1  68    0 15062M   818M CPU2   2   0:44 20.65%
> perl5.8.9

At this point, have your cd to a partition large enough to hold a few 100 
megs, and type:
ktrace -p 1611

where 1611 is the PID of the perl process. You may want to be a bit earlier 
then this point. After a few seconds, type ktrace -C. Then kdump|less. There 
should be plenty of allocations there (*alloc* functions).

>
> and from dmesg:
>
> <snip>
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(3): failed
> swap_pager_getswapspace(16): failed
> pid 1611 (perl5.8.9), uid 1030, was killed: out of swap space
>
>
> The computer gets pretty unresponsive during this, unless swapoff my
> swap, then the process eats the memory faster and gets killed
> faster.   Did I type in the wrong variables?

Nope, I'm now reading up myself, I would think this works on amd64. You can 
see defaults in /boot/defaults/loader.conf.

Maybe it only works on what top shows as RES memory (truely used memory, so 
overallocation is possible). In that case, set kern.defdsiz to 512M, perl 
should bug out earlier and probably doesn't affect normal operations (busy 
MySQL server might).

I do hope this feature isn't obsolete on amd64, I kinda like knowing there's a 
guard against my programming errors ;)

-- 
Mel

Problem with today's modular software: they start with the modules
    and never get to the software part.