perl 5.8 and malloc (and mod_perl)

[Mike Harding]

(much of the following is, of course, FreeBSD specific, if you found
this via Google)

Note that perl 5.8+ has "safe signals" which defer the signal processing
until the next perl opcode - apparently entering the signal information
into a queue.  This should allow non-perl malloc to be used.  See
PERLIPC(1) for 5.8.2 - it discusses this in detail.
Note that Perl signals were never really -ever- safe before, as far as I
can tell, unless you went to great lengths to make sure that no
non-reentrant code was called - and the Perl system code is/was
apparently not very reentrant.

So, I think that I am (much) better off using system malloc with
mod_perl, at least if you do log rotation with grace restarts.  If you
don't use mod_perl then either should work, and the perl_malloc should
be faster, at least in some edge cases, at the expense of some memory.

There was some discussion of using system malloc with Perl degenerating
into n^2 performance if buffers were extended 1 unit at a time, as unzip
apparently does, as perl would call realloc n times - the Perl malloc
use(s/d) exponential reallocation to avoid this.  This could be fixed in
the Perl system source code so that both systems work the same, but the
edge goes to Perl's malloc for speed for now.

I'm just trying to give mod_perl users this info because, at least for
the DSO mod_perl case, using the system malloc appears to fix a massive
memory leak during an apache graceful restart, which is a common way to
rotate Apache log files.  For other uses, Perl's malloc is probably
superior.

It may also be possible to stop the leak by building Apache or Perl some
other way so that Perl's malloc can be used, but I'm not interested in
looking at this unless some problems arise from using the system
malloc.  I don't think that the speed problem mentioned above will
affect systems that I work on - the systems, at least initially, appear
as fast or faster than before I used the system malloc with
Perl/mod_perl.

Note that this should not affect people who don't use graceful restarts
for log rotation, although it does appear that the httpd process is much
smaller (1/2 or 2/3 of the size) if the system malloc is used.  This may
be a genuine leak, or just how Perl uses memory or appears to use
memory, but I am happy to see Mason running over mod_perl with a 14 meg
httpd.

I found this interesting tidbit here:

http://www.nntp.perl.org/group/perl.perl5.summary/61

"Using perl's malloc() on FreeBSD produce spurious failures in the
threading stress tests. Consequently, Jarkko reverted perl to use
FreeBSD's malloc(), in spite of its slowness (see Dan Kogai's analysis
in last week summary.)"

On Fri, 2004-01-16 at 03:28, Alexander Leidinger wrote:
> On Thu, 15 Jan 2004 15:31:28 -0800 (PST)
> Mike Harding <mvh at ix.netcom.com> wrote:
> 
> > 
> > (This is a copy of a mail already sent to the perl5.8 port maintainer,
> > I wanted to make sure that this info is available to those who run
> > into the same issue)
> > 
> > I have a few client sites where recently (since a perl upgrade)
> > Mason/mod_perl based httpd processes had 'swollen' to huge sizes -
> > 300+ meg.  After a lot of coffee and googling and such, it looks like
> > the culprit is somehow related to the use of perl's malloc.  When
> > apache gets a graceful restart signal (for rotating logs) the root
> > apache process (and all it's children) grow by 6 megs or so.  I'm
> > rotating the logs every day at midnight so it took a while for me to
> > notice.
> 
> I don't know if the situation changed with perl 5.8, but with 5.6 you
> have to compile with perls malloc if your application makes use of
> signals, else you will see heisenbugs. It isn't safe to call malloc in a
> signal handler -- see sigaction(2) -- and a perl program which makes
> use of signals may just do this.
> 
> Bye,
> Alexander.