Squid memory leaks in -stable using libc malloc

[Mike Harding]

There can of course be a difference between what malloc allocates and
what the application asked for.  Various malloc implementations will do
things like round allocations up to a power of 2, etc.  The application
is only aware of the memory it asked for - the OS knows how much memory
was actually allocated.  'Wasted' memory isn't an indication of a bug,
it's a characteristic of the malloc used.  It's only leaking memory if
the memory usage grows without bound, and I've never seen that with
squid, in a few years anyway.

Different malloc implementations optimize for different things - the
FreeBSD malloc, looking at the man page, appears to try to minimize page
faults.  It also looks like it uses buckets with various powers of 2. 
dlmalloc looks to have a maximum 'wastage' of 15 bytes per chunk.  There
is a good writeup of dlmalloc here:

http://g.oswego.edu/dl/html/malloc.html

Sounds like you might as well use dlmalloc with squid - squid does
generate a lot of memory allocations, and the characteristics of squid
may function better with this malloc, which is likely why they provide
it as an option.  You may want to suggest to the port maintainer that
dlmalloc be used as the default...

- Mike H.

On Sun, 2003-09-07 at 08:20, Anders Nordby wrote:
> Hi,
> 
> On Sat, Sep 06, 2003 at 08:17:19AM -0700, Mike Harding wrote:
> > Squid uses more memory than you assign to cache_mem, this is
> > documented in the Squid FAQ, section 8.  cache_mem is sort of a
> > 'suggested' value, it's normal for squid to use a lot more memory than
> > cache_mem.
> 
> I know this, and I know the contents of the FAQ. However, there's a huge
> difference between what top reports as memory used (SIZE) and what cache
> manager reports as total allocated KB. In a running process I have just
> now, top says Squid uses 1235 MB of memory, while cache manager reports
> 663 MB allocated (contents of /proc/<pid>/map attached). That's after
> running for around 10 hours. If I use dlmalloc (I do on one system), the
> numbers are 2291 MB according to top and 2221 MB according to cache
> manager after running for 30 hours. Do you see the difference? To me,
> there seems to be a leak, or at least very unfavourable results with
> phkmalloc and Squid.
> 
> BTW, in the first example I gave here, I (still) use phkmalloc with H
> configured in malloc.conf, although H did not seem to make a difference.
> 
> > If you are happy with dmalloc, though, go ahead and use it, just check
> > out the squid FAQ about memory usage.  I don't think that there is
> > anything wrong with FreeBSD's malloc, it just has different
> > performance characteristics.
> 
> I agree that there doesn't have to be something technically wrong with
> (phk)malloc. However, if there is, someone may have an interest in
> following this up (tjr@ seems to be - I talked to him on IRC about
> this).
> 
> Cheers,