Squid memory leaks in -stable using libc malloc

[David Schultz]

On Sun, Sep 07, 2003, Anders Nordby wrote:
> Hi,
> 
> On Sat, Sep 06, 2003 at 08:17:19AM -0700, Mike Harding wrote:
> > Squid uses more memory than you assign to cache_mem, this is
> > documented in the Squid FAQ, section 8.  cache_mem is sort of a
> > 'suggested' value, it's normal for squid to use a lot more memory than
> > cache_mem.
> 
> I know this, and I know the contents of the FAQ. However, there's a huge
> difference between what top reports as memory used (SIZE) and what cache
> manager reports as total allocated KB. In a running process I have just
> now, top says Squid uses 1235 MB of memory, while cache manager reports
> 663 MB allocated (contents of /proc/<pid>/map attached). That's after
> running for around 10 hours. If I use dlmalloc (I do on one system), the
> numbers are 2291 MB according to top and 2221 MB according to cache
> manager after running for 30 hours. Do you see the difference? To me,
> there seems to be a leak, or at least very unfavourable results with
> phkmalloc and Squid.

phkmalloc satisfies allocations smaller than the system page size
from power-of-two buckets, and larger allocations are rounded up
to a multiple of the page size.  While simple, this scheme works
poorly for some applications.  For example, postgresql makes some
allocations that are slightly larger than a power of two due to
its use of inline buffer tags.  If you use malloc(3)'s 'U' option
in combination with ktrace(1), you can get a good idea of exactly
what sizes Squid is trying to allocate.

Note also that there's a difference between heap size and physical
memory usage.  In a demand-paged virtual memory system, you can
allocate just about as much heap space as you want practically for
free; you only pay for the pages you actually touch.  A better
metric for the memory pressure that an application is exerting on
the system is its resident set size---'RES' in top(1).  phkmalloc
by default is not aggressive about returning memory to the system,
but it usually exhibits very good behavior from the point of view
of the VM system.