malloc options

[Giorgos Keramidas]

On Sun, 27 Jul 2008 21:55:39 -0400 (EDT), vogelke+software at pobox.com (Karl Vogel) wrote:
>>> On Sat, 26 Jul 2008 17:36:35 -0700,
>>> Doug Hardie <bc979 at lafn.org> wrote:
>
> D> The program has worked under considerable load for many years with
> D> versions 3.7 to 6.2.  Problems only occur with 7.0.  The program is
> D> quite complex and big.  It uses probably hundreds of mallocs in a
> D> typical use.  The problems only occur reasonably randomly and only
> D> under quite heavy load.  The developer is looking into it, but the
> D> problem only occurs on FreeBSD 7.0, not any other Unix systems.  In
> D> the meantime I am losing money because of it.
>
>>> On Sun, 27 Jul 2008 05:03:58 +0300,
>>> Giorgos Keramidas <keramida at ceid.upatras.gr> said:
>
> G> While that's understandable, the current malloc() has undergone
> G> quite extensive testing by Jason Evans and a lot of people who use
> G> it in FreeBSD 7.X or later.  Its ability to expose bugs in this way
> G> was deemed important enough that it is now used by other projects
> G> too.
>
> I ran into a similar problem with the BSD allocator running under
> heavy load that didn't happen under any Solaris or Linux system I
> used.  I finally fixed it by using Doug Lea's malloc just for this one
> application:
>
>      http://shell.siscom.net/~vogelke/Software/Languages/C/Libraries/malloc/
>
> This was under FreeBSD 6.*, but it might provide another data point if
> you want to give it a try.

I'm not sure how similar the two problems are.  I quite frankly know
_very_ little of what the original problem was, other than "I am
encountering issues where values just seem to arbitrarily change".

Memory exhaustion is a potential problem with almost any sort of
allocator that fragments memory in any way, but random corruption of
user data is probably a different issue :/

If you have some sort of description of the workload that triggered the
memory exhaustion with jemalloc (the current malloc implementation in
FreeBSD), it's probably a good idea to talk to Jason Evans about it (his
email is "jasone" at FreeBSD.org).  He may be able to help you tune
malloc or even make changes to the system version of malloc that make it
less vulnerable to this sort of problem.