Multiple malloc(3)s (was Re: HEADS DOWN)

[Jason Evans]

Robert Watson wrote:
> I have to admit, 
> however, that I do wish we'd retained an easy switch to let us go back 
> and forth between phkmalloc and jemalloc for comparison purposes.

It is pretty easy to go back and forth for comparison purposes, because 
revision 1.92 of src/lib/libc/stdlib/malloc.c is a source-compatible 
version of phkmalloc.  You can update just that one file and get a libc 
with phkmalloc instead of jemalloc.  I continue to make use of this for 
performance testing purposes, and it remains a sound strategy for 
diagnosing possible regressions.

In my opinion, keeping only one malloc implementation in the base system 
was the right approach.  Doing so put a lot of extra pressure on 
jemalloc to handle all conceivable edge cases, rather than punting here 
and there.  (jemalloc underwent a massive redesign after it was 
initially committed, so there is physical evidence of how things could 
have gone without this pressure.)  Giving users a choice between 
"predictable low memory usage and horribly slow multi-threaded 
performance", versus, "unpredictable memory usage and fast 
multi-threaded performance" puts users in a bad spot, because they have 
to choose the lesser of two evils.

Jason