Proposed addition of malloc_size_np()

[Jason Evans]

On Mar 27, 2006, at 10:50 AM, John-Mark Gurney wrote:
> Jason Evans wrote this message on Mon, Mar 27, 2006 at 08:34 -0800:
>>
>> I can think of a few straightforward uses:
>
> [...]
>
>> 2) [...]
>
> This use case is incorrect...

Fair enough.  That leaves three use cases.

>> 4) Porting code from Linux.  Case in point: Xara Xtreme, currently  
>> being
>> ported by Vasil Dimov.  At the moment, he has to use dlmalloc.
>
> sad to say, but we seem to always inherit Linux's warts in the name of
> portability.. :(  I'd say adopt a method similar to the ports tree,  
> once
> we have 10+ ports depending upon it, then we can consider this.. but
> importing this function for a single program seems a bit excessive..

malloc_usable_size() is *not* a Linux wart.  Instead, it mostly[1]  
addresses a fundamental flaw in the malloc(3) API.  We must go back  
much farther than Linux if we really want to start disparaging malloc 
(3).  Consider that the allocator must know how large allocations are  
so that it can do the right thing for realloc() and free().  Consider  
also that the application must know how large its objects are so that  
it can stay in bounds.  That means that such information has to be  
stored in two places.  Following is a much more reasonable API that  
doesn't burden the allocator with information that the application  
should be able to provide:

void *
malloc(size_t size);

void *
calloc(size_t size);

void *
realloc(void *ptr, size_t size, size_t old_size);

void
free(void *ptr, size_t size);

That's not what we have though, and we're certainly not going to  
completely redesign malloc(3) at this point.  So, instead, we're  
faced with trying to patch things up as well as is feasible without  
breaking compatibility.  Let's re-visit Poul-Henning's suggestion,  
which involves a parallel API that exposes the internal sizes to the  
application.  The following is not identical to what he proposed, but  
it has the same semantics:

void *
malloc_np(size_t *size);

void *
calloc_np(size_t *size);

void *
realloc_np(void *ptr, size_t *size, size_t *old_size);

void *
memalign_np(size_t alignment, size_t *size);

void *
free_np(void *ptr, size_t *size);

This serves much the same purpose as the idealized malloc(3) API I  
outlined above but with the advantage that it can co-exist with the  
current malloc(3) API.  From an engineering perspective, this is  
probably actually a better solution than malloc_usable_size(), but 1)  
it's a much broader API, 2) it's nothing like what any other OSes  
have done, and 3) in practice it really doesn't gain us anything over  
malloc_usable_size() (in fact it doesn't solve the problem of porting  
Linux and OS X software).

For my own personal needs, I really don't care whether we add  
malloc_usable_size() or the broader API described above.  However, I  
do feel like the latter is an over-engineered solution to the  
problem.  To me, this isn't about reinventing a perfect wheel -- it's  
about filing off a rough edge on the stone wheel we're stuck with.

Jason

[1] In a slightly more ideal world, we would add malloc_size()  
instead of malloc_usable_size(), which would return the size that was  
passed to malloc()/calloc()/realloc().  However, that would impose  
implementation constraints on the allocator that would have far- 
reaching implications.  I don't want to get into the details here,  
but suffice it to say that requiring support for malloc_size() would  
require a lot of extra space overhead for many allocator designs,  
including for phkmalloc and jemalloc.