Multi-zone malloc(9)

[mdf at FreeBSD.org]

On Thu, Jul 22, 2010 at 10:41 AM, Kostik Belousov <kostikbel at gmail.com> wrote:
> On Thu, Jul 22, 2010 at 09:54:51AM -0700, mdf at freebsd.org wrote:
>> Occasionally we run into use-after-free and malloc'd buffer overrun
>> scenarios.  When this happens it can be rather difficult to determine
>> what code is at fault, since e.g. every 64 byte allocation, regardless
>> of malloc type, comes from the same UMA zone.  This means that an
>> overflow in M_TEMP will affect M_DEVBUF, etc.  Adding multiple uma
>> zones for each bucket size means that we can hash on the malloc type's
>> shortdesc field so that there are fewer collisions and misused memory
>> from one malloc type only affects a subset of other malloc types.
>> Varying the hash means that, with several crashes due to memory stomp,
>> a single malloc type can usually be determined as the culprit.  If the
>> bug isn't obvious from inspection at this point, MemGuard will help
>> catch the offender.
>>
>> The patch at:
>>
>>    http://people.freebsd.org/~mdf/multizone_malloc.patch
>>
>> implements an optional multi-zone malloc(9).  By default there is a
>> single zone, and MALLOC_DEBUG_MAXZONES can be specified in the kernel
>> configuration file.  A ddb function will print all the malloc types
>> that have a hash collision with the specified type.
>>
>> A few questions for -arch@:
>>
>>  - We found this very useful at Isilon.  Should this go into CURRENT?
>>
>>  - Should this be on by default for GENERIC?  The memory overhead of 8
>> uma zones per malloc allocation size shouldn't be very large.
>>
>>  - would a __FreeBSD_version bump be needed since the malloc_internal
>> type is known by user-space?
>
> Can you quantify the overhead, both in CPU time and memory usage terms
> ? I would much prefer to have debug and non-debug kernels to run
> similar code, in other words, can the multizone allocation be enabled
> unconditionally ?

CPU usage should be lost in the noise, since the subzone hash class is
computed once at boot.

Memory overhead is a bit hard to quantify, since with multiple zones
there are more partially allocated pages in each malloc size.  Worst
case would be, I think, 1 page per CPU per multi-zone per allocation
size, so for 4 CPUs with 4k pages, that would be 4*8*9 or a little
over 1MB.

The actual number of multi-zones is also a TUNABLE, so the same code
could be built for debug and non-debug, and a loader.conf change would
enable the actual use of multiple zones during boot.

Thanks,
matthew