malloc does not return null when out of memory

[Don Lewis]

On 24 Jul, Chuck Swiger wrote:
> Muttley wrote:
>> Yes, I thought briefly about something like this.
>> 
>> Then I thought 'there's a race condition'.
> 
> Where?  The FreeBSD implementation is wrapped in a THREAD_LOCK()...?
> 
>> Then I realised that other processes might not link against this malloc.
> 
> Perhaps.
> 
>> Then I realised the race condition doesn't even matter; processes will 
>> still be killed, as the kernel doesn't care that you're still in 
>> malloc() when the overcommitted memory is touched, it just knows you've 
>> touched it and there's no actual memory there. This will result in far 
>> more processes being killed. I believe that's a bad thing.
> 
> Someone stated that it was a problem that malloc() returned pointers to virtual 
> address space that had been mapped but not allocated.  This patch does not 
> guarantee that malloc() will return, but, if malloc() does returns a pointer, 
> using the memory being pointed to will refer to memory that is allocated.
> 
> As Barny Wolff said:
>  > Won't this merely die in malloc, not return 0?
> 
> True.  This isn't a perfect solution, but given the choice between:
> 
> 1) malloc(LOTS) returning a pointer, and then sometime later the program dies 
> with a bus error when using that memory because no more VM is available, or
> 
> 2) malloc(LOTS) causing an immediate failure in malloc(),
> 
> ...choice #2 appears to be significantly better.
> 
> Figuring out what went wrong from a coredump or backtrace for #2 when the signal 
> happens in malloc() should be obvious; determining why the program crashed in 
> the middle of referencing memory in some large buffer is potentially misleading.
> 
> Programs which take care to preallocate regions of memory they need before they 
> start doing a transaction or some other operation that needs to be atomic would 
> also prefer #2; the patch I proposed could have a beneficial impact on data 
> integrity for such programs.

I believe that the problem isn't confined to dynamically allocated
memory.  I think it is also possible to run into problems when accessing
a large static array, or even when not accessing memory at all if the
kernel wants to free up some swap space and the process in question is
sufficently large.