malloc does not return null when out of memory

[Chuck Swiger]

Peter Jeremy wrote:
> On 2003-Jul-24 04:35:14 +0200, Matthias Buelow <mkb at mukappabeta.de> wrote:
 > Cleanly recovering in all cases when there is no additional memory
 > available is a very hard problem.  Most of the suggested fixes
 > generally rely on the kernel providing a warning when the VM system
 > is under stress but can still satisfy some requests.  Unfortunately,
 > I don't think any of them included the necessary patches.
[ ... ]

I agree with this statement: handling system-wide resource shortages is a 
tricky, failure-prone, and dangerous environment to try and program for.  I'm 
very familar with the Mach project, Mach memory management, and task/thread 
semantics; am I okay using that terminology, or am I going to confuse myself (or 
other people :-)?

For instance, I know that FreeBSD's VM system includes some degree of 
copy-on-write semantics, but I don't believe there is support for user-mode 
pagers, right?  Mach uses message-passing, heavily, and supported both 
kernel-mode and user-mode paging algorithms; it would let you set VM_NORMAL, 
VM_SEQUENTIAL, VM_ANOLMALOUS, or your own paging algorithm.

There is a difference between "I want another page of VM" and "my process wants 
another frame of physical memory", but for those tasks using VM backing store 
which they can flush and make available to the rest of the system, being able to 
free up physical RAM can help make memory available for error-handling or 
whatever in the faulting task.  In other words, Mach messaging served as a 
mechanism for the kernel to let tasks (processes) know that the VM system was 
under stress.

	--

Mach relies heavily on COW semantics and lazy allocation of resources, and I 
fully understand the benefits of VM overcommit in general, but Mach let you 
control allocation of VM very explicitly if you needed to.  Mach also did some 
clever optimizations with regard to zero-filling pages: mapped but unallocated 
regions of memory read as zero, and the Mach exception handler which trapped 
writing to mapped but unreferenced memory, or memory shared copy-on-write, and 
would only allocate more VM if the write actually changed something.  Writing 
zeros to an unallocated page did not cause the page to be allocated.

Which is why my patch used SOME_JUNK, of course.  :-)

[ This is in answer to another message by Jeremy, which said: ]
 > This comes up every time this thread starts.  As I said before - read
 > the archives.  If you think you have a solution that works and avoids
 > at least the larger pitfalls (see the archives), you need to provide
 > patches (or show a willingness to pay someone else to write the code).

Let me criticise the proposed patch I made:

1) To force pages to be allocated, you only need to write one byte per page: 
mine writes at least one byte per malloc().  malloc_pages() could use the 
existing stride algorithm, and malloc_bytes() should do something else appropriate.

2) I use malloc_pagesize rather than, what, getpagesize(), or god only knows 
what else is appropriate.

3) What to do with regard to error-handling, as discussed above and elsewhere.

-- 
-Chuck