malloc(3) ignores RLIMIT_DATA

[Jason Evans]

Igor Sysoev wrote:
> malloc(3) in FreeBSD 7 uses mmap() (then sbrk() on 32-bit platform)
> and ignores RLIMIT_DATA. FreeBSD 8's malloc() can be configured
> to use sbrk() only ("Dm"), but default setting is "DM".

I plan to merge these changes to RELENG_7 shortly after FreeBSD 7.0 is 
released.  As for the default, "DM", there is a strong argument that 
malloc() should try hard to succeed, and only fail prematurely if the 
user has added constraints via resource limits.  Unfortunately, 
RELENG_7_0 will not have the proper MALLOC_OPTIONS support to disable 
mmap()-based allocation, but the release engineers prudently deemed the 
necessary malloc(3) changes too risky so late in the release cycle.

> As sbrk() is less preferable because of framentation and race conditions,
> why not to create mmap() flag MMAP_DSS to check RLIMIT_DATA and to use it
> in malloc(3) ?

There has been general agreement among the people I've discussed this 
issue with that the correct solution is to add a separate resource limit 
for anonymously mapped memory, which would provide capabilities similar 
to what your suggestion would provide.

Jason