Question about process rlimits

[Matthew Fleming]

On Thu, Dec 2, 2010 at 12:51 PM, Andrew Duane <aduane at juniper.net> wrote:
>
> I've been poking at some bugs we have around pushing user memory to/past the limits of our box, and decided to try seeing what happens on a stock FreeBSD system (7.1 in this case).
>
> Basically I have a program that mallocs big memory chunks and zeros them to consume both physical and virtual memory. I had expected the program to stop malloc'ing when brk() reaches the process' RLIMIT_DATA (512MB cur and max). It didn't. It happily malloc'd many gigabytes of memory until I stopped it.
>
> On our 6.2 based product boxes, RLIMIT_DATA correctly stops the malloc from continuing, just like the manuals say.
>
> Am I missing something?

Starting on FreeBSD 7, the default malloc(3) implementation uses
mmap(2), not brk(2) to manage its address space.

There are two ways to deal with this:

1) edit /etc/malloc.conf and add the 'D' option to force malloc to use
sbrk(2).  I haven't tried this one.

2) limit the total virtual memory allowed by a process, RLIMIT_VMEM.
This is what we used when migrating from FreeBSD 6 to 7.

Cheers,
matthew