mmap()

[Michael Conlen]

On Nov 23, 2005, at 5:21 PM, Robert Watson wrote:

>
> On Wed, 23 Nov 2005, Michael Conlen wrote:
>
>> Sorry if this is the wrong place for this, but I haven't been  
>> getting answers elsewhere.
>>
>> I'm trying to tune the system to allow very large mmap()'s in a  
>> single process space, something on the order of 1.5 GB so I can  
>> pass very large values for -Xms and -Xmx to java. I know I had  
>> been able to do this on FreeBSD in the past but recent versions of  
>> either Java or FreeBSD aren't playing nicely.  currently..
>
> BTW, you may find it useful to use procfs to inspect the address  
> space layout of your process.  You can d:
>
>   mkdir /proc
>   mount -t procfs proc /proc
>   cd /proc/pid
>   dd if=map of=/dev/stdout bs=20k
>
> This can help you look for fragmentation of process address space,  
> among other things.

Thanks. I forgot when you mmap you have to have contiguous space... ...

0x8048000 0x8049000 0 0 0xc3c79630 r-x 1 0 0x0 COW NC vnode /tmp/foo
0x8049000 0x804a000 1 0 0xc5ad718c rw- 2 0 0x2180 NCOW NNC default -
0x804a000 0x804c000 2 0 0xc5ad718c rwx 2 0 0x2180 NCOW NNC default -
0x87f49000 0x87f69000 32 0 0xc10406b4 r-x 80 29 0x4 COW NC vnode / 
libexec/ld-elf

That looks like about 2 GB of space between the last two lines. The  
rest is as follows

0x87f69000 0x87f6a000 1 0 0xc5ae1d68 rw- 1 0 0x2180 COW NNC vnode / 
libexec/ld-elf.so.1
0x87f6a000 0x87f6f000 5 0 0xc3b4db58 rw- 2 0 0x2180 NCOW NNC default -
0x87f6f000 0x87f77000 7 0 0xc3b4db58 rwx 2 0 0x2180 NCOW NNC default -
0x87f77000 0x88041000 108 0 0xc3b3bd68 r-x 1 0 0x2180 COW NNC vnode / 
lib/libc.so.5
0x88041000 0x88042000 1 0 0xc5ad6840 r-x 1 0 0x2180 COW NNC vnode / 
lib/libc.so.5
0x88042000 0x88047000 5 0 0xc5ad418c rwx 1 0 0x2180 COW NNC vnode / 
lib/libc.so.5
0x88047000 0x8805c000 6 0 0xc5ad75ac rwx 1 0 0x2180 NCOW NNC default -
0xbfbe0000 0xbfc00000 3 0 0xc3c9718c rwx 1 0 0x2180 NCOW NNC default -

If I change from using mmap to malloc() I get the following first  
four lines show malloc() on the heap

0x8048000 0x8049000 1 0 0xc3b2c318 r-x 1 0 0x0 COW NC vnode /tmp/foo
0x8049000 0x804a000 1 0 0xc5ae3108 rw- 2 0 0x2180 NCOW NNC default -
0x804a000 0x3fc4c000 2 0 0xc5ae3108 rwx 2 0 0x2180 NCOW NNC default -
0x87f49000 0x87f69000 32 0 0xc10406b4 r-x 83 31 0x4 COW NC vnode / 
libexec/ld-elf

and it looks like a smaller mmap() shows up at the end


0x88041000 0x88042000 1 0 0xc5adcce4 r-x 1 0 0x2180 COW NNC vnode / 
lib/libc.so.5
0x88042000 0x88047000 5 0 0xc5ad6420 rwx 1 0 0x2180 COW NNC vnode / 
lib/libc.so.5
0x88047000 0x8805b000 5 0 0xc5ae1ce4 rwx 3 0 0x2180 NCOW NNC default -
0x8805b000 0xbfb5b000 0 0 0xc5ae1ce4 --- 3 0 0x2180 NCOW NNC default -
0xbfb5b000 0xbfb5c000 1 0 0xc5ae1ce4 rwx 3 0 0x2180 NCOW NNC default -

I presume the stack is coming up from the bottom.  The end looks  
about 5 MB short of 3 GB.

In all it looks like about 1 GB off the top (kernel I presume), the  
heap, followed by the libraries which IIRC are mmap()'ed coming down  
after the end of the heap and I presume the stack coming up from the  
bottom.

If I reduce the limit for the datasize locally using limit it doesn't  
seem to actually free up space for the stack. If I change loader.conf  
to reduce the datasize then the space is freed up to do the mmap().  
If I leave the max datasize and reduce the default to 1 GB I don't  
get any change in the memory map at all.

So here's the problem, I've got a DB server that needs a large  
datasize and a tomcat server which occasionally needs to use a lot of  
memory, which java allocates from a memory mapped space. Any ideas  
how to get the system to allow processes to have either/or?

--
Michael Conlen