two 4GB mallocs => SEGV

[James R. Van Artsalen]

David O'Brien wrote:

>malloc.c:map_pages() calls brk(2) and this is where it goes to la-la land.
>
>  
>
The brk() kernel call is probably failing due to ulimit being exceeded 
and not anything mysterious.

A few months ago I posted this bug in the libc brk(2) code - the stack 
is not balanced if the kernel returns an error.  I'm not running current 
code at the moment but see if you brk.S has a stack issue at the err: 
label.  Stick in this pop if so and report if malloc(3c) then returns 
NULL instead of crashing, then up your ulimit and try again and see if 
all works without error.

--- lib/libc/amd64/sys/brk.S.~1~        Sat May 24 12:35:23 2003
+++ lib/libc/amd64/sys/brk.S    Fri Apr  9 02:02:22 2004
@@ -78,6 +78,7 @@
        popq    %rdi
        ret
 err:
+       popq    %rdi
 #ifdef PIC
        movq    PIC_GOT(HIDENAME(cerror)),%rdx
        jmp     *%rdx