sbrk(2) broken

[Jason Evans]

Poul-Henning noticed today that xchat fails to start if malloc uses sbrk 
internally.  This failure happens during the first call to malloc, with 
the following message:

Fatal error 'Can't allocate initial thread' at line 335 in file 
/usr/src/lib/libthr/thread/thr_init.c (errno = 12)

This can be worked around with MALLOC_OPTIONS=dM .

The problem does not appear to be specific to jemalloc; I reverted 
src/lib/libc/stdlib/malloc.c to revision 1.92 (last phkmalloc revision), 
which also uses sbrk, and the failure mode is the same.

The failure occurs on both i386 and amd64.  It appears that sbrk(0) 
returns an address that is in the address range normally used by mmap. 
So, the first call to sbrk with a non-zero increment is fantastically 
wrong.  On i386 (ktrace output):

   1013 xchat    CALL  break(0x28200000)
   1013 xchat    RET   break -1 errno 12 Cannot allocate memory

On amd64 (truss ouput):

   break(0x800900000)  ERR#12 'Cannot allocate memory'

sbrk is not a true system call, so it seems like the problem should have 
something to do with the _end data symbol.  I looked at it in gdb though 
and never saw an unreasonable value, despite bogus sbrk(0) results.  I 
do not know offhand how to get the addresses of .minbrk and .curbrk 
(register inspection within gdb while stepping through sbrk?), which are 
what sbrk actually uses (see src/lib/libc/amd64/sys/sbrk.S).  Perhaps 
the loader isn't initializing them correctly...

I am quite pressed for time at the moment, and cannot look into this in 
any more detail for at least a couple of weeks.  If anyone knows what 
the problem is, please let me know.

Thanks,
Jason