mmap(2) segaults with certain len values and MAP_ANON|MAP_FIXED

[Robert Watson]

On Wed, 21 Oct 2009, Alexander Best wrote:

> this code serves only one purpose: to trigger a segfault. i don't use the 
> code for any other purpose. i was under the impression that mmap() should 
> either succeed or fail (tertium non datur). mmap's manual doesn't say 
> anything about mmap() causing segfaults.

Have you tried ktracing the application?  I think you'll find that mmap(2) 
system call succeeded fine, and that the segfault comes from attempting to 
execute the address in libc on return to userspace, as a result of libc not 
being at that address anymore (since you removed its mapping).  You can use 
procstat -v to inspect address space use by processes, but as a general rule 
you don't want to pass anything other than an address of 0x0 to mmap(2) unless 
you're very carefully managing the address space of the process.  Many 
userspace libraries are involved in using that address space, but especially 
the runtime linker which begins execution in userspace when a binary is 
started.

Robert N M Watson
Computer Laboratory
University of Cambridge