mmap(2) MAP_FIXED isn't thread-safe (+testcase)

[Roman Divacky]

On Sun, Aug 05, 2007 at 04:56:46PM +0200, Tijl Coosemans wrote:
> Hi all,
> 
> While investigating ports/115092 and other reports of seemingly random
> page faults when running Wine, I think I've found the cause to be mmap
> not being thread-safe when MAP_FIXED is used. It causes mmap(MAP_FIXED)
> to return -1(ENOMEM) sometimes when it shouldn't, but also to return an
> address with wrong protections, hence the protection faults occuring.
> 
> Attached is a test program that shows this. It runs two threads. The
> first mmap()'s a region, starts a second thread and then goes in a loop
> calling mmap(PROT_WRITE,MAP_FIXED) on that region, essentially
> replacing that mapping. This is basically what rtld does to map an ELF
> object for instance when dlopen(3) is called. The second thread tries
> to steal the mapping from the first by calling mmap(PROT_NONE) in a
> loop. After a while the program segfaults when the first thread tries
> to write to the mapped region.
> 
> Some lines are commented out. If you remove the commenting, I hit on
> the case where mmap(MAP_FIXED) returns -1.
> 
> The problem is in sys/vm/vm_mmap.c:vm_mmap(). In case of MAP_FIXED
> first vm_map_remove() is called and then later vm_map_find(). This
> would need some locking, but I don't know which lock or how to approach
> this, so can somebody have a look at this?

man, you are fantastic! this www.vlakno.cz/~rdivacky/tijl.patch
simple/stupid patch makes flash9 work. this is of course wrong solution
but can be used as a proof of concept.. I wonder what else is broken by
the MPunSAFEness of the mmap() MAP_FIXED.

thnx a lot for investigating this!

roman