mlockall() on freebsd 7.2 + amd64 returns EAGAIN

Fri Apr 13 21:45:42 UTC 2012

On Fri, Apr 13, 2012 at 11:37:44AM -0700, Sushanth Rai wrote:
> I've attached the simple program that creates 5 threads. Following is the o/p of /proc/<pid>/map when this program is running. Note that I modified 
> sys/fs/procfs/procfs_map.c to print whether a region is wired. As you can see from this o/p, none of stack areas get wired. 
> 
> 0x400000 0x401000 1 0 0xffffff002d943bd0 r-x 1 0 0x1000 COW NC wired vnode /var/tmp/thread1
> 0x500000 0x501000 1 0 0xffffff002dd13e58 rw- 2 0 0x3100 NCOW NNC wired default -
> 0x501000 0x600000 255 0 0xffffff002dd13e58 rwx 2 0 0x3100 NCOW NNC wired default -
> 0x800500000 0x800526000 38 0 0xffffff0025574000 r-x 192 46 0x1004 COW NC wired vnode /libexec/ld-elf.so.1
> 0x800526000 0x800537000 17 0 0xffffff002d9f81b0 rw- 1 0 0x3100 NCOW NNC wired default -
> 0x800626000 0x80062d000 7 0 0xffffff002dd13bd0 rw- 1 0 0x3100 COW NNC wired vnode /libexec/ld-elf.so.1
> 0x80062d000 0x800633000 6 0 0xffffff002dd145e8 rw- 1 0 0x3100 NCOW NNC wired default -
> 0x800633000 0x800645000 18 0 0xffffff00256d71b0 r-x 63 42 0x4 COW NC wired vnode /lib/libthr.so.3
> 0x800645000 0x800646000 1 0 0xffffff002d975510 r-x 1 0 0x3100 COW NNC wired vnode /lib/libthr.so.3
> 0x800646000 0x800746000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC not-wired default -
> 0x800746000 0x80074a000 4 0 0xffffff002572a288 rw- 1 0 0x3100 COW NNC wired vnode /lib/libthr.so.3
> 0x80074a000 0x80074c000 2 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wired default -
> 0x80074c000 0x80083e000 242 0 0xffffff001cd226c0 r-x 238 92 0x1004 COW NC wired vnode /lib/libc.so.7
> 0x80083e000 0x80083f000 1 0 0xffffff002dd12000 r-x 1 0 0x3100 COW NNC wired vnode /lib/libc.so.7
> 0x80083f000 0x80093e000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC not-wired default -
> 0x80093e000 0x80095d000 31 0 0xffffff002dddc360 rw- 1 0 0x3100 COW NNC wired vnode /lib/libc.so.7
> 0x80095d000 0x800974000 23 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wired default -
> 0x800a00000 0x800b00000 256 0 0xffffff002dbd1798 rw- 1 0 0x3100 NCOW NNC wired default -
> 0x800b00000 0x800c00000 256 0 0xffffff002dd14948 rw- 1 0 0x3100 NCOW NNC wired default -
> 0x7fffff3db000 0x7fffff3fb000 1 0 0xffffff002dbb4360 rw- 1 0 0x3100 NCOW NNC not-wired default -
> 0x7fffff5dc000 0x7fffff5fc000 1 0 0xffffff002dc66af8 rw- 1 0 0x3100 NCOW NNC not-wired default -
> 0x7fffff7dd000 0x7fffff7fd000 1 0 0xffffff002dbea438 rw- 1 0 0x3100 NCOW NNC not-wired default -
> 0x7fffff9de000 0x7fffff9fe000 1 0 0xffffff002dd7fd80 rw- 1 0 0x3100 NCOW NNC not-wired default -
> 0x7fffffbdf000 0x7fffffbff000 1 0 0xffffff002dbe9438 rw- 1 0 0x3100 NCOW NNC not-wired default -
> 0x7fffffbff000 0x7fffffc00000 0 0 0 --- 0 0 0x0 NCOW NNC not-wired none -
> 0x7ffffffe0000 0x800000000000 32 0 0xffffff002dd125e8 rwx 1 0 0x3100 NCOW NNC wired default -
> 
> --- On Fri, 4/13/12, Konstantin Belousov <kostikbel at gmail.com> wrote:
> 
> > From: Konstantin Belousov <kostikbel at gmail.com>
> > Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN
> > To: "Sushanth Rai" <sushanth_rai at yahoo.com>
> > Cc: freebsd-hackers at freebsd.org
> > Date: Friday, April 13, 2012, 1:11 AM
> > On Thu, Apr 12, 2012 at 08:10:26PM
> > -0700, Sushanth Rai wrote:
> > > > 
> > > > Then it should be fixed in r190885.
> > > > 
> > > 
> > > Thanks. That works like a charm. 
> > > 
> > > mlockall() mostly works now. There is still a, issue in
> > wiring the stacks of multithreaded program when the program
> > uses default stack allocation scheme. Thread library
> > allocates stack for each thread by calling mmap() and
> > sending address and size to be mapped. The kernel adjusts
> > the start address to sgrowsz in  vm_map_stack() and
> > maps at the adjusted address. But the subsequent wiring is
> > done using the original address, which fails. 
Oh, I see. The problem is the VM_MAP_WIRE_NOHOLES flag. Since we
map only the initial stack fragment even for the MCL_WIREFUTURE maps,
there is a hole in the stack region.

In fact, for MCL_WIREFUTURE, we probably should map the whole
stack at once, prefaulting all pages.

Below are two patches. The change for vm_mmap.c would fix your immediate
problem by allowing holes in wired region.

The change for vm_map.c prefaults the whole stack instead of the
initial fragment. The single-threaded programs still get a fault
on stack growth.

diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
index 6198629..2fd18d1 100644
--- a/sys/vm/vm_map.c
+++ b/sys/vm/vm_map.c
@@ -3259,7 +3259,10 @@ vm_map_stack(vm_map_t map, vm_offset_t addrbos, vm_size_t max_ssize,
 	    addrbos + max_ssize < addrbos)
 		return (KERN_NO_SPACE);
 
-	init_ssize = (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz;
+	if (map->flags & MAP_WIREFUTURE)
+		init_ssize = max_ssize;
+	else
+		init_ssize = (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz;
 
 	PROC_LOCK(curthread->td_proc);
 	vmemlim = lim_cur(curthread->td_proc, RLIMIT_VMEM);
diff --git a/sys/vm/vm_mmap.c b/sys/vm/vm_mmap.c
index 2588c85..3fccd9e 100644
--- a/sys/vm/vm_mmap.c
+++ b/sys/vm/vm_mmap.c
@@ -1561,9 +1561,11 @@ vm_mmap(vm_map_t map, vm_offset_t *addr, vm_size_t size, vm_prot_t prot,
 		 * If the process has requested that all future mappings
 		 * be wired, then heed this.
 		 */
-		if (map->flags & MAP_WIREFUTURE)
+		if (map->flags & MAP_WIREFUTURE) {
 			vm_map_wire(map, *addr, *addr + size,
-			    VM_MAP_WIRE_USER | VM_MAP_WIRE_NOHOLES);
+			    VM_MAP_WIRE_USER | ((flags & MAP_STACK) ?
+			    VM_MAP_WIRE_HOLESOK : VM_MAP_WIRE_NOHOLES));
+		}
 	} else {
 		/*
 		 * If this mapping was accounted for in the vnode's
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20120413/a086811a/attachment.pgp
