mlockall() on freebsd 7.2 + amd64 returns EAGAIN

[Sushanth Rai]

Wiring entire address space seems to have interesting side effect. The libc memory allocator calls madvise() to free the dirty unused pages, which does nothing when the pages are wired. The allocator unmaps only when entire chunk is free (default size of 1MB). That leaves lots for free pages which cannot reclaimed even when the system is under memory pressure.

Sushanth

--- On Mon, 4/16/12, Sushanth Rai <sushanth_rai at yahoo.com> wrote:

> From: Sushanth Rai <sushanth_rai at yahoo.com>
> Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN
> To: "Konstantin Belousov" <kostikbel at gmail.com>
> Cc: alc at freebsd.org, freebsd-hackers at freebsd.org
> Date: Monday, April 16, 2012, 11:41 AM
> Many thanks. I verified the patch you
> provided and it works fine.
> 
> Sushanth
> 
> 
> > Oh, I see. The problem is the VM_MAP_WIRE_NOHOLES
> flag.
> > Since we
> > map only the initial stack fragment even for the
> > MCL_WIREFUTURE maps,
> > there is a hole in the stack region.
> > 
> > In fact, for MCL_WIREFUTURE, we probably should map
> the
> > whole
> > stack at once, prefaulting all pages.
> > 
> > Below are two patches. The change for vm_mmap.c would
> fix
> > your immediate
> > problem by allowing holes in wired region.
> > 
> > The change for vm_map.c prefaults the whole stack
> instead of
> > the
> > initial fragment. The single-threaded programs still
> get a
> > fault
> > on stack growth.
> > 
> > diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
> > index 6198629..2fd18d1 100644
> > --- a/sys/vm/vm_map.c
> > +++ b/sys/vm/vm_map.c
> > @@ -3259,7 +3259,10 @@ vm_map_stack(vm_map_t map,
> > vm_offset_t addrbos, vm_size_t max_ssize,
> >          addrbos + max_ssize <
> > addrbos)
> >          return
> > (KERN_NO_SPACE);
> >  
> > -    init_ssize = (max_ssize < sgrowsiz) ?
> > max_ssize : sgrowsiz;
> > +    if (map->flags & MAP_WIREFUTURE)
> > +        init_ssize =
> > max_ssize;
> > +    else
> > +        init_ssize =
> > (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz;
> >  
> >      PROC_LOCK(curthread->td_proc);
> >      vmemlim = lim_cur(curthread->td_proc,
> > RLIMIT_VMEM);
> > diff --git a/sys/vm/vm_mmap.c b/sys/vm/vm_mmap.c
> > index 2588c85..3fccd9e 100644
> > --- a/sys/vm/vm_mmap.c
> > +++ b/sys/vm/vm_mmap.c
> > @@ -1561,9 +1561,11 @@ vm_mmap(vm_map_t map,
> vm_offset_t
> > *addr, vm_size_t size, vm_prot_t prot,
> >           * If the
> > process has requested that all future mappings
> >           * be
> > wired, then heed this.
> >           */
> > -        if (map->flags
> > & MAP_WIREFUTURE)
> > +        if (map->flags
> > & MAP_WIREFUTURE) {
> >             
> > vm_map_wire(map, *addr, *addr + size,
> > -           
> >     VM_MAP_WIRE_USER | VM_MAP_WIRE_NOHOLES);
> > +           
> >     VM_MAP_WIRE_USER | ((flags & MAP_STACK) ?
> > +           
> >     VM_MAP_WIRE_HOLESOK : VM_MAP_WIRE_NOHOLES));
> > +        }
> >      } else {
> >          /*
> >           * If this
> > mapping was accounted for in the vnode's
> >
>