mlock(2), unprivileged users, and RLIMIT_MEMLOCK

[Coleman Kane]

On Sat, 2008-04-12 at 15:23 -0400, Joe Marcus Clarke wrote:
> On Sat, 2008-04-12 at 15:09 -0400, Coleman Kane wrote:
> > Hello,
> > 
> > Recently we've been having a discussion on the GNOME list about fixing
> > the seahorse breakage introduced with the latest GNOME 2.22, rooted in
> > the fact that FreeBSD's mlock(2) implementation is only usable if you
> > have superuser privileges. Due to bugs in seahorse, the lack of mlock(2)
> > causes many seahorse applications to die. I've posted a suggested patch
> > to 
> > 
> > From my understanding, a significant reasoning for this is because if
> > unprivileged users could mlock(2), then they could incur a DoS attack on
> > a system by spawning off at most RLIMIT_NPROC processes, each
> > wiring-down RLIMIT_MEMLOCK bytes of memory in an effort to steal away
> > all real system RAM from the rest of the system, and bring usage to a
> > screeching halt.
> > 
> > I've posted up a short page about it on my site here:
> > http://www.cokane.org/dokuwiki/freebsd/mlock-support
> > 
> > I'd like to know if there are any other patches that are floating around
> > for the same thing, or even if there are some good alternatives to
> > mlock(2) that yield similar results (secure memory accessible by the
> > user). I'd also welcome any comments that others have on the topic, as I
> > am looking for approaches to implement the support under FreeBSD without
> > compromising the security of the OS.
> 
> As mezz pointed out, Peter Jeremy commented on this a while ago:
> 
> http://lists.freebsd.org/pipermail/freebsd-arch/2006-July/005496.html
> 
> > 
> > An idea that came to mind, but I am less familiar with, is to also
> > support some sort of MAC policy checks that can be enforced by the
> > administrator on the system to provide some users with secure access
> > support, while preventing others from using it.
> > 
> > A second idea might be to turn RLIMIT_MEMLOCK into a per-user (or even
> > system-wide) resource limit, rather than a per-process limit.
> > 
> > As a third idea, we could leave the per-process limit (to abide by
> > historical documentation), but also add a sysctl that enforces a
> > system-wide "max mlock pages" which can be tested by the mlock(2)
> > syscall, refusing to mlock(2) more memory if the limit is hit.
> 
> I think this already exists in -CURRENT: vm.max_wired ("System-wide
> limit to wired page count").  This is tested by mlock(2) in addition to
> RLIMIT_MEMLOCK.
> 
> I also looked through the kernel for instances where RLIMIT_MEMLOCK is
> checked, and the only other place is in the vslock() function.  The only
> consumer of this function I could find is sysctl_wire_old_buffer() which
> is used by quite a few sysctl handlers.  If the rlimit is changed from
> infinity, users might have problems getting results from certain
> sysctls.
> 
> Joe
> 

Another thing that we're going to want to keep in mind is that the
mlock(2)-memory is probably allocated on a page-by-page basis (so
mlock(2) pointers point into mlock(2) pages). I *think* this means that
the minimum per-process mlock(2) size (as far as in-kernel usage is
concerned) is going to be 4096 Bytes. I'm setting up a kernel now with
your patch so that I can test using rlimits on the system.

I am guessing that mlock(2) pages are going to be distinct between
processes, meaning that two processes that only want to mlock(2)
512-bytes of memory will end up incurring an 8192 Byte impact on on
mlock(2) availability.

Correct me if I am mistaken.

--
Coleman Kane