FreeBSD memguard + spinlocks

[Robert Watson]

On Sat, 11 Apr 2009, Andrew Brampton wrote:

> I'm having a problem with memguard(9) on FreeBSD 7.1 but before I ask about 
> that I just need to check my facts about malloc.
>
> When in interrupt context malloc must be called with M_NOWAIT, this is 
> because I can't sleep inside a interrupt. Now when I hold a spinlock 
> (MTX_SPIN) I am also not allowed to sleep or obtain a sleepable mutex (such 
> as MTX_DEF). So I assume while holding a spin lock any mallocs I do must 
> have the M_NOWAIT flag? This was not clear from the manual pages, but at 
> least makes sense to me.
>
> So my problem with memguard stems from the fact that I am locking a 
> spinlock, and then I'm calling malloc with M_NOWAIT. But inside 
> memguard_alloc a MTX_DEF is acquired causing WITNESS to panic.
>
> So I think fundamental memguard is flawed and should be using MTX_SPIN 
> instead of MTX_DEF otherwise it can't be called from inside a interrupt or 
> when a spin lock is held. But maybe I'm missing something?
>
> Also on a related note, I see that MTX_SPIN disables interrupts, making it a 
> rather "heavy" spinlock. Is there a lighter spin lock that literally just 
> spins? I read that MTX_DEF are far quicker to acquire , but surely a light 
> spinlock would be easier to acquire than sleeping?
>
> I think for the moment I will fix my code by not using a MTX_SPIN (since the 
> code is not in a interrupt), however, I think memguard should change its 
> lock.

Your understanding is mostly right.  The missing bit is this: there are two 
kinds of interrupt contexts -- fast/filter interrupt handlers, which borrow 
the stack and execution context of the kernel thread they preempt, and 
interrupt threads, which get their own complete thread context.

Fast interrupt handlers are allowed unlock to acquire spinlocks so as to avoid 
deadlock because of the borrowed context.  This means they can't perform any 
sort of sleep, or acquire any locks that might sleep, since the thread they've 
preempted may hold conflicting locks, or be the one that would have woken up 
the sleep that the handler performed.  Almost no code will run in fast 
handlers -- perhaps checking some device registers, doing work on a lockless 
or spinlock-protected queue, and waking up a worker thread.

This is why, BTW, spin locks disable interrupt: they need to control 
preemption by other interrupt handlers to avoid deadlock, but they are not 
intended for use except when either in the scheduler, in a few related IPI 
contexts, or when synchronizing between normal kernel code and a fast handler.

Full interrupt thread contexts are permitted to perform short lock sleeps, 
such as those performed when contending default mutexes, rwlocks, and rmlocks. 
They are permitted to invoke kernel services such as malloc(9), UMA(9), the 
network stack, etc, as long as they use M_NOWAIT and don't invoke msleep(9) or 
similar unbounded sleeps -- again to avoid the possibility of deadlocks, since 
you don't want an interrupt thread sleeping waiting for an event that only it 
can satisfy.

So the first question, really, is whether you are or mean to be using 
fast/filter interrupt handler.  Device drivers will never call memory 
allocation, free, etc, from there, but will defer it to an ithread using the 
filter mechanism in 8.x, or to a task queue or other worker in 7.x and 
earlier.  If you're using a regular INTR_MPSAFE ithread, you should be able to 
use only default mutexes (a single atomic operation if uncontended) without 
disabling interrupts, etc.

Robert N M Watson
Computer Laboratory
University of Cambridge