FreeBSD memguard + spinlocks

[Andrew Brampton]

2009/4/11 Robert Watson <rwatson at freebsd.org>:
> On Sat, 11 Apr 2009, Andrew Brampton wrote:
>
> Your understanding is mostly right.  The missing bit is this: there are two
> kinds of interrupt contexts -- fast/filter interrupt handlers, which borrow
> the stack and execution context of the kernel thread they preempt, and
> interrupt threads, which get their own complete thread context.
>
> Fast interrupt handlers are allowed unlock to acquire spinlocks so as to
> avoid deadlock because of the borrowed context.  This means they can't
> perform any sort of sleep, or acquire any locks that might sleep, since the
> thread they've preempted may hold conflicting locks, or be the one that
> would have woken up the sleep that the handler performed.  Almost no code
> will run in fast handlers -- perhaps checking some device registers, doing
> work on a lockless or spinlock-protected queue, and waking up a worker
> thread.
>
> This is why, BTW, spin locks disable interrupt: they need to control
> preemption by other interrupt handlers to avoid deadlock, but they are not
> intended for use except when either in the scheduler, in a few related IPI
> contexts, or when synchronizing between normal kernel code and a fast
> handler.
>
> Full interrupt thread contexts are permitted to perform short lock sleeps,
> such as those performed when contending default mutexes, rwlocks, and
> rmlocks. They are permitted to invoke kernel services such as malloc(9),
> UMA(9), the network stack, etc, as long as they use M_NOWAIT and don't
> invoke msleep(9) or similar unbounded sleeps -- again to avoid the
> possibility of deadlocks, since you don't want an interrupt thread sleeping
> waiting for an event that only it can satisfy.
>
> So the first question, really, is whether you are or mean to be using
> fast/filter interrupt handler.  Device drivers will never call memory
> allocation, free, etc, from there, but will defer it to an ithread using the
> filter mechanism in 8.x, or to a task queue or other worker in 7.x and
> earlier.  If you're using a regular INTR_MPSAFE ithread, you should be able
> to use only default mutexes (a single atomic operation if uncontended)
> without disabling interrupts, etc.
>
> Robert N M Watson
> Computer Laboratory
> University of Cambridge
>

Thanks very much for your detailed reply. I'm slowly understanding how
everything in FreeBSD fits together, and I appreciate your help.

I've been given a project to take over, and all of the design
decisions were made before I started working on it, thus I'm playing
catch up. One of the decisions was to implement their own version of a
spin lock, which literally looks something like this:

lock_aquire() {
  critical_enter();
  while (! lockHeld ) {}
  lockHeld++;
}

This was actually the code tripping up MemGuard, as it is inside a
critical section, which MemGuard is unable to sleep within. This is
all running inside a kthread_create thread (I'm unsure of the proper
name of this type of thread).

Anyway, that is why I also asked about a lighter weight spin lock
(perhaps similar to this one). I tempted to replace this custom
spinlock with the standard MTX_DEF, however I'm unsure of its impact.
The custom spin lock seems quick and light to acquire, and it does not
concern me that a interrupt can potentially interrupt the code.

On a related note, if you change the lock in memguard to a MTX_SPIN,
it panics the kernel during boot. So that is not an option :) I was
only using memguard because I suspected memory being used after it was
freed. However, I think I will either change my locks to MTX_DEF or
live without memguard.

I realise I've not really asked any questions, but I would be grateful
for any insights anyone may have.
Andrew