FreeBSD 5.2.1: Mutex/Spinlock starvation?

[Robert Watson]

On Thu, 3 Jun 2004, Ali Niknam wrote:

> First of all: this is my first posting in this group so please be gentil
> :) 

Welcome :-).

<snip>
> Now i unfortunately do not know enough about the internals of BSD to do a
> very estimated guess, but i'll give a shot nevertheless: my estimate is that
> due to the tremendous amount of 'locked' processes the system simply starves
> of CPU to do anything. My guess is the Locking mechanism probably uses
> some kind of 'spin' to wait until the resource is unlocked (whichever
> resource it is, probably something network related, though).
</snip>

Actually, by default, most mutexes in the system are sleep mutexes, so
they sleep on contention rather than spinning.  In some cases, this
actually hurts more than spinning, because if the mutex is released
quickly by the holder, then you pay the context switches which cost more
than spinning for the short period of time.

You might want to try adding "options ADAPTIVE_MUTEXES" to your kernel
configuration, which will cause mutexes to spin briefly on SMP systems
before sleeping, and has been observed to improve performance quite a bit.

> I would be very interested to hear what this problem could be; perhaps i
> can test a little if someone has solutions (i cant test much
> unfortunately, it's a production system).

As you may or may not be aware, improving locking and parallelism in
FreeBSD 5.x is a big on-going task, with a lot of activity.  A moderate
quantity of recent locking work has occurred since 5.2.1 release, so
depending on your tolerance for experimentation on this system, you might
wish to give 5-CURRENT a try.  Be warned that 5-CURRENT, while having a
number of performance enhancements, also has some stability regressions,
more recent ACPI code, etc.  I'm using older snapshots of 5-CURRENT in
production today, but generally not newer than about April or early May.
If you do try -CURRENT, take a look at UPDATING, and make sure to disable
a lot of the debugging features present if you're interested specifically
in performance.

If you have a lower tolerance for instability, there are a number of minor
performance tweaks that can be easily back-ported to 5.2.1, such as the
change to proc.h to make grabbing and releasing the proc lock conditional
on p_stops having events defined.  This removes several mutex operations
from each system call, and I've observed the difference in a pretty
measurable way on micro-benchmarks.  It's also pretty low risk.  The
change is src/sys/sys/proc.h:1.366.  There are some other related changes
that can probably be dug up, including changes to improve the performance
of the scheduler in the presence of threads, etc. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Senior Research Scientist, McAfee Research