sched_lock && thread_lock()
Jeff Roberson
jroberson at chesapeake.net
Sun May 20 23:07:57 UTC 2007
Attilio and I have been working on addressing the increasing problem of
sched_lock contention on -CURRENT. Attilio has been addressing the parts
of the kernel which do not need to fall under the scheduler lock and
moving them into seperate locks. For example, the ldt/gdt lock and clock
lock which were committed earlier. Also, using atomics for the vmcnt
structure.
I have been working on an approach to using thread locks rather than a
global scheduler lock. The design is similar to Solaris's container
locks, but the details are different. The basic idea is to have a pointer
in the thread structure that points at a spinlock that protects the
thread. This spinlock may be one of the scheduler lock, a turnstile lock,
or a sleep queue lock. As the thread changes state from running to
blocked on a lock or sleeping the lock changes with it.
This has several advantages. The majority of the kernel simply calls
thread_lock() which figures out the details. The kernel then knows
nothing of the particulars of the scheduler locks, and the schedulers are
free to implement them in any way that they like. Furthermore, in some
cases the locking is reduced, because locking the thread has the side
effect of locking the container.
This patch does not implement per-cpu scheduler locks. It just changes
the kernel to support this model. I have a fork of ULE in development
that runs with per-cpu locks, but it is not ready yet. This means that
there should be very little change in system performance until the
scheduler catches up. In fact, on a 2cpu system the difference is
immeasurable or almost so on every workload I have tested. On an 8way
opteron system the results vary between +10% on some reasonable workloads
and -15% on super-smack, which has some inherent problems that I believe
are not exposing real performance problems with this patch.
This has also been tested extensively by Kris and myself on a variety of
machines and I believe it to be fairly solid. The only thing remaining to
do is fix rusage so that it does not rely on a global scheduler lock.
I am posting the patch here in case anyone with specific knowledge of
turnstiles, sleepqueues, or signals would like to review it, and as a
general heads up to people interested in where the kernel is headed.
This will apply to current just prior to my kern_clock.c commits. I will
re-merge and update again in the next few days, probably after we sort out
rusage.
http://people.freebsd.org/~jeff/threadlock.diff
Thanks,
Jeff
More information about the freebsd-arch
mailing list