Advice on a multithreaded netisr patch?
rwatson at FreeBSD.org
Mon Apr 6 05:09:15 PDT 2009
On Mon, 6 Apr 2009, Barney Cordoba wrote:
> Is there a way to give a kernel thread exclusive use of a core? I know you
> can pin a kernel thread with sched_bind(), but is there a way to keep other
> threads from using the core? On an 8 core system it almost seems that the
> randomness of more cores is a negative in some situations.
> Also, I've noticed that calling sched_bind() during bootup is a bad thing in
> that it locks the system. I'm not certain but I suspect its the thread_lock
> that is the culprit. Is there a clean way to determine that its safe to lock
> curthread and do a cpu bind?
There isn't an interface to cleanly express "Use CPUs 4-7 for only network
processing". You can configure the system this way using the cpuset command
(including directing the low-level interrupts to specific CPUs in 8.x), but if
we think this is going to be a frequently desired policy, a bit more
abstraction will be required.
I'm not familiar with the problem you're seeing with sched_bind() -- I'm using
it from within some of my code without a problem, and that's fairly early in
the boot. A number of deadlocks are possible if one isn't very careful early
in the boot though, so I might look specifically for some of those: if you
migrate a thread to a CPU that isn't yet started, it won't be able to run
until the CPU has started. This means it's important not to migrate threads
that might lead to priority version-like deadlocks:
- Be careful not to migrate threads that hold locks the system requires to get
to the point where multiple CPUs run.
- Be careful not to migrate threads that will signal a resource being
available, such as a device driver, required to get to the point where
multiple CPUs run.
- Be careful not to migrate the main boot thread.
Could you be running into one of those cases? Usually they're fairly easy to
diagnose using DDB, if you can get into it, because you can see what the main
boot thread is waiting for, and reason about what's holding it. Are you able
to get into DDB when this occurs? (Perhaps using an NMI?)
Robert N M Watson
University of Cambridge
More information about the freebsd-net