Advice on a multithreaded netisr patch?

Barney Cordoba barney_cordoba at
Mon Apr 6 06:41:02 PDT 2009

--- On Mon, 4/6/09, Robert Watson <rwatson at> wrote:

> From: Robert Watson <rwatson at>
> Subject: Re: Advice on a multithreaded netisr  patch?
> To: "Barney Cordoba" <barney_cordoba at>
> Cc: freebsd-net at, "Ivan Voras" <ivoras at>
> Date: Monday, April 6, 2009, 8:09 AM
> On Mon, 6 Apr 2009, Barney Cordoba wrote:
> > Is there a way to give a kernel thread exclusive use
> of a core? I know you can pin a kernel thread with
> sched_bind(), but is there a way to keep other threads from
> using the core? On an 8 core system it almost seems that the
> randomness of more cores is a negative in some situations.
> > 
> > Also, I've noticed that calling sched_bind()
> during bootup is a bad thing in that it locks the system.
> I'm not certain but I suspect its the thread_lock that
> is the culprit. Is there a clean way to determine that its
> safe to lock curthread and do a cpu bind?
> There isn't an interface to cleanly express "Use
> CPUs 4-7 for only network processing".  You can
> configure the system this way using the cpuset command
> (including directing the low-level interrupts to specific
> CPUs in 8.x), but if we think this is going to be a
> frequently desired policy, a bit more abstraction will be
> required.
> I'm not familiar with the problem you're seeing
> with sched_bind() -- I'm using it from within some of my
> code without a problem, and that's fairly early in the
> boot.  A number of deadlocks are possible if one isn't
> very careful early in the boot though, so I might look
> specifically for some of those: if you migrate a thread to a
> CPU that isn't yet started, it won't be able to run
> until the CPU has started.  This means it's important
> not to migrate threads that might lead to priority
> version-like deadlocks:
> - Be careful not to migrate threads that hold locks the
> system requires to get
>   to the point where multiple CPUs run.
> - Be careful not to migrate threads that will signal a
> resource being
>   available, such as a device driver, required to get to
> the point where
>   multiple CPUs run.
> - Be careful not to migrate the main boot thread.
> Could you be running into one of those cases?  Usually
> they're fairly easy to diagnose using DDB, if you can
> get into it, because you can see what the main boot thread
> is waiting for, and reason about what's holding it.  Are
> you able to get into DDB when this occurs?  (Perhaps using
> an NMI?)

Yes, the cpus are launched quite late, so that must be it. I guess
the mp_ncpus is set before they are launched. Is there a way to determine
that a specific core has been lauched?

Regarding using cpuset, John B indicated that you couldn't allocate
"sets" for kernel threads; and that sched_bind() was the only function
available. So that brings 2 questions:

1) How do you get the thread ID for a process from user space to use with
cpuset? I don't see that ps displays it.

2) Can cpu sets be manipulated / setup from within the kernel?



More information about the freebsd-net mailing list