Kernel threads inherit CPU affinity from random sibling

Attilio Rao attilio at
Sat Jan 28 03:41:47 UTC 2012

2012/1/28 Ryan Stone <rysto32 at>:
> Right now, whenever a thread is spawned, it inherits CPU affinity from
> its "parent" thread.  I put parent in scare quotes because as far as I
> can tell, for a kernel thread the parent is essentially chosen
> arbitrarily (it looks like it is the most recent thread spawned in
> that kernel process).  Inheriting affinity is arguably correct for
> userland threads (at the very least, an explicit design decision to
> implement inheritance was clearly implemented).  However for kernel
> threads, this behaviour leads to bizarre results that clearly weren't
> intended.  For example, I added a printf to sched_fork_thread() that
> prints a message every time a thread inherits affinity from its
> "parent".  Here's the results from booting a dual-core VM with
> net.isr.bindthreads=1:
> Thread 100006 inheriting affinity from parent swi1: netisr 0(100005)
> Thread 100007 inheriting affinity from parent swi3: vm(100006)
> Thread 100008 inheriting affinity from parent swi4: clock(100007)
> Thread 100014 inheriting affinity from parent swi4: clock(100008)
> Thread 100017 inheriting affinity from parent swi5: +(100014)
> Thread 100018 inheriting affinity from parent swi6: Giant taskq(100017)
> Thread 100022 inheriting affinity from parent swi6: task queue(100018)
> Thread 100025 inheriting affinity from parent swi2: cambio(100022)
> Thread 100026 inheriting affinity from parent irq14: ata0(100025)
> Thread 100027 inheriting affinity from parent irq15: ata1(100026)
> Thread 100028 inheriting affinity from parent irq19: le0(100027)
> Thread 100029 inheriting affinity from parent irq21: pcm0(100028)
> Thread 100034 inheriting affinity from parent irq22: ohci0(100029)
> Thread 100035 inheriting affinity from parent irq1: atkbd0(100034)
> The result is that every thread in intr kernel process ends up
> inheriting the affinity(my favourite part is that both softclock
> threads are now forced to fight over the same CPU).  I am working on
> the following patch that adds a flag to
> sched_fork()/sched_fork_thread() that can be passed to force the child
> thread to *not* inherit affinity.  However, this doesn't address
> affinity inheritance from calls to kproc_create().  I suppose that I
> could push the flag up into fork1(), but I'd prefer not to.  I am
> wondering if it even makes sense for affinity to be inherited from a
> parent process even for userland processes.
> What are peoples thoughts?  Is this the right approach?

I think what you found out is very sensitive.
However, the patch is not correct as you cannot call
cpuset_setthread() with thread_lock held. Also, I don't think we need
the flag bloat, to be honest.

IMHO, we should choose a default policy for sched_fork{_thread}() and
have the opposite case to fix the thread inherited affinity manually.
For example, for the shortest patch, we can go with
sched_fork{_thread}() functions to directly inherited parent affinity
which is ok in the userland case but needs to be fixed in the
kthread/kproc case. Hence this is my fix:

(just test-compiled, so if you agree and want to test further, be my guest).
Maybe we can also add a comment to sched_fork() guys explaining the
default policy, but the code seems self-explanatory enough to me.

What do you think about it?


Peace can only be achieved by understanding - A. Einstein

More information about the freebsd-hackers mailing list