can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code?

[John Baldwin]

On Friday, February 14, 2014 1:26:07 pm Andrey Chernov wrote:
> On 14.02.2014 22:18, John Baldwin wrote:
> > On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote:
> >> On 14 February 2014 08:39, John Baldwin <jhb at freebsd.org> wrote:
> >>> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote:
> >>>> Ok, so now I remember the other odd thing.
> >>>>
> >>>> I was seeing the sending context(s) jumping from one CPU to another
> >>>> during flowtable_insert_common(), around the locking bits.
> >>>>
> >>>> But I thread pinned all the sender user threads!
> >>>>
> >>>> So, why would the senders still be scheduled on other CPUs if I've
> >>>> pinned the userland threads?
> >>>>
> >>>> (and yes, I verified that the userland threads weren't moving around.)
> >>>
> >>> Can you clarify a bit?  It's not clear how sender thraeds differ from
> >>> userland threads differ from sender user threads.  (I.e. one reading
> >>> is that these are all the same thing and should thus all be pinned
> >>> (I assume you mean using cpuset to bind them to specific cores rather
> >>> than sched_pin))
> >>
> >> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts 
stuff:
> >>
> >> * the userland threads are using the cpuset call to map a thread into
> >> a cpuset, yes
> >> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as
> >> the userland threads
> > 
> > If they are all cpuset to a single CPU, they should not migrate, though
> > I think sched_bind() can override that.  However, that requires code to
> > explicitly call sched_bind() which should be rare.
> > 
> 
> Due to this bug, not fixed yet, the real picture is more complex:
> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585

Eh, that bug report has no useful details, as in, it doesn't list the
actual commands run.  If you do 'cpuset -l 6 -s 1' to force all
processes to only use CPU6, then yes, of course the other CPUs are idle
because that's what you _asked_ for.  AFAICT, that is all the original
reporter did.  At work we regularly add and remove CPUs from the
default set (set 1) on hundreds of machines every day with ULE without
any issues.

-- 
John Baldwin