sched_pin() bug in SCHED_ULE

[mdf at FreeBSD.org]

On Thu, Aug 26, 2010 at 3:10 PM, Andriy Gapon <avg at icyb.net.ua> wrote:
> on 27/08/2010 00:20 mdf at FreeBSD.org said the following:
>>
>> I tried making sched_pin() a real function which used
>> intr_disable/intr_restore around saving off td->td_oncpu,
>> td->td_lastcpu and ts->ts_cpu, and the stack at the time of call.  In
>> sched_switch when I saw an unexpected migration I printed all that
>> out.  I found that on my boxes, at sched_pin() time ts_cpu was already
>> different from td->td_oncpu, so the specific problem I was having was
>> that while another thread can change ts_cpu it has no way to force
>> that thread to immediately migrate.
>
> Like e.g. in sched_affinity where ts_cpu is first changed and then the old cpu
> is ipi-ed?

That could do it.  I didn't follow the stack of the places that were
touching ts_cpu for non-curthread.

>> I believe the below patch fixes the issue, though I'm open to other methods:
>>
>>
>> Index: kern/sched_ule.c
>> ===================================================================
>> --- kern/sched_ule.c  (.../head/src/sys)      (revision 158279)
>> +++ kern/sched_ule.c  (.../branches/BR_BUG_67957/src/sys)     (revision 158279)
>> @@ -112,6 +112,7 @@
>>  /* flags kept in ts_flags */
>>  #define      TSF_BOUND       0x0001          /* Thread can not migrate. */
>>  #define      TSF_XFERABLE    0x0002          /* Thread was added as transferable. */
>> +#define      TSF_BINDING     0x0004          /* Thread is being bound. */
>
> I don't really follow why TSF_BINDING is needed, i.e. why TSF_BOUND is not
> sufficient in this case?

I wanted to tighten up the asserts, so that the only time it was okay
for a td_pinned thread to migrate was the one time it was moving to
the target cpu.  Having a separate flag allows that.

Thanks,
matthew