sched_pickcpu() and a pinned thread.

[Miłosz Kaniewski]

Hi,

I have an application which exchanges packets between two NIC ports.
Most of a CPU time is therefore consumed by the userspace process and
two IRQ threads. All of them are pinned to the same CPU using
cpu_setaffinity() function so I expect that the scheduler usage should
be minimal even on a high network load when a lot of interrupts are
generated.
I recently find out that my application spends a lot of time in
cpu_search_lowest() function.  A backtrace shown me that
cpu_search_lowest() is called by sched_pickcpu(). I tried to analyze
this second function but I don't know what exactly should I expect
from it if my thread is pinned to a specific CPU. I assume that the
function should get to a point where ts->ts_cpu is returned. However
it doesn't do it and sched_pickcpu() always ends looking for the
lowest busy CPU using cpu_search_lowest(). In result it finds the CPU
that the thread is pinned to but it is very time consuming process.
Therefore I suspect that sched_pickcpu() should return before that
happens.

For interrupt threads I see that there is a special case at the
beginning of sched_pickcpu():
/*
     * Prefer to run interrupt threads on the processors that generate
     * the interrupt.
     */
    pri = td->td_priority;
    if (td->td_priority <= PRI_MAX_ITHD && THREAD_CAN_SCHED(td, self) &&
        curthread->td_intr_nesting_level && ts->ts_cpu != self) {
        SCHED_STAT_INC(pickcpu_intrbind);
        ts->ts_cpu = self;
        if (TDQ_CPU(self)->tdq_lowpri > pri) {
            SCHED_STAT_INC(pickcpu_affinity);
            return (ts->ts_cpu);
        }
    }
However IRQ threads from the NIC doesn't fall into this case because
ts->ts_cpu == self (these variables are equal to the CPU to which my
threads are pinned). Is this check required?

For non-interrupt threads there are some other possible options where
ts->ts_cpu is returned however I am not sure which should handle a
pinned thread.

I would be grateful if someone would give me some clue how should I
understand the sched_pickcpu() function and when it should return in
case of a pinned thread.

Thanks
Miłosz Kaniewski