CPU selection for ithreads on 8.3

[John Baldwin]

On Wednesday, May 02, 2012 5:16:17 pm Navdeep Parhar wrote:
> There seems to be a regression in 8.3 in the way the kernel selects CPUs
> for interrupts.  For example, cxgb(4) on 8.3 ends up with all
> its ithreads on the same CPU (CPU7 in this case).
> 
> 12 root     -68    -     0K   816K WAIT    7   0:55  0.00% intr{irq279: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:52  0.00% intr{irq275: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:47  0.00% intr{irq278: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:43  0.00% intr{irq277: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:43  0.00% intr{irq282: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:41  0.00% intr{irq281: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:32  0.00% intr{irq276: 
> cxgbc0}
> 12 root     -68    -     0K   816K WAIT    7   0:31  0.00% intr{irq280: 
> cxgbc0}
> 
> Back in the day there used to be code in cxgb to bind different
> interrupts to different CPUs but it was removed because the kernel
> distributed them across CPUs anyway.  So what changed?  This appears 8.3
> specific.  I don't see it on head and I don't have a 9 system readily
> available right now.

Hmmm, that seems odd.  It is true that the round-robin that the OS does only
pins the low-level message from the APIC/MSI vector to the CPU.  It does not
affect the thread.  However, ULE prefers to run ithreads on the CPU that
processes the interrupt:

static int
sched_pickcpu(struct thread *td, int flags)
{
	...
	/*
	 * Prefer to run interrupt threads on the processors that generate
	 * the interrupt.
	 */
	if (td->td_priority <= PRI_MAX_ITHD && THREAD_CAN_SCHED(td, self) &&
	    curthread->td_intr_nesting_level && ts->ts_cpu != self) {
		SCHED_STAT_INC(pickcpu_intrbind);
		ts->ts_cpu = self;
	}
}

-- 
John Baldwin