pthread switch (was Odd KSE panic)

[Andrew Gallatin]

Daniel Eischen writes:
 > 
 > With a small number of threads, it probably doesn't make sense
 > to have more than one KSE (unless they are mostly CPU-bound).
 > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads.

Actually, there are 2 procs, each with 2 threads. 

To elaborate, there are two processes, each running one instance of a
pingpong test for each of 2 NICs in the box.

 > Each time a thread sleeps, the KSE enters the kernel to sleep
 > (kse_release()) because there are no other threads to run.
 > 
 > Drew, can you try lowering the concurrency?  You can
 > either try using pthread_setconcurrency(1) or setting
 > kern.threads.virtual_cpu=1.


Here is the raw data for a ping-pong between 2 nics on the same box
with various configurations of cpu_idle_hlt and virtual_cpu.



Polling	        5.0us
Linux          20.4us
linulator      82.5us
thr	       75.1us
kse	      123.6us

machdep.cpu_idle_hlt=0
linulator      74.0us
thr	       64.6us
kse	      111.9

kern.threads.virtual_cpu=1
machdep.cpu_idle_hlt=0
kse	      100.8

kern.threads.virtual_cpu=1
machdep.cpu_idle_hlt=0
kse	      93.9


Key:
Linux: kernel 2.66, Debian libc6  2.2.5-11.5 (no sysenter, or NPTL)
linulator: static binary used above, run under COMPAT_LINUX
thr: libthr (via libmap.conf)
kse: libpthread


BTW, there's a little more fat in the interrupt/wakeup path in the
FreeBSD MD driver code (condvars, MPSAFE intr, rather than an
INTR_FAST and some homebrew technique).  Based on other measurements, I
don't think its 54us worth of fat (more like 15 or 20usec).

Drew