machdep.cpu_idle_hlt and SMP perf?

[Andrew Gallatin]

John Baldwin writes:
 > On Monday 06 February 2006 17:37, Andrew Gallatin wrote:
 > > John Baldwin writes:
 > >  > On Monday 06 February 2006 14:46, Andrew Gallatin wrote:
 > >  > > Andre Oppermann writes:
 > >  > >  > Andrew Gallatin wrote:
 > >  > >  > > Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network rx
 > >  > >  > > performance by a considerable amount (7.5Gbs -> 5.5Gbs)?
 > >  > >
 > >  > You may be seeing problems because it might simply take a while for the
 > >  > CPU to wake up from HLT when an interrupt comes in.  The 4BSD scheduler
 > >  > tries to do IPIs to wakeup any sleeping CPUs when it schedules a new
 > >  > thread, but that would add higher latency for ithreads than just
 > >  > preempting directly to the ithread.  Oh, you have to turn that on, it's
 > >  > off by default
 > >  > (kern.sched.ipiwakeup.enabled=1).
 > >
 > > Hmm..  It seems to be on by default.  Unfortunately, it does not seem
 > > to help.
 > 
 > I'm not sure.

One thing which really helps is disabling preemption.  If I do that,
I get 7.7Gb/sec with machdep.cpu_idle_hlt=1.  This is slightly better
than machdep.cpu_idle_hlt=0 and no PREEMPTION.

BTW, net.isr.direct=1 in all testing.


Drew