machdep.cpu_idle_hlt and SMP perf?

[Andrew Gallatin]

John Baldwin writes:
 > On Monday 06 February 2006 14:46, Andrew Gallatin wrote:
 > > Andre Oppermann writes:
 > >  > Andrew Gallatin wrote:
 > >  > > Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network rx
 > >  > > performance by a considerable amount (7.5Gbs -> 5.5Gbs)?
 > >
 > > <...>
 > >
 > >  > This may be the same problem OpenBSD has fixed last year in the handling
 > >  > of the idle loop.  From the kerneltrap posting:
 > >
 > > <....>
 > >
 > >  > First commit message:
 > >  >   http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111692513727274&w=2
 > >  >
 > >  > The MFC with all changes in one commit message:
 > >  >   http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111859519015510&w=2
 > >
 > > The bug they fixes was missing interrupts by both calling APM's idle
 > > routine, which may hlt, and hlt'ing in the idle loop itself.  Since I
 > > have no idea what acpi is doing, I got excited about this.
 > >
 > > Alas, it seems like this isn't it.  I pointed cpu_idle_hook back to
 > > cpu_idle_default and away from acpi_cpu_idle, but that made no
 > > difference.
 > 
 > You may be seeing problems because it might simply take a while for the CPU to 
 > wake up from HLT when an interrupt comes in.  The 4BSD scheduler tries to do 
 > IPIs to wakeup any sleeping CPUs when it schedules a new thread, but that 
 > would add higher latency for ithreads than just preempting directly to the 
 > ithread.  Oh, you have to turn that on, it's off by default 
 > (kern.sched.ipiwakeup.enabled=1).

Hmm..  It seems to be on by default.  Unfortunately, it does not seem
to help.

Would you expect ULE to do better?  I've noticed that if I screw up
the time state of the machine by switching between ACPI-fast and TSC
timecounters, performance for TCP ping-pongs goes all over the map...

Drew