polling(4) and idle time/cpu usage percentages

[Bruce Evans]

On Sun, 4 May 2003, Kevin Day wrote:

> I've got a FreeBSD system acting as a router, it's passing 250-600mbps of
> traffic through it most of the time.
>
> Yesterday it was running 4.6-RELEASE without polling. I've upgraded it to
> 4.8 and enabled polling. Before it was showing 30-50% CPU use in interrupt
> and system combined. Now it's showing 0-1% (99% idle).
>
> Is this because it's polling in the idle loop, and time spent doing this
> isn't getting accounted for anywhere, or is polling THAT much more efficient?
>
> If it's the former, is it supposed to work this way? Now I've got no clear
> way of knowing how busy the system is. (It's just routing packets, really
> nothing more)

The former.  It's hard for it to work better without wasting too many
cycles for the accounting.  In RELENG_4, everything done in the "idle"
loop is counted as idle time using the single counter cp_time[CP_IDLE].
This is very efficient.

This is "fixed" in -current by wasting too many cycles for the accounting.
-current can't do anything that might block on a mutex in its pure
idle routine.  This means that it can do very little in its pure idle
routine, since even polling-like things that don't want to block tend
to need mutexes for synchronization.  So -current uses idle priority
kernel threads instead of the idle loop for polling.  Since all threads
are heavyweight, normal accounting for threads (processes) gives their
CPU usage very accurately for "free" (once you have paid for their weight).
Context switching and accounting for even idle priority threads isn't
free even if there is idle time to spare, since the time to switch back
to non-idle priority threads is taken from non-idle time.

Hack for getting more useful idle time statistics in RELENG_4:
- in the idle loop, set a flag while calling _idle_poll and
  _vm_page_zero_idle.
- in hardclock(), bump cp_time[CP_SYS] instead of cp_time[CP_IDLE]
  when the flag is set.  A new counter would give more detail but
  wouldn't be reported automatically by utilities.

Bruce