Re-engineer the wheel: a rejuvenation of BSD callout(9) and timer facilities - report weeks 9-12

[Davide Italiano]

Howdy,
This last week I'm mainly focusing on writing some  microbenchmarks in
order to profile the performance of the work I've recently done, as
well as discussing on the ML about the current status of my project

Here's my report for the last week.

Week 9:
- Introduce tsleep_bt() function as part of the sleep(9) KPI. Use it
in kern_nanosleep() so that we don't need to duplicate _sleep() code.
- General code polish and bugfixing
- Start implementing properly direct callout execution (until now it
was a bit rudimentary) so that it may have all the functionalities
that normal callout has (for example, migration)

Week 10:
- Fix a LOR deadlock dropping the callout lock while executing the
handler directly from hardware interrupt context.
- Finish adding migration support for callouts which runs from hw
interrupt context.
- Refactor a couple of comments to reflect the new world order.
- Implement statistics for direct execution callouts, providing a set
of new SYSCTLs.

Week 11:
- Modify the KTR code so that the size of the circular event buffer is
not anymore constrained to be a power of two (merged to HEAD)
- Investigate about the causes of a panic in early boot
deterministically reproducible also on HEAD if the value of
KTR_ENTRIES set is over a given threshold (depending on the amount of
RAM available). Increasing the value of NKPT (number of kernel page
tables) in pmap code acts as workaround, but, still I'm discussing
with Alan Cox (alc@) a proper solution to address the problem. This is
not directly related to my work so I don't consider it as a priority.
- Remove a spurious brace which cause build fail in case BCE_DEBUG
option is turned on. (merged to HEAD).

Week 12:
- Some optimizations: simplify the logic of softclock() removing a
check which is likely to be false in most of the cases, after recent
changes. Also, remove an unneeded and duplicated bitwise and
operation.  Optimize a bit callout_process() routine avoiding some
calculations if they're redundant.
- Remove some unneeded checks. For non-direct callouts
softclock_call_cc() is only called for callouts on cc_expireq queue,
and all the callouts there have CALLOUT_PROCESSED flag. Similarly, for
direct callouts the aforementioned function is called for callouts on
the wheel, which haven't CALLOUT_PROCESS flag set.
- General code polishment and bugfixing.
- Start writing amicrobenchmark in order to profile the performance of
the last three months changes. It may be found here:
http://people.freebsd.org/~davide/test_cycles_sleep.c

A diff between HEAD and the branch, applicable without conflict to
recent HEAD revision, may be found here:
http://people.freebsd.org/~davide/callout_patch.2.diff

We're mostly at the end of the program, thanks for your attention, folks.

Davide