Strawman proposal: making libthr default thread implementation?

[Robert Watson]

I know this has been discussed in the past, but I figured with 7.x trundling 
forward, it was time to think about it again.  In benchmarks for many common 
applications and scenarios, libthr demonstrates significantly better 
performance over libpthread -- this is not a coincidence, as David Xu has 
invested a lot of time in improving libthr and application performance, and 
libthr has benefitted significantly from the last ten years of threading work 
on FreeBSD.  libthr is also implemented across a larger number of our 
platforms, and is already libpthread on several.  The first recommendation we 
make to MySQL and other heavy thread users is "Switch to libthr", which is 
suggestive, also!  Likewise, I know David has worked hard to eliminate 
technical and standards obstacles that have been raised in the past.  This 
e-mail is a strawman proposal, intended to raise discussion, and possibly lead 
to action.

So the strawman proposal is: make libthr the default threading library on 7.x. 
A few questions given the proposal:

- Are there technical features present in libpthread that aren't yet in
   libthr, and are required?  In the past system/local thread support has been
   the complaint, but I believe that is now long fixed.  This is useful
   regardless of a switch.

- A further thought: it would be really nice to be able to benchmark the
   behavior of applications like Mozilla, Tor, KDE, etc, which also depend on
   threads, but which don't come with easy-to-use potted benchmarks (that I
   know of).  Can we get some hands to help explore this issue?  This is also
   useful regardless of a switch.

- What mechanism should be used?  Should libthr simply be compiled as
   libpthread, and libpthread as libkse, so that they can both be around and
   libkse can be used for benchmarking still?

- Are there significant objections to doing this?

BTW, the one technical concern I have with libthr is that under high 
contention kernel activity involving a small number of kernel locks, libkse 
implicitly reduces kernel lock contention by bounding the number of threads 
that can be in the run state in the kernel.  If, for example, you have 32 
threads, all performing accept() and then file descriptor I/O in a tight loop, 
libpthread will often perform significantly better, as libthr allows them all 
into the kernel at once.  This is most likely a scheduler issue, but it's 
significantly mitigated by M:N threading.  To experiment with this problem, 
which I may not have characterized, you can try out the 
src/tools/tools/netrate/{http,httpd} micro-benchmarks using the -t argument to 
force threaded instead of process-based behavior.  This effect seems not to 
come out as a practical issue in macro-benchmarks I've tried, perhaps as they 
are generally tuned for Linux threading, which presumably faces the same 
problem as our 1:1 package.  Exploring this issue is a strong argument to 
continue to support multiple thread libraries in the development branch, as 
you can't do a comparative evaluation if you don't have choices to compare!

If we're going to think about this seriously for 7.0, increasing libthr 
exposure sooner rather than later is probably a good idea.

Robert N M Watson
Computer Laboratory
University of Cambridge