What is the PREEMPTION option good for?

[Bruce Evans]

PREEMPTION may be needed for correctness, including for very low latency
switching to high priority tasks, but it is a pessimization for most
things that I tried (mainly makeworld), and the details of the
pessimizations indicate that it doesn't actually give correctness either.

Makeworld in a certain SMP configuration here takes 853+-5 seconds without
PREEMPTION or IPI_PREEMPTION, and 863+-5 seconds with PREEMPTION and
IPI_PREEMPTION (both without KSE; KSE gives another pessimization in
the 5-10 second range).

Most of the difference is caused by pgzero becoming too active with
PREEMPTION.  The behaviour with PREEMPTION under SMP is similar to
that under both UP and SMP when the PREEMPTION ifdef was first added
to vm_zeroidle.c a couple of years ago.  pgzero is active for about 3
times as long (18 seconds instead of 6 for my current makeworld
benchmark.  This reduces the reported system time by even more than
the extra time spent in pagezero, but for some reason (probably cache
thrashing, though pgzero uses nontemporal writes on the benchmark
machine) it increases the real time by about the same amount as the
extra time spent in pgzero.

A couple of years ago, pgzero did this even for !SMP because it had a
very broken priority so it rarely (never?) got preempted.  Now its
preemption is broken similarly under SMP with PREEMPTION but without
IPI_PREEMPTION, since it takes an IPI to preempt it in many cases.
Its code is:

% 	for (;;) {
% 		if (vm_page_zero_check()) {
% 			vm_page_zero_idle();
% #ifndef PREEMPTION
% 			if (sched_runnable()) {
% 				mtx_lock_spin(&sched_lock);
% 				mi_switch(SW_VOL, NULL);
% 				mtx_unlock_spin(&sched_lock);
% 			}
% #endif

without PREEMPTION, it yields voluntarily, and this works fine.  With
PREEMPTION and !SMP, it gets preempted, and this works not so fine (it
has slightly higher overheads).  With PREEMPTION and SMP and >1 CPU
but no IPI_PREEMPTION, this cannot work, and even with IPI_PREEMPTION
it doesn't work now in practice (IPI_PREEMPTION gives a small pessimization
due to more context switches without significantly affecting the time
spent in pgzero).

If PREEMPTION should actually be best here, why doesn't the main idle
thread depend on it?

Does anyone have an example where PREEMPTION makes a useful difference?

Bruce