Removing kernel thread stack swapping

[David Schultz]

Any objections to the idea of removing the feature of swapping out
kernel stacks?  Unlike disabling UAREA swapping, this has the
small downside that it wastes 16K (give or take a power of 2) of
wired memory per kernel thread that we would otherwise have
swapped out.  However, this disadvantage is probably negligible by
today's standards, and there are several advantages:

1. David Xu found that some kernel code stores externally-accessible
   data structures on the stack, then goes to sleep and allows the
   stack to become swappable.  This can result in a kernel panic.

2. We don't know how many instances of the above problem there are.
   Selectively disabling swapping for the right threads at the
   right times would decrease maintainability.

3. Thread stack swapping doesn't work very well anymore anyway.
   KSE introduced the idea that a process could have multiple
   kernel threads and hence multiple kernel stacks, but the
   swapper was not taught about this very well.  All the threads
   in a process get swapped out at the same time, and this only
   happens if all the threads happen to be swappable at the same time.

4. The code isn't well maintained.  That's not to say that it
   doesn't work, but it's conceivable that changes in locking
   elsewhere might introduce new bugs.  What if swapout races
   with exit() while the latter is trying to kill off the threads
   in the process, for instance?  If there were a bug, it could
   be very difficult to reproduce and track down.

At least for now, I would retain the swapper daemon and its
medium-term scheduling feature.  (That is, under high load, it
could mark some processes as unrunnable, thereby reducing
contention and allowing their user space pages to be paged out.)
This feature does not add much complexity to other parts of the
kernel.  If it turns out not to be very useful by itself, it can
be removed later, but I'm not as zealous with the axe as phk.  ;-)