Bogus signal handler causes kernel panic (5.2.1-p8/i386)

[Bruce Evans]

On Wed, 16 Jun 2004, Simon Barner wrote:

> I tried the local denial of service attack described in [1], that was
> reported for Linux 2.4 and 2.6 some days ago (see [2] for the original
> thread in linux.kernel)  on my FreeBSD 5.2.1-p8 system.
>
> The result is a kernel panic (back trace attached).
>
> Since des@ told me in a private mail, that he could not reprocduce the
> panic on -CURRENT, I'd like to ask how to proceed from here.

I couldn't reproduce it either, but I think this is just accidental.
It takes a particular combination of FPU exceptions and masks to cause
the panic.

> Is the problem known to be fixed in current?
> Is somebody able to reproduce this on FreeBSD 5.2.1 (I am sorry,
> upgrading to -CURRENT is out of question for me)?

Try the following quick fix.  It is for -current but should work for
RELENG_5 and RELENG_4 too.  (Note that it changes fpurstor(), not
fpusave().  The patch context is not large enough to be unambigous.)
It might be incomplete.

%%%
Index: npx.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/isa/npx.c,v
retrieving revision 1.149
diff -u -2 -r1.149 npx.c
--- npx.c	6 Jun 2004 15:17:44 -0000	1.149
+++ npx.c	17 Jun 2004 03:40:18 -0000
@@ -963,4 +964,5 @@
 {

+	fnclex();
 #ifdef CPU_ENABLE_SSE
 	if (cpu_fxsr)
%%%

> Please note, that the problem does not exist on FreeBSD 4.9 (the test
> program simply dumps core (bt attached)).

I think RELENG_4 has the problem too in the (CPU_ENABLE_SSE && cpu_fxsr)
case.  Then fpusave() doesn't reset the npx state, so there may be a
pending exception from the previous process.  fpurstor() then traps if
it happens to restore a state that has the exception unmasked.  There
used to be no problem because the previous state was always put away
using the fnsave instruction, and fnsave has the side effect of
initializing a clean state, in particular a state that doesn't have
any pending exceptions.  This has been broken in 2 ways:
- in RELENG_4 and -current, fxsave is used instead of fnsave in the
  (CPU_ENABLE_SSE && cpu_fxsr) case).  fxsave doesn't have the side
  effect.
- in -current, the previous state is sometimes dropped instead of
  saved.  This is entirely in software, so it doesn't have the side
  effect.

> [1] http://linuxreviews.org/news/2004-06-11_kernel_crash/#toc1
> [2] http://groups.google.de/groups?hl=de&lr=&ie=UTF-8&frame=right&th=f7580d647408b95b&seekm=26hGq-Zr-31%40gated-at.bofh.it#link1

The bug is a little different in Linux.  Linux uses more synchronization
instructions, perhaps unnecessarily.  Its version of dropping the state
isn't entirely in software.  It has an fwait that was not preceded by an
fnclex, so it paniced if there were any pending exceptions in the state
being dropped.

The amd64 programmer's manual volume 5 needs the following fix for frstor.
In:

"To avoid generating exceptions when loading a new environment, use the
FCLEX instruction to clear the exception flags in the x87 status word
before storing that environment",

change "environment" to "x87 state" or similar, and change "storing"
to "loading".  The second change is also needed for fldenv.  The use
of FCLEX is correctly described for fldcw.

Bruce