Undesirable FPU tag word value after PT_SETFPREGS on i386

[Konstantin Belousov]

On Fri, Jun 16, 2017 at 03:19:35PM +0530, karnajit wangkhem wrote:
> Hi All,
> 
> Please help me understand why NaN is reported after PT_SETFPREGS. As a
> result of this, if the inferior process contains an float operations, they
> result in NaN.
> This behavior is not seen in case of 32/64 bit binary on amd64.
> 
> My env
> kern.ostype: FreeBSD
> kern.osrelease: 11.0-RELEASE-p1
> kern.version: FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 03:40:55 UTC
> 2016
>     root at releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC
> hw.machine_arch: i386
> 
> Result of the below program on an i386 box
> FPU TAG = ffff
> FPU TAG = fe00  -> 11 11 11 10 00 00 00 00

FreeBSD 11 started translating between abridged and full FPU tags, but
the translation appeared to be broken.  Try this.

diff --git a/sys/i386/isa/npx.c b/sys/i386/isa/npx.c
index 5ca1ee507fe..de29adc4101 100644
--- a/sys/i386/isa/npx.c
+++ b/sys/i386/isa/npx.c
@@ -1108,7 +1108,7 @@ npx_fill_fpregs_xmm1(struct savexmm *sv_xmm, struct save87 *sv_87)
 		sv_87->sv_ac[i] = sv_xmm->sv_fp[i].fp_acc;
 		if ((penv_xmm->en_tw & (1 << i)) != 0)
 			/* zero and special are set as valid */
-			penv_87->en_tw &= ~(3 << i);
+			penv_87->en_tw &= ~(3 << i * 2);
 	}
 }
 
@@ -1143,7 +1143,7 @@ npx_set_fpregs_xmm(struct save87 *sv_87, struct savexmm *sv_xmm)
 	penv_xmm->en_tw = 0;
 	for (i = 0; i < 8; ++i) {
 		sv_xmm->sv_fp[i].fp_acc = sv_87->sv_ac[i];
-		if ((penv_87->en_tw && (3 << i)) != (3 << i))
+		if ((penv_87->en_tw & (3 << i * 2)) != (3 << i * 2))
 		    penv_xmm->en_tw |= 1 << i;
 	}
 }