UB in various hypot() implementations (left-shifting a negative, number)

[Steve Kargl]

On Mon, Nov 18, 2019 at 09:55:03PM -0800, Jeff Walden wrote:
> On 11/16/19 1:14 PM, Steve Kargl wrote:
> > Well, clearly, the patch to e_hypotl.c is wrong.  It
> > clear the significand when t1 = 0 whereas t1 = 1 leaves
> > one bit set in the significand.  Simply looking at the
> > value of t1 under a poor man's debugger shows the 
> > difference.  Adding "printf("%Le %La\n", t1, t1);" after
> > the SET_HIGH_WORD gives
> > 
> > 2.962347e-2493 0x1p-8280  <-- t1 = 1
> > 0.000000e+00 0x1p-8280    <-- t1 = 0
> > 
> > for hypotl(ldexpl(1.1,-16000), ldexpl(2.1, -16000)).
> 
> Hrm.  My understanding based on an earlier line in e_hypotl.c
> was that that change should be identical, but something mildly
> subtle is going on and my understanding of |long double| is not
> quite as good as I thought it was.  I've removed that file's
> changes and left just the original two in, then resubmitted the diff.
> 
> Jeff

In general, float, double, and long double all use the same
general approach where a value is scaled if required.  The
details of unraveling the scaling do differ.  float operates
on the 32-bit integer representation of the float value.  double
operates on the 32-bit integer portion of the double that contains
the exponent.  For at least ld80 long double, the macro SET_HIGH_WORD
assigns directly to the bitfield of struct within a union.  The bitfield
represents the sign and exponent.  So, setting t1=0 clears the significand,
which has an undesirable effect. 

-- 
Steve