UB in various hypot() implementations (left-shifting a negative number)
jwalden at mit.edu
Wed Nov 13 22:43:13 UTC 2019
Just wanted to note here that I filed https://reviews.freebsd.org/D22354 to fix a bit of undefined behavior in the hypot() function implementations.
`hypot(x, y)` computes `sqrt(x*x + y*y)`, with IEEE-754-aware precision. For very large or small `x` or `y`, the naive implementation would lose precision -- so in such cases the calculation is performed after multiplying the numbers by a very large (or very small) power of two, then the multiplication is undone at end.
Undoing the multiplication involves multiplying a quantity `w` by `2**k`, where `k` (which may be positive or negative) was computed depending on the particular `x` and `y` values provided. Current algorithms generally take `t1=0.0`, extract the high word, add `k<<20` or `k<<23` to it to appropriately bump the exponent, then overwrite `t1`'s high word. But it seems equally effective to take `t1=0.0`, then write `(1023+k)<<20` or `(127+k)<<23` to it for identical effect -- and `k` is never so negative to compute a negative value. My changes do this. (I wish there were named constants I could use for all these numbers, but as there don't seem to be any, magic numbers seems like the best I can do.)
More information about the freebsd-numerics