standards/130067: Wrong numeric limits in system headers?

[Bruce Evans]

On Tue, 30 Dec 2008, Vaclav Haisman wrote:

> There seems to be a problem with definition of long double limits on FreeBSD i386/6.x.
>
> shell::wilx:~/packed_vector> echo | g++ -dD -E -  | sort | grep LDBL_MAX
> #define __LDBL_MAX_10_EXP__ 4932
> #define __LDBL_MAX_EXP__ 16384
> #define __LDBL_MAX__ 1.1897314953572316e+4932L
>
> shell::wilx:~/packed_vector> fgrep -rn LDBL_MAX /usr/include
> [...]
> /usr/include/machine/float.h:75:#define LDBL_MAX_EXP    16384
> /usr/include/machine/float.h:76:#define LDBL_MAX        1.1897314953572317650E+4932L
> /usr/include/machine/float.h:77:#define LDBL_MAX_10_EXP 4932
> /usr/include/float.h:75:#define LDBL_MAX_EXP    16384
> /usr/include/float.h:76:#define LDBL_MAX        1.1897314953572317650E+4932L
> /usr/include/float.h:77:#define LDBL_MAX_10_EXP 4932
>
> Notice the difference in definition of LDBL_MAX, the values in system
> headers are tiny bit larger than that defined by GCC itself.
> ...
> machine/float.h:76:#define LDBL_MAX        1.1897314953572317650E+4932L
> float.h:76:#define LDBL_MAX                1.1897314953572317650E+4932L
> GCC: __LDBL_MAX__                          1.1897314953572316e+4932L

This has never worked.  However, since gcc became aware of the limited
precision of long doubles under FreeBSD a few years ago, it shouldn't
be as broken as it is.  Gcc defines __LDBL_MAX__ to have the correct
value (1-2^-53)*2^1024 (rounded to 17 digits, which is enough for the
actual precision of 53 bits), while FreeBSD defines LDBL_MAX as
(1-2^-64)*2^1024 (rounded to 20 digits, which is perhaps not quite
enough for the non-actual precision of 64 bits (either this 20 or
DECIMAL_DIG's value of 21 is dubious).  The wrong value in FreeBSD
may have worked accidentally a few years ago, but now hit has no
chance of working:
- a few years ago: gcc didn't know about FreeBSD's limited precision,
   so it evaluated the constant in 64-bit precision and got precisely
   (1-2^-64)*2^1024 (only the decimal constant is imprecise).  Static
   initialization to this value preserved the value.  Actual use of
   the value caused it to be rounded to 53 bits.  I think that still
   made it Infinity in most cases.
- now: gcc evaluates it in 53-bit precision and gets Infinity for it
   consistently.

Many other long double constants in FreeBSD's <float.h> have never
worked, and are much further from neing correct, but are fixed in
gcc:

>From gcc4.2 -E -dM:
% #define __LDBL_MAX__ 1.1897314953572316e+4932L
% #define __LDBL_MAX_EXP__ 16384
% #define __LDBL_HAS_INFINITY__ 1
% #define __LDBL_MIN__ 3.3621031431120935e-4932L
% #define __LDBL_HAS_QUIET_NAN__ 1
% #define __LDBL_HAS_DENORM__ 1
% #define __LDBL_EPSILON__ 2.2204460492503131e-16L
% #define __LDBL_DIG__ 15
% #define __LDBL_MANT_DIG__ 53
% #define __LDBL_MIN_EXP__ (-16381)
% #define __LDBL_MAX_10_EXP__ 4932
% #define __LDBL_DENORM_MIN__ 7.4653686412953080e-4948L
% #define __LDBL_MIN_10_EXP__ (-4931)

>From float.h:
% #define LDBL_MANT_DIG	64

Wrong; should be 53.  All the other errors are derived from this
(see the definitions of FLT_* for the derivations -- p must be 53
but is 64).

% #define LDBL_EPSILON	1.0842021724855044340E-19L

Wrong: too small by a factor of 2^13.

% #define LDBL_DIG	18

Wrong: unnecessarily large (fairly harmless).

% #define LDBL_MIN_EXP	(-16381)

Correct.

% #define LDBL_MIN	3.3621031431120935063E-4932L

Correct (just has more precision than needed for rounding to 53 bits).

% #define LDBL_MIN_10_EXP	(-4931)
% #define LDBL_MAX_EXP	16384

Correct.

% #define LDBL_MAX	1.1897314953572317650E+4932L

Wrong: see above.

% #define LDBL_MAX_10_EXP	4932

Correct.

Bruce