Whats up with KCalc?

[John Oxley]

On Wed, Jul 13, 2005 at 01:13:24AM -0500, Nikolas Britton wrote:
> On 7/12/05, Nikolas Britton <nikolas.britton at gmail.com> wrote:
> Ok... this thing is WAY OFF... if you type in 85.49 and then hit the +
> key the thing automatically jumps to
> 85.48999999999999488409230252727866172791!!! My SuSE box shows "85.49"
> but if you copy and paste the number into a text editor it shows
> 85.48999... too.

IANA Mathematician, but my guess would be that there is a rounding
error.  KCalc as you have all said uses 96bit long double precision.
This means that floating point numbers are stored as 96bit doubles.
And yes 46 is a floating point (46.000).  My varsity days are a bit
hazy, but I believe a double is stored as X bits ^ Y bits.  Not sure of
the ratios of X and Y though.  This means you can only get a certain
accuracy.  If X is 8 for example, the accuracy is 1/(268435456) =~
3.725290e-09.  

Therefore your smallest unit number is that and any other number will be
a multiple of that, raised to a power which is where the Y bits comes
in.  Oh I forgot to mention that X+Y = 96.

This is probably a very bad explanation and hopefully someone can do it
better :)

-John
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 2198 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20050713/d3bbd0f5/smime.bin