How do calculators work with precision?

Posted by zoul on Stack Overflow See other posts from Stack Overflow or by zoul
Published on 2010-04-26T13:12:47Z Indexed on 2010/04/26 13:13 UTC
Read the original article Hit count: 258

Hello! I wonder how calculators work with precision. For example the value of sin(M_PI) is not exactly zero when computed in double precision:

#include <math.h>
#include <stdio.h>

int main() {
    double x = sin(M_PI);
    printf("%.20f\n", x); // 0.00000000000000012246
    return 0;
}

Now I would certainly want to print zero when user enters sin(p). I can easily round somewhere on 1e–15 to make this particular case work, but that’s a hack, not a solution. When I start to round like this and the user enters something like 1e–20, they get a zero back (because of the rounding). The same thing happens when the user enters 1/10 and hits the = key repeatedly — when he reaches the rounding treshold, he gets zero.

And yet some calculators return plain zero for sin(p) and at the same time they can work with expressions such as (1e–20)/10 comfortably. Where’s the trick?

© Stack Overflow or respective owner

Related posts about calculator

Related posts about floating-point