Print 0.1 + 0.2 in most programming languages and you get 0.30000000000000004. Nothing is broken; this is how computers store real numbers. This guide explains the IEEE 754 double-precision format used by SciKey and most numerical software, how its limits show up in results, and how the calculator handles them.
1. IEEE 754 double precision
A double (binary64) uses 64 bits.
| Part | Bits | Role |
|---|---|---|
| Sign | 1 | Positive or negative |
| Exponent | 11 | Which power of 2 |
| Fraction | 52 | Significant digits (53 in effect, with a hidden leading 1) |
53 significant bits correspond to about 15 to 17 decimal digits. The largest representable value is about 1.797×10³⁰⁸, and the gap between 1 and the next double (machine epsilon) is 2⁻⁵² ≈ 2.22×10⁻¹⁶.
2. 0.1 has no exact binary form
Just as 1/3 = 0.333… never ends in decimal, 1/10 never ends in binary: 0.1 = 0.0001100110011…₂ repeating forever. It is rounded to 53 bits, so the stored value is actually
0.1000000000000000055511151231257827…
0.2 is also stored slightly high. Their sum carries both errors to 0.3000000000000000444…, which is a different double from the one that stores 0.3 (0.2999999999999999888…). The shortest decimal that identifies the sum is therefore 0.30000000000000004.
For the same reason, 0.1×3 is also 0.30000000000000004. Decimals whose denominators are powers of two, such as 0.5, 0.25 and 0.375, are exact in binary and carry no error.
3. Why the calculator still shows 0.3
SciKey does not display the raw internal value; it rounds to your significant-digit setting (12 by default). Rounded to 12 digits, 0.30000000000000004 is 0.300000000000, i.e. 0.3. The error sits in the 17th digit and never reaches the display.
Handheld scientific calculators work the same way: they typically compute with around 15 internal digits and display 10, hiding tiny trailing errors at the display stage. Only the display changes, so the stored value is still 0.30000000000000004. Carrying it into further calculations usually leaves errors around 10⁻¹⁶, which is irrelevant in practice.
4. Why sin(π) is not zero
In RAD mode, sin(π) should be exactly 0. But π is irrational, and the double closest to π is about 1.22×10⁻¹⁶ smaller than the true value. Near π, sine has slope −1, so sin(double π) ≈ 1.2246×10⁻¹⁶. The sine itself was computed accurately; the input was never exactly π.
SciKey corrects for this with the following rules:
- In DEG and GRAD modes, the argument is reduced exactly by 360° (400 gon), and multiples of 90° return exact values:
sin(180)= 0,cos(90)= 0, andtan(90)is undefined, so it returns an error. - In RAD mode, if the argument matches an integer multiple of π within a relative error of 10⁻¹³, sin returns 0; if it matches an odd multiple of π/2, cos returns 0 and tan returns an error.
So sin(π) shows 0 in SciKey. The correction applies only near multiples of π and does not affect any other calculation.
5. Big numbers: factorials and overflow
Factorials grow fast. 170! ≈ 7.25741561531×10³⁰⁶ still fits in a double, but 171! is about 1.24×10³⁰⁹, beyond the maximum of 1.797×10³⁰⁸. That is why SciKey's n! accepts integers from 0 to 170 and 171! returns an overflow error.
In practice, large factorials usually appear inside ratios. The combination C(200, 3) is 200!/(3!·197!), but nCr(200, 3) = 1313400 computes it directly without huge intermediate values. In probability, taking logarithms and adding them is another common technique.
6. The integer limit: 2⁵³
With 53 significant bits, every integer up to 2⁵³ = 9007199254740992 is exact. Beyond that, doubles skip integers. 2⁵³ + 1 cannot be represented and rounds back to 2⁵³, so 2^53+1-2^53 gives 0, not 1: the added 1 vanished when the sum was stored.
This is why SciKey shows binary, octal and hex only for integers with absolute value up to 2⁵³; beyond that, the low-order digits are not guaranteed. It is also why card numbers or IDs longer than 16 digits change their last digits when treated as numbers.
7. Catastrophic cancellation: the most dangerous error
Subtracting two nearly equal numbers wipes out the leading digits and leaves mostly error. This is called catastrophic cancellation.
Example 1. (1+1E-15)-1 should be 10⁻¹⁵, but double precision gives 1.11022302463×10⁻¹⁵. Storing 1 + 10⁻¹⁵ introduces a rounding error of order 10⁻¹⁶, and after subtracting 1 that error is 11% of the result.
Example 2. For x = 10⁸, √(x²+1) − x is about 5×10⁻⁹. Computed directly, √(10¹⁶+1) is indistinguishable from 10⁸ and the result is 0. Rationalising gives an equivalent form without the subtraction:
√(x²+1) − x = 1 / (√(x²+1) + x)
Store 10⁸ in x and 1/(sqrt(x^2+1)+x) returns 5×10⁻⁹ accurately.
Example 3. 1 − cos(10⁻⁸) is about 5×10⁻¹⁷, but cos(10⁻⁸) rounds to exactly 1 and the direct result is 0. Using the identity 1 − cos x = 2 sin²(x/2), 2sin(0.5E-8)^2 gives 5×10⁻¹⁷ (RAD mode).
No number of display digits fixes this. You have to rewrite the formula.
8. Practical tips
- Do not test for exact equality. To compare two computed results, check whether their difference is small enough (for example, a relative error of 10⁻¹²).
- Handle money in whole units. Work in the smallest currency unit as an integer so fractional errors cannot accumulate.
- Avoid subtracting nearly equal numbers. Rationalise, use identities or rearrange to remove the subtraction.
- Work with ratios for big numbers. When factorials or powers explode, use nCr, logarithms or cancel terms first.
- Do not round intermediate results. Double-precision error is around 10⁻¹⁶, but rounding by hand to four digits introduces errors around 10⁻⁴. The bigger errors usually come from people, not the machine.
- Try raising the display to 15 digits. If a result looks odd, show more digits to see the internal value and tell a trailing-digit error from a genuinely wrong formula.