16 Feb 2010
Old rounding errors never die, they just fade into images…

My friend Matt Owen posted a snapshot of the stock results on his Facebook wall today, which showed some interesting results on NASDAQ.

It’s amusing to see little math slip ups appear from time to time like this on commercial web sites.  Seeing this error reminded me of other experiences I have had in coding where numbers would throw wildly inaccurate answers every 3rd blue moon or so.

Since I’ve experienced this type of error before, I was thinking to myself, “Wow, that floating point rounding error is still around !”  It’s probably not the problem with these invalid values, but you might be asking yourself “what’s a floating point rounding error?”  Not everyone who has learned coding is aware of it, and it’s more likely to affect calculations requiring precise levels of resolution, rather than applications written for everyday use.

Nonetheless, it’s a good thing to be aware of.  Back in my junior high school years, our math teacher used the following code in BASIC to demonstrate the problem:

1
2
3
4
5

X = 0.0
For I = 1 To 1000
X = X + 0.1
Next I
Print X

The code’s purpose is simple: start at 0, and add 1/10th to it for 1,000 times.  The expected result is the same as 0.1 * 1,000… or 100.

Since X is not type cast, the parser would find a suitable default floating-point type for X due to the decimal point in the number.  Depending on the computer it ran on, it would produce slightly different results, but never the exact 100.00 which is the correct result.  On an HP2000C mainframe, the answer would be 99.996, while on a Radio Shack TRS-80 the answer was 100.02.

So out of curiosity, I had to see what more modern languages like VBScript and C# would produce.  Sure enough, neither produce the expected result of 100.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

x = cdbl (0.0)
for i = 1 to 1000
x = x + cdbl(0.1)
next
WScript.Echo x
 
y = csng (0.0)
for i = 1 to 1000
y = y + csng(0.1)
next
WScript.Echo y
 
z = 0.0
for i = 1 to 1000
z = z + 0.1
next
WScript.Echo z

The result is:
99.9999999999986
99.99905
99.9999999999986 (uses a default cast of double)

In CSharp, create a windows form with a label and a button, and insert this code at the button click event:

1
2
3

double total = 0.0;
for (int i = 1; i <= 1000; i++) total += 0.1;
this.label1.Text = total.ToString();

The result is: 99.9999999999986

The problem is widely known and well-documented. Still, I like the way my high school math teacher put it.  This is a card to put in your pocket, and play when you encounter the person who swears up and down that math is precise and computers are perfect.  You can even use it on a person who laughs at the occasional “numbers gone wild” event in their portal… as in the stock result above.   🙂