8 replies to this topic

[5te4lthX]

My VB program thinks that
0.95 <> (0.1 * 7) + (0.25 * 1)
is true, when it definitely isn't. When I had VB 6.0 display the value of
(0.1 * 7) + (0.25 * 1)
it displayed 0.95

Anyone know why this is happening, and if the same happens to you?

[scsefrmr]

VBA inside Excel 2007 can't do math either...

Typing in the immediate window:
? 0.95 <> (0.1 * 7) + (0.25 * 1)

Excel says:
True

So it shows its not just VB :(

[WingedPanther]

If you display the result of (0.1*7) + (0.25*1) you will probably get something like 0.95000000001 which does not equal 0.95.

[scsefrmr]

WingedPanther said:

If you display the result of (0.1*7) + (0.25*1) you will probably get something like 0.95000000001 which does not equal 0.95.

That is exactly what I thought... but Excel says 0.95 as the total when I try it that way. Very odd.

Tried it in OpenOffice Calc (Laptop is Excel/Word free :)) and when I put this into a cell...

=(0.1*7) + (0.25 * 1) = 0.95

It responded with TRUE. Sounds like an MS thing like the 65535 bug in Excel 2007.

Edited by scsefrmr, 18 June 2008 - 11:32 AM.

[WingedPanther]

You have to remember that Excel rounds the displayed result by default, so you may not see the difference.

[scsefrmr]

With Excel doing that, it is a bit misleading to the user then isn't it?
Especially when there are examples like above, as it's only going to drive people crazy wondering why it doesn't work properly.

One of Excel's jobs is to do calculations, so it should really do it correctly.

[WingedPanther]

The issue lies in the internal representation of floating point values in binary and their imprecise (limited precision) conversion to decimal for display.

[Xav]

Well, that shut them up! :D

[adora25]

It likes fractions better. Here is an article I found on this issue:
Troubleshooting Data Types

Here is some code that works:
(95 / 100) <> (1 / 10 * 7) + (1 / 4 * 1)