开发者

Flex/AS3 very strange simple Number operation issue

开发者 https://www.devze.com 2023-03-09 05:57 出处:网络
My problem is quite simple to describe, in flex: 0.8 - 0.2 = 0.6000000000000001 Anyone got this before, I\'m sure the first two members are 0.8 开发者_开发知识库and 0.2 and are Number class, why wo

My problem is quite simple to describe, in flex:

0.8 - 0.2 = 0.6000000000000001

Anyone got this before, I'm sure the first two members are 0.8 开发者_开发知识库and 0.2 and are Number class, why would this happen??

And another thing, I cast the value from * type like this:

var value:*=0.8;
var castValue:Number = Number(value);

But nothing more, if I trace value I get 0.8 not 0.800000000000001 or something.


This is a fairly common problem in all languages because of Floating Point Number Precision. There's not much you can do about as it is an error value that's possible on all machine.

What you can do is set a precision that you want from the number:

trace(castValue.toFixed(5)); // Gives five decimals after the point

Often times, Flash rounds off these number for you in the conversion of Number into a String (for visual displaying) which removes most of these errors from being viewable to the user.


Essentially, there are some numbers that just can't be represented exactly in binary.

Although this page about floating point numbers is about Java and not Actionscript, the difficulty exists in a lot of languages. There's a section in that article (see "Rounding errors") that mentions the problem with numbers like .6

It's a long read, but I found it quite helpful (and not as long or math-heavy as this page about Floating-Point Arithmetic).

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号