开发者

converting values after decimal point to lesser precision [duplicate]

开发者 https://www.devze.com 2023-03-01 14:51 出处:网络
This question already has answers here: Closed 11 years ago. Possible Duplicate: Why can't decimal numbers be represented exactly in binary?
This question already has answers here: Closed 11 years ago.

Possible Duplicate:

Why can't decimal numbers be represented exactly in binary?

I have a probl开发者_开发知识库em,am solving a simplex problem,while several iterations,in 1 iteration the value turned out to be .400000,now i am multiplying this value with 100 to obtain 40.000000.Now am subtracting this value with 40,but the solution is coming out to be .000001.Please help me out with this,am not able to figure out how this value is turning out to .000001?? As this .000001 is giving problems while checking value,i.e i am comparing with 0,but compiler is treating this as value greater than 0,and am getting wrong answer.How can i correct this error?


It has to do with the machine precision in Floating points. Have a read about it here: http://en.wikipedia.org/wiki/Floating_point#Machine_precision

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号