floating-accuracy

相关标签：javascript jquery android 多少钱 iPhone

"Dynamic" display of floats?

I\'m trying to write an app for iphone in ObjectiveC. I need to display a statustext that tells the user the value of a few variables. These values are 开发者_开发技巧changeable by dragging a marker w[详细]

2023-02-27 08:37 分类：问答
How accurate/precise is java.lang.Math.pow(x, n) for large n?

I would like to calculate (1.0-p)^n where p is a double be开发者_如何学Ctween 0 and 1 (often very close to 0) and n is a positive integer that might be on the order of hundreds or thousands (perhaps l[详细]

2023-02-22 13:11 分类：问答
Ruby number precision with simple arithmetic

I\'m learning Ruby for fun, and for creating websites also (but that\'s irrelevant). While playing with it, i noticed something \"weird\"[详细]

2023-02-19 17:09 分类：问答
Addition error with ruby-1.9.2 [duplicate]

This question already has answers here: 开发者_运维问答ruby: converting from float to integer in ruby produces strange results[详细]

2023-02-19 04:23 分类：问答
How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basic开发者_运维百科ally the idea is that if I execute:[详细]

2023-02-16 10:50 分类：问答
ruby: converting from float to integer in ruby produces strange results

ree-1.8.7-2010.02 :003 > (10015.8*100.0).to_i => 1001579 ree-1.8.7-2开发者_如何学Go010.02 :004 > 10015.8*100.0[详细]

2023-02-16 02:39 分类：问答
.NET trouble when using doubles

In .NET, when I subtract 1.35 from 1.35072 it shows .000719999999. How could I get .00072 when using a double?[详细]

2023-02-15 16:00 分类：问答
Why would a variable of type double have an unexpected result?

My sanity check fails because a double variable does not contain the expected result, it\'s really bizarre.[详细]

2023-02-14 05:29 分类：问答
Rounded numbers returned as '0.999999999992345' sometimes

I have a report that should return something along the lines of SELECT brand, ROUND(SUM(count * price) / SUM(count), 2)[详细]

2023-02-11 03:14 分类：问答
Floating point comparison irreproducibility

I and my Ph.D. student have encountered a problem in a physics data analysis context that I could use some insight on. We have code that analyzes data from one of the LHC experiments that gives irrepr[详细]

2023-02-10 11:53 分类：问答