floating-accuracy
"Dynamic" display of floats?
I\'m trying to write an app for iphone in ObjectiveC. I need to display a statustext that tells the user the value of a few variables. These values are 开发者_开发技巧changeable by dragging a marker w[详细]
2023-02-27 08:37 分类:问答How accurate/precise is java.lang.Math.pow(x, n) for large n?
I would like to calculate (1.0-p)^n where p is a double be开发者_如何学Ctween 0 and 1 (often very close to 0) and n is a positive integer that might be on the order of hundreds or thousands (perhaps l[详细]
2023-02-22 13:11 分类:问答Ruby number precision with simple arithmetic
I\'m learning Ruby for fun, and for creating websites also (but that\'s irrelevant). While playing with it, i noticed something \"weird\"[详细]
2023-02-19 17:09 分类:问答Addition error with ruby-1.9.2 [duplicate]
This question already has answers here: 开发者_运维问答ruby: converting from float to integer in ruby produces strange results[详细]
2023-02-19 04:23 分类:问答How to avoid floating point precision errors with floats or doubles in Java?
I have a very annoying problem with long sums of floats or doubles in Java. Basic开发者_运维百科ally the idea is that if I execute:[详细]
2023-02-16 10:50 分类:问答ruby: converting from float to integer in ruby produces strange results
ree-1.8.7-2010.02 :003 > (10015.8*100.0).to_i => 1001579 ree-1.8.7-2开发者_如何学Go010.02 :004 > 10015.8*100.0[详细]
2023-02-16 02:39 分类:问答.NET trouble when using doubles
In .NET, when I subtract 1.35 from 1.35072 it shows .000719999999. How could I get .00072 when using a double?[详细]
2023-02-15 16:00 分类:问答Why would a variable of type double have an unexpected result?
My sanity check fails because a double variable does not contain the expected result, it\'s really bizarre.[详细]
2023-02-14 05:29 分类:问答Rounded numbers returned as '0.999999999992345' sometimes
I have a report that should return something along the lines of SELECT brand, ROUND(SUM(count * price) / SUM(count), 2)[详细]
2023-02-11 03:14 分类:问答Floating point comparison irreproducibility
I and my Ph.D. student have encountered a problem in a physics data analysis context that I could use some insight on. We have code that analyzes data from one of the LHC experiments that gives irrepr[详细]
2023-02-10 11:53 分类:问答