开发者

Why is it in vb.net if I assign a number to a single variable it does not equal the same value

开发者 https://www.devze.com 2023-01-23 08:22 出处:网络
I have been searching the internet trying to understand in its simplist form why this behavior happens.

I have been searching the internet trying to understand in its simplist form why this behavior happens.

开发者_运维问答
Dim mysingle As Single = 456.11
Dim mybool As Boolean = mysingle = 456.11

In the lines above mybool becomes false. I found this behavior when putting the single into a double I found extra digits showing. The .net documentations states a single is an approimate value :S

I gatehr a single is a 32bit floating point number? But why are extra digits appearing when I have explicitly said what the number is.. surely the memory should store that numbers either side of my number are 0 to fill up the memory location?!

My brain is fried on this one :(


The value being compared in the second statement isn't considered a Single, rather it's being treated as a Double. Since you're using VB.NET you can suffix it with a ! to force it to a Single and this will return True:

Dim mysingle As Single = 456.11
Dim mybool As Boolean = mysingle = 456.11!


I recommend reading: What Every Computer Scientist Should Know About Floating-Point Arithmetic.

It explains the issues with precision in floating point math in detail. For a simpler variation on the above, see The Floating Point Guide.


Another good reference for understanding floating-point in .NET: https://csharpindepth.com/articles/FloatingPoint

Let me highlight his point on comparisons:

One consequence of all of this is that you should very, very rarely be comparing binary floating point numbers for equality directly. [...] One simple way of doing this is to subtract one from the other, use Math.Abs to find the absolute value of the difference, and then check whether this is lower than a certain tolerance level.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号