Fixing VB6 inaccurate mathmetical results due to rounding

Posted by jay on Stack Overflow See other posts from Stack Overflow or by jay
Published on 2010-03-29T04:08:21Z Indexed on 2010/03/29 4:13 UTC
Read the original article Hit count: 233

Filed under:

vb6

|

math

Try running this in a .VBS file

MsgBox(545.14-544.94)

You get a neat little answer of 0.199999999999932!

This rounding issue also occurs unfortunately in Sin(2 * pi) since VB can only ever see the (user defined) variable pi as accurate as 3.14159265358979.

Is rounding it manually (and loosing accuracy) the only way to improve the result? What is the most effective way of dealing with this kind of problem?

© Stack Overflow or respective owner

Related posts about vb6

VB6 App + .Net component working as compiled app but not in VB6 IDE

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a VB6 App that uses a .Net component (via a .tlb reference in the VB6 app) which is working fine when executed as a compiled app, but it produces an error from the VB6 IDE a certain point when it is trying to use the .NET component. I should note that the error occurs when the .NET component… >>> More
Tutoriel VBA/VB6 : Les extensions OpenGL en VBA et VB6, par Thierry Gasperment (Arkham46)

as seen on Developper.com - Search for 'Developper.com'
Bonjour à tous! Voici un article sur la programmation des extensions OpenGL, en VB6/VBA Cet article décrit l'utilisation de quelques extensions fréquemment utilisées : - Les VBO (vertex buffer objects) pour améliorer les performances - Les textures 3D pour réaliser des textures continue sur… >>> More
Is it possible to simultaneously debug VB6 and a C++ COM dll?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a VB6 dll that is loaded by a VB6 frontend. This VB6 dll calls a C++ ATL dll via its COM interface. So, I can run from code in VB6 and I can debug in C++ also, however I can't seem to step through the VB6 code and then get into the C++ code. I feel that this should be possible. Currently I… >>> More
Writing excel file to vb6

as seen on Stack Overflow - Search for 'Stack Overflow'
i have an excel file template and i want save the written value in textboxes to the cells of excel using vb6 language. can anyone help me? >>> More
Trace not working in a .NET DLL loaded from VB6 EXE

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello I have a .NET DLL that writes to the Trace. But seems that when I call my DLL from a VB6 EXE the trace is not working. I have created an myApp.config file in the EXE folder with the trace configuration, but this does not solves the issue. I've also tried creating the Trace objects in code… >>> More

Related posts about math

how to make a function recursive

as seen on Stack Overflow - Search for 'Stack Overflow'
i have this huge function and i am wondering how to make it recursive. i have the base case which should never come true, so it should always go to else and keep calling itself with the variable t increases. any help would be great thanks def draw(x, y, t, planets): if 'Satellites' in planets["Moon"]: … >>> More
Sunrise / set calculations

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to calculate the sunset / rise times using python based on the link provided below. My results done through excel and python do not match the real values. Any ideas on what I could be doing wrong? My Excel sheet can be found under .. http://transpotools.com/sun_time.xls # Created on… >>> More
Android-Java: Constructing a triangle based on Coordinates on a map and your bearing

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm constructing a geolocation based application and I'm trying to figure out a way to make my application realise when a user is facing the direction of the given location (a particular long / lat co-ord). I've got the math figured, I just have the triangle to construct. //UPDATE So… >>> More
Math.max and Math.min outputting highest and lowest values allowed

as seen on Stack Overflow - Search for 'Stack Overflow'
so I'm trying to make a program that will output the sum, average, and smallest and largest values. I have everything basically figured out except the smallest and largest values are outputting 2147483647 and -2147483647, which I believe are the absolute smallest and largest values that Java will… >>> More
Is Programming == Math?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've heard many times that all programming is really a subset of math. Some suggest that OO, at its roots, is mathematically based. I don't get the connection. Aside from some obvious examples: using induction to prove a recursive algorithm formal correctness proofs functional languages lambda calculus asymptotic… >>> More