Search Results

Search found 680 results on 28 pages for 'precision'.

Page 4/28 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • Compare Date objects with different levels of precision

    - by brainimus
    I have a JUnit test that fails because the milliseconds are different. In this case I don't care about the milliseconds. How can I change the precision of the assert to ignore milliseconds (or any precision I would like it set to)? Example of a failing assert that I would like to pass: Date dateOne = new Date(); dateOne.setTime(61202516585000L); Date dateTwo = new Date(); dateTwo.setTime(61202516585123L); assertEquals(dateOne, dateTwo);

    Read the article

  • c++ floating point precision loss: 3015/0.00025298219406977296

    - by SigTerm
    The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == 0x40a78e0000000000 //*((unsigned __int64*)(&b)) == 0x3f30945640000000 double f = a/b;//3015/0.00025298219406977296; the result of calculation (i.e. "f") is 11917835.000000000 (*((unsigned __int64*)(&f)) == 0x4166bb4160000000) although it should be 11917834.814763514 (i.e. *((unsigned __int64*)(&f)) == 0x4166bb415a128aef). I.e. fractional part is lost. Unfortunately, I need fractional part to be correct. Questions: 1) Why does this happen? 2) How can I fix the problem? Additional info: 0) The result is taken directly from "watch" window (it wasn't printed, and I didn't forget to set printing precision). I also provided hex dump of floating point variable, so I'm absolutely sure about calculation result. 1) The disassembly of f = a/b is: fld qword ptr [a] fdiv qword ptr [b] fstp qword ptr [f] 2) f = 3015/0.00025298219406977296; yields correct result (f == 11917834.814763514 , *((unsigned __int64*)(&f)) == 0x4166bb415a128aef ), but it looks like in this case result is simply calculated during compile-time: fld qword ptr [__real@4166bb415a128aef (828EA0h)] fstp qword ptr [f] So, how can I fix this problem? P.S. I've found a temporary workaround (i need only fractional part of division, so I simply use f = fmod(a/b)/b at the moment), but I still would like to know how to fix this problem properly - double precision is supposed to be 16 decimal digits, so such calculation isn't supposed to cause problems.

    Read the article

  • Why don't computers store decimal numbers as a second whole number?

    - by SomeKittens
    Computers have trouble storing fractional numbers where the denominator is something other than a solution to 2^x. This is because the first digit after the decimal is worth 1/2, the second 1/4 (or 1/(2^1) and 1/(2^2)) etc. Why deal with all sorts of rounding errors when the computer could have just stored the decimal part of the number as another whole number (which is therefore accurate?) The only thing I can think of is dealing with repeating decimals (in base 10), but there could have been an edge solution to that (like we currently have with infinity).

    Read the article

  • Handling extremely large numbers in a language which can't?

    - by Mallow
    I'm trying to think about how I would go about doing calculations on extremely large numbers (to infinitum - intergers no floats) if the language construct is incapable of handling numbers larger than a certain value. I am sure I am not the first nor the last to ask this question but the search terms I am using aren't giving me an algorithm to handle those situations. Rather most suggestions offer a language change or variable change, or talk about things that seem irrelevant to my search. So I need a little guideance. I would sketch out an algorithm like this: Determine the max length of the integer variable for the language. If a number is more than half the length of the max length of the variable split it in an array. (give a little play room) Array order [0] = the numbers most to the right [n-max] = numbers most to the left Ex. Num: 29392023 Array[0]:23, Array[1]: 20, array[2]: 39, array[3]:29 Since I established half the length of the variable as the mark off point I can then calculate the ones, tenths, hundredths, etc. Place via the halfway mark so that if a variable max length was 10 digits from 0 to 9999999999 then I know that by halfing that to five digits give me some play room. So if I add or multiply I can have a variable checker function that see that the sixth digit (from the right) of array[0] is the same place as the first digit (from the right) of array[1]. Dividing and subtracting have their own issues which I haven't thought about yet. I would like to know about the best implementations of supporting larger numbers than the program can.

    Read the article

  • Problem installing iATKOS S3 Version 2 Snow Leopard 10.6.3 on DELL Precision T5500 Desktop

    - by Matias Dominoni
    Someone managed to install this right? I've used the following parameters: -v -x -f cpus=1 busratio=22 After installation, boot fails with a Kernel Panic. The exact error here: http://www.insanelymac.com/forum/index.php?showtopic=182609&mode=linearplus I'm aware that is very annoying installation. http://www.insanelymac.com/forum/index.php?showtopic=222386 Does anyone knows a guide to follow up or any other distrubution that works?

    Read the article

  • Dell Precision 670 no booting

    - by fristson
    Our dell running xp did not boot one day and was making a beep code I had a hard time finding on the internet. It was one of the RAM sticks so i replaced it. Before that i removed the motherboard battery to see if there was a problem with the battery. Now at start up it tells me that No Logical Drives Found and then BIOS not installed!. I have read over alot of post here and other places but everyone is talking about installation of the OS. I just want it to start with what it had. Please help.

    Read the article

  • Possible loss of precision; extracting char from string

    - by Troy
    I am getting a string from the user and then doing some checking to make sure it is valid, here is the code I have been using; char digit= userInput.charAt(0) - '0'; This had been working fine until I did some work on another method, I went to compile and have been receiving a 'possible loss of precision' error since then. What am I doing wrong?

    Read the article

  • Change the precision of all decimal columns in every table in the database

    - by Jon
    Hi all, I have a rather large database that has alot of decimal columns in alot of tables, the customer has now changed their mind and wants all the numbers (decimals) to have a precision of 3 d.p. instead of the original two. Is there any quick way of going through all the tables in a database and changing any decimal column in that table to have 3.d.p instead of 2 d.p? The db is on sql 2005. Any help would be great.

    Read the article

  • Arbitrary-precision arithmetic Explanation

    - by TT
    I'm trying to learn C and have come across the inability to work with REALLY big numbers (i.e., 100 digits, 1000 digits, etc.). I am aware that there exist libraries to do this, but I want to attempt to implement it myself. I just want to know if anyone has or can provide a very detailed, dumbed down explanation of arbitrary-precision arithmetic. Thanks!

    Read the article

  • Desktop installation fails on Dell Precision T1600 with Nvidia Quadro card?

    - by Masao
    I'm trying to install Ubuntu 12.04 Desktop on a Dell Precision T1600, however, I get a blank screen with only a blinking cursor in the top left hand corner (this is after the grey screen with the small Ubuntu logo at the bottom). If I remove the Nvidia Quadro 2000 card and plug a monitor into the built-in VGA port, then the installation works fine. Also, I've tried an Nvidia Quadro 2000D by PNY, but I get the same problem. Has anyone installed Ubuntu 12.04 Desktop on a Dell Precision workstation?

    Read the article

  • Oracle PeopleSoft Mobile Inventory Management, une solution mobile pour améliorer précision des stocks et productivité des utilisateurs

    Oracle PeopleSoft Mobile Inventory Management Une application mobile pour améliorer la précision des stocks et la productivité des utilisateurs Oracle possède déjà un portefeuille très complet de solutions de gestion des stocks et de solutions mobiles. Ce portefeuille s'enrichit encore plus aujourd'hui avec une nouvelle solution : Oracle PeopleSoft Mobile Inventory Management. Il s'agit en fait d'une nouvelle application de la suite Oracle PeopleSoft Enterprise Supply Chain Management qui permet d'améliorer la précision des stocks et la productivité des opérateurs en automatisant les transactions mobiles de gestion pour les utilisateurs de terminaux mobiles équipés de Microsoft W...

    Read the article

  • Why doesn't MySQL support millisecond precision?

    - by Byron Whitlock
    So I just found the most frustrating bug ever in MySQL. Apparently the TIMESTAMP field, and supporting functions do not support any greater precision than seconds!? So I am using PHP and Doctrine, and I really need those microseconds (I am using the actAs: [Timestampable] property). I found a that I can use a BIGINT field to store the values. But will doctrine add the milliseconds? I think it just assigns NOW() to the field. I am also worried the date manipulation functions (in SQL) sprinkled through the code will break. I also saw something about compiling a UDF extension. This is not an acceptable workaround because I or a future maintainer will upgrade and poof, change gone. Has anyone found a suitable workaround?

    Read the article

  • best way to output a full precision double into a text file

    - by flevine100
    Hi, I need to use an existing text file to store some very precise values. When read back in, the numbers essentially need to be exactly equivalent to the ones that were originally written. Now, a normal person would use a binary file... for a number of reasons, that's not possible in this case. So... do any of you have a good way of encoding a double as a string of characters (aside from increasing the precision). My first thought was to cast the double to a char[] and write out the chars. I don't think that's going to work because some of the characters are not visible, produce sounds, and even terminate strings ('\0'... I'm talkin to you!) Thoughts?

    Read the article

  • Why doesn't MySQL support millisecond / microsecond precision?

    - by Byron Whitlock
    So I just found the most frustrating bug ever in MySQL. Apparently the TIMESTAMP field, and supporting functions do not support any greater precision than seconds!? So I am using PHP and Doctrine, and I really need those microseconds (I am using the actAs: [Timestampable] property). I found a that I can use a BIGINT field to store the values. But will doctrine add the milliseconds? I think it just assigns NOW() to the field. I am also worried the date manipulation functions (in SQL) sprinkled through the code will break. I also saw something about compiling a UDF extension. This is not an acceptable because I or a future maintainer will upgrade and poof, change gone. Has anyone found a suitable workaround?

    Read the article

  • Possible loss of precision / [type] cannot be dereferenced

    - by Samuel
    I have been looking around a lot but i simply can't find a nice solution to this... Point mouse = MouseInfo.getPointerInfo().getLocation(); int dx = (BULLET_SPEED*Math.abs(x - mouse.getX()))/ (Math.abs(y - mouse.getY()) + Math.abs(x - mouse.getX()))* (x - mouse.getX())/Math.abs(x - mouse.getX()); In this constellation i get: Possible loss of precision, when i change e.g (x - mouse.getX()) to (x - mouse.getX()).doubleValue() it says double cannot be dereferenced, when i add intValue() somewhere it says int cannot be dereferenced. What's my mistake? [x, y are integers | BULLET_SPEED is a static final int] Thanks!

    Read the article

  • Floating point precision nuances.

    - by user247077
    Hi, I found this code in NVIDIA's cuda SDK samples. void computeGold( float* reference, float* idata, const unsigned int len) { reference[0] = 0; double total_sum = 0; unsigned int i; for( i = 1; i < len; ++i) { total_sum += idata[i-1]; reference[i] = idata[i-1] + reference[i-1]; } // Here it should be okay to use != because we have integer values // in a range where float can be exactly represented if (total_sum != reference[i-1]) printf("Warning: exceeding single-precision accuracy. Scan will be inaccurate.\n"); } (C) Nvidia Corp Can somebody please tell me a case where the warning would be printed, and most importantly, why. Thank you very much.

    Read the article

  • Setting precision on std::cout in entire file scope - C++ iomanip

    - by Ivan
    Hi all, I'm doing some calculations, and the results are being save in a file. I have to output very precise results, near the precision of the double variable, and I'm using the iomanip setprecision(int) for that. The problem is that I have to put the setprecision everywhere in the output, like that: func1() { cout<<setprecision(12)<<value; cout<<setprecision(10)<<value2; } func2() { cout<<setprecision(17)<<value4; cout<<setprecision(3)<<value42; } And that is very cumbersome. Is there a way to set more generally the cout fixed modifier? Thanks

    Read the article

  • Reading variable with double float precision from a text file with gnuplot

    - by user3636322
    I have a text file, containing data in 3 columns like below: 0.0100000000 | 0.0058077299 | -0.0000000288 0.0110000000 | 0.0075128707 | -0.0000000373 0.0120000000 | 0.0093579693 | -0.0000000465 I want to get the variables from this file in gnuplot and use them to draw graphs: What I exactly do is like below (e.g: to pick the variable from row 2 column 3): ii= 2 a_0 = system("awk '{ if (NR == " . ii . ") printf \"%f\", $3}' " .datafile) a_0 = a_0+0. but what is written as a_0 is zero! How can I increase the precision to get the exact value?

    Read the article

  • Using * Width & Precision Specifiers With boost::format

    - by John Dibling
    I am trying to use width and precision specifiers with boost::format, like this: #include <boost\format.hpp> #include <string> int main() { int n = 5; std::string s = (boost::format("%*.*s") % (n*2) % (n*2) % "Hello").str(); return 0; } But this doesn't work because boost::format doesn't support the * specifier. Boost throws an exception when parsing the string. Is there a way to accomplish the same goal, preferably using a drop-in replacement?

    Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >