Search Results

Search found 6604 results on 265 pages for 'double precision'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 2/265 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • x86-64 long double precision

    - by aaa
    hello. What is the actual precision of long double on Intel 64-bit platforms? is it 80 bits padded to 128 or actual 128 bit? if former, besides going gmp, is there another option to achieve true 128 precision?

    Read the article

  • Difference between float and double

    - by VaioIsBorn
    I know, i've read about the difference between double precision and single precision etc. But they should give the same results on most cases right ? I was solving a problem on a programming contest and there were calculations with floating point numbers that were not really big so i decided to use float instead of double, and i checked it - i was getting the correct results. But when i send the solution, it said only 1 of 10 tests was correct. I checked again and again, until i found that using float is not the same using double. I put double for the calculations and double for the output, and the program gave the SAME results, but this time it passed all the 10 tests correctly. I repeat, the output was the SAME, the results were the SAME, but putting float didn't work - only double. The values were not so big too, and the program gave the same results on the same tests both with float and double, but the online judge accepted only the double-provided solution. Why ? What is the difference ?

    Read the article

  • How to calculate a operation's time in micro second precision

    - by Sanjeet Daga
    I want to calculate performance of a function in micro second precision on Windows platform. Now Windows itself has milisecond granuality, so how can I achieve this. I tried following sample, but not getting correct results. LARGE_INTEGER ticksPerSecond = {0}; LARGE_INTEGER tick_1 = {0}; LARGE_INTEGER tick_2 = {0}; double uSec = 1000000; // Get the frequency QueryPerformanceFrequency(&ticksPerSecond); //Calculate per uSec freq double uFreq = ticksPerSecond.QuadPart/uSec; // Get counter b4 start of op QueryPerformanceCounter(&tick_1); // The ope itself Sleep(10); // Get counter after opfinished QueryPerformanceCounter(&tick_2); // And now the op time in uSec double diff = (tick_2.QuadPart/uFreq) - (tick_1.QuadPart/uFreq);

    Read the article

  • When do you use float and when do you use double

    - by Jakub Zaverka
    Frequently in my programming experience I need to make a decision whether I should use float or double for my real numbers. Sometimes I go for float, sometimes I go for double, but really this feels more subjective. If I would be confronted to defend my decision, I would probably not give sound reasons. When do you use float and when do you use double? Do you always use double, only when memory constraints are present you go for float? Or you use always float unless the precision requirement requires you to use double? Are there some substantial differences regarding computational complexity of basic arithemtics between float and double? What are the pros and cons of using float or double? And have you even used long double?

    Read the article

  • About floating point precision and why do we still use it

    - by system_is_b0rken
    Floating point has always been troublesome for precision on large worlds. This article explains behind-the-scenes and offers the obvious alternative - fixed point numbers. Some facts are really impressive, like: "Well 64 bits of precision gets you to the furthest distance of Pluto from the Sun (7.4 billion km) with sub-micrometer precision. " Well sub-micrometer precision is more than any fps needs (for positions and even velocities), and it would enable you to build really big worlds. My question is, why do we still use floating point if fixed point has such advantages? Most rendering APIs and physics libraries use floating point (and suffer it's disadvantages, so developers need to get around them). Are they so much slower? Additionally, how do you think scalable planetary engines like outerra or infinity handle the large scale? Do they use fixed point for positions or do they have some space dividing algorithm?

    Read the article

  • C++ floating point precision

    - by Davinel
    double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = 0; for (char i = 1; i <= 50; i++) { b = b + a; }; std::cout.precision(20); std::cout << b << std::endl; result: 15.000000000000014211 So.. 'a' is smaller than it should be. But if we take 'a' 50 times - result will be bigger than it should be. Why is this? And how to get correct result in this case?

    Read the article

  • Setting minimum number of decimal places for std::ostream precision

    - by Phil Boltt
    Hi, Is there a way to set the "minimum" number of decimal places that a std::ostream will output? For example, say I have two doubles that I want to print: double a = 0; double b = 0.123456789; I can set my maximum decimal precision so that I output b exactly std::cout << std::setprecision(9) << b << std::endl; Is there a way to set "minimum" precision so that std::cout << a << std::endl; yields "0.0", not just "0"? Thanks! Phil

    Read the article

  • C precision of double: compiler dependent?

    - by yCalleecharan
    Hi,on my 32-bit machine (with an Intel T7700 duo core), I have 15 precision digits for both double and long double types for the C language. I compared the parameters LDBL_DIG for long doubles and DBL_DIG for doubles and they are both 15. I got these answers using MVS2008. I was wondering if these results can be compiler dependent or do they just depend on my processor? Thanks a lot...

    Read the article

  • Numerical precision of double type in Visual C++ 2008 Express debugger

    - by damik
    I'm using Visual C++ 2008 Express Edition and when i debug code: double x = 0.2; I see in debugging tooltip on x 0.20000000000000001 but: typedef numeric_limits< double > double_limit; int a = double_limit::digits10 gives me: a = 15 Why results in debugger are longer than maybe ? What is this strange precision based on ? My CPU is Intel Core 2 Duo T7100

    Read the article

  • Return Double from Boost thread

    - by Benedikt Wutzi
    Hi I have an Boost thread which should return a double. The function looks like this: void analyser::findup(const double startwl, const double max, double &myret){ this->data.begin(); for(int i = (int)data.size() ; i >= 0;i--){ if(this->data[i].lambda > startwl){ if(this->data[i].db >= (max-30)) { myret = this->data[i+1].lambda; std::cout <<"in thread " << myret << std::endl; return; } } } } this function is called by another function: void analyser::start_find_up(const double startwl, const double max){ double tmp = -42.0; boost::thread up(&analyser::findup,*this, startwl,max,tmp); std::cout << "before join " << tmp << std::endl; up.join(); std::cout << "after join " << tmp << std::endl; } Anyway I've tried and googled almost anything but i can't get it to return a value. The output looks like this right now. before join -42 in thread 843.487 after join -42 Thanks for any help.

    Read the article

  • c++ floating point precision loss: 3015/0.00025298219406977296

    - by SigTerm
    The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == 0x40a78e0000000000 //*((unsigned __int64*)(&b)) == 0x3f30945640000000 double f = a/b;//3015/0.00025298219406977296; the result of calculation (i.e. "f") is 11917835.000000000 (*((unsigned __int64*)(&f)) == 0x4166bb4160000000) although it should be 11917834.814763514 (i.e. *((unsigned __int64*)(&f)) == 0x4166bb415a128aef). I.e. fractional part is lost. Unfortunately, I need fractional part to be correct. Questions: 1) Why does this happen? 2) How can I fix the problem? Additional info: 0) The result is taken directly from "watch" window (it wasn't printed, and I didn't forget to set printing precision). I also provided hex dump of floating point variable, so I'm absolutely sure about calculation result. 1) The disassembly of f = a/b is: fld qword ptr [a] fdiv qword ptr [b] fstp qword ptr [f] 2) f = 3015/0.00025298219406977296; yields correct result (f == 11917834.814763514 , *((unsigned __int64*)(&f)) == 0x4166bb415a128aef ), but it looks like in this case result is simply calculated during compile-time: fld qword ptr [__real@4166bb415a128aef (828EA0h)] fstp qword ptr [f] So, how can I fix this problem? P.S. I've found a temporary workaround (i need only fractional part of division, so I simply use f = fmod(a/b)/b at the moment), but I still would like to know how to fix this problem properly - double precision is supposed to be 16 decimal digits, so such calculation isn't supposed to cause problems.

    Read the article

  • MS SQL datetime precision problem

    - by Nailuj
    I have a situation where two persons might work on the same order (stored in an MS SQL database) from two different computers. To prevent data loss in the case where one would save his copy of the order first, and then a little later the second would save his copy and overwrite the first, I've added a check against the lastSaved field (datetime) before saving. The code looks roughly like this: private bool orderIsChangedByOtherUser(Order localOrderCopy) { // Look up fresh version of the order from the DB Order databaseOrder = orderService.GetByOrderId(localOrderCopy.Id); if (databaseOrder != null && databaseOrder.LastSaved > localOrderCopy.LastSaved) { return true; } else { return false; } } This works for most of the time, but I have found one small bug. If orderIsChangedByOtherUser returns false, the local copy will have its lastSaved updated to the current time and then be persisted to the database. The value of lastSaved in the local copy and the DB should now be the same. However, if orderIsChangedByOtherUser is run again, it sometimes returns true even though no other user has made changes to the DB. When debugging in Visual Studio, databaseOrder.LastSaved and localOrderCopy.LastSaved appear to have the same value, but when looking closer they some times differ by a few milliseconds. I found this article with a short notice on the millisecond precision for datetime in SQL: Another problem is that SQL Server stores DATETIME with a precision of 3.33 milliseconds (0. 00333 seconds). The solution I could think of for this problem, is to compare the two datetimes and consider them equal if they differ by less than say 10 milliseconds. My question to you is then: are there any better/safer ways to compare two datetime values in MS SQL to see if they are exactly the same?

    Read the article

  • How do calculators work with precision?

    - by zoul
    Hello! I wonder how calculators work with precision. For example the value of sin(M_PI) is not exactly zero when computed in double precision: #include <math.h> #include <stdio.h> int main() { double x = sin(M_PI); printf("%.20f\n", x); // 0.00000000000000012246 return 0; } Now I would certainly want to print zero when user enters sin(p). I can easily round somewhere on 1e–15 to make this particular case work, but that’s a hack, not a solution. When I start to round like this and the user enters something like 1e–20, they get a zero back (because of the rounding). The same thing happens when the user enters 1/10 and hits the = key repeatedly — when he reaches the rounding treshold, he gets zero. And yet some calculators return plain zero for sin(p) and at the same time they can work with expressions such as (1e–20)/10 comfortably. Where’s the trick?

    Read the article

  • Opening password protected Excel 2007 documents by double clicking from My documents does not work u

    - by erik-van-gorp
    When all of the following conditions are true, excel will open (most of the time) but will not open the document itself. No error is displayed. This only occurs with Excel files, Word and powerpoint do open perfectly. Conditions : OS is "Windows 7 Professional 64-bit" office is "Office 2007 Ultimate". excel file is in .xls (2003 format) excel file is password protected excel file is in "My Documents" (or a subfolder of it) file is double-clicked from explorer under Windows 7. Following options do open the excel file as it should : right click and selecting the (bold) open action single click the file and pressing enter moving the file to the desktop and double-click it. non password protected files do open from the same directory. Actions taken not resolving the problem: - reboot - repair office installation - system restore does not work because of Antivirus application installed (message from system restore, using "Symantec Internet Security 2010") Anyone any idea ?

    Read the article

  • Determine precision and scale of particular number in Python

    - by jrdioko
    I have a variable in Python containing a floating point number (e.g. num = 24654.123), and I'd like to determine the number's precision and scale values (in the Oracle sense), so 123.45678 should give me (8,5), 12.76 should give me (4,2), etc. I was first thinking about using the string representation (via str or repr), but those fail for large numbers: >>> num = 1234567890.0987654321 >>> str(num) = 1234567890.1 >>> repr(num) = 1234567890.0987654

    Read the article

  • C++ double division by 0.0 versus DBL_MIN

    - by wonsungi
    When finding the inverse square root of a double, is it better to clamp invalid non-positive inputs at 0.0 or MIN_DBL? (In my example below double b may end up being negative due to floating point rounding errors and because the laws of physics are slightly slightly fudged in the game.) Both division by 0.0 and MIN_DBL produce the same outcome in the game because 1/0.0 and 1/DBL_MIN are effectively infinity. My intuition says MIN_DBL is the better choice, but would there be any case for using 0.0? Like perhaps sqrt(0.0), 1/0.0 and multiplication by 1.#INF000000000000 execute faster because they are special cases. double b = 1 - v.length_squared()/(c*c); #ifdef CLAMP_BY_0 if (b < 0.0) b = 0.0; #endif #ifdef CLAMP_BY_DBL_MIN if (b <= 0.0) b = DBL_MIN; #endif double lorentz_factor = 1/sqrt(b); double division in MSVC: 1/0.0 = 1.#INF000000000000 1/DBL_MIN = 4.4942328371557898e+307

    Read the article

  • Swap bits in c++ for a double

    - by hidayat
    Im trying to change from big endian to little endian on a double. One way to go is to use double val, tmp = 5.55; ((unsigned int *)&val)[0] = ntohl(((unsigned int *)&tmp)[1]); ((unsigned int *)&val)[1] = ntohl(((unsigned int *)&tmp)[0]); But then I get a warning: "dereferencing type-punned pointer will break strict-aliasing rules" and I dont want to turn this warning off. Another way to go is: #define ntohll(x) ( ( (uint64_t)(ntohl( (uint32_t)((x << 32) >> 32) )) << 32) | ntohl( ((uint32_t)(x >> 32)) ) ) val = (double)bswap_64(unsigned long long(tmp)); //or val = (double)ntohll(unsigned long long(tmp)); But then a lose the decimals. Anyone know a good way to swap the bits on a double without using a for loop?

    Read the article

  • EF4 Code First Control Unicode and Decimal Precision, Scale with Attributes

    - by Dane Morgridge
    There are several attributes available when using code first with the Entity Framework 4 CTP5 Code First option.  When working with strings you can use [MaxLength(length)] to control the length and [Required] will work on all properties.  But there are a few things missing. By default all string will be created using unicode so you will get nvarchar instead of varchar.  You can change this using the fluent API or you can create an attribute to make the change.  If you have a lot of properties, the attribute will be much easier and require less code. You will need to add two classes to your project to create the attribute itself: 1: public class UnicodeAttribute : Attribute 2: { 3: bool _isUnicode; 4:  5: public UnicodeAttribute(bool isUnicode) 6: { 7: _isUnicode = isUnicode; 8: } 9:  10: public bool IsUnicode { get { return _isUnicode; } } 11: } 12:  13: public class UnicodeAttributeConvention : AttributeConfigurationConvention<PropertyInfo, StringPropertyConfiguration, UnicodeAttribute> 14: { 15: public override void Apply(PropertyInfo memberInfo, StringPropertyConfiguration configuration, UnicodeAttribute attribute) 16: { 17: configuration.IsUnicode = attribute.IsUnicode; 18: } 19: } The UnicodeAttribue class gives you a [Unicode] attribute that you can use on your properties and the UnicodeAttributeConvention will tell EF how to handle the attribute. You will need to add a line to the OnModelCreating method inside your context for EF to recognize the attribute: 1: protected override void OnModelCreating(System.Data.Entity.ModelConfiguration.ModelBuilder modelBuilder) 2: { 3: modelBuilder.Conventions.Add(new UnicodeAttributeConvention()); 4: base.OnModelCreating(modelBuilder); 5: } Once you have this done, you can use the attribute in your classes to make sure that you get database types of varchar instead of nvarchar: 1: [Unicode(false)] 2: public string Name { get; set; }   Another option that is missing is the ability to set the precision and scale on a decimal.  By default decimals get created as (18,0).  If you need decimals to be something like (9,2) then you can once again use the fluent API or create a custom attribute.  As with the unicode attribute, you will need to add two classes to your project: 1: public class DecimalPrecisionAttribute : Attribute 2: { 3: int _precision; 4: private int _scale; 5:  6: public DecimalPrecisionAttribute(int precision, int scale) 7: { 8: _precision = precision; 9: _scale = scale; 10: } 11:  12: public int Precision { get { return _precision; } } 13: public int Scale { get { return _scale; } } 14: } 15:  16: public class DecimalPrecisionAttributeConvention : AttributeConfigurationConvention<PropertyInfo, DecimalPropertyConfiguration, DecimalPrecisionAttribute> 17: { 18: public override void Apply(PropertyInfo memberInfo, DecimalPropertyConfiguration configuration, DecimalPrecisionAttribute attribute) 19: { 20: configuration.Precision = Convert.ToByte(attribute.Precision); 21: configuration.Scale = Convert.ToByte(attribute.Scale); 22:  23: } 24: } Add your line to the OnModelCreating: 1: protected override void OnModelCreating(System.Data.Entity.ModelConfiguration.ModelBuilder modelBuilder) 2: { 3: modelBuilder.Conventions.Add(new UnicodeAttributeConvention()); 4: modelBuilder.Conventions.Add(new DecimalPrecisionAttributeConvention()); 5: base.OnModelCreating(modelBuilder); 6: } Now you can use the following on your properties: 1: [DecimalPrecision(9,2)] 2: public decimal Cost { get; set; } Both these options use the same concepts so if there are other attributes that you want to use, you can create them quite simply.  The key to it all is the PropertyConfiguration classes.   If there is a class for the datatype, then you should be able to write an attribute to set almost everything you need.  You could also create a single attribute to encapsulate all of the possible string combinations instead of having multiple attributes on each property. All in all, I am loving code first and having attributes to control database generation instead of using the fluent API is huge and saves me a great deal of time.

    Read the article

  • Dividing a double with integer

    - by hardcoder
    I am facing an issue while dividing a double with an int. Code snippet is : double db = 10; int fac = 100; double res = db / fac; The value of res is 0.10000000000000001 instead of 0.10. Does anyone know what is the reason for this? I am using cc to compile the code.

    Read the article

  • Java string to double conversion.

    - by wretrOvian
    Hi, I've been reading up on the net about the issues with handling float and double types in java. Unfortunately, the image is still not clear. Hence, i'm asking here direct. :( My MySQL table has various DECIMAL(m,d) columns. The m may range from 5 to 30. d stays a constant at 2. Question 1. What equivalent data-type should i be using in Java to work (i.e store, retrieve, and process) with the size of the values in my table? (I've settled with double - hence this post). Question 2. While trying to parse a double from a string, i'm getting errors Double dpu = new Double(dpuField.getText()); for example - "1" -> java.lang.NumberFormatException: empty String "10" -> 1.0 "101" -> 10.0 "101." -> 101.0 "101.1" -> 101.0 "101.19" -> 101.1 What am i doing wrong? What is the correct way to convert a string to a double value? And what measures should i take to perform operations on such values?

    Read the article

  • [C#] Convert string to double with 2 digit after decimal separator

    - by st.stoqnov
    All began with these simple lines of code: string s = "16.9"; double d = Convert.ToDouble(s); d*=100; The result should be 1690.0, but it's not. d is equal to 1689.9999999999998. All I want to do is to round a double to value with 2 digit after decimal separator. Here is my function. private double RoundFloat(double Value) { float sign = (Value < 0) ? -0.01f : 0.01f; if (Math.Abs(Value) < 0.00001) Value = 0; string SVal = Value.ToString(); string DecimalSeparator = System.Globalization.CultureInfo.CurrentCulture.NumberFormat.CurrencyDecimalSeparator; int i = SVal.IndexOf(DecimalSeparator); if (i > 0) { int SRnd; try { // ????? ??????? ????? ???? ?????????? ?????????? SRnd = Convert.ToInt32(SVal.Substring(i + 3, 1)); } catch { SRnd = 0; } if (SVal.Length > i + 3) SVal = SVal.Substring(0, i + 3); //SVal += "00001"; try { double result = (SRnd >= 5) ? Convert.ToDouble(SVal) + sign : Convert.ToDouble(SVal); //result = Math.Round(result, 2); return result; } catch { return 0; } } else { return Value; } But again the same problem, converting from string to double is not working as I want. A workaround to this problem is to concatenate "00001" to the string and then use the Math.Round function (commented in the example above). This double value multiplied to 100 (as integer) is send to a device (cash register) and this values must be correct. I am using VS2005 + .NET CF 2.0 Is there another more "elegant" solution, I am not happy with this one.

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >