Search Results

Search found 218 results on 9 pages for 'rounding'.

Page 1/9 | 1 2 3 4 5 6 7 8 9  | Next Page >

  • SQL Rounding Problems in 2005 and 2000

    - by azamsharp
    I have a value in the database which is 2.700000002. When I run a query in Management studio in SQL SERVER 2005 I get 2.7. But when I run in SQL SERVER 2000 query analyzer it comes 2.700000002. 2.70000002 is correct why is SQL SERVER 2005 trying to change the value by rounding it or selecting the floor value?

    Read the article

  • Rounding values up or down in C#

    - by c11ada
    Hey all, I've created a game which gives a score at the end of the game, but the problem is that this score is sometimes a number with a lot of digits after the decimal point (like 87.124563563566). How would I go about rounding up or down the value so that I could have something like 87.12? Thanks!

    Read the article

  • Rounding floating-point numbers to 4 decimal points

    - by Himadri
    I have two decimal numbers. I want those number to be same upto 4 decimal points without rounding. If numbers are different I want 2nd number to be replaced by 1st. What if condition should I write? Eg, 1. num1 = 0.94618976 num2 = 0.94620239 If we round these numbers upto 4 decimal then we get 0.9462 same number, but I don't want to round these numbers. 2. num1 = 0.94620239 num2 = 0.94639125 The one way I found is take absolute difference of both numbers say diff and then check the value. My problem is of checking the range of diff. Thank You.

    Read the article

  • Decimal rounding strategies in enterprise applications

    - by Sapphire
    Well, I am wondering about a thing with rounding decimals, and storing them in DB. Problem is like this: Let's say we have a customer and a invoice. The invoice has total price of $100.495 (due to some discount percentage which is not integer number), but it is shown as $100.50 (when rounded, just for print on invoice). It is stored in the DB with the price of $100.495, which means that when customer makes a deposit of $100.50 it will have $0.005 extra on the account. If this is rounded, it will appear as $0, but after couple of invoices it would keep accumulating, which would appear wrong (although it actually is not). What is best to do in this case. Store the value of $100.50, or leave everything as-is?

    Read the article

  • Why Is ToString() Rounding My Double Value?

    - by user163757
    How do I prevent my double value from being rounded when converting to a string? I have tried both Convert.ToString and ToString() with the same result. For example my double may look something like 77.987654321, and the two strings conversions convert to to 77.98765. I need to keep the precision of the value as is.

    Read the article

  • Error in rounding off values using .round in Ruby

    - by Shreyas Satish
    The following piece of code works perfectly in script/console but returns the following error when i compile the same in a ruby script.: :in `round': wrong number of arguments (1 for 0) (ArgumentError) tf={"ph"={0=1.33333333333333, 1=1.5}, "fee"={0=1.66666666666667}, "test"={0=1.16666666666667, 1=1.25}, "what"={0=2.0, 1=2.0}, "for"={0=1.5}, "is"={0=1.83333333333333, 1=1.75}} tf.each{|k,v| v.each{|k1,v1| tf[k][k1]=(v1.round(5))}} Any Ideas ? Cheers !

    Read the article

  • C - floating point rounding

    - by hatorade
    I'm trying to understand how floating point numbers work. I think I'd like to test out what I know / need to learn by evaluating the following: I would like to find the smallest x such that x + 1 = x, where x is a floating point number. As I understand it, this would happen in the case where x is large enough so that x + 1 is closer to x than the next number higher than x representable by floating point. So intuitively it seems it would be the case where I don't have enough digits in the significand. Would this number x then be the number where the significand is all 1's. But then I can't seem to figure out what the exponent would have to be. Obviously it would have to be big (relative to 10^0, anyway).

    Read the article

  • Rounding of calculated measure in MDX

    - by Espo
    How can i round a calculated mdx measure up to the nearest integer without having Excel on the server? The Excel-function is CEILING(number, significance), but it is not possible to install Excel on the production ssas-server.

    Read the article

  • JFreeChart Ugly Rounding Error?

    - by billynomates
    Using series.add(180, 1); produces a perfectly valid chart like this (little red dot at the bottom with some PolarItemRenderer Mods!) but using series.add(3000/(6000/360), 1); produces this beast: I assume it's because somewhere, 6000/360 = 16.6... is getting rounded? How can I stop this happening? Thanks :)

    Read the article

  • Java Ugly Rounding Error?

    - by billynomates
    Using series.add(180, 1); produces a perfectly valid chart like this (little red dot at the bottom with some PolarItemRenderer Mods!) but using series.add(3000/(6000/360), 1); produces this beast: I assume it's because somewhere, 6000/360 = 16.6... is getting rounded? How can I stop this happening? Thanks :)

    Read the article

  • Rounding a positive number to a power of another number

    - by Sagekilla
    I'm trying to round a number to the next smallest power of another number. The number I'm trying to round is always positive. I'm not particular on which direction it rounds, but I prefer downwards if possible. I would like to be able to round towards arbitrary bases, but the ones I'm most concerned with at the moment is base 2 and fractional powers of 2 like 2^(1/2), 2^(1/4), and so forth. Here's my current algorithm for base 2. The log2 I multiply by is actually the inverse of log2: double roundBaseTwo(double x) { return 1.0 / (1 << (int)((log(x) * log2)) } Any help would be appreciated!

    Read the article

  • Rounding functions in DAX

    - by Marco Russo (SQLBI)
    Today I prepared a table of the many rounding functions available in DAX (yes, it’s part of the book we’re writing), so that I have a complete schema of the better function to use, depending on the round operation I need to do. Here is the list of functions used and then the results shown for a relevant set of values. FLOOR = FLOOR( Tests[Value], 0.01 ) TRUNC = TRUNC( Tests[Value], 2 ) ROUNDDOWN = ROUNDDOWN( Tests[Value], 2 ) MROUND = MROUND( Tests[Value], 0.01 ) ROUND = ROUND( Tests[Value], 2 )...(read more)

    Read the article

  • Rounding functions in DAX

    - by Marco Russo (SQLBI)
    Today I prepared a table of the many rounding functions available in DAX (yes, it’s part of the book we’re writing), so that I have a complete schema of the better function to use, depending on the round operation I need to do. Here is the list of functions used and then the results shown for a relevant set of values. FLOOR = FLOOR( Tests[Value], 0.01 ) TRUNC = TRUNC( Tests[Value], 2 ) ROUNDDOWN = ROUNDDOWN( Tests[Value], 2 ) MROUND = MROUND( Tests[Value], 0.01 ) ROUND = ROUND( Tests[Value], 2 )...(read more)

    Read the article

  • Is there a way to do 'correct' arithmetical rounding in .NET? / C#

    - by Markus
    I'm trying to round a number to it's first decimal place and, considering the different MidpointRounding options, that seems to work well. A problem arises though when that number has sunsequent decimal places that would arithmetically affect the rounding. An example: With 0.1, 0.11..0.19 and 0.141..0.44 it works: Math.Round(0.1, 1) == 0.1 Math.Round(0.11, 1) == 0.1 Math.Round(0.14, 1) == 0.1 Math.Round(0.15, 1) == 0.2 Math.Round(0.141, 1) == 0.1 But with 0.141..0.149 it always returns 0.1, although 0.146..0.149 should round to 0.2: Math.Round(0.145, 1, MidpointRounding.AwayFromZero) == 0.1 Math.Round(0.146, 1, MidpointRounding.AwayFromZero) == 0.1 Math.Round(0.146, 1, MidpointRounding.ToEven) == 0.1 Math.Round(0.146M, 1, MidpointRounding.ToEven) == 0.1M Math.Round(0.146M, 1, MidpointRounding.AwayFromZero) == 0.1M I tried to come up with a function that addresses this problem, and it works well for this case, but of course it glamorously fails if you try to round i.e. 0.144449 to it's first decimal digit (which should be 0.2, but results 0.1.) (That doesn't work with Math.Round() either.) private double round(double value, int digit) { // basically the old "add 0.5, then truncate to integer" trick double fix = 0.5D/( Math.Pow(10D, digit+1) )*( value = 0 ? 1D : -1D ); double fixedValue = value + fix; // 'truncate to integer' - shift left, round, shift right return Math.Round(fixedValue * Math.Pow(10D, digit)) / Math.Pow(10D, digit); } I assume a solution would be to enumerate all digits, find the first value larger than 4 and then round up, or else round down. Problem 1: That seems idiotic, Problem 2: I have no idea how to enumerate the digits without a gazillion of multiplications and subtractios. Long story short: What is the best way to do that?

    Read the article

  • iPhoneOS SDK - Remove Corner Rounding from views (iPad problem)

    - by norskben
    Hi Guys This might be a little bit picky, but in the iPad SplitViewController setup, there are 2 views. Each of the views has a very small black corner rounding. (This is probably the same with iPhone apps too). This rounding is visible in the image below. What I would like to do is remove the black rounding, so the UI doesnt get these two little bumps along the bottom. Has anyone done this, or know how to? -Its surely possible. Hopefully some one has seen this before. Thanks Image Link Mirror

    Read the article

  • Fixing VBSCRIPT inaccurate mathematical results due to rounding

    - by jay
    Try running this in a .VBS file MsgBox(545.14-544.94) You get a neat little answer of 0.199999999999932! This rounding issue also occurs unfortunately in Sin(2 * pi) since VB can only ever see the (user defined) variable pi as accurate as 3.14159265358979. Is rounding it manually (and loosing accuracy) the only way to improve the result? What is the most effective way of dealing with this kind of problem?

    Read the article

  • Fixing VB6 inaccurate mathmetical results due to rounding

    - by jay
    Try running this in a .VBS file MsgBox(545.14-544.94) You get a neat little answer of 0.199999999999932! This rounding issue also occurs unfortunately in Sin(2 * pi) since VB can only ever see the (user defined) variable pi as accurate as 3.14159265358979. Is rounding it manually (and loosing accuracy) the only way to improve the result? What is the most effective way of dealing with this kind of problem?

    Read the article

  • JavaScript: Rounding to two decimal places. Not less than two

    - by Abs
    Hello all, I have this line of code which rounds my numbers to 2 decimal places. But the thing is I get numbers like this. 10.8, 2.4 etc. These are not my idea of 2 decimal places so how I can improve this: Math.round(price*Math.pow(10,2))/Math.pow(10,2); I want numbers like 10.80, 2.40 etc. Use of JQuery is fine with me. Thanks for any help.

    Read the article

  • Why do you need float/double?

    - by acidzombie24
    I was watching http://www.joelonsoftware.com/items/2011/06/27.html and laughed at Jon Skeet joke about 0.3 not being 0.3. I personally never had problems with floats/decimals/doubles but then I remember I learned 6502 very early and never needed floats in most of my programs. The only time I used it was for graphics and math where inaccurate numbers were ok and the output was for the screen and not to be stored (in a db, file) or dependent on. My question is, where are places were you typically use floats/decimals/double? So I know to watch out for these gotchas. With money I use longs and store values by the cent, for speed of an object in a game I add ints and divide (or bitshift) the value to know if I need to move a pixel or not. (I made object move in the 6502 days, we had no divide nor floats but had shifts). So I was mostly curious.

    Read the article

  • sql rounding value

    - by fishhead
    hello, I have been sucsessful at rounding my value to one decimal point but I would like to trim off the multitude oz trailing zeros. ROUND(SUM(Duration ),1) ends up giving me 16.9000000000000000 how do I trim all those trailing zeros. thanks mssql

    Read the article

1 2 3 4 5 6 7 8 9  | Next Page >