Search Results

Search found 487 results on 20 pages for '100000'.

Page 15/20 | < Previous Page | 11 12 13 14 15 16 17 18 19 20  | Next Page >

  • Comparing two strings in excel, add value for common variables

    - by overtime
    I'm comparing two large datasets containing strings in excel. Column A contains the numbers 1-1,000,000. Column B contains 1,000,000 strings, neatly organized in the desired order. Column C contains 100,000 randomly organized strings, that have identical values somewhere in column B. Example: A B C D 1 String1 String642 2 String2 String11 3 String3 String8000 4 String4 String78 What I'd like to do is find duplicate values in columns B and C then output the Column A value that corresponds with the string in Column C into Column D. Desired Output: A B C D 1 String1 String642 642 2 String2 String11 11 3 String3 String8000 8000 4 String4 String78 78

    Read the article

  • Options to efficiently synchronize 1 million files with remote servers?

    - by Zilvinas
    At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These playlists need to be uploaded to 10 other remote servers on different continents every hour and it needs to happen quick in under 2 mins ideally. It's very important that files that are deleted on the master are also deleted on all the replicas. We currently use Linux for our infrastructure. I was thinking about trying rsync with the -W option to copy whole files without comparing contents. I haven't tried it yet but maybe people who have more experience with rsync could tell me if it's a viable option? What other options are worth considering?

    Read the article

  • Simple C# CSV Excel export class

    - by Chris
    Thought this might be handy for someone, this is an extremely simple CSV export class that I needed. Features: Extremely simple to use Escapes commas and quotes so excel handles them fine Exports date and datetimes in timezone-proof format Without further ado: using System; using System.Data.SqlTypes; using System.IO; using System.Text; using System.Collections.Generic; /// <summary> /// Simple CSV export /// Example: /// CsvExport myExport = new CsvExport(); /// /// myExport.AddRow(); /// myExport["Region"] = "New York, USA"; /// myExport["Sales"] = 100000; /// myExport["Date Opened"] = new DateTime(2003, 12, 31); /// /// myExport.AddRow(); /// myExport["Region"] = "Sydney \"in\" Australia"; /// myExport["Sales"] = 50000; /// myExport["Date Opened"] = new DateTime(2005, 1, 1, 9, 30, 0); /// /// Then you can do any of the following three output options: /// string myCsv = myExport.Export(); /// myExport.ExportToFile("Somefile.csv"); /// byte[] myCsvData = myExport.ExportToBytes(); /// </summary> public class CsvExport { /// <summary> /// To keep the ordered list of column names /// </summary> List<string> fields = new List<string>(); /// <summary> /// The list of rows /// </summary> List<Dictionary<string, object>> rows = new List<Dictionary<string, object>>(); /// <summary> /// The current row /// </summary> Dictionary<string, object> currentRow { get { return rows[rows.Count - 1]; } } /// <summary> /// Set a value on this column /// </summary> public object this[string field] { set { // Keep track of the field names, because the dictionary loses the ordering if (!fields.Contains(field)) fields.Add(field); currentRow[field] = value; } } /// <summary> /// Call this before setting any fields on a row /// </summary> public void AddRow() { rows.Add(new Dictionary<string, object>()); } /// <summary> /// Converts a value to how it should output in a csv file /// If it has a comma, it needs surrounding with double quotes /// Eg Sydney, Australia -> "Sydney, Australia" /// Also if it contains any double quotes ("), then they need to be replaced with quad quotes[sic] ("") /// Eg "Dangerous Dan" McGrew -> """Dangerous Dan"" McGrew" /// </summary> string MakeValueCsvFriendly(object value) { if (value == null) return ""; if (value is INullable && ((INullable)value).IsNull) return ""; if (value is DateTime) { if (((DateTime)value).TimeOfDay.TotalSeconds==0) return ((DateTime)value).ToString("yyyy-MM-dd"); return ((DateTime)value).ToString("yyyy-MM-dd HH:mm:ss"); } string output = value.ToString(); if (output.Contains(",") || output.Contains("\"")) output = '"' + output.Replace("\"", "\"\"") + '"'; return output; } /// <summary> /// Output all rows as a CSV returning a string /// </summary> public string Export() { StringBuilder sb = new StringBuilder(); // The header foreach (string field in fields) sb.Append(field).Append(","); sb.AppendLine(); // The rows foreach (Dictionary<string, object> row in rows) { foreach (string field in fields) sb.Append(MakeValueCsvFriendly(row[field])).Append(","); sb.AppendLine(); } return sb.ToString(); } /// <summary> /// Exports to a file /// </summary> public void ExportToFile(string path) { File.WriteAllText(path, Export()); } /// <summary> /// Exports as raw UTF8 bytes /// </summary> public byte[] ExportToBytes() { return Encoding.UTF8.GetBytes(Export()); } }

    Read the article

  • parallel computation for an Iterator of elements in Java

    - by Brian Harris
    I've had the same need a few times now and wanted to get other thoughts on the right way to structure a solution. The need is to perform some operation on many elements on many threads without needing to have all elements in memory at once, just the ones under computation. As in, Iterables.partition is insufficient because it brings all elements into memory up front. Expressing it in code, I want to write a BulkCalc2 that does the same thing as BulkCalc1, just in parallel. Below is sample code that illustrates my best attempt. I'm not satisfied because it's big and ugly, but it does seem to accomplish my goals of keeping threads highly utilized until the work is done, propagating any exceptions during computation, and not having more than numThreads instances of BigThing necessarily in memory at once. I'll accept the answer which meets the stated goals in the most concise way, whether it's a way to improve my BulkCalc2 or a completely different solution. interface BigThing { int getId(); String getString(); } class Calc { // somewhat expensive computation double calc(BigThing bigThing) { Random r = new Random(bigThing.getString().hashCode()); double d = 0; for (int i = 0; i < 100000; i++) { d += r.nextDouble(); } return d; } } class BulkCalc1 { final Calc calc; public BulkCalc1(Calc calc) { this.calc = calc; } public TreeMap<Integer, Double> calc(Iterator<BigThing> in) { TreeMap<Integer, Double> results = Maps.newTreeMap(); while (in.hasNext()) { BigThing o = in.next(); results.put(o.getId(), calc.calc(o)); } return results; } } class SafeIterator<T> { final Iterator<T> in; SafeIterator(Iterator<T> in) { this.in = in; } synchronized T nextOrNull() { if (in.hasNext()) { return in.next(); } return null; } } class BulkCalc2 { final Calc calc; final int numThreads; public BulkCalc2(Calc calc, int numThreads) { this.calc = calc; this.numThreads = numThreads; } public TreeMap<Integer, Double> calc(Iterator<BigThing> in) { ExecutorService e = Executors.newFixedThreadPool(numThreads); List<Future<?>> futures = Lists.newLinkedList(); final Map<Integer, Double> results = new MapMaker().concurrencyLevel(numThreads).makeMap(); final SafeIterator<BigThing> it = new SafeIterator<BigThing>(in); for (int i = 0; i < numThreads; i++) { futures.add(e.submit(new Runnable() { @Override public void run() { while (true) { BigThing o = it.nextOrNull(); if (o == null) { return; } results.put(o.getId(), calc.calc(o)); } } })); } e.shutdown(); for (Future<?> future : futures) { try { future.get(); } catch (InterruptedException ex) { // swallowing is OK } catch (ExecutionException ex) { throw Throwables.propagate(ex.getCause()); } } return new TreeMap<Integer, Double>(results); } }

    Read the article

  • Get rid of jfreechart chartpanel unnecessary space

    - by ryvantage
    I am trying to get a JFreeChart ChartPanel to remove unwanted extra space between the edge of the panel and the graph itself. To best illustrate, here's a SSCCE (with JFreeChart installed): public static void main(String[] args) { JPanel panel = new JPanel(new GridBagLayout()); GridBagConstraints gbc = new GridBagConstraints(); gbc.fill = GridBagConstraints.BOTH; gbc.gridwidth = 1; gbc.gridheight = 1; gbc.weightx = 1; gbc.weighty = 1; gbc.gridy = 1; gbc.gridx = 1; panel.add(createChart("Sales", Chart_Type.DOLLARS, 100000, 115000), gbc); gbc.gridx = 2; panel.add(createChart("Quotes", Chart_Type.DOLLARS, 250000, 240000), gbc); gbc.gridx = 3; panel.add(createChart("Profits", Chart_Type.PERCENTAGE, 40.00, 38.00), gbc); JFrame frame = new JFrame(); frame.add(panel); frame.setSize(800, 300); frame.setLocationRelativeTo(null); frame.setVisible(true); frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); } private static ChartPanel createChart(String title, Chart_Type type, double goal, double actual) { double maxValue = goal * 2; double yellowToGreenNum = goal; double redToYellowNum = goal * .75; DefaultValueDataset dataset = new DefaultValueDataset(actual); JFreeChart jfreechart = createChart(dataset, Math.max(actual, maxValue), redToYellowNum, yellowToGreenNum, title, type); ChartPanel chartPanel = new ChartPanel(jfreechart); chartPanel.setBorder(new LineBorder(Color.red)); return chartPanel; } private static JFreeChart createChart(ValueDataset valuedataset, Number maxValue, Number redToYellowNum, Number yellowToGreenNum, String title, Chart_Type type) { MeterPlot meterplot = new MeterPlot(valuedataset); meterplot.setRange(new Range(0.0D, maxValue.doubleValue())); meterplot.addInterval(new MeterInterval(" Goal Not Met ", new Range(0.0D, redToYellowNum.doubleValue()), Color.lightGray, new BasicStroke(2.0F), new Color(255, 0, 0, 128))); meterplot.addInterval(new MeterInterval(" Goal Almost Met ", new Range(redToYellowNum.doubleValue(), yellowToGreenNum.doubleValue()), Color.lightGray, new BasicStroke(2.0F), new Color(255, 255, 0, 64))); meterplot.addInterval(new MeterInterval(" Goal Met ", new Range(yellowToGreenNum.doubleValue(), maxValue.doubleValue()), Color.lightGray, new BasicStroke(2.0F), new Color(0, 255, 0, 64))); meterplot.setNeedlePaint(Color.darkGray); meterplot.setDialBackgroundPaint(Color.white); meterplot.setDialOutlinePaint(Color.gray); meterplot.setDialShape(DialShape.CHORD); meterplot.setMeterAngle(260); meterplot.setTickLabelsVisible(false); meterplot.setTickSize(maxValue.doubleValue() / 20); meterplot.setTickPaint(Color.lightGray); meterplot.setValuePaint(Color.black); meterplot.setValueFont(new Font("Dialog", Font.BOLD, 0)); meterplot.setUnits(""); if(type == Chart_Type.DOLLARS) meterplot.setTickLabelFormat(NumberFormat.getCurrencyInstance()); else if(type == Chart_Type.PERCENTAGE) meterplot.setTickLabelFormat(NumberFormat.getPercentInstance()); JFreeChart jfreechart = new JFreeChart(title, JFreeChart.DEFAULT_TITLE_FONT, meterplot, false); return jfreechart; } enum Chart_Type { DOLLARS, PERCENTAGE } If you resize the frame, you can see that you cannot make the edge of the graph go to the edge of the panel (the panels are outlined in red). Especially on the bottom - there is always a gap between the bottom the graph and the bottom of the panel. Is there a way to make the graph fill the entire area? Is there a way to at least guarantee that it is touching one edge of the panel (i.e., it is touching the top and bottom or the left and right) ??

    Read the article

  • How to reduce virtual memory by optimising my PHP code?

    - by iCeR
    My current code (see below) uses 147MB of virtual memory! My provider has allocated 100MB by default and the process is killed once run, causing an internal error. The code is utilising curl multi and must be able to loop with more than 150 iterations whilst still minimizing the virtual memory. The code below is only set at 150 iterations and still causes the internal server error. At 90 iterations the issue does not occur. How can I adjust my code to lower the resource use / virtual memory? Thanks! <?php function udate($format, $utimestamp = null) { if ($utimestamp === null) $utimestamp = microtime(true); $timestamp = floor($utimestamp); $milliseconds = round(($utimestamp - $timestamp) * 1000); return date(preg_replace('`(?<!\\\\)u`', $milliseconds, $format), $timestamp); } $url = 'https://www.testdomain.com/'; $curl_arr = array(); $master = curl_multi_init(); for($i=0; $i<150; $i++) { $curl_arr[$i] = curl_init(); curl_setopt($curl_arr[$i], CURLOPT_URL, $url); curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, 1); curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYHOST, FALSE); curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYPEER, FALSE); curl_multi_add_handle($master, $curl_arr[$i]); } do { curl_multi_exec($master,$running); } while($running > 0); for($i=0; $i<150; $i++) { $results = curl_multi_getcontent ($curl_arr[$i]); $results = explode("<br>", $results); echo $results[0]; echo "<br>"; echo $results[1]; echo "<br>"; echo udate('H:i:s:u'); echo "<br><br>"; usleep(100000); } ?> Processor Information Total processors: 8 Processor #1 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #2 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #3 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #4 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #5 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #6 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #7 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Processor #8 Vendor GenuineIntel Name Intel(R) Xeon(R) CPU E5405 @ 2.00GHz Speed 1995.120 MHz Cache 6144 KB Memory Information Memory for crash kernel (0x0 to 0x0) notwithin permissible range Memory: 8302344k/9175040k available (2176k kernel code, 80272k reserved, 901k data, 228k init, 7466304k highmem) System Information Linux server3.server.com 2.6.18-194.17.1.el5PAE #1 SMP Wed Sep 29 13:31:51 EDT 2010 i686 i686 i386 GNU/Linux Physical Disks SCSI device sda: 1952448512 512-byte hdwr sectors (999654 MB) sda: Write Protect is off sda: Mode Sense: 03 00 00 08 SCSI device sda: drive cache: write back SCSI device sda: 1952448512 512-byte hdwr sectors (999654 MB) sda: Write Protect is off sda: Mode Sense: 03 00 00 08 SCSI device sda: drive cache: write back sd 0:1:0:0: Attached scsi disk sda sd 4:0:0:0: Attached scsi removable disk sdb sd 0:1:0:0: Attached scsi generic sg4 type 0 sd 4:0:0:0: Attached scsi generic sg7 type 0 Current Memory Usage total used free shared buffers cached Mem: 8306672 7847384 459288 0 487912 6444548 -/+ buffers/cache: 914924 7391748 Swap: 4095992 496 4095496 Total: 12402664 7847880 4554784 Current Disk Usage Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 898G 307G 546G 36% / /dev/sda1 99M 19M 76M 20% /boot none 4.0G 0 4.0G 0% /dev/shm /var/tmpMnt 4.0G 1.8G 2.0G 48% /tmp

    Read the article

  • C# performance varying due to memory

    - by user1107474
    Hope this is a valid post here, its a combination of C# issues and hardware. I am benchmarking our server because we have found problems with the performance of our quant library (written in C#). I have simulated the same performance issues with some simple C# code- performing very heavy memory-usage. The code below is in a function which is spawned from a threadpool, up to a maximum of 32 threads (because our server has 4x CPUs x 8 cores each). This is all on .Net 3.5 The problem is that we are getting wildly differing performance. I run the below function 1000 times. The average time taken for the code to run could be, say, 3.5s, but the fastest will only be 1.2s and the slowest will be 7s- for the exact same function! I have graphed the memory usage against the timings and there doesnt appear to be any correlation with the GC kicking in. One thing I did notice is that when running in a single thread the timings are identical and there is no wild deviation. I have also tested CPU-bound algorithms and the timings are identical too. This has made us wonder if the memory bus just cannot cope. I was wondering could this be another .net or C# problem, or is it something related to our hardware? Would this be the same experience if I had used C++, or Java?? We are using 4x Intel x7550 with 32GB ram. Is there any way around this problem in general? Stopwatch watch = new Stopwatch(); watch.Start(); List<byte> list1 = new List<byte>(); List<byte> list2 = new List<byte>(); List<byte> list3 = new List<byte>(); int Size1 = 10000000; int Size2 = 2 * Size1; int Size3 = Size1; for (int i = 0; i < Size1; i++) { list1.Add(57); } for (int i = 0; i < Size2; i = i + 2) { list2.Add(56); } for (int i = 0; i < Size3; i++) { byte temp = list1.ElementAt(i); byte temp2 = list2.ElementAt(i); list3.Add(temp); list2[i] = temp; list1[i] = temp2; } watch.Stop(); (the code is just meant to stress out the memory) I would include the threadpool code, but we used a non-standard threadpool library. EDIT: I have reduced "size1" to 100000, which basically doesn't use much memory and I still get a lot of jitter. This suggests it's not the amount of memory being transferred, but the frequency of memory grabs?

    Read the article

  • can't find what's wrong with my code :(

    - by blood
    the point of my code is for me to press f1 and it will scan 500 pixels down and 500 pixels and put them in a array (it just takes a box that is 500 by 500 of the screen). then after that when i hit end it will click on only on the color black or... what i set it to. anyway it has been doing odd stuff and i can't find why: #include <iostream> #include <windows.h> using namespace std; COLORREF rgb[499][499]; HDC hDC = GetDC(HWND_DESKTOP); POINT main_coner; BYTE rVal; BYTE gVal; BYTE bVal; int red; int green; int blue; int ff = 0; int main() { for(;;) { if(GetAsyncKeyState(VK_F1)) { cout << "started"; int a1 = 0; int a2 = 0; GetCursorPos(&main_coner); int x = main_coner.x; int y = main_coner.y; for(;;) { //cout << a1 << "___" << a2 << "\n"; rgb[a1][a2] = GetPixel(hDC, x, y); a1++; x++; if(x > main_coner.x + 499) { y++; x = main_coner.x; a1 = 0; a2++; } if(y > main_coner.y + 499) { ff = 1; break; } } cout << "done"; break; } if(ff == 1) break; } for(;;) { if(GetAsyncKeyState(VK_END)) { GetCursorPos(&main_coner); int x = main_coner.x; int y = main_coner.y; int a1 = -1; int a2 = -1; for(;;) { x++; a1++; rVal = GetRValue(rgb[a1][a2]); gVal = GetGValue(rgb[a1][a2]); bVal = GetBValue(rgb[a1][a2]); red = (int)rVal; // get the colors into __int8 green = (int)gVal; // get the colors into __int8 blue = (int)bVal; // get the colors into __int8 if(red == 0 && green == 0 && blue == 0) { SetCursorPos(main_coner.x + x, main_coner.y + y); mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0); Sleep(10); mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0); Sleep(100); } if(x > main_coner.x + 499) { a1 = 0; a2++; } if(y > main_coner.y + 499) { Sleep(100000000000); break; } if(GetAsyncKeyState(VK_CONTROL)) { Sleep(100000); break; } } } } for(;;) { if(GetAsyncKeyState(VK_END)) { break; } } return 0; } anyone see what's wrong with my code :( (feel free to add tags)

    Read the article

  • Not sure I am using inheritance/polymorphism issue?

    - by planker1010
    So for this assignment I have to create a car class(parent) and a certifiedpreowned (child) and I need to have the parent class have a method to check if it is still under warranty. *checkWarrantyStatus(). that method calls the boolean isCoveredUnderWarranty() to veryify if the car still has warranty. My issue is in the certifiedpreowned class I have to call the isCoveredUnderWarranty() as well to see if it is covered under the extended warranty and then have it be called via the checkWarrantyStatus() in the car method. I hope this makes sense. So to sum it up I need to in the child class have it check the isCoveredUnderWarranty with extended warranty info. Then it has to move to the parent class so it can be called via checkWarrantyStatus. Here is my code, I have 1 error. public class Car { public int year; public String make; public String model; public int currentMiles; public int warrantyMiles; public int warrantyYears; int currentYear =java.util.Calendar.getInstance().get(java.util.Calendar.YEAR); /** construct car object with specific parameters*/ public Car (int y, String m, String mod, int mi){ this.year = y; this.make = m; this.model = mod; this.currentMiles = mi; } public int getWarrantyMiles() { return warrantyMiles; } public void setWarrantyMiles(int warrantyMiles) { this.warrantyMiles = warrantyMiles; } public int getWarrantyYears() { return warrantyYears; } public void setWarrantyYears(int warrantyYears) { this.warrantyYears = warrantyYears; } public boolean isCoveredUnderWarranty(){ if (currentMiles < warrantyMiles){ if (currentYear < (year+ warrantyYears)) return true; } return false; } public void checkWarrantyStatus(){ if (isCoveredUnderWarranty()){ System.out.println("Your car " + year+ " " + make+ " "+ model+ " With "+ currentMiles +" is still covered under warranty"); } else System.out.println("Your car " + year+ " " + make+ " "+ model+ " With "+ currentMiles +" is out of warranty"); } } public class CertifiedPreOwnCar extends Car{ public CertifiedPreOwnCar(int y, String m, String mod, int mi) { super(mi, m, mod, y); } public int extendedWarrantyYears; public int extendedWarrantyMiles; public int getExtendedWarrantyYears() { return extendedWarrantyYears; } public void setExtendedWarrantyYears(int extendedWarrantyYears) { this.extendedWarrantyYears = extendedWarrantyYears; } public int getExtendedWarrantyMiles() { return extendedWarrantyMiles; } public void setExtendedWarrantyMiles(int extendedWarrantyMiles) { this.extendedWarrantyMiles = extendedWarrantyMiles; } public boolean isCoveredUnderWarranty() { if (currentMiles < extendedWarrantyMiles){ if (currentYear < (year+ extendedWarrantyYears)) return true; } return false; } } public class TestCar { public static void main(String[] args) { Car car1 = new Car(2014, "Honda", "Civic", 255); car1.setWarrantyMiles(60000); car1.setWarrantyYears(5); car1.checkWarrantyStatus(); Car car2 = new Car(2000, "Ferrari", "F355", 8500); car2.setWarrantyMiles(20000); car2.setWarrantyYears(7); car2.checkWarrantyStatus(); CertifiedPreOwnCar car3 = new CertifiedPreOwnCar(2000, "Honda", "Accord", 65000); car3.setWarrantyYears(3); car3.setWarrantyMiles(30000); car3.setExtendedWarrantyMiles(100000); car3.setExtendedWarrantyYears(7); car3.checkWarrantyStatus(); } }

    Read the article

  • Pig: Count number of keys in a map

    - by Donald Miner
    I'd like to count the number of keys in a map in Pig. I could write a UDF to do this, but I was hoping there would be an easier way. data = LOAD 'hbase://MARS1' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'A:*', '-loadKey true -caching=100000') AS (id:bytearray, A_map:map[]); In the code above, I want to basically build a histogram of id and how many items in column family A that key has. In hoping, I tried c = FOREACH data GENERATE id, COUNT(A_map); but that unsurprisingly didn't work. Or, perhaps someone can suggest a better way to do this entirely. If I can't figure this out soon I'll just write a Java MapReduce job or a Pig UDF.

    Read the article

  • C#/.NET Little Wonders: Interlocked CompareExchange()

    - by James Michael Hare
    Once again, in this series of posts I look at the parts of the .NET Framework that may seem trivial, but can help improve your code by making it easier to write and maintain. The index of all my past little wonders posts can be found here. Two posts ago, I discussed the Interlocked Add(), Increment(), and Decrement() methods (here) for adding and subtracting values in a thread-safe, lightweight manner.  Then, last post I talked about the Interlocked Read() and Exchange() methods (here) for safely and efficiently reading and setting 32 or 64 bit values (or references).  This week, we’ll round out the discussion by talking about the Interlocked CompareExchange() method and how it can be put to use to exchange a value if the current value is what you expected it to be. Dirty reads can lead to bad results Many of the uses of Interlocked that we’ve explored so far have centered around either reading, setting, or adding values.  But what happens if you want to do something more complex such as setting a value based on the previous value in some manner? Perhaps you were creating an application that reads a current balance, applies a deposit, and then saves the new modified balance, where of course you’d want that to happen atomically.  If you read the balance, then go to save the new balance and between that time the previous balance has already changed, you’ll have an issue!  Think about it, if we read the current balance as $400, and we are applying a new deposit of $50.75, but meanwhile someone else deposits $200 and sets the total to $600, but then we write a total of $450.75 we’ve lost $200! Now, certainly for int and long values we can use Interlocked.Add() to handles these cases, and it works well for that.  But what if we want to work with doubles, for example?  Let’s say we wanted to add the numbers from 0 to 99,999 in parallel.  We could do this by spawning several parallel tasks to continuously add to a total: 1: double total = 0; 2:  3: Parallel.For(0, 10000, next => 4: { 5: total += next; 6: }); Were this run on one thread using a standard for loop, we’d expect an answer of 4,999,950,000 (the sum of all numbers from 0 to 99,999).  But when we run this in parallel as written above, we’ll likely get something far off.  The result of one of my runs, for example, was 1,281,880,740.  That is way off!  If this were banking software we’d be in big trouble with our clients.  So what happened?  The += operator is not atomic, it will read in the current value, add the result, then store it back into the total.  At any point in all of this another thread could read a “dirty” current total and accidentally “skip” our add.   So, to clean this up, we could use a lock to guarantee concurrency: 1: double total = 0.0; 2: object locker = new object(); 3:  4: Parallel.For(0, count, next => 5: { 6: lock (locker) 7: { 8: total += next; 9: } 10: }); Which will give us the correct result of 4,999,950,000.  One thing to note is that locking can be heavy, especially if the operation being locked over is trivial, or the life of the lock is a high percentage of the work being performed concurrently.  In the case above, the lock consumes pretty much all of the time of each parallel task – and the task being locked on is relatively trivial. Now, let me put in a disclaimer here before we go further: For most uses, lock is more than sufficient for your needs, and is often the simplest solution!    So, if lock is sufficient for most needs, why would we ever consider another solution?  The problem with locking is that it can suspend execution of your thread while it waits for the signal that the lock is free.  Moreover, if the operation being locked over is trivial, the lock can add a very high level of overhead.  This is why things like Interlocked.Increment() perform so well, instead of locking just to perform an increment, we perform the increment with an atomic, lockless method. As with all things performance related, it’s important to profile before jumping to the conclusion that you should optimize everything in your path.  If your profiling shows that locking is causing a high level of waiting in your application, then it’s time to consider lighter alternatives such as Interlocked. CompareExchange() – Exchange existing value if equal some value So let’s look at how we could use CompareExchange() to solve our problem above.  The general syntax of CompareExchange() is: T CompareExchange<T>(ref T location, T newValue, T expectedValue) If the value in location == expectedValue, then newValue is exchanged.  Either way, the value in location (before exchange) is returned. Actually, CompareExchange() is not one method, but a family of overloaded methods that can take int, long, float, double, pointers, or references.  It cannot take other value types (that is, can’t CompareExchange() two DateTime instances directly).  Also keep in mind that the version that takes any reference type (the generic overload) only checks for reference equality, it does not call any overridden Equals(). So how does this help us?  Well, we can grab the current total, and exchange the new value if total hasn’t changed.  This would look like this: 1: // grab the snapshot 2: double current = total; 3:  4: // if the total hasn’t changed since I grabbed the snapshot, then 5: // set it to the new total 6: Interlocked.CompareExchange(ref total, current + next, current); So what the code above says is: if the amount in total (1st arg) is the same as the amount in current (3rd arg), then set total to current + next (2nd arg).  This check and exchange pair is atomic (and thus thread-safe). This works if total is the same as our snapshot in current, but the problem, is what happens if they aren’t the same?  Well, we know that in either case we will get the previous value of total (before the exchange), back as a result.  Thus, we can test this against our snapshot to see if it was the value we expected: 1: // if the value returned is != current, then our snapshot must be out of date 2: // which means we didn't (and shouldn't) apply current + next 3: if (Interlocked.CompareExchange(ref total, current + next, current) != current) 4: { 5: // ooops, total was not equal to our snapshot in current, what should we do??? 6: } So what do we do if we fail?  That’s up to you and the problem you are trying to solve.  It’s possible you would decide to abort the whole transaction, or perhaps do a lightweight spin and try again.  Let’s try that: 1: double current = total; 2:  3: // make first attempt... 4: if (Interlocked.CompareExchange(ref total, current + i, current) != current) 5: { 6: // if we fail, go into a spin wait, spin, and try again until succeed 7: var spinner = new SpinWait(); 8:  9: do 10: { 11: spinner.SpinOnce(); 12: current = total; 13: } 14: while (Interlocked.CompareExchange(ref total, current + i, current) != current); 15: } 16:  This is not trivial code, but it illustrates a possible use of CompareExchange().  What we are doing is first checking to see if we succeed on the first try, and if so great!  If not, we create a SpinWait and then repeat the process of SpinOnce(), grab a fresh snapshot, and repeat until CompareExchnage() succeeds.  You may wonder why not a simple do-while here, and the reason it’s more efficient to only create the SpinWait until we absolutely know we need one, for optimal efficiency. Though not as simple (or maintainable) as a simple lock, this will perform better in many situations.  Comparing an unlocked (and wrong) version, a version using lock, and the Interlocked of the code, we get the following average times for multiple iterations of adding the sum of 100,000 numbers: 1: Unlocked money average time: 2.1 ms 2: Locked money average time: 5.1 ms 3: Interlocked money average time: 3 ms So the Interlocked.CompareExchange(), while heavier to code, came in lighter than the lock, offering a good compromise of safety and performance when we need to reduce contention. CompareExchange() - it’s not just for adding stuff… So that was one simple use of CompareExchange() in the context of adding double values -- which meant we couldn’t have used the simpler Interlocked.Add() -- but it has other uses as well. If you think about it, this really works anytime you want to create something new based on a current value without using a full lock.  For example, you could use it to create a simple lazy instantiation implementation.  In this case, we want to set the lazy instance only if the previous value was null: 1: public static class Lazy<T> where T : class, new() 2: { 3: private static T _instance; 4:  5: public static T Instance 6: { 7: get 8: { 9: // if current is null, we need to create new instance 10: if (_instance == null) 11: { 12: // attempt create, it will only set if previous was null 13: Interlocked.CompareExchange(ref _instance, new T(), (T)null); 14: } 15:  16: return _instance; 17: } 18: } 19: } So, if _instance == null, this will create a new T() and attempt to exchange it with _instance.  If _instance is not null, then it does nothing and we discard the new T() we created. This is a way to create lazy instances of a type where we are more concerned about locking overhead than creating an accidental duplicate which is not used.  In fact, the BCL implementation of Lazy<T> offers a similar thread-safety choice for Publication thread safety, where it will not guarantee only one instance was created, but it will guarantee that all readers get the same instance.  Another possible use would be in concurrent collections.  Let’s say, for example, that you are creating your own brand new super stack that uses a linked list paradigm and is “lock free”.  We could use Interlocked.CompareExchange() to be able to do a lockless Push() which could be more efficient in multi-threaded applications where several threads are pushing and popping on the stack concurrently. Yes, there are already concurrent collections in the BCL (in .NET 4.0 as part of the TPL), but it’s a fun exercise!  So let’s assume we have a node like this: 1: public sealed class Node<T> 2: { 3: // the data for this node 4: public T Data { get; set; } 5:  6: // the link to the next instance 7: internal Node<T> Next { get; set; } 8: } Then, perhaps, our stack’s Push() operation might look something like: 1: public sealed class SuperStack<T> 2: { 3: private volatile T _head; 4:  5: public void Push(T value) 6: { 7: var newNode = new Node<int> { Data = value, Next = _head }; 8:  9: if (Interlocked.CompareExchange(ref _head, newNode, newNode.Next) != newNode.Next) 10: { 11: var spinner = new SpinWait(); 12:  13: do 14: { 15: spinner.SpinOnce(); 16: newNode.Next = _head; 17: } 18: while (Interlocked.CompareExchange(ref _head, newNode, newNode.Next) != newNode.Next); 19: } 20: } 21:  22: // ... 23: } Notice a similar paradigm here as with adding our doubles before.  What we are doing is creating the new Node with the data to push, and with a Next value being the original node referenced by _head.  This will create our stack behavior (LIFO – Last In, First Out).  Now, we have to set _head to now refer to the newNode, but we must first make sure it hasn’t changed! So we check to see if _head has the same value we saved in our snapshot as newNode.Next, and if so, we set _head to newNode.  This is all done atomically, and the result is _head’s original value, as long as the original value was what we assumed it was with newNode.Next, then we are good and we set it without a lock!  If not, we SpinWait and try again. Once again, this is much lighter than locking in highly parallelized code with lots of contention.  If I compare the method above with a similar class using lock, I get the following results for pushing 100,000 items: 1: Locked SuperStack average time: 6 ms 2: Interlocked SuperStack average time: 4.5 ms So, once again, we can get more efficient than a lock, though there is the cost of added code complexity.  Fortunately for you, most of the concurrent collection you’d ever need are already created for you in the System.Collections.Concurrent (here) namespace – for more information, see my Little Wonders – The Concurent Collections Part 1 (here), Part 2 (here), and Part 3 (here). Summary We’ve seen before how the Interlocked class can be used to safely and efficiently add, increment, decrement, read, and exchange values in a multi-threaded environment.  In addition to these, Interlocked CompareExchange() can be used to perform more complex logic without the need of a lock when lock contention is a concern. The added efficiency, though, comes at the cost of more complex code.  As such, the standard lock is often sufficient for most thread-safety needs.  But if profiling indicates you spend a lot of time waiting for locks, or if you just need a lock for something simple such as an increment, decrement, read, exchange, etc., then consider using the Interlocked class’s methods to reduce wait. Technorati Tags: C#,CSharp,.NET,Little Wonders,Interlocked,CompareExchange,threading,concurrency

    Read the article

  • Problems with Widgets in dojox DataGrid

    - by Kitson
    I am trying to include some editing Widgets in my dojox.grid.DataGrid seem to be having a lot of difficulty. I have tried everything I can think of to get it to work, but something just isn't going right. When I started having problems, I tried to copy almost exactly from the grid tests and model my "breakout" of code just like that, but without success. Basic editing of the Grid seems to work. In the example below, the "Events" column allows edits, but the two columns that are using the cellType attribute don't work. In fact they also seem to ignore the other attributes (like the styles) which would seem to indicate that some sort of issue was run into, but there is nothing in FireBug. Also I get the same behaviour between Chrome and Firefox. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <head> <title>Insert title here</title> <link id="themeStyles" rel="stylesheet" href="javascript/dojotoolkit/dijit/themes/tundra/tundra.css"> <style type="text/css"> @import "css/gctilog.css"; @import "javascript/dojotoolkit/dojo/resources/dojo.css"; @import "javascript/dojotoolkit/dijit/themes/tundra/tundra.css"; @import "javascript/dojotoolkit/dojox/grid/resources/Grid.css"; @import "javascript/dojotoolkit/dojox/grid/resources/tundraGrid.css"; @import "javascript/dojotoolkit/ocp/resources/MultiStateCheckBox.css"; </style> <script type="text/javascript" src="javascript/dojotoolkit/dojo/dojo.js" djConfig="parseOnLoad:true, isDebug:true, locale:'en-gb'"></script> <script type="text/javascript"> dojo.require("dojo.currency"); dojo.require("dijit.dijit"); dojo.require("dijit.form.HorizontalSlider"); dojo.require("dojox.data.JsonRestStore"); dojo.require("dojox.grid.DataGrid"); dojo.require("dojox.layout.ExpandoPane"); dojo.require("dojox.timing"); dojo.require("ocp.MultiStateCheckBox"); dojo.require("dojo.parser"); formatCurrency = function(inDatum){ return isNaN(inDatum) ? '...' : dojo.currency.format(inDatum, this.constraint); } </script> <script type="text/javascript" src="javascript/formatter.js"></script> <script type="text/javascript" src="javascript/utilities.js"></script> </head> <body class="tundra"> <div name="labelCallids">Call IDs</div> <div dojoType="dojox.data.JsonRestStore" id="callidStore4" jsId="callidStore4" target="logmap/maps.php/maps/4/callids/" idAttribute="callid"></div> <table dojoType="dojox.grid.DataGrid" id="callidGrid4" store="callidStore4" query="{ callid: '*' }" style="width: 950px; border: 1px solid rgb(0,156,221); margin-left: 15px;" clientSort="false" autoHeight="10" noDataMessage="No Call IDs Available..."> <thead> <tr> <th field="callid" width="375px">Call ID</th> <th cellType="dojox.grid.cells.ComboBox" field="type" options="SIP,TLib" editable="true" width="10em" styles='text-align: center;'>Type</th> <th field="event_count" width="40px" editable="true" styles="text-align: right;">Events</th> <th field="start_ts" width="75px" formatter="secToHourMinSecMS">Start</th> <th field="end_ts" width="75px" formatter="secToHourMinSecMS">End</th> <th field="duration" width="75px" formatter="secToHourMinSecMS">Duration</th> <th cellType="dojox.grid.cells._Widget" widgetClass="dijit.form.HorizontalSlider" field="include" formatter="formatCurrency" constraint="{currency:'EUR'}" editable="true" width="10em" styles='text-align: right;'>Amount</th> </tr> </thead> </table> </body> </html> Is there anything that I am missing. It would seem to be fundamental, but I just can't seem to see it. [EDIT] What I have done instead is return a dijit Widget using the formatter to return a widget. So in the declarative model, I specify something like this: <th field="type" formatter="getMultiField" width="10em" styles='text-align: center;'>Type</th> And then I wrote a JavaScript function like the below to return the widget I wanted. function getMultiField(value) { var jsonValue = JSON.parse(value); //I provide the value of the widget as JSON //from my data store, so I need to parse it var control = new ocp.MultiStateCheckBox({ //my custom widget id : "dMSCB"+(new Date).getTime()+Math.ceil(Math.random()*100000), //generate a unique ID value : jsonValue.value, onChange : function (value {...}) //code to manipulate the underlying data store }); return control; //The dojo 1.4 grid can handle a returned Widget }

    Read the article

  • Shadow volume shader optimization (GLSL)

    - by Soubok
    I wondering if there is a way to optimize this vertex shader. This vertex shader projects (in the light direction) a vertex to the far plane if it is in the shadow. void main(void) { vec3 lightDir = (gl_ModelViewMatrix * gl_Vertex - gl_LightSource[0].position).xyz; // if the vertex is lit if ( dot(lightDir, gl_NormalMatrix * gl_Normal) < 0.01 ) { // don't move it gl_Position = ftransform(); } else { // move it far, is the light direction vec4 fin = gl_ProjectionMatrix * ( gl_ModelViewMatrix * gl_Vertex + vec4(normalize(lightDir) * 100000.0, 0.0) ); if ( fin.z > fin.w ) // if fin is behind the far plane fin.z = fin.w; // move to the far plane (needed for z-fail algo.) gl_Position = fin; } }

    Read the article

  • Java ThreadPoolExecutor getting stuck while using ArrayBlockingQueue

    - by Ravi Rao
    Hi, I'm working on some application and using ThreadPoolExecutor for handling various tasks. ThreadPoolExecutor is getting stuck after some duration. To simulate this in a simpler environment, I've written a simple code where I'm able to simulate the issue. import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.RejectedExecutionHandler; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; public class MyThreadPoolExecutor { private int poolSize = 10; private int maxPoolSize = 50; private long keepAliveTime = 10; private ThreadPoolExecutor threadPool = null; private final ArrayBlockingQueue&lt;Runnable&gt; queue = new ArrayBlockingQueue&lt;Runnable&gt;( 100000); public MyThreadPoolExecutor() { threadPool = new ThreadPoolExecutor(poolSize, maxPoolSize, keepAliveTime, TimeUnit.SECONDS, queue); threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() { @Override public void rejectedExecution(Runnable runnable, ThreadPoolExecutor threadPoolExecutor) { System.out .println(&quot;Execution rejected. Please try restarting the application.&quot;); } }); } public void runTask(Runnable task) { threadPool.execute(task); } public void shutDown() { threadPool.shutdownNow(); } public ThreadPoolExecutor getThreadPool() { return threadPool; } public void setThreadPool(ThreadPoolExecutor threadPool) { this.threadPool = threadPool; } public static void main(String[] args) { MyThreadPoolExecutor mtpe = new MyThreadPoolExecutor(); for (int i = 0; i &lt; 1000; i++) { final int j = i; mtpe.runTask(new Runnable() { @Override public void run() { System.out.println(j); } }); } } } Try executing this code a few times. It normally print outs the number on console and when all threads end, it exists. But at times, it finished all task and then is not getting terminated. The thread dump is as follows: MyThreadPoolExecutor [Java Application] MyThreadPoolExecutor at localhost:2619 (Suspended) Daemon System Thread [Attach Listener] (Suspended) Daemon System Thread [Signal Dispatcher] (Suspended) Daemon System Thread [Finalizer] (Suspended) Object.wait(long) line: not available [native method] ReferenceQueue&lt;T&gt;.remove(long) line: not available ReferenceQueue&lt;T&gt;.remove() line: not available Finalizer$FinalizerThread.run() line: not available Daemon System Thread [Reference Handler] (Suspended) Object.wait(long) line: not available [native method] Reference$Lock(Object).wait() line: 485 Reference$ReferenceHandler.run() line: not available Thread [pool-1-thread-1] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-2] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-3] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-4] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-6] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-8] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-5] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-10] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-9] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [pool-1-thread-7] (Suspended) Unsafe.park(boolean, long) line: not available [native method] LockSupport.park(Object) line: not available AbstractQueuedSynchronizer$ConditionObject.await() line: not available ArrayBlockingQueue&lt;E&gt;.take() line: not available ThreadPoolExecutor.getTask() line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available Thread [DestroyJavaVM] (Suspended) C:\Program Files\Java\jre1.6.0_07\bin\javaw.exe (Jun 17, 2010 10:42:33 AM) In my actual application,ThreadPoolExecutor threads go in this state and then it stops responding. Regards, Ravi Rao

    Read the article

  • javamail vs sendmail performance during bulk email

    - by Glenn2041
    Hi all, I'm writing a Java mass emailer application to send out emails to send between 50,000 to 100,000 a day to users. The current plan is to delegate the sending to delegate to sendmail (on the local unix server). From our testing sendmail is sending a maximum of 5 emails per second. Would JavaMail be a faster option? Does anyone know what a faster way to send emails. We want to get this process out as quick as possible. Edit: BTW, a pdf will be attached too

    Read the article

  • Where can I download a free, text-rich dataset?

    - by blee
    I want to do a bit of lightweight testing and bench-marking for full-text search, so the dataset should have the qualities: 10,000 - 100,000 records. good dispersion of English words. In CSV or Excel format--i.e. I don't want to access it via API. Something like books or movies with title and description fields would be perfect. I browsed the UCI Machine Learning Repo, but it was too number-oriented. Thanks!

    Read the article

  • WCF SSL secure transfer or large payloads without changing firewall.

    - by Sir Mix
    I need to transfer small amounts of data intermittently from clients to our server in a secure fashion and pull down large binary files from the server ocassionally. It's important for all this to be reliable. I'm anticipating 100,000 clients. I control both ends, but I want to deliver a solution that doesn't require changing the firewall for the majority of customers. A lag of one or two minutes before the information migrates to the server or comes down seems to be acceptable at this time. We need to make the connection secure, so was thinking about SSL, but open to suggestions. Basically, what is the best binding to use in this situation so that we have a secure transmission and the system handles the stress and load in a way that works for 95% of clients out of the box (firewalls will not block in majority of firewall configurations).

    Read the article

  • Adding 90000 XElement to XDocument

    - by Jon
    I have a Dictionary<int, MyClass> It contains 100,000 items 10,000 items value is populated whilst 90,000 are null. I have this code: var nullitems = MyInfoCollection.Where(x => x.Value == null).ToList(); nullitems.ForEach(x => LogMissedSequenceError(x.Key + 1)); private void LogMissedSequenceError(long SequenceNumber) { DateTime recordTime = DateTime.Now; var errors = MyXDocument.Descendants("ERRORS").FirstOrDefault(); if (errors != null) { errors.Add( new XElement("ERROR", new XElement("DATETIME", DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss:fff")), new XElement("DETAIL", "No information was read for expected sequence number " + SequenceNumber), new XAttribute("TYPE", "MISSED"), new XElement("PAGEID", SequenceNumber) ) ); } } This seems to take about 2 minutes to complete. I can't seem to find where the bottleneck might be or if this timing sounds about right? Can anyone see anything to why its taking so long?

    Read the article

  • relational database: how to design this table

    - by donpal
    I'm a database newbie designing a database. I'll use SO to ask my question because it's easier to ask it on something that you can see already, but it's not the same, it will just help me understand the right approach. As you can see, there are many questions here and each can have many answers. How should I store the answers in a table? Should I store all the answers in the SAME table with a unique id (make it the key) and just a new field for the question id? What if there are 100,000 answers like there is here? Do I still store them in 1 table? What keys should I use to minimize search time when I want to search for the answers of a specific question? The database is both read and write if that makes any difference in this case.

    Read the article

  • Slow insert speed in Postgresql memory tablespace

    - by Prashant
    Hi, I have a requirement where I need to store the records at rate of 10,000 records/sec into a database (with indexing on a few fields). Number of columns in one record is 25. I am doing a batch insert of 100,000 records in one transaction block. To improve the insertion rate, I changed the tablespace from disk to RAM.With that I am able to achieve only 5,000 inserts per second. I have also done the following tuning in the postgres config: Indexes : no fsync : false logging : disabled Other information: - Tablespace : RAM - Number of columns in one row : 25 (mostly integers) - CPU : 4 core, 2.5 GHz - RAM : 48 GB I am wondering why a single insert query is taking around 0.2 msec on average when database is not writing anything on disk (as I am using RAM based tablespace). Is there something I am doing wrong? Help appreciated. Prashant

    Read the article

  • Shipping Java code with data baked into the .jar

    - by Andrew
    I need to ship some Java code that has an associated set of data. It's a simulator for a device, and I want to be able to include all of the data used for the simulated records in the one .JAR file. In this case, each simulated record contains four fields (calling party, called party, start of call, call duration). What's the best way to do that? I've gone down the path of generating the data as Java statements, but IntelliJ doesn't seem particularly happy dealing with a 100,000 line Java source file! Is there a smarter way to do this? In the C#/.NET world I'd create the data as a separate file, embed it in the assembly as a resource, and then use reflection to pull that out at runtime and access it. I'm unsure of what the appropriate analogy is in the Java world. FWIW, Java 1.6, shipping for Solaris.

    Read the article

  • Yahoo YQL Rate Limits

    - by catlan
    I'm a bit unsure about the Usage Information and Limits of Yahoo YQL. Per application limit (identified by your Access Key): 100,000 calls per day. Per IP limits: /v1/public/: 1,000 calls per hour; /v1/yql/: 10,000 calls per hour. Do I require an application/access key for the /v1/public/ interface, non of the examples uses one. If I don't need an application key and only access the /v1/public/ interface I only have do worry about the IP limits of 1,000 calls per hour, right?

    Read the article

  • F# performance question: what is the compiler doing?

    - by Stephen Swensen
    Referencing this code: http://stackoverflow.com/questions/2840714/f-static-member-type-constraints/2842037#2842037 Why is, for example, [1L..100000L] |> List.map (fun n -> factorize gL n) significantly slower than [1L..100000L] |> List.map (fun n -> factorize (G_of 1L) n) By looking at Reflector, I can see that the compiler is treating each of these in very different ways, but there is too much going on for me to decipher the essential difference. Naively I assumed the former would perform better than the later because gL is precomputed whereas G_of 1L has to be computed 100,000 times (at least it appears that way).

    Read the article

  • Create date efficiently

    - by Dave Jarvis
    On Pavel's page is the following function: CREATE OR REPLACE FUNCTION makedate(year int, dayofyear int) RETURNS date AS $$ SELECT (date '0001-01-01' + ($1 - 1) * interval '1 year' + ($2 - 1) * interval '1 day'):: date $$ LANGUAGE sql; I have the following code: makedate(y.year,1) What is the fastest way in PostgreSQL to create a date for January 1st of a given year? Pavel's function would lead me to believe it is: date '0001-01-01' + y.year * interval '1 year' + interval '1 day'; My thought would be more like: to_date( y.year||'-1-1', 'YYYY-MM-DD'); Am looking for the fastest way using PostgreSQL 8.4. (The query that uses the date function can select between 100,000 and 1 million records, so it needs speed.) Thank you!

    Read the article

  • Web service performance testing plan, Microsoft .NET WS, SQL

    - by zxed
    Trying to answer a question to come up with a testing plan. It has to do with using a website and/or webservice that queries a sql server to get data and display to user. * Solution must be able to handle an estimated 2000 users, approximately 700 concurrent users, 10,000 + website hits a month. Database calls should handle 100,000 queries via the website/webservice a month. The system is used at multiple times during a 24 hour period; however networking and bandwidth traffic decreases after 5 pm * two windows 2003 servers are used, one for web, another for sql. Both are located in the same room. User access is varied and users can be far/near (its a centralized system), users access via www

    Read the article

< Previous Page | 11 12 13 14 15 16 17 18 19 20  | Next Page >