Search Results

Search found 25204 results on 1009 pages for 'event stream processing'.

Page 396/1009 | < Previous Page | 392 393 394 395 396 397 398 399 400 401 402 403  | Next Page >

  • Downloading a webpage in C# example

    - by Chris
    I am trying to understand some example code on this web page: (http://www.csharp-station.com/HowTo/HttpWebFetch.aspx) that downloads a file from the internet. The piece of code quoted below goes through a loop getting chunks of data and saving them to a string until all the data has been downloaded. As I understand it, "count" contains the size of the downloaded chunk and the loop runs until count is 0 (an empty chunk of data is downloaded). My question is, isn't it possible that count could be 0 without the file being completely downloaded? Say if the network connection is interrupted, the stream may not have any data to read on a pass of the loop and count should be 0, ending the download prematurely. Or does ResStream.Read stop the program until it gets data? Is this the correct way to save a stream? int count = 0; do { // fill the buffer with data count = resStream.Read(buf, 0, buf.Length); // make sure we read some data if (count != 0) { // translate from bytes to ASCII text tempString = Encoding.ASCII.GetString(buf, 0, count); // continue building the string sb.Append(tempString); } } while (count 0); // any more data to read?

    Read the article

  • reporting tool/viewer for large datasets

    - by FrustratedWithFormsDesigner
    I have a data processing system that generates very large reports on the data it processes. By "large" I mean that a "small" execution of this system produces about 30 MB of reporting data when dumped into a CSV file and a large dataset is about 130-150 MB (I'm sure someone out there has a bigger idea of "large" but that's not the point... ;) Excel has the ideal interface for the report consumers in the form of its Data Lists: users can filter and segment the data on-the-fly to see the specific details that they are interested in - they can also add notes and markup to the reports, create charts, graphs, etc... They know how to do all this and it's much easier to let them do it if we just give them the data. Excel was great for the small test datasets, but it cannot handle these large ones. Does anyone know of a tool that can provide a similar interface as Excel data lists, but that can handle much larger files? The next tool I tried was MS Access, and found that the Access file bloats hugely (30 MB input file leads to about 70 MB Access file, and when I open the file, run a report and close it the file's at 120-150 MB!), the import process is slow and very manual (currently, the CSV files are created by the same plsql script that runs the main process so there's next to no intervention on my part). I also tried an Access database with linked tables to the database tables that store the report data and that was many times slower (for some reason, sqlplus could query and generate the report file in a minute or soe while Access would take anywhere from 2-5 minutes for the same data) (If it helps, the data processing system is written in PL/SQL and runs on Oracle 10g.)

    Read the article

  • Best Functional Approach

    - by dbyrne
    I have some mutable scala code that I am trying to rewrite in a more functional style. It is a fairly intricate piece of code, so I am trying to refactor it in pieces. My first thought was this: def iterate(count:Int,d:MyComplexType) = { //Generate next value n //Process n causing some side effects return iterate(count - 1, n) } This didn't seem functional at all to me, since I still have side effects mixed throughout my code. My second thought was this: def generateStream(d:MyComplexType):Stream[MyComplexType] = { //Generate next value n return Stream.cons(n, generateStream(n)) } for (n <- generateStream(initialValue).take(2000000)) { //process n causing some side effects } This seemed like a better solution to me, because at least I've isolated my functional value-generation code from the mutable value-processing code. However, this is much less memory efficient because I am generating a large list that I don't really need to store. This leaves me with 3 choices: Write a tail-recursive function, bite the bullet and refactor the value-processing code Use a lazy list. This is not a memory sensitive app (although it is performance sensitive) Come up with a new approach. I guess what I really want is a lazily evaluated sequence where I can discard the values after I've processed them. Any suggestions?

    Read the article

  • How to properly handle signals when using the worker thread pattern?

    - by ipartola
    I have a simple server that looks something like this: void *run_thread(void *arg) { // Communicate via a blocking socket } int main() { // Initialization happens here... // Main event loop while (1) { new_client = accept(socket, ...); pthread_create(&thread, NULL, &run_thread, *thread_data*); pthread_detach(thread); } // Do cleanup stuff: close(socket); // Wait for existing threads to finish exit(0); ) Thus when a SIGINT or SIGTERM is received I need to break out of the main event loop to get to the clean up code. Moreover most likely the master thread is waiting on the accept() call so it's not able to check some other variable to see if it should break;. Most of the advice I found was along the lines of this: http://devcry.blogspot.com/2009/05/pthreads-and-unix-signals.html (creating a special signal handling thread to catch all the signals and do processing on those). However, it's the processing portion that I can't really wrap my head around: how can I possibly tell the main thread to return from the accept() call and check on an external variable to see if it should break;?

    Read the article

  • Android: Problems downloading images and converting to bitmaps

    - by Mike
    Hi all, I am working on an application that downloads images from a url. The problem is that only some images are being correctly downloaded and others are not. First off, here is the problem code: public Bitmap downloadImage(String url) { HttpClient client = new DefaultHttpClient(); HttpResponse response = null; try { response = client.execute(new HttpGet(url)); } catch (ClientProtocolException cpe) { Log.i(LOG_FILE, "client protocol exception"); return null; } catch (IOException ioe) { Log.i(LOG_FILE, "IOE downloading image"); return null; } catch (Exception e) { Log.i(LOG_FILE, "Other exception downloading image"); return null; } // Convert images from stream to bitmap object try { Bitmap image = BitmapFactory.decodeStream(response.getEntity().getContent()); if(image==null) Log.i(LOG_FILE, "image conversion failed"); return image; } catch (Exception e) { Log.i(LOG_FILE, "Other exception while converting image"); return null; } } So what I have is a method that takes the url as a string argument and then downloads the image, converts the HttpResponse stream to a bitmap by means of the BitmapFactory.decodeStream method, and returns it. The problem is that when I am on a slow network connection (almost always 3G rather than Wi-Fi) some images are converted to null--not all of them, only some of them. Using a Wi-Fi connection works perfectly; all the images are downloaded and converted properly. Does anyone know why this is happening? Or better, how can I fix this? How would I even go about testing to determine the problem? Any help is awesome; thank you!

    Read the article

  • How Best to Replace PL/SQL with C#?

    - by Mike
    Hi, I write a lot of one-off Oracle SQL queries/reports (in Toad), and sometimes they can get complex, involving lots of unions, joins, and subqueries, and/or requiring dynamic SQL, and/or procedural logic. PL/SQL is custom made for handling these situations, but as a language it does not compare to C#, and even if it did, it's tooling does not, and if even that did, forcing yet another language on the team is something to be avoided whenever possible. Experience has shown me that using SQL for the set based processing, coupled with C# for the procedural processing, is a powerful combination indeed, and far more readable, maintainable and enhanceable than PL/SQL. So, we end up with a number of small C# programs which typically would construct a SQL query string piece by piece and/or run several queries and process them as needed. This kind of code could easily be a disaster, but a good developer can make this work out quite well, and end up with very readable code. So, I don't think it's a bad way to code for smaller DB focused projects. My main question is, how best to create and package all these little C# programs that run ad hoc queries and reports against the database? Right now I create little report objects in a DLL, developed and tested with NUnit, but then I continue to use NUnit as the GUI to select and run them. NUnit just happens to provide a nice GUI for this kind of thing, even after testing has been completed. I'm also interested in suggestions for reporting apps generally. I feel like there is a missing piece or product. The perfect tool would allow writing and running C# inside of Toad, or SQL inside of Visual Studio, along with simple reporting facilities. All ideas will be appreciated, but let's make the assumption that PL/SQL is not the solution.

    Read the article

  • How can I kill off a Python web app on GAE early following a redirect?

    - by Mike Hayes
    Hi Disclaimer: completely new to Python from a PHP background Ok I'm using Python on Google App Engine with Google's webapp framework. I have a function which I import as it contains things which need to be processed on each page. def some_function(self): if data['user'].new_user and not self.request.path == '/main/new': self.redirect('/main/new') This works fine when I call it, but how can I make sure the app is killed off after the redirection. I don't want anything else processing. For example I will do this: class Dashboard(webapp.RequestHandler): def get(self): some_function(self) #Continue with normal code here self.response.out.write('Some output here') I want to make sure that once the redirection is made in some_function() (which works fine), that no processing is done in the get() function following the redirection, nor is the "Some output here" outputted. What should I be looking at to make this all work properly? I can't just exit the script because the webapp framework needs to run. I realise that more than likely I'm just doing things in completely the wrong way any way for a Python app, so any guidance would be a great help. Hopefully I have explained myself properly and someone will be able to point me in the right direction. Thanks

    Read the article

  • How can I take advantage of IObservable/IObserver to get rid of my "god object"?

    - by Will
    In a system I'm currently working on, I have many components which are defined as interfaces and base classes. Each part of the system has some specific points where they interact with other parts of the system. For example, the data readying component readies some data which eventually needs to go to the data processing portion, the communications component needs to query different components for their status for relaying to the outside, etc. Currently, I glue these parts of the system together using a "god object", or an object with intimate knowledge of different parts of the system. It registers with events over here and shuttles the results to methods over there, creates a callback method here and returns the result of that method over there, and passes many requests through a multi-threaded queue for processing because it "knows" certain actions have to run on STA threads, etc. While its convenient, it concerns me that this one type knows so much about how everybody else in the system is designed. I'd much prefer a more generic hub that can be given instances which can expose events or methods or callbacks or that can consume these. I've been seeing more about the IObservable/IObserver features of the reactive framework and that are being rolled into .NET 4.0 (I believe). Can I leverage this pattern to help replace my "god object"? How should I go about doing this? Are there any resources for using this pattern for this specific purpose?

    Read the article

  • c# Unable to open file for reading

    - by Maks
    I'm writing a program that uses FileSystemWatcher to monitor changes to a given directory, and when it recieves OnCreated or OnChanged event, it copies those created/changed files to a specified directorie(s). At first I had problems with the fact that OnChanged/OnCreated events can be sent twice (not acceptable in case it needed to process 500MB file) but I made a way around this and with what I'm REALLY STUCKED with is getting the following IOException: The process cannot access the file 'C:\Where are Photos\bookmarks (11).html' because it is being used by another process. Thus, preventing the program from copying all the files it should. So as I mentioned, when user uses this program he/she specifes monitored directory, when user copies/creates/changes file in that directory, program should get OnCreated/OnChanged event and then copy that file to few other directories. Above error happens in all casess, if user copies few files that needs to owerwrite other ones in folder being monitored or when copying bulk of several files or even sometimes when copying one file in a monitored directory. Whole program is quite big so I'm sending the most important parts. OnCreated: private void OnCreated(object source, FileSystemEventArgs e) { AddLogEntry(e.FullPath, "created", ""); // Update last access data if it's file so the same file doesn't // get processed twice because of sending another event. if (fileType(e.FullPath) == 2) { lastPath = e.FullPath; lastTime = DateTime.Now; } // serves no purpose now, it will be remove soon string fileName = GetFileName(e.FullPath); // copies file from source to few other directories Copy(e.FullPath, fileName); Console.WriteLine("OnCreated: " + e.FullPath); } OnChanged: private void OnChanged(object source, FileSystemEventArgs e) { // is it directory if (fileType(e.FullPath) == 1) return; // don't mind directory changes itself // Only if enough time has passed or if it's some other file // because two events can be generated int timeDiff = ((TimeSpan)(DateTime.Now - lastTime)).Seconds; if ((timeDiff < minSecsDiff) && (e.FullPath.Equals(lastPath))) { Console.WriteLine("-- skipped -- {0}, timediff: {1}", e.FullPath, timeDiff); return; } // Update last access data for above to work lastPath = e.FullPath; lastTime = DateTime.Now; // Only if size is changed, the rest will handle other handlers if (e.ChangeType == WatcherChangeTypes.Changed) { AddLogEntry(e.FullPath, "changed", ""); string fileName = GetFileName(e.FullPath); Copy(e.FullPath, fileName); Console.WriteLine("OnChanged: " + e.FullPath); } } fileType: private int fileType(string path) { if (Directory.Exists(path)) return 1; // directory else if (File.Exists(path)) return 2; // file else return 0; } Copy: private void Copy(string srcPath, string fileName) { foreach (string dstDirectoy in paths) { string eventType = "copied"; string error = "noerror"; string path = ""; string dirPortion = ""; // in case directory needs to be made if (srcPath.Length > fsw.Path.Length) { path = srcPath.Substring(fsw.Path.Length, srcPath.Length - fsw.Path.Length); int pos = path.LastIndexOf('\\'); if (pos != -1) dirPortion = path.Substring(0, pos); } if (fileType(srcPath) == 1) { try { Directory.CreateDirectory(dstDirectoy + path); //Directory.CreateDirectory(dstDirectoy + fileName); eventType = "created"; } catch (IOException e) { eventType = "error"; error = e.Message; } } else { try { if (!overwriteFile && File.Exists(dstDirectoy + path)) continue; // create new dir anyway even if it exists just to be sure Directory.CreateDirectory(dstDirectoy + dirPortion); // copy file from where event occured to all specified directories using (FileStream fsin = new FileStream(srcPath, FileMode.Open, FileAccess.Read, FileShare.Read)) { using (FileStream fsout = new FileStream(dstDirectoy + path, FileMode.Create, FileAccess.Write)) { byte[] buffer = new byte[32768]; int bytesRead = -1; while ((bytesRead = fsin.Read(buffer, 0, buffer.Length)) > 0) fsout.Write(buffer, 0, bytesRead); } } } catch (Exception e) { if ((e is IOException) && (overwriteFile == false)) { eventType = "skipped"; } else { eventType = "error"; error = e.Message; // attempt to find and kill the process locking the file. // failed, miserably System.Diagnostics.Process tool = new System.Diagnostics.Process(); tool.StartInfo.FileName = "handle.exe"; tool.StartInfo.Arguments = "\"" + srcPath + "\""; tool.StartInfo.UseShellExecute = false; tool.StartInfo.RedirectStandardOutput = true; tool.Start(); tool.WaitForExit(); string outputTool = tool.StandardOutput.ReadToEnd(); string matchPattern = @"(?<=\s+pid:\s+)\b(\d+)\b(?=\s+)"; foreach (Match match in Regex.Matches(outputTool, matchPattern)) { System.Diagnostics.Process.GetProcessById(int.Parse(match.Value)).Kill(); } Console.WriteLine("ERROR: {0}: [ {1} ]", e.Message, srcPath); } } } AddLogEntry(dstDirectoy + path, eventType, error); } } I checked everywhere in my program and whenever I use some file I use it in using block so even writing event to log (class for what I ommited since there is probably too much code already in post) wont lock the file, that is it shouldn't since all operations are using using statement block. I simply have no clue who's locking the file if not my program "copy" process from user through Windows or something else. Right now I have two possible "solutions" (I can't say they are clean solutions since they are hacks and as such not desireable). Since probably the problem is with fileType method (what else could lock the file?) I tried changing it to this, to simulate "blocking-until-ready-to-open" operation: fileType: private int fileType(string path) { FileStream fs = null; int ret = 0; bool run = true; if (Directory.Exists(path)) ret = 1; else { while (run) { try { fs = new FileStream(path, FileMode.Open); ret = 2; run = false; } catch (IOException) { } finally { if (fs != null) { fs.Close(); fs.Dispose(); } } } } return ret; } This is working as much as I could tell (test), but... it's hack, not to mention other deficients. The other "solution" I could try (I didn't test it yet) is using GC.Collect() somewhere at the end of fileType() method. Maybe even worse "solution" than previous one. Can someone pleas tell me, what on earth is locking the file, preventing it from opening and how can I fix that? What am I missing to see? Thanks in advance.

    Read the article

  • iOS UIImageView dataWithContentsOfURL returning empty

    - by user761389
    I'm trying to display an image from a URL in a UIImageView and I'm seeing some very peculiar results. The bit of code that I'm testing with is below imageURL = @"http://images.shopow.co.uk/image/user_dyn/1073/32/32"; imageURL = @"http://images.shopow.co.uk/assets/profile_images/default/32_32/avatar-male-01.jpg"; NSURL *imageURLRes = [NSURL URLWithString:imageURL]; NSData *imageData = [NSData dataWithContentsOfURL:imageURLRes]; UIImage *image = [UIImage imageWithData:imageData]; NSLog(@"Image Data: %@", imageData); In it's current form I can see data in the output window which is what I'd expect. However if comment out the second imageURL so I'm referencing the first I'm getting empty data and therefore nil is being returned by imageWithData. What is possibly more confusing is that the first image is basically the same as the second but it's been through a PHP processing script. I'm nearly certain that it isn't the script that's causing the issue because if I use this instead imageURL = @"http://images.shopow.co.uk/image/product_dynimg/389620/32/32" the image is displayed and this uses the same image processing script. I'm struggling to find any difference in the images that would cause this to occur. Any help would be appreciated.

    Read the article

  • Help file not working

    - by meryl
    Hi, can anyone help me ? I wanto to play an audio file and whenever I press the stop button , the already played part of the file should be saved. Unfortunately , what I get is an audio file (.wav) which actually is unplayable. Thanks //**************************** void play_cut() { try { // First, we get the format of the input file final AudioFileFormat.Type fileType = AudioSystem.getAudioFileFormat(inputAudio).getType(); // Then, we get a clip for playing the audio. c = AudioSystem.getClip(); // We get a stream for playing the input file. AudioInputStream ais = AudioSystem.getAudioInputStream(inputAudio); // We use the clip to open (but not start) the input stream c.open(ais); // We get the format of the audio codec (not the file format we got above) final AudioFormat audioFormat = ais.getFormat(); c.start(); AudioInputStream startStream = new AudioInputStream(new FileInputStream(inputAudio), audioFormat, c.getLongFramePosition()); AudioSystem.write(startStream, fileType, outputAudio); } catch (UnsupportedAudioFileException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } catch (LineUnavailableException e) { e.printStackTrace(); } }// end play_cut //****************************

    Read the article

  • F# ref-mutable vars vs object fields

    - by rwallace
    I'm writing a parser in F#, and it needs to be as fast as possible (I'm hoping to parse a 100 MB file in less than a minute). As normal, it uses mutable variables to store the next available character and the next available token (i.e. both the lexer and the parser proper use one unit of lookahead). My current partial implementation uses local variables for these. Since closure variables can't be mutable (anyone know the reason for this?) I've declared them as ref: let rec read file includepath = let c = ref ' ' let k = ref NONE let sb = new StringBuilder() use stream = File.OpenText file let readc() = c := stream.Read() |> char // etc I assume this has some overhead (not much, I know, but I'm trying for maximum speed here), and it's a little inelegant. The most obvious alternative would be to create a parser class object and have the mutable variables be fields in it. Does anyone know which is likely to be faster? Is there any consensus on which is considered better/more idiomatic style? Is there another option I'm missing?

    Read the article

  • ASP.NET Image Upload Parameter Not Valid. Exception

    - by pennylane
    Hi Guys, Im just trying to save a file to disk using a posted stream from jquery uploadify I'm also getting Parameter not valid. On adding to the error message so i can tell where it blew up in production im seeing it blow up on: var postedBitmap = new Bitmap(postedFileStream) any help would be most appreciated public string SaveImageFile(Stream postedFileStream, string fileDirectory, string fileName, int imageWidth, int imageHeight) { string result = ""; string fullFilePath = Path.Combine(fileDirectory, fileName); string exhelp = ""; if (!File.Exists(fullFilePath)) { try { using (var postedBitmap = new Bitmap(postedFileStream)) { exhelp += "got past bmp creation" + fullFilePath; using (var imageToSave = ImageHandler.ResizeImage(postedBitmap, imageWidth, imageHeight)) { exhelp += "got past resize"; if (!Directory.Exists(fileDirectory)) { Directory.CreateDirectory(fileDirectory); } result = "Success"; postedBitmap.Dispose(); imageToSave.Save(fullFilePath, GetImageFormatForFile(fileName)); } exhelp += "got past save"; } } catch (Exception ex) { result = "Save Image File Failed " + ex.Message + ex.StackTrace; Global.SendExceptionEmail("Save Image File Failed " + exhelp, ex); } } return result; }

    Read the article

  • public class ImageHandler : IHttpHandler

    - by Ken
    cmd.Parameters.AddWithValue("@id", new system.Guid (imageid)); What using System reference would this require? Here is the handler: using System; using System.Collections.Specialized; using System.Web; using System.Web.Configuration; using System.Web.Security; using System.Globalization; using System.Configuration; using System.Data.SqlClient; using System.Data; using System.IO; using System.Web.Profile; using System.Drawing; public class ImageHandler : IHttpHandler { public void ProcessRequest(HttpContext context) { string imageid; if (context.Request.QueryString["id"] != null) imageid = (context.Request.QueryString["id"]); else throw new ArgumentException("No parameter specified"); context.Response.ContentType = "image/jpeg"; Stream strm = ShowProfileImage(imageid.ToString()); byte[] buffer = new byte[8192]; int byteSeq = strm.Read(buffer, 0, 8192); while (byteSeq > 0) { context.Response.OutputStream.Write(buffer, 0, byteSeq); byteSeq = strm.Read(buffer, 0, 8192); } //context.Response.BinaryWrite(buffer); } public Stream ShowProfileImage(String imageid) { string conn = ConfigurationManager.ConnectionStrings["MyConnectionString1"].ConnectionString; SqlConnection connection = new SqlConnection(conn); string sql = "SELECT image FROM Profile WHERE UserId = @id"; SqlCommand cmd = new SqlCommand(sql, connection); cmd.CommandType = CommandType.Text; cmd.Parameters.AddWithValue("@id", new system.Guid (imageid));//Failing Here!!!! connection.Open(); object img = cmd.ExecuteScalar(); try { return new MemoryStream((byte[])img); } catch { return null; } finally { connection.Close(); } } public bool IsReusable { get { return false; } } }

    Read the article

  • Uploading an xml direct to ftp

    - by Joshua Maerten
    i put this direct below a button: XmlDocument doc = new XmlDocument(); XmlElement root = doc.CreateElement("Login"); XmlElement id = doc.CreateElement("id"); id.SetAttribute("userName", usernameTxb.Text); id.SetAttribute("passWord", passwordTxb.Text); XmlElement name = doc.CreateElement("Name"); name.InnerText = nameTxb.Text; XmlElement age = doc.CreateElement("Age"); age.InnerText = ageTxb.Text; XmlElement Country = doc.CreateElement("Country"); Country.InnerText = countryTxb.Text; id.AppendChild(name); id.AppendChild(age); id.AppendChild(Country); root.AppendChild(id); doc.AppendChild(root); // Get the object used to communicate with the server. FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://users.skynet.be"); request.Method = WebRequestMethods.Ftp.UploadFile; request.UsePassive = false; // This example assumes the FTP site uses anonymous logon. request.Credentials = new NetworkCredential("fa490002", "password"); // Copy the contents of the file to the request stream. StreamReader sourceStream = new StreamReader(); byte[] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd()); sourceStream.Close(); request.ContentLength = fileContents.Length; Stream requestStream = request.GetRequestStream(); requestStream.Write(fileContents, 0, fileContents.Length); requestStream.Close(); FtpWebResponse response = (FtpWebResponse)request.GetResponse(); response.Close(); MessageBox.Show("Created SuccesFully!"); this.Close(); but i always get an error of the streamreader path, what do i need to place there ? the meening is, creating an account and when i press the button, an xml file is saved to, ftp://users.skynet.be/testxml/ the filename is from usernameTxb.Text + ".xml".

    Read the article

  • How Best to Replace Ugly Queries and Dynamic PL/SQL with C#?

    - by Mike
    Hi, I write a lot of one-off Oracle SQL queries (in Toad), and sometimes they can get complex, involving lots of unions, joins, and subqueries, and sometimes requiring dynamic SQL. That is, sometimes SQL queries require set based processing along with significant procedural processing. This is what PL/SQL is custom made for, but as a language it does not begin to compare to C#. Now and then I convert a PL/SQL procedure to C#, and am always amazed at how much cleaner and easier to both read and write the C# version is. The C# program might for example construct a SQL query string piece by piece and/or run several queries and process them as needed. The C# version is usually much faster as well, which must mean that I'm not very good at PL/SQL either. I do not currently have access to LINQ. My question is, how best to package all these little C# programs, which are really just mini reports, that is, replacements for ugly SQL queries? Right now I'm actually using NUnit to hold them, and calling each report a [Test], even though they aren't really tests. NUnit just happens to provide a convenient packaging framework.

    Read the article

  • How can I efficiently manipulate 500k records in SQL Server 2005?

    - by cdeszaq
    I am getting a large text file of updated information from a customer that contains updates for 500,000 users. However, as I am processing this file, I often am running into SQL Server timeout errors. Here's the process I follow in my VB application that processes the data (in general): Delete all records from temporary table (to remove last month's data) (eg. DELETE * FROM tempTable) Rip text file into the temp table Fill in extra information into the temp table, such as their organization_id, their user_id, group_code, etc. Update the data in the real tables based on the data computed in the temp table The problem is that I often run commands like UPDATE tempTable SET user_id = (SELECT user_id FROM myUsers WHERE external_id = tempTable.external_id) and these commands frequently time out. I have tried bumping the timeouts up to as far as 10 minutes, but they still fail. Now, I realize that 500k rows is no small number of rows to manipulate, but I would think that a database purported to be able to handle millions and millions of rows should be able to cope with 500k pretty easily. Am I doing something wrong with how I am going about processing this data? Please help. Any and all suggestions welcome.

    Read the article

  • Ftp throws WebException only in .NET 4.0

    - by Trakhan
    I have following c# code. It runs just fine when compiled against .NET framework 3.5 or 2.0 (I did not test it against 3.0, but it will most likely work too). The problem is, that it fails when built against .NET framework 4.0. FtpWebRequest Ftp = (FtpWebRequest)FtpWebRequest.Create(Url_ + '/' + e.Name); Ftp.Method = WebRequestMethods.Ftp.UploadFile; Ftp.Credentials = new NetworkCredential(Login_, Password_); Ftp.UseBinary = true; Ftp.KeepAlive = false; Ftp.UsePassive = true; Stream S = Ftp.GetRequestStream(); byte[] Content = null; bool Continue = false; do { try { Continue = false; Content = File.ReadAllBytes(e.FullPath); } catch (Exception) { Continue = true; System.Threading.Thread.Sleep(500); } } while (Continue); S.Write(Content, 0, Content.Length); S.Close(); FtpWebResponse Resp = (FtpWebResponse)Ftp.GetResponse(); if (Resp.StatusCode != FtpStatusCode.CommandOK) Console.WriteLine(Resp.StatusDescription); The problem is in call Stream S = Ftp.GetRequestStream();, which throws an en instance of WebException with message “The remote server returned an error: (500) Syntax error, command unrecognized”. Does anybody know why it is so? PS. I communicate virtual ftp server in ECM Alfresco.

    Read the article

  • suppress error using fread()

    - by Mikey1980
    I wrote a script for screen pops from our soft phone that locates a directory listing for the caller but occasionally they get "Can't read input stream" and the rest of the script quits. Does anyone have any suggestions on how to suppress error the error message and allow the rest of the script to run? Thanks! $i=0; $open = fopen("http://www.411.ca/whitepages/?n=".$_GET['phone'], "r"); $read = fread($open, 9024); fclose($open); eregi("'/(.*)';",$read,$got); $tv = ereg_replace('[[:blank:]]',' ',$got[1]); $url = "http://www.411.ca/".$tv; while ($name=="unknown" && $i < 15) { ## try 15 times before giving up $file = @ fopen($fn=$url,"r") or die ("Can't read input stream"); $text = fread($file,16384); if (preg_match('/"name">(.*?)<\/div>/is',$text,$found)) { $name = $found[1]; } if (preg_match('/"phone">(.*?)<\/div>/is',$text,$found)) { $phone = $found[1]; } if (preg_match('/"address">(.*?)<\/div>/is',$text,$found)) { $address = $found[1]; } fclose($file); $i++; }

    Read the article

  • Parsing data from txt file in J2ME

    - by CSFYPMAIL
    Basically I'm creating an indoor navigation system in J2ME. I've put the location details in a .txt file i.e. Locations names and their coordinates. Edges with respective start node and end node as well as the weight (length of the node). I put both details in the same file so users dont have to download multiple files to get their map working (it could become time consuming and seem complex). So what i did is to seperate the deferent details by typing out location Names and coordinates first, After that I seperated that section from the next section which is the edges by drawing a line with multiple underscores. Now the problem I'm having is parsing the different details into seperate arrays by setting up a command (while manually tokenizing the input stream) to check wether the the next token is an underscore. If it is, (in pseudocode terms), move to the next line in the stream, create a new array and fill it up with the next set of details. I found a some explanation/code HERE that does something similar but still parses into one array, although it manually tokenizes the input. Any ideas on what to do? Thanks Text File Explanation The text has the following format... <--1stSection--  /**   * Section one has the following format   * xCoordinate;yCoordinate;LocationName   */ 12;13;New York City 40;12;Washington D.C. ...e.t.c _________________________ <--(underscore divider) <--2ndSection--  /**   * Its actually an adjacency list but indirectly provides "edge" details.   * Its in this form   * StartNode/MainReferencePoint;Endnode1;distance2endNode1;Endnode2;distance2endNode2;...e.t.c   */ philadelphia;Washington D.C.;7;New York City;2 New York City;Florida;24;Illinois;71 ...e.t.c

    Read the article

  • Datastructure choices for highspeed and memory efficient detection of duplicate of strings

    - by Jonathan Holland
    I have a interesting problem that could be solved in a number of ways: I have a function that takes in a string. If this function has never seen this string before, it needs to perform some processing. If the function has seen the string before, it needs to skip processing. After a specified amount of time, the function should accept duplicate strings. This function may be called thousands of time per second, and the string data may be very large. This is a highly abstracted explanation of the real application, just trying to get down to the core concept for the purpose of the question. The function will need to store state in order to detect duplicates. It also will need to store an associated timestamp in order to expire duplicates. It does NOT need to store the strings, a unique hash of the string would be fine, providing there is no false positives due to collisions (Use a perfect hash?), and the hash function was performant enough. The naive implementation would be simply (in C#): Dictionary<String,DateTime> though in the interest of lowering memory footprint and potentially increasing performance I'm evaluating a custom data structures to handle this instead of a basic hashtable. So, given these constraints, what would you use? EDIT, some additional information that might change proposed implementations: 99% of the strings will not be duplicates. Almost all of the duplicates will arrive back to back, or nearly sequentially. In the real world, the function will be called from multiple worker threads, so state management will need to be synchronized.

    Read the article

  • std::ifstream buffer caching

    - by ledokol
    Hello everybody, In my application I'm trying to merge sorted files (keeping them sorted of course), so I have to iterate through each element in both files to write the minimal to the third one. This works pretty much slow on big files, as far as I don't see any other choice (the iteration has to be done) I'm trying to optimize file loading. I can use some amount of RAM, which I can use for buffering. I mean instead of reading 4 bytes from both files every time I can read once something like 100Mb and work with that buffer after that, until there will be no element in buffer, then I'll refill the buffer again. But I guess ifstream is already doing that, will it give me more performance and is there any reason? If fstream does, maybe I can change size of that buffer? added My current code looks like that (pseudocode) // this is done in loop int i1 = input1.read_integer(); int i2 = input2.read_integer(); if (!input1.eof() && !input2.eof()) { if (i1 < i2) { output.write(i1); input2.seek_back(sizeof(int)); } else input1.seek_back(sizeof(int)); output.write(i2); } } else { if (input1.eof()) output.write(i2); else if (input2.eof()) output.write(i1); } What I don't like here is seek_back - I have to seek back to previous position as there is no way to peek 4 bytes too much reading from file if one of the streams is in EOF it still continues to check that stream instead of putting contents of another stream directly to output, but this is not a big issue, because chunk sizes are almost always equal. Can you suggest improvement for that? Thanks.

    Read the article

  • Benefits of "Don't Fragment" on TCP Packets?

    - by taspeotis
    One of our customers is having trouble submitting data from our application (on their PC) to a server (different geographical location). When sending packets under 1100 bytes everything works fine, but above this we see TCP retransmitting the packet every few seconds and getting no response. The packets we are using for testing are about 1400 bytes (but less than 1472). I can send an ICMP ping to www.google.com that is 1472 bytes and get a response (so it's not their router/first few hops). I found that our application sets the DF flag for these packets, and I believe a router along the way to the server has an MTU less than/equal to 1100 and dropping the packet. This affects 1 client in 5000, but since everybody's routes will be different this is expected. The data is a SOAP envelope and we expect a SOAP response back. I can't justify WHY we do it, the code to do this was written by a previous developer. So... Are there are benefits OR justification to setting the DF flag on TCP packets for application data? I can think of reasons it is needed for network diagnostics applications but not in our situation (we want the data to get to the endpoint, fragmented or not). One of our sysadmins said that it might have something to do with us using SSL, but as far as I know SSL is like a stream and regardless of fragmentation, as long as the stream is rebuilt at the end, there's no problem. If there's no good justification I will be changing the behaviour of our application. Thanks in advance.

    Read the article

  • Protocol specification in XML

    - by Mathijs
    Is there a way to specify a packet-based protocol in XML, so (de)serialization can happen automatically? The context is as follows. I have a device that communicates through a serial port. It sends and receives a byte stream consisting of 'packets'. A packet is a collection of elementary data types and (sometimes) other packets. Some elements of packets are conditional; their inclusion depends on earlier elements. I have a C# application that communicates with this device. Naturally, I don't want to work on a byte-level throughout my application; I want to separate the protocol from my application code. Therefore I need to translate the byte stream to structures (classes). Currently I have implemented the protocol in C# by defining a class for each packet. These classes define the order and type of elements for each packet. Making class members conditional is difficult, so protocol information ends up in functions. I imagine XML that looks like this (note that my experience designing XML is limited): <packet> <field name="Author" type="int32" /> <field name="Nickname" type="bytes" size="4"> <condition type="range"> <field>Author</field> <min>3</min> <max>6</min> </condition> </field> </packet> .NET has something called a 'binary serializer', but I don't think that's what I'm looking for. Is there a way to separate protocol and code, even if packets 'include' other packets and have conditional elements?

    Read the article

  • using ftpWebRequest with an error: the remote server returned error 530 not loged in

    - by user1361207
    I am trying to use the ftpWebRequest in c# my code is // Get the object used to communicate with the server. FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://192.168.20.10/file.txt"); request.Method = WebRequestMethods.Ftp.UploadFile; // This example assumes the FTP site uses anonymous logon. request.Credentials = new NetworkCredential("dev\ftp", "devftp"); // Copy the contents of the file to the request stream. StreamReader sourceStream = new StreamReader(@"\file.txt"); byte[] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd()); sourceStream.Close(); request.ContentLength = fileContents.Length; request.UsePassive = true; Stream requestStream = request.GetRequestStream(); requestStream.Write(fileContents, 0, fileContents.Length); requestStream.Close(); FtpWebResponse response = (FtpWebResponse)request.GetResponse(); Console.WriteLine("Upload File Complete, status {0}", response.StatusDescription); response.Close(); and I get an Error in request.GetRequestStream(); the error is: the remote server returned error 530 not loged in if I try to go in to a browser page and in the url I write ftp://192.168.20.10/ the brows page is asking me for a name and password, I put the same name and password and I see all the files and folders in the ftp folder. can any one help me with this problem?

    Read the article

< Previous Page | 392 393 394 395 396 397 398 399 400 401 402 403  | Next Page >