Issue in Webscrapping in C# : Downloading and parsing zipped text files

Posted by user64094 on Programmers See other posts from Programmers or by user64094
Published on 2012-09-12T07:42:04Z Indexed on 2012/09/12 9:48 UTC
Read the original article Hit count: 231

Filed under:

c#

|

web-scraping

I am writing an webscrapper, to do the download content from a website. Traversing to the website/URL, triggers the creation of a temporary URL. This new URL has a zipped text file. This zipped file is to be downloaded and parsed. I have written a scrapper in C# using WebClient and its function - DownloadFileAsync(). The zipped file is read from the designated location on a trapped DownloadFileCompleted event. My issue : The Windows 'Open/Save dialog is triggered". This requires user input and the automation is disrupted.

Can you suggest a way to bypass the issue ? I am cool with rewriting the code using any alternate libraries. :) Thanks for reading,

© Programmers or respective owner

Related posts about c#

.NET WebRequest.PreAuthenticate not quite what it sounds like

as seen on West-Wind - Search for 'West-Wind'
I’ve run into the problem a few times now: How to pre-authenticate .NET WebRequest calls doing an HTTP call to the server – essentially send authentication credentials on the very first request instead of waiting for a server challenge first? At first glance this sound like it should be easy:… >>> More
HttpWebRequest and Ignoring SSL Certificate Errors

as seen on West-Wind - Search for 'West-Wind'
Man I can't believe this. I'm still mucking around with OFX servers and it drives me absolutely crazy how some these servers are just so unbelievably misconfigured. I've recently hit three different 3 major brokerages which fail HTTP validation with bad or corrupt certificates at least according to… >>> More
Dynamic Type to do away with Reflection

as seen on West-Wind - Search for 'West-Wind'
The dynamic type in C# 4.0 is a welcome addition to the language. One thing I’ve been doing a lot with it is to remove explicit Reflection code that’s often necessary when you ‘dynamically’ need to walk and object hierarchy. In the past I’ve had a number of ReflectionUtils that used string based expressions… >>> More
The dynamic Type in C# Simplifies COM Member Access from Visual FoxPro

as seen on West-Wind - Search for 'West-Wind'
I’ve written quite a bit about Visual FoxPro interoperating with .NET in the past both for ASP.NET interacting with Visual FoxPro COM objects as well as Visual FoxPro calling into .NET code via COM Interop. COM Interop with Visual FoxPro has a number of problems but one of them at least got a lot… >>> More
Finding a Relative Path in .NET

as seen on West-Wind - Search for 'West-Wind'
Here’s a nice and simple path utility that I’ve needed in a number of applications: I need to find a relative path based on a base path. So if I’m working in a folder called c:\temp\templates\ and I want to find a relative path for c:\temp\templates\subdir\test.txt I want to receive back subdir\test… >>> More

Related posts about web-scraping

Webscraping Google tasks via Google Calendar

as seen on Stack Overflow - Search for 'Stack Overflow'
As gmail and the task api is not available everywhere (eg: some companies block gmail but not calendar), is there a way to scrap google task through the calendar web interface ? The solution can be to use jQuery/Jaxer or a pointer to a browser script/plugin. >>> More
Source for Names to use in web scraping

as seen on Stack Overflow - Search for 'Stack Overflow'
Can anyone suggest a good source of names that I can use to help analyze some tables on web pages. The first column of the tables I am scraping have names alone, names and titles or just titles. The names can be as varied as John Smith to Vikram Saksena. I have been poking around for a compiled… >>> More
Web scraping with Python

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm currently trying to scrape a website that has fairly poorly-formatted HTML (often missing closing tags, no use of classes or ids so it's incredibly difficult to go straight to the element you want, etc.). I've been using BeautifulSoup with some success so far but every once and a while (though… >>> More
Web scraping with Python

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm currently trying to scrape a website that has fairly poorly-formatted HTML (often missing closing tags, no use of classes or ids so it's incredibly difficult to go straight to the element you want, etc.). I've been using BeautifulSoup with some success so far but every once and a while (though… >>> More
Python web scraping involving HTML tags with attributes

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to make a web scraper that will parse a web-page of publications and extract the authors. The skeletal structure of the web-page is the following: <html> <body> <div id="container"> <div id="contents"> <table> <tbody> <tr> <td class="author">####I… >>> More