Automated download of website content using ASP.net

Posted by Yaaqov on Stack Overflow See other posts from Stack Overflow or by Yaaqov
Published on 2010-03-24T20:20:31Z Indexed on 2010/03/24 20:23 UTC
Read the original article Hit count: 256

Filed under:

parse

|

ASP.NET

|

web-applications

|

screen-scraping

Using ASP.net, what methods can I use to do the following:

Open up a connection to a given URL to read HTML content
Parse the given URL for hyperlinks, and place them in an array
Loop through each hyperlink (only 1 level down), opening each one, saving the HTML contents in a table, and move to the next hyperlink until done.

If ASP.net is not up to the task, other languages or free scripts/toolkits would be acceptable.

Thanks.

© Stack Overflow or respective owner

Related posts about parse

System.Uri can't parse when password field contains a "#"

as seen on Stack Overflow - Search for 'Stack Overflow'
The following code throws System.UriFormatException: var uri = new UriBuilder("ftp://user:pass#[email protected]:21/fu/bar.zip"); System.UriFormatException: Invalid URI: A port was expected because of there is a colon (':') present but the port could not be parsed. Removing the # symbol from… >>> More
Method extension for safely type convert

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Recently I read good Russian post with many interesting extensions methods after then I remembered that I too have one good extension method “Safely type convert”. Idea of this method I got at last job. We often write code like this: int intValue; if (obj == null || !int.TryParse(obj… >>> More
LINQ 4 XML - What is the proper way to query deep in the tree structure?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have an XML structure that is 4 deep: <?xml version="1.0" encoding="utf-8"?> <EmailRuleList xmlns:xsd="EmailRules.xsd"> <TargetPST name="Tech Communities"> <Parse emailAsList="true" useJustDomain="false" fromAddress="false" toAddress="true"> <EmailRule address="@aspadvice… >>> More
Real-time log parsing and reporting

as seen on Server Fault - Search for 'Server Fault'
We have a small project we are working on part-time that runs on Nginx/MongoDB on Ubuntu 10.04 LTS Server. We'd like to be able to see reports on things like server load, requests/sec, response time, DB load, DB response time, etc. Is there an open source or free (as in beer) tool that can parse such… >>> More
php file downloads instead of being processed with ajax on apache

as seen on Server Fault - Search for 'Server Fault'
I have a small website where some content is displayed within a HTML tag using AJAX. The content is simply taken from another page on the same web site. However, sometimes instead of loading the parsed PHP file, the browser displays a download box instead. I downloaded the file and this is what it… >>> More

Related posts about ASP.NET

Migrating ASP.NET MVC 1.0 applications to ASP.NET MVC 2 RTM

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Note: ASP.NET MVC 2 RTM isn’t yet released! But this tool will help you get your ASP.NET MVC 1.0 applications ready for when it is! I have updated the MVC App Converter to convert projects from ASP.NET MVC 1.0 to ASP.NET MVC 2 RTM. This should be last the last major change to the MVC App Converter… >>> More
April 14th Links: ASP.NET, ASP.NET MVC, ASP.NET Web API and Visual Studio

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Here is the latest in my link-listing blog series: ASP.NET Easily overlooked features in VS 11 Express for Web: Good post by Scott Hanselman that highlights a bunch of easily overlooked improvements that are coming to VS 11 (and specifically the free express editions) for web development: unit… >>> More
Use ASP.NET 4 Browser Definitions with ASP.NET 3.5

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
We updated the browser definitions files included with ASP.NET 4 to include information on recent browsers and devices such as Google Chrome and the iPhone. You can use these browser definition files with earlier versions of ASP.NET such as ASP.NET 3 Read More......(read more) >>> More
ASP.NET webforms + ASP.NET Ajax versus ASP.NET MVC and Ajax framework freedom

as seen on Stack Overflow - Search for 'Stack Overflow'
If given the choice, which path would you take? ASP.NET Webforms + ASP.NET AJAX or ASP.NET MVC + JavaScript Framework of your Choice Are there any limitations that ASP.NET Webforms / ASP.NET AJAX has vis-a-vis MVC? >>> More
ASP.NET MVC 2 Released

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
I’m happy to announce that the final release of ASP.NET MVC 2 is now available for VS 2008/Visual Web Developer 2008 Express with ASP.NET 3.5. You can download and install it from the following locations: Download ASP.NET MVC 2 using the Microsoft Web Platform Installer Download… >>> More