Web Crawling Sites with Javascripts or web forms

Posted by Jojo on Stack Overflow See other posts from Stack Overflow or by Jojo
Published on 2010-03-30T10:51:25Z Indexed on 2010/03/30 10:53 UTC
Read the original article Hit count: 337

Filed under:

JavaScript

|

webforms

|

c#

|

Windows

|

Development

Hello everyone, I have a webcrawler application. It successfully crawled most common and simple sites. Now i ran into some types of websites wherein HTML documents are dynamically generated through FORMS or javascripts. I believe they can be crawled and I just don't know how. Now, these websites do not show the actual HTML page. I mean if I browse that page in IE or firefox, the HTML code does not match what's actually in the IE or firefox. These sites contain textboxes, checkboxes, etc... so I believe they are what they call "Web Forms". Actually I am not much familiar with web development so correct me if I'm wrong.

My question is, does anyone in similar situation as I am now and have successfully solved these types of "challenges"? Does anyone know of a certain book or article regarding web crawling? Those that pertains to these advanced type of websites?

Thanks.

© Stack Overflow or respective owner

Related posts about JavaScript

CHAT ROOMs 7 by 6

as seen on Stack Overflow - Search for 'Stack Overflow'
I am looking for chatroom on one page with 7 loggedin users and 6+rows for say 42 users.these users will keep on adding wthnew users.Need urgent help.A PRETTY UNUSUAL Q FOR MOST OF U.What is MORE REQ new features: Usernames are unique to users currently chatting You can see a "currently chatting"… >>> More
Integrating JavaScript Unit Tests with Visual Studio

as seen on Stephen Walter - Search for 'Stephen Walter'
Modern ASP.NET web applications take full advantage of client-side JavaScript to provide better interactivity and responsiveness. If you are building an ASP.NET application in the right way, you quickly end up with lots and lots of JavaScript code. When writing server code, you should be writing… >>> More
Has Javascript developed beyond what it was originally designed to do?

as seen on Programmers - Search for 'Programmers'
I've been talking with a friend about the purpose of Javascript, when and how it should be used, etc. He quoted that: JavaScript was designed to add interactivity to HTML pages [...] JavaScript gives HTML designers a programming tool HTML authors are normally not programmers… >>> More
PHP, javascript, single quote problems with IE when passing variable from ajax post to javascript fu

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi! I have been trying to get this to work for a while, and I suspect there's an easy solution that I just can't find. My head feels like jelly and I would really appreciate any help. My main page.php makes a .post() to backend.php and fetches a list of cities which it echoes in the form of: <li… >>> More
Javascript in XSL that is loaded by Javascript

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there anyway to have javascript run when a XSL sheet has been applied to an XML file by Javascript? I am using a JQuery plugin to apply the sheet to the xml but the javascript that is located inside of the XSL file will not run. I put the Javascript at the bottom of the file and it still does… >>> More

Related posts about webforms

WebForms vs MVC (again)

as seen on Dot net Slackers - Search for 'Dot net Slackers'
There's a new video up on www.asp.net which aims to help developers pick between ASP.NET WebForms and ASP.NET MVC. The video boils down to 5 benefits per technology which Microsoft thinks you should consider. Let's go over the points, shall we? First, ASP.NET WebForms: 1 - Familiar control… >>> More
WebForms vs MVC (again)

as seen on Dot net Slackers - Search for 'Dot net Slackers'
There's a new video up on www.asp.net which aims to help developers pick between ASP.NET WebForms and ASP.NET MVC. The video boils down to 5 benefits per technology which Microsoft thinks you should consider. Let's go over the points, shall we? First, ASP.NET WebForms: 1 - Familiar control… >>> More
Simple ‘In-Progress’ Message for Time Consuming Tasks in ASP.NET Pages

as seen on Code Project - Search for 'Code Project'
‘In-Progress’ Message & Disabling controls in ASP.NET Pages using JavaScript >>> More
ASP.NET Podcast Show #148 - ASP.NET WebForms to build a Mobile Web Application

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Check the podcast site for the original url. This is the video and source code for an ASP.NET WebForms app that I wrote that is optimized for the iPhone and mobile environments. Subscribe to everything. Subscribe to WMV. Subscribe to M4V for iPhone/iPad. Subscribe to MP3. Download… >>> More
How to connect a WordPress contact form to another database which uses a form script on a static site?

as seen on Programmers - Search for 'Programmers'
Static Site B has two separate contact form scripts. One script processes leads via a script developed using Cold Fusion. Another script processes leads via a script using MySql Database. New Site A is being developed using WordPress. How do I use a WordPress Contact Form plug-in to integrate these… >>> More