Scraping ASP.NET site with Ruby
        Posted  
        
            by JillianK
        on Stack Overflow
        
        See other posts from Stack Overflow
        
            or by JillianK
        
        
        
        Published on 2010-05-12T21:36:50Z
        Indexed on 
            2010/05/12
            21:54 UTC
        
        
        Read the original article
        Hit count: 270
        
I would like to scrape the search results of this ASP.NET site using Ruby and preferably just using Hpricot (I cannot open an instance of Firefox): http://www.ngosinfo.gov.pk/SearchResults.aspx?name=&foa=0
However, I am having trouble figuring out how to go through each page of results. Basically, I need simulate clicking on links like these:
<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$2','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">2</a>
<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$3','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">3</a>
etc.
I tried using Net::HTTP to handle the post, but while that received the correct HTML, there were no search results (I'm probably not doing that correctly). In addition, the URL of the page does not contain any parameters indicating page, so it is not possible to force the results that way.
Any help would be greatly appreciated.
© Stack Overflow or respective owner