Search Results

Search found 27 results on 2 pages for 'webscraping'.

Page 1/2 | 1 2  | Next Page >

  • Webscraping Google tasks via Google Calendar

    - by ideotop
    As gmail and the task api is not available everywhere (eg: some companies block gmail but not calendar), is there a way to scrap google task through the calendar web interface ? The solution can be to use jQuery/Jaxer or a pointer to a browser script/plugin.

    Read the article

  • R webscraping: interrogating for date and importance

    - by adam.888
    I am able to webscrape a table from a webpage containing news library(XML) webpage <- "http://www.tradingeconomics.com/calendar" tables <- readHTMLTable(webpage ) n.rows <- unlist(lapply(tables, function(t) dim(t)[1])) dfcal <- as.data.frame(tables$calendar) However I do not know how to interrogate for date or for importance. For example how could I webscrape news from Jan 2014? I am able to do this on the webpage by altering button settings, but how can I do it from within R? I was also not able to collect the importance column data. Also are there better ways for collecting economic news from within R? I have looked on http://www.rseek.org/ but could not find anything. Thank you for your help.

    Read the article

  • How can I use R (Rcurl/XML packages ?!) to scrap this webpage ?

    - by Tal Galili
    Hi all, I have a (somewhat complex) webscraping challenge that I wish to accomplish and would love for some direction (to whatever level you feel like sharing) here goes: I would like to go through all the "species pages" present in this link: http://gtrnadb.ucsc.edu/ So for each of them I will go to: The species page link (for example: http://gtrnadb.ucsc.edu/Aero_pern/) And then to the "Secondary Structures" page link (for example: http://gtrnadb.ucsc.edu/Aero_pern/Aero_pern-structs.html) Inside that link I wish to scrap the data in the page so that I will have a long list containing this data (for example): chr.trna3 (1-77) Length: 77 bp Type: Ala Anticodon: CGC at 35-37 (35-37) Score: 93.45 Seq: GGGCCGGTAGCTCAGCCtGGAAGAGCGCCGCCCTCGCACGGCGGAGGcCCCGGGTTCAAATCCCGGCCGGTCCACCA Str: >>>>>>>..>>>>.........<<<<.>>>>>.......<<<<<.....>>>>>.......<<<<<<<<<<<<.... Where each line will have it's own list (inside the list for each "trna" inside the list for each animal) I remember coming across the packages Rcurl and XML (in R) that can allow for such a task. But I don't know how to use them. So what I would love to have is: 1. Some suggestion on how to build such a code. 2. And recommendation for how to learn the knowledge needed for performing such a task. Thanks for any help, Tal

    Read the article

  • HTML Agility Pack Screen Scraping XPATH isn't returning data

    - by Matthias Welsh
    I'm attempting to write a screen scraper for Digikey that will allow our company to keep accurate track of pricing, part availability and product replacements when a part is discontinued. There seems to be a discrepancy between the XPATH that I'm seeing in Chrome Devtools as well as Firebug on Firefox and what my C# program is seeing. The code I'm currently using is pretty quick and dirty... //This function retrieves data from the digikey private static List<string> ExtractProductInfo(HtmlDocument doc) { List<HtmlNode> m_unparsedProductInfoNodes = new List<HtmlNode>(); List<string> m_unparsedProductInfo = new List<string>(); //Base Node for part info string m_baseNode = @"//html[1]/body[1]/div[2]"; //Write part info to list m_unparsedProductInfoNodes.Add(doc.DocumentNode.SelectSingleNode(m_baseNode + @"/table[1]/tr[1]/td[1]/table[1]/tr[1]/td[1]")); //More lines of similar form will go here for more info //this retrieves digikey PN foreach(HtmlNode node in m_unparsedProductInfoNodes) { m_unparsedProductInfo.Add(node.InnerText); } return m_unparsedProductInfo; } Although the path I'm using appears to be "correct" I keep getting NULL when I look at the list "m_unparsedProductInfoNodes" Any idea what's going on here? I'll also add that if I do a "SelectNodes" on the baseNode it only returns a div... not sure what that indicates but it doesn't seem right.

    Read the article

  • Is there a jQuery webscraper out there?

    - by victor hugo
    I'm trying to pullout some info from an external site using jQuery and Adobe AIR. Right now I'm using a hidden div and jQuery's load function to load fragments of the external site, once the info is loaded I parse some info with selectors. This is fine but it's kinda dirty and I need to perform this several times (don't want to need many hidden divs). Just wondering if anybody knows a good webscrapper written in jQuery or maybe another method I'm missing

    Read the article

  • What are the best measures to protect content from being crawled?

    - by Moak
    I've been crawling a lot of websites for content recently and am surprised how no site so far was able to put up much resistance. Ideally the site I'm working on should not be able to be harvested so easily. So I was wondering what are the best methods to stop bots from harvesting your web content. Obvious solutions: Robots.txt (yea right) IP blacklists What can be done to catch bot activity? What can be done to make data extraction difficult? What can be done to give them crap data? Just looking for ideas, no right/wrong answer

    Read the article

  • Source for Names to use in web scraping

    - by PyNEwbie
    Can anyone suggest a good source of names that I can use to help analyze some tables on web pages. The first column of the tables I am scraping have names alone, names and titles or just titles. The names can be as varied as John Smith to Vikram Saksena. I have been poking around for a compiled list of words that can be found in proper names.

    Read the article

  • Web scraping with Python

    - by Jack
    I'm currently trying to scrape a website that has fairly poorly-formatted HTML (often missing closing tags, no use of classes or ids so it's incredibly difficult to go straight to the element you want, etc.). I've been using BeautifulSoup with some success so far but every once and a while (though quite rarely), I run into a page where BeautifulSoup creates the HTML tree a bit differently from (for example) Firefox or Webkit. While this is understandable as the formatting of the HTML leaves this ambiguous, if I were able to get the same parse tree as Firefox or Webkit produces I would be able to parse things much more easily. The problems are usually something like the site opens a <b> tag twice and when BeautifulSoup sees the second <b> tag, it immediately closes the first while Firefox and Webkit nest the <b> tags. Is there a web scraping library for Python (or even any other language (I'm getting desperate)) that can reproduce the parse tree generated by Firefox or WebKit (or at least get closer than BeautifulSoup in cases of ambiguity).

    Read the article

  • Web scraping with Python

    - by Jack
    I'm currently trying to scrape a website that has fairly poorly-formatted HTML (often missing closing tags, no use of classes or ids so it's incredibly difficult to go straight to the element you want, etc.). I've been using BeautifulSoup with some success so far but every once and a while (though quite rarely), I run into a page where BeautifulSoup creates the HTML tree a bit differently from (for example) Firefox or Webkit. While this is understandable as the formatting of the HTML leaves this ambiguous, if I were able to get the same parse tree as Firefox or Webkit produces I would be able to parse things much more easily. The problems are usually something like the site opens a <b> tag twice and when BeautifulSoup sees the second <b> tag, it immediately closes the first while Firefox and Webkit nest the <b> tags. Is there a web scraping library for Python (or even any other language (I'm getting desperate)) that can reproduce the parse tree generated by Firefox or WebKit (or at least get closer than BeautifulSoup in cases of ambiguity).

    Read the article

  • Is there a jQuery webscrapper out there?

    - by victor hugo
    I'm trying to pullout some info from an external site using jQuery and Adobe AIR. Right now I'm using a hidden div and jQuery's load function to load fragments of the external site, once the info is loaded I parse some info with selectors. This is fine but it's kinda dirty and I need to perform this several times (don't want to need many hidden divs). Just wondering if anybody knows a good webscrapper written in jQuery or maybe another method I'm missing

    Read the article

  • Scraping *.aspx content using Python

    - by tomato
    I'm having difficulties scraping dynamically generated table in ASPX. Trying to scrape the gas prices from a site like this GasPrices. I can extract all the information in the gas price table (address, time submitted etc.), except for the actual gas price. Is there a way I could scrape the gas prices? i.e. somehow get a text representation of it. I'm not very familiar with ASP/ASPX - but what's being generated now is not showing up in the final HTML. I'm using Python to do the scraping, but that's irrelevant unless there's a specific library... Thanks in advance.

    Read the article

  • Scraping *.aspx content using Python

    - by tomato
    I'm having difficulties scrapping dynamically generated table in ASPX. Trying to scrap the gas prices from a site like these GasPrices. I can extract all the information in the gas price table (address, time submitted etc.), except for the actual gas price. Is there a way I could scrap the gas prices? i.e. somehow get a text representation of it. I'm not very familiar with ASP/ASPX - but what's being generated now is not showing up in the final HTML. I'm using Python to do the scrapping, but that's irrelevant unless there's a specific library...

    Read the article

  • How to isolate a single element from a scraped web page in R

    - by PaulHurleyuk
    Hello, I'm trying to do soemone a favour, and it's a tad outside my comfort zone, so I'm stuck. I want to use R to scrape this page (http://www.fifa.com/worldcup/archive/germany2006/results/matches/match=97410001/report.html ) and others, to get the goal scorers and times. So far, this is what I've got require(RCurl) require(XML) theURL <-"http://www.fifa.com/worldcup/archive/germany2006/results/matches/match=97410001/report.html" webpage <- getURL(theURL, header=FALSE, verbose=TRUE) webpagecont <- readLines(tc <- textConnection(webpage)); close(tc) pagetree <- htmlTreeParse(webpagecont, error=function(...){}, useInternalNodes = TRUE) and the pagetree object now contains a pointer to my parsed html (I think). The part I want is <div class="cont")<ul> <div class="bold medium">Goals scored</div> <li>Philipp LAHM (GER) 6', </li> <li>Paulo WANCHOPE (CRC) 12', </li> <li>Miroslav KLOSE (GER) 17', </li> <li>Miroslav KLOSE (GER) 61', </li> <li>Paulo WANCHOPE (CRC) 73', </li> <li>Torsten FRINGS (GER) 87'</li> </ul></div> but I'm now lost as to how to isolate them, and frankly xpathSApply, xpathApply confuse the beejeebies out of me !. So, does anyone know how to fomulate a command to suck out the element conmtaiend within the tags ? Thanks Paul.

    Read the article

  • Facebook fan page photo's scraping

    - by Daan Poron
    Hi, We want to add a facebook fan page photo competition to our fan page. The meaning is that ppl can upload photo's and others can like them. The person with the most likes on his photo wins a price. Now i was wondering if anyone knows a good idea on how to get a snapshot of all the photo's on a given moment. So that when we want to stop the contest we get an overview of the number of likes of all the persons. Some good website scraping tools? maybe a usefull facebook app? some other alternatives? greets, Daan

    Read the article

  • Problem pulling data from website in .NET and C#

    - by Cptcecil
    I have written a web scraping program to go to a list of pages and write all the html to a file. The problem is that when I pull a block of text some of the characters get written as '?'. How do I pull those characters into my text file? Here is my code: string baseUri = String.Format("http://www.rogersmushrooms.com/gallery/loadimage.asp?did={0}&blockName={1}", id.ToString(), name.Trim()); // our third request is for the actual webpage after the login. HttpWebRequest request = (HttpWebRequest)WebRequest.Create(baseUri); request.Method = "GET"; request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1)"; //get the response object, so that we may get the session cookie. HttpWebResponse response = (HttpWebResponse)request.GetResponse(); StreamReader reader = new StreamReader(response.GetResponseStream()); // and read the response string page = reader.ReadToEnd(); StreamWriter SW; string filename = string.Format("{0}.txt", id.ToString()); SW = File.AppendText("C:\\Share\\" + filename); SW.Write(page); reader.Close(); response.Close();

    Read the article

  • Yahoo Web Scrapes: What are the limits?

    - by bvandrunen
    We are using a web scraper and have it set up to have a sleep function which has a random function set up (so that it isn't the same time between each scrape) but we are still getting blocked from Yahoo after 20-30 requests. Does any one know if there is a limit (i.e: 20 requests per minutes, 200 an hour) Right now our average between each request is around 3-6 seconds. Thanks for any help

    Read the article

  • Web scraping advice/help with java for android app!

    - by Capsud
    Hey there, I've heard about web scraping software that can take data from a webpage. i'm building an android app and I want to take information from this site www.menupages.ie All I need is the names of the restaurants, and typing them in myself would be very tedious. Can someone tell me how i'd go about doing this in eclipse, what methods i need etc. I dont know anything about it. Thanks alot.

    Read the article

  • How to use a loop to download HTML with paging?

    - by Nai
    I want to loop through this URL and download the HTML. https://www.googleapis.com/customsearch/v1?key=AIzaSyAAoPQprb6aAV-AfuVjoCdErKTiJHn-4uI&cx=017576662512468239146:omuauf_lfve&q=" + searchTermFormat + "&num=10" +"&start=" + i start and num controls the paging of the URL. So if &start=2, and &num=10, it will scrape 10 results from page 2. Given that Google has a max limit of num = 10, how can I write a loop that loops through the HTML and scrape the results for the first 10 pages? This is what I have so far which just scrapes the first page. //input search term Console.WriteLine("What is your search query?:"); string searchTerm = Console.ReadLine(); //concantenate the strings using + symbol to make it URL friendly for google string searchTermFormat = searchTerm.Replace(" ", "+"); //create a new instance of Webclient and use DownloadString method from the Webclient class to extract download html WebClient client = new WebClient(); int i = 1; string Json = client.DownloadString("https://www.googleapis.com/customsearch/v1?key=AIzaSyAAoPQprb6aAV-AfuVjoCdErKTiJHn-4uI&cx=017576662512468239146:omuauf_lfve&q=" + searchTermFormat + "&num=10" + "&start=" + i); //create a new instance of JavaScriptSerializer and deserialise the desired content JavaScriptSerializer js = new JavaScriptSerializer(); GoogleSearchResults results = js.Deserialize<GoogleSearchResults>(Json); //output results to console Console.WriteLine(js.Serialize(results)); Console.ReadLine();

    Read the article

  • What's the fastest way to scrape a lot of pages in php?

    - by Yegor
    I have a data aggregator that relies on scraping several sites, and indexing their information in a way that is searchable to the user. I need to be able to scrape a vast number of pages, daily, and I have ran into problems using simple curl requests, that are fairly slow when executed in rapid sequence for a long time (the scraper runs 24/7 basically). Running a multi curl request in a simple while loop is fairly slow. I speeded it up by doing individual curl requests in a background process, which works faster, but sooner or later the slower requests start piling up, which ends up crashing the server. Are there more efficient ways of scraping data? perhaps command line curl?

    Read the article

  • Extract Data from a Website using PHP

    - by 01jayss
    I am trying to get PHP to extract the TOKEN (the uppercase one), USERID (uppercase), and the USER NAME (uppercase) from a web page with the following text. {"rsp":{"stat":"ok","auth":{"token":"**TOKEN**","perms":"read","user":{"id":"**USERID**","username":"**USER NAME**","fullname":"**NAME OF USER**"}}}} (This is from the RTM api, getting the authentication token of the user). How would I go about doing this? Thanks!

    Read the article

  • How to know if the website being scraped has changed?

    - by Lost_in_code
    I'm using PHP to scrape a website and collect some data. It's all done without using regex. I'm using php's explode() method to find particular HTML tags instead. It is possible that if the structure of the website changes (CSS, HTML), then wrong data may be collected by the scraper. So the question is - how do I know if the HTML structure has changed? How to identify this before storing any data to my database to avoid wrong data being stored.

    Read the article

1 2  | Next Page >