rcurl web scraping timeout exits program

Posted by user1742368 on Stack Overflow See other posts from Stack Overflow or by user1742368
Published on 2012-10-12T21:33:48Z Indexed on 2012/10/12 21:36 UTC
Read the original article Hit count: 188

Filed under:
|
|
|

I am using a loop and rcurl scrape data from multiple pages which seems to work fine at certain times but fails when there is a timeout due to the server not responding. I am using a timeout=30 which traps the timeout error however the program stops after the timeout. i would like the progrm to continue to the next page when the timeout occurrs but cant figureout how to do this? url = getCurlHandle(cookiefile = "", verbose = TRUE)

Here is the statement I am using that causes the timeout. I am happy to share the code if there is interest. webpage = getURLContent(url, followlocation=TRUE, curl = curl,.opts=list( verbose = TRUE, timeout=90, maxredirs = 2))

woodwardjj

© Stack Overflow or respective owner

Related posts about timeout

Related posts about scrape