What's the requests/second standard for scraping websites?

Posted by feydr on Stack Overflow See other posts from Stack Overflow or by feydr
Published on 2010-05-29T22:24:43Z Indexed on 2010/05/29 22:32 UTC
Read the original article Hit count: 223

Filed under:

screen-scraping

|

etiquette

This was the closest question to my question and it wasn't really answered very well imo:

http://stackoverflow.com/questions/2022030/web-scraping-etiquette

I'm looking for the answer to #1:

How many requests/second should you be doing to scrape?

Right now I pull from a queue of links. Every site that gets scraped has it's own thread and sleeps for 1 second in between requests. I ask for gzip compression to save bandwidth.

Are there standards for this? Surely all the big search engines have some set of guidelines they follow in regards to this.

© Stack Overflow or respective owner

Related posts about screen-scraping

PHP Screen Scraping Class

as seen on Bradino - Search for 'Bradino'
After some positive feedback I have decided to continue to develop the PHP Screen Scraping class. This post will server as the permanent home for the class. Download PHP Screen Scraping Class Updates 20009-07-30 Added setHeader() function >>> More
Screen scraping over SSL with .NET

as seen on Stack Overflow - Search for 'Stack Overflow'
What solutions exist for screen scraping a site over SSL for use with .NET? My use case is that I need to login to a partner website (https), navigate through a dynamic hierarchy, and download a zipped file of reports. I certainly could use other screen scrapers if there are no good viable options… >>> More
looking for alternative to Webzinc .NET , screen scraping, web automation library for .net

as seen on Stack Overflow - Search for 'Stack Overflow'
i came across this .net library http://www.webzinc.com/online/faq.aspx however, i was wondering if there was a free alternative out there ? >>> More
Screen-scraping of a secure page of any site on https:// with asp.net in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page. Please advice what should i do for scraping a secure page of any site on https://. >>> More
How different is mashup from screenscraping and consuming webservices

as seen on Stack Overflow - Search for 'Stack Overflow'
From what I understand, Mashup is aggregating data from separate sources and providing a single view. How different is mashup when compared to screenscraping or using webservices to get data from external sources? >>> More

Related posts about etiquette

Email etiquette checker?

as seen on Super User - Search for 'Super User'
Sometimes, in the heat of getting a lot of things done, I don't read emails I send as closely as I should. Is there an automated way to pop up warnings before a potentially-unpolite email is sent? I'm thinking about something that would say "wait, why don't you have a 'please' or 'would you' in… >>> More
Etiquette for refactoring other people's sourcecode?

as seen on Stack Overflow - Search for 'Stack Overflow'
Our team of software developers consists of a bunch of experienced programmers with a variety of programming styles and preferences. We do not have standards for everything, just the bare necessities to prevent total chaos. Recently, I bumped into some refactoring done by a colleague. My code looked… >>> More
Does professionalism in job postings matter?

as seen on Programmers - Search for 'Programmers'
I came across this job posting: http://www.justin.tv/jobs/jobs. The page contains some bad language ("No bulls--t"). I personally find this vaguely offensive, and it would certainly put me off applying to work there: It made me wonder what kind/personality of people work at this company. At least… >>> More
What rules of etiquette should be followed at software conferences?

as seen on Programmers - Search for 'Programmers'
Whether as an attendee, a speaker, or a vendor I wanted to know what the unspoken rules of etiquette are at software conferences. Other than the blindingly obvious ones (like don't assault the winner of the iPad raffle because you didn't win). What are some of the rules that should be followed… >>> More
Is the term "web portal" obselete?

as seen on Programmers - Search for 'Programmers'
Firstly, sorry if this is the wrong place: I've looked at all the programming-related boards and this one seems like the best fit - correct me if I'm wrong. My boss uses the term "portal" for the project I work on all the time. To me, the word makes me think of Yahoo in the late 90s. Does the word… >>> More