Is this Anti-Scraping technique viable with Crawl-Delay?

Posted by skibulk on Pro Webmasters See other posts from Pro Webmasters or by skibulk
Published on 2012-10-17T23:06:19Z Indexed on 2012/10/17 23:21 UTC
Read the original article Hit count: 223

Filed under:

googlebot

|

crawling

|

web-crawlers

|

scraper-sites

I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for users that access an abnormal number of pages per minute.

I don't want search engine spiders to ever receive the error. My inclination is to set a robots.txt crawl-delay which will ensure spiders access a number of pages per minute under my 503 threshold.

Is this an appropriate solution? Do all major search engines support the directive? Could it negatively affect SEO? Are there any other solutions or recommendations?

© Pro Webmasters or respective owner

Related posts about googlebot

Screen resolution of Googlebot mobile?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly when serving different image resolution to different viewports via JavaScript. While Googlebot has… >>> More
ASP.NET MVC GoogleBot Issues

as seen on Stack Overflow - Search for 'Stack Overflow'
I wrote a site using ASP.NET MVC, and although it is not completely SEO optimized at this point I figured it is a good start. What I'm finding is that when I use Google's Webmaster Tools to fetch my site (to see what a GoogleBot sees) it sees this. HTTP/1.1 200 OK Cache-Control: public, max-age=1148 Content-Type:… >>> More
Where Googlebot starts crawling?

as seen on Stack Overflow - Search for 'Stack Overflow'
Say if I register a domain and have developed it into a complete website. From where and how Googlebot knows that the new domain is up? Does it always start with the domain registry? If it starts with the registry, does that mean that anyone can have complete access to the registry's database? Thanks… >>> More
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools: Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index… >>> More
Google-Bot fell in love with my 404-page

as seen on Server Fault - Search for 'Server Fault'
Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.78.140 - - [21/Oct/2013:14:37:01 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5… >>> More

Related posts about crawling

https & ajax crawling

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We made on our webpage https://www.1point618.com a transition to ssl and now we using nearly entirely ajax to load the content. Therefore all urls of existing pages have changed. We used the 301 redirect as recommended, also we have implemented google's specification that the webpage is still crawl-able… >>> More
Understanding Ajax crawling of search site

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a couple of questions about Ajax crawling of site, which is kind of search engine itself. The base article explains the mechanism of making AJAX application crawlable. All this stuff with HTML-snapshots is clear and easy to implement, but I cant understand where will Google bot will get "the… >>> More
Ajax site not being crawled - have escaped fragment, what's wrong? [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
My site is anonkun.com. You can see that it's "ajax" and doesn't load much HTML. Here are some example pages: http://anonkun.com http://anonkun.com/?_escaped_fragment_= http://anonkun.com/stories/Dev-kun---FAQ/6ef881f8-cf48-4f87-a688-c585f23809c5 http://anonkun.com/stories/Dev-kun---FAQ/6ef881f8-cf48-4f87-a688-c585f23809c5… >>> More
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm not mad about "not ranking high." I think something is (accidentally) rejecting search engine indexing… >>> More
How to interpret number of URL errors in Google webmaster tools

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out is how to interpret the number of errors over time. At the end of February we've recently migrated… >>> More