Detecting well behaved / well known bots

Posted by Simon_Weaver on Stack Overflow See other posts from Stack Overflow or by Simon_Weaver
Published on 2009-06-06T23:45:21Z Indexed on 2010/05/25 8:41 UTC
Read the original article Hit count: 173

Filed under:

bots

|

googlebot

I found this question very interesting : Programmatic Bot Detection I have a very similar question, but I'm not bothered about 'badly behaved bots'.

I am tracking (in addition to google analytics) the following per visit :

Entry URL
Referer
UserAgent
Adwords (by means of query string)
Whether or not the user made a purchase
etc.

The problem is that to calculate any kind of conversion rate I'm ending up with lots of 'bot' visits that are greatly skewing my results.

I'd like to ignore as many as possible bot visits, but I want a solution that I don't need to monitor too closely, and that won't in itself be a performance hog and preferably still work if someone has javascript disabled.

Are there good published lists of the top 100 bots or so? I did find a list at http://www.user-agents.org/ but that appears to contain hundreds if not thousands of bots. I don't want to check every referer against thousands of links.

Here is the current googlebot UserAgent. How often does it change?

 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

© Stack Overflow or respective owner

Related posts about bots

How to remove bots from my computer?

as seen on Super User - Search for 'Super User'
Because trend micro's rubotted detected that there is a bot running on my system and recommends to run Trend micro's house call to remove the bot. But when I ran house call, it does not detect anything. By the way,my current antivirus software is Microsoft Security Essentials. >>> More
Java, how to make bots? Beginner

as seen on Stack Overflow - Search for 'Stack Overflow'
I am new in the world of programming. I am student of J2ME(Java) beginner. And I am interested to make bot, how to make bots easily. Any guide? Link or keywords which I can google to learn things? >>> More
Twitter traffic might not be what it seems

as seen on Hadermann.be - Search for 'Hadermann.be'
Are you using bit.ly stats to measure interest in the links you post on twitter? I’ve been hearing for a while about people claiming to get the majority of their traffic originating from twitter these days. Now, I’ve been playing with the twitter ruby gem recently, doing various experiments… >>> More
Searchengine bots and meta refresh for disabled Javascript

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi! I have a website that must have javascript turned on so it can work there is a < noscript tag that have a meta to redirect the user to a page that alerts him about the disabled javascript... I am wondering, is this a bad thing for search engine crawlers? Because I send an e-mail to myself… >>> More
Video Game Bots?

as seen on Stack Overflow - Search for 'Stack Overflow'
Something I've always wondered, especially since it inspired me to start programming when I was a kid, was how video game bots work? I'm sure there are a lot of different methods, but what about automation for MMORPGs? Or even FPS-type bots? >>> More

Related posts about googlebot

Screen resolution of Googlebot mobile?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly when serving different image resolution to different viewports via JavaScript. While Googlebot has… >>> More
ASP.NET MVC GoogleBot Issues

as seen on Stack Overflow - Search for 'Stack Overflow'
I wrote a site using ASP.NET MVC, and although it is not completely SEO optimized at this point I figured it is a good start. What I'm finding is that when I use Google's Webmaster Tools to fetch my site (to see what a GoogleBot sees) it sees this. HTTP/1.1 200 OK Cache-Control: public, max-age=1148 Content-Type:… >>> More
Where Googlebot starts crawling?

as seen on Stack Overflow - Search for 'Stack Overflow'
Say if I register a domain and have developed it into a complete website. From where and how Googlebot knows that the new domain is up? Does it always start with the domain registry? If it starts with the registry, does that mean that anyone can have complete access to the registry's database? Thanks… >>> More
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools: Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index… >>> More
Google-Bot fell in love with my 404-page

as seen on Server Fault - Search for 'Server Fault'
Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.78.140 - - [21/Oct/2013:14:37:01 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5… >>> More