How do I block a user-agent from Apache

Posted by rubo77 on Pro Webmasters See other posts from Pro Webmasters or by rubo77
Published on 2013-06-26T05:46:14Z Indexed on 2013/06/26 10:28 UTC
Read the original article Hit count: 460

Filed under:

googlebot

|

apache2

|

apache-log-files

|

spider

How do I realize a UA string block by regular expression in the config files of my Apache webserver?

For example: if I would like to block out all bots from Apache on my debian server, that have the regular expression /\b\w+[Bb]ot\b/ or /Spider/ in their user-agent.

Those bots should not be able to see any page on my server and they should not appear neither in the accesslogs nor in the errorlogs.

http://global-security.blogspot.de/2009/06/how-to-block-robots-before-they-hit.html supposes to uses mod_security for that, but isn't there a simple directive for http.conf?

© Pro Webmasters or respective owner

Related posts about googlebot

Screen resolution of Googlebot mobile?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly when serving different image resolution to different viewports via JavaScript. While Googlebot has… >>> More
ASP.NET MVC GoogleBot Issues

as seen on Stack Overflow - Search for 'Stack Overflow'
I wrote a site using ASP.NET MVC, and although it is not completely SEO optimized at this point I figured it is a good start. What I'm finding is that when I use Google's Webmaster Tools to fetch my site (to see what a GoogleBot sees) it sees this. HTTP/1.1 200 OK Cache-Control: public, max-age=1148 Content-Type:… >>> More
Where Googlebot starts crawling?

as seen on Stack Overflow - Search for 'Stack Overflow'
Say if I register a domain and have developed it into a complete website. From where and how Googlebot knows that the new domain is up? Does it always start with the domain registry? If it starts with the registry, does that mean that anyone can have complete access to the registry's database? Thanks… >>> More
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools: Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index… >>> More
Google-Bot fell in love with my 404-page

as seen on Server Fault - Search for 'Server Fault'
Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.78.140 - - [21/Oct/2013:14:37:01 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5… >>> More

Related posts about apache2

apache2: Could not open configuration file /etc/apache2/apache2.conf: Permission denied

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I recently upgraded Ubuntu to the latest LTS edition on my work laptop, which I use as a LAMP development platform. The upgrade was from 12.4 to 14.4. Now I'm having trouble getting apache up and running again. Here is the output from an attempt: antonc@antonc-laptop:/etc/apache2$ sudo service apache2… >>> More
Apache segfault glibc segfault

as seen on Server Fault - Search for 'Server Fault'
I keep getting (about every 5-6 hours) this segfault in apache: [Tue Jun 26 12:43:10 2012] [notice] child pid 26810 exit signal Aborted (6) *** glibc detected *** /usr/sbin/apache2: free(): invalid pointer: 0xb68c2628 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6ff22)[0xb75aef22] /lib/i386-linux-gnu/libc… >>> More
Localhost not working after installing PHP on Mountain Lion

as seen on Server Fault - Search for 'Server Fault'
I've installed php using brew install php54 --with-mysql, I've set up all the path correctly. which php will give me /usr/local/bin/php php -v will give me PHP 5.4.8 (cli) (built: Nov 20 2012 09:29:31) php --ini will give me: Configuration File (php.ini) Path: /usr/local/etc/php/5.4 Loaded Configuration… >>> More
Apache 2 Virtual Hosts no working on OSX 10.6

as seen on Server Fault - Search for 'Server Fault'
This is my first MacBook and I'm trying to get virtual hosts up and running so as it's going to be my dev machine. I've got apache/php/mysql running fine, the problem is that what ever address I go to I just get one of the virtual hosts I've setup. I can't even get to the root site anymore. I had… >>> More
apache2 error Could not open configuration file /etc/apache2/conf.d/: No such file or directory

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have just upgraded my Ubuntu 13.10 and apache2 is not working. When I try to start the apache2 server it is printing following errors: * Starting web server apache2 * The apache2 configtest failed. Output of config test was: apache2: Syntax error on line 263 of /etc/apache2/apache2.conf: Could… >>> More