How to identify the client is a search robot?

Posted by Yau Leung on Pro Webmasters See other posts from Pro Webmasters or by Yau Leung
Published on 2010-12-01T09:54:54Z Indexed on 2012/10/01 21:52 UTC
Read the original article Hit count: 210

Filed under:

apache

|

AJAX

I have built my entire site using AJAX (indeed it's GWT). I have also implemented AJAX crawling proposed by Google. However, after the implementation, I found that neither Yahoo , Bing, nor Baidu implemented that scheme!

I'm wondering if there is a way to identify the web client is a search robot. If they are, they will be shown the HTML snapshot I created.

It will be best if I can identify them in APACHE level, then I can just do a mod_rewrite. But it's still ok if I can do that in PHP or GWT.

© Pro Webmasters or respective owner

Related posts about apache

Web site not responding

as seen on Super User - Search for 'Super User'
I have website working fine before. But now its not able to connect to the server(I believe that is the problem). But its strange that the message not able to connect to the server is not coming and its keep connecting... for infinite time. Here is the screenshot. Here are some of the useful… >>> More
SVN Error 403 Forbidden

as seen on Stack Overflow - Search for 'Stack Overflow'
I can't figure this out. I try to import a new project into a svn repository from Netbeans and get 403 Forbidden. I just setup svn on my serverbox today. I can get to it through a browser just fine, though its empty as I haven't imported my project yet. Apache's path for html files is /var/www I… >>> More
How can I setup dependencies for Axis2 / Axiom on Maven2

as seen on Stack Overflow - Search for 'Stack Overflow'
I've tried the following settings on pom.xml to use Axis2 wsdl2code: <dependencies> <dependency> <groupId>org.apache.axis2</groupId> <artifactId>axis2</artifactId> <version>1.5.1</version> </dependency> </dependencies> ... <build> … >>> More
Problem compiling hive with ant

as seen on Stack Overflow - Search for 'Stack Overflow'
I compiling with Solaris 10 SPARC, jdk 1.6 from Sun, Ant 1.7.1 from OpenCSW. I have no problem running hadoop 0.17.2.1 However, I have problem compiling/integrating hive with the error 'cannot find symbol', although I followed the tutorial. I have the hive source code from SVN exactly from tutorial… >>> More
War deployment error related to classloading

as seen on Stack Overflow - Search for 'Stack Overflow'
hello when i am deploying my war file and run it it will give error like org.springframework.instrument.classloading.tomcat.TomcatInstrumentableClassLoader Jan 6, 2011 3:16:04 PM org.apache.catalina.startup.HostConfig deployDescriptor INFO: Deploying configuration descriptor servlet.xml Jan… >>> More

Related posts about AJAX

Ajax call from a form rendered as Ajax response (jQuery + Grails: chaining ajax requests)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I was expecting the below scenario common, but couldn't find much help online. I have a form loaded through Ajax (say, create entity form). It is loaded through a button click (load) event $("#bt-create").click(function(){ $ ('#pid').load('/controller/vehicleModel/create3'); … >>> More
Ajax Control Toolkit Now Supports jQuery

as seen on Stephen Walter - Search for 'Stephen Walter'
I’m excited to announce the September 2013 release of the Ajax Control Toolkit, which now supports building new Ajax Control Toolkit controls with jQuery. You can download the latest release of the Ajax Control Toolkit from http://AjaxControlToolkit.CodePlex.com or you can install the Ajax Control… >>> More
The Microsoft Ajax Library and Visual Studio Beta 2

as seen on Stephen Walter - Search for 'Stephen Walter'
Visual Studio 2010 Beta 2 was released this week and one of the first things that I hope you notice is that it no longer contains the latest version of ASP.NET AJAX. What happened? Where did AJAX go? Just like Sting and The Police, just like Phil Collins and Genesis, just like Greg Page and the Wiggles… >>> More
Video Aulas gratuitas de ASP.NET AJAX

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Olá pessoal, A Microsoft publicou 14 vídeos que cedi a eles a respeito do ASP.NET AJAX. Todos os vídeos são gratuitos, e caso você ainda use o AJAX estas lições podem ajudá-lo bastante. Tenha um excelente estudo. Instalação do AJAX Control Tookit ASP.NET AJAX: Controle calendário ASP.NET… >>> More
July 2013 Release of the Ajax Control Toolkit

as seen on Stephen Walter - Search for 'Stephen Walter'
I’m super excited to announce the July 2013 release of the Ajax Control Toolkit. You can download the new version of the Ajax Control Toolkit from CodePlex (http://ajaxControlToolkit.CodePlex.com) or install the Ajax Control Toolkit from NuGet: With this release, we have completely rewritten the… >>> More