Why is Yahoo Indexing Bot considered as "evil"?

Posted by bigstylee on Stack Overflow See other posts from Stack Overflow or by bigstylee
Published on 2010-04-01T18:41:55Z Indexed on 2010/04/01 18:53 UTC
Read the original article Hit count: 305

Filed under:

search-engine

|

indexing

|

yahoo

|

bot

After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow's sitemap. This returned a 404 error which I am guessing is just a protected page by determining if your are a Index Bot or simply doesnt exists.

This then lead me to look at the robots.txt for StackOverFlow. I was surprised to see the comment "Yahoo bot is evil" along with a couple other Indexing bots (Spinn3r and KSCrawler) . I am unfamilular with Spinn3r and KSCrawler but my question is, why are these bots (particular Yahoo) considered as evil? Surely any and all indexing of any Search Engine is a good thing?

© Stack Overflow or respective owner

Related posts about search-engine

Search Engine Optimisation - Google Replaces Yahoo As T-Mobile's Default Search Engine

as seen on Ezine Articles - Search for 'Ezine Articles'
Competition has been stoked in the mobile search engine optimisation (SEO) sector with the announcement that Google is to replace Yahoo as T-Mobile's default engine. While T-Mobile USA has replaced Yahoo with Google as its default search on mobile phones such as RIM BlackBerrys, some Yahoo-based services… >>> More
Search Engine Optimization - The Importance of Page Optimization in Search Engine Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
In order for your website to rank well, your internal linking structure is critical to your success. This is covered some of the theory for this in various articles and blogs about Page Structure of a website, which said how you should map out the physical linking structure, but in this guide I will… >>> More
How to Improve Search Engine Ranking by On-Page Search Engine Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
There are two types of search engine optimization (SEO) techniques: on-page and off-page SEO. With on-page SEO, you have full control of all the techniques and methods you can use on your own sites. With off-page SEO, you will try some techniques to get other site owners to refer to your site which… >>> More
Search Engine Optimization For Great Search Engine Placement!

as seen on Ezine Articles - Search for 'Ezine Articles'
Do you do enough search engine optimization to get the search engine placement that you want for the keywords that you want to rank for? If not then read on and I will give you information that you need to know to start getting those rankings that you want, and start receiving traffic! There are a… >>> More
Search Engine Optimization - Use of META Tag in Search Engine Marketing

as seen on Ezine Articles - Search for 'Ezine Articles'
Want your WEBPAGE to land in the 1st PAGES of Google and Yahoo search engines? Then, make use of this powerful META TAG in your webpage/s, and you will have the result you have been aching for ages. >>> More

Related posts about indexing

IIS6 Indexing Service indexing asp.net codebehind (.aspx.cs) files

as seen on Server Fault - Search for 'Server Fault'
I've setup a few catalogs on an Windows Server 2003 IIS6 install, each tracking files within a website. In the Properties - Generation Dialog for each catalog, 'Index files with unknown extensions' is turned OFF. 'Inherit above settings from Service' in that dialog is also turned off. However, the… >>> More
Outlook 2007 OST File Indexing and OneNote 2007 Indexing are Broken

as seen on Super User - Search for 'Super User'
I'm running Outlook 2007 under Windows 7 Home Premium RTM. My OST file was previously being properly indexed but eventually searches significantly slowed down so I suspected a problem. Searching and indexing appears broken in OneNote 2007 as well as search time is now significantly longer. I brought… >>> More
Indexing vs. no indexing when inserting records

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a few questions about whether or not it would be best to not use indexing. BACKGROUND: My records have a timestamp attribute, and the records will be inserted in order of their timestamps (i.e., inserted chronologically). QUESTIONS: If I DON'T use indexing is it typical for the database… >>> More
SQL SERVER – Transcript of Learning SQL Server Performance: Indexing Basics – Interview of Vinod Kumar by Pinal Dave

as seen on SQL Authority - Search for 'SQL Authority'
Recently I just wrote a blog post on about Learning SQL Server Performance: Indexing Basics and I received lots of request that if we can share some insight into the course. Here is 200 seconds interview of Vinod Kumar I took right after completing the course. We have few free codes to watch the course… >>> More
Can the Windows Indexing Service restart an app pool if it doesn't index the Web.Config?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am having a slight debate with a colleague of mine on this subject. Scenario: A web site has a sub directory that is indexed using the Windows Indexing Service. There is not a web.config in this directory and the indexing service is not indexing the parent directory which is the site root. Something… >>> More