开发者

Why is Yahoo Indexing Bot considered as "evil"?

开发者 https://www.devze.com 2022-12-25 03:45 出处:网络
After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow\'s sitemap.This returned a 404 error which I am guess

After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow's sitemap. This returned a 404 error which I am guessing is just a protected page by determining if your are a Index Bot or simply doesnt exists.

This then开发者_JAVA百科 lead me to look at the robots.txt for StackOverFlow. I was surprised to see the comment "Yahoo bot is evil" along with a couple other Indexing bots (Spinn3r and KSCrawler) . I am unfamilular with Spinn3r and KSCrawler but my question is, why are these bots (particular Yahoo) considered as evil? Surely any and all indexing of any Search Engine is a good thing?


These are very aggressive bots that do not throttle requests and can bring a site down.

See this forum thread for an example for the yahoo bot.

0

精彩评论

暂无评论...
验证码 换一张
取 消