开发者

How to detect crawlers in Asp.net MVC

开发者 https://www.devze.com 2023-04-08 16:33 出处:网络
I\'m using an action filter that checks what browser version is being used on my site, if it\'s an older browser I put up a div at the top asking them to upgrade. I don\'t want web crawlers to get the

I'm using an action filter that checks what browser version is being used on my site, if it's an older browser I put up a div at the top asking them to upgrade. I don't want web crawlers to get the div message, so I've implemented HttpBrowserCapabilitiesBase.Crawler and it looks like it works for Google, but Bing and the others don't seem to register as crawlers. Strange for a Microsoft product to not notice Bing as a crawler!

Is there some way to add user agents to the crawler property or something?

Thanks!

Edited: I'm using asp.net mvc 3, it looks like I need to user .Browser files(?). Anyone know of a comprehensive set of .Browser files out there for Bing and the r开发者_如何学JAVAest of the crawlers?


You will probably need to update your browscap.ini file as the one shipped with IIS is probably old. You can get a new one at one of the following URLs:

  • http://browsers.garykeith.com/downloads.asp
  • http://owenbrady.net/browsercaps/

browscap.ini usually lives at: c:\windows\system32\inetsrv\browscap.ini


We're not using MVC but we do this:

Regex.IsMatch(Request.UserAgent, @"bot|crawler|baiduspider|80legs|ia_archiver|voyager|curl|wget|yahoo! slurp|mediapartners-google", RegexOptions.IgnoreCase);

More options in my answer here:

Detecting honest web crawlers

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号