Where can I obtain a list of User Agents for SEO bots? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.

Closed 6 years ago.

Improve this question

I am implementing a simplistic filter on how much of my site unregistered users can acces. Natu开发者_运维知识库rally, I want to give SEO bots free reign/access to most of the site.

I know this is simplistic. But its not worth doing anything more complicated. I need to compile a list of names of user agents I will allow, for this, I need a list of the names of the bots, starting with Googlebot (I don't even know if that is the official spelling of Gogles web crawling bot).

Anyone I would like a link to a definite resource that gives the names of the SEO indexing bots?. I tried http://www.user-agents.org/ but the granularity is not fine enough - it appears to list every user agent ever created!

---

Try this list: http://www.useragentstring.com/pages/useragentstring.php?typ=Crawler

Although the combination of Google, Yahoo, Bing, Baidu, Ask, and AOL represent virtually 100% of the search engine market. So I would recommend adding the crawler user agents for just those to your filter and you really don't need to worry about the rest.

---

You should also remember to avoid seeing your content through search engine's cache then:

<meta name=”robots” content=”noarchive”>