The best and most efficient methods of detecting web crawlers

There are many ways to pre开发者_如何转开发tend to be a human being. So what are the best methods to see past it?

---

The most efficient method is Captha but can make your website less user friendly. Thus, the best approach is analyze your traffic programmaticaly and no sooner your code detects something unusual, just ask the user to answer the captha in a strict time line.

Just understand the basic funda. No one writes a crawler to read just one page or article. Some want it all, some want it fast while some want it regular. Some may even hit from different IP's but then they hit in the same style with same time gap. The biggest difference between a crawler and a human is crawler are discipline about when to hit, humans are lazy. Crawlers are efficient enough to many hits together, humans are not and so on .. . Anything which sounds as if a pattern in your traffic log is nothing else but a crawler.