开发者

Robots.txt priority question

开发者 https://www.devze.com 2023-02-02 21:28 出处:网络
If I have these lines in robots.txt: Disallow /folder/ Allow /folder/filename.php Will the filename.php be allowed then?

If I have these lines in robots.txt:

Disallow /folder/
Allow /folder/filename.php

Will the filename.php be allowed then?

Which order does google pri开发者_运维问答oritize the lines?

What will happen here for example?:

Allow /
Disallow /

I am mainly referring to Google here as a search engine.

Thanks


For your first example, yes, the file /folder/filename.php will be allowed, as Google will prioritize the most specific rule. See here or check Wikipedia's article, for example, for more details.

Furthermore, Google has a detailed documentation on how it uses robots.txt over here.

Edit : For the second example, the Allow will be prioritized as per the Wikipedia quote Byron Whitlock posted in his answer.


It depends on the search engine:

While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that Allow patterns with equal or more characters in the directive path win over a matching Disallow pattern. Bing uses the Allow or Disallow directive which is the most specific.

In order to be compatible to all robots, if one wants to allow single files inside an otherwise disallowed directory, it is necessary to place the Allow directive(s) first, followed by the Disallow.

http://en.wikipedia.org/wiki/Robots_exclusion_standard

0

精彩评论

暂无评论...
验证码 换一张
取 消