开发者

Is noindex valid in robots.txt? [duplicate]

开发者 https://www.devze.com 2023-03-30 16:07 出处:网络
This question already has answers here: Noindex in a robots.txt 开发者_如何学JAVA (2 answers) Closed 1 year ago.
This question already has answers here: Noindex in a robots.txt 开发者_如何学JAVA (2 answers) Closed 1 year ago.

Is noindex an optional directive in a robots.txt file, or are user-agent, disallow, allow and crawl-delay the only options?

For example, is this valid for the contents of a robots.txt file?

user-agent: *  
disallow: /  
noindex: /


noindex is not a valid directive for a robots.txt file. It is a valid directive for a META robots tag, though.

The only standard directives for robots.txt are "User-agent" and "Disallow". Some browsers support an extended set of directives including "Crawl-delay", "Allow" and "Sitemap". http://rield.com/cheat-sheets/robots-exclusion-standard-protocol seems to have a thorough explanation of the standard and extended directives.

0

精彩评论

暂无评论...
验证码 换一张
取 消