开发者

How do you create a robots.txt file that blocks all but the root

开发者 https://www.devze.com 2023-04-04 05:20 出处:网络
How do you create a valid robots.txt file that blocks all crawler 开发者_StackOverflow社区requests except for the root, aka landing http://www.mysite.comAssuming your default page for the root is name

How do you create a valid robots.txt file that blocks all crawler 开发者_StackOverflow社区requests except for the root, aka landing http://www.mysite.com


Assuming your default page for the root is named index.htm, I believe this will accomplish what you're looking for.

User-agent: *
Allow: /index.htm
Disallow: /

Google's Webmaster Tools has some great help for formulating a robots.txt and if you use the Webmaster Tools, you also get a robots.txt builder/tester.

0

精彩评论

暂无评论...
验证码 换一张
取 消