Is this the correct way to do this - below is my txt file, would this prevent Google from indexing my admin d开发者_JS百科irectory as well as oldpage.php?
User-agent: *
Allow: /
Disallow: /admin/
Disallow: http://www.mysite.com/oldpage.php
Yes you are absolutely correct except single file restriction.
User-agent: *
: means for all crawler
Allow: /
: allow access of full site
Disallow: /admin/
: restrict to admin directory
Disallow: /oldpage.php
: restrict to oldpage.php
精彩评论