I have a public page that is not supposed be possible for users to sign into. So I have a url that there is no link to and you have to enter manually and then sign in.
The url is multilanguage however, so it can be "/SV/Account/Logon" or开发者_Python百科 "/EN/Account/Logon" etc etc. Can I disable this url to be indexed for all languages?
This url might help you http://www.robotstxt.org/robotstxt.html. There will be no wildcard exclusion in your case you will have to add all the language specific login urls in robot.txt
Update
you can put Disallow: /folder/subfolder/
or Disallow: /folder/
in your robots.txt file
精彩评论