codeigniter prevent system folder and folders from being indexed

The system folder, img folder, css folder and js folder contain codes that unrelate to the content of the website. Should we d开发者_如何学Goisallow them in the robots.text ?

---

Disallowing directories using a robots.txt file means that :

• Their content will not be indexed by major search-engines
  • If those file have content that's not related to your website, and/or should not be returned in searches, you can do that.
• But their content will still be available to users

robots.txt is just an indication to crawlers -- and absolutely not a way to forbid access to anyone.

---

First, it's robots.txt, not robots.text.

Second, CSS files and Javascripts won't ever be indexed anyway, and images are usually not indexed directly, they are indexed when they are found in a page, and the system folder shouldn't even be accessible via the internet.

Finally, as Martin said, the robots.txt file is only a recommendation to search engines, they need not follow it (though typically the big ones do).

---

By default, the system folder in Codeigniter is not accessible to the users so you should't worry about that.

Just as Andrew wrote, the contents of the img, css and js folders will not be indexed by search engines so you should't worry about that either.

However, if you have any reason to block the contents of the above files for the search engines, you can do that without any worries that your site's rankings and visibility will be affected by this.

Here's a SEO recommendation that Google gives: Seo Fact no. 19.