robots.txt

相关标签：javascript jquery android 多少钱 iPhone

robots.txt ignrore all folders but crawl all files in root

should i then do 开发者_运维知识库User-agent: * Disallow: / is it as simple as that? or will that not crawl the files in the root either?[详细]

2022-12-19 10:17 分类：问答
Implementing "Report this content" and detecting spammer or robot triggered event

I\'m creating a forum for a website, and plan on implementing a \"Report this content\" function. In all honesty, I\'m not sure how useful (lit. necessary) the feature will be, since a user account ([详细]

2022-12-19 06:35 分类：问答
Can I do a 301 redirect in robots.txt?

I have a site which has a whole host of legacy links, which now need to be ma开发者_C百科pped to new links.[详细]

2022-12-17 18:32 分类：问答
can i use robots.txt while handling my site with htaccess

I am using htaccess in my site, such that all the request to my site will be redirected to index page in开发者_运维百科 my root directory. No other file in my site can be accessed because my htaccess[详细]

2022-12-13 17:17 分类：问答
robots.txt to restrict search engines indexing specified keywords for privacy

I have a large directory of individual names along with generic publicaly available and category specific information that I want indexed as much as possible in search engines. Listing these names on[详细]

2022-12-12 11:04 分类：问答
blocked links in sitemap

i\'m using a online sitemap generator tool which generates links even for which are blocked in robots.txt. Is these block开发者_StackOverflow社区ed links affect site ranking ? . Is there anyway to ove[详细]

2022-12-09 15:06 分类：问答
How to allow crawlers access to index.php only, using robots.txt?

If i want to only allow crawlers to access index.php, will this 开发者_JAVA技巧work? User-agent: *[详细]

2022-12-09 12:30 分类：问答
How do I modify robots.txt in Plone?

I\'ve got a Plone site that I administer and I\'d like to add some pages to the Disallow of a robots.txt.[详细]

2022-12-08 06:19 分类：问答
robots.txt and wildcard at the end od disallow [closed]

Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow.[详细]

2022-12-08 03:34 分类：问答