开发者

Disable google from indexing redirect urls

开发者 https://www.devze.com 2023-04-03 04:48 出处:网络
i have a news aggregation site with tons of news... if a single news on the home page has text then the link is with a redirect to the sources site www.site.com/red/23545, but if the same news were no

i have a news aggregation site with tons of news... if a single news on the home page has text then the link is with a redirect to the sources site www.site.com/red/23545, but if the same news were not to have any text just the title then it gets this link which is not a redirect and goes to a different page on my site www.site.com/23545/some_news_title

So the same news can on some pages be with text and then get redirected on the source page or on some pages with only title and get a normal link to a page within the site.

The problem is that google is indexing the redirect links (www.site.com/red/23545).

I added

Disallow: /red/

to my robots.txt

and i also added

rel="nofollow"

to all redirect links

but non of it didn't work... its still indexed...

The reaso开发者_如何学Cn i use the redirect link is to prevent screen scraping of my site.. if someone were to go on the redirect link without a my domain as a referrer he would be redirected to my sit and not the news source page...

Any help would be appreciated.

Thanks!


You can add the following header on your redirect pages to prevent them from being indexed:

X-Robots-Tag: noindex

Still, I'm surprised that your robots.txt didn't work. How long have you had the robots.txt file there? It does tend to take some time for those sorts of things to take effect. You can speed up the process by removing the links from Google's index via the Google Webmaster Tools.

0

精彩评论

暂无评论...
验证码 换一张
取 消