HttpWebRequest returns empty text

When I try to fetch th开发者_运维知识库e content of this URL http://www.yellowpages.com.au/qld/gatton/a-12000029-listing.html

using System.Net;

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader objSR;
objSR = new StreamReader(resStream, System.Text.Encoding.GetEncoding("utf-8"));
string sResponse = objSR.ReadToEnd();

I don't get any response from the server. Please help me find out why this happens.

Thanks in advance!

---

It may well be looking at the user agent and refusing to serve content to a client that doesn't identify itself. Try setting the UserAgent property on your request object.

---

Looks to me like that site is checking the referrer url and may be serving up empty content if an invalid referrer is specified.

Try setting request.Referer = "http://www.google.com";. Experiment with the referrer to see if that changes the response. I'd also try the UserAgent property as Matthew suggested.

---

I had the same problem and the cause was that I previously had set the method to HEAD and in later revisions had the need to parse the body.