开发者

Both WebClient and HttpWebRequest suddenly failing to pull basic Amazon pages?

开发者 https://www.devze.com 2023-03-16 05:59 出处:网络
We have an in-house application that pulls some data from some of Amazon\'s pages occassionally (We know they have APIs for certain operations... what we\'re doing requires some custom info not includ

We have an in-house application that pulls some data from some of Amazon's pages occassionally (We know they have APIs for certain operations... what we're doing requires some custom info not included in the APIs). We have never had a problem pulling their pages, but suddenly Amazon is returning "(503) Server Unavailable" on pretty much every request, and this has been happening for several days, so we doubt it's a temporary thing. Even something as simple as this:

System.Net.WebClient client = new System.Net.WebClient();
string data = client.DownloadString(new Uri("http://www.amazon.com/Bose-Companion-multimedia-speaker-Graphite/dp/B000HZBR64/"));

The strange thing is that these pages load just fine in a web browser, but any time we try to p开发者_如何学编程ull them through code, it is failing.

What could cause these functions to fail? Is it possible that they changed something on their end and that we need to do some custom logic with our calls?


After some further testing, it turns out that this was happening because Amazon needs the Accept parameter of the HttpWebRequest to be specifically set. When setting it to:

request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"

Everything worked fine. This is a recent change, so they must have altered something on their end.


check the user-agent of you request. make the user-agent the same as your browser. And check if you set any proxy for your app? maybe your browser and your app are using different proxies

0

精彩评论

暂无评论...
验证码 换一张
取 消