开发者

Different results for the same RSS feed fetching from different user agents

开发者 https://www.devze.com 2023-02-03 02:56 出处:网络
If I add a feed URL to Google Reader or to a desktop feed aggregator, I receive nice results. The URL is:

If I add a feed URL to Google Reader or to a desktop feed aggregator, I receive nice results. The URL is:

http://estaticos03.marca.com/rss/futbol_1adivision.xml

But when I fetch the same URL from a script (python script, using feedparser library) I am getting slightly different content for the same results (the title for each entry, for example, is different and all in uppercase).

I believe something is done on the server-side to try to discourage people like me to parse the content for my own projects (the feed is from a popular football newspaper), but I am not sure about it. I tried to pass some user agents (like the google reader one) but still no luck, so maybe they check the IP as well? I am really confused.

开发者_运维问答Any idea why is this happening to me?

Thanks!


AFAIK Google Reader does some "magic" in the content to beautify it. They strip some tags and styles to avoid breaking their interface.

Can you provide more details on the differences?


Did you changed the user agent of your script? Try to mimic Firefox and see what happen.


All right folks, I found it. I analyzed the source XML received (as @TryPyPy). I had been trusting too much the feedparser library. Latest official version (4.1) has a bug related to mistakeing the title tag from media namespace instead of the original one:

http://code.google.com/p/feedparser/issues/detail?id=76

So I reinstalled from trunk and now everything is OK. Thanks for helping anyway!

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号