开发者

Downloading a webpage using urllib2 results in garbled junk? (only sometimes)

开发者 https://www.devze.com 2023-01-18 21:33 出处:网络
How come I hit this webpage, I get HTML text: http://itunes.apple.com/us/app/mobile/id381057839 But when I hit this webpage, I get garbled junk?

How come I hit this webpage, I get HTML text:

http://itunes.apple.com/us/app/mobile/id381057839

But when I hit this webpage, I get garbled junk?

http://itunes.apple.com/us/app/mobile/id375562663

I use the same download() function in python, which is here:

def download(source_url):
    try:
        socket.setdefaulttimeout(10)
        agent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AlexaToolbar/alxf-1.54 Firefox/3.6.10 GTB7.1"
     开发者_Go百科   ree = urllib2.Request(source_url)
        ree.add_header('User-Agent',agent)
        ree.add_header("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
        ree.add_header("Accept-Language","en-us,en;q=0.5")
        ree.add_header("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7")
        ree.add_header("Accept-Encoding","gzip,deflate")
        ree.add_header("Host","itunes.apple.com")
        resp = urllib2.urlopen(ree)
        htmlSource = resp.read()
        return htmlSource
    except Exception, e:
        print e


Solved. It was compression issue.

def download(source_url):
    try:
        socket.setdefaulttimeout(10)
        agents = ['Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)','Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1)','Microsoft Internet Explorer/4.0b1 (Windows 95)','Opera/8.00 (Windows NT 5.1; U; en)']
        ree = urllib2.Request(source_url)
        ree.add_header('User-Agent',random.choice(agents))
        ree.add_header('Accept-encoding', 'gzip')
        opener = urllib2.build_opener()
        h = opener.open(ree).read()
        import StringIO
        import gzip

        compressedstream = StringIO.StringIO(h)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        data = gzipper.read()
        return data

    except Exception, e:
        print e
        return ""
0

精彩评论

暂无评论...
验证码 换一张
取 消