Why urllib2 doesn't work for me?_问答_开发者_运维开发者技术经验分享

I have installed 3 different python script on my ubuntu 10.04 32 bit machine with python 2.6.5.

All of these use the urllib2 and I always get this error:

urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

Why ?

Examples:

>>> import urllib2
>>> response = urllib2.urlopen("http://www.google.com")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>



>>> response = urllib2.urlopen("http://search.twitter.com/search.atom?q=hello&rpp=10&page=1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line开发者_运维百科 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

UPDATE:

$ ping google.com
PING google.com (72.14.234.104) 56(84) bytes of data.
64 bytes from google.com (72.14.234.104): icmp_seq=1 ttl=54 time=25.3 ms
64 bytes from google.com (72.14.234.104): icmp_seq=2 ttl=54 time=24.6 ms
64 bytes from google.com (72.14.234.104): icmp_seq=3 ttl=54 time=25.1 ms
64 bytes from google.com (72.14.234.104): icmp_seq=4 ttl=54 time=25.0 ms
64 bytes from google.com (72.14.234.104): icmp_seq=5 ttl=54 time=23.9 ms
^C
--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 23.959/24.832/25.365/0.535 ms


$ w3m http://www.google.com
w3m: Can't load http://www.google.com.

$ telnet google.com 80
Trying 1.0.0.0...
telnet: Unable to connect to remote host: Connection timed out

UPDATE 2:

I am at home and I am using a router and an Access point :-. However I have just noticed that Firefox doesn't work for me. But chrome, synaptic and other browsers like Midori and Epiphany, etc does work.

UPDATE 3:

>>> useragent = 'Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Ubuntu/10.04 Chromium/6.0.472.62 Chrome/6.0.472.62 Safari/534.3)'
>>> request = urllib2.Request('http://www.google.com/')
>>> request.add_header('User-agent', useragent )
>>> urllib2.urlopen(request)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

UPDATE 4:

>>> socket.setdefaulttimeout(50)
>>> urllib2.urlopen('http://www.google.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1136, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

UPDATE 5:

Wireshark results ( packet sniffer ):

Firefox: http://bit.ly/chtynm

Chrome: http://bit.ly/9ZjILK

Midori: http://bit.ly/cKilC4

midori is another browser that works for me. Only Firefox doesn't work.

As suggested, troubleshoot the network setup first.

First, check that you can ping the host you're trying to connect to:

$ ping www.google.com

Then try a HTTP connection using for instance w3m:

$ w3m http://www.google.com

i can think only in one raeson right now , XRobot they don't trust you .

woh they ? they :)

when you want to do some crawling or scraping and you see that they don't trust you , you just have to dump them , how is that ?

First of all you should know that some web server filter they contain for malicious software like robot (maybe they know you are a robot, hmmm XRobot :) ), how they do that? there is many way to filter : like using captcha in the webpage , filtering by User-Agent ...

And because your ICMP ping work ,chrome browser work but not w3m i suggest you change the User-Agent like this:

user_agent = 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.10) Gecko/20100915\
              Ubuntu/10.04 (lucid) Firefox/3.6.10'

request = urllib2.Request('http://www.google.com/')
request.add_header('User-agent', user_agent )

opener.open(request)

maybe i'm getting paranoia here, but hopefully this can help you :)

To what URL are you trying to connect? There could be any number of reasons for this error, most of them having to do with either an incorrect name or IP address or a problem with your link to the remote host.

Sounds like chrome and synaptic might be using an HTTP proxy. In chromium, go to Options/Under the hood/Change Proxy Settings. Check the gnome proxy settings with:

$ gconftool-2 -R /system/proxy

Do these steps one by one-

Check if you are connected & it's working. ping google.com
If all fine, & your internet connection is just slow then do this -

import socket
socket.setdefaulttimeout(300) #in seconds.

This will extend the timeout of your socket.

Have you tested your network connection? Something on the other end is not responding, due to a severance of connection or a refusal of connection.

Also, post the version of python you're using.

UPDATE:

This is almost certainly a network issue. I also have an Ubuntu 10.04 machine (32-bit) with Python 2.6.5 that's a nearly pristine install, and I am unable to reproduce the problem.

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> response = urllib2.urlopen("http://www.google.com")
>>> print response.read(100)
<!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"><

I was experiencing similar behavior. Eventually I remembered that I'd run a script earlier that installed a proxy. Removing the proxy from urllib2 solved my problem. This doesn't explain your telnet or w3m mysteries, but it may help someone with the urllib2 part.

This page helped me figure out how to remove the proxy.

http://www.decalage.info/en/python/urllib2noproxy

Here's the code:

proxy_handler = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy_handler)
urllib2.install_opener(opener)

I think there is some issue with permissions. I had the same problem on my Ubuntu 11.10. Calling python with sudo did the trick for me. Give it a try;

jeffisabelle:~ $ python
Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import urllib2
>>> response = urllib2.urlopen("http://www.google.com")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 394, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 412, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1201, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1171, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>


jeffisabelle:~ $ sudo python
Python 2.7.2+ (default, Oct  4 2011, 20:03:08) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> response = urllib2.urlopen("http://www.google.com")
>>>