i have some python proxy checker.
and to speed up check, i was decided change to multithreaded version,
and thread module is first for me, i was tried several times to convert to thread version
and look for many info, but it not so much easy for novice python programmer.
if anyone can help me really much appreciate!!
thanks in advance!
import urllib2, socket
socket.setdefaulttimeout(180)
# read the list of proxy IPs in proxyList
proxyList = open('listproxy.txt').read()
def is_bad_proxy(pip):
try:
proxy_handler = urllib2.ProxyHandler({'http': pip}) 开发者_StackOverflow中文版
opener = urllib2.build_opener(proxy_handler)
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib2.install_opener(opener)
req=urllib2.Request('http://www.yahoo.com') # <---check whether proxy alive
sock=urllib2.urlopen(req)
except urllib2.HTTPError, e:
print 'Error code: ', e.code
return e.code
except Exception, detail:
print "ERROR:", detail
return 1
return 0
for item in proxyList:
if is_bad_proxy(item):
print "Bad Proxy", item
else:
print item, "is working"
urllib2.install_opener()
function installs global opener, i.e. it's not thread safe. So don't use it and call opener.open()
method instead of global urllib2.urlopen()
function. Also use Queue
class from Queue
module to hold the list of proxies to check. The rest of your code is OK to use in threaded mode.
精彩评论