How to crawl a web page for files of certain size

I need to crawl a list of several thousand hosts and f开发者_Python百科ind at least two files rooted there that are larger than some value, given as an argument. Can any popular (python based?) tool possibly help?

---

Here is an example of how you can get the filesize of an file on a HTTP server.

import urllib2

def sizeofURLResource(url):
    """
    Return the size of an resource at 'url' in bytes
    """
    info = urllib2.urlopen(url).info()
    return info.getheaders("Content-Length")[0]

There is also an library for building web scrapers here: http://dev.scrapy.org/ but I don't know much about it(just googled honestly).

---

Here is how I did it. See the code below.

import urllib2
url = 'http://www.ueseo.org'
r = urllib2.urlopen(url)
print len(r.read())