retrieving just the title of a webpage in python_问答_开发者

retrieving just the title of a webpage in python

开发者 https://www.devze.com 2023-03-04 20:21 出处：网络

I have more than 5000 webpages i want the titles of all of them. In my project i am using the BeautifulSoup html parser like this.

相关专题：python

I have more than 5000 webpages i want the titles of all of them. In my project i am using the BeautifulSoup html parser like this.

soup = BeautifulSoup(open(url).read())
soup('title')[0].string

But its taking lots of time. Just for the title of a webpage i am reading the entire file and building the parse tree(I thought this is the reaso开发者_JAVA技巧n for delay, correct me if i am wrong).

Is there in any other simple way to do this in python.

It would certainly be faster if you just used a simple regular expression, BeautifulSoup is pretty slow. You could do something like:

import re
regex = re.compile('<title>(.*?)</title>', re.IGNORECASE|re.DOTALL)
regex.search(string_to_search).group(1)

You could always use a regular expression to do it, but that might break if you get a badly formatted page. That'd be something like this:

import re
titleRE = re.compile("<title>(.+?)</title>")
title = titleRE.search(s).group(1)

You could even use the simple string methods:

html = '<html> lots of crap <title>Title</title> even more crap </html>'
start = html.find('<title>') + 7 # Add length of <title> tag
end = html.find('</title>', start)
title = html[start:end]

However, this only guarantees that <title> is found before </title> in the page. Not that it is in the <head> section or anything.

Also, you should validate your assumption that it actually is the BeautifulSoup parsing that comsumes the lion's share of the time. (My guess is that open(url).read() for 5,000 resources takes quite some time, too. This you will not eliminate, no matter how you "parse" the HTML.)

Try

>> hearders = {'headers':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0'}
>>> n = requests.get('http://www.imdb.com/title/tt0108778/', headers=hearders)
>>> al = n.text
>>> al[al.find('<title>') + 7 : al.find('</title>')]
u'Friends (TV Series 1994\u20132004) - IMDb'