Find and append each reference to a html link - Python

I have a HTML file I g开发者_如何转开发ot from Wikipedia and would like to find every link on the page such as /wiki/Absinthe and replace it with the current directory added to the front such as /home/fergus/wikiget/wiki/Absinthe so for:

<a href="/wiki/Absinthe">Absinthe</a>

becomes:

<a href="/home/fergus/wikiget/wiki/Absinthe">Absinthe</a>

and this is throughout the whole document.

Do you have any ideas? I'm happy to use BeautifulSoup or Regex!

---

If that's really all you have to do, you could do it with sed and its -i option to rewrite the file in-place:

sed -e 's,href="/wiki,href="/home/fergus/wikiget/wiki,' wiki-file.html

However, here's a Python solution using the lovely lxml API, in case you need to do anything more complex or you might have badly formed HTML, etc.:

from lxml import etree
import re

parser = etree.HTMLParser()

with open("wiki-file.html") as fp:
    tree = etree.parse(fp, parser)

for e in tree.xpath("//a[@href]"):
    link = e.attrib['href']
    if re.search('^/wiki',link):
        e.attrib['href'] = '/home/fergus/wikiget'+link

# Or you can just specify the same filename to overwrite it:
with open("wiki-file-rewritten.html","w") as fp:
    fp.write(etree.tostring(tree))

Note that lxml is probably a better option than BeautifulSoup for this kind of task nowadays, for the reasons given by BeautifulSoup's author.

---

This is solution using re module:

#!/usr/bin/env python
import re

open('output.html', 'w').write(re.sub('href="http://en.wikipedia.org', 'href="/home/fergus/wikiget/wiki/Absinthe', open('file.html').read()))

Here's another one without using re:

#!/usr/bin/env python
open('output.html', 'w').write(open('file.html').read().replace('href="http://en.wikipedia.org', 'href="/home/fergus/wikiget/wiki/Absinthe'))

---

You can use a function with re.sub:

def match(m):
    return '<a href="/home/fergus/wikiget' + m.group(1) + '">'

r = re.compile(r'<a\shref="([^"]+)">')
r.sub(match, yourtext)

An example:

>>> s = '<a href="/wiki/Absinthe">Absinthe</a>'
>>> r.sub(match, s)
'<a href="/home/fergus/wikiget/wiki/Absinthe">Absinthe</a>'

---

from lxml import html

el = html.fromstring('<a href="/wiki/word">word</a>')
# or `el = html.parse(file_or_url).getroot()`

def repl(link):
    if link.startswith('/'):
       link = '/home/fergus/wikiget' + link
    return link

print(html.tostring(el))
el.rewrite_links(repl)
print(html.tostring(el))

Output

<a href="/wiki/word">word</a>
<a href="/home/fergus/wikiget/wiki/word">word</a>

You could also use the function lxml.html.rewrite_links() directly:

from lxml import html

def repl(link):
    if link.startswith('/'):
       link = '/home/fergus/wikiget' + link
    return link

print html.rewrite_links(htmlstr, repl)

---

I would do

import re

ch = '<a href="/wiki/Absinthe">Absinthe</a>'

r = re.compile('(<a\s+href=")(/wiki/[^"]+">[^<]+</a>)')

print ch
print
print r.sub('\\1/home/fergus/wikiget\\2',ch)

EDIT:

this solution have been said not to capture tags with additional attribute. I thought it was a narrow pattern of string that was aimed, such as <a href="/wiki/WORD">WORD</a>

If not, well, no problem, a solution with a simpler RE is easy to write

r = re.compile('(<a\s+href="/)([^>]+">)')

ch = '<a href="/wiki/Aide:Homonymie" title="Aide:Homonymie">'
print ch
print r.sub('\\1home/fergus/wikiget/\\2',ch)

or why not:

r = re.compile('(<a\s+href="/)')

ch = '<a href="/wiki/Aide:Homonymie" title="Aide:Homonymie">'
print ch
print r.sub('\\1home/fergus/wikiget/',ch)