Parsing a website with a javascript call using Python

Since I counldn't find an API func开发者_开发问答tion in common wikimedia to get alicense of an image, the only thing left to do it to fetch the webpage and parse it myself.

For each image, there is a nice popup in wikimedia that lists the "Attribution" field which I need. For example, in the page http://commons.wikimedia.org/wiki/File:Brad_Pitt_Cannes_2011.jpg there is a link on the right saying "Use this file on the web". When clicking on it I can see the "Attribution" field which I need.

Using Python, how can I fetch the webpage and initiate a javascript call to open that pop up in order to retrieve the text inside the "Attribution" field?

Thanks!

meir

---

using unutbu's answer, I converted it to use Selenium WebDriver (rather than the older Selenium-RC).

import codecs
import lxml.html as lh
from selenium import webdriver

browser = webdriver.Firefox()
browser.get('http://commons.wikimedia.org/wiki/File%3aBrad_Pitt_Cannes_2011.jpg')
content = browser.page_source
browser.quit()

doc = lh.fromstring(content)
for elt in doc.xpath('//span[a[contains(@title,"Use this file")]]/text()'):
    print elt

output:

on the web
on a wiki

---

Assuming you can read Javascript, you can look at this Javascript file: http://commons.wikimedia.org/w/index.php?title=MediaWiki:Stockphoto.js&action=raw&ctype=text/javascript

You can see what the Javascript does in order to get it's info (look at get_author_attribution and get_license. You can port this to Python using BeautifulSoup to parse the HTML.

---

I'd be interested to see how this is done using other tools. Using Selenium RC and lxml, it can be done like this:

import selenium

sel=selenium.selenium("localhost",4444,"*firefox", "file://")   
sel.start()
sel.open('http://commons.wikimedia.org/wiki/File%3aBrad_Pitt_Cannes_2011.jpg')

sel.click('//a[contains(@title,"Use this file on the web")]')
print(sel.get_value('//input[@id="stockphoto_attribution"]'))
sel.stop()

yields

Georges Biard [CC-BY-SA-3.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons