开发者

beautifulsoup and mechanize to get ajax call result

开发者 https://www.devze.com 2022-12-26 01:33 出处:网络
hi im building a scraper using python 2.5 and beautifulsoup but im stuble upon a problem ... part of the web page is generating

hi im building a scraper using python 2.5 and beautifulsoup but im stuble upon a problem ... part of the web page is generating a开发者_运维问答fter user click on some button, whitch start an ajax request by calling specific javacsript function using proper parameters

is there a way to simulate user interaction and get this result? i come across a mechanize module but it seems to me that this is mostly used to work with forms ...

i would appreciate any links or some code samples thanks


ok so i have figured it out ... it was quite simple after i realised that i could use combination of urllib, ulrlib2 and beautifulsoup

import urllib, urllib2
from BeautifulSoup import BeautifulSoup as bs_parse

data = urllib.urlencode(values)
req  = urllib2.Request(url, data)
res  = urllib2.urlopen(req)
page = bs_parse(res.read())


No, you can't do that easily. AFAIK your options are, easiest first:

  1. Read the AJAX javascript code yourself, as a human programmer, understand it, and then write python code to simulate the AJAX calls by hand. You can also use some capture software to capture requests/responses made live and try to reproduce them with code;
  2. Use selenium or some other browser automation tool to fetch the page on a real web browser;
  3. Use some python javascript runner like spidermonkey or pyv8 to run the javascript code, and hook it to your copy of the HTML dom;
0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号