开发者

Python- is there a module that will automatically scrape the content of an article off a webpage?

开发者 https://www.devze.com 2023-03-16 10:24 出处:网络
I know there is lxml and BeautifulSoup, but that won\'t work for my project, because I don\'t know in advance what the H开发者_如何转开发TML format of the site I am trying to scrape an article off of

I know there is lxml and BeautifulSoup, but that won't work for my project, because I don't know in advance what the H开发者_如何转开发TML format of the site I am trying to scrape an article off of will be. Is there a python-type module similar to Readability that does a pretty good job at finding the content of an article and returning it?


It's possible to do using PhantomJS (C++) or PyPhantomJS (Python).

They're both headless WebKit based browsers, which you can fully control from JavaScript. Because you can control it from JavaScript, I find it is really easy to do stuff such as scrape the content of an article.

PyPhantomJS also has a plugin system, so that's definitely a plus. :)


Extracting the real content from a content-page can not be done automatically - at least not with the standard tools. You have to define/identify where the real content is stored (by specifying the related CSS ID or class in your own HTML extraction code).


Using HTQL, the query is: &html_main_text

0

精彩评论

暂无评论...
验证码 换一张
取 消