I would like to scrape web app with authentication. I am planning on using django/python to build the app. The app should log开发者_运维百科 a user in the backend, and pick list of tables and store those in Django.
BeautifulSoup will be most likely used to get list of tables.
What would be best approach to log a user with username and password, and crawl different pages of the backend.
There's also Scrapy with this django-related feature :
http://doc.scrapy.org/experimental/djangoitems.html
精彩评论