I have a project where I am trying to login to sprint and then do some screen scraping to get data about the different lines that the company controls. I have tried passing the cookies that are provided by the initial website call in the initial HttpWebRequest form post, but I do not get any cookies back that will denote user or session or anything. In fact, if I then try to use the WebClient class to get the landing page, the response url that I get back is the login page.
I think it is due to the fact that when you login, you get redirected to a page that does some processing and then redirects you to the landing page. I am passing in correct credentials and don't know where it is failing. Can anyone help me so that I do not need to use Watin or any other browser control to scrape that data as that will 开发者_开发百科be too slow.
Use Selenium.
It is normally for website testing, but you easily use it for your situation.
It allows you to launch a browser and programmatically control mouse clicks and keyboard presses to do exactly what you need.
You also run xpath on the HTML to read data, or even run custom javascript on pages if you need to get more complicated.
精彩评论