Replicate steps in downloading file

I'm trying to automate the download of a file from a website. Normally to download the file, I login with a username and password. Navigate to a particular screen then click a button.

I've been trying to watch the sequence of POSTs using Chrome's developer mode, and then replicate all the steps using .Net WebClient class, but to no success. I've derived from the WebClient class and added cookie handling. Which seems to be working. I go to the login page and post using WebClient.UploadValues. About half the times it seems to work. The next step appears to make another POST action to a reporting URL. Once again I use WebClient.UploadValues, but the response from the server is a page showing an internal error.

I have a couple of questions. 1) Are there better tools than hand coding C# code to replicate a bunch of web browser interactions? I really only care about being able to download the file at a particular time each day onto a Windows box. 2) The WebClient does not seem to be the best class to use for this. Perhaps it's a bit to simplistic. I tried using HttpWebRequest, but it has no facilities for encoding POST requests. Any other recommendations? 3) Although Chrome's developer plugin appears to show all interaction, I find it开发者_高级运维 a bit cumbersome to use. I'd be interested in seeing all of the raw communication (unencrypted though, the site is only accesses via https), so I can see if I'm really replicating all of the steps.

I can even post the exact code I'm using. The site I'm pulling data from, specifically is the Standard and Poors website. They have the ability to create custom reports for downloading historical data which I need for reporting, not republishing.

---

Using IE to download the file would be a much easier, as compared to writing C# / Perl / Java code to replicate http requests.

Reason is, even a slight change in JavaScript code can break the flow.

With IE, you can automate it using COM. Following VBA example opens IS and performs a google search:

Sub Search_Google()
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")

IE.Navigate "http://www.google.com" 'load web page google.com

While IE.Busy
  DoEvents  'wait until IE is done loading page.
Wend

IE.Document.all("q").Value = "what you want to put in text box"
ie.Document.all("btnG").Click 
'clicks the button named "btng" which is google's "google search" button

While ie.Busy
  DoEvents  'wait until IE is done loading page.
Wend

End Sub

---

3) Although Chrome's developer plugin appears to show all interaction, I find it a bit cumbersome to use. I'd be interested in seeing all of the raw communication (unencrypted though, the site is only accesses via https), so I can see if I'm really replicating all of the steps.

For this you can use Fiddler to view all the interaction going on and the RAW data going back and forth. To make it work with HTTPS you will need to install the Certificates to enable decryption of trafffic.