Web Scraper design

can we make application that searches google for a word and navigates to various pages? Using Httpwebresponse or and search for a word on rendered page?and it should have multiple proxy usage i.e all ab开发者_运维百科ove is multi threaded and each thread has different proxy.

So far I have failed to do so in return GetResponse says "Method not allowed"

---

You might want to look at Google API as that would be much more easy to access but I do no know if your usage would require you to buy the service license.

Also, verify that you get the right response as Google sometimes uses iframes for content, maybe you only get the outer frame?