reducing loading time of 100 pages of google

for my project i need to access entire pages(100) of google at a time for a particular keyword.I used 'for' loop for accessing pages in url written in my c# code.But it is taking more time to access.So开发者_开发百科me times it showing HttpRequest error.Any way to increase the speed?

Query them in parallel. HTTP is asynchronous by nature, so should be your request code.

In your case, speed is limited by the time it takes to fulfill an I/O request. You can speed up the total task by accessing servers in parallel (i.e. using ThreadPool). A browser will generally use a couple (2-8) parallel I/O requests to a serer, so so could you (for instance usefull if you also need image files or css files referenced by the google result). Since you'll have up to 100 servers, you can do it massively parallel; again a task the Threadpool will help you with.