DataGridView Optimize Performance using Paging

In my app Datagridview displays object Proxy

• Proxy has two properties Address, Status

• DataGridview is bound to List which holds the Proxy objects.

• The DataGridView and UI becomes unresponsive due to the heavy load on the memory, as the list reaches 1 million proxy count.

• The app is harvesting proxies from diffrent websites, how do I scale the application to handle huge lists.

开发者_如何学Go

My concern, is harvesting, and implementing paging at the same time.

Paging with SQLCe, is it a good solution?? or will sql ce slow the harvesting process, or is there a better solution, i don't know.

the app harvests arround 500 - 1700 proxies per second, it is a feature, to extract "as fast as possible", I now there are other obvious limitations, bottle necks, but i am ignoring them for now.

Please advice how do i keep the speed, and make it scale, best practices., I am not sure about SQLCe

---

Now why would you ever want to display 1 million records to the user?! Even if paged, he'd still have to click through, let's say, 10000 pages!

Implement filtering, only display what's necessary and limit it to 7 records. Add float Score to Proxy; express it as a percentage - 0% means google.com didn't load at all, 100% means no slowdown compared to direct connection (haha).

Then it's

var displayedProxies = myProxies.OrderByDescending(Score).Take(7);

Think of potential usage scenarios and make the UI fit. In example, if it's targeted at spammers wanting to send out billions of emails, you just need one button - "Export in (machine-readable format name here)". However, if it's just some user wanting to surf anonymously, you can give him a list of "7 random proxies" with a message, that the scores are updating. Then just replace those 7 random ones in real-time with a list of the 7 best found so far.

---

I agree, the best approach is to get the data in chunks, calling a stored proc that receives the page number and the number of records that you want to be returned and then binding the records to the grid.

If there are filters applied to the grid, I would also pass them in to the stored proc.

I would disable VIEWSTATE on the datagrid if you are still passing many records (say more than a thousand per page); in fact, if you have too many records and you want this thing to fly, I would prefer a mix of ajax calls to a web service to get the data, coupled with the jquery datatables plugin, which I find fantastic and fairly well documented. Here's the link.

Edit: If you do the jquery datatables/webservice approach, try to convince people not to use IE Version < 9. IE Javascript engine sucks on IE 6 and 7 and less so on IE8 but still pretty bad compared to FF, Chrome, etc.