how can i make my program more responsive? (program that loads atleast 200 files) - i might have 1 idea

first - i want to say sorry for my butchered English.

I am building a program that uses a lot of files. i have a lot of foreach loops that loops through the harddisk and those files (atleast 200 files - 600 bytes each file in average), the loop is using XPath to search for values in the file (the files are XML files of course)

I need to find a way to m开发者_开发技巧ake my program more responsive - i thought of one which is the following: Computers memory has a faster speed of loading than computer harddisk - and i thought - maybe i should load those files to the memory and than loop the memory instead of looping the harddisk.., by the way if someone can tell me how much faster computers memory are (from harddisks) than thanks

Thanks in advanced.. Din

if someone didn't understand my English i will try to explain again

---

The best approach I think of is PLINQ in C#4.0. Group these XML files and query them with LINQ-to-XML parallelly. The following is a simple example, which loads all xml files in C:\xmlFolder and choose those documents which contains an element whose name is "key".

List<XDocument> xmls = Directory.EnumerateFiles(@"C:\XmlFolder").AsParallel()
                          .Select(path => XDocument.Load(path))
                          .Where(doc => doc.Descendants()
                                           .Any(ele => ele.Name.Equals("key")))
                          .ToList(); 

---

You should parse the XML files in a different thread and create objects with the required information, this way you will have instant access to the information.

---

Define "responsive." Do you mean that you want UI cues to continue to happen, or that you want to continue to be able to do other things in the UI while it's processing the files?

The former is easy, you can just toss in the occasional Application.DoEvents() in your loops. This will prompt the UI to perform any cues that are waiting (such as draw the window, etc.).

The latter is going to involve multi-threading. Diving into that is a bit more complex than can be taught in a paragraph or two, but some Google searches for "c# .net multi threading tutorial" should yield a ton of results. If you're not familiar with the basic concept of what multi-threading provides, I can further explain it.

---

Use a BackgroundWorker or a ThreadPool to spawn off multiple threads for the I/O, and have then read the data into a Queue (this is assuming the total size of your data is not too large). Have another thread(s) reading off of that Queue, and doing your internal xPath logic to pull whatever you need from those files.

Essentially, think of it as an instance of the Producer/Consumer design pattern, wherein your I/O reader threads are producers, and your XPath logic threads are consumers.

The type of the object in the queue could be just a byte-array, but I'd suggest a custom C# class that contains the byte array, as well as some of the file metadata in case you need it for whatever reason.

---

You can use database for storing XML files, it will be faster, more secure and reliable than you current schema. You can build indexes, concurrent access is enabled, XQuery/Xpath is supported and much more "pluses".

If you have only XML files, you can consider Native XML Databases, or if you have other types as well you can consider XML enabled DBMLS (such as Oracle or DB2).