C# speed of File.AppendAllText

I fetch data from a database and use File.AppendAllText to write each database row into one file totalling 8000 files. This whole process repeats in a loop(of queries) and it seems to be taking forever. Is there a better way to go about doing t开发者_StackOverflow中文版his? Does it make sense to store all the file handlers?

---

The documentation says "Opens a file, appends the specified string to the file, and then closes the file".
That is you open and close your file for each line.

Use something like this

string[] dbRows = new string[] { "1", "2", "3" };
using (var file = new StreamWriter("file.txt", true))
{
    foreach (var row in dbRows)
    {
        file.WriteLine(row);
    }
}

---

It may be related to several parameters and in the following cases parallelism won't help:

• Disk I/O: you will just make the I/O even slower
• Network capacity: once you reach the network limit, you will just get several files at the same time but overall speed won't change

You could also try to keep the target files open for the duration of the process and close all the files after all the source data is processed. This will avoid the whole open/write/close loop on each user/file.

---

When using a explicit filestream you can set the blocksize. Sometimes ago I found out that changing the blocksize some times has a dramatically impact on perfomance.

Perhaps you can also parallelize reading vom SQL Server and writng to disk. For example using a DataReader.

Also have a look at http://blogs.msdn.com/b/blogdoezequiel/archive/2011/02/11/best-practices-on-filestream-implementations.aspx

The like is mainly on SQL Server but also has some hints on FileStreams in Generic

---

I' m working with the same issue... writinng less more often is a lot faster, so it's something to do with byte allocation perhaps more than open/close streamreader.

Check out different frequencies and sizes of the write process...

if you write only strings of 500/1000/10000/100000 length, it's different performance. In an XYZ loop, i have found the fastest performance is writing in Strings of 200-300 characters takes my test about 5 seconds, and writing 10000 characters takes about 30 seconds for an SSD card.

That contradicts the idea that the bottleneck is due to write frequency! C# Performance - Chunking Write of file with AppendAllText