开发者

How to increase the writing speed of the streamwriter?

开发者 https://www.devze.com 2023-03-25 06:07 出处:网络
How to raise the speed of the stre开发者_如何转开发amwriter to write a 83MB csv file. I have inceresed the buffersize to 65536 but its also consuming more time. How to improve the speed.

How to raise the speed of the stre开发者_如何转开发amwriter to write a 83MB csv file. I have inceresed the buffersize to 65536 but its also consuming more time. How to improve the speed.


StreamWriter writer =new streamWriter(
      new FileStream(filePath, FileMode.CreateNew), Encoding.UTF8, 65536))
string str=string.Empty;
while((str = reader.ReadLine())!=null)
    writer.WriteLine(str)}        
writer.Close()


Depending on the number of lines your CSV file contains, you could possibly end up with a loop that executes millions of times. Not a good idea to access the disk that many times.

The easy way out (if you have memory) is to read the entire CSV file in a string[] in memory (using File.ReadAll() i think), do your processing and write it all once (File.WriteAll() i think). This will greatly increase your performance.

The other way out is to use asynchronous read/writes, increase buffer size AND create a mechanism to read bigger chunks of data. Having a big buffer if you are only reading 1 line will not help you.


I would try raising the buffer even more. At that buffer size it's going to take over 8200 writes to create the whole file. Try a buffer around 256K or 512K.


Based on your comment showing what your code is actualy doing:

File.WriteAllText( filePath, reader.ReadToEnd() );
0

精彩评论

暂无评论...
验证码 换一张
取 消