Processing and updating a large file row by row

So I am processing a 200 mb txt file and I have to read each row in the file update one or开发者_如何转开发 two columns and then save the same. What is the best way to achieve the same?

I was thinking of lading into a datatable but holding that big of a file in memory is a big pain.

I realise I should do it in batches but what is the best way to achieve the same?

I dont think I want to load into a dB first cos I cant do a mass update anyways. i Have to do a line by line read there too.

Just as an update my files basically have columns in any order and I need to update two or more columns all the time.

Thanks.

---

Read a line, parse it, and write fields into a temp file. When all the lines are done, delete the original file and rename the temp file.

---

To add to what Ants said...

You have options ...

• Line by line:

  StreamReader fileStream = new StreamReader( sourceFileName );
  StreamWriter ansiWriter = new StreamWriter( destinationFileName,  
    false, Encoding.GetEncoding( 20127 ) );  
  string fileContent;  
  while ( ( fileContent = fileStream.ReadLine() ) != null )  
  {  
      YourReplaceMethod( fileContent );  
      ansiWriter.WriteLine( fileContent );  
  }
  fileStream.Close();  
  ansiWriter.Close();  
  

• Bulk (today's boxes should be able to handle 200MB w/o problems):

  byte[] bytes = File.ReadAllBytes( sourceFileName );
  byte[] writeMeBytes = YourReplaceMethod( bytes );
  File.WriteAllBytes( destinationFileName, writeMeBytes );