Edit very large xml files

I would like to create a text box which loads xml files and let users edit them. However, I cannot use XmlDocument to load since the files can be very large. I am looking for options to stream/load the xml document in chunks so that I do not get ou开发者_高级运维t of memory errors -- at the same time, performance is important too. Could you let me know what would be good options?

---

Try Scintilla.NET, it is miles better than a TextBox!

http://scintillanet.codeplex.com/

Loading the document is easy:

using (TextReader reader = new StreamReader(myFilePath, Encoding.UTF8))
{
    scintillaDocument.Text = reader.ReadToEnd();
}

Or:

scintillaDocument.Text = File.ReadAllText(myFilePath);

---

Why bother with reading the xml into an xmldocument at all if all you're doing is pushing it into a textbox?

How big are you talking about here? Have you tried streaming it into a textbox yet?

sometextarea.Text = System.IO.File.ReadAllText(Server.MapPath("somexml.xml"));

Now, saving it back to the filesystem is a different story, especially if you want it to be 1. Valid xml and 2. Valid against a schema.

---

I too had to deal with large XML files (1+ gig) and had to parse elements out to import into a mySql database. I was successful by using a text-based stream reader. What I did was to keep reading in chunks until I had one complete single "record" of an xml based on the known

`<perRecordTag>`

   <other data / node elements>

`</perRecordTag>`

Then, I would do an XMLDocument to read from the string ( after the leading and trailing of the record ) stripped. I could then parse, review, whatever of that single record and move on.

Obviously, I had to retain all that after the end of xml record to start the beginning of the next read record element, but that was no problem.

---

I've had simimlar issues doing this type of thing with CSV file data.

DRapp is right, it's likely the cleanest way to approach the situation assuming the user isn't expecting to read everything at root level in one hit.

In theory all you need to be careful of is what is open or closed but you can store just this core info in a string and it shouldn't be too bulky.

and as DRapp suggests you simply load the data in to a stream and with a bit of careful position management you should be able to read and write.

your biggest issue is that if say at point x you want to replace the data in node y with some data of a different length you would either end up with a gap in the file or you would overwrite the next node / a portion of it.

so each time a chnage is made essentially you need to stream the file in to another file up to the point where the edit starts then stream in the edit then stream in the rest of the file.

You should be able to do all this with stream reader and stream writer objects that you sit on top of 1 stream instance on the original file plus a stream writer on a second temp file.

It's never gonna be fast though, purely because writing updates to a 1.x gig file takes time on the hard drive and no optimising is gonna change that.

---

You can use memory-mapped files to handle huge files. See MemoryMappedFile Class on MSDN. OK, it's low level, but can help. Available from .NET 4.

You can use readers with XMLNameTable support. It will enable string interning, so if a string appears frequently in a document the same string will represent it in memory for you.

You can also try a 3rd party XML parser. E.g. Altova XML is used in the company's products, so maybe it can do more things than the built-in .NET classes. BTW it's free.

---

I think you're trying to do too much in your text box here. Why not have the users edit the XML document in a tool that's meant for editing XML? Such a tool might even be able to handle large XML files.

Then, when the XML has been edited, the users can upload the complete XML document to your site.

---

I've not tried it with files that big, but you should look at Microsoft's XML Notepad 2007. It claims under a second load times for a 3mb document.

http://www.microsoft.com/download/en/details.aspx?id=7973

---

You're probably looking for an XmlTextReader.