Is it possible to make a shallow copy of very large STL strings?

Good afternoon, We are building a prototype of a deduper. We are using a array of STL strings to store the records to be depuped. The array looks like this:

std::string* StringArray = new std::string[NumberDedupeRecor开发者_JS百科ds]

The records are very large, as large as 160,000,000 bytes. When we try to store a std::string version of a record to deduped in the std::string* StringArray, STL makes a deep copy of the string and mallocs a new buffer of at least 160,000,000 bytes. We quickly run out of heap memory and get a std::bad_alloc exception. Is there a workaround to avoid the deep copy and std::bad_alloc? Perhaps we should use a new data structure for storing the std::string records to be deduped or maybe we should save auto_ptr's.

We show a code snippet here:

std::string clara5(curr.getPtr()); 
char* const maryptr = (curr.getPtr() + n - curr.low()); 
maryptr[54] = '\x0'; 
StringArray[StringArrayCount] = clara5; 
curr.mPtr = (char*)StringArray[StringArrayCount].c_str(); 

std::multiset<Range>::iterator miter5 = ranges_type.lower_bound(Range(n));
(*miter5).mPtr = curr.mPtr; StringArrayCount += 1;

Thank you.

---

You can simply take a pointer or reference to the original std::string- including smart pointer if you find it necessary to enforce various ownership strategems.

---

If possible, rather than trying to use smart pointers, you may want to change your code so that you only have a few instances of std::string in memory at a time. This of course will depend on your access patterns, but you may be able to load and process one string (record) at a time rather than allocating an array for all of them at once.

EDIT: Given that the OP is trying to remove duplicates, this may not work very well.

---

I think the real answer to your problem is to use a rope - see http://www.sgi.com/tech/stl/Rope.html - std::string is not really designed to be use for very large strings.