how can I speed up insertion of many rows to a table via ADO.NET?

I have a table that has 5 columns: AcctId (int), Address1 (varchar), Address2 (varchar)开发者_JAVA技巧, Person1 (varchar), Person2 (varchar) . I'm generating random data to insert into this table via a C# console application. I've tried doing this random data insert via SQL-Server and decided it was not a good solution -- SQL is not good at random on an each-row basis. Generating the random data -- 975k rows of it -- takes a minimal amount of time. It's in a List of custom objects.

I need to take this random data and update many rows in the database with the new random data. I tried updating the rows one at a time, very slow because of the repeated searching of the List object in code. So I think the best approach is to put all the randomized data into a table in the database, then update all the other tables that use this data. I.e. UPDATE t SET t.Address1=d.Address1 FROM Table1 t INNER JOIN RandomizedData d ON d.AcctId = t.Acct_ID. The database is very un-normalized so this Acct data is sprinkled all over the place. I've got no control of the normalization.

So, having decided to insert all of the randomized data into a single table, I set out to create insert scripts:

USE TheDatabase
Insert tmp_RandomizedData
 SELECT 1,'4392 EIGHTH AVE','','JENNIFER CARTER','BARBARA CARTER'  UNION ALL  
 SELECT 2,'2168 MAIN ST','HNGR F','DANIEL HERNANDEZ','SUSAN MARTIN'
// etc another 98 times... 
// FYI, this is not real data! 

I'm building this INSERT script in batches of 100. It's taking on average 175 ms to run each insert. Does this seem like a long time? It's going to take about 35 mins to run the whole insert.

The table doesn't have a primary key or any indexes. I was planning on adding those after all the data is inserted (thinking that that would be faster).

Is there a better way to do this?

---

The SQLBulkCopy class in .net can blast records in pretty quickly. I used this to transfer data from an i-Series database to SQL Tables very rapidly.

---

Use BCP. You can use this article as a guide. It's for VB6 but the gist is exactly the same. The trick is to use the BULK INSERT command.

---

... Read more of your question, you might also want to look at Sql RedGates sample data generator, it generates tons of data really, really, fast.

Use larger batches, 50,000 to 75,000 rows. On SQL 2000 on hardware from 2000, the sweet spot for inserts was 50,000 rows. This was on a live production database, with indexes, during the day on a very large table.

Small batch sizes are better for inserts into a highly active table and where there is a high deadlock risk. Is anyone else using this table while your doing inserts?

Is this a one time import? Let it run over night.

Finally, INSERT statements executed via ADO.NET isn't really an optimal ETL solution. SSIS, DTS, (or any other ETL solution, such as Talend) would be more appropriate for heavy duty data moving. On the other hand, if all you have is a hammer...