Pass object to thread and get it back after thread has run

I will say this right off the bat. I am an amateur at threading. I am a senior c# web developer, but I have a project that requires me to populate a lot of objects that take a long time to populate as they require WebRequests and Responses to populate. I have everything working without threading, but it does not run fast enough for my requirements. I would like to pass everything to a ThreadPool to have the threading managed for me as I may be queuing up 20,000 threads at the same time and for obvious reasons. I do not want to hit a website with the requests needed to populate all of them at once.

What I would like to do is to pass in an object, populate it, and then add it to a collection in the main thread once it is populated. Then once all the objects are populated, continue on with execution of the program. I do not know how many objects will need to be populated until they are all populated either.

My question...What is the best approach to doing this?

Here is the loop that I am trying to speed up:

foreach (HElement hElement in repeatingTag.RunRepeatingTagInstruction())
{
    object newObject = Activator.CreateInstance(curr开发者_如何学PythonentObject.GetType().GetGenericArguments()[0]);
    List<XElement> ordering = GetOrdering(tagInstructions.Attribute("type").Value);
    RunOrdering(ordering, newObject, hElement);
    MethodInfo method = currentObject.GetType().GetMethod("Add");
    method.Invoke(currentObject, new[] { newObject });
}

I don't know what the object is beforehand so I create it using the Activator. The RunOrdering method runs through the instructions that I pass that tell it how to populate the object. Then I add it to the collection. Also, the object itself may have properties that will require this method to run through and populate their data.

---

Since you probably have to wait for them all to be complete, all you need is a Parallel.ForEach() or equivalent. And a Thread-safe collection. Note that for I/O intensive tasks you would want to limit the number of Threads. 20.00 threads would be insane in any situation.

But we would need to see more details (code). Note that there is no such thing as "a collection in the main thread".

---

populate a lot of objects that take a long time to populate as they require WebRequests and Responses

Avoid Threading if you are doing requests. No speedup after two threads, merely existent with the two. A lot of truble for nothing.

---

Couple of suggestions:

If you are on .net 4 try using Tasks instead. You would have much better control over scheduling. Try to not share any objects, make them immutable and all the warnings and best practices about synchronisation, shared data etc.

And secondly you might want to think of an out of process solution like message queues (xMQ products or poor man's database table as queue) so you would have the chance to distribute your task over multiple machines if you need to.