multi-threaded proxy checker having problems

I am trying to create a proxy checker. This is my first attempt at multithreading and it's not going so well, the threads seem to be waiting for one to complete before initializing the next.

Imports System.Net
Imports System.IO
Imports System.Threading

Public Class Form1
    Public sFileName As String
    Public srFileReader As System.IO.StreamReader
    Public sInputLine As String

    Public Class WebCall
        Public proxy As String
        Public htmlout As String

        Public Sub New(ByVal proxy As String)
            Me.proxy = proxy
        End Sub

        Public Event ThreadComplete(ByVal htmlout As String)

        Public Sub send()
            Dim myWebRequest As HttpWebRequest = CType(WebRequest.Create("http://www.myserver.com/ip.php"), HttpWebRequest)
            myWebRequest.Proxy = New WebProxy(proxy, False)
            Try
                Dim myWebResponse As HttpWebResponse = CType(myWe开发者_如何学GobRequest.GetResponse(), HttpWebResponse)
                Dim loResponseStream As StreamReader = New StreamReader(myWebResponse.GetResponseStream())
                htmlout = loResponseStream.ReadToEnd()
                Debug.WriteLine("Finished - " & htmlout)
                RaiseEvent ThreadComplete(htmlout)
            Catch ex As WebException
                If (ex.Status = WebExceptionStatus.ConnectFailure) Then
                End If
                Debug.WriteLine("Failed - " & proxy)
            End Try
        End Sub
    End Class

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim proxy As String
        Dim webArray As New ArrayList()
        Dim n As Integer
        For n = 0 To 2
            proxy = srFileReader.ReadLine()
            webArray.Add(New WebCall(proxy))
        Next

        Dim w As WebCall
        For Each w In webArray
            Threading.ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf w.send), w)
        Next w

    End Sub

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        srFileReader = System.IO.File.OpenText("proxies.txt")
    End Sub
End Class

---

I'll put this as an answer, rather than a comment:

the threads seem to be waiting for one to complete before initializing the next

The ThreadPool is exactly what it sounds like: a small pool of threads. For our purposes, let's imagine the pool has 10 threads. If you queue up 1000 tasks, the first 10 tasks are executed in the 10 available threads, while the others are queued up and wait for a task to complete.

HTTP calls are slow, and it holds one of your threads hostage until it completes. Understandably, kicking off lots of long-running tasks very quickly starves the ThreadPool of all available threads immediately, and the remaining tasks are queued up until another thread becomes available. In other words, what you're seeing is the ThreadPools intended behavior.

For what its worth, you can use WebClient.DownloadStringAsync(Uri uri, object token) and WebClient.DownloadStringCompleted event to make HTTP calls without blocking the current thread, and therefore release it back to the threadpool for your other tasks to use.

I was under the assumption that the threads can run simultaneously "side-by-side"

Threads do indeed run side-by-side, but you won't actually have 1000 threads in flight at once. Threads have a huge overhead, since each thread has its own stack, and by default the stack size is about one megabyte in size.

Since threads are so expensive to create, .NET pre-allocates a handful of threads and puts them in the ThreadPool. (The default number of threads depends on the size of your virtual address space, number of CPU cores, etc.) Tasks are executed on a threadpool thread, once completed, the thread is returned back to the pool so it can be re-used by another task -- which is better than creating a new thread for each task.

so do I need to set the number of threads, with setminthreads/maxthreads?

No. Don't fiddle with the SetMinThreads/SetMaxThreads methods -- the ThreadPool is already highly optimized to add more threads to the pool when needed. Unless you're an expert in the way the CLR handles threads, there is a very high probability that messing with the TheadPool defaults with decrease performance.

You may find this useful: http://www.albahari.com/threading/

---

Why don't you try to develop a web or online proxy checker? Nowadays it's very easy using AJAX and you don't need to worry or manage threads, it's automatically done between your browser and server with asynchronous JavaScript.