c - WSAWaitForMultipleObjects blocking any thread but last

i have a problem with a multi-thread SMTP/POP3 server. The server starts a pool of threads to handle incoming connections. The main thread create the sockets and the the threads, passing the sockets as parameters in a proper structure. The loop function for the threads is the following:

SOCKET SMTP_ListenSocket = (SOCKET) data->SMTPconn;
SOCKET POP3_ListenSocket = (SOCKET) data->POP3conn;
static struct sockaddr_in ClntAddr;
unsigned int clntLen = sizeof(ClntAddr);    
hEvents[0] = CreateEvent(NULL, FALSE, FALSE, NULL); 
hEvents[1] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[2] = exitEvent; //HANDLE FOR A MANUAL RESET EVENT
WSAEventSelect(SMTP_ListenSocket, hEvents[0], FD_ACCEPT); 
WSAEventSelect(POP3_ListenSocket, hEvents[1], FD_ACCEPT); 

while(1){

      DWORD res = WaitForMultipleObjects(3, hEvents, FALSE, INFINITE);
      switch(res){

            case WAIT_OBJECT_0: {      
                ClientSocket = my_accept(SMTP_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
                /*  ...  */                    
                my_shutdown(ClientSocket,2);
                my_closesocket(ClientSocket);
                ClientSocket = INVALID_SOCKET;
        break;
             }

             case WAIT_OBJECT_0 + 1: {

                  ClientSocket = my_accept(POP3_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
                  /* ... */                        
                  my_shutdown(ClientSocket,2);
                  my_closesocket(ClientSocket);
                  ClientSocket = INVALID_SOCKET;
                  break;
             }

             case WAIT_OBJECT_0 + 2:
             {
                  exitHandler(0);
                  break;
     开发者_如何转开发        } 
      }//end switch

 }//end while

When the pool contains only one thread there's no problem. When the pool consist of more threads, only one thread accepts the incoming connections

---

Do you have the pooled threads all calling this same code? If so, then don't use WaitForMultipleObjects() (or WSAWaitForMultipleEvents()) like this. This kind of model only works reliably if one thread is polling connections. If you have multiple threads polling at the same time, then you have race conditions.

Instead, you should use AcceptEx() with Overlapped I/O or Completion Ports instead. The thread that creates the sockets can call AcceptEx() on each socket to queue a new operation on each one, then the pooled threads can use GetQueuedCompletionStatus() or GetOverlappedResult() to dequeue a pending connection without worrying about trampling on other threads. Once a connection is accepted, the receiving thread can process it as needed and then call AcceptEx() to queue a new operation for that socket.

---

Each thread here is setting a new WSAEventSelect prior to entering the wait. This overwrites any existing event selects. This means that, once a thread (call it thread A) accepts a connection, there is no event associated with the socket.

To solve this, you should call WSAEventSelect again within your switch, immediately after the accept(). This will restore the event binding immediately before going into any potentially lengthy processing.

Note that it's possible that two threads may be awoken for the same event, if the timing works out just right. You can hack around that by going back to your wait loop if the accept fails, but this is a bit unsatisfying.

So, instead of rolling your own version, use IO completion ports here. I/O completion ports have a number of additional features, and avoid potential race conditions in which two threads might pick up the same event. They also take steps to reduce context switches when your code is not CPU bound.