Reuse a Client TCP Socket for Multiple HTTP Connections_问答_开发者

Greetings all.

I am making a ANSI C multi-threaded www-crawler (HTTP 1.1 compatible) on Linux 2.6.29-3.ydl61.3 and fairly progressed well. I have '000 of domains in a MySQL database to collect pages from. I can open any/all the domains in the crawler in keep-alive mode, as desired. I use POSIX threading and there are no contentions or data races whatsoever.

While the target servers seem ready to allow me to issue multiple concurrent or sequential requests for pages on each server socket (since each server returns 'Connection: Keep-Alive' as expected), I cannot actually do so... I can only fetch one page per socket connection... i.e. I can write a typical HTTP GET request to the socket via a file-descriptor and read down the response. Then immediately after that, I can only write to the fd BUT NOT read anymore! So while I have multiple (some into hundreds) urls per domain... it seems that I have to keep recreating socket connections to the same servers for each write/read (extremely memory wasteful and slow) rather than create only one client TCP connection and keep reusing the fd/socket until am done with the domain.

See below a partial output of 'netstat --inet -a' (note that as undesired I have multiple local socket connections to the same domain - these are not concurrent per domain):

tcp 0 0 gcell1:38614 tcp 0 0 gcell1:34678 tcp 0 0 gcell11:34768 tcp 0 0 gcell11:56085 tcp 0 0 gcell11:34661 tcp 0 0 gcell11:34785 tcp 0 0 gcell11:46660 tcp 0 0 gcell11:34697 tcp 0 0 gcell11:37510 tcp 0 0 gcell11:37516 tcp 0 0 gcell11:34710 tcp 0 0 gcell11:34711 tcp 0 0 gcell11:46677 tcp 0 0 gcell11:56513 tcp 0 0 gcell11:57560 tcp 0 0 gcell11:4663开发者_运维问答4 67.225.194.54:http CLOSE_WAIT 67.225.194.54:http CLOSE_WAIT 67.225.194.54:http CLOSE_WAIT www.kenic.or.ke:http CLOSE_WAIT 67.225.194.54:http CLOSE_WAIT x2web02.myhosting.com:http CLOSE_WAIT x2web02.myhosting.com:http CLOSE_WAIT www.kenic.or.ke:http CLOSE_WAIT x2web02.myhosting.com:http CLOSE_WAIT x2web02.myhosting.com:http CLOSE_WAIT x2web02.myhosting.com:http CLOSE_WAIT ip-72-167-251-99.ip.se:http CLOSE_WAIT

The client socket is created as below (partial code only)

if((http_socket_fd = socket (PF_INET, SOCK_STREAM, IPPROTO_TCP))!=SKMG_FAILURE) //typical
...
fcntl(http_socket_fd,SOCK_NONBLOCK); //set to non-block
...
setsockopt(http_socket_fd, SOL_SOCKET, SO_KEEPALIVE, &optval, optlen); //local TCP keep-alive used
...
while(connect(http_socket_fd, (struct sockaddr *)&http_name, sizeof (struct sockaddr_in)) == (-1))
...
return http_socket_fd;

After this I just use write/read on the fd. And it works perfectly BUT for ONLY one round trip.

1) How can I reuse http_socket_fd for each HTTP GET write/read per domain without needing to create a new local TCP socket for each url?? Merely passing http_socket_fd to every page fetch call per domain is exactly what has failed to work. [CRITICAL]

2) How can I make asynchronous requests to these servers on this one thread per socket per domain paradigm? I run 4 concurrent threads (my server is dual-threaded), i.e. 4 different concurrent domain fetches. [NON-CRITICAL]

The usual practice is to create one client socket per connection. It is also a bad idea to share sockets among threads.

Instead of writing your own HTTP client, have you ever considered using a library like libcurl that provide many advanced features? The libcurl site has a sample program that downloads content using multiple threads. Also have a look at ZeroMQ, a high-performance messaging framework. A ZeroMQ socket could be used to connect to multiple servers and efficiently download data. (See The Guide).