Why doesn't Python socket receive a final blank value during recv loop of HTTP socket?

I've written a miniature proxy module in Python 3 to simply sit between my browser and the web. 开发者_运维技巧My goal is to merely proxy the traffic going back and forth. One behavior of the program is to save the website responses I get in a local directory.

Everything works the way I expect, except for the simple fact that using socket.recv() in a loop seems to never yield the blank bytes object implied in the examples provided in the docs. Virtually every online example talks about the blank string coming through the socket when the server closes it.

My assumption is that something is going on via the keep-alive header, where the remote server never closes the socket unless its own timeout threshold is reached. Is this correct? If so, how on earth am I to detect when a payload is finished being sent? Observing that received data is smaller than my declared chunk size does not work at all, due to the way TCP functions.

To demonstrate, the following code opens a socket at an image file on Google's web server. I copied the actual request string from my browser's own requests. Running the code (remember, Python 3!) shows that binary image data is received to completion, but then the code never is capable of hitting the break statement. Only when the server closes the socket (after some 3 minutes of idle time) does this code actually reach the print command at the end of the file.

How on earth does one get around this? My goal is to not modify the behavior of my browser's requests—I don't want to have to set the keep-alive header to false or something gaudy like that. Is the answer to use some ugly timeouts (via socket.settimeout())? Seems laughable, but I don't know what else could be done.

Thanks in advance.

import socket

remote_host = 'www.google.com'
remote_port = 80

remote_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote_socket.connect((remote_host, remote_port))
remote_socket.sendall(b'GET http://www.google.com/images/logos/ps_logo2a_cp.png HTTP/1.1\r\nHost: www.google.com\r\nCache-Control: max-age=0\r\nPragma: no-cache\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip,deflate,sdch\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n\r\n')

content = b''
while True:
    msg = remote_socket.recv(1024)
    if not msg:
        break
    print(msg)
    content += msg

print("DONE: %d" % len(content))

---

If you have a keep-alive connection, there will be some indication of the message length in the headers of the response. See HTTP Message. Buffer recv until you have the complete header (terminated by a blank line), determine the message body length, and read exactly that much information.

Here's a simple class to buffer TCP reads until a message terminator or a specific number of bytes has been read. I added it to your example:

import socket
import re

class MessageError(Exception): pass

class MessageReader(object):
    def __init__(self,sock):
        self.sock = sock
        self.buffer = b''

    def get_until(self,what):
        while what not in self.buffer:
            if not self._fill():
                return b''
        offset = self.buffer.find(what) + len(what)
        data,self.buffer = self.buffer[:offset],self.buffer[offset:]
        return data

    def get_bytes(self,size):
        while len(self.buffer) < size:
            if not self._fill():
                return b''
        data,self.buffer = self.buffer[:size],self.buffer[size:]
        return data

    def _fill(self):
        data = self.sock.recv(1024)
        if not data:
            if self.buffer:
                raise MessageError('socket closed with incomplete message')
            return False
        self.buffer += data
        return True

remote_host = 'www.google.com'
remote_port = 80

remote_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote_socket.connect((remote_host, remote_port))
remote_socket.sendall(b'GET http://www.google.com/images/logos/ps_logo2a_cp.png HTTP/1.1\r\nHost: www.google.com\r\nCache-Control: max-age=0\r\nPragma: no-cache\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip,deflate,sdch\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n\r\n')
mr = MessageReader(remote_socket)
header = mr.get_until(b'\r\n\r\n')
print(header.decode('ascii'))
m = re.search(b'Content-Length: (\d+)',header)
if m:
    length = int(m.group(1))
    data = mr.get_bytes(length)
    print(data)
remote_socket.close()

Output

HTTP/1.1 200 OK
Content-Type: image/png
Last-Modified: Thu, 12 Aug 2010 00:42:08 GMT
Date: Tue, 21 Jun 2011 05:03:35 GMT
Expires: Tue, 21 Jun 2011 05:03:35 GMT
Cache-Control: private, max-age=31536000
X-Content-Type-Options: nosniff
Server: sffe
Content-Length: 6148
X-XSS-Protection: 1; mode=block


b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01l\x00\x00\x00~\x08\x03\x00\ (rest omitted)

---

One very easy way to get the server to close the connection is to add this header to your HTTP request:

Connection: close

By default, HTTP/1.1 servers are permitted to keep the connection open so you can create a second request. You should still create a timeout so you don't get starved for sockets when servers ignore the header.

---

When a tcp connection is closed, it will send a final blank message indicating that the socket has been closed. When you receive the message, You should most likely close the socket on your end as well.

---

Quite honestly the easiest and most reliable solution is still going to be using socket timeouts and encapsulating it in a try/except and utilizing the socket.timeout exception. You could probably look at the last bit of data recieved to see if it should or shouldn't have died.

remote_socket.setblocking(True) # not really needed but to emphasize this 
                                #is a blocking socket until the timeout
remote_socket.settimeout(15) # 15 second timeout
while True:
  try
    msg = remote_socket.recv(1024)
    if not msg:
        break
    print(msg)
    content += msg
  except socket.timeout:
    #do some checking on last received data
  else:
    #socket died for another reason or ended the way it was supposed to.