[python] Python socket receive - incoming packets always have a different size

I'm using the SocketServer module for a TCP server. I'm experiencing some issue here with the recv() function, because the incoming packets always have a different size, so if I specify recv(1024) (I tried with a bigger value, and smaller), it gets stuck after 2 or 3 requests because the packet length will be smaller (I think), and then the server gets stuck until a timeout.

class Test(SocketServer.BaseRequestHandler):

def handle(self):

   print "From:", self.client_address

   while True:    

     data = self.request.recv(1024)
     if not data: break

     if data[4] == "\x20":              
       self.request.sendall("hello")
     if data[4] == "\x21":
       self.request.sendall("bye")
     else:
       print "unknow packet"
   self.request.close()
   print "Disconnected", self.client_address

launch = SocketServer.ThreadingTCPServer(('', int(sys.argv[1])),Test)

launch.allow_reuse_address= True;

launch.serve_forever()

If the client sends multiples requests over the same source port, but the server gets stuck, any help would be very appreciated, thank !

You can alternatively use recv(x_bytes, socket.MSG_WAITALL), which seems to work only on Unix, and will return exactly x_bytes.

You could try always sending the first 4 bytes of your data as data size and then read complete data in one shot. Use the below functions on both client and server-side to send and receive data.

def send_data(conn, data):
    serialized_data = pickle.dumps(data)
    conn.sendall(struct.pack('>I', len(serialized_data)))
    conn.sendall(serialized_data)


def receive_data(conn):
    data_size = struct.unpack('>I', conn.recv(4))[0]
    received_payload = b""
    reamining_payload_size = data_size
    while reamining_payload_size != 0:
        received_payload += conn.recv(reamining_payload_size)
        reamining_payload_size = data_size - len(received_payload)
    data = pickle.loads(received_payload)

    return data

you could find sample program at https://github.com/vijendra1125/Python-Socket-Programming.git

I know this is old, but I hope this helps someone.

Using regular python sockets I found that you can send and receive information in packets using sendto and recvfrom

# tcp_echo_server.py
import socket

ADDRESS = ''
PORT = 54321

connections = []
host = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host.setblocking(0)
host.bind((ADDRESS, PORT))
host.listen(10)  # 10 is how many clients it accepts

def close_socket(connection):
    try:
        connection.shutdown(socket.SHUT_RDWR)
    except:
        pass
    try:
        connection.close()
    except:
        pass

def read():
    for i in reversed(range(len(connections))):
        try:
            data, sender = connections[i][0].recvfrom(1500)
            return data
        except (BlockingIOError, socket.timeout, OSError):
            pass
        except (ConnectionResetError, ConnectionAbortedError):
            close_socket(connections[i][0])
            connections.pop(i)
    return b''  # return empty if no data found

def write(data):
    for i in reversed(range(len(connections))):
        try:
            connections[i][0].sendto(data, connections[i][1])
        except (BlockingIOError, socket.timeout, OSError):
            pass
        except (ConnectionResetError, ConnectionAbortedError):
            close_socket(connections[i][0])
            connections.pop(i)

# Run the main loop
while True:
    try:
        con, addr = host.accept()
        connections.append((con, addr))
    except BlockingIOError:
        pass

    data = read()
    if data != b'':
        print(data)
        write(b'ECHO: ' + data)
        if data == b"exit":
            break

# Close the sockets
for i in reversed(range(len(connections))):
    close_socket(connections[i][0])
    connections.pop(i)
close_socket(host)

The client is similar

# tcp_client.py
import socket

ADDRESS = "localhost"
PORT = 54321

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ADDRESS, PORT))
s.setblocking(0)

def close_socket(connection):
    try:
        connection.shutdown(socket.SHUT_RDWR)
    except:
        pass
    try:
        connection.close()
    except:
        pass

def read():
    """Read data and return the read bytes."""
    try:
        data, sender = s.recvfrom(1500)
        return data
    except (BlockingIOError, socket.timeout, AttributeError, OSError):
        return b''
    except (ConnectionResetError, ConnectionAbortedError, AttributeError):
        close_socket(s)
        return b''

def write(data):
    try:
        s.sendto(data, (ADDRESS, PORT))
    except (ConnectionResetError, ConnectionAbortedError):
        close_socket(s)

while True:
    msg = input("Enter a message: ")
    write(msg.encode('utf-8'))

    data = read()
    if data != b"":
        print("Message Received:", data)

    if msg == "exit":
        break

close_socket(s)

Note that exact reason why your code is frozen is not because you set too high request.recv() buffer size. Here is explained What means buffer size in socket.recv(buffer_size)

This code will work until it'll receive an empty TCP message (if you'd print this empty message, it'd show b''):

while True:    
  data = self.request.recv(1024)
  if not data: break

And note, that there is no way to send empty TCP message. socket.send(b'') simply won't work.

Why? Because empty message is sent only when you type socket.close(), so your script will loop as long as you won't close your connection. As Hans L pointed out here are some good methods to end message.

That's the nature of TCP: the protocol fills up packets (lower layer being IP packets) and sends them. You can have some degree of control over the MTU (Maximum Transfer Unit).

In other words: you must devise a protocol that rides on top of TCP where your "payload delineation" is defined. By "payload delineation" I mean the way you extract the unit of message your protocol supports. This can be as simple as "every NULL terminated strings".

The answer by Larry Hastings has some great general advice about sockets, but there are a couple of mistakes as it pertains to how the recv(bufsize) method works in the Python socket module.

So, to clarify, since this may be confusing to others looking to this for help:

1. The bufsize param for the recv(bufsize) method is not optional. You'll get an error if you call recv() (without the param).
2. The bufferlen in recv(bufsize) is a maximum size. The recv will happily return fewer bytes if there are fewer available.

See the documentation for details.

Now, if you're receiving data from a client and want to know when you've received all of the data, you're probably going to have to add it to your protocol -- as Larry suggests. See this recipe for strategies for determining end of message.

As that recipe points out, for some protocols, the client will simply disconnect when it's done sending data. In those cases, your while True loop should work fine. If the client does not disconnect, you'll need to figure out some way to signal your content length, delimit your messages, or implement a timeout.

I'd be happy to try to help further if you could post your exact client code and a description of your test protocol.