Python Socket Receive Large Amount of Data

Question

When I try to receive larger amounts of data it gets cut off and I have to press enter to get the rest of the data  At first I was able to increase it a little bit but it still won t receive all of it  As you can see I have increased the buffer on the conn recv   but it still doesn t get all of the data  It cuts it off at a certain point  I have to press enter on my raw input in order to receive the rest of the data  Is there anyway I can get all of the data at once  Here s the code    port   7777 sock   socket socket socket AF INET  socket SOCK STREAM  sock bind   0 0 0 0   port   sock listen 1  print   Listening on port    str port   while 1      conn  sock addr   sock accept       print  accepted connection from   sock addr     while 1          command   raw input  shell gt                 conn send command                  data   conn recv 8000                  if not data  break                 print data      conn close

User · Answer

Disclaimer  There are very rare cases in which you really need to do this  If possible use an existing application layer protocol or define your own eg  precede each message with a fixed length integer indicating the length of data that follows or terminate each message with a   n  character   Adam Rosenfield s answer does a really good job at explaining that   With that said  there is a way to read all of the data available on a socket  However  it is a bad idea to rely on this kind of communication as it introduces the risk of loosing data  Use this solution with extreme caution and only after reading the explanation below   def recvall sock       BUFF SIZE   4096     data   bytearray       while True          packet   sock recv BUFF SIZE          if not packet     Important               break         data extend packet      return data   Now the if not packet  line is absolutely critical  Many answers here suggested using a condition like if len packet   lt  BUFF SIZE  which is broken and will most likely cause you to close your connection prematurely and loose data  It wrongly assumes that one send on one end of a TCP socket corresponds to one receive of sent number of bytes on the other end  It does not  There is a very good chance that sock recv BUFF SIZE  will return a chunk smaller than BUFF SIZE even if there s still data waiting to be received  There is a good explanation of the issue here and here   By using the above solution you are still risking data loss if the other end of the connection is writing data slower than you are reading  You may just simply consume all data on your end and exit when more is on the way  There are ways around it that require the use of concurrent programming  but that s another topic of its own

User · Answer

You can use it as  data   recvall sock   def recvall sock       BUFF SIZE   4096   4 KiB     data   b       while True          part   sock recv BUFF SIZE          data    part         if len part   lt  BUFF SIZE                either 0 or end of data             break     return data

User · Answer

Modifying Adam Rosenfield s code   import sys   def send msg sock  msg       size of package   sys getsizeof msg      package   str size of package       msg  Create our package size     message     sock sendall package   def recv msg sock       try          header   sock recv 2  Magic  small number to begin with          while     not in header              header    sock recv 2   Keep looping  picking up two bytes each time          size of package  separator  message fragment   header partition              message   sock recv int size of package           full message   message fragment   message         return full message      except OverflowError          return  OverflowError       except          print  Unexpected error    sys exc info   0          raise   I would  however  heavily encourage using the original approach

User · Answer

You may need to call conn recv   multiple times to receive all the data   Calling it a single time is not guaranteed to bring in all the data that was sent  due to the fact that TCP streams don t maintain frame boundaries  i e  they only work as a stream of raw bytes  not a structured stream of messages    See this answer for another description of the issue   Note that this means you need some way of knowing when you have received all of the data   If the sender will always send exactly 8000 bytes  you could count the number of bytes you have received so far and subtract that from 8000 to know how many are left to receive  if the data is variable-sized  there are various other methods that can be used  such as having the sender send a number-of-bytes header before sending the message  or if it s ASCII text that is being sent you could look for a newline or NUL character

User · Answer

A variation using a generator function  which I consider more pythonic    def recvall sock  buffer size 4096       buf   sock recv buffer size      while buf          yield buf         if len buf   lt  buffer size  break         buf   sock recv buffer size        with socket create connection  host  port   as sock      sock sendall command      response   b   join recvall sock

User · Answer

For anyone else who s looking for an answer in cases where you don t know the length of the packet prior  Here s a simple solution that reads 4096 bytes at a time and stops when less than 4096 bytes were received  However  it will not work in cases where the total length of the packet received is exactly 4096 bytes - then it will call recv   again and hang   def recvall sock       data   b       bufsize   4096     while True          packet   sock recv bufsize          data    packet         if len packet   lt  bufsize              break     return data

User · Answer

The accepted answer is fine but it will be really slow with big files -string is an immutable class this means more objects are created every time you use the   sign  using list as a stack structure will be more efficient   This should work better   while True       chunk   s recv 10000      if not chunk           break     fragments append chunk   print    join fragments

User · Answer

You can do it using Serialization  from socket import   from json import dumps  loads  def recvall conn       data          while True      try          data   conn recv 1024          return json loads data      except ValueError          continue  def sendall conn       conn sendall json dumps data     NOTE  If you want to shara a file using code above you need to encode   decode it into base64

User · Answer

I think this question has been pretty well answered  but I just wanted to add a method using Python 3 8 and the new assignment expression  walrus operator  since it is stylistically simple   import socket  host    127 0 0 1  port   31337 s   socket socket socket AF INET  socket SOCK STREAM  s bind  host port   s listen   con  addr   s accept   msg list       while  walrus msg    con recv 3      b  r n       msg list append walrus msg   print msg list    In this case  3 bytes are received from the socket and immediately assigned to walrus msg  Once the socket receives a b  r n  it breaks the loop  walrus msg are added to a msg list and printed after the loop breaks  This script is basic but was tested and works with a telnet session   NOTE  The parenthesis around the  walrus msg    con recv 3   are needed  Without this  while walrus msg    con recv 3     b  r n   evaluates walrus msg to True instead of the actual data on the socket

User · Answer

Most of the answers describe some sort of recvall   method  If your bottleneck when receiving data is creating the byte array in a for loop  I benchmarked three approaches of allocating the received data in the recvall   method   Byte string method   arr   b   while len arr   lt  msg len      arr    sock recv max msg size    List method   fragments      while True       chunk   sock recv max msg size      if not chunk           break     fragments append chunk  arr   b   join fragments    Pre-allocated bytearray method   arr   bytearray msg len  pos   0 while pos  lt  msg len      arr pos pos max msg size    sock recv max msg size      pos    max msg size   Results

User · Answer

TCP IP is a stream-based protocol  not a message-based protocol   There s no guarantee that every send   call by one peer results in a single recv   call by the other peer receiving the exact data sent   it might receive the data piece-meal  split across multiple recv   calls  due to packet fragmentation   You need to define your own message-based protocol on top of TCP in order to differentiate message boundaries   Then  to read a message  you continue to call recv   until you ve read an entire message or an error occurs   One simple way of sending a message is to prefix each message with its length   Then to read a message  you first read the length  then you read that many bytes   Here s how you might do that   def send msg sock  msg         Prefix each message with a 4-byte length  network byte order      msg   struct pack   gt I   len msg     msg     sock sendall msg   def recv msg sock         Read message length and unpack it into an integer     raw msglen   recvall sock  4      if not raw msglen          return None     msglen   struct unpack   gt I   raw msglen  0        Read the message data     return recvall sock  msglen   def recvall sock  n         Helper function to recv n bytes or return None if EOF is hit     data   bytearray       while len data   lt  n          packet   sock recv n - len data           if not packet              return None         data extend packet      return data   Then you can use the send msg and recv msg functions to send and receive whole messages  and they won t have any problems with packets being split or coalesced on the network level

[python] Python Socket Receive Large Amount of Data

Examples related to python

Examples related to sockets