Is there a limit on number of tcp ip connections between machines on linux

Question

I have a very simple program written in 5 min that opens a sever socket and loops through the request and prints to the screen the bytes sent to it    I then tried to benchmark how many connections I can hammer it with to try to find out how many concurrent users I can support with this program   On another machine  where the network between them is not saturated  I created a simple program that goes into a loop and connects to the server machine and send the bytes  hello world     When the loop is 1000-3000 the client finishes with all requests sent  When the loop goes beyond 5000 it starts to have time outs after finish the first X number of requests  Why is this  I have made sure to close my socket in the loop    Can you only create so many connections within a certain period of time    Is this limit only applicable between the same machines and I need not worry about this in production where 5000  requests are all coming from different machines

User · Answer

Yep  the limit is set by the kernel  check out this thread on Stack Overflow for more details  Increasing the maximum number of tcp ip connections in linux

User · Answer

When looking for the max performance you run into a lot of issue and potential bottlenecks   Running a simple hello world test is not necessarily going to find them all   Possible limitations include      Kernel socket limitations  look in  proc sys net  for lots of kernel tuning   process limits   check out ulimit as others have stated here as your application grows in complexity  it may not have enough CPU power to keep up with the number of connections coming in   Use top to see if your CPU is maxed number of threads  I m not experienced with threading  but this may come into play in conjunction with the previous items

User · Answer

Is your server single-threaded  If so  what polling   multiplexing function are you using   Using select   does not work beyond the hard-coded maximum file descriptor limit set at compile-time  which is hopeless  normally 256  or a few more    poll   is better but you will end up with the scalability problem with a large number of FDs repopulating the set each time around the loop   epoll   should work well up to some other limit which you hit   10k connections should be easy enough to achieve  Use a recent ish  2 6 kernel   How many client machines did you use  Are you sure you didn t hit a client-side limit

User · Answer

The quick answer is 2 16 TCP ports  64K   The issues with system imposed limits is a configuration issue  already touched upon in previous comments   The internal implications to TCP is not so clear  to me    Each port requires memory for it s instantiation  goes onto a list and needs network buffers for data in transit   Given 64K TCP sessions the overhead for instances of the ports might be an issue on a 32-bit kernel  but not a 64-bit kernel  correction here gladly accepted    The lookup process with 64K sessions can slow things a bit and every packet hits the timer queues  which can also be problematic   Storage for in transit data can theoretically swell to the window size times ports  maybe 8 GByte    The issue with connection speed  mentioned above  is probably what you are seeing   TCP generally takes time to do things   However  it is not required   A TCP connect  transact and disconnect can be done very efficiently  check to see how the TCP sessions are created and closed    There are systems that pass tens of gigabits per second  so the packet level scaling should be OK   There are machines with plenty of physical memory  so that looks OK   The performance of the system  if carefully configured should be OK   The server side of things should scale in a similar fashion   I would be concerned about things like memory bandwidth   Consider an experiment where you login to the local host 10 000 times   Then type a character   The entire stack through user space would be engaged on each character   The active footprint would likely exceed the data cache size   Running through lots of memory can stress the VM system   The cost of context switches could approach a second   This is discussed in a variety of other threads  https   serverfault com questions 69524 im-designing-a-system-to-handle-10000-tcp-connections-per-second-what-problems

User · Answer

You might wanna check out  etc security limits conf

User · Answer

There is a limit  yes   See ulimit   Also you need to consider the TIMED WAIT state   Once a TCP socket is closed  by default  the port remains occupied in TIMED WAIT status for 2 minutes   This value is tunable   This will also  run you out of sockets  even though they are closed   Run netstat to see the TIMED WAIT stuff in action   P S  The reason for TIMED WAIT is to handle the case of packets arriving after the socket is closed   This can happen because packets are delayed or the other side just doesn t know that the socket has been closed yet   This allows the OS to silently drop those packets without a chance of  infecting  a different  unrelated socket connection

[linux] Is there a limit on number of tcp/ip connections between machines on linux?

Examples related to linux

Examples related to sockets

Examples related to tcp