How do SO REUSEADDR and SO REUSEPORT differ

Question

The man pages and programmer documentations for the socket options SO REUSEADDR and SO REUSEPORT are different for different operating systems and often highly confusing  Some operating systems don t even have the option SO REUSEPORT  The WEB is full of contradicting information regarding this subject and often you can find information that is only true for one socket implementation of a specific operating system  which may not even be explicitly mentioned in the text   So how exactly is SO REUSEADDR different than SO REUSEPORT   Are systems without SO REUSEPORT more limited   And what exactly is the expected behavior if I use either one on different operating systems

User · Answer

Mecki s answer is absolutly perfect  but it s worth adding that FreeBSD also supports SO REUSEPORT LB  which mimics Linux  SO REUSEPORT behaviour - it balances the load  see setsockopt 2

User · Answer

Welcome to the wonderful world of portability    or rather the lack of it  Before we start analyzing these two options in detail and take a deeper look how different operating systems handle them  it should be noted that the BSD socket implementation is the mother of all socket implementations  Basically all other systems copied the BSD socket implementation at some point in time  or at least its interfaces  and then started evolving it on their own  Of course the BSD socket implementation was evolved as well at the same time and thus systems that copied it later got features that were lacking in systems that copied it earlier  Understanding the BSD socket implementation is the key to understanding all other socket implementations  so you should read about it even if you don t care to ever write code for a BSD system  There are a couple of basics you should know before we look at these two options  A TCP UDP connection is identified by a tuple of five values    lt protocol gt    lt src addr gt    lt src port gt    lt dest addr gt    lt dest port gt   Any unique combination of these values identifies a connection  As a result  no two connections can have the same five values  otherwise the system would not be able to distinguish these connections any longer  The protocol of a socket is set when a socket is created with the socket   function  The source address and port are set with the bind   function  The destination address and port are set with the connect   function  Since UDP is a connectionless protocol  UDP sockets can be used without connecting them  Yet it is allowed to connect them and in some cases very advantageous for your code and general application design  In connectionless mode  UDP sockets that were not explicitly bound when data is sent over them for the first time are usually automatically bound by the system  as an unbound UDP socket cannot receive any  reply  data  Same is true for an unbound TCP socket  it is automatically bound before it will be connected  If you explicitly bind a socket  it is possible to bind it to port 0  which means  quot any port quot   Since a socket cannot really be bound to all existing ports  the system will have to choose a specific port itself in that case  usually from a predefined  OS specific range of source ports   A similar wildcard exists for the source address  which can be  quot any address quot   0 0 0 0 in case of IPv4 and    in case of IPv6   Unlike in case of ports  a socket can really be bound to  quot any address quot  which means  quot all source IP addresses of all local interfaces quot   If the socket is connected later on  the system has to choose a specific source IP address  since a socket cannot be connected and at the same time be bound to any local IP address  Depending on the destination address and the content of the routing table  the system will pick an appropriate source address and replace the  quot any quot  binding with a binding to the chosen source IP address  By default  no two sockets can be bound to the same combination of source address and source port  As long as the source port is different  the source address is actually irrelevant  Binding socketA to ipA portA and socketB to ipB portB is always possible if ipA    ipB holds true  even when portA    portB  E g  socketA belongs to a FTP server program and is bound to 192 168 0 1 21 and socketB belongs to another FTP server program and is bound to 10 0 0 1 21  both bindings will succeed  Keep in mind  though  that a socket may be locally bound to  quot any address quot   If a socket is bound to 0 0 0 0 21  it is bound to all existing local addresses at the same time and in that case no other socket can be bound to port 21  regardless which specific IP address it tries to bind to  as 0 0 0 0 conflicts with all existing local IP addresses  Anything said so far is pretty much equal for all major operating system  Things start to get OS specific when address reuse comes into play  We start with BSD  since as I said above  it is the mother of all socket implementations  BSD SO REUSEADDR If SO REUSEADDR is enabled on a socket prior to binding it  the socket can be successfully bound unless there is a conflict with another socket bound to exactly the same combination of source address and port  Now you may wonder how is that any different than before  The keyword is  quot exactly quot   SO REUSEADDR mainly changes the way how wildcard addresses   quot any IP address quot   are treated when searching for conflicts  Without SO REUSEADDR  binding socketA to 0 0 0 0 21 and then binding socketB to 192 168 0 1 21 will fail  with error EADDRINUSE   since 0 0 0 0 means  quot any local IP address quot   thus all local IP addresses are considered in use by this socket and this includes 192 168 0 1  too  With SO REUSEADDR it will succeed  since 0 0 0 0 and 192 168 0 1 are not exactly the same address  one is a wildcard for all local addresses and the other one is a very specific local address  Note that the statement above is true regardless in which order socketA and socketB are bound  without SO REUSEADDR it will always fail  with SO REUSEADDR it will always succeed  To give you a better overview  let s make a table here and list all possible combinations   SO REUSEADDR       socketA        socketB       Result ---------------------------------------------------------------------   ON OFF       192 168 0 1 21   192 168 0 1 21    Error  EADDRINUSE    ON OFF       192 168 0 1 21      10 0 0 1 21    OK   ON OFF          10 0 0 1 21   192 168 0 1 21    OK    OFF             0 0 0 0 21   192 168 1 0 21    Error  EADDRINUSE     OFF         192 168 1 0 21       0 0 0 0 21    Error  EADDRINUSE     ON              0 0 0 0 21   192 168 1 0 21    OK    ON          192 168 1 0 21       0 0 0 0 21    OK   ON OFF           0 0 0 0 21       0 0 0 0 21    Error  EADDRINUSE   The table above assumes that socketA has already been successfully bound to the address given for socketA  then socketB is created  either gets SO REUSEADDR set or not  and finally is bound to the address given for socketB  Result is the result of the bind operation for socketB  If the first column says ON OFF  the value of SO REUSEADDR is irrelevant to the result  Okay  SO REUSEADDR has an effect on wildcard addresses  good to know  Yet that isn t it s only effect it has  There is another well known effect which is also the reason why most people use SO REUSEADDR in server programs in the first place  For the other important use of this option we have to take a deeper look on how the TCP protocol works  A socket has a send buffer and if a call to the send   function succeeds  it does not mean that the requested data has actually really been sent out  it only means the data has been added to the send buffer  For UDP sockets  the data is usually sent pretty soon  if not immediately  but for TCP sockets  there can be a relatively long delay between adding data to the send buffer and having the TCP implementation really send that data  As a result  when you close a TCP socket  there may still be pending data in the send buffer  which has not been sent yet but your code considers it as sent  since the send   call succeeded  If the TCP implementation was closing the socket immediately on your request  all of this data would be lost and your code wouldn t even know about that  TCP is said to be a reliable protocol and losing data just like that is not very reliable  That s why a socket that still has data to send will go into a state called TIME WAIT when you close it  In that state it will wait until all pending data has been successfully sent or until a timeout is hit  in which case the socket is closed forcefully  At most  the amount of time the kernel will wait before it closes the socket  regardless if it still has data in flight or not  is called the Linger Time  The Linger Time is globally configurable on most systems and by default rather long  two minutes is a common value you will find on many systems   It is also configurable per socket using the socket option SO LINGER which can be used to make the timeout shorter or longer  and even to disable it completely  Disabling it completely is a very bad idea  though  since closing a TCP socket gracefully is a slightly complex process and involves sending forth and back a couple of packets  as well as resending those packets in case they got lost  and this whole close process is also limited by the Linger Time  If you disable lingering  your socket may not only lose data in flight  it is also always closed forcefully instead of gracefully  which is usually not recommended  The details about how a TCP connection is closed gracefully are beyond the scope of this answer  if you want to learn more about  I recommend you have a look at this page  And even if you disabled lingering with SO LINGER  if your process dies without explicitly closing the socket  BSD  and possibly other systems  will linger nonetheless  ignoring what you have configured  This will happen for example if your code just calls exit    pretty common for tiny  simple server programs  or the process is killed by a signal  which includes the possibility that it simply crashes because of an illegal memory access   So there is nothing you can do to make sure a socket will never linger under all circumstances  The question is  how does the system treat a socket in state TIME WAIT  If SO REUSEADDR is not set  a socket in state TIME WAIT is considered to still be bound to the source address and port and any attempt to bind a new socket to the same address and port will fail until the socket has really been closed  which may take as long as the configured Linger Time  So don t expect that you can rebind the source address of a socket immediately after closing it  In most cases this will fail  However  if SO REUSEADDR is set for the socket you are trying to bind  another socket bound to the same address and port in state TIME WAIT is simply ignored  after all its already  quot half dead quot   and your socket can bind to exactly the same address without any problem  In that case it plays no role that the other socket may have exactly the same address and port  Note that binding a socket to exactly the same address and port as a dying socket in TIME WAIT state can have unexpected  and usually undesired  side effects in case the other socket is still  quot at work quot   but that is beyond the scope of this answer and fortunately those side effects are rather rare in practice  There is one final thing you should know about SO REUSEADDR  Everything written above will work as long as the socket you want to bind to has address reuse enabled  It is not necessary that the other socket  the one which is already bound or is in a TIME WAIT state  also had this flag set when it was bound  The code that decides if the bind will succeed or fail only inspects the SO REUSEADDR flag of the socket fed into the bind   call  for all other sockets inspected  this flag is not even looked at  SO REUSEPORT SO REUSEPORT is what most people would expect SO REUSEADDR to be  Basically  SO REUSEPORT allows you to bind an arbitrary number of sockets to exactly the same source address and port as long as all prior bound sockets also had SO REUSEPORT set before they were bound  If the first socket that is bound to an address and port does not have SO REUSEPORT set  no other socket can be bound to exactly the same address and port  regardless if this other socket has SO REUSEPORT set or not  until the first socket releases its binding again  Unlike in case of SO REUESADDR the code handling SO REUSEPORT will not only verify that the currently bound socket has SO REUSEPORT set but it will also verify that the socket with a conflicting address and port had SO REUSEPORT set when it was bound  SO REUSEPORT does not imply SO REUSEADDR  This means if a socket did not have SO REUSEPORT set when it was bound and another socket has SO REUSEPORT set when it is bound to exactly the same address and port  the bind fails  which is expected  but it also fails if the other socket is already dying and is in TIME WAIT state  To be able to bind a socket to the same addresses and port as another socket in TIME WAIT state requires either SO REUSEADDR to be set on that socket or SO REUSEPORT must have been set on both sockets prior to binding them  Of course it is allowed to set both  SO REUSEPORT and SO REUSEADDR  on a socket  There is not much more to say about SO REUSEPORT other than that it was added later than SO REUSEADDR  that s why you will not find it in many socket implementations of other systems  which  quot forked quot  the BSD code before this option was added  and that there was no way to bind two sockets to exactly the same socket address in BSD prior to this option  Connect   Returning EADDRINUSE  Most people know that bind   may fail with the error EADDRINUSE  however  when you start playing around with address reuse  you may run into the strange situation that connect   fails with that error as well  How can this be  How can a remote address  after all that s what connect adds to a socket  be already in use  Connecting multiple sockets to exactly the same remote address has never been a problem before  so what s going wrong here  As I said on the very top of my reply  a connection is defined by a tuple of five values  remember  And I also said  that these five values must be unique otherwise the system cannot distinguish two connections any longer  right  Well  with address reuse  you can bind two sockets of the same protocol to the same source address and port  That means three of those five values are already the same for these two sockets  If you now try to connect both of these sockets also to the same destination address and port  you would create two connected sockets  whose tuples are absolutely identical  This cannot work  at least not for TCP connections  UDP connections are no real connections anyway   If data arrived for either one of the two connections  the system could not tell which connection the data belongs to  At least the destination address or destination port must be different for either connection  so that the system has no problem to identify to which connection incoming data belongs to  So if you bind two sockets of the same protocol to the same source address and port and try to connect them both to the same destination address and port  connect   will actually fail with the error EADDRINUSE for the second socket you try to connect  which means that a socket with an identical tuple of five values is already connected  Multicast Addresses Most people ignore the fact that multicast addresses exist  but they do exist  While unicast addresses are used for one-to-one communication  multicast addresses are used for one-to-many communication  Most people got aware of multicast addresses when they learned about IPv6 but multicast addresses also existed in IPv4  even though this feature was never widely used on the public Internet  The meaning of SO REUSEADDR changes for multicast addresses as it allows multiple sockets to be bound to exactly the same combination of source multicast address and port  In other words  for multicast addresses SO REUSEADDR behaves exactly as SO REUSEPORT for unicast addresses  Actually  the code treats SO REUSEADDR and SO REUSEPORT identically for multicast addresses  that means you could say that SO REUSEADDR implies SO REUSEPORT for all multicast addresses and the other way round  FreeBSD OpenBSD NetBSD All these are rather late forks of the original BSD code  that s why they all three offer the same options as BSD and they also behave the same way as in BSD  macOS  MacOS X  At its core  macOS is simply a BSD-style UNIX named  quot Darwin quot   based on a rather late fork of the BSD code  BSD 4 3   which was then later on even re-synchronized with the  at that time current  FreeBSD 5 code base for the Mac OS 10 3 release  so that Apple could gain full POSIX compliance  macOS is POSIX certified   Despite having a microkernel at its core   quot Mach quot    the rest of the kernel   quot XNU quot   is basically just a BSD kernel  and that s why macOS offers the same options as BSD and they also behave the same way as in BSD  iOS   watchOS   tvOS iOS is just a macOS fork with a slightly modified and trimmed kernel  somewhat stripped down user space toolset and a slightly different default framework set  watchOS and tvOS are iOS forks  that are stripped down even further  especially watchOS   To my best knowledge they all behave exactly as macOS does  Linux Linux  lt  3 9 Prior to Linux 3 9  only the option SO REUSEADDR existed  This option behaves generally the same as in BSD with two important exceptions   As long as a listening  server  TCP socket is bound to a specific port  the SO REUSEADDR option is entirely ignored for all sockets targeting that port  Binding a second socket to the same port is only possible if it was also possible in BSD without having SO REUSEADDR set  E g  you cannot bind to a wildcard address and then to a more specific one or the other way round  both is possible in BSD if you set SO REUSEADDR  What you can do is you can bind to the same port and two different non-wildcard addresses  as that s always allowed  In this aspect Linux is more restrictive than BSD   The second exception is that for client sockets  this option behaves exactly like SO REUSEPORT in BSD  as long as both had this flag set before they were bound  The reason for allowing that was simply that it is important to be able to bind multiple sockets to exactly to the same UDP socket address for various protocols and as there used to be no SO REUSEPORT prior to 3 9  the behavior of SO REUSEADDR was altered accordingly to fill that gap  In that aspect Linux is less restrictive than BSD    Linux  gt   3 9 Linux 3 9 added the option SO REUSEPORT to Linux as well  This option behaves exactly like the option in BSD and allows binding to exactly the same address and port number as long as all sockets have this option set prior to binding them  Yet  there are still two differences to SO REUSEPORT on other systems   To prevent  quot port hijacking quot   there is one special limitation  All sockets that want to share the same address and port combination must belong to processes that share the same effective user ID  So one user cannot  quot steal quot  ports of another user  This is some special magic to somewhat compensate for the missing SO EXCLBIND SO EXCLUSIVEADDRUSE flags   Additionally the kernel performs some  quot special magic quot  for SO REUSEPORT sockets that isn t found in other operating systems  For UDP sockets  it tries to distribute datagrams evenly  for TCP listening sockets  it tries to distribute incoming connect requests  those accepted by calling accept     evenly across all the sockets that share the same address and port combination  Thus an application can easily open the same port in multiple child processes and then use SO REUSEPORT to get a very inexpensive load balancing    Android Even though the whole Android system is somewhat different from most Linux distributions  at its core works a slightly modified Linux kernel  thus everything that applies to Linux should apply to Android as well  Windows Windows only knows the SO REUSEADDR option  there is no SO REUSEPORT  Setting SO REUSEADDR on a socket in Windows behaves like setting SO REUSEPORT and SO REUSEADDR on a socket in BSD  with one exception  Prior to Windows 2003  a socket with SO REUSEADDR could always been bound to exactly the same source address and port as an already bound socket  even if the other socket did not have this option set when it was bound  This behavior allowed an application  quot to steal quot  the connected port of another application  Needless to say that this has major security implications  Microsoft realized that and added another important socket option  SO EXCLUSIVEADDRUSE  Setting SO EXCLUSIVEADDRUSE on a socket makes sure that if the binding succeeds  the combination of source address and port is owned exclusively by this socket and no other socket can bind to them  not even if it has SO REUSEADDR set  This default behavior was changed first in Windows 2003  Microsoft calls that  quot Enhanced Socket Security quot   funny name for a behavior that is default on all other major operating systems   For more details just visit this page  There are three tables  The first one shows the classic behavior  still in use when using compatibility modes    the second one shows the behavior of Windows 2003 and up when the bind   calls are made by the same user  and the third one when the bind   calls are made by different users  Solaris Solaris is the successor of SunOS  SunOS was originally based on a fork of BSD  SunOS 5 and later was based on a fork of SVR4  however SVR4 is a merge of BSD  System V  and Xenix  so up to some degree Solaris is also a BSD fork  and a rather early one  As a result Solaris only knows SO REUSEADDR  there is no SO REUSEPORT  The SO REUSEADDR behaves pretty much the same as it does in BSD  As far as I know there is no way to get the same behavior as SO REUSEPORT in Solaris  that means it is not possible to bind two sockets to exactly the same address and port  Similar to Windows  Solaris has an option to give a socket an exclusive binding  This option is named SO EXCLBIND  If this option is set on a socket prior to binding it  setting SO REUSEADDR on another socket has no effect if the two sockets are tested for an address conflict  E g  if socketA is bound to a wildcard address and socketB has SO REUSEADDR enabled and is bound to a non-wildcard address and the same port as socketA  this bind will normally succeed  unless socketA had SO EXCLBIND enabled  in which case it will fail regardless the SO REUSEADDR flag of socketB  Other Systems In case your system is not listed above  I wrote a little test program that you can use to find out how your system handles these two options  Also if you think my results are wrong  please first run that program before posting any comments and possibly making false claims  All that the code requires to build is a bit POSIX API  for the network parts  and a C99 compiler  actually most non-C99 compiler will work as well as long as they offer inttypes h and stdbool h  e g  gcc supported both long before offering full C99 support   All that the program needs to run is that at least one interface in your system  other than the local interface  has an IP address assigned and that a default route is set which uses that interface  The program will gather that IP address and use it as the second  quot specific address quot   It tests all possible combinations you can think of   TCP and UDP protocol Normal sockets  listen  server  sockets  multicast sockets SO REUSEADDR set on socket1  socket2  or both sockets SO REUSEPORT set on socket1  socket2  or both sockets All address combinations you can make out of 0 0 0 0  wildcard   127 0 0 1  specific address   and the second specific address found at your primary interface  for multicast it s just 224 1 2 3 in all tests   and prints the results in a nice table  It will also work on systems that don t know SO REUSEPORT  in which case this option is simply not tested  What the program cannot easily test is how SO REUSEADDR acts on sockets in TIME WAIT state as it s very tricky to force and keep a socket in that state  Fortunately most operating systems seems to simply behave like BSD here and most of the time programmers can simply ignore the existence of that state  Here s the code  I cannot include it here  answers have a size limit and the code would push this reply over the limit

[linux] How do SO_REUSEADDR and SO_REUSEPORT differ?

Examples related to linux

Examples related to windows

Examples related to sockets

Examples related to unix

Examples related to portability