I think you should use SO_LINGER options (with timeout 0). In this case, you connection will close immediately after closing your program; and next restart will be able to bind again.
example:
linger lin;
lin.l_onoff = 0;
lin.l_linger = 0;
setsockopt(fd, SOL_SOCKET, SO_LINGER, (const char *)&lin, sizeof(int));
see definition: http://man7.org/linux/man-pages/man7/socket.7.html
SO_LINGER
Sets or gets the SO_LINGER option. The argument is a linger
structure.
struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};
When enabled, a close(2) or shutdown(2) will not return until
all queued messages for the socket have been successfully sent
or the linger timeout has been reached. Otherwise, the call
returns immediately and the closing is done in the background.
When the socket is closed as part of exit(2), it always
lingers in the background.
More about SO_LINGER: TCP option SO_LINGER (zero) - when it's required