Okay, my problem wasn't the port, but the binding address. My server has an internal address (10.0.0.4) and an external address (52.175.223.XX). When I tried connecting with:
$sock = @stream_socket_server('tcp://52.175.223.XX:123', $errNo, $errStr, STREAM_SERVER_BIND|STREAM_SERVER_LISTEN);
It failed because the local socket was 10.0.0.4 and not the external 52.175.223.XX. You can checkout the local available interfaces with sudo ifconfig
.