I solved this by going into Site Manager -> selected the connection that Failed to retrieve directory listing
-> Switched to tab "Transfer Settings" and set "Transfer Mode" to "Active" instead of "Default". Also check if you are connected via VPN or anything similar, this can also interfere.