In my case, restarting my router which I used to connect to the internet worked. I think too much of connections were going from the same IP Address and when I restarted my router, possibly a new IP was assigned and now everything works fine, and passive mode gives good speed in directory listing.