Virtualenv is your friend
Even if you want to add a package to your primary install, it's still best to do it in a virtual environment first, to ensure compatibility with your other packages. However, if you get familiar with virtualenv, you'll probably find there's really no reason to install anything in your base install.