[python] What is the purpose of "pip install --user ..."?

From pip install --help:

--user  Install to the Python user install directory for your platform. 
        Typically ~/.local/, or %APPDATA%\Python on Windows. 
        (See the Python documentation for site.USER_BASE for full details.)

The documentation for site.USER_BASE is a terrifying wormhole of interesting *NIX subject matter that I don't understand.

What is the purpose of --user in plain english? Why would intalling the package to ~/.local/ matter? Why not just put an executable somewhere in my $PATH?

Without Virtual Environments

pip <command> --user changes the scope of the current pip command to work on the current user account's local python package install location, rather than the system-wide package install location, which is the default.

• See User Installs in the PIP User Guide.

This only really matters on a multi-user machine. Anything installed to the system location will be visible to all users, so installing to the user location will keep that package installation separate from other users (they will not see it, and would have to install it themselves separately to use it). Because there can be version conflicts, installing a package with dependencies needed by other packages can cause problems, so it's best not to push all packages a given user uses to the system install location.

• If it is a single-user machine, there is little or no difference to installing to the --user location. It will be installed to a different folder, that may or may not need to be added to the path, depending on the package and how it's used (many packages install command-line tools that must be on the path to run from a shell).
• If it is a multi-user machine, --user is preferred to using root/sudo or requiring administrator installation and affecting the Python environment of every user, except in cases of general packages that the administrator wants to make available to all users by default.
  • Note: Per comments, on most Unix/Linux installs it has been pointed out that system installs should use the general package manager, such as apt, rather than pip.

With Virtual Environments

• See more about installing packages with virtual environments in the Python Packaging documentation.
• Read about how to create and use virtual environments, and the venv command in the Python VENV docs.

The --user option in an active venv/virtualenv environment will install to the local user python location (same as without a virtual environment).

Packages are installed to the virtual environment by default, but if you use --user it will force it to install outside the virtual environments, in the users python script directory (in Windows, this currently is c:\users\<username>\appdata\roaming\python\python37\scripts for me with Python 3.7).

However, you won't be able to access a system or user install from within virtual environment (even if you used --user while in a virtual environment).

If you install a virtual environment with the --system-site-packages argument, you will have access to the system script folder for python. I believe this included the user python script folder as well, but I'm unsure. However, there may be unintended consequences for this and it is not the intended way to use virtual environments.

Location of the Python System and Local User Install Folders

You can find the location of the user install folder for python with python -m site --user-base. I'm finding conflicting information in Q&A's, the documentation and actually using this command on my PC as to what the defaults are, but they are underneath the user home directory (~ shortcut in *nix, and c:\users\<username> typically for Windows).

Other Details

The --user option is not a valid for every command. For example pip uninstall will find and uninstall packages wherever they were installed (in the user folder, virtual environment folder, etc.) and the --user option is not valid.

Things installed with pip install --user will be installed in a local location that will only be seen by the current user account, and will not require root access (on *nix) or administrator access (on Windows).

The --user option modifies all pip commands that accept it to see/operate on the user install folder, so if you use pip list --user it will only show you packages installed with pip install --user.

Just a warning:

According to this issue, --user is currently not valid inside a virtual env's pip, since a user location doesn't really make sense for a virtual environment.

So do not use pip install --user some_pkg inside a virtual environment, otherwise, virtual environment's pip will be confused. See this answer for more details.

--user installs in site.USER_SITE.

For my case, it was /Users/.../Library/Python/2.7/bin. So I have added that to my PATH (in ~/.bash_profile file):

export PATH=$PATH:/Users/.../Library/Python/2.7/bin

Why not just put an executable somewhere in my $PATH

~/.local/bin directory is theoretically expected to be in your $PATH.

According to these people it's a bug not adding it in the $PATH when using systemd.

This answer explains it more extensively.

But even if your distro includes the ~/.local/bin directory to the $PATH, it might be in the following form (inside ~/.profile):

if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

which would require you to logout and login again, if the directory was not there before.

Best way to is install virtualenv and not require the --user confusion. You will get more flexibility and not worry about clobbering the different python versions and projects everytime you pip install a package.

https://virtualenv.pypa.io/en/stable/

Other answers mention site.USER_SITE as where Python packages get placed. If you're looking for binaries, these go in {site.USER_BASE}/bin.

If you want to add this directory to your shell's search path, use:

export PATH="${PATH}:$(python3 -c 'import site; print(site.USER_BASE)')/bin"

On macOS, the reason for using the --user flag is to make sure we don't corrupt the libraries the OS relies on. A conservative approach for many macOS users is to avoid installing or updating pip with a command that requires sudo. Thus, this includes installing to /usr/local/bin...

Ref: Installing python for Neovim (https://github.com/zchee/deoplete-jedi/wiki/Setting-up-Python-for-Neovim)

I'm not all clear why installing into /usr/local/bin is a risk on a Mac given the fact that the system only relies on python binaries in /Library/Frameworks/ and /usr/bin. I suspect it's because as noted above, installing into /usr/local/bin requires sudo which opens the door to making a costly mistake with the system libraries. Thus, installing into ~/.local/bin is a sure fire way to avoid this risk.

Ref: Using python on a Mac (https://docs.python.org/2/using/mac.html)

Finally, to the degree there is a benefit of installing packages into the /usr/local/bin, I wonder if it makes sense to change the owner of the directory from root to user? This would avoid having to use sudo while still protecting against making system-dependent changes.* Is this a security default a relic of how Unix systems were more often used in the past (as servers)? Or at minimum, just a good way to go for Mac users not hosting a server?

*Note: Mac's System Integrity Protection (SIP) feature also seems to protect the user from changing the system-dependent libraries.

- E