[python] tell pip to install the dependencies of packages listed in a requirement file

Developing a Django web app, I have a list of packages I need to install in a virtualenv. Say:

Django==1.3.1
--extra-index-url=http://dist.pinaxproject.com/dev/
Pinax==0.9b1.dev10
git+git://github.com/pinax/pinax-theme-bootstrap.git@cff4f5bbe9f87f0c67ee9ada9aa8ae82978f9890
# and other packages

Initially I installed them manually, one by one, along the development. This installed the required dependencies and I finally used pip freeze before deploying the app.

Problem is, as I upgraded some packages, some dependencies are no longer used nor required but they keep being listed by pip freeze.

Now, I'd like to set up a new virtualenv this way:

  • put the needed packages (without their dependencies) in a requirement file,
    like manual-requirements.txt
  • install them with their dependencies
    pip install -r manual-requirement.txt (? problem, this does not install the dependencies)
  • freeze the full virtualenv
    pip freeze -r manual-requirements.txt > full-requirements.txt
    and deploy.

Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I'd like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.

edit: actually, pip does install dependencies not explicitly listed in the requirement file, even if the documentation tells us that such files are flat. I was wrong about which dependencies i expected to be installed. I'll let this question for anyone in doubt about pip not installing all dependencies.

This question is related to python pip requirements.txt

The answer is


As @Ming mentioned:

pip install -r file.txt

Here's a simple line to force update all dependencies:

while read -r package; do pip install --upgrade --force-reinstall $package;done < pipfreeze.txt

Extending Piotr's answer, if you also need a way to figure what to put in requirements.in, you can first use pip-chill to find the minimal set of required packages you have. By combining these tools, you can show the dependency reason why each package is installed. The full cycle looks like this:

  1. Create virtual environment:
    $ python3 -m venv venv
  2. Activate it:
    $ . venv/bin/activate
  3. Install newest version of pip, pip-tools and pip-chill:
    (venv)$ pip install --upgrade pip
    (venv)$ pip install pip-tools pip-chill
  4. Build your project, install more pip packages, etc, until you want to save...
  5. Extract minimal set of packages (ie, top-level without dependencies):
    (venv)$ pip-chill --no-version > requirements.in
  6. Compile list of all required packages (showing dependency reasons):
    (venv)$ pip-compile requirements.in
  7. Make sure the current installation is synchronized with the list:
    (venv)$ pip-sync

Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I'd like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.

That's what pip-tools package is for (from https://github.com/jazzband/pip-tools):

Installation

$ pip install --upgrade pip  # pip-tools needs pip==6.1 or higher (!)
$ pip install pip-tools

Example usage for pip-compile

Suppose you have a Flask project, and want to pin it for production. Write the following line to a file:

# requirements.in
Flask

Now, run pip-compile requirements.in:

$ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# Make changes in requirements.in, then run this to update:
#
#    pip-compile requirements.in
#
flask==0.10.1
itsdangerous==0.24        # via flask
jinja2==2.7.3             # via flask
markupsafe==0.23          # via jinja2
werkzeug==0.10.4          # via flask

And it will produce your requirements.txt, with all the Flask dependencies (and all underlying dependencies) pinned. Put this file under version control as well and periodically re-run pip-compile to update the packages.

Example usage for pip-sync

Now that you have a requirements.txt, you can use pip-sync to update your virtual env to reflect exactly what's in there. Note: this will install/upgrade/uninstall everything necessary to match the requirements.txt contents.

$ pip-sync
Uninstalling flake8-2.4.1:
  Successfully uninstalled flake8-2.4.1
Collecting click==4.1
  Downloading click-4.1-py2.py3-none-any.whl (62kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 65kB 1.8MB/s
  Found existing installation: click 4.0
    Uninstalling click-4.0:
      Successfully uninstalled click-4.0
Successfully installed click-4.1

simplifily, use:

pip install -r requirement.txt

it can install all listed in requirement file.


Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to pip

How to fix error "ERROR: Command errored out with exit status 1: python." when trying to install django-heroku using pip "E: Unable to locate package python-pip" on Ubuntu 18.04 How to Install pip for python 3.7 on Ubuntu 18? What is the meaning of "Failed building wheel for X" in pip install? Could not install packages due to an EnvironmentError: [Errno 13] How do I install Python packages in Google's Colab? Conda version pip install -r requirements.txt --target ./lib pip: no module named _internal AttributeError: Module Pip has no attribute 'main' Error after upgrading pip: cannot import name 'main'

Examples related to requirements.txt

How to use requirements.txt to install all dependencies in a python project Could not find a version that satisfies the requirement <package> How to state in requirements.txt a direct github source Reference requirements.txt for the install_requires kwarg in setuptools setup.py file tell pip to install the dependencies of packages listed in a requirement file