I have a requirements.txt
file that I'm using with Travis-CI. It seems silly to duplicate the requirements in both requirements.txt
and setup.py
, so I was hoping to pass a file handle to the install_requires
kwarg in setuptools.setup
.
Is this possible? If so, how should I go about doing it?
Here is my requirements.txt
file:
guessit>=0.5.2
tvdb_api>=1.8.2
hachoir-metadata>=1.3.3
hachoir-core>=1.3.3
hachoir-parser>=1.3.4
This question is related to
python
pip
setuptools
requirements.txt
Requirements files use an expanded pip format, which is only useful if you need to complement your setup.py
with stronger constraints, for example specifying the exact urls some of the dependencies must come from, or the output of pip freeze
to freeze the entire package set to known-working versions. If you don't need the extra constraints, use only a setup.py
. If you feel like you really need to ship a requirements.txt
anyway, you can make it a single line:
.
It will be valid and refer exactly to the contents of the setup.py
that is in the same directory.
from pip.req import parse_requirements
did not work for me and I think it's for the blank lines in my requirements.txt, but this function does work
def parse_requirements(requirements):
with open(requirements) as f:
return [l.strip('\n') for l in f if l.strip('\n') and not l.startswith('#')]
reqs = parse_requirements(<requirements_path>)
setup(
...
install_requires=reqs,
...
)
I did this:
import re
def requirements(filename):
with open(filename) as f:
ll = f.read().splitlines()
d = {}
for l in ll:
k, v = re.split(r'==|>=', l)
d[k] = v
return d
def packageInfo():
try:
from pip._internal.operations import freeze
except ImportError:
from pip.operations import freeze
d = {}
for kv in freeze.freeze():
k, v = re.split(r'==|>=', kv)
d[k] = v
return d
req = getpackver('requirements.txt')
pkginfo = packageInfo()
for k, v in req.items():
print(f'{k:<16}: {v:<6} -> {pkginfo[k]}')
Most of the other answers above don't work with the current version of pip's API. Here is the correct* way to do it with the current version of pip (6.0.8 at the time of writing, also worked in 7.1.2. You can check your version with pip -V).
from pip.req import parse_requirements
from pip.download import PipSession
install_reqs = parse_requirements(<requirements_path>, session=PipSession())
reqs = [str(ir.req) for ir in install_reqs]
setup(
...
install_requires=reqs
....
)
* Correct, in that it is the way to use parse_requirements with the current pip. It still probably isn't the best way to do it, since, as posters above said, pip doesn't really maintain an API.
Install the current package in Travis. This avoids the use of a requirements.txt
file.
For example:
language: python
python:
- "2.7"
- "2.6"
install:
- pip install -q -e .
script:
- python runtests.py
Yet another parse_requirements
hack that also parses environment markers into extras_require
:
from collections import defaultdict
from pip.req import parse_requirements
requirements = []
extras = defaultdict(list)
for r in parse_requirements('requirements.txt', session='hack'):
if r.markers:
extras[':' + str(r.markers)].append(str(r.req))
else:
requirements.append(str(r.req))
setup(
...,
install_requires=requirements,
extras_require=extras
)
It should support both sdist and binary dists.
As stated by others, parse_requirements
has several shortcomings, so this is not what you should do on public projects, but it may suffice for internal/personal projects.
On the face of it, it does seem that requirements.txt
and setup.py
are silly duplicates, but it's important to understand that while the form is similar, the intended function is very different.
The goal of a package author, when specifying dependencies, is to say "wherever you install this package, these are the other packages you need, in order for this package to work."
In contrast, the deployment author (which may be the same person at a different time) has a different job, in that they say "here's the list of packages that we've gathered together and tested and that I now need to install".
The package author writes for a wide variety of scenarios, because they're putting their work out there to be used in ways they may not know about, and have no way of knowing what packages will be installed alongside their package. In order to be a good neighbor and avoid dependency version conflicts with other packages, they need to specify as wide a range of dependency versions as can possibly work. This is what install_requires
in setup.py
does.
The deployment author writes for a very different, very specific goal: a single instance of an installed application or service, installed on a particular computer. In order to precisely control a deployment, and be sure that the right packages are tested and deployed, the deployment author must specify the exact version and source-location of every package to be installed, including dependencies and dependency's dependencies. With this spec, a deployment can be repeatably applied to several machines, or tested on a test machine, and the deployment author can be confident that the same packages are deployed every time. This is what a requirements.txt
does.
So you can see that, while they both look like a big list of packages and versions, these two things have very different jobs. And it's definitely easy to mix this up and get it wrong! But the right way to think about this is that requirements.txt
is an "answer" to the "question" posed by the requirements in all the various setup.py
package files. Rather than write it by hand, it's often generated by telling pip to look at all the setup.py
files in a set of desired packages, find a set of packages that it thinks fits all the requirements, and then, after they're installed, "freeze" that list of packages into a text file (this is where the pip freeze
name comes from).
So the takeaway:
setup.py
should declare the loosest possible dependency versions that are still workable. Its job is to say what a particular package can work with.requirements.txt
is a deployment manifest that defines an entire installation job, and shouldn't be thought of as tied to any one package. Its job is to declare an exhaustive list of all the necessary packages to make a deployment work.Using parse_requirements
is problematic because the pip API isn't publicly documented and supported. In pip 1.6, that function is actually moving, so existing uses of it are likely to break.
A more reliable way to eliminate duplication between setup.py
and requirements.txt
is to specific your dependencies in setup.py
and then put -e .
into your requirements.txt
file. Some information from one of the pip
developers about why that's a better way to go is available here: https://caremad.io/blog/setup-vs-requirement/
This simple approach reads the requirements file from setup.py
. It is a variation of the answer by Dmitiry S.. This answer is compatible only with Python 3.6+.
Per D.S., requirements.txt
can document concrete requirements with specific version numbers, whereas setup.py
can document abstract requirements with loose version ranges.
Below is an excerpt of my setup.py
.
import distutils.text_file
from pathlib import Path
from typing import List
def _parse_requirements(filename: str) -> List[str]:
"""Return requirements from requirements file."""
# Ref: https://stackoverflow.com/a/42033122/
return distutils.text_file.TextFile(filename=str(Path(__file__).with_name(filename))).readlines()
setup(...
install_requires=_parse_requirements('requirements.txt'),
...)
Note that distutils.text_file.TextFile
will strip comments. Also, per my experience, you apparently do not need to take any special step to bundle in the requirements file.
I created a reusable function for this. It actually parses an entire directory of requirements files and sets them to extras_require.
Latest always available here: https://gist.github.com/akatrevorjay/293c26fefa24a7b812f5
import glob
import itertools
import os
# This is getting ridiculous
try:
from pip._internal.req import parse_requirements
from pip._internal.network.session import PipSession
except ImportError:
try:
from pip._internal.req import parse_requirements
from pip._internal.download import PipSession
except ImportError:
from pip.req import parse_requirements
from pip.download import PipSession
def setup_requirements(
patterns=[
'requirements.txt', 'requirements/*.txt', 'requirements/*.pip'
],
combine=True):
"""
Parse a glob of requirements and return a dictionary of setup() options.
Create a dictionary that holds your options to setup() and update it using this.
Pass that as kwargs into setup(), viola
Any files that are not a standard option name (ie install, tests, setup) are added to extras_require with their
basename minus ext. An extra key is added to extras_require: 'all', that contains all distinct reqs combined.
Keep in mind all literally contains `all` packages in your extras.
This means if you have conflicting packages across your extras, then you're going to have a bad time.
(don't use all in these cases.)
If you're running this for a Docker build, set `combine=True`.
This will set `install_requires` to all distinct reqs combined.
Example:
>>> import setuptools
>>> _conf = dict(
... name='mainline',
... version='0.0.1',
... description='Mainline',
... author='Trevor Joynson <[email protected],io>',
... url='https://trevor.joynson.io',
... namespace_packages=['mainline'],
... packages=setuptools.find_packages(),
... zip_safe=False,
... include_package_data=True,
... )
>>> _conf.update(setup_requirements())
>>> # setuptools.setup(**_conf)
:param str pattern: Glob pattern to find requirements files
:param bool combine: Set True to set install_requires to extras_require['all']
:return dict: Dictionary of parsed setup() options
"""
session = PipSession()
# Handle setuptools insanity
key_map = {
'requirements': 'install_requires',
'install': 'install_requires',
'tests': 'tests_require',
'setup': 'setup_requires',
}
ret = {v: set() for v in key_map.values()}
extras = ret['extras_require'] = {}
all_reqs = set()
files = [glob.glob(pat) for pat in patterns]
files = itertools.chain(*files)
for full_fn in files:
# Parse
reqs = {
str(r.req)
for r in parse_requirements(full_fn, session=session)
# Must match env marker, eg:
# yarl ; python_version >= '3.0'
if r.match_markers()
}
all_reqs.update(reqs)
# Add in the right section
fn = os.path.basename(full_fn)
barefn, _ = os.path.splitext(fn)
key = key_map.get(barefn)
if key:
ret[key].update(reqs)
extras[key] = reqs
extras[barefn] = reqs
if 'all' not in extras:
extras['all'] = list(all_reqs)
if combine:
extras['install'] = ret['install_requires']
ret['install_requires'] = list(all_reqs)
def _listify(dikt):
ret = {}
for k, v in dikt.items():
if isinstance(v, set):
v = list(v)
elif isinstance(v, dict):
v = _listify(v)
ret[k] = v
return ret
ret = _listify(ret)
return ret
__all__ = ['setup_requirements']
if __name__ == '__main__':
reqs = setup_requirements()
print(reqs)
Another possible solution...
def gather_requirements(top_path=None):
"""Captures requirements from repo.
Expected file format is: requirements[-_]<optional-extras>.txt
For example:
pip install -e .[foo]
Would require:
requirements-foo.txt
or
requirements_foo.txt
"""
from pip.download import PipSession
from pip.req import parse_requirements
import re
session = PipSession()
top_path = top_path or os.path.realpath(os.getcwd())
extras = {}
for filepath in tree(top_path):
filename = os.path.basename(filepath)
basename, ext = os.path.splitext(filename)
if ext == '.txt' and basename.startswith('requirements'):
if filename == 'requirements.txt':
extra_name = 'requirements'
else:
_, extra_name = re.split(r'[-_]', basename, 1)
if extra_name:
reqs = [str(ir.req) for ir in parse_requirements(filepath, session=session)]
extras.setdefault(extra_name, []).extend(reqs)
all_reqs = set()
for key, values in extras.items():
all_reqs.update(values)
extras['all'] = list(all_reqs)
return extras
and then to use...
reqs = gather_requirements()
install_reqs = reqs.pop('requirements', [])
test_reqs = reqs.pop('test', [])
...
setup(
...
'install_requires': install_reqs,
'test_requires': test_reqs,
'extras_require': reqs,
...
)
The following interface became deprecated in pip 10:
from pip.req import parse_requirements
from pip.download import PipSession
So I switched it just to simple text parsing:
with open('requirements.txt', 'r') as f:
install_reqs = [
s for s in [
line.split('#', 1)[0].strip(' \t\n') for line in f
] if s != ''
]
Cross posting my answer from this SO question for another simple, pip version proof solution.
try: # for pip >= 10
from pip._internal.req import parse_requirements
from pip._internal.download import PipSession
except ImportError: # for pip <= 9.0.3
from pip.req import parse_requirements
from pip.download import PipSession
requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())
if __name__ == '__main__':
setup(
...
install_requires=[str(requirement.req) for requirement in requirements],
...
)
Then just throw in all your requirements under requirements.txt
under project root directory.
If you don't want to force your users to install pip, you can emulate its behavior with this:
import sys
from os import path as p
try:
from setuptools import setup, find_packages
except ImportError:
from distutils.core import setup, find_packages
def read(filename, parent=None):
parent = (parent or __file__)
try:
with open(p.join(p.dirname(parent), filename)) as f:
return f.read()
except IOError:
return ''
def parse_requirements(filename, parent=None):
parent = (parent or __file__)
filepath = p.join(p.dirname(parent), filename)
content = read(filename, parent)
for line_number, line in enumerate(content.splitlines(), 1):
candidate = line.strip()
if candidate.startswith('-r'):
for item in parse_requirements(candidate[2:].strip(), filepath):
yield item
else:
yield candidate
setup(
...
install_requires=list(parse_requirements('requirements.txt'))
)
While not an exact answer to the question, I recommend Donald Stufft's blog post at https://caremad.io/2013/07/setup-vs-requirement/ for a good take on this problem. I've been using it to great success.
In short, requirements.txt
is not a setup.py
alternative, but a deployment complement. Keep an appropriate abstraction of package dependencies in setup.py
. Set requirements.txt
or more of 'em to fetch specific versions of package dependencies for development, testing, or production.
E.g. with packages included in the repo under deps/
:
# fetch specific dependencies
--no-index
--find-links deps/
# install package
# NOTE: -e . for editable mode
.
pip executes package's setup.py
and installs the specific versions of dependencies declared in install_requires
. There's no duplicity and the purpose of both artifacts is preserved.
It can't take a file handle. The install_requires
argument can only be a string or a list of strings.
You can, of course, read your file in the setup script and pass it as a list of strings to install_requires
.
import os
from setuptools import setup
with open('requirements.txt') as f:
required = f.read().splitlines()
setup(...
install_requires=required,
...)
BEWARE OF parse_requirements
BEHAVIOUR!
Please note that pip.req.parse_requirements
will change underscores to dashes. This was enraging me for a few days before I discovered it. Example demonstrating:
from pip.req import parse_requirements # tested with v.1.4.1
reqs = '''
example_with_underscores
example-with-dashes
'''
with open('requirements.txt', 'w') as f:
f.write(reqs)
req_deps = parse_requirements('requirements.txt')
result = [str(ir.req) for ir in req_deps if ir.req is not None]
print result
produces
['example-with-underscores', 'example-with-dashes']
Here is a complete hack (tested with pip 9.0.1
) based on Romain's answer that parses requirements.txt
and filters it according to current environment markers:
from pip.req import parse_requirements
requirements = []
for r in parse_requirements('requirements.txt', session='hack'):
# check markers, such as
#
# rope_py3k ; python_version >= '3.0'
#
if r.match_markers():
requirements.append(str(r.req))
print(requirements)
I would not recommend doing such a thing. As mentioned multiple times install_requires
and requirements.txt
are definitely not supposed to be the same list. But since there are a lot of misleading answers all around involving private internal APIs of pip, it might be worth looking at saner alternatives...
There is no need for pip to parse a requirements.txt
file from a setuptools setup.py
script. The setuptools project already contains all the necessary tools in its top level package pkg_resources
.
It could more or less look like this:
#!/usr/bin/env python3
import pathlib
import pkg_resources
import setuptools
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
setuptools.setup(
install_requires=install_requires,
)
Notes:
Source: Stackoverflow.com