Reference requirements txt for the install requires kwarg in setuptools setup py file

Question

I have a requirements txt file that I m using with Travis-CI   It seems silly to duplicate the requirements in both requirements txt and setup py  so I was hoping to pass a file handle to the install requires kwarg in setuptools setup   Is this possible  If so  how should I go about doing it   Here is my requirements txt file   guessit gt  0 5 2 tvdb api gt  1 8 2 hachoir-metadata gt  1 3 3 hachoir-core gt  1 3 3 hachoir-parser gt  1 3 4

User · Answer

If you don t want to force your users to install pip  you can emulate its behavior with this   import sys  from os import path as p  try      from setuptools import setup  find packages except ImportError      from distutils core import setup  find packages   def read filename  parent None       parent    parent or   file         try          with open p join p dirname parent   filename   as f              return f read       except IOError          return      def parse requirements filename  parent None       parent    parent or   file        filepath   p join p dirname parent   filename      content   read filename  parent       for line number  line in enumerate content splitlines    1           candidate   line strip            if candidate startswith  -r                for item in parse requirements candidate 2   strip    filepath                   yield item         else              yield candidate  setup          install requires list parse requirements  requirements txt

User · Answer

Yet another parse requirements hack that also parses environment markers into extras require   from collections import defaultdict from pip req import parse requirements  requirements      extras   defaultdict list  for r in parse requirements  requirements txt   session  hack        if r markers          extras       str r markers   append str r req       else          requirements append str r req    setup               install requires requirements      extras require extras     It should support both sdist and binary dists   As stated by others  parse requirements has several shortcomings  so this is not what you should do on public projects  but it may suffice for internal personal projects

User · Answer

While not an exact answer to the question  I recommend Donald Stufft s blog post at https   caremad io 2013 07 setup-vs-requirement  for a good take on this problem  I ve been using it to great success   In short  requirements txt is not a setup py alternative  but a deployment complement  Keep an appropriate abstraction of package dependencies in setup py  Set requirements txt or more of  em to fetch specific versions of package dependencies for development  testing  or production   E g  with packages included in the repo under deps      fetch specific dependencies --no-index --find-links deps     install package   NOTE  -e   for editable mode     pip executes package s setup py and installs the specific versions of dependencies declared in install requires  There s no duplicity and the purpose of both artifacts is preserved

User · Answer

BEWARE OF parse requirements BEHAVIOUR   Please note that pip req parse requirements will change underscores to dashes   This was enraging me for a few days before I discovered it   Example demonstrating   from pip req import parse requirements    tested with v 1 4 1  reqs       example with underscores example-with-dashes      with open  requirements txt    w   as f      f write reqs   req deps   parse requirements  requirements txt   result    str ir req  for ir in req deps if ir req is not None  print result   produces    example-with-underscores    example-with-dashes

User · Answer

Here is a complete hack  tested with pip 9 0 1  based on Romain s answer that parses requirements txt and filters it according to current environment markers   from pip req import parse requirements  requirements      for r in parse requirements  requirements txt   session  hack          check markers  such as                 rope py3k      python version  gt    3 0            if r match markers            requirements append str r req    print requirements

User · Answer

On the face of it  it does seem that requirements txt and setup py are silly duplicates  but it s important to understand that while the form is similar  the intended function is very different   The goal of a package author  when specifying dependencies  is to say  wherever you install this package  these are the other packages you need  in order for this package to work    In contrast  the deployment author  which may be the same person at a different time  has a different job  in that they say  here s the list of packages that we ve gathered together and tested and that I now need to install    The package author writes for a wide variety of scenarios  because they re putting their work out there to be used in ways they may not know about  and have no way of knowing what packages will be installed alongside their package   In order to be a good neighbor and avoid dependency version conflicts with other packages  they need to specify as wide a range of dependency versions as can possibly work   This is what install requires in setup py does   The deployment author writes for a very different  very specific goal  a single instance of an installed application or service  installed on a particular computer   In order to precisely control a deployment  and be sure that the right packages are tested and deployed  the deployment author must specify the exact version and source-location of every package to be installed  including dependencies and dependency s dependencies   With this spec  a deployment can be repeatably applied to several machines  or tested on a test machine  and the deployment author can be confident that the same packages are deployed every time   This is what a requirements txt does   So you can see that  while they both look like a big list of packages and versions  these two things have very different jobs   And it s definitely easy to mix this up and get it wrong   But the right way to think about this is that requirements txt is an  answer  to the  question  posed by the requirements in all the various setup py package files   Rather than write it by hand  it s often generated by telling pip to look at all the setup py files in a set of desired packages  find a set of packages that it thinks fits all the requirements  and then  after they re installed   freeze  that list of packages into a text file  this is where the pip freeze name comes from      So the takeaway     setup py should declare the loosest possible dependency versions that are still workable   Its job is to say what a particular package can work with  requirements txt is a deployment manifest that defines an entire installation job  and shouldn t be thought of as tied to any one package   Its job is to declare an exhaustive list of all the necessary packages to make a deployment work  Because these two things have such different content and reasons for existing  it s not feasible to simply copy one into the other    References    install requires vs Requirements files from the Python packaging user guide

User · Answer

Cross posting my answer from this SO question for another simple  pip version proof solution   try     for pip  gt   10     from pip  internal req import parse requirements     from pip  internal download import PipSession except ImportError     for pip  lt   9 0 3     from pip req import parse requirements     from pip download import PipSession  requirements   parse requirements os path join os path dirname   file      requirements txt    session PipSession     if   name         main         setup                      install requires  str requirement req  for requirement in requirements                        Then just throw in all your requirements under requirements txt under project root directory

User · Answer

I did this   import re  def requirements filename       with open filename  as f          ll   f read   splitlines       d          for l in ll          k  v   re split r     gt     l          d k    v     return d  def packageInfo        try          from pip  internal operations import freeze     except ImportError          from pip operations import freeze      d          for kv in freeze freeze            k  v   re split r     gt     kv          d k    v     return d  req   getpackver  requirements txt   pkginfo   packageInfo    for k  v in req items        print f  k  lt 16    v  lt 6  - gt   pkginfo k

User · Answer

Requirements files use an expanded pip format  which is only useful if you need to complement your setup py with stronger constraints  for example specifying the exact urls some of the dependencies must come from  or the output of pip freeze to freeze the entire package set to known-working versions  If you don t need the extra constraints  use only a setup py  If you feel like you really need to ship a requirements txt anyway  you can make it a single line       It will be valid and refer exactly to the contents of the setup py that is in the same directory

User · Answer

I would not recommend doing such a thing  As mentioned multiple times install requires and requirements txt are definitely not supposed to be the same list  But since there are a lot of misleading answers all around involving private internal APIs of pip  it might be worth looking at saner alternatives    There is no need for pip to parse a requirements txt file from a setuptools setup py script  The setuptools project already contains all the necessary tools in its top level package pkg resources  It could more or less look like this     usr bin env python3  import pathlib  import pkg resources import setuptools  with pathlib Path  requirements txt   open   as requirements txt      install requires             str requirement          for requirement         in pkg resources parse requirements requirements txt         setuptools setup      install requires install requires      Notes   See also this other answer  https   stackoverflow com a 59971236 11138259

User · Answer

Another possible solution     def gather requirements top path None          Captures requirements from repo       Expected file format is  requirements -   lt optional-extras gt  txt      For example           pip install -e   foo       Would require           requirements-foo txt          or          requirements foo txt              from pip download import PipSession     from pip req import parse requirements     import re      session   PipSession       top path   top path or os path realpath os getcwd        extras          for filepath in tree top path           filename   os path basename filepath          basename  ext   os path splitext filename          if ext      txt  and basename startswith  requirements                if filename     requirements txt                   extra name    requirements              else                     extra name   re split r  -     basename  1              if extra name                  reqs    str ir req  for ir in parse requirements filepath  session session                   extras setdefault extra name      extend reqs      all reqs   set       for key  values in extras items            all reqs update values      extras  all     list all reqs      return extras   and then to use     reqs   gather requirements   install reqs   reqs pop  requirements       test reqs   reqs pop  test           setup               install requires   install reqs       test requires   test reqs       extras require   reqs

User · Answer

The following interface became deprecated in pip 10   from pip req import parse requirements from pip download import PipSession   So I switched it just to simple text parsing   with open  requirements txt    r   as f      install reqs             s for s in               line split      1  0  strip    t n   for line in f           if s

User · Answer

Install the current package in Travis  This avoids the use of a requirements txt file  For example   language  python python    -  2 7    -  2 6  install    - pip install -q -e   script    - python runtests py

User · Answer

You can flip it around and list the dependencies in setup py and have a single character  mdash  a dot    mdash  in requirements txt instead     Alternatively  even if not advised  it is still possible to parse the requirements txt file  if it doesn t refer any external requirements by URL  with the following hack  tested with pip 9 0 1    install reqs   parse requirements  requirements txt   session  hack     This doesn t filter environment markers though     In old versions of pip  more specifically older than 6 0  there is a public API that can be used to achieve this  A requirement file can contain comments     and can include some other files  --requirement or -r   Thus  if you really want to parse a requirements txt you can use the pip parser   from pip req import parse requirements    parse requirements   returns generator of pip req InstallRequirement objects install reqs   parse requirements  lt requirements path gt      reqs is a list of requirement   e g    django  1 5 1    mezzanine  1 4 6   reqs    str ir req  for ir in install reqs   setup              install requires reqs

User · Answer

from pip req import parse requirements did not work for me and I think it s for the blank lines in my requirements txt  but this function does work  def parse requirements requirements       with open requirements  as f          return  l strip   n   for l in f if l strip   n   and not l startswith        reqs   parse requirements  lt requirements path gt    setup              install requires reqs

User · Answer

Using parse requirements is problematic because the pip API isn t publicly documented and supported   In pip 1 6  that function is actually moving  so existing uses of it are likely to break   A more reliable way to eliminate duplication between setup py and requirements txt is to specific your dependencies in setup py and then put -e   into your requirements txt file   Some information from one of the pip developers about why that s a better way to go is available here  https   caremad io blog setup-vs-requirement

User · Answer

It can t take a file handle  The install requires argument can only be a string or a list of strings   You can  of course  read your file in the setup script and pass it as a list of strings to install requires   import os from setuptools import setup  with open  requirements txt   as f      required   f read   splitlines    setup     install requires required

User · Answer

I created a reusable function for this  It actually parses an entire directory of requirements files and sets them to extras require   Latest always available here  https   gist github com akatrevorjay 293c26fefa24a7b812f5  import glob import itertools import os    This is getting ridiculous try      from pip  internal req import parse requirements     from pip  internal network session import PipSession except ImportError      try          from pip  internal req import parse requirements         from pip  internal download import PipSession     except ImportError          from pip req import parse requirements         from pip download import PipSession   def setup requirements          patterns                requirements txt    requirements   txt    requirements   pip                     combine True               Parse a glob of requirements and return a dictionary of setup   options      Create a dictionary that holds your options to setup   and update it using this      Pass that as kwargs into setup    viola      Any files that are not a standard option name  ie install  tests  setup  are added to extras require with their     basename minus ext  An extra key is added to extras require   all   that contains all distinct reqs combined       Keep in mind all literally contains  all  packages in your extras      This means if you have conflicting packages across your extras  then you re going to have a bad time       don t use all in these cases        If you re running this for a Docker build  set  combine True       This will set  install requires  to all distinct reqs combined       Example        gt  gt  gt  import setuptools      gt  gt  gt   conf   dict              name  mainline               version  0 0 1               description  Mainline               author  Trevor Joynson  lt github trevor joynson io gt                url  https   trevor joynson io               namespace packages   mainline                packages setuptools find packages                zip safe False              include package data True                 gt  gt  gt   conf update setup requirements         gt  gt  gt    setuptools setup    conf        param str pattern  Glob pattern to find requirements files      param bool combine  Set True to set install requires to extras require  all        return dict  Dictionary of parsed setup   options             session   PipSession          Handle setuptools insanity     key map              requirements    install requires            install    install requires            tests    tests require            setup    setup requires             ret    v  set   for v in key map values        extras   ret  extras require            all reqs   set        files    glob glob pat  for pat in patterns      files   itertools chain  files       for full fn in files            Parse         reqs                 str r req              for r in parse requirements full fn  session session                Must match env marker  eg                  yarl   python version  gt    3 0              if r match markers                     all reqs update reqs             Add in the right section         fn   os path basename full fn          barefn      os path splitext fn          key   key map get barefn           if key              ret key  update reqs              extras key    reqs          extras barefn    reqs      if  all  not in extras          extras  all     list all reqs       if combine          extras  install     ret  install requires           ret  install requires     list all reqs       def  listify dikt           ret               for k  v in dikt items                if isinstance v  set                   v   list v              elif isinstance v  dict                   v    listify v              ret k    v          return ret      ret    listify ret       return ret     all       setup requirements    if   name         main         reqs   setup requirements       print reqs

User · Answer

Most of the other answers above don t work with the current version of pip s API  Here is the correct  way to do it with the current version of pip  6 0 8 at the time of writing  also worked in 7 1 2  You can check your version with pip -V    from pip req import parse requirements from pip download import PipSession  install reqs   parse requirements  lt requirements path gt   session PipSession     reqs    str ir req  for ir in install reqs   setup              install requires reqs                Correct  in that it is the way to use parse requirements with the current pip  It still probably isn t the best way to do it  since  as posters above said  pip doesn t really maintain an API

User · Answer

This simple approach reads the requirements file from setup py  It is a variation of the answer by Dmitiry S   This answer is compatible only with Python 3 6    Per D S   requirements txt can document concrete requirements with specific version numbers  whereas setup py can document abstract requirements with loose version ranges   Below is an excerpt of my setup py   import distutils text file from pathlib import Path from typing import List  def  parse requirements filename  str  - gt  List str          Return requirements from requirements file           Ref  https   stackoverflow com a 42033122      return distutils text file TextFile filename str Path   file    with name filename    readlines    setup           install requires  parse requirements  requirements txt              Note that distutils text file TextFile will strip comments  Also  per my experience  you apparently do not need to take any special step to bundle in the requirements file

[python] Reference requirements.txt for the install_requires kwarg in setuptools setup.py file

Examples related to python

Examples related to pip

Examples related to setuptools

Examples related to requirements.txt