Sibling package imports

Question

I ve tried reading through questions about sibling imports and even the package documentation  but I ve yet to find an answer   With the following structure    -- LICENSE md  -- README md  -- api         --   init   py         -- api py         -- api key py  -- examples         --   init   py         -- example one py         -- example two py  -- tests         --   init   py         -- test one py   How can the scripts in the  examples and tests directories import from the api module and be run from the commandline   Also  I d like to avoid the ugly sys path insert hack for every file  Surely this can be done in Python  right

User · Accepted Answer

Seven years after

Since I wrote the answer below, modifying sys.path is still a quick-and-dirty trick that works well for private scripts, but there has been several improvements

Installing the package (in a virtualenv or not) will give you what you want, though I would suggest using pip to do it rather than using setuptools directly (and using setup.cfg to store the metadata)
Using the -m flag and running as a package works too (but will turn out a bit awkward if you want to convert your working directory into an installable package).
For the tests, specifically, pytest is able to find the api package in this situation and takes care of the sys.path hacks for you

So it really depends on what you want to do. In your case, though, since it seems that your goal is to make a proper package at some point, installing through pip -e is probably your best bet, even if it is not perfect yet.

Old answer

As already stated elsewhere, the awful truth is that you have to do ugly hacks to allow imports from siblings modules or parents package from a __main__ module. The issue is detailed in PEP 366. PEP 3122 attempted to handle imports in a more rational way but Guido has rejected it one the account of

The only use case seems to be running scripts that happen to be living inside a module's directory, which I've always seen as an antipattern.

(here)

Though, I use this pattern on a regular basis with

# Ugly hack to allow absolute import from the root folder
# whatever its name is. Please forgive the heresy.
if __name__ == "__main__" and __package__ is None:
    from sys import path
    from os.path import dirname as dir

    path.append(dir(path[0]))
    __package__ = "examples"

import api

Here path[0] is your running script's parent folder and dir(path[0]) your top level folder.

I have still not been able to use relative imports with this, though, but it does allow absolute imports from the top level (in your example api's parent folder).

User · Answer

For siblings package imports  you can use either the insert or the append method of the  sys path  2  module   if   name         main    and if   package   is None      import sys     from os import path     sys path append  path dirname  path dirname  path abspath   file              import api   This will work if you are launching your scripts as follows   python examples example one py python tests test one py   On the other hand  you can also use the relative import   if   name         main    and if   package   is not None      import   api api   In this case you will have to launch your script with the  -m  argument  note that  in this case  you must not give the   py  extension    python -m packageName examples example one python -m packageName tests test one   Of course  you can mix the two approaches  so that your script will work no matter how it is called   if   name         main         if   package   is None          import sys         from os import path         sys path append  path dirname  path dirname  path abspath   file                  import api     else          import   api api

User · Answer

Project  1 1 User 1 1 1 about py 1 1 2 init py 1 2 Tech 1 2 1 info py 1 1 2 init py Now  if you want to access about py module in the User package  from the info py module in Tech package then you have to bring the cmd  in windows  path to Project i e    C  Users Personal Desktop Project gt   as per the above Package example  And from this path you have to enter  python -m Package name module name For example for the above Package we have to do  C  Users Personal Desktop Project gt python -m Tech info Imp Points  Don t use  py extension after info module i e  python -m Tech info py Enter this  where the siblings packages are in the same level  -m is the flag  to check about it you can type from the cmd python --help

User · Answer

I wanted to comment on the solution provided by np8 but I don t have enough reputation so I ll just mention that you can create a setup py file exactly as they suggested  and then do pipenv install --dev -e   from the project root directory to turn it into an editable dependency  Then your absolute imports will work e g  from api api import foo and you don t have to mess around with system-wide installations  Documentation

User · Answer

Tired of sys path hacks  There are plenty of sys path append -hacks available  but I found an alternative way of solving the problem in hand  Summary  Wrap the code into one folder  e g  packaged stuff  Create setup py script where you use setuptools setup     see minimal setup py below  Pip install the package in editable state with pip install -e  lt myproject folder gt  Import using from packaged stuff modulename import function name   Setup The starting point is the file structure you have provided  wrapped in a folder called myproject     -- myproject      -- api           -- api key py           -- api py           --   init   py      -- examples           -- example one py           -- example two py           --   init   py      -- LICENCE md      -- README md      -- tests          --   init   py          -- test one py  I will call the   the root folder  and in my example case it is located at C  tmp test imports   api py As a test case  let s use the following   api api py def function from api        return  I am the return value from api api    test one py from api api import function from api  def test function        print function from api     if   name         main         test function    Try to run test one  PS C  tmp test imports gt  python   myproject tests test one py Traceback  most recent call last     File  quot   myproject tests test one py quot   line 1  in  lt module gt      from api api import function from api ModuleNotFoundError  No module named  api   Also trying relative imports wont work  Using from   api api import function from api would result into PS C  tmp test imports gt  python   myproject tests test one py Traceback  most recent call last     File  quot   tests test one py quot   line 1  in  lt module gt      from   api api import function from api ValueError  attempted relative import beyond top-level package   Steps  Make a setup py file to the root level directory   The contents for the setup py would be  from setuptools import setup  find packages  setup name  myproject   version  1 0   packages find packages      Use a virtual environment   If you are familiar with virtual environments  activate one  and skip to the next step  Usage of virtual environments are not absolutely required  but they will really help you out in the long run  when you have more than 1 project ongoing     The most basic steps are  run in the root folder   Create virtual env  python -m venv venv   Activate virtual env  source   venv bin activate  Linux  macOS  or   venv Scripts activate  Win     To learn more about this  just Google out  quot python virtual env tutorial quot  or similar  You probably never need any other commands than creating  activating and deactivating  Once you have made and activated a virtual environment  your console should give the name of the virtual environment in parenthesis PS C  tmp test imports gt  python -m venv venv PS C  tmp test imports gt    venv Scripts activate  venv  PS C  tmp test imports gt   and your folder tree should look like this      -- myproject       -- api            -- api key py            -- api py            --   init   py       -- examples            -- example one py            -- example two py            --   init   py       -- LICENCE md       -- README md       -- tests           --   init   py           -- test one py  -- setup py  -- venv      -- Include      -- Lib      -- pyvenv cfg      -- Scripts  87 entries exceeds filelimit  not opening dir    pip install your project in editable state   Install your top level package myproject using pip  The trick is to use the -e flag when doing the install  This way it is installed in an editable state  and all the edits made to the  py files will be automatically included in the installed package  In the root directory  run pip install -e    note the dot  it stands for  quot current directory quot   You can also see that it is installed by using pip freeze  venv  PS C  tmp test imports gt  pip install -e   Obtaining file    C  tmp test imports Installing collected packages  myproject   Running setup py develop for myproject Successfully installed myproject  venv  PS C  tmp test imports gt  pip freeze myproject  1 0   Add myproject  into your imports   Note that you will have to add myproject  only into imports that would not work otherwise  Imports that worked without the setup py  amp  pip install will work still work fine  See an example below   Test the solution Now  let s test the solution using api py defined above  and test one py defined below  test one py from myproject api api import function from api  def test function        print function from api     if   name         main         test function    running the test  venv  PS C  tmp test imports gt  python   myproject tests test one py I am the return value from api api      See the setuptools docs for more verbose setup py examples     In reality  you could put your virtual environment anywhere on your hard disk

User · Answer

First  you should avoid having files with the same name as the module itself  It may break other imports   When you import a file  first the interpreter checks the current directory and then searchs global directories   Inside examples or tests you can call   from   api import api

User · Answer

for the main question  call sibling folder as module  from    import siblingfolder call a file py from sibling folder as module  from   siblingfolder import a file call a function inside a file in sibling folder as module  from  siblingmodule a file import func name exists in a file The easiest way  go to lib site-packages folder  if exists  easy install pth  file  just edit it and add your directory that you have script that you want make it as module  if not exists  just make it one   and put your folder that you want there after you add it     python will be automatically perceive that folder as similar like site-packages and you can call every script from that folder or subfolder as a module  i wrote this by my phone  and hard to set it to make everyone comfortable to read

User · Answer

I don t yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling relative import hack   Until that day  this is my solution   For examples or tests to import stuff from    api  it would look like    import sys path import os path   Import from sibling directory    api sys path append os path dirname os path abspath   file              import api api import api api key

User · Answer

Here is another alternative that I insert at top of the Python files in tests folder     Path hack  import sys  os sys path insert 0  os path abspath

User · Answer

in your main file add this  import sys import os  sys path append os path abspath os path join   file   mainScriptDepth     mainScriptDepth   the depth of the main file from the root of the project  Here in your case  mainScriptDepth    quot        quot

User · Answer

TLDR  This method does not require setuptools  path hacks  additional command line arguments  or specifying the top level of the package in every single file of your project   Just make a script in the parent directory of whatever your are calling to be your   main   and run everything from there   For further explanation continue reading   Explanation  This can be accomplished without hacking a new path together  extra command line args  or adding code to each of your programs to recognize its siblings   The reason this fails as I believe was mentioned before is the programs being called have their   name   set as   main     When this occurs the script being called accepts itself to be on the top level of the package and refuses to recognize scripts in sibling directories   However  everything under the top level of the directory will still recognize ANYTHING ELSE under the top level   This means the ONLY thing you have to do to get files in sibling directories to recognize utilize each other is to call them from a script in their parent directory   Proof of Concept In a dir with the following structure        Main py      Siblings             sib1                    call py             sib2                    callsib py   Main py contains the following code   import sib1 call as call   def main        call Call     if   name         main         main     sib1 call py contains   import sib2 callsib as callsib   def Call        callsib CallSib     if   name         main         Call     and sib2 callsib py contains   def CallSib        print  Got Called    if   name         main         CallSib     If you reproduce this example you will notice that calling Main py will result in  Got Called  being printed as is defined in sib2 callsib py  even though sib2 callsib py got called through sib1 call py   However if one were to directly call sib1 call py  after making appropriate changes to the imports  it throws an exception   Even though it worked when called by the script in its parent directory  it will not work if it believes itself to be on the top level of the package

User · Answer

You need to look to see how the import statements are written in the related code   If examples example one py uses the following import statement   import api api      then it expects the root directory of the project to be in the system path   The easiest way to support this without any hacks  as you put it  would be to run the examples from the top level directory  like this   PYTHONPATH  PYTHONPATH   python examples example one py

User · Answer

I made a sample project to demonstrate how I handled this  which is indeed another sys path hack as indicated above  Python Sibling Import Example  which relies on   if   name         main         import os     import sys     sys path append os getcwd     This seems to be pretty effective so long as your working directory remains at the root of the Python project  If anyone deploys this in a real production environment it d be great to hear if it works there as well

User · Answer

You don t need and shouldn t hack sys path unless it is necessary and in this case it is not  Use   import api api key   in tests  examples   Run from the project directory  python -m tests test one   You should probably move tests  if they are api s unittests  inside api and run python -m api test to run all tests  assuming there is   main   py  or python -m api test test one to run test one instead    You could also remove   init   py from examples  it is not a Python package  and run the examples in a virtualenv where api is installed e g   pip install -e   in a virtualenv would install inplace api package if you have proper setup py

User · Answer

Just in case someone using Pydev on Eclipse end up here  you can add the sibling s parent path  and thus the calling module s parent  as an external library folder using Project- Properties and setting External Libraries under the left menu Pydev-PYTHONPATH  Then you can import from your sibling  e  g  from sibling import some class

[python] Sibling package imports

Seven years after

Old answer

Examples related to python

Examples related to packages

Examples related to python-import

Examples related to siblings