[python] Sibling package imports

I've tried reading through questions about sibling imports and even the package documentation, but I've yet to find an answer.

With the following structure:

+-- LICENSE.md
+-- README.md
+-- api
¦   +-- __init__.py
¦   +-- api.py
¦   +-- api_key.py
+-- examples
¦   +-- __init__.py
¦   +-- example_one.py
¦   +-- example_two.py
+-- tests
¦   +-- __init__.py
¦   +-- test_one.py

How can the scripts in the examples and tests directories import from the api module and be run from the commandline?

Also, I'd like to avoid the ugly sys.path.insert hack for every file. Surely this can be done in Python, right?

This question is related to python packages python-import siblings

The answer is


Just in case someone using Pydev on Eclipse end up here: you can add the sibling's parent path (and thus the calling module's parent) as an external library folder using Project->Properties and setting External Libraries under the left menu Pydev-PYTHONPATH. Then you can import from your sibling, e. g. from sibling import some_class.


Here is another alternative that I insert at top of the Python files in tests folder:

# Path hack.
import sys, os
sys.path.insert(0, os.path.abspath('..'))

  1. Project

1.1 User

1.1.1 about.py

1.1.2 init.py

1.2 Tech

1.2.1 info.py

1.1.2 init.py

Now, if you want to access about.py module in the User package, from the info.py module in Tech package then you have to bring the cmd (in windows) path to Project i.e. **C:\Users\Personal\Desktop\Project>**as per the above Package example. And from this path you have to enter, python -m Package_name.module_name For example for the above Package we have to do,

C:\Users\Personal\Desktop\Project>python -m Tech.info

Imp Points

  1. Don't use .py extension after info module i.e. python -m Tech.info.py
  2. Enter this, where the siblings packages are in the same level.
  3. -m is the flag, to check about it you can type from the cmd python --help

I made a sample project to demonstrate how I handled this, which is indeed another sys.path hack as indicated above. Python Sibling Import Example, which relies on:

if __name__ == '__main__': import os import sys sys.path.append(os.getcwd())

This seems to be pretty effective so long as your working directory remains at the root of the Python project. If anyone deploys this in a real production environment it'd be great to hear if it works there as well.


I don't yet have the comprehension of Pythonology necessary to see the intended way of sharing code amongst unrelated projects without a sibling/relative import hack. Until that day, this is my solution. For examples or tests to import stuff from ..\api, it would look like:

import sys.path
import os.path
# Import from sibling directory ..\api
sys.path.append(os.path.dirname(os.path.abspath(__file__)) + "/..")
import api.api
import api.api_key

TLDR

This method does not require setuptools, path hacks, additional command line arguments, or specifying the top level of the package in every single file of your project.

Just make a script in the parent directory of whatever your are calling to be your __main__ and run everything from there. For further explanation continue reading.

Explanation

This can be accomplished without hacking a new path together, extra command line args, or adding code to each of your programs to recognize its siblings.

The reason this fails as I believe was mentioned before is the programs being called have their __name__ set as __main__. When this occurs the script being called accepts itself to be on the top level of the package and refuses to recognize scripts in sibling directories.

However, everything under the top level of the directory will still recognize ANYTHING ELSE under the top level. This means the ONLY thing you have to do to get files in sibling directories to recognize/utilize each other is to call them from a script in their parent directory.

Proof of Concept In a dir with the following structure:

.
|__Main.py
|
|__Siblings
   |
   |___sib1
   |   |
   |   |__call.py
   |
   |___sib2
       |
       |__callsib.py

Main.py contains the following code:

import sib1.call as call


def main():
    call.Call()


if __name__ == '__main__':
    main()

sib1/call.py contains:

import sib2.callsib as callsib


def Call():
    callsib.CallSib()


if __name__ == '__main__':
    Call()

and sib2/callsib.py contains:

def CallSib():
    print("Got Called")

if __name__ == '__main__':
    CallSib()

If you reproduce this example you will notice that calling Main.py will result in "Got Called" being printed as is defined in sib2/callsib.py even though sib2/callsib.py got called through sib1/call.py. However if one were to directly call sib1/call.py (after making appropriate changes to the imports) it throws an exception. Even though it worked when called by the script in its parent directory, it will not work if it believes itself to be on the top level of the package.


for the main question:

call sibling folder as module:

from .. import siblingfolder

call a_file.py from sibling folder as module:

from ..siblingfolder import a_file

call a_function inside a file in sibling folder as module:

from..siblingmodule.a_file import func_name_exists_in_a_file

The easiest way.

go to lib/site-packages folder.

if exists 'easy_install.pth' file, just edit it and add your directory that you have script that you want make it as module.

if not exists, just make it one...and put your folder that you want there

after you add it..., python will be automatically perceive that folder as similar like site-packages and you can call every script from that folder or subfolder as a module.

i wrote this by my phone, and hard to set it to make everyone comfortable to read.


You don't need and shouldn't hack sys.path unless it is necessary and in this case it is not. Use:

import api.api_key # in tests, examples

Run from the project directory: python -m tests.test_one.

You should probably move tests (if they are api's unittests) inside api and run python -m api.test to run all tests (assuming there is __main__.py) or python -m api.test.test_one to run test_one instead.

You could also remove __init__.py from examples (it is not a Python package) and run the examples in a virtualenv where api is installed e.g., pip install -e . in a virtualenv would install inplace api package if you have proper setup.py.


First, you should avoid having files with the same name as the module itself. It may break other imports.

When you import a file, first the interpreter checks the current directory and then searchs global directories.

Inside examples or tests you can call:

from ..api import api

Tired of sys.path hacks?

There are plenty of sys.path.append -hacks available, but I found an alternative way of solving the problem in hand.

Summary

  • Wrap the code into one folder (e.g. packaged_stuff)
  • Create setup.py script where you use setuptools.setup(). (see minimal setup.py below)
  • Pip install the package in editable state with pip install -e <myproject_folder>
  • Import using from packaged_stuff.modulename import function_name

Setup

The starting point is the file structure you have provided, wrapped in a folder called myproject.

.
+-- myproject
    +-- api
    ¦   +-- api_key.py
    ¦   +-- api.py
    ¦   +-- __init__.py
    +-- examples
    ¦   +-- example_one.py
    ¦   +-- example_two.py
    ¦   +-- __init__.py
    +-- LICENCE.md
    +-- README.md
    +-- tests
        +-- __init__.py
        +-- test_one.py

I will call the . the root folder, and in my example case it is located at C:\tmp\test_imports\.

api.py

As a test case, let's use the following ./api/api.py

def function_from_api():
    return 'I am the return value from api.api!'

test_one.py

from api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

Try to run test_one:

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\myproject\tests\test_one.py", line 1, in <module>
    from api.api import function_from_api
ModuleNotFoundError: No module named 'api'

Also trying relative imports wont work:

Using from ..api.api import function_from_api would result into

PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
Traceback (most recent call last):
  File ".\tests\test_one.py", line 1, in <module>
    from ..api.api import function_from_api
ValueError: attempted relative import beyond top-level package

Steps

  1. Make a setup.py file to the root level directory

The contents for the setup.py would be*

from setuptools import setup, find_packages

setup(name='myproject', version='1.0', packages=find_packages())
  1. Use a virtual environment

If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)

  • Create virtual env
    • python -m venv venv
  • Activate virtual env
    • source ./venv/bin/activate (Linux, macOS) or ./venv/Scripts/activate (Win)

To learn more about this, just Google out "python virtual env tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.

Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis

PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>

and your folder tree should look like this**

.
+-- myproject
¦   +-- api
¦   ¦   +-- api_key.py
¦   ¦   +-- api.py
¦   ¦   +-- __init__.py
¦   +-- examples
¦   ¦   +-- example_one.py
¦   ¦   +-- example_two.py
¦   ¦   +-- __init__.py
¦   +-- LICENCE.md
¦   +-- README.md
¦   +-- tests
¦       +-- __init__.py
¦       +-- test_one.py
+-- setup.py
+-- venv
    +-- Include
    +-- Lib
    +-- pyvenv.cfg
    +-- Scripts [87 entries exceeds filelimit, not opening dir]
  1. pip install your project in editable state

Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.

In the root directory, run

pip install -e . (note the dot, it stands for "current directory")

You can also see that it is installed by using pip freeze

(venv) PS C:\tmp\test_imports> pip install -e .
Obtaining file:///C:/tmp/test_imports
Installing collected packages: myproject
  Running setup.py develop for myproject
Successfully installed myproject
(venv) PS C:\tmp\test_imports> pip freeze
myproject==1.0
  1. Add myproject. into your imports

Note that you will have to add myproject. only into imports that would not work otherwise. Imports that worked without the setup.py & pip install will work still work fine. See an example below.


Test the solution

Now, let's test the solution using api.py defined above, and test_one.py defined below.

test_one.py

from myproject.api.api import function_from_api

def test_function():
    print(function_from_api())

if __name__ == '__main__':
    test_function()

running the test

(venv) PS C:\tmp\test_imports> python .\myproject\tests\test_one.py
I am the return value from api.api!

* See the setuptools docs for more verbose setup.py examples.

** In reality, you could put your virtual environment anywhere on your hard disk.


in your main file add this:

import sys
import os 
sys.path.append(os.path.abspath(os.path.join(__file__,mainScriptDepth)))

mainScriptDepth = the depth of the main file from the root of the project.

Here in your case mainScriptDepth = "../../".


You need to look to see how the import statements are written in the related code. If examples/example_one.py uses the following import statement:

import api.api

...then it expects the root directory of the project to be in the system path.

The easiest way to support this without any hacks (as you put it) would be to run the examples from the top level directory, like this:

PYTHONPATH=$PYTHONPATH:. python examples/example_one.py 

For siblings package imports, you can use either the insert or the append method of the [sys.path][2] module:

if __name__ == '__main__' and if __package__ is None:
    import sys
    from os import path
    sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
    import api

This will work if you are launching your scripts as follows:

python examples/example_one.py
python tests/test_one.py

On the other hand, you can also use the relative import:

if __name__ == '__main__' and if __package__ is not None:
    import ..api.api

In this case you will have to launch your script with the '-m' argument (note that, in this case, you must not give the '.py' extension):

python -m packageName.examples.example_one
python -m packageName.tests.test_one

Of course, you can mix the two approaches, so that your script will work no matter how it is called:

if __name__ == '__main__':
    if __package__ is None:
        import sys
        from os import path
        sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
        import api
    else:
        import ..api.api

I wanted to comment on the solution provided by np8 but I don't have enough reputation so I'll just mention that you can create a setup.py file exactly as they suggested, and then do pipenv install --dev -e . from the project root directory to turn it into an editable dependency. Then your absolute imports will work e.g. from api.api import foo and you don't have to mess around with system-wide installations.

Documentation


Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to packages

How to install Python packages from the tar.gz file without using pip install Package name does not correspond to the file path - IntelliJ Can I force pip to reinstall the current version? The import android.support cannot be resolved How to solve java.lang.NoClassDefFoundError? How to list all installed packages and their versions in Python? Importing packages in Java Check for installed packages before running install.packages() How do I find a list of Homebrew's installable packages? Installing a local module using npm?

Examples related to python-import

Python: 'ModuleNotFoundError' when trying to import module from imported package ImportError: libSM.so.6: cannot open shared object file: No such file or directory Relative imports - ModuleNotFoundError: No module named x ModuleNotFoundError: What does it mean __main__ is not a package? Reading file using relative path in python project What does from __future__ import absolute_import actually do? Why does using from __future__ import print_function breaks Python2-style print? Automatically create requirements.txt Error importing Seaborn module in Python Why can't Python import Image from PIL?

Examples related to siblings

How to display and hide a div with CSS? XPath:: Get following Sibling Sibling package imports What is syntax for selector in CSS for next element? jQuery: How to get to a particular child of a parent? Is there a way to select sibling nodes?