[python] Is __init__.py not required for packages in Python 3.3+

I am using Python 3.5.1. I read the document and the package section here: https://docs.python.org/3/tutorial/modules.html#packages

Now, I have the following structure:

/home/wujek/Playground/a/b/module.py

module.py:

class Foo:
    def __init__(self):
        print('initializing Foo')

Now, while in /home/wujek/Playground:

~/Playground $ python3
>>> import a.b.module
>>> a.b.module.Foo()
initializing Foo
<a.b.module.Foo object at 0x100a8f0b8>

Similarly, now in home, superfolder of Playground:

~ $ PYTHONPATH=Playground python3
>>> import a.b.module
>>> a.b.module.Foo()
initializing Foo
<a.b.module.Foo object at 0x10a5fee10>

Actually, I can do all kinds of stuff:

~ $ PYTHONPATH=Playground python3
>>> import a
>>> import a.b
>>> import Playground.a.b

Why does this work? I though there needed to be __init__.py files (empty ones would work) in both a and b for module.py to be importable when the Python path points to the Playground folder?

This seems to have changed from Python 2.7:

~ $ PYTHONPATH=Playground python
>>> import a
ImportError: No module named a
>>> import a.b
ImportError: No module named a.b
>>> import a.b.module
ImportError: No module named a.b.module

With __init__.py in both ~/Playground/a and ~/Playground/a/b it works fine.

This question is related to python python-3.x package

The answer is


I would say that one should omit the __init__.py only if one wants to have the implicit namespace package. If you don't know what it means, you probably don't want it and therefore you should continue to use the __init__.py even in Python 3.


Based on my experience, even with python 3.3+, an empty __init__.py is still needed sometimes. One situation is when you want to refer a subfolder as a package. For example, when I ran python -m test.foo, it didn't work until I created an empty __init__.py under the test folder. And I'm talking about 3.6.6 version here which is pretty recent.

Apart from that, even for reasons of compatibility with existing source code or project guidelines, its nice to have an empty __init__.py in your package folder.


If you have setup.py in your project and you use find_packages() within it, it is necessary to have an __init__.py file in every directory for packages to be automatically found.

Packages are only recognized if they include an __init__.py file

UPD: If you want to use implicit namespace packages without __init__.py you just have to use find_namespace_packages() instead

Docs


Overview

@Mike's answer is correct but too imprecise. It is true that Python 3.3+ supports Implicit Namespace Packages that allows it to create a package without an __init__.py file. This is called a namespace package in contrast to a regular package which does have an __init__.py file (empty or not empty).

However, creating a namespace package should ONLY be done if there is a need for it. For most use cases and developers out there, this doesn't apply so you should stick with EMPTY __init__.py files regardless.

Namespace package use case

To demonstrate the difference between the two types of python packages, lets look at the following example:

google_pubsub/              <- Package 1
    google/                 <- Namespace package (there is no __init__.py)
        cloud/              <- Namespace package (there is no __init__.py)
            pubsub/         <- Regular package (with __init__.py)
                __init__.py <- Required to make the package a regular package
                foo.py

google_storage/             <- Package 2
    google/                 <- Namespace package (there is no __init__.py)
        cloud/              <- Namespace package (there is no __init__.py)
            storage/        <- Regular package (with __init__.py)
                __init__.py <- Required to make the package a regular package
                bar.py

google_pubsub and google_storage are separate packages but they share the same namespace google/cloud. In order to share the same namespace, it is required to make each directory of the common path a namespace package, i.e. google/ and cloud/. This should be the only use case for creating namespace packages, otherwise, there is no need for it.

It's crucial that there are no __init__py files in the google and google/cloud directories so that both directories can be interpreted as namespace packages. In Python 3.3+ any directory on the sys.path with a name that matches the package name being looked for will be recognized as contributing modules and subpackages to that package. As a result, when you import both from google_pubsub and google_storage, the Python interpreter will be able to find them.

This is different from regular packages which are self-contained meaning all parts live in the same directory hierarchy. When importing a package and the Python interpreter encounters a subdirectory on the sys.path with an __init__.py file, then it will create a single directory package containing only modules from that directory, rather than finding all appropriately named subdirectories outside that directory. This is perfectly fine for packages that don't want to share a namespace. I highly recommend taking a look at Traps for the Unwary in Python’s Import System to get a better understanding of how Python importing behaves with regular and namespace package and what __init__.py traps to watch out for.

Summary

  • Only skip __init__.py files if you want to create namespace packages. Only create namespace packages if you have different libraries that reside in different locations and you want them each to contribute a subpackage to the parent package, i.e. the namespace package.
  • Keep on adding empty __init__py to your directories because 99% of the time you just want to create regular packages. Also, Python tools out there such as mypy and pytest require empty __init__.py files to interpret the code structure accordingly. This can lead to weird errors if not done with care.

Resources

My answer only touches the surface of how regular packages and namespace packages work so take a look at the following resources for further information:


Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to python-3.x

Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation Replace specific text with a redacted version using Python Upgrade to python 3.8 using conda "Permission Denied" trying to run Python on Windows 10 Python: 'ModuleNotFoundError' when trying to import module from imported package What is the meaning of "Failed building wheel for X" in pip install? How to downgrade python from 3.7 to 3.6 I can't install pyaudio on Windows? How to solve "error: Microsoft Visual C++ 14.0 is required."? Iterating over arrays in Python 3 How to upgrade Python version to 3.7?

Examples related to package

ModuleNotFoundError: No module named 'sklearn' Python: How to pip install opencv2 with specific version 2.4.9? Relative imports - ModuleNotFoundError: No module named x Is __init__.py not required for packages in Python 3.3+ "pip install unroll": "python setup.py egg_info" failed with error code 1 Unable to Install Any Package in Visual Studio 2015 beyond top level package error in relative import How can I specify the required Node.js version in package.json? "installation of package 'FILE_PATH' had non-zero exit status" in R Error in installation a R package