[python] How to write a Python module/package?

I've been making Python scripts for simple tasks at work and never really bothered packaging them for others to use. Now I have been assigned to make a Python wrapper for a REST API. I have absolutely no idea on how to start and I need help.

What I have:

(Just want to be specific as possible) I have the virtualenv ready, it's also up in github, the .gitignore file for python is there as well, plus, the requests library for interacting with the REST API. That's it.

Here's the current directory tree

.
+-- bin
¦   +-- /the usual stuff/
+-- include
¦   +-- /the usual stuff/
+-- lib
¦   +-- python2.7
¦       +-- /the usual stuff/
+-- local
¦   +-- /the usual stuff/
+-- README.md

27 directories, 280 files

I don't even know where to put the .py files, if I ever make one.

What I wanted to do:

Make a python module install-able with "pip install ..."

If possible, I want a general step by step process on writing Python modules.

This question is related to python python-2.7 python-module

The answer is


A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py

create hello.py then write the following function as its content:

def helloworld():
   print "hello"

Then you can import hello:

>>> import hello
>>> hello.helloworld()
'hello'
>>>

To group many .py files put them in a folder. Any folder with an __init__.py is considered a module by python and you can call them a package

|-HelloModule
  |_ __init__.py
  |_ hellomodule.py

You can go about with the import statement on your module the usual way.

For more information, see 6.4. Packages.


Python 3 - UPDATED 18th November 2015

Found the accepted answer useful, yet wished to expand on several points for the benefit of others based on my own experiences.

Module: A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py appended.

Module Example: Assume we have a single python script in the current directory, here I am calling it mymodule.py

The file mymodule.py contains the following code:

def myfunc():
    print("Hello!")

If we run the python3 interpreter from the current directory, we can import and run the function myfunc in the following different ways (you would typically just choose one of the following):

>>> import mymodule
>>> mymodule.myfunc()
Hello!
>>> from mymodule import myfunc
>>> myfunc()
Hello!
>>> from mymodule import *
>>> myfunc()
Hello!

Ok, so that was easy enough.

Now assume you have the need to put this module into its own dedicated folder to provide a module namespace, instead of just running it ad-hoc from the current working directory. This is where it is worth explaining the concept of a package.

Package: Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name A.B designates a submodule named B in a package named A. Just like the use of modules saves the authors of different modules from having to worry about each other’s global variable names, the use of dotted module names saves the authors of multi-module packages like NumPy or the Python Imaging Library from having to worry about each other’s module names.

Package Example: Let's now assume we have the following folder and files. Here, mymodule.py is identical to before, and __init__.py is an empty file:

.
+-- mypackage
    +-- __init__.py
    +-- mymodule.py

The __init__.py files are required to make Python treat the directories as containing packages. For further information, please see the Modules documentation link provided later on.

Our current working directory is one level above the ordinary folder called mypackage

$ ls
mypackage

If we run the python3 interpreter now, we can import and run the module mymodule.py containing the required function myfunc in the following different ways (you would typically just choose one of the following):

>>> import mypackage
>>> from mypackage import mymodule
>>> mymodule.myfunc()
Hello!
>>> import mypackage.mymodule
>>> mypackage.mymodule.myfunc()
Hello!
>>> from mypackage import mymodule
>>> mymodule.myfunc()
Hello!
>>> from mypackage.mymodule import myfunc
>>> myfunc()
Hello!
>>> from mypackage.mymodule import *
>>> myfunc()
Hello!

Assuming Python 3, there is excellent documentation at: Modules

In terms of naming conventions for packages and modules, the general guidelines are given in PEP-0008 - please see Package and Module Names

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.


Since nobody did cover this question of the OP yet:

What I wanted to do:

Make a python module install-able with "pip install ..."

Here is an absolute minimal example, showing the basic steps of preparing and uploading your package to PyPI using setuptools and twine.

This is by no means a substitute for reading at least the tutorial, there is much more to it than covered in this very basic example.

Creating the package itself is already covered by other answers here, so let us assume we have that step covered and our project structure like this:

.
+-- hellostackoverflow/
    +-- __init__.py
    +-- hellostackoverflow.py

In order to use setuptools for packaging, we need to add a file setup.py, this goes into the root folder of our project:

.
+-- setup.py
+-- hellostackoverflow/
    +-- __init__.py
    +-- hellostackoverflow.py

At the minimum, we specify the metadata for our package, our setup.py would look like this:

from setuptools import setup

setup(
    name='hellostackoverflow',
    version='0.0.1',
    description='a pip-installable package example',
    license='MIT',
    packages=['hellostackoverflow'],
    author='Benjamin Gerfelder',
    author_email='[email protected]',
    keywords=['example'],
    url='https://github.com/bgse/hellostackoverflow'
)

Since we have set license='MIT', we include a copy in our project as LICENCE.txt, alongside a readme file in reStructuredText as README.rst:

.
+-- LICENCE.txt
+-- README.rst
+-- setup.py
+-- hellostackoverflow/
    +-- __init__.py
    +-- hellostackoverflow.py

At this point, we are ready to go to start packaging using setuptools, if we do not have it already installed, we can install it with pip:

pip install setuptools

In order to do that and create a source distribution, at our project root folder we call our setup.py from the command line, specifying we want sdist:

python setup.py sdist

This will create our distribution package and egg-info, and result in a folder structure like this, with our package in dist:

.
+-- dist/
+-- hellostackoverflow.egg-info/
+-- LICENCE.txt
+-- README.rst
+-- setup.py
+-- hellostackoverflow/
    +-- __init__.py
    +-- hellostackoverflow.py

At this point, we have a package we can install using pip, so from our project root (assuming you have all the naming like in this example):

pip install ./dist/hellostackoverflow-0.0.1.tar.gz

If all goes well, we can now open a Python interpreter, I would say somewhere outside our project directory to avoid any confusion, and try to use our shiny new package:

Python 3.5.2 (default, Sep 14 2017, 22:51:06) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from hellostackoverflow import hellostackoverflow
>>> hellostackoverflow.greeting()
'Hello Stack Overflow!'

Now that we have confirmed the package installs and works, we can upload it to PyPI.

Since we do not want to pollute the live repository with our experiments, we create an account for the testing repository, and install twine for the upload process:

pip install twine

Now we're almost there, with our account created we simply tell twine to upload our package, it will ask for our credentials and upload our package to the specified repository:

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

We can now log into our account on the PyPI test repository and marvel at our freshly uploaded package for a while, and then grab it using pip:

pip install --index-url https://test.pypi.org/simple/ hellostackoverflow

As we can see, the basic process is not very complicated. As I said earlier, there is a lot more to it than covered here, so go ahead and read the tutorial for more in-depth explanation.


Once you have defined your chosen commands, you can simply drag and drop the saved file into the Lib folder in your python program files.

>>> import mymodule 
>>> mymodule.myfunc()

I created a project to easily initiate a project skeleton from scratch. https://github.com/MacHu-GWU/pygitrepo-project.

And you can create a test project, let's say, learn_creating_py_package.

You can learn what component you should have for different purpose like:

  • create virtualenv
  • install itself
  • run unittest
  • run code coverage
  • build document
  • deploy document
  • run unittest in different python version
  • deploy to PYPI

The advantage of using pygitrepo is that those tedious are automatically created itself and adapt your package_name, project_name, github_account, document host service, windows or macos or linux.

It is a good place to learn develop a python project like a pro.

Hope this could help.

Thank you.


Make a file named "hello.py"

If you are using Python 2.x

def func():
    print "Hello"

If you are using Python 3.x

def func():
    print("Hello")

Run the file. Then, you can try the following:

>>> import hello
>>> hello.func()
Hello

If you want a little bit hard, you can use the following:

If you are using Python 2.x

def say(text):
    print text

If you are using Python 3.x

def say(text):
    print(text)

See the one on the parenthesis beside the define? That is important. It is the one that you can use within the define.

Text - You can use it when you want the program to say what you want. According to its name, it is text. I hope you know what text means. It means "words" or "sentences".

Run the file. Then, you can try the following if you are using Python 3.x:

>>> import hello
>>> hello.say("hi")
hi
>>> from hello import say
>>> say("test")
test

For Python 2.x - I guess same thing with Python 3? No idea. Correct me if I made a mistake on Python 2.x (I know Python 2 but I am used with Python 3)


A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py

Create a file called hello.py with the following function as its content:

def helloworld():
   print "hello"

Then you can

import hello
hello.helloworld()

To group many .py files, put them in a folder. Any folder with an init.py is considered a module by python and you can call them a package.

|-HelloModule |_ init.py |_ hellomodule.py


Questions with python tag:

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation Upgrade to python 3.8 using conda Unable to allocate array with shape and data type How to fix error "ERROR: Command errored out with exit status 1: python." when trying to install django-heroku using pip How to prevent Google Colab from disconnecting? "UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure." when plotting figure with pyplot on Pycharm How to fix 'Object arrays cannot be loaded when allow_pickle=False' for imdb.load_data() function? "E: Unable to locate package python-pip" on Ubuntu 18.04 Tensorflow 2.0 - AttributeError: module 'tensorflow' has no attribute 'Session' Jupyter Notebook not saving: '_xsrf' argument missing from post How to Install pip for python 3.7 on Ubuntu 18? Python: 'ModuleNotFoundError' when trying to import module from imported package OpenCV TypeError: Expected cv::UMat for argument 'src' - What is this? Requests (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.") Error in PyCharm requesting website How to setup virtual environment for Python in VS Code? Pylint "unresolved import" error in Visual Studio Code Pandas Merging 101 Numpy, multiply array with scalar What is the meaning of "Failed building wheel for X" in pip install? Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed Could not install packages due to an EnvironmentError: [Errno 13] OpenCV !_src.empty() in function 'cvtColor' error ConvergenceWarning: Liblinear failed to converge, increase the number of iterations How to downgrade python from 3.7 to 3.6 I can't install pyaudio on Windows? How to solve "error: Microsoft Visual C++ 14.0 is required."? Iterating over arrays in Python 3 How do I install opencv using pip? How do I install Python packages in Google's Colab? How do I use TensorFlow GPU? How to upgrade Python version to 3.7? How to resolve TypeError: can only concatenate str (not "int") to str How can I install a previous version of Python 3 in macOS using homebrew? Flask at first run: Do not use the development server in a production environment TypeError: only integer scalar arrays can be converted to a scalar index with 1D numpy indices array What is the difference between Jupyter Notebook and JupyterLab? Pytesseract : "TesseractNotFound Error: tesseract is not installed or it's not in your path", how do I fix this? Could not install packages due to a "Environment error :[error 13]: permission denied : 'usr/local/bin/f2py'" How do I resolve a TesseractNotFoundError? Trying to merge 2 dataframes but get ValueError Authentication plugin 'caching_sha2_password' is not supported Python Pandas User Warning: Sorting because non-concatenation axis is not aligned

Questions with python-2.7 tag:

Numpy, multiply array with scalar Not able to install Python packages [SSL: TLSV1_ALERT_PROTOCOL_VERSION] How to create a new text file using Python Could not find a version that satisfies the requirement tensorflow Python: Pandas pd.read_excel giving ImportError: Install xlrd >= 0.9.0 for Excel support Display/Print one column from a DataFrame of Series in Pandas How to calculate 1st and 3rd quartiles? How can I read pdf in python? How to completely uninstall python 2.7.13 on Ubuntu 16.04 Check key exist in python dict Visual Studio Code pylint: Unable to import 'protorpc' WinError 2 The system cannot find the file specified (Python) How to plot vectors in python using matplotlib how to update spyder on anaconda python pip - install from local dir "pip install json" fails on Ubuntu Checking whether the pip is installed? pip or pip3 to install packages for Python 3? What is a good practice to check if an environmental variable exists or not? Invalid http_host header Append an empty row in dataframe using pandas What is the difference between json.load() and json.loads() functions pandas: find percentile stats of a given column how to run python files in windows command prompt? How to get the latest file in a folder? ln (Natural Log) in Python Drop all data in a pandas dataframe ImportError: No module named google.protobuf Using Keras & Tensorflow with AMD GPU Is it ok having both Anacondas 2.7 and 3.5 installed in the same time? ImportError: cannot import name NUMPY_MKL Opencv - Grayscale mode Vs gray color conversion What is the difference between json.dump() and json.dumps() in python? Read .doc file with python Pip install - Python 2.7 - Windows 7 Python Traceback (most recent call last) Switch between python 2.7 and python 3.5 on Mac OS X In Flask, What is request.args and how is it used? Python: how to capture image from webcam on click using OpenCV Download and save PDF file with Python requests module Python 101: Can't open file: No such file or directory How do I install Keras and Theano in Anaconda Python on Windows? Make new column in Panda dataframe by adding values from other columns Why I get 'list' object has no attribute 'items'? RuntimeError: module compiled against API version a but this version of numpy is 9 What does from __future__ import absolute_import actually do? The most efficient way to remove first N elements in a list? Python - Extracting and Saving Video Frames Python for and if on one line What is the right way to debug in iPython notebook?

Questions with python-module tag:

ImportError: libSM.so.6: cannot open shared object file: No such file or directory How to write a Python module/package? Python: OSError: [Errno 2] No such file or directory: '' Unable to import a module that is definitely installed Installing python module within code How to import other Python files? How do I find out my python path using python? What does if __name__ == "__main__": do? How to do relative imports in Python? How to import a module given the full path?