What is the best project structure for a Python application

Question

Imagine that you want to develop a non-trivial end-user desktop  not web  application in Python  What is the best way to structure the project s folder hierarchy   Desirable features are ease of maintenance  IDE-friendliness  suitability for source control branching merging  and easy generation of install packages   In particular    Where do you put the source  Where do you put application startup scripts  Where do you put the IDE project cruft  Where do you put the unit acceptance tests  Where do you put non-Python data such as config files  Where do you put non-Python sources such as C   for pyd so binary extension modules

User · Answer

This blog post by Jean-Paul Calderone is commonly given as an answer in #python on Freenode.

Filesystem structure of a Python project

Do:

name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files Twisted. When you do releases, you should include a version number suffix: Twisted-2.5.

create a directory Twisted/bin and put your executables there, if you have any. Don't give them a .py extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects. (Slight wrinkle: since on Windows, the interpreter is selected by the file extension, your Windows users actually do want the .py extension. So, when you package for Windows, you may want to add it. Unfortunately there's no easy distutils trick that I know of to automate this process. Considering that on POSIX the .py extension is a only a wart, whereas on Windows the lack is an actual bug, if your userbase includes Windows users, you may want to opt to just have the .py extension everywhere.)

If your project is expressable as a single Python source file, then put it into the directory and name it something related to your project. For example, Twisted/twisted.py. If you need multiple source files, create a package instead (Twisted/twisted/, with an empty Twisted/twisted/__init__.py) and place your source files in it. For example, Twisted/twisted/internet.py.

put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example, Twisted/twisted/test/. Of course, make it a package with Twisted/twisted/test/__init__.py. Place tests in files like Twisted/twisted/test/test_internet.py.

add Twisted/README and Twisted/setup.py to explain and install your software, respectively, if you're feeling nice.

Don't:

put your source in a directory called src or lib. This makes it hard to run without installing.

put your tests outside of your Python package. This makes it hard to run the tests against an installed version.

create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module instead of a package, it's simpler.

try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.

User · Answer

Non-python data is best bundled inside your Python modules using the package data support in setuptools  One thing I strongly recommend is using namespace packages to create shared namespaces which multiple projects can use -- much like the Java convention of putting packages in com yourcompany yourproject  and being able to have a shared com yourcompany utils namespace    Re branching and merging  if you use a good enough source control system it will handle merges even through renames  Bazaar is particularly good at this   Contrary to some other answers here  I m  1 on having a src directory top-level  with doc and test directories alongside   Specific conventions for documentation directory trees will vary depending on what you re using  Sphinx  for instance  has its own conventions which its quickstart tool supports   Please  please leverage setuptools and pkg resources  this makes it much easier for other projects to rely on specific versions of your code  and for multiple versions to be simultaneously installed with different non-code files  if you re using package data

User · Answer

Check out Open Sourcing a Python Project the Right Way  Let me excerpt the project layout part of that excellent article   When setting up a project  the layout  or directory structure  is important to get right  A sensible layout means that potential contributors don t have to spend forever hunting for a piece of code  file locations are intuitive  Since we re dealing with an existing project  it means you ll probably need to move some stuff around  Let s start at the top  Most projects have a number of top-level files  like setup py  README md  requirements txt  etc   There are then three directories that every project should have   A docs directory containing project documentation A directory named with the project s name which stores the actual Python package A test directory in one of two places  Under the package directory containing test code and resources As a stand-alone top level directory To get a better sense of how your files should be organized  here s a simplified snapshot of the layout for one of my projects  sandman        pwd   code sandman   tree    - LICENSE  - README md  - TODO md  - docs      -- conf py      -- generated      -- index rst      -- installation rst      -- modules rst      -- quickstart rst      -- sandman rst  - requirements txt  - sandman      --   init   py      -- exception py      -- model py      -- sandman py      -- test          -- models py          -- test sandman py  - setup py   As you can see  there are some top level files  a docs directory  generated is an empty directory where sphinx will put the generated documentation   a sandman directory  and a test directory under sandman

User · Answer

Doesn t too much matter   Whatever makes you happy will work   There aren t a lot of silly rules because Python projects can be simple     scripts or  bin for that kind of command-line interface stuff  tests for your tests  lib for your C-language libraries  doc for most documentation  apidoc for the Epydoc-generated API docs    And the top-level directory can contain README s  Config s and whatnot   The hard choice is whether or not to use a  src tree   Python doesn t have a distinction between  src   lib  and  bin like Java or C has   Since a top-level  src directory is seen by some as meaningless  your top-level directory can be the top-level architecture of your application     foo  bar  baz   I recommend putting all of this under the  name-of-my-product  directory   So  if you re writing an application named quux  the directory that contains all this stuff is named   quux   Another project s PYTHONPATH  then  can include  path to quux foo to reuse the QUUX foo module     In my case  since I use Komodo Edit  my IDE cuft is a single  KPF file   I actually put that in the top-level  quux directory  and omit adding it to SVN

User · Answer

In my experience  it s just a matter of iteration   Put your data and code wherever you think they go   Chances are  you ll be wrong anyway   But once you get a better idea of exactly how things are going to shape up  you re in a much better position to make these kinds of guesses   As far as extension sources  we have a Code directory under trunk that contains a directory for python and a directory for various other languages   Personally  I m more inclined to try putting any extension code into its own repository next time around   With that said  I go back to my initial point   don t make too big a deal out of it   Put it somewhere that seems to work for you   If you find something that doesn t work  it can  and should  be changed

User · Answer

According to Jean-Paul Calderone s Filesystem structure of a Python project   Project   -- bin       -- project    -- project       -- test           --   init   py          -- test main py               --   init   py      -- main py    -- setup py  -- README

User · Answer

Non-python data is best bundled inside your Python modules using the package data support in setuptools  One thing I strongly recommend is using namespace packages to create shared namespaces which multiple projects can use -- much like the Java convention of putting packages in com yourcompany yourproject  and being able to have a shared com yourcompany utils namespace    Re branching and merging  if you use a good enough source control system it will handle merges even through renames  Bazaar is particularly good at this   Contrary to some other answers here  I m  1 on having a src directory top-level  with doc and test directories alongside   Specific conventions for documentation directory trees will vary depending on what you re using  Sphinx  for instance  has its own conventions which its quickstart tool supports   Please  please leverage setuptools and pkg resources  this makes it much easier for other projects to rely on specific versions of your code  and for multiple versions to be simultaneously installed with different non-code files  if you re using package data

User · Answer

Doesn t too much matter   Whatever makes you happy will work   There aren t a lot of silly rules because Python projects can be simple     scripts or  bin for that kind of command-line interface stuff  tests for your tests  lib for your C-language libraries  doc for most documentation  apidoc for the Epydoc-generated API docs    And the top-level directory can contain README s  Config s and whatnot   The hard choice is whether or not to use a  src tree   Python doesn t have a distinction between  src   lib  and  bin like Java or C has   Since a top-level  src directory is seen by some as meaningless  your top-level directory can be the top-level architecture of your application     foo  bar  baz   I recommend putting all of this under the  name-of-my-product  directory   So  if you re writing an application named quux  the directory that contains all this stuff is named   quux   Another project s PYTHONPATH  then  can include  path to quux foo to reuse the QUUX foo module     In my case  since I use Komodo Edit  my IDE cuft is a single  KPF file   I actually put that in the top-level  quux directory  and omit adding it to SVN

User · Answer

Non-python data is best bundled inside your Python modules using the package data support in setuptools  One thing I strongly recommend is using namespace packages to create shared namespaces which multiple projects can use -- much like the Java convention of putting packages in com yourcompany yourproject  and being able to have a shared com yourcompany utils namespace    Re branching and merging  if you use a good enough source control system it will handle merges even through renames  Bazaar is particularly good at this   Contrary to some other answers here  I m  1 on having a src directory top-level  with doc and test directories alongside   Specific conventions for documentation directory trees will vary depending on what you re using  Sphinx  for instance  has its own conventions which its quickstart tool supports   Please  please leverage setuptools and pkg resources  this makes it much easier for other projects to rely on specific versions of your code  and for multiple versions to be simultaneously installed with different non-code files  if you re using package data

User · Answer

Doesn t too much matter   Whatever makes you happy will work   There aren t a lot of silly rules because Python projects can be simple     scripts or  bin for that kind of command-line interface stuff  tests for your tests  lib for your C-language libraries  doc for most documentation  apidoc for the Epydoc-generated API docs    And the top-level directory can contain README s  Config s and whatnot   The hard choice is whether or not to use a  src tree   Python doesn t have a distinction between  src   lib  and  bin like Java or C has   Since a top-level  src directory is seen by some as meaningless  your top-level directory can be the top-level architecture of your application     foo  bar  baz   I recommend putting all of this under the  name-of-my-product  directory   So  if you re writing an application named quux  the directory that contains all this stuff is named   quux   Another project s PYTHONPATH  then  can include  path to quux foo to reuse the QUUX foo module     In my case  since I use Komodo Edit  my IDE cuft is a single  KPF file   I actually put that in the top-level  quux directory  and omit adding it to SVN

User · Answer

Check out Open Sourcing a Python Project the Right Way  Let me excerpt the project layout part of that excellent article   When setting up a project  the layout  or directory structure  is important to get right  A sensible layout means that potential contributors don t have to spend forever hunting for a piece of code  file locations are intuitive  Since we re dealing with an existing project  it means you ll probably need to move some stuff around  Let s start at the top  Most projects have a number of top-level files  like setup py  README md  requirements txt  etc   There are then three directories that every project should have   A docs directory containing project documentation A directory named with the project s name which stores the actual Python package A test directory in one of two places  Under the package directory containing test code and resources As a stand-alone top level directory To get a better sense of how your files should be organized  here s a simplified snapshot of the layout for one of my projects  sandman        pwd   code sandman   tree    - LICENSE  - README md  - TODO md  - docs      -- conf py      -- generated      -- index rst      -- installation rst      -- modules rst      -- quickstart rst      -- sandman rst  - requirements txt  - sandman      --   init   py      -- exception py      -- model py      -- sandman py      -- test          -- models py          -- test sandman py  - setup py   As you can see  there are some top level files  a docs directory  generated is an empty directory where sphinx will put the generated documentation   a sandman directory  and a test directory under sandman

User · Answer

The  Python Packaging Authority  has a sampleproject   https   github com pypa sampleproject  It is a sample project that exists as an aid to the Python Packaging User Guide s Tutorial on Packaging and Distributing Projects

User · Answer

This blog post by Jean-Paul Calderone is commonly given as an answer in #python on Freenode.

Filesystem structure of a Python project

Do:

name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files Twisted. When you do releases, you should include a version number suffix: Twisted-2.5.

create a directory Twisted/bin and put your executables there, if you have any. Don't give them a .py extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects. (Slight wrinkle: since on Windows, the interpreter is selected by the file extension, your Windows users actually do want the .py extension. So, when you package for Windows, you may want to add it. Unfortunately there's no easy distutils trick that I know of to automate this process. Considering that on POSIX the .py extension is a only a wart, whereas on Windows the lack is an actual bug, if your userbase includes Windows users, you may want to opt to just have the .py extension everywhere.)

If your project is expressable as a single Python source file, then put it into the directory and name it something related to your project. For example, Twisted/twisted.py. If you need multiple source files, create a package instead (Twisted/twisted/, with an empty Twisted/twisted/__init__.py) and place your source files in it. For example, Twisted/twisted/internet.py.

put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example, Twisted/twisted/test/. Of course, make it a package with Twisted/twisted/test/__init__.py. Place tests in files like Twisted/twisted/test/test_internet.py.

add Twisted/README and Twisted/setup.py to explain and install your software, respectively, if you're feeling nice.

Don't:

put your source in a directory called src or lib. This makes it hard to run without installing.

put your tests outside of your Python package. This makes it hard to run the tests against an installed version.

create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module instead of a package, it's simpler.

try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.

User · Answer

In my experience  it s just a matter of iteration   Put your data and code wherever you think they go   Chances are  you ll be wrong anyway   But once you get a better idea of exactly how things are going to shape up  you re in a much better position to make these kinds of guesses   As far as extension sources  we have a Code directory under trunk that contains a directory for python and a directory for various other languages   Personally  I m more inclined to try putting any extension code into its own repository next time around   With that said  I go back to my initial point   don t make too big a deal out of it   Put it somewhere that seems to work for you   If you find something that doesn t work  it can  and should  be changed

User · Answer

Try starting the project using the python boilerplate template  It largely follows the best practices  e g  those here   but is better suited in case you find yourself willing to split your project into more than one egg at some point  and believe me  with anything but the simplest projects  you will  One common situation is where you have to use a locally-modified version of someone else s library     Where do you put the source    For decently large projects it makes sense to split the source into several eggs  Each egg would go as a separate setuptools-layout under PROJECT ROOT src  lt egg name gt    Where do you put application startup scripts    The ideal option is to have application startup script registered as an entry point in one of the eggs   Where do you put the IDE project cruft    Depends on the IDE  Many of them keep their stuff in PROJECT ROOT   lt something gt  in the root of the project  and this is fine   Where do you put the unit acceptance tests    Each egg has a separate set of tests  kept in its PROJECT ROOT src  lt egg name gt  tests directory  I personally prefer to use py test to run them   Where do you put non-Python data such as config files    It depends  There can be different types of non-Python data     Resources   i e  data that must be packaged within an egg  This data goes into the corresponding egg directory  somewhere within package namespace  It can be used via the pkg resources package from setuptools  or since Python 3 7 via the importlib resources module from the standard library   Config-files   i e  non-Python files that are to be regarded as external to the project source files  but have to be initialized with some values when application starts running  During development I prefer to keep such files in PROJECT ROOT config  For deployment there can be various options  On Windows one can use  APP DATA   lt app-name gt  config  on Linux   etc  lt app-name gt  or  opt  lt app-name gt  config  Generated files  i e  files that may be created or modified by the application during execution  I would prefer to keep them in PROJECT ROOT var during development  and under  var during Linux deployment    Where do you put non-Python sources such as C   for pyd so binary extension modules    Into PROJECT ROOT src  lt egg name gt  native    Documentation would typically go into PROJECT ROOT doc or PROJECT ROOT src  lt egg name gt  doc  this depends on whether you regard some of the eggs to be a separate large projects   Some additional configuration will be in files like PROJECT ROOT buildout cfg and PROJECT ROOT setup cfg

User · Answer

The  Python Packaging Authority  has a sampleproject   https   github com pypa sampleproject  It is a sample project that exists as an aid to the Python Packaging User Guide s Tutorial on Packaging and Distributing Projects

User · Answer

Try starting the project using the python boilerplate template  It largely follows the best practices  e g  those here   but is better suited in case you find yourself willing to split your project into more than one egg at some point  and believe me  with anything but the simplest projects  you will  One common situation is where you have to use a locally-modified version of someone else s library     Where do you put the source    For decently large projects it makes sense to split the source into several eggs  Each egg would go as a separate setuptools-layout under PROJECT ROOT src  lt egg name gt    Where do you put application startup scripts    The ideal option is to have application startup script registered as an entry point in one of the eggs   Where do you put the IDE project cruft    Depends on the IDE  Many of them keep their stuff in PROJECT ROOT   lt something gt  in the root of the project  and this is fine   Where do you put the unit acceptance tests    Each egg has a separate set of tests  kept in its PROJECT ROOT src  lt egg name gt  tests directory  I personally prefer to use py test to run them   Where do you put non-Python data such as config files    It depends  There can be different types of non-Python data     Resources   i e  data that must be packaged within an egg  This data goes into the corresponding egg directory  somewhere within package namespace  It can be used via the pkg resources package from setuptools  or since Python 3 7 via the importlib resources module from the standard library   Config-files   i e  non-Python files that are to be regarded as external to the project source files  but have to be initialized with some values when application starts running  During development I prefer to keep such files in PROJECT ROOT config  For deployment there can be various options  On Windows one can use  APP DATA   lt app-name gt  config  on Linux   etc  lt app-name gt  or  opt  lt app-name gt  config  Generated files  i e  files that may be created or modified by the application during execution  I would prefer to keep them in PROJECT ROOT var during development  and under  var during Linux deployment    Where do you put non-Python sources such as C   for pyd so binary extension modules    Into PROJECT ROOT src  lt egg name gt  native    Documentation would typically go into PROJECT ROOT doc or PROJECT ROOT src  lt egg name gt  doc  this depends on whether you regard some of the eggs to be a separate large projects   Some additional configuration will be in files like PROJECT ROOT buildout cfg and PROJECT ROOT setup cfg

User · Answer

Doesn t too much matter   Whatever makes you happy will work   There aren t a lot of silly rules because Python projects can be simple     scripts or  bin for that kind of command-line interface stuff  tests for your tests  lib for your C-language libraries  doc for most documentation  apidoc for the Epydoc-generated API docs    And the top-level directory can contain README s  Config s and whatnot   The hard choice is whether or not to use a  src tree   Python doesn t have a distinction between  src   lib  and  bin like Java or C has   Since a top-level  src directory is seen by some as meaningless  your top-level directory can be the top-level architecture of your application     foo  bar  baz   I recommend putting all of this under the  name-of-my-product  directory   So  if you re writing an application named quux  the directory that contains all this stuff is named   quux   Another project s PYTHONPATH  then  can include  path to quux foo to reuse the QUUX foo module     In my case  since I use Komodo Edit  my IDE cuft is a single  KPF file   I actually put that in the top-level  quux directory  and omit adding it to SVN

User · Answer

Non-python data is best bundled inside your Python modules using the package data support in setuptools  One thing I strongly recommend is using namespace packages to create shared namespaces which multiple projects can use -- much like the Java convention of putting packages in com yourcompany yourproject  and being able to have a shared com yourcompany utils namespace    Re branching and merging  if you use a good enough source control system it will handle merges even through renames  Bazaar is particularly good at this   Contrary to some other answers here  I m  1 on having a src directory top-level  with doc and test directories alongside   Specific conventions for documentation directory trees will vary depending on what you re using  Sphinx  for instance  has its own conventions which its quickstart tool supports   Please  please leverage setuptools and pkg resources  this makes it much easier for other projects to rely on specific versions of your code  and for multiple versions to be simultaneously installed with different non-code files  if you re using package data

User · Answer

In my experience  it s just a matter of iteration   Put your data and code wherever you think they go   Chances are  you ll be wrong anyway   But once you get a better idea of exactly how things are going to shape up  you re in a much better position to make these kinds of guesses   As far as extension sources  we have a Code directory under trunk that contains a directory for python and a directory for various other languages   Personally  I m more inclined to try putting any extension code into its own repository next time around   With that said  I go back to my initial point   don t make too big a deal out of it   Put it somewhere that seems to work for you   If you find something that doesn t work  it can  and should  be changed

User · Answer

According to Jean-Paul Calderone s Filesystem structure of a Python project   Project   -- bin       -- project    -- project       -- test           --   init   py          -- test main py               --   init   py      -- main py    -- setup py  -- README

User · Answer

In my experience  it s just a matter of iteration   Put your data and code wherever you think they go   Chances are  you ll be wrong anyway   But once you get a better idea of exactly how things are going to shape up  you re in a much better position to make these kinds of guesses   As far as extension sources  we have a Code directory under trunk that contains a directory for python and a directory for various other languages   Personally  I m more inclined to try putting any extension code into its own repository next time around   With that said  I go back to my initial point   don t make too big a deal out of it   Put it somewhere that seems to work for you   If you find something that doesn t work  it can  and should  be changed

[python] What is the best project structure for a Python application?

Filesystem structure of a Python project

Examples related to python

Examples related to directory-structure

Examples related to organization

Examples related to project-structure