Import local function from a module housed in another directory with relative imports in Jupyter Notebook using Python 3

Question

I have a directory structure similar to the following  meta project     project1           init   py         lib             module py               init   py     notebook folder         notebook jpynb   When working in notebook jpynb if I try to use a relative import to access a function function   in module py with   from   project1 lib module import function   I get the following error   SystemError                               Traceback  most recent call last   lt ipython-input-7-6393744d93ab gt  in  lt module gt    ---- gt  1 from   project1 lib module import function  SystemError  Parent module    not loaded  cannot perform relative import   Is there any way to get this to work using relative imports    Note  the notebook server is instantiated at the level of the meta project directory  so it should have access to the information in those files    Note  also  that at least as originally intended project1 wasn t thought of as a module and therefore does not have an   init   py file  it was just meant as a file-system directory  If the solution to the problem requires treating it as a module and including an   init   py file  even a blank one  that is fine  but doing so is not enough to solve the problem   I share this directory between machines and relative imports allow me to use the same code everywhere   amp  I often use notebooks for quick prototyping  so suggestions that involve hacking together absolute paths are unlikely to be helpful     Edit  This is unlike Relative imports in Python 3  which talks about relative imports in Python 3 in general and     in particular     running a script from within a package directory  This has to do with working within a jupyter notebook trying to call a function in a local module in another directory which has both different general and particular aspects

User · Answer

I have found that python-dotenv helps solve this issue pretty effectively. Your project structure ends up changing slightly, but the code in your notebook is a bit simpler and consistent across notebooks.

For your project, do a little install.

pipenv install python-dotenv

Then, project changes to:

+-- .env (this can be empty)
+-- ipynb
¦   +-- 20170609-Examine_Database_Requirements.ipynb
¦   +-- 20170609-Initial_Database_Connection.ipynb
+-- lib
    +-- __init__.py
    +-- postgres.py

And finally, your import changes to:

import os
import sys

from dotenv import find_dotenv


sys.path.append(os.path.dirname(find_dotenv()))

A +1 for this package is that your notebooks can be several directories deep. python-dotenv will find the closest one in a parent directory and use it. A +2 for this approach is that jupyter will load environment variables from the .env file on startup. Double whammy.

User · Answer

I have just found this pretty solution   import sys  sys path insert 0          add parent folder path where lib folder is import lib store load   store load is a file on my library folder   You just want some functions of that file  from lib store load import your function name   If python version    3 3 you do not need init py file in the folder

User · Answer

So far  the accepted answer has worked best for me  However  my concern has always been that there is a likely scenario where I might refactor the notebooks directory into subdirectories  requiring to change the module path in every notebook  I decided to add a python file within each notebook directory to import the required modules    Thus  having the following project structure    project    notebooks       explore           notebook1 ipynb           notebook2 ipynb           project path py        explain           notebook1 ipynb           project path py    lib          init   py        module py   I added the file project path py in each notebook subdirectory  notebooks explore and notebooks explain   This file contains the code for relative imports  from  metakermit    import sys import os  module path   os path abspath os path join os pardir  os pardir   if module path not in sys path      sys path append module path    This way  I just need to do relative imports within the project path py file  and not in the notebooks  The notebooks files would then just need to import project path before importing lib  For example in 0 0-notebook ipynb   import project path import lib   The caveat here is that reversing the imports would not work  THIS DOES NOT WORK    import lib import project path   Thus care must be taken during imports

User · Answer

Here s my 2 cents  import sys map the path where the module file is located  In my case it was the desktop sys path append   Users John Desktop   Either import the whole mapping module BUT then you have to use the  notation to map the classes like mapping Shipping   import mapping  mapping py is the name of my module file shipit   mapping Shipment    Shipment is the name of the class I need to use in the mapping module Or import the specific class from the mapping module from mapping import Mapping shipit   Shipment    Now you don t have to use the  notation

User · Answer

All other answers here depends on adding code the the notebook    In my opinion is bad practice to hardcode a specific path into the notebook code  or otherwise depend on the location  since this makes it really hard to refactor you code later on  Instead I would recommend you to add the root project folder to PYTHONPATH when starting up your Jupyter notebook server  either directly from the project folder like so env PYTHONPATH  pwd  jupyter notebook or if you are starting it up from somewhere else  use the absolute path like so env PYTHONPATH  Users foo bar project  jupyter notebook

User · Answer

Researching this topic myself and having read the answers I recommend using the path py library since it provides a context manager for changing the current working directory   You then have something like  import path if path Path     lib   isdir        with path Path                import lib   Although  you might just omit the isdir statement   Here I ll add print statements to make it easy to follow what s happening  import path import pandas  print path Path getcwd    print path Path     lib   isdir    if path Path     lib   isdir        with path Path                print path Path getcwd            import lib         print  Success    print path Path getcwd      which outputs in this example  where lib is at  home jovyan shared notebooks by-team data-vis demos lib     home jovyan shared notebooks by-team data-vis demos custom-chart  home jovyan shared notebooks by-team data-vis demos  home jovyan shared notebooks by-team data-vis demos custom-chart   Since the solution uses a context manager  you are guaranteed to go back to your previous working directory  no matter what state your kernel was in before the cell and no matter what exceptions are thrown by importing your library code

User · Answer

Came here searching for best practices in abstracting code to submodules when working in Notebooks  I m not sure that there is a best practice  I have been proposing this    A project hierarchy as such    -- ipynb         -- 20170609-Examine Database Requirements ipynb         -- 20170609-Initial Database Connection ipynb  -- lib      --   init   py      -- postgres py   And from 20170609-Initial Database Connection ipynb       In  1   cd         In  2   from lib postgres import database connection   This works because by default the Jupyter Notebook can parse the cd command  Note that this does not make use of Python Notebook magic  It simply works without prepending  bash    Considering that 99 times out of a 100 I am working in Docker using one of the Project Jupyter Docker images  the following modification is idempotent      In  1   cd  home jovyan      In  2   from lib postgres import database connection

User · Answer

I had almost the same example as you in this notebook where I wanted to illustrate the usage of an adjacent module s function in a DRY manner   My solution was to tell Python of that additional module import path by adding a snippet like this one to the notebook   import os import sys module path   os path abspath os path join        if module path not in sys path      sys path append module path    This allows you to import the desired function from the module hierarchy   from project1 lib module import function   use the function normally function        Note that it is necessary to add empty   init   py files to project1  and lib  folders if you don t have them already

[python] Import local function from a module housed in another directory with relative imports in Jupyter Notebook using Python 3

Examples related to python

Examples related to jupyter-notebook

Examples related to relative-import