[python] pytest cannot import module while python can

I am working on a package in Python. I use virtualenv. I set the path to the root of the module in a .pth path in my virtualenv, so that I can import modules of the package while developing the code and do testing (Question 1: is it a good way to do?). This works fine (here is an example, this is the behavior I want):

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py 
issued command: echo hello
command output: hello

However, if I try to use PyTest, I get some import error messages:

(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile: 
collected 0 items / 1 errors 

================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
    from rc import ns
E   ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz@zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest

I am a bit puzzled, it looks like this indicates an import error, but Python does it fine so why is there a problem specifically with PyTest? Any suggestion to the reason / remedy (Question 2)? I googled and stack-overflowed the 'ImportError: cannot import' error for PyTest, but the hits I got were related to missing python path and remedy to this, which does not seem to be the problem here. Any suggestions?

I had a similar issue, exact same error, but different cause. I was running the test code just fine, but against an old version of the module. In the previous version of my code one class existed, while the other did not. After updating my code, I should have run the following to install it.

sudo pip install ./ --upgrade

Upon installing the updated module running pytest produced the correct results (because i was using the correct code base).

In my case I am working in a container and unfortuantely pytest has the tendency to use python2.7 rather than my python3 interpreter of choice.

In my case this worked:

python3 -m pytest

My folder structure

/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md

I disagree with the posts saying that you must remove any __init__.py files. What you must instead do is alter the sys.path.

Run an experiment where you print sys.path when running the code normally. Then print sys.path while running the code via pytest. I think you will find that there is a difference between these two paths, hence why pytest breaks.

To fix this, insert the path from the first experiment at the 0th index of the second.

Let '/usr/exampleUser/Documents/foo' be the first element of print(sys.path) for experiment 1.

Below is code that should fix your issue:

import sys sys.path[0] = '/usr/exampleUser/Documents/foo'

Put this at the top of your file, before your actual import statement.

Source: I was dealing with this myself and the above process solved it.

Kept everything same and just added a blank test file at the root folder .. Solved it

Here are the findings, this problem really bugged me for a while. My folder structure was

mathapp/
    - server.py  
    - configuration.py 
    - __init__.py 
    - static/ 
       - home.html  
tests/            
    - functional 
       - test_errors.py 
    - unit  
       - test_add.py

and pytest would complain with the ModuleNotFoundError and gives the HINT - make sure your test modules/packages have valid Python names.

I introduced a mock test file at the same level as mathsapp and tests directory. The file contained nothing. Now pytest does not complain.

Result without the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 1 item / 1 error

=================================== ERRORS ====================================
_______________ ERROR collecting tests/functional/test_func.py ________________
ImportError while importing test module 'C:\mainak\workspace\0github\python-rest-app-cont\tests\functional\test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\functional\test_func.py:4: in <module>
    from mathapp.service import sum
E   ModuleNotFoundError: No module named 'mathapp'
=========================== short test summary info ===========================
ERROR tests/functional/test_func.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.24s ===============================

Results with the file

$ pytest
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\mak2006\workspace\0github\python-rest-app-cont
collected 2 items

tests\functional\test_func.py .                                          [ 50%]
tests\unit\test_unit.py .                                                [100%]

============================== 2 passed in 0.11s ==============================

Install the packages into Your virtual environment.
Then start a new shell and source Your virtual environment again.

Had a similar issue and it worked when I added __init__.py file under tests directory.

I had the same problem but for another reason than the ones mentioned:

I had py.test installed globally, while the packages were installed in a virtual environment.

The solution was to install pytest in the virtual environment. (In case your shell hashes executables, as Bash does, use hash -r, or use the full path to py.test)

I was getting this using VSCode. I have a conda environment. I don't think the VScode python extension could see the updates I was making.

python c:\Users\brig\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear test
Test Discovery failed:

I had to run pip install ./ --upgrade

In my case, the import error occurred because the package is pointing to another package/directory with the same name and its path is one level above the folder I actually wanted. I think this also explains why some people need to remove _ init _.py while others need to add back.

I just put print(the_root_package.__path__) (after import the_root_package) in both python console and pytest scripts to compare the difference

BOTTOM LINE: When you do python, the package you import may be different from the package when you run pytest.

I just solved this by removing the __init__.py in my project root:

.
+-- __init__.py <--- removed
+-- models
¦   +-- __init__.py
¦   +-- address.py
¦   +-- appointment.py
¦   +-- client.py
+-- requirements.txt
+-- setup.cfg
+-- tests
¦   +-- __init__.py
¦   +-- models
¦   ¦   +-- __init__.py
¦   ¦   +-- appointment_test.py
¦   ¦   +-- client_test.py
¦   +-- other_test.py
+-- script.py

Yet another massive win for Python's import system. I think the reason there is no consensus is that what works probably depends on your environment and the tools you are using on top of it.

I'm using this from VS Code, in the test explorer under Windows in a conda environment, Python 3.8.

The setup I have got to work is:

mypkg/
    __init__.py
    app.py
    view.py
tests/
    test_app.py
    test_view.py

Under this setup intellisense works and so does test discovery.

Note that I originally tried the following, as recommended here.

src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    test_app.py
    test_view.py

I could find no way of getting this to work from VS Code because the src folder just blew the mind of the import system. I can imagine there is a way of getting this to work from the command line. As a relatively new convert to Python programming it gives me a nostalgic feeling of working with COM, but being slightly less fun.

My 2 cents on this: pytest will fail at chance if you are not using virtual environments. Sometimes it will just work, sometimes not.

Therefore, the solution is:

• remove pytest with pip uninstall
• create your venv
• activate your venv
• pip install your project path in editable mode, so it will be treated by pytest as a module (otherwise, pytest wont find your internal imports). You will need a setup.py file for that
• install your packages, including pytest
• finally, run your tests

The code, using windows PowerShell:

pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy

Then finally

(my_env) pytest --html="my_testing_report.html"

An example of setup.py, for pip install -e:

import setuptools

setuptools.setup(
    name='my_package',
    version='devel',
    author='erickfis',
    author_email='[email protected]',
    description='My package',
    long_description='My gooood package',
    packages=setuptools.find_packages(),
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
    ],
    include_package_data=True
)

if you need a init.py file in your folder make a copy of the folder and delete init.py in that one to run your tests it works for local projects. If you need to run test regularly see if you can move your init.py to a separate file.

I can't say I understand why this works, but I had the same problem and the tests work fine if I run python -m pytest.

I'm in a virtualenv, with pytest also available globally:

(proj)tom@neon ~/dev/proj$ type -a python
python is /home/tom/.virtualenvs/proj/bin/python
python is /usr/bin/python

(proj)tom@neon ~/dev/proj$ python -V
Python 3.5.2

(proj)tom@neon ~/dev/proj$ type -a pytest
pytest is /home/tom/.virtualenvs/proj/bin/pytest
pytest is /usr/bin/pytest

(proj)tom@neon ~/dev/proj$ pytest --version
This is pytest version 3.5.0, imported from /home/tom/.virtualenvs/proj/lib/python3.5/site-packages/pytest.py

This problem will happen if you have a tests.py file and a tests folder with tests/__init__.py.

During the collection pytest finds the folder, but when it tries to import the test files from the folder, tests.py file will cause the import problem.

To fix simply remove the tests.py file and put all your tests inside the tests/ folder.

For your specific case the fix will be precisely:

• Remove the file /home/zz/Desktop/GitFolders/rc/tests.py
• Make sure /home/zz/Desktop/GitFolders/rc/tests/__init__.py is present

Simply put an empty conftest.py file in the project root directory, because when pytest discovers a conftest.py, it modifies sys.path so it can import stuff from the conftest module. A general directory structure can be:

Root
+-- conftest.py
+-- module1
¦   +-- __init__.py
¦   +-- sample.py
+-- tests
    +-- test_sample.py

For anyone who tried everything and still getting error,I have a work around.

In the folder where pytest is installed,go to pytest-env folder.

Open pyvenv.cfg file.

In the file change include-system-site-packages from false to true.

home = /usr/bin
include-system-site-packages = true
version = 3.6.6

Hope it works .Don't forget to up vote.

I was experiencing this issue today and solved it by calling python -m pytest from the root of my project directory.

Calling pytest from the same location still caused issues.

My Project dir is organized as:

api/
 - server/
  - tests/
      - test_routes.py
  - routes/
      - routes.py
 - app.py

The module routes was imported in my test_routes.py as: from server.routes.routes import Routes

Hope that helps!

The answer above not work for me. I just solved it by appending the absolute path of the module which not found to the sys.path at top of the test_xxx.py (your test module), like:

import sys
sys.path.append('path')

I had placed all my tests in a tests folder and was getting the same error. I solved this by adding an init.py in that folder like so:

.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
    |-- __init__.py <------
    |-- conftest.py
    `-- test_sample.py

I had a similar problem just recently. The way it worked for me it was realizing that "setup.py" was wrong

Previously I deleted my previous src folder, and added a new one with other name, but I didn't change anything on the setup.py (newbie mistake I guess).

So pointing setup.py to the right packages folder did the trick for me

from setuptools import find_packages, setup

setup(
    name="-> YOUR SERVICE NAME <-",
    extras_Require=dict(test=["pytest"]),
    packages=find_packages(where="->CORRECT FOLDER<-"),
    package_dir={"": "->CORRECT FOLDER<-"},
)

Also, not init.py in test folder nor in the root one.

Hope it helps someone =)

Best!

It could be that Pytest is not reading the package as a Python module while Python is (likely due to path issues). Try changing the directory of the pytest script or adding the module explicitly to your PYTHONPATH.

Or it could be that you have two versions of Python installed on your machine. Check your Python source for pytest and for the python shell that you run. If they are different (i.e. Python 2 vs 3), use source activate to make sure that you are running the pytest installed for the same python that the module is installed in.

Another special case:

I had the problem using tox. So my program ran fine, but unittests via tox kept complaining. After installing packages (needed for the program) you need to additionally specify the packages used in the unittests in the tox.ini

[testenv]
deps =
    package1
    package2 
...

[Resolved] Before directly jumping into the solution of removing/ adding __init__.py, we might also want to look at how the imports are been done in your classes. Actually, I lost a day playing around with just __init__.py thinking that might be the problem :) However, that was quite informative.

In my case, it was the wrong way of calling classes from one python class to another python class which was throwing ImportError. Fixed the way classes/ modules to be called and it worked like charm. Hope, this helps others as well.

And yes, for similar error, we might have different solutions depending on how the code is written. Better to spend more time on self debugging. Lesson Learnt :) Happy coding!!!

I solved my problem by setting the PYTHONPATH in Environment Variables for the specific configuration I'm running my tests with.

While you're viewing the test file on PyCharm:

1. Ctrl + Shift + A
2. Type Edit Configurations
3. Set your PYTHONPATH under Environment > Environment variables.

If it is related to python code that was originally developed in python 2.7 and now migrated into python 3.x than the problem is probably related to an import issue.

e.g. when importing an object from a file: base that is located in the same directory this will work in python 2.x:

from base import MyClass

in python 3.x you should replace with base full path or .base not doing so will cause the above problem. so try:

from .base import MyClass

If you run Pytest from a terminal:

Run pytest with the --import-mode=append command-line flag.

Argument description in the official documentation: https://docs.pytest.org/en/stable/pythonpath.html

UPD: Before I also wrote how to do the same if you use PyCharm, but community does not like extendend answers, so I removed additional information that probably was helpful to someone who have a similar issue.

Edit your conftest.py and add following lines of code:

import os, sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(file), '..')))

And if trying to run the test case through terminal, use following ex:

python -m pytest test_some_step_file_steps.py --html=HTML_step_file_output.html --self-contained-html