I have a Python package that at first appears to install just fine, but when calling on one of the entry points raises a ModuleNotFoundException. The module is otherwise found just fine with both import package from the interactive interpreter as well as with python -m package.etc. But if I try to call on the entry-point directly (flike python -m package.etc.main) it will raise an AttributeError saying that the module has no attribute __path__.
I can see the package if I do pip list.
The project is currently set up with the "template" pyproject.toml and only setup.cfg, but the behaviour is essentially the same (the traceback looks slightly different but the error is the same) when using setup.py over pyproject.toml, both with pip but also if I invoke setup.py directly. The structure of the project is:
package
├── __init__.py
├── cli
│ ├── __init__.py
│ ├── entry.py
├── file.py
I get the same behaviour if doing this in a virtual environment as when I do it with a userspace (--user) install.
Modifying the environment variable ${PYTHONPATH} fixes the issue, and installing the package in editable mode works just fine.
Turns out that the issue was that I had something like:
[options]
packages = find:
[options.packages.find]
include =
README.md
in my setup.cfg, and it appears as if the declaration of include was exclusive which led to the package not being included in the installation, which still worked when installed in editable mode (presumably because editable mode only sets up some sort of links or appends the source code directories to some path).
I used easy_install to install pytest on a Mac and started writing tests for a project with a file structure likes so:
repo/
|--app.py
|--settings.py
|--models.py
|--tests/
|--test_app.py
Run py.test while in the repo directory, and everything behaves as you would expect.
But when I try that same thing on either Linux or Windows (both have pytest 2.2.3 on them), it barks whenever it hits its first import of something from my application path. For instance, from app import some_def_in_app.
Do I need to be editing my PATH to run py.test on these systems?
I'm not sure why py.test does not add the current directory in the PYTHONPATH itself, but here's a workaround (to be executed from the root of your repository):
python -m pytest tests/
It works because Python adds the current directory in the PYTHONPATH for you.
Recommended approach for pytest>=7: use the pythonpath setting
Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:
pyproject.toml example:
[tool.pytest.ini_options]
pythonpath = [
"."
]
pytest.ini example:
[pytest]
pythonpath = .
The path entries are calculated relative to the rootdir, thus . adds repo directory to sys.path in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── app.py
└── tests
├── test_app.py
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
".", "src",
]
or
[pytest]
pythonpath = . src
will add both app and lib modules to sys.path, so
import app
import lib
will both work.
Original answer (not recommended for recent pytest versions; use for pytest<7 only): conftest solution
The least invasive solution is adding an empty file named conftest.py in the repo/ directory:
$ touch repo/conftest.py
That's it. No need to write custom code for mangling the sys.path or remember to drag PYTHONPATH along, or placing __init__.py into dirs where it doesn't belong (using python -m pytest as suggested in Apteryx's answer is a good solution though!).
The project directory afterwards:
repo
├── conftest.py
├── app.py
├── settings.py
├── models.py
└── tests
└── test_app.py
Explanation
pytest looks for the conftest modules on test collection to gather custom hooks and fixtures, and in order to import the custom objects from them, pytest adds the parent directory of the conftest.py to the sys.path (in this case the repo directory).
Other project structures
If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an __init__.py), for example:
repo
├── conftest.py
├── spam
│ ├── __init__.py
│ ├── bacon.py
│ └── egg.py
├── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
src layout
Although this approach can be used with the src layout (place conftest.py in the src dir):
repo
├── src
│ ├── conftest.py
│ ├── spam
│ │ ├── __init__.py
│ │ ├── bacon.py
│ │ └── egg.py
│ └── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
beware that adding src to PYTHONPATH mitigates the meaning and benefits of the src layout! You will end up with testing the code from repository and not the installed package. If you need to do it, maybe you don't need the src dir at all.
Where to go from here
Of course, conftest modules are not just some files to help the source code discovery; it's where all the project-specific enhancements of the pytest framework and the customization of your test suite happen. pytest has a lot of information on conftest modules scattered throughout their docs; start with conftest.py: local per-directory plugins
Also, SO has an excellent question on conftest modules: In py.test, what is the use of conftest.py files?
I had the same problem. I fixed it by adding an empty __init__.py file to my tests directory.
Yes, the source folder is not in Python's path if you cd to the tests directory.
You have two choices:
Add the path manually to the test files. Something like this:
import sys, os
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
Run the tests with the env var PYTHONPATH=../.
Run pytest itself as a module with:
python -m pytest tests
This happens when the project hierarchy is, for example, package/src package/tests and in tests you import from src. Executing as a module will consider imports as absolute rather than relative to the execution location.
You can run with PYTHONPATH in project root
PYTHONPATH=. py.test
Or use pip install as editable import
pip install -e . # install package using setup.py in editable mode
I had the same problem in Flask.
When I added:
__init__.py
to the tests folder, the problem disappeared :)
Probably the application couldn't recognize folder tests as a module.
I created this as an answer to your question and my own confusion. I hope it helps. Pay attention to PYTHONPATH in both the py.test command line and in the tox.ini.
https://github.com/jeffmacdonald/pytest_test
Specifically: You have to tell py.test and tox where to find the modules you are including.
With py.test you can do this:
PYTHONPATH=. py.test
And with tox, add this to your tox.ini:
[testenv]
deps= -r{toxinidir}/requirements.txt
commands=py.test
setenv =
PYTHONPATH = {toxinidir}
I fixed it by removing the top-level __init__.py in the parent folder of my sources.
I started getting weird ConftestImportFailure: ImportError('No module named ... errors when I had accidentally added __init__.py file to my src directory (which was not supposed to be a Python package, just a container of all source).
It is a bit of a shame that this is an issue in Python... But just adding this environment variable is the most comfortable way, IMO:
export PYTHONPATH=$PYTHONPATH:.
You can put this line in you .zshrc or .bashrc file.
I was having the same problem when following the Flask tutorial and I found the answer on the official Pytest documentation.
It's a little shift from the way I (and I think many others) are used to do things.
You have to create a setup.py file in your project's root directory with at least the following two lines:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
where PACKAGENAME is your app's name. Then you have to install it with pip:
pip install -e .
The -e flag tells pip to install the package in editable or "develop" mode. So the next time you run pytest it should find your app in the standard PYTHONPATH.
I had a similar issue. pytest did not recognize a module installed in the environment I was working in.
I resolved it by also installing pytest into the same environment.
Also if you run pytest within your virtual environment make sure pytest module is installed within your virtual environment. Activate your virtual environment and run pip install pytest.
For me the problem was tests.py generated by Django along with tests directory. Removing tests.py solved the problem.
I got this error as I used relative imports incorrectly. In the OP example, test_app.py should import functions using e.g.
from repo.app import *
However liberally __init__.py files are scattered around the file structure, this does not work and creates the kind of ImportError seen unless the files and test files are in the same directory.
from app import *
Here's an example of what I had to do with one of my projects:
Here’s my project structure:
microbit/
microbit/activity_indicator/activity_indicator.py
microbit/tests/test_activity_indicator.py
To be able to access activity_indicator.py from test_activity_indicator.py I needed to:
start test_activity_indicatory.py with the correct relative import:
from microbit.activity_indicator.activity_indicator import *
put __init__.py files throughout the project structure:
microbit/
microbit/__init__.py
microbit/activity_indicator/__init__.py
microbit/activity_indicator/activity_indicator.py
microbit/tests/__init__.py
microbit/tests/test_activity_indicator.py
According to a post on Medium by Dirk Avery (and supported by my personal experience) if you're using a virtual environment for your project then you can't use a system-wide install of pytest; you have to install it in the virtual environment and use that install.
In particular, if you have it installed in both places then simply running the pytest command won't work because it will be using the system install. As the other answers have described, one simple solution is to run python -m pytest instead of pytest; this works because it uses the environment's version of pytest. Alternatively, you can just uninstall the system's version of pytest; after reactivating the virtual environment the pytest command should work.
I was getting this error due to something even simpler (you could even say trivial). I hadn't installed the pytest module. So a simple apt install python-pytest fixed it for me.
'pytest' would have been listed in setup.py as a test dependency. Make sure you install the test requirements as well.
Since no one has suggested it, you could also pass the path to the tests in your pytest.ini file:
[pytest]
...
testpaths = repo/tests
See documentation: https://docs.pytest.org/en/6.2.x/customize.html#pytest-ini
Side effect for Visual Studio Code: it should pick up the unit test in the UI.
We have fixed the issue by adding the following environment variable.
PYTHONPATH=${PYTHONPATH}:${PWD}/src:${PWD}/test
As pointed out by Luiz Lezcano Arialdi, the correct solution is to install your package as an editable package.
Since I am using Pipenv, I thought about adding to his answer a step-by-step how to install the current path as an edible with Pipenv, allowing to run pytest without the need of any mangling code or lose files.
You will need to have the following minimal folder structure (documentation):
package/
package/
__init__.py
module.py
tests/
module_test.py
setup.py
setup.py mostly has the following minium code (documentation):
import setuptools
setuptools.setup(name='package', # Change to your package name
packages=setuptools.find_packages())
Then you just need to run pipenv install --dev -e . and Pipenv will install the current path as an editable package (the --dev flag is optional) (documentation).
Now you should be able to run pytest without problems.
If this pytest error appears not for your own package, but for a Git-installed package in your package's requirements.txt, the solution is to switch to editable installation mode.
For example, suppose your package's requirements.txt had the following line:
git+https://github.com/foo/bar.git
You would instead replace it with the following:
-e git+https://github.com/foo/bar.git#egg=bar
If nothing works, make sure your test_module.py is listed under the correct src directory.
Sometimes it will give ModuleNotFoundError not because modules are misplaced or export PYTHONPATH="${PWD}:${PYTHONPATH}" is not working, its because test_module.py is placed into a wrong directory under the tests folder.
it should be 1-to-1 mapping relation recursively instead of the root folder should be named as "tests" and the name of the file that include test code should starts with "test_",
for example,
./nlu_service/models/transformers.py
./tests/models/test_transformers.py
This was my experience.
Very often the tests were interrupted due to module being unable to be imported.
After research, I found out that the system is looking at the file in the wrong place and we can easily overcome the problem by copying the file, containing the module, in the same folder as stated, in order to be properly imported.
Another solution proposal would be to change the declaration for the import and show MutPy the correct path of the unit. However, due to the fact that multiple units can have this dependency, meaning we need to commit changes also in their declarations, we prefer to simply move the unit to the folder.
My solution:
Create the conftest.py file in the test directory containing:
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + "/relative/path/to/code/")
This will add the folder of interest to the Python interpreter path without modifying every test file, setting environment variable or messing with absolute/relative paths.
I am learning Test Driven development for the first time. I have no experience of software development, but have some experience with scripting.
I have been following LinuxAcademy Python 3 for Sys Admin tutorial.
I created the following structure,
├── Makefile
├── Pipfile
├── Pipfile.lock
├── README.rst
├── setup.py
├── src
│ └── pgbackup
│ ├── cli.py
│ └── __init__.py
└── tests
└── test_cli.py
setup.py file,
from setuptools import setup, find_packages
with open('README.rst', 'r') as f:
readme = f.read()
setup(
name='pgbackup',
version='0.1.0',
description='Database backups locally or to AWS S3.',
long_description=readme,
author='Keith Thompson',
author_email='keith#linuxacademy.com',
packages=find_packages('src'),
package_dir={'': 'src'},
)
Makefile file,
.PHONY: install test
default: test
install:
pipenv install --dev --skip-lock
test:
PYTHONPATH=./src pytest
tests/test_cli.py file,
import pytest
from pgbackup import cli
def test_helloworld():
"""
JUST A HELLO WORLD TEST
"""
assert cli.hello() == "helloworld"
and src/pgbackup/cli.py file,
def hello():
return "helloworld"
I wrote helloworld as my first sample test it is not the part of the tutorial. Now when I run make command from project root directory, my test is passed,
========================================== test session starts ===========================================platform linux -- Python 3.6.6, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
rootdir: /root/code/pgbackup, inifile:
collected 1 item
tests/test_cli.py . [100%]
======================================== 1 passed in 0.04 seconds ========================================
I know the make command is setting PYTHONPATH to ./src pytest but not get my head around how its running actual test? I know its only setting a search path to import python modules.
If I try to run pytest command from tests dir, my test if failed,
================================================= ERRORS =================================================___________________________________ ERROR collecting tests/test_cli.py ___________________________________ImportError while importing test module '/root/code/pgbackup/tests/test_cli.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_cli.py:2: in <module>
from pgbackup import cli
E ModuleNotFoundError: No module named 'pgbackup'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!======================================== 1 error in 0.35 seconds =========================================
If i run test from src dir, it doesn't run anything,
====================================== no tests ran in 0.01 seconds ======================================
Can someone please explain how running make runs the test since Makefile is just setting PYTHONPATH variable?
Can someone please explain how running make runs the test since Makefile is just setting PYTHONPATH variable?
It's not only setting variable. pytest is being run here because your current line for test:
PYTHONPATH=./src pytest
is equivalent to:
export PYTHONPATH=./src; pytest
Check 3rd example here: Command not found error in Bash variable assignment for explanation
You will probably want to change it to look accordingly,
If I try to run pytest command from tests dir, my test if failed,
As for running pytest from different directories. Indeed, this will give different results. pytest searches for tests (looks for all files in test folder, or current one named test_*). So if you run it inside src, it won't find any tests.
If I try to run pytest command from tests dir, my test if failed
pgbackup seem to be imported from current directory so when you move inside 'tests' folder it won't be found.
Hope this helps.
I've been googling this issue for over an hour and have absolutely 0 idea what to do - I'm trying to set up travis ci on my public repo and for some reason every time I commit and build, I consistently get this:
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /home/travis/build/epitone/digitron, inifile:
plugins: cov-2.5.1
collected 0 items
========================= no tests ran in 0.01 seconds =========================
The command "pytest" exited with 5.
Done. Your build exited with 1.
For some reason, my tests aren't being collected even though on my local machine, pytest runs just fine and even passes my (single) test that I've set up. Does anyone know exactly what's going on here?
My directory setup is something like this:
digitron/
├── bot.py
├── lib
│ ├── _config.py
│ ├── config.py
│ ├── __init__.py
│ └── utils.py
├── README.md
├── requirements.txt
└── tests
└── auth_test.py
my .travis.yml file:
language: python
python:
- "2.7"
- "3.2"
- "3.3"
- "3.4"
- "3.5"
- "3.6"
- "nightly" # currently points to 3.7-dev
before_install:
- pip install pytest pytest-cov
# command to install dependencies
install: "pip install -r requirements.txt"
# command to run tests
script: pytest
my requirements.txt file contains:
pytest>=3.2.1
py>=1.4.31
pluggy>=0.4.0
and my auth_test.py file is simply:
"""
Testing for Digitron
"""
# Necessary to find parent directory
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import bot
import pytest
def test_connect():
"""Test connect() method in bot.py
Returns True or false
"""
assert(bot.connect("irc.chat.twitch.tv", 6667, "oauth:key", "username", "#channel")) == True
I'm basically pulling my hair out over here trying to figure out what I'm doing wrong - googling error code 5 gets me nothing, and I'm even following along with someone elses setup and still getting nowhere - is there something I'm missing?
edit: tried explicitly calling the tests/ directory in the pytest command - works locally, fails yet again on travis.ci
While I can't tell if this is the cause of your problem, at least the method to coerce sys.path is not very orthodox. pytest performs this automatically: https://docs.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code so there may be import issues.
The error number 5 is caused by pytest not finding any test, see the documentation
Possible tests for resolution:
Rename your test file to test_auth.py
Add diagnostic information to your script: pwd, ls, etc. Try simply executing the file: python tests/auth_test.py.
This is a bit long and annoying to format in a comment, hence the answer.
You should add a pytest.ini file with the contents:
[pytest]
testpaths = <path to your tests here>
Without this, Travis cannot find your tests, hence why you are getting the error number 5.
I'm writing a couple of packages that I'd like to release on PyPi for other people to use.
I've not released to PyPi before so I have been mocking up a submission template: https://github.com/chris-brown-nz/pypi-package-template
Here's a tree of the project template:
| MANIFEST.in
| README.rst
| setup.cfg
| setup.py
|
\---package
module_one.py
module_three.py
module_two.py
__init__.py
In terms of interacting with the package, this is what I would usually do - is it the best way?
To run a method:
from package import module_one
module_one.ClassOne().method_a()
To get a value from a method:
from package import module_two
print(module_two.ClassFive().method_e())
To set then use an attribute of an instance:
from package import module_three
cls = module_three.ClassSeven("Hello World")
print(cls.value)
'package' is a reserved name obviously and won't be used in the final project.
I'd be grateful for some feedback on how I've structured my project and whether it is considered standard, or if it should be modified in some way.
There are different approaches to this, whether one or the other is better is depending on a how you want to develop, usage of the package (e.g. if you ever install it using pip install -e packag_name), etc.
What is missing from your tree is the name of the directory where the setup.py resides, and that is usually the package name:
└── package
├── package
│ ├── __init__.py
│ ├── module_one.py
│ ├── module_three.py
│ └── module_two.py
├── MANIFEST.in
├── README.rst
├── setup.cfg
└── setup.py
as you can see you are doubling the 'package' name, and that means that your setup.py has to be adapted for each package, or dynamically determine the name of the directory where the module.py files resides. If you go for this route, I would suggest you put the module.py files in a generically named directory 'src' or 'lib'.
I don't like the above "standard" setup for multiple reasons:
it doesn't map well to how python programs "grow" before they are split up into packages. Before splitting up having such a 'src' directory would mean using:
from package.src.module_one import MyModuleOneClass
Instead you would have your module.py files directly under package
Having a setup.py to control installation, a README.rst for documentation and an __init__.py to satisfy Python's import is one thing, but all other stuff, apart from your module.py files containing the actual functionality, is garbage. Garbage that might be needed at some point during the package creation process, but is not necessary for the package functionality.
There are other considerations, such as being able to access the version number of the package from the setup.py as well as from the program, without the former having to import the package itself (which may lead to install complications), nor having another extra version.py file that needs importing.
In particular I always found the transition from using a directory structure under site-packages that looked like:
└── organisation
├── package1
└── package2
├── subpack1
└── subpack2
and that could intuitively be used for both importing and navigation to source files, to something like:
├── organisation_package1
│ └── src
├── organisation_package2_subpack1
│ └── src
└── organisation_package2_subpack2
└── src
unnatural. To rearrange and break a working structure to be able to package things seems wrong.
For my set of published packages I followed another way:
- I kept the natural tree structure that you can use "before packaging", 'src' or 'lib' directories.
- I have a generic setup.py which reads and parses (it does not import) the metadata (such as version number, package name, license information, whether to install a utility (and its name)), from a dictionary in the __init__.py file. A file you need anyway.
- The setup.py is smart enough to distinguish subdirectories containing other packages from subdirectories that are part of the parent package.
- setup.py generates files that are needed during package generation only (like setup.cfg), on the fly, and deletes them when no longer needed.
The above allows you to have nested namespaced packages (i.e. package2 can be a package you upload to PyPI, in addition to package2.subpack1 and package2.subpack2). The major thing it (currently) doesn't allow is using pip install -e to edit a single package (and not have the others editable). Given the way I develop, that is not a restriction.
The above embraces namespace packages, where many other approaches have problems with these (remember the last line of Zen of Python: Namespaces are one honking great idea – let’s do more of those)
Examples of the above can e.g be found in my packages ruamel.yaml (and e.g. ruamel.yaml.cmd), or generically by searching PyPI for ruamel.
As is probably obvious, the standard disclaimer: I am the author of those packages
As I use a utility to start packaging, which also runs the tests and does other sanity checks, the generic setup.py could be removed from the setup and inserted by that utility as well. But since subpackage detection is based upon setup.py availability or not, this requires some rework of the generic setup.py.