Related
I have a small team of people working on a small new Python project.
I created a small package called Environment.
This has two files:
Setup.py
from setuptools import setup, find_packages
setup(name='Environment', version='1.0', packages=find_packages())
root.py
ROOT_DIR = "STATIC_VALUE_15"
When I use pip install -e . and then run the project within the anaconda enviroment I use for this project, I can run files that use this Environment package. However when other people use the same branch and do the same steps, they get the error:
Exception has occurred: ModuleNotFoundError
They are also using their conda environment and running the project the same way in vsc.
When anyone uses pip list, for this package our paths are all correct.
Are there any suggestions for things that I could try and test to get it to work on their system, or why it works on mine and not theirs?
Setup.py is deprecated, use pyproject.toml instead.
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "environment"
version = "1.0.0"
You also need to follow this file hierarchy:
package_project/
└── src/
└── environment/
├── __init__.py
└── root.py
You may read Packaging Python Projects. There is plenty of room to make mistake, I struggled a lot in the past because I forgot a file, or didn't follow the file hierarchy, and if the build fail, sometime you can still run or import the code because of your python environment.
If you want to test if it works for someone else without having someone else, you can build the package and install it in an other environment where the sources are not easily accessible for python, using the .whl.
I used easy_install to install pytest on a Mac and started writing tests for a project with a file structure likes so:
repo/
|--app.py
|--settings.py
|--models.py
|--tests/
|--test_app.py
Run py.test while in the repo directory, and everything behaves as you would expect.
But when I try that same thing on either Linux or Windows (both have pytest 2.2.3 on them), it barks whenever it hits its first import of something from my application path. For instance, from app import some_def_in_app.
Do I need to be editing my PATH to run py.test on these systems?
I'm not sure why py.test does not add the current directory in the PYTHONPATH itself, but here's a workaround (to be executed from the root of your repository):
python -m pytest tests/
It works because Python adds the current directory in the PYTHONPATH for you.
Recommended approach for pytest>=7: use the pythonpath setting
Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:
pyproject.toml example:
[tool.pytest.ini_options]
pythonpath = [
"."
]
pytest.ini example:
[pytest]
pythonpath = .
The path entries are calculated relative to the rootdir, thus . adds repo directory to sys.path in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── app.py
└── tests
├── test_app.py
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
".", "src",
]
or
[pytest]
pythonpath = . src
will add both app and lib modules to sys.path, so
import app
import lib
will both work.
Original answer (not recommended for recent pytest versions; use for pytest<7 only): conftest solution
The least invasive solution is adding an empty file named conftest.py in the repo/ directory:
$ touch repo/conftest.py
That's it. No need to write custom code for mangling the sys.path or remember to drag PYTHONPATH along, or placing __init__.py into dirs where it doesn't belong (using python -m pytest as suggested in Apteryx's answer is a good solution though!).
The project directory afterwards:
repo
├── conftest.py
├── app.py
├── settings.py
├── models.py
└── tests
└── test_app.py
Explanation
pytest looks for the conftest modules on test collection to gather custom hooks and fixtures, and in order to import the custom objects from them, pytest adds the parent directory of the conftest.py to the sys.path (in this case the repo directory).
Other project structures
If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an __init__.py), for example:
repo
├── conftest.py
├── spam
│ ├── __init__.py
│ ├── bacon.py
│ └── egg.py
├── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
src layout
Although this approach can be used with the src layout (place conftest.py in the src dir):
repo
├── src
│ ├── conftest.py
│ ├── spam
│ │ ├── __init__.py
│ │ ├── bacon.py
│ │ └── egg.py
│ └── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
beware that adding src to PYTHONPATH mitigates the meaning and benefits of the src layout! You will end up with testing the code from repository and not the installed package. If you need to do it, maybe you don't need the src dir at all.
Where to go from here
Of course, conftest modules are not just some files to help the source code discovery; it's where all the project-specific enhancements of the pytest framework and the customization of your test suite happen. pytest has a lot of information on conftest modules scattered throughout their docs; start with conftest.py: local per-directory plugins
Also, SO has an excellent question on conftest modules: In py.test, what is the use of conftest.py files?
I had the same problem. I fixed it by adding an empty __init__.py file to my tests directory.
Yes, the source folder is not in Python's path if you cd to the tests directory.
You have two choices:
Add the path manually to the test files. Something like this:
import sys, os
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
Run the tests with the env var PYTHONPATH=../.
Run pytest itself as a module with:
python -m pytest tests
This happens when the project hierarchy is, for example, package/src package/tests and in tests you import from src. Executing as a module will consider imports as absolute rather than relative to the execution location.
You can run with PYTHONPATH in project root
PYTHONPATH=. py.test
Or use pip install as editable import
pip install -e . # install package using setup.py in editable mode
I had the same problem in Flask.
When I added:
__init__.py
to the tests folder, the problem disappeared :)
Probably the application couldn't recognize folder tests as a module.
I created this as an answer to your question and my own confusion. I hope it helps. Pay attention to PYTHONPATH in both the py.test command line and in the tox.ini.
https://github.com/jeffmacdonald/pytest_test
Specifically: You have to tell py.test and tox where to find the modules you are including.
With py.test you can do this:
PYTHONPATH=. py.test
And with tox, add this to your tox.ini:
[testenv]
deps= -r{toxinidir}/requirements.txt
commands=py.test
setenv =
PYTHONPATH = {toxinidir}
I fixed it by removing the top-level __init__.py in the parent folder of my sources.
I started getting weird ConftestImportFailure: ImportError('No module named ... errors when I had accidentally added __init__.py file to my src directory (which was not supposed to be a Python package, just a container of all source).
It is a bit of a shame that this is an issue in Python... But just adding this environment variable is the most comfortable way, IMO:
export PYTHONPATH=$PYTHONPATH:.
You can put this line in you .zshrc or .bashrc file.
I was having the same problem when following the Flask tutorial and I found the answer on the official Pytest documentation.
It's a little shift from the way I (and I think many others) are used to do things.
You have to create a setup.py file in your project's root directory with at least the following two lines:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
where PACKAGENAME is your app's name. Then you have to install it with pip:
pip install -e .
The -e flag tells pip to install the package in editable or "develop" mode. So the next time you run pytest it should find your app in the standard PYTHONPATH.
I had a similar issue. pytest did not recognize a module installed in the environment I was working in.
I resolved it by also installing pytest into the same environment.
Also if you run pytest within your virtual environment make sure pytest module is installed within your virtual environment. Activate your virtual environment and run pip install pytest.
For me the problem was tests.py generated by Django along with tests directory. Removing tests.py solved the problem.
I got this error as I used relative imports incorrectly. In the OP example, test_app.py should import functions using e.g.
from repo.app import *
However liberally __init__.py files are scattered around the file structure, this does not work and creates the kind of ImportError seen unless the files and test files are in the same directory.
from app import *
Here's an example of what I had to do with one of my projects:
Here’s my project structure:
microbit/
microbit/activity_indicator/activity_indicator.py
microbit/tests/test_activity_indicator.py
To be able to access activity_indicator.py from test_activity_indicator.py I needed to:
start test_activity_indicatory.py with the correct relative import:
from microbit.activity_indicator.activity_indicator import *
put __init__.py files throughout the project structure:
microbit/
microbit/__init__.py
microbit/activity_indicator/__init__.py
microbit/activity_indicator/activity_indicator.py
microbit/tests/__init__.py
microbit/tests/test_activity_indicator.py
According to a post on Medium by Dirk Avery (and supported by my personal experience) if you're using a virtual environment for your project then you can't use a system-wide install of pytest; you have to install it in the virtual environment and use that install.
In particular, if you have it installed in both places then simply running the pytest command won't work because it will be using the system install. As the other answers have described, one simple solution is to run python -m pytest instead of pytest; this works because it uses the environment's version of pytest. Alternatively, you can just uninstall the system's version of pytest; after reactivating the virtual environment the pytest command should work.
I was getting this error due to something even simpler (you could even say trivial). I hadn't installed the pytest module. So a simple apt install python-pytest fixed it for me.
'pytest' would have been listed in setup.py as a test dependency. Make sure you install the test requirements as well.
Since no one has suggested it, you could also pass the path to the tests in your pytest.ini file:
[pytest]
...
testpaths = repo/tests
See documentation: https://docs.pytest.org/en/6.2.x/customize.html#pytest-ini
Side effect for Visual Studio Code: it should pick up the unit test in the UI.
We have fixed the issue by adding the following environment variable.
PYTHONPATH=${PYTHONPATH}:${PWD}/src:${PWD}/test
As pointed out by Luiz Lezcano Arialdi, the correct solution is to install your package as an editable package.
Since I am using Pipenv, I thought about adding to his answer a step-by-step how to install the current path as an edible with Pipenv, allowing to run pytest without the need of any mangling code or lose files.
You will need to have the following minimal folder structure (documentation):
package/
package/
__init__.py
module.py
tests/
module_test.py
setup.py
setup.py mostly has the following minium code (documentation):
import setuptools
setuptools.setup(name='package', # Change to your package name
packages=setuptools.find_packages())
Then you just need to run pipenv install --dev -e . and Pipenv will install the current path as an editable package (the --dev flag is optional) (documentation).
Now you should be able to run pytest without problems.
If this pytest error appears not for your own package, but for a Git-installed package in your package's requirements.txt, the solution is to switch to editable installation mode.
For example, suppose your package's requirements.txt had the following line:
git+https://github.com/foo/bar.git
You would instead replace it with the following:
-e git+https://github.com/foo/bar.git#egg=bar
If nothing works, make sure your test_module.py is listed under the correct src directory.
Sometimes it will give ModuleNotFoundError not because modules are misplaced or export PYTHONPATH="${PWD}:${PYTHONPATH}" is not working, its because test_module.py is placed into a wrong directory under the tests folder.
it should be 1-to-1 mapping relation recursively instead of the root folder should be named as "tests" and the name of the file that include test code should starts with "test_",
for example,
./nlu_service/models/transformers.py
./tests/models/test_transformers.py
This was my experience.
Very often the tests were interrupted due to module being unable to be imported.
After research, I found out that the system is looking at the file in the wrong place and we can easily overcome the problem by copying the file, containing the module, in the same folder as stated, in order to be properly imported.
Another solution proposal would be to change the declaration for the import and show MutPy the correct path of the unit. However, due to the fact that multiple units can have this dependency, meaning we need to commit changes also in their declarations, we prefer to simply move the unit to the folder.
My solution:
Create the conftest.py file in the test directory containing:
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + "/relative/path/to/code/")
This will add the folder of interest to the Python interpreter path without modifying every test file, setting environment variable or messing with absolute/relative paths.
The following is my directory structure.
ankur
├── ankur1
│ ├── __init__.py
│ └── util.py
├── ankur2
│ └── main.py
└── __init__.py
In main.py, I am importing the following.
import ankur.ankur1.util
When I execute the code in Windows, it works perfectly fine. But in Linux, I get the following error.
ImportError: No module named ankur.ankur1.util
I also read the official python doc on Modules and Packages.
Your package structure is OK. Your import statement is OK. The only thing missing is for the package to be visible in sys.path, a list of locations where import statements can be resolved.
Usually we do this by "installing" the package locally with pip, which copies your code into site-packages†. This directory is one of the entries in sys.path, so when your code is installed in site-packages, the import statements can now be resolved as usual.
However, to install your code you'll need an installer (setup.py script) or a build system (pyproject.toml file) defined for the package. Your project doesn't appear to have any installer or build system, so you'll need to create one (see the Python Packaging User Guide for details about that) and then install the package with pip. If you don't want to learn Python packaging just yet, you'll need to find another way around.
It is possible to modify sys.path directly in main.py, which is subsequently enabling the statement import ankur.ankur1.util to be resolved. This is hacky and I recommend against that. It would add the restriction that executing main.py is the only entry point to the rest of the package, and so any other code wanting to import ankur will first need to know the path to main.py on the filesystem. That's a messy approach and should be avoided.
Another way around is to use the environment - there is an environment variable PYTHONPATH which can be used to augment the default search path for module files. In your shell:
export PYTHONPATH=/path/to/parent # linux/macOS
SET PYTHONPATH=C:/path/to/parent # Windows
Where parent is the directory containing ankur subdirectory.
† The exact location of site-packages depends on your OS/platform, but you can check with import sysconfig; sysconfig.get_paths()["purelib"]
Consider the following Python project skeleton:
proj/
├── foo
│ └── __init__.py
├── README.md
└── scripts
└── run.py
In this case foo holds the main project files, for example
# foo/__init__.py
class Foo():
def run(self):
print('Running...')
And scripts holds auxiliary scripts that need to import files from foo, which are then invoked via:
[~/proj]$ python scripts/run.py
There are two ways of importing Foo which both fail:
If a relative import is attempted from ..foo import Foo then the error is ValueError: attempted relative import beyond top-level package
If an absolute import is attempted from foo import Foo then the error is ModuleNotFoundError: No module named 'foo'
My current workaround is to append the running path to sys.path:
import sys
sys.path.append('.')
from foo import Foo
Foo().run()
But this feels like a hack, and has to be added to every new script in scripts/.
Is there a better way to structure scripts in such projects?
There's two ways you could resolve this.
(1) Turn your project into an installable package
Add a proj/setup.py file with the following contents:
import setuptools
setuptools.setup(
name="my-project",
version="1.0.0",
author="You",
author_email="you#example.com",
description="This is my project",
packages=["foo"],
)
create a virtualenv:
python3 -m venv virtualenv # this creates a directory "virtualenv" in your project
source ./virtualenv/bin/activate # this switches you into the new environment
python setup.py develop # this places your "foo" package in the environment
inside the virtualenv, foo behaves as an installed package and is importable via import foo.
So you can use absolute imports in your scripts.
To make them run from anywhere, without needing to activate the virtualenv, you can then specify the path as a shebang.
In scripts/run.py (the first line is important):
#!/path/to/proj/virtualenv/bin/python
import foo
print(foo.callfunc())
(2) Make the scripts part of the foo package
Instead of a separate subdirectory scripts, make a subpackage. In proj/foo/commands/run.py:
from .. import callfunc()
def main():
print(callfunc())
if __name__ == "__main__":
main()
Then execute the script from the top-level proj/ directory with:
python -m foo.commands.run
If you combine this with (1) and install your package, you can then run python -m foo.commands.run from anywhere.
Solution
There are multiple ways to achieve this. Both require creating a python package by adding a setup.py (building on #matejcik's answer).
Option 1 (recommended): entry_point + console_scripts register a function in your project as the entry point to script execution (ie: proj:foo:cli:run).
Option 2: scripts: Use this keyword argument in the setup() method to reference the path to your script (ie: `bin/script.py).
Note
I recommend using a CLI library/framework like Click so that your codebase is only concerned with maintaining application specific business logic rather than CLI robust framework feature logic. Also, click recommends using entry_point + console_scripts method of script integration due to cross-platform compatibility.
Setup Tools - Automatic script creation: https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation
Setup Tools - keyword arguments: https://setuptools.readthedocs.io/en/latest/setuptools.html#new-and-changed-setup-keywords
Click GitHub: https://github.com/pallets/click/
Click Setuptools integration: https://click.palletsprojects.com/en/master/setuptools/
Best practice? Put a single entry-point in the root
I know this might sound absurd, if you have lots of scripts you want to be able to execute... But it's actually the cleanest option and it's the one that is most often used in big Python projects like magage.py in Django, for example. It also doesn't need to be a huge undertaking. Even more importantly, it is always more secure to have a single entry point than several smaller ones.
proj/
├── run.py
├── foo
│ └── __init__.py
├── README.md
└── scripts
└── my_script.py
When run.py lives in the root directory, it can be very lightweight... Basically just a wrapper to call the function you need from my_scripts.py. It just ties everything together so now all of your imports just work.
Just keep in mind that your entrypoint is your root. The parent of a root doesn't exist. So put your entrypoint in the root, and then import packages relative to the root, aka import foo from scripts.
But how do I call multiple scripts!?
If you need to be able to call multiple scripts, this is a good argument for... Well... arguments! Keep run.py as your single entrypoint/command, and leverage subcommands to pass functionality to the script you care about.
Reinventing the wheel?
Generally, frameworks have already done the architecture for you to add your own subcommands, such as Django and, for a smaller footprint, Flask.
You can easily wrap up a small project without that help, though, as I've illustrated.
Security
No one ever wishes their code was less refactorable after a few years of working with it. No one ever wishes their codebase has less security. As we drive to more secure systems in general, it would make sense to create some gatekeeper script that determines what is and isn’t a safe operation and by whom. Moving the code to an LDAP based system, and need to lock things down by group? No problem. You can either change the single file or add LDAP security in your codebase, even creating your own internal API.
With distributed scripts, security options are much less flexible and much harder to maintain, and a single vulnerability could leave you wide open to exploit.
Bonus advantage
You're adding abstraction to your script base. If you ever want to change the structure of your codebase (maybe you want scripts to have subfolders with more organization), you/your users don't need to do any refactoring for any dependencies, or change paths to longer, more verbose names. Your package is self-contained, and the only thing a user will ever need to touch is your proj/run.py entry-point.
And, obviously, you don't need to play with Python paths as much!
You need to add __init__.py files to scripts and to proj folders for those to be considered Python packages and for you to be able to import from those.
One way this is also commonly done, is to place your foo and scripts folders into a proj/src folder, which then has a __init__.py file, and thus is a Python package.
If you like simplicity, and there are no additional restrictions on what you asked, add one __init__.py to the scripts folder, and to any other sibling folders, making them packages, then always use the absolute import form, as you said you do not want proj as a parent package of those and so there is no __init__.py there, and then call your scripts (instead) from inside the proj folder with:
python -m scripts.run
or whatever name you give to other scripts other than run.py
This is similar to option 2 of #matejcik answer, but even simpler.
another solution is you add a.pth file on your Python directory
and write the content of the following,
# your.pth
#↓ input the directory of proj
C:\...\proj
done
# scripts.py
from foo import Foo
Foo().run()
It will work well.
.. note:: If your IDE is PyCharm, then you can use the Source roots to help you too.
Python looks for packages/modules in the directories listed in sys.path. There are several ways of ensuring that your directories of interest, in this case proj, is one of those directories:
Move your scripts to the proj directory. Python adds the directory containing the input script to sys.path.
Put the directory proj into the contents of the PYTHONPATH environment variable.
Make the module part of an installable package and install it, either in a virtual environment or not.
At run time, dynamically add the directory proj to sys.path.
Option 1 is the most logical and requires no source changes. If you are afraid that might break something, you can perhaps make scripts a symbolic link pointing back to proj?
If you are unwilling to do that, then ...
You may consider it a hack, but I would recommend that you do modify your scripts to update sys.path at runtime. But instead append an absolute path so that the scripts can be executed regardless of what the current directory is. In your case, directory proj is the parent directory of directory scripts, where the scripts reside, so:
import sys
import os.path
parent_directory = os.path.split(os.path.dirname(__file__))[0]
if parent_directory not in sys.path:
#sys.path.insert(0, parent_directory) # the first entry is directory of the running script, so maybe insert after that at index 1
sys.append(parent_directory)
I am working on a package in Python. I use virtualenv. I set the path to the root of the module in a .pth path in my virtualenv, so that I can import modules of the package while developing the code and do testing (Question 1: is it a good way to do?). This works fine (here is an example, this is the behavior I want):
(VEnvTestRc) zz#zz:~/Desktop/GitFolders/rc$ python
Python 2.7.12 (default, Jul 1 2016, 15:12:24)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rc import ns
>>> exit()
(VEnvTestRc) zz#zz:~/Desktop/GitFolders/rc$ python tests/test_ns.py
issued command: echo hello
command output: hello
However, if I try to use PyTest, I get some import error messages:
(VEnvTestRc) zz#zz:~/Desktop/GitFolders/rc$ pytest
=========================================== test session starts ============================================
platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
rootdir: /home/zz/Desktop/GitFolders/rc, inifile:
collected 0 items / 1 errors
================================================== ERRORS ==================================================
________________________________ ERROR collecting tests/test_ns.py ________________________________
ImportError while importing test module '/home/zz/Desktop/GitFolders/rc/tests/test_ns.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_ns.py:2: in <module>
from rc import ns
E ImportError: cannot import name ns
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 error in 0.09 seconds ==========================================
(VEnvTestRc) zz#zz:~/Desktop/GitFolders/rc$ which pytest
/home/zz/Desktop/VirtualEnvs/VEnvTestRc/bin/pytest
I am a bit puzzled, it looks like this indicates an import error, but Python does it fine so why is there a problem specifically with PyTest? Any suggestion to the reason / remedy (Question 2)? I googled and stack-overflowed the 'ImportError: cannot import' error for PyTest, but the hits I got were related to missing python path and remedy to this, which does not seem to be the problem here. Any suggestions?
Found the answer:
DO NOT put a __init__.py file in a folder containing TESTS if you plan on using pytest. I had one such file, deleting it solved the problem.
This was actually buried in the comments to the second answer of PATH issue with pytest 'ImportError: No module named YadaYadaYada' so I did not see it, hope it gets more visibility here.
I can't say I understand why this works, but I had the same problem and the tests work fine if I run python -m pytest.
I'm in a virtualenv, with pytest also available globally:
(proj)tom#neon ~/dev/proj$ type -a python
python is /home/tom/.virtualenvs/proj/bin/python
python is /usr/bin/python
(proj)tom#neon ~/dev/proj$ python -V
Python 3.5.2
(proj)tom#neon ~/dev/proj$ type -a pytest
pytest is /home/tom/.virtualenvs/proj/bin/pytest
pytest is /usr/bin/pytest
(proj)tom#neon ~/dev/proj$ pytest --version
This is pytest version 3.5.0, imported from /home/tom/.virtualenvs/proj/lib/python3.5/site-packages/pytest.py
I just solved this by removing the __init__.py in my project root:
.
├── __init__.py <--- removed
├── models
│ ├── __init__.py
│ ├── address.py
│ ├── appointment.py
│ └── client.py
├── requirements.txt
├── setup.cfg
├── tests
│ ├── __init__.py
│ ├── models
│ │ ├── __init__.py
│ │ ├── appointment_test.py
│ │ └── client_test.py
│ └── other_test.py
└── script.py
I had the same problem but for another reason than the ones mentioned:
I had py.test installed globally, while the packages were installed in a virtual environment.
The solution was to install pytest in the virtual environment. (In case your shell hashes executables, as Bash does, use hash -r, or use the full path to py.test)
Had a similar issue and it worked when I added __init__.py file under tests directory.
Simply put an empty conftest.py file in the project root directory, because when pytest discovers a conftest.py, it modifies sys.path so it can import stuff from the conftest module.
A general directory structure can be:
Root
├── conftest.py
├── module1
│ ├── __init__.py
│ └── sample.py
└── tests
└── test_sample.py
In my case I am working in a container and unfortuantely pytest has the tendency to use python2.7 rather than my python3 interpreter of choice.
In my case this worked:
python3 -m pytest
My folder structure
/
app/
-module1.py
-module2.py
-tests/
--test_module1.py
--test_module2.py
requirements.txt
README.md
ANOTHER SUGGESTION
I explored this question and various others on SO and elsewhere... all the stuff about adding (or removing) an empty __init__.py in and/or conftest.py in various parts of the project directory structure, all the stuff about PYTHONPATH, etc., etc.: NONE of these solutions worked for me, in what is actually a very simple situation, and which shouldn't be causing any grief.
I regard this as a flaw in pytest's current setup. In fact I got a message recently from someone on SO who clearly knew his stuff. He said that pytest is not designed to work with (as per Java/Groovy/Gradle) separate "src" and "test" directory structures, and that test files should be mingled in amongst the application directories and files. This perhaps goes some way to providing an explanation ... however, tests, particularly integration/functional tests, don't always necessarily correspond neatly to particular directories, and I think pytest should give users more choice in this regard.
Structure of my project:
project_root
src
core
__init__.py
__main__.py
my_other_file.py
tests
basic_tests
test_first_tests.py
The import problem posed: very simply, __main__.py has a line import my_other_file. This (not surprisingly) works OK when I just run the app, i.e. run python src/core from the root directory.
But when I run pytest with a test which imports __main__.py I get
ModuleNotFoundError: No module named 'my_other_file'
on the line in __main__.py that tries to import "my_other_file". Note that the problem here, in my case, is that, in the course of a pytest test, one application file fails to find another application file in the same package.
USING PYTHONPATH
After a lot of experimentation, and putting an __init__.py file and a confest.py file in just about every directory I could find (I think the crucial files were __init__.py added to "tests" and "basic_tests", see above directory structure), and then setting PYTHONPATH to as follows
PYTHONPATH=D:\My Documents\software projects\EclipseWorkspace\my_project\src\core
... I found it worked. Imports in the testing files had to be tweaked a bit, the general pattern being from core import app, project, but the test files were able to "see" core, and crucially there was no need to mess around with the import commands in the app files.
HOWEVER... for some reason the tests now run much slower using this method! Compared to my solution below, where the core package can be seen to be loaded just once, my suspicion is that the PYTHONPATH solution probably results in vast amounts of code being reloaded again and again. I can't yet confirm this, but I can't see any other explanation.
THE ALTERNATIVE
My alternative fairly simple solution is this:
1 - in __init__.py in that application package (i.e. directory "core"), put the following two lines:
import pathlib, sys
sys.path.append(str(pathlib.Path(__file__).parent))
NB normally there isn't anything in an __init__.py of course. It turns out, as I confirmed by experimentation, that pytest usually (see update below) executes __init__.py in this situation, after pytest has done whatever it has done to mess up the sys.path entries.
2 - UPDATE 2022-01:
The original solution I had found involved putting a conftest.py file in the application directory(ies) - without which things didn't work. This is obviously undesirable. I find that another solution is to put this code in your conftest.py file in your root directory:
def pytest_configure(config):
import src.core # NB this causes `src/core/__init__.py` to run
# set up any "aliases" (optional...)
import sys
sys.modules['core'] = sys.modules['src.core']
... indeed, from my experiments, the effect of putting conftest.py in the application directory seems to be that pytest then runs __init__.py in that directory. This appears to imply that the module is being imported...
(previous suggestion:)
yes, you also HAVE to include an empty "conftest.py" file in the directory "core". Hopefully this should be the only conftest.py you'll need: I experimented with all this, and putting one in the root directory was not necessary (nor did it solve the problem without the suggested code in __init__.py).
3 - finally, in my test function, before calling core.__main__ in my example, I have to import the file I know is about to be imported:
import core.my_other_file
import core.__main__
If you do this in the first test in your file, you will find that sys.modules is set up for all other tests in that file. Better yet, put import core.my_other_file at the very start of the file, before the first test. Unfortunately, from core import * does not seem to work.
Later: this method has some idiosyncracies and limitations. For example, although the -k switch works OK to filter in/out tests or entire files, if you do something like pytest tests/tests_concerning_module_x, it appears that core.__init__.py does NOT get run... so the files in the core module are once again mutually unimportable during testing. Other limitations will probably come to light...
As I say, I regard this as a flaw in pytest's setup. I have absolutely no idea what pytest does to establish a common-sense setup for sys.path, but it's obviously getting it wrong. There should be no need to rely on PYTHONPATH, or whatever, and if indeed this is the "official" solution, the documentation on this subject is sorely lacking.
NB this suggestion of mine has a problem: by adding to sys.path every time pytest runs __init__.py in a module, it means that this new path thereafter becomes permanent in sys.path, both during testing and, more worryingly, during runs of the application itself, if there is anything which actually calls __init__.py. (Incidentally, just going python src/core (as in my example) does NOT cause this to happen. But other things might.)
To cater for this I have a clunky but effective solution:
import pathlib, sys, traceback
frame_list = traceback.extract_stack()
if len(frame_list) > 2:
path_parts = pathlib.Path(frame_list[2].filename).parts
if len(path_parts) > 2:
module_dir_path_str = str(pathlib.Path(__file__).parent)
if sys.platform.startswith('win'):
if path_parts[-3:-1] == ('Scripts', 'pytest.exe'):
sys.testing_context = True
sys.path.append(module_dir_path_str)
elif sys.platform.startswith('lin'):
if path_parts[-3:-1] == ('_pytest', 'config'):
sys.testing_context = True
sys.path.append(module_dir_path_str)
This is based on my examination of the stacktrace when pytest runs something, in a W10 context and Linux Mint 20 context. It means that in an application run there will be no messing with sys.path.
Of course, this may break with future versions of pytest. My version is 6.2.5.
This problem will happen if you have a tests.py file and a tests folder with tests/__init__.py.
During the collection pytest finds the folder, but when it tries to import the test files from the folder, tests.py file will cause the import problem.
To fix simply remove the tests.py file and put all your tests inside the tests/ folder.
For your specific case the fix will be precisely:
Remove the file /home/zz/Desktop/GitFolders/rc/tests.py
Make sure /home/zz/Desktop/GitFolders/rc/tests/__init__.py is present
I solved my problem by setting the PYTHONPATH in Environment Variables for the specific configuration I'm running my tests with.
While you're viewing the test file on PyCharm:
Ctrl + Shift + A
Type Edit Configurations
Set your PYTHONPATH under Environment > Environment variables.
UPDATE
Move into your project's directory root
Create a virtual environment
Activate your newly created virtual environment
Set the variable $PYTHONPATH to the root of your project and export it:
export PYTHONPATH=$(pwd)
Do not remove the __init__.py from the tests/ directory or from the src/ directory.
Also note:
The root of your directory is not a python module so do NOT add an __init__.py
conftest.py is not necessary in the root of your project.
The $PYTHONPATH var will only be available during the current terminal/console session; so you will need to set this each time.
(You can follow the steps previous to this update if you are working in pycharm).
I had a similar issue, exact same error, but different cause. I was running the test code just fine, but against an old version of the module. In the previous version of my code one class existed, while the other did not. After updating my code, I should have run the following to install it.
sudo pip install ./ --upgrade
Upon installing the updated module running pytest produced the correct results (because i was using the correct code base).
Please check here: https://docs.pytest.org/en/documentation-restructure/background/pythonpath.html
I has an issue with pytest (that was solved using python -m pytest); the error was
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/site-packages/...
I found the problem was missing __init__.py in tests/ and tests/subfolders.
In my case, the import error occurred because the package is pointing to another package/directory with the same name and its path is one level above the folder I actually wanted.
I think this also explains why some people need to remove _ init _.py while others need to add back.
I just put print(the_root_package.__path__) (after import the_root_package) in both python console and pytest scripts to compare the difference
BOTTOM LINE: When you do python, the package you import may be different from the package when you run pytest.
I had placed all my tests in a tests folder and was getting the same error. I solved this by adding an __init__.py in that folder like so:
.
|-- Pipfile
|-- Pipfile.lock
|-- README.md
|-- api
|-- app.py
|-- config.py
|-- migrations
|-- pull_request_template.md
|-- settings.py
`-- tests
|-- __init__.py <------
|-- conftest.py
`-- test_sample.py
Install the packages into Your virtual environment.
Then start a new shell and source Your virtual environment again.
As of pytest 7.0, you can now add pythonpath in pytest.ini. No need to add __init__.py or conftest.py in your root directory.
[pytest]
minversion = 7.0
addopts = --cov=src
pythonpath = src
testpaths =
tests
You can run pytest without any parameters.
https://docs.pytest.org/en/7.0.x/reference/reference.html#confval-pythonpath
Here's a medium article! describing the problem!
The issue is which pytest you are using and your use of a virtual environment.
If you have installed pytest system-wide, in other words, outside of a virtual environment, pytest has a nasty habit of only looking outside your virtual environment for modules! If your project is using a module that is only installed in your virtual environment, and you’re using a system-wide pytest, it won’t find the module, even if you’ve activated the virtual environment.1
Here’s the step-by-step:1
Exit any virtual environment
Use Pip to uninstall pytest
Activate your project’s virtual environment
Install pytest inside the virtual environment
pytest will now find your virtual-environment-only packages!
The answer above not work for me. I just solved it by appending the absolute path of the module which not found to the sys.path at top of the test_xxx.py (your test module), like:
import sys
sys.path.append('path')
I was getting this using VSCode. I have a conda environment. I don't think the VScode python extension could see the updates I was making.
python c:\Users\brig\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear test
Test Discovery failed:
I had to run pip install ./ --upgrade
I was experiencing this issue today and solved it by calling python -m pytest from the root of my project directory.
Calling pytest from the same location still caused issues.
My Project dir is organized as:
api/
- server/
- tests/
- test_routes.py
- routes/
- routes.py
- app.py
The module routes was imported in my test_routes.py as: from server.routes.routes import Routes
Hope that helps!
I disagree with the posts saying that you must remove any __init__.py files. What you must instead do is alter the sys.path.
Run an experiment where you print sys.path when running the code normally.
Then print sys.path while running the code via pytest. I think you will find that there is a difference between these two paths, hence why pytest breaks.
To fix this, insert the path from the first experiment at the 0th index of the second.
Let '/usr/exampleUser/Documents/foo' be the first element of print(sys.path) for experiment 1.
Below is code that should fix your issue:
import sys
sys.path[0] = '/usr/exampleUser/Documents/foo'
Put this at the top of your file, before your actual import statement.
Source: I was dealing with this myself and the above process solved it.
Another special case:
I had the problem using tox. So my program ran fine, but unittests via tox kept complaining.
After installing packages (needed for the program) you need to additionally specify the packages used in the unittests in the tox.ini
[testenv]
deps =
package1
package2
...
If it is related to python code that was originally developed in python 2.7 and now migrated into python 3.x than the problem is probably related to an import issue.
e.g.
when importing an object from a file: base that is located in the same directory this will work in python 2.x:
from base import MyClass
in python 3.x you should replace with base full path or .base
not doing so will cause the above problem.
so try:
from .base import MyClass
Yet another massive win for Python's import system. I think the reason there is no consensus is that what works probably depends on your environment and the tools you are using on top of it.
I'm using this from VS Code, in the test explorer under Windows in a conda environment, Python 3.8.
The setup I have got to work is:
mypkg/
__init__.py
app.py
view.py
tests/
test_app.py
test_view.py
Under this setup intellisense works and so does test discovery.
Note that I originally tried the following, as recommended here.
src/
mypkg/
__init__.py
app.py
view.py
tests/
test_app.py
test_view.py
I could find no way of getting this to work from VS Code because the src folder just blew the mind of the import system. I can imagine there is a way of getting this to work from the command line. As a relatively new convert to Python programming it gives me a nostalgic feeling of working with COM, but being slightly less fun.
I find the answer in there :Click here
If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an init.py)
update PYTHONPATH till src folder
export PYTHONPATH=/tmp/pycharm_project_968/src
The TestCase directory must be a Python Package
For anyone who tried everything and still getting error,I have a work around.
In the folder where pytest is installed,go to pytest-env folder.
Open pyvenv.cfg file.
In the file change include-system-site-packages from false to true.
home = /usr/bin
include-system-site-packages = true
version = 3.6.6
Hope it works .Don't forget to up vote.
My 2 cents on this: pytest will fail at chance if you are not using virtual environments. Sometimes it will just work, sometimes not.
Therefore, the solution is:
remove pytest with pip uninstall
create your venv
activate your venv
pip install your project path in editable mode, so it will be treated by pytest as a module (otherwise, pytest wont find your internal imports). You will need a setup.py file for that
install your packages, including pytest
finally, run your tests
The code, using windows PowerShell:
pip uninstall pytest
python.exe -m venv my_env
.\my_env\Scripts\activate
(my_env) pip install -e .
(my_env) pip install pytest pytest-html pandas numpy
Then finally
(my_env) pytest --html="my_testing_report.html"
An example of setup.py, for pip install -e:
import setuptools
setuptools.setup(
name='my_package',
version='devel',
author='erickfis',
author_email='erickfis#gmail.com',
description='My package',
long_description='My gooood package',
packages=setuptools.find_packages(),
classifiers=[
'Programming Language :: Python :: 3',
'Operating System :: OS Independent',
],
include_package_data=True
)
If you run Pytest from a terminal:
Run pytest with the --import-mode=append command-line flag.
Argument description in the official documentation: https://docs.pytest.org/en/stable/pythonpath.html
UPD:
Before I also wrote how to do the same if you use PyCharm, but community does not like extendend answers, so I removed additional information that probably was helpful to someone who have a similar issue.