I have this project structure:
/project_name
main.py
----- __init__.py
------ /modules
-------- __init__.py
-------- module1.py
-------- module2.py
I've edited to add more information. After working and testing a lot of recomendations to solve the problem, nothing works.
Enviroment
Windows
Conda virtual enviroment project python 3.10
VSCode
Problem
When running main.py from VScode
from modules.module1 import *
if __name__ == "__main__":
pass
this error raise
from module1 import *
ModuleNotFoundError: No module named 'module2'
Modules
module1.py
from module2 import *
module2.py
def test():
print("just testing")
So the problem always occurs when from main.py i import a module that imports another module. The second module imported from the module that i have imported from main.py is not found.
Still looking for the solution
You could try to set PythonPath first. If you use vscode to develop, you could set this PythonPath in setting.json
if module1.py and module2.py are in the same directory, you could try to use relative import. Please pay attention to cycle import as well.
from .module1 import Module1
main.py had better to move into src directory.
Pycharm working directory
If you use Pycharm, you can configure the run configuration working directory to the root of your project.
Import your package in editable mode
Create a pyproject.toml file in the root of the project with this content:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "package_name"
version = "0.0.1"
requires-python = ">=3.7"
Your package need to follow this structure:
project_name/
└── src/
└── package_name/
├── __init__.py
└── example.py
To install it in your environment from the root of the project:
user/projects/package_name$ pip install -e .
By doing that, you won't have to worry about PYTHONPATH, workind directory, relative/absolute import. You just import and use your package using the intended path you created, Python will know how to look thanks of the pip install command.
run the file from project_name folder
If you run a file from inside a folder, python will be able to look local module from this folder. You need to call it at the root of the project.
user/projects/package_name$ python -m src/modules/module2.py
Thanks all. Finally i've solved it including this line before importing my file module:
sys.path.append("X:\\path\\root_folder\\")
I used easy_install to install pytest on a Mac and started writing tests for a project with a file structure likes so:
repo/
|--app.py
|--settings.py
|--models.py
|--tests/
|--test_app.py
Run py.test while in the repo directory, and everything behaves as you would expect.
But when I try that same thing on either Linux or Windows (both have pytest 2.2.3 on them), it barks whenever it hits its first import of something from my application path. For instance, from app import some_def_in_app.
Do I need to be editing my PATH to run py.test on these systems?
I'm not sure why py.test does not add the current directory in the PYTHONPATH itself, but here's a workaround (to be executed from the root of your repository):
python -m pytest tests/
It works because Python adds the current directory in the PYTHONPATH for you.
Recommended approach for pytest>=7: use the pythonpath setting
Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:
pyproject.toml example:
[tool.pytest.ini_options]
pythonpath = [
"."
]
pytest.ini example:
[pytest]
pythonpath = .
The path entries are calculated relative to the rootdir, thus . adds repo directory to sys.path in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── app.py
└── tests
├── test_app.py
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
".", "src",
]
or
[pytest]
pythonpath = . src
will add both app and lib modules to sys.path, so
import app
import lib
will both work.
Original answer (not recommended for recent pytest versions; use for pytest<7 only): conftest solution
The least invasive solution is adding an empty file named conftest.py in the repo/ directory:
$ touch repo/conftest.py
That's it. No need to write custom code for mangling the sys.path or remember to drag PYTHONPATH along, or placing __init__.py into dirs where it doesn't belong (using python -m pytest as suggested in Apteryx's answer is a good solution though!).
The project directory afterwards:
repo
├── conftest.py
├── app.py
├── settings.py
├── models.py
└── tests
└── test_app.py
Explanation
pytest looks for the conftest modules on test collection to gather custom hooks and fixtures, and in order to import the custom objects from them, pytest adds the parent directory of the conftest.py to the sys.path (in this case the repo directory).
Other project structures
If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an __init__.py), for example:
repo
├── conftest.py
├── spam
│ ├── __init__.py
│ ├── bacon.py
│ └── egg.py
├── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
src layout
Although this approach can be used with the src layout (place conftest.py in the src dir):
repo
├── src
│ ├── conftest.py
│ ├── spam
│ │ ├── __init__.py
│ │ ├── bacon.py
│ │ └── egg.py
│ └── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
beware that adding src to PYTHONPATH mitigates the meaning and benefits of the src layout! You will end up with testing the code from repository and not the installed package. If you need to do it, maybe you don't need the src dir at all.
Where to go from here
Of course, conftest modules are not just some files to help the source code discovery; it's where all the project-specific enhancements of the pytest framework and the customization of your test suite happen. pytest has a lot of information on conftest modules scattered throughout their docs; start with conftest.py: local per-directory plugins
Also, SO has an excellent question on conftest modules: In py.test, what is the use of conftest.py files?
I had the same problem. I fixed it by adding an empty __init__.py file to my tests directory.
Yes, the source folder is not in Python's path if you cd to the tests directory.
You have two choices:
Add the path manually to the test files. Something like this:
import sys, os
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
Run the tests with the env var PYTHONPATH=../.
Run pytest itself as a module with:
python -m pytest tests
This happens when the project hierarchy is, for example, package/src package/tests and in tests you import from src. Executing as a module will consider imports as absolute rather than relative to the execution location.
You can run with PYTHONPATH in project root
PYTHONPATH=. py.test
Or use pip install as editable import
pip install -e . # install package using setup.py in editable mode
I had the same problem in Flask.
When I added:
__init__.py
to the tests folder, the problem disappeared :)
Probably the application couldn't recognize folder tests as a module.
I created this as an answer to your question and my own confusion. I hope it helps. Pay attention to PYTHONPATH in both the py.test command line and in the tox.ini.
https://github.com/jeffmacdonald/pytest_test
Specifically: You have to tell py.test and tox where to find the modules you are including.
With py.test you can do this:
PYTHONPATH=. py.test
And with tox, add this to your tox.ini:
[testenv]
deps= -r{toxinidir}/requirements.txt
commands=py.test
setenv =
PYTHONPATH = {toxinidir}
I fixed it by removing the top-level __init__.py in the parent folder of my sources.
I started getting weird ConftestImportFailure: ImportError('No module named ... errors when I had accidentally added __init__.py file to my src directory (which was not supposed to be a Python package, just a container of all source).
It is a bit of a shame that this is an issue in Python... But just adding this environment variable is the most comfortable way, IMO:
export PYTHONPATH=$PYTHONPATH:.
You can put this line in you .zshrc or .bashrc file.
I was having the same problem when following the Flask tutorial and I found the answer on the official Pytest documentation.
It's a little shift from the way I (and I think many others) are used to do things.
You have to create a setup.py file in your project's root directory with at least the following two lines:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
where PACKAGENAME is your app's name. Then you have to install it with pip:
pip install -e .
The -e flag tells pip to install the package in editable or "develop" mode. So the next time you run pytest it should find your app in the standard PYTHONPATH.
I had a similar issue. pytest did not recognize a module installed in the environment I was working in.
I resolved it by also installing pytest into the same environment.
Also if you run pytest within your virtual environment make sure pytest module is installed within your virtual environment. Activate your virtual environment and run pip install pytest.
For me the problem was tests.py generated by Django along with tests directory. Removing tests.py solved the problem.
I got this error as I used relative imports incorrectly. In the OP example, test_app.py should import functions using e.g.
from repo.app import *
However liberally __init__.py files are scattered around the file structure, this does not work and creates the kind of ImportError seen unless the files and test files are in the same directory.
from app import *
Here's an example of what I had to do with one of my projects:
Here’s my project structure:
microbit/
microbit/activity_indicator/activity_indicator.py
microbit/tests/test_activity_indicator.py
To be able to access activity_indicator.py from test_activity_indicator.py I needed to:
start test_activity_indicatory.py with the correct relative import:
from microbit.activity_indicator.activity_indicator import *
put __init__.py files throughout the project structure:
microbit/
microbit/__init__.py
microbit/activity_indicator/__init__.py
microbit/activity_indicator/activity_indicator.py
microbit/tests/__init__.py
microbit/tests/test_activity_indicator.py
According to a post on Medium by Dirk Avery (and supported by my personal experience) if you're using a virtual environment for your project then you can't use a system-wide install of pytest; you have to install it in the virtual environment and use that install.
In particular, if you have it installed in both places then simply running the pytest command won't work because it will be using the system install. As the other answers have described, one simple solution is to run python -m pytest instead of pytest; this works because it uses the environment's version of pytest. Alternatively, you can just uninstall the system's version of pytest; after reactivating the virtual environment the pytest command should work.
I was getting this error due to something even simpler (you could even say trivial). I hadn't installed the pytest module. So a simple apt install python-pytest fixed it for me.
'pytest' would have been listed in setup.py as a test dependency. Make sure you install the test requirements as well.
Since no one has suggested it, you could also pass the path to the tests in your pytest.ini file:
[pytest]
...
testpaths = repo/tests
See documentation: https://docs.pytest.org/en/6.2.x/customize.html#pytest-ini
Side effect for Visual Studio Code: it should pick up the unit test in the UI.
We have fixed the issue by adding the following environment variable.
PYTHONPATH=${PYTHONPATH}:${PWD}/src:${PWD}/test
As pointed out by Luiz Lezcano Arialdi, the correct solution is to install your package as an editable package.
Since I am using Pipenv, I thought about adding to his answer a step-by-step how to install the current path as an edible with Pipenv, allowing to run pytest without the need of any mangling code or lose files.
You will need to have the following minimal folder structure (documentation):
package/
package/
__init__.py
module.py
tests/
module_test.py
setup.py
setup.py mostly has the following minium code (documentation):
import setuptools
setuptools.setup(name='package', # Change to your package name
packages=setuptools.find_packages())
Then you just need to run pipenv install --dev -e . and Pipenv will install the current path as an editable package (the --dev flag is optional) (documentation).
Now you should be able to run pytest without problems.
If this pytest error appears not for your own package, but for a Git-installed package in your package's requirements.txt, the solution is to switch to editable installation mode.
For example, suppose your package's requirements.txt had the following line:
git+https://github.com/foo/bar.git
You would instead replace it with the following:
-e git+https://github.com/foo/bar.git#egg=bar
If nothing works, make sure your test_module.py is listed under the correct src directory.
Sometimes it will give ModuleNotFoundError not because modules are misplaced or export PYTHONPATH="${PWD}:${PYTHONPATH}" is not working, its because test_module.py is placed into a wrong directory under the tests folder.
it should be 1-to-1 mapping relation recursively instead of the root folder should be named as "tests" and the name of the file that include test code should starts with "test_",
for example,
./nlu_service/models/transformers.py
./tests/models/test_transformers.py
This was my experience.
Very often the tests were interrupted due to module being unable to be imported.
After research, I found out that the system is looking at the file in the wrong place and we can easily overcome the problem by copying the file, containing the module, in the same folder as stated, in order to be properly imported.
Another solution proposal would be to change the declaration for the import and show MutPy the correct path of the unit. However, due to the fact that multiple units can have this dependency, meaning we need to commit changes also in their declarations, we prefer to simply move the unit to the folder.
My solution:
Create the conftest.py file in the test directory containing:
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + "/relative/path/to/code/")
This will add the folder of interest to the Python interpreter path without modifying every test file, setting environment variable or messing with absolute/relative paths.
With a project setup as follows:
---------------------
root
FrameworkPackage1
__init__.py
sourcefile1.py
FrameworkPackage2
__init__.py
sourcefile2.py
apps
Project
src
MyApp
__init__.py
__main__.py
setup.py
README.md
---------------------
When I'm creating the setup.py, from what I understand, I use package_dir to set the location of these packages.
---------------------
packages=['MyApp', 'FrameworkPackage1', 'FrameworkPackage2'],
package_dir={'': 'src',
'FrameworkPackage1': '../../FrameworkPackage1',
'FrameworkPackage2': '../../FrameworkPackage2'}
---------------------
So this correctly builds a package with all the required files. However, when I try to install, it fails, and if I just try to untar/gz the package file it puts FrameworkPackage1/2 in the "../../.." dir from where the unzip happens.
Ideally I'd like the package to work as follows and install from pip so I could run the following:
import MyApp as ma
import FrameworkPackage1 as fp1
import FrameworkPackage2 as fp2
print(ma.Function())
print(fp1.OtherFunction())
print(fp2.OtherFunction())
Is there a way to set the frameworks to be found in "../../../" but install into the root of the distribution?
Firstly, as #a_guest suggested, shouldn't the package_dir look like this?
packages=['MyApp', 'FrameworkPackage1', 'FrameworkPackage2'],
package_dir={'': 'src',
'FrameworkPackage1': '../../FrameworkPackage1',
'FrameworkPackage2': '../../FrameworkPackage2'}
Alternatively, you could try adding a __init__.py to the root folder so that it is recognized as a python folder (based on this question)
Secondly, instead of using this bundled structure for your package, you could either:
If the Framework packages are used elsewhere: Treat each package separately. This would allow you to evolve them separately, and add them to your MyApp by simply including them in the requirements.txt (or equivalents). A con of this is that each one would have its own setup.py, but this is a much better packaging practice.
If the Framework packages are not used elsewhere (or you just want your local copy): Switch to a project setup with the setup.py directly in the main folder ( package_dir={'': 'src', 'FrameworkPackage1': 'src', 'FrameworkPackage2': 'src'}, with a structure looking like:
---------------------
...
Project
src
MyApp
__init__.py
__main__.py
FrameworkPackage1
__init__.py
sourcefile1.py
FrameworkPackage2
__init__.py
sourcefile2.py
setup.py
README.md
---------------------
I’m looking for solution for designing my program.
My program consists of 3 blocks:
Classes
Functions
Other utilities
I want to structure my program this way:
program_folder/
main.py
classes_folder/
class_1.py
class_2.py
functions_folder/
set_of_func_1.py
set_of_func_1.py
utilities_folder/
set_of_utilities_1.py
set_of_utilities_1.py
I want to:
any scripts in «classes_folder» were able to import any of scripts in
«functions_folder».
any scripts in «functions_folder» were able
to import any of scripts in «utilities_folder».
all scripts were
normally used by main.py.
all scripts in «classes_folder»,
«functions_folder» and «utilities_folder» could be tested when worked
as «main» (if __name__ == “__main__”: some tests)
«program_folder»
could be in any place in my computer (there shouldn’t be dependency
on exact path to «program_folder»).
From all the above I thought I have to:
Change import search path for all scripts in «classes_folder»,
«functions_folder» and «utilities_folder».
Set current working
directory to «program_folder» for all scripts?
Is there a way I can do it?
Does my idea look good or have I put there some unexpected problems?
You can create a skeleton project like the following:
/path/to/project/
setup.py
my_project/
__init__.py
a/
__init__.py
b/
__init__.py
==> ./my_project/__init__.py <==
print('my_project/__init__.py')
==> ./my_project/a/__init__.py <==
import my_project
print('my_project/a/__init__.py')
==> ./my_project/b/__init__.py <==
import my_project.a
print('my_project/b/__init__.py')
==> ./setup.py <==
from distutils.core import setup
setup(name='my_project',
version='1.0',
description='my_project',
author='author',
packages=['my_project'])
Then you can install the project locally using pip install -e /path/to/project/ (the project folder is not copied, just gets registered; there's a dependency on the exact path, but this dependency is not hard-coded in project files themselves).
As the result, import my_project, import my_project.a etc. do that they mean:
$ python my_project/b/__init__.py
my_project/__init__.py
my_project/a/__init__.py
my_project/b/__init__.py
A common Python project structure could look like this:
project_name/
setup.py
requirements.txt
project_name/
__main__.py
classes/
__init__.py
class1.py
class2.py
functions/
__init__.py
functions.py
utils/
__init__.py
utils.py
Then, you could modify your imports from absolute to relative and run your package using something like:
$ /path/to/project_name> python -m project_name
Note that setup.py is only required if you want to install your package under some of your interpreters.
Note: see comments below also
My project contains three Python applications. Application 1 is a web app. Applications 2 and 3 contain scripts downloading some data.
All three apps need to use a module Common containing a "model" (classes that are saved to database) and common settings.
I have no clue how to structure this project. I could create three directories, one for each application, and copy Common three times into their directories (doesn't seem right).
Another idea that comes to mind is; create a main directory and put there all files from Common, including __init__.py. Then, crete three subdirectories (submodules), one for each application.
Another way would be installing Common using pip, but that means I would have to reinstall every time I change something in that module.
In previous projects I used .NET - the equivalent in that world would be a Solution with four projects, one of them being Common.
Any suggestions?
I have a similar project that is set up like this
project_root/
App1/
__init__.py
FlaskControlPanel/
app.py
static/
templates/
models/
__init__.py
mymodels.py
Then, I run everything from project_root. I have a small script (either batch or shell depending on my environment) that sets PYTHONPATH=. so that imports work correctly. This is done because I usually develop using PyCharm, where the imports "just work", but when I deploy the final product the path doesn't match what it did in my IDE.
Once the PYTHONPATH is set to include everything from your project root, you can do standard imports.
For example, from my FlaskControlPanel app.py, I have this line:
from models.mymodels import Model1, Model2, Model3
From the App1 __init__.py I have the exact same import statement:
from models.mymodels import Model1, Model2, Model3
I can start the Flask application by running this from my command line (in Windows) while I am in the project_root directory:
setlocal
SET PYTHONPATH=.
python FlaskControlPanel\app.py
The setlocal is used to ensure the PYTHONPATH is only modified for this session.
I like this approach
projects/
__init__.py
project1/
__init__.py
project2/
__init__.py
lib1/
__init__.py
libfile.py
lib2/
__init__.py
So, I need to cd into the projects folder.
To start a projects use
python -m project_name
This allows me to easily import from any external lib like
from lib1.libfile import [imoprt what you want]
or
from lib1 import libfile
Make standard Python modules from your apps. I recommend structure like this:
apps/
common/
setup.py
common/
__init__.py
models.py
app1/
setup.py
app1/
__init__.py
models.py
project/
requirements.txt
Basic setup.py for app common:
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(
name='common',
version='1.0.0',
packages=find_packages(),
zip_safe=False,
)
Make similar setup.py for other apps.
Set editable "-e" option for your apps in requirements.txt:
-e apps/common
-e apps/app1
Install requirements with pip:
$ pip install -r requirements.txt
Editable option means that source files will be linked into Python enviroment. Any change in source files of your apps will have immediate effect without reinstalling them.
Now you can import models from your common app (or any other app) anywhere (in other apps, project files, ...).
I would create a structure like this:
project_root/
app1/
__init__.py
script.py
common/
__init__.py
models.py (all "common" models)
app1/script.py
import os, sys
# add parent directory to pythonpath
basepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
if basepath not in sys.path:
sys.path.append(basepath)
from common.models VeryCommonModel
print VeryCommonModel
If you don't want to set the python path at runtime, set the python path before running the script:
$ export PYTHONPATH=$PYTHONPATH:/path/to/project_root
And then you can do:
python app1/script.py