My source files are located under src and my test files are located under tests. When I want to run a test file, say python myTest.py, I get an import error: "No module named ASourceModule.py".
How do I import all the modules from source needed to run my tests?
You need to add that directory to the path:
import sys
sys.path.append('../src')
Maybe put this into a module if you are using it a lot.
If you don't want to add the source path to each test file or change your PYTHONPATH, you can use nose to run the tests.
Suppose your directory structure is like this:
project
package
__init__.py
module.py
tests
__init__.py
test_module.py
You should import the module normally in the test_module.py (e.g. from package import module). Then run the tests by running nosetests in the project folder. You can also run specific tests by doing nosetests tests/test_module.py.
The __init__.py in the tests directory is necessary if you want to run the tests from inside it.
You can install nose easily with easy_install or pip:
easy_install nose
or
pip install nose
nose extends unittest in a lot more ways, to learn more about it you can check their website: https://nose.readthedocs.org/en/latest/
On my system (Windows 10), I was required to do something like this:
import sys
import os
sys.path.append(os.path.dirname(os.path.realpath(__file__)) + "/../src")
Appending the relative directory directly to sys.path did not work
The best (most manageable) solution appears to be using a virtualenv and setuptools/distribute to install a development copy of your (src) package. That way your tests execute against a fully "installed" system.
In the pystest docs there is a section on "good practices" explaining this approach, see here.
For those using Pytest:
Make sure src is recognized as a package by putting an empty__init__.py inside.
Put an empty conftest.py in the project folder.
Make sure there is no __init__.py in the test directory.
Looks like this:
project
conftest.py
src
__init__.py
module.py
test
test_module.py
And make sure module.py contains the line:
from src import module
Source: Pytest docs
Related
I am an experienced java enterprise developer but very new to python enterprise development shop. I am currently, struggling to understand why some imports work while others don't.
Some background: Our dev team recently upgraded python from 3.6 to 3.10.5 and following is our package structure
src/
bunch of files (dockerfile, Pipfile, requrirements.txt, shell scripts, etc)
package/
__init__.py
moduleA.py
subpackage1/
__init__.py
moduleX.py
moduleY.py
subpackage2/
__init__.py
moduleZ.py
tests/
__init__.py
test1.py
Now, inside the moduleA.py, I am trying to import subpackage2/moduleZ.py like so
from .subpackage2 import moduleZ
But, I get the error saying
ImportError: attempted relative import with no known parent package
The funny thing is that if I move moduleA.py out of package/ and into src/ then it is able to find everything. I am not sure why is this the case.
I run the moduleA.py by executiong python package/moduleA.py.
Now, I read that maybe there is a problem becasue you have you give a -m parameter if running a module as a script (or something on those lines). But, if I do that, I get the following error:
ModuleNotFoundError: No module names 'package/moduleA.py'
I even try running package1/moduleA and remove the .py, but that does not work either. I can understand why as I technically never installed it ?
All of this happened because apparently, the tests broke and to make it work they added relative imports. They changed the import from "from subpackage2 import moduleZ" to "from .subpackage2 import moduleZ" and the tests started working, but the app started failing.
Any understanding I can get would be much appreciated.
The -m parameter is used with the import name, not the path. So you'd use python3 -m package.moduleA (with . instead of /, and no .py), not python3 -m package/moduleA.py.
That said, it only works if package.moduleA is locatable from one of the roots in sys.path. Shy of installing the package, the simplest way to make it work is to ensure your working directory is src (so package exists in the working directory):
$ cd path/to/src
$ python3 -m package.moduleA
and, with your existing setup, if moduleA.py includes a from .subpackage2 import moduleZ, the import should work; Python knows package.moduleA is a module within package, so it can use a relative import to look for a sibling package to moduleA named subpackage2, and then inside it it can find moduleZ.
Obviously, this is brittle (it only works if you cd to the src root directory before running Python, or hack the path to src in PYTHONPATH, which is terrible hack if the code ever has to be run by anyone else); ideally you make this an installable package, install it (in global site-packages, user site-packages, or within a virtual environment created with the built-in venv module or the third-party virtualenv module), and then your working directory no longer matters (since the site-packages will be part of your sys.path automatically). For simple testing, as long as the working directory is correct (not sure what it was for you), and you use -m correctly (you were using it incorrectly), relative imports will work, but it's not the long term solution.
So first of all - the root importing directory is the directory from which you're running the main script.
This directory by default is the root for all imports from all scripts.
So if you're executing script from directory src you can do such imports:
from package.moduleA import *
from package.subpackage1.moduleX import *
But now in files moduleA and moduleX you need to make imports based on root folder. If you want to import something from module moduleY inside moduleX you need to do:
# this is inside moduleX
from package.subpackage1.moduleY import *
This is because python is looking for modules in specific locations.
First location is your root directory - directory from which you execute your main script.
Second location is directory with modules installed by PIP.
You can check all directories using following:
import sys
for p in sys.path:
print(p)
Now to solve your problem there are couple solutions.
The fast one but IMHO not the best one is to add all paths with submodules to sys.path - list variable with all directories where python is looking for modules.
new_path = "/path/to/application/app/folder/src/package/subpackage1"
if new_path not in sys.path:
sys.path.append(new_path)
Another solution is to use full path for imports in all package modules:
from package.subpackage1.moduleX import *
I think in your case it will be the correct solution.
You can also combine 2 solutions.
First add folders with subpackages to sys.path and use subpackage folders as a root folders for imports. But it's good solution only if you have complex submodule structure. And it's not the best solution if in future you will need to deploy your package as a wheel or share between multiple projects.
I have a project where I want to VS Code's discover tests and other testing features to make testing easier. I have a problem that imports in test files break when I try to discover tests.
I have a file structure like so:
project\
__init__.py
package1\
module1.py
__init__.py
tests\
test.py
__init__.py
In test.py I have a line:
import project.package1.module1 as module1
I run my project by calling python -m project in the root folder, and I am able to run tests successfully by calling python -m pytest project from the root folder.
When I run VS Code's "discover tests" feature or try to step through a file with the debugger, I receive an error 'ModuleNotFoundError: No module named project'.
Does anyone know how to solve this problem?
I had the same issue. The solution that worked for me was to introduce a .envfile that holds my PYTHONPATH entries, relative to my workspace folder.
PYTHONPATH="path1:path2:pathN"
Then I added a line to my workspace settings that specifies the location of my .env file.
// ...
"python.envFile": "${workspaceFolder}/.env",
// ...
I had the same issue where I was able to run pytest and python -m pytest successfully in the terminal within VSCode but the discovery was failing. My solution was to implement the failing import in the following way
import sys
sys.path.insert(0, '/full/path/to/package1/')
from package1.module1 import module1
Note that VSCode was opened with the project folder being the root.
Next solution works for Linux and Windows,
import sys
from pathlib import Path
sys.path.insert(0, str(Path('package1/').resolve()))
It's based on #Chufolon answer. My StackOverflow reputation doesn't allow me to just comment on his answer. I prefer his solution because in the .env there could be sensitive information (passwords, ...) that shouldn't be shared (omit it in .gitignore file) for security reasons; and also because __init__.py is shared by default through Git.
I have some Python package and some tests. The files are layed out following http://pytest.org/latest/goodpractices.html#choosing-a-test-layout-import-rules
Putting tests into an extra directory outside your actual application
code, useful if you have many functional tests or for other reasons
want to keep tests separate from actual application code (often a good
idea):
setup.py # your distutils/setuptools Python package metadata
mypkg/
__init__.py
appmodule.py
tests/
test_app.py
My problem is, when I run the tests py.test, I get an error
ImportError: No module named 'mypkg'
I can solve this by installing the package python setup.py install but this means the tests run against the installed package, not the local one, which makes development very tedious. Whenever I make a change and want to run the tests, I need to reinstall, else I am testing the old code.
What can I do?
I know this question has been already closed, but a simple way I often use is to call pytest via python -m, from the root (the parent of the package).
$ python -m pytest tests
This works because -m option adds the current directory to the python path, and hence mypkg is detected as a local package (not as the installed).
See:
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-python-m-pytest
The normal approach for development is to use a virtualenv and use pip install -e . in the virtualenv (this is almost equivalent to python setup.py develop). Now your source directory is used as installed package on sys.path.
There are of course a bunch of other ways to get your package on sys.path for testing, see Ensuring py.test includes the application directory in sys.path for a question with a more complete answer for this exact same problem.
On my side, while developing, I prefer to run tests from the IDE (using a runner extension) rather than using the command line. However, before pushing my code or prior to a release, I like to use the command line.
Here is a way to deal with this issue, allowing you to run tests from both the test runner used by your IDE and the command line.
My setup:
IDE: Visual Studio Code
Testing: pytest
Extension (test runner): https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
Work directory structure (my solution should be easily adaptable to your context):
project_folder/
src/
mypkg/
__init__.py
appmodule.py
tests/
mypkg/
appmodule_test.py
pytest.ini <- Use so pytest can locate pkgs from ./src
.env <- Use so VsCode and its extention can locate pkgs from ./src
.env:
PYTHONPATH="${PYTHONPATH};./src;"
pytest.ini (tried with pytest 7.1.2):
[pytest]
pythonpath = . src
./src/mypkg/appmodule.py:
def i_hate_configuring_python():
return "Finally..."
./tests/mypkg/appmodule_test.py:
from mypkg import app_module
def test_demo():
print(app_module.i_hate_configuring_python())
This should do the trick
Import the package using from .. import mypkg. For this to work you will need to add (empty) __init__.py files to the tests directory and the containing directory. py.test should take care of the rest.
I have the following folder structure;
myapp\
myapp\
__init__.py
tests\
test_myapp.py
and my pwd is
C:\Users\wwerner\programming\myapp\
I have the following test setup:
import sys
import pprint
def test_cool():
pprint.pprint(sys.path)
assert False
That produces the following paths:
['C:\\Users\\wwerner\\programming\\myapp\\tests',
'C:\\Users\\wwerner\\programming\\envs\\myapp\\Scripts',
'C:\\Windows\\system32\\python34.zip',
'C:\\Python34\\DLLs',
'C:\\Python34\\lib',
'C:\\Python34',
'C:\\Users\\wwerner\\programming\\envs\\myapp',
'C:\\Users\\wwerner\\programming\\envs\\myapp\\lib\\site-packages']
And when I try to import myapp I get the following error:
ImportError: No module named 'myapp'
So it looks like it's not adding the current directory to my path.
By changing my import line to look like this:
import sys
sys.path.insert(0, '.')
import myapp
I am then able to import myapp with no problems.
Why does my current directory not show up in the path when running pytest? Is my only workaround to insert . into the sys.path? (I'm using Python 3.4 if it matters)
Ahah!
After comparing the layout of my cookiecutter repo, it turns out to be way more simple (and better) than that.
tests/
__init__.py
test_myapp.py
A simple addition of the __init__.py file to my test dir allows me to run py.test from my main directory.
Using an installable package
If you have an installable package (setup.py or pyproject.toml file with a build-system defined) then you want to test against the installed package.
pip install --editable .
pytest
The simplest possible way to make the project shown in the question into an installable package would be by adding this setup.py:
from setuptools import setup
setup(
name="myapp",
version="0.1",
packages=["myapp"],
)
This will put the myapp code at /path/to/myapp/.venv/lib/python3.XY/site-packages, which is in the sys.path of the virtual environment. Now myapp can be imported from the site-packages dir, just as it would be for a user installation. It is neither necessary nor desirable for the current working directory to be present on sys.path during test execution.
Not using an installable package
The project shown in the question does not have any installer, so it can't be installed. It can still be tested by making sure the project root (i.e. the directory which contains both myapp and tests as subdirectories) is present on sys.path.
The best way to do this is to use python -m pytest, rather than invoking the bare pytest command. When you use python -m pytest it adds the current working directory to the start of sys.path. That's the normal Python behavior when executing a package as __main__ (documented here) and it's also a documented usage for pytest - see Invoking pytest versus python -m pytest.
Why does adding an __init__.py to the tests subdirectory (not) work?
The directory structure shown in the question is the "Tests outside application code" pattern, documented here. This is also the directory structure I recommend, since it creates a clear distinction between library/application code and test code.
It's not recommended to add __init__.py files inside the test directories when using a "Tests outside application code" structure, since the test files aren't intended to be "packaged" (e.g. test files do not really need to import from other test files, and they do not need to be installed at all for end users of your package).
The reason adding a myapp/__init__.py actually allows myapp to be imported by pytest, as described in Wayne's answer is actually an accident due to the way test discovery appends sys.path during the test collection phase. This is described as "problematic" in the docs
... this introduces a subtle problem: in order to load the test modules from the tests directory, pytest prepends the root of the repository to sys.path, which adds the side-effect that now mypkg is also importable
They go on to strongly recommend using the src-layout if you intend to have __init__.py files inside test directories, to avoid this confusion of the import system.
But perhaps the best reason not to rely on this side-effect is that pytest collection actually can work in multiple modes (see import modes), and Wayne's answer relies upon pytest using the default "prepend" mode. It is currently mentioned that a future version will switch to "importlib" mode as default:
We intend to make importlib the default in future releases.
The accepted answer does not work with pytest --import-mode=importlib and so will stop working altogether at some stage.
sys.path automatically has the script's directory in it, and not the current working directory.
I am guessing that your script in placed in tests directory. Based on this assumption, your code should look like this:
import sys
import os
ROOT_DIR = os.path.dirname(os.path.dirname(__file__))
sys.path.append(ROOT_DIR)
import myapp # Should work now
Use the environment variable PYTHONPATH.
In Windows:
set PYTHONPATH=.
py.test
In Unix:
PYTHONPATH=. py.test
I have a folder structure like this
App
--App
--app.py
--Docs
--Tests
--test_app.py
In my test_app.py file, I have a line to import my app module. When I run py.test on the root folder, I get this error about no module named app. How should I configure this?
Working with Python 3 and getting the same error on a similar project layout, I solved it by adding an __init__ file to my tests module.
$ touch tests/__init__.py
I'm not great at packaging and importing, but I think that this helps pytest work out where the target App module is located.
I already had an __init__.py file in the /App/App directory and wanted to run tests from the project root without any path-mangling magic:
python -m pytest tests
The output immediately looks like this:
➟ python -m pytest tests
====================================== test session starts ======================================
platform linux -- Python 3.5.1, pytest-2.9.0, py-1.4.31, pluggy-0.3.1
rootdir: /home/andrew/code/app, inifile:
plugins: teamcity-messages-1.17
collected 46 items
... lines omitted ...
============================= 44 passed, 2 skipped in 1.61 seconds ==============================
I had a similar problem and had to delete __init__.py from the root and add an __init__.py to the tests folder.
So you are running py.test from /App. Are you sure /App/App is in your $PYTHONPATH?
If it's not, code that tries to import app will fail with such a message.
EDIT0: including the info from my comment below, for completeness.
An attempt to import app will only succeed if it was executed inside /App/App, which is not the case here. You probably want to make /App/App a package by putting __init__.py inside it, and change your import to qualify app as from App import app.
EDIT1: by request, adding further explanation from my second comment below.
By putting __init__.py inside /App/App, that directory becomes a package. Which means you can import from it, as long as it - the directory - is visible in the $PYTHONPATH. I.e. you can do from App import app if /App is in the $PYTHONPATH. Your current working directory gets automatically added to $PYTHONPATH, so when you run a script from /App, the import will work.
Running pytest with the python -m pytest command helps with this exact thing.
Since your current package is not yet in your $PYTHONPATH or sys.path - pytest gets this error.
By using python -m pytest you automatically add the working directory into sys.path for running pytest. Their documentation also mentions:
This is almost equivalent to invoking the command line script pytest
I also got same error while running test cases for my app located as below
myproject
--app1
--__init.py__
--test.py
--app2
--__init.py__
--test.py
--__init.py__
I deleted my myproject's init.py file to run my test cases.
TL;DR
You might as well add an empty conftest.py file to your root app folder.
(if you take a look at the question folder structure, that would be the same level as the "Tests" folder, not inside of it).
More info:
Pytest looks for conftest.py files inside all your project folders.
conftest.py provides configuration for the file tree pytest finds it in. Because pytest somehow scans all subdirectories starting from conftest.py folder, it should find packages/modules outside the tests folder (as long as a conftest.py file is in your app root folder).
Eventually, you might want to write some code in your empty conftest.py, specially to share fixtures among different tests files, in order to avoid duplicate code. Afterall, the DRY principle (Don't Repeat Yourself) should also be follwed when writing tests.
Adding __init.py__ to the tests folder also should help pytest to find modules throughout your application. However, note that Python 3.3+ has implicit namespace packages that allow it to create a packages without an __init__.py file. That been said, creating __init__.py files for this specific purpose seems more like a a workaround for pytest than a python requirement. More about that in: Is __init__.py not required for packages in Python 3.3+
I got the similar issue. And after trying multiple things including installing pytest on virtual environment and adding/removing __init__.py file from the package, none worked for me.
Solution that worked for me is(windows solution):
Added Project folder(not package folder) to python path(set PYTHONPATH=%PYTHONPATH%;%CD%)
Ran my script from Project Folder and boom, it worked.
I hit the same issue.
my-app
--conf
--my-app
--tests
I set the __init__.py files. I added a conftest.py ( for sharing pytest.fixtures ). I added this to my Poetry file ( pyproject.toml:
[tool.pytest.ini_options]
pythonpath = [
"."
Turned out it was my use of hyphens and not underscores ! Noo...
# pytest can't find Module
--my-app
# works
--my_app
What worked for me: I had to make absolute imports in my test file and call python -m test in the root folder.
This worked for me:
Went to parent app, and pip install -e . (install a local and editable app).