I have a package with a directory "tests" in which I'm storing my unit tests. My package looks like:
.
├── LICENSE
├── models
│ └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│ ├── db
│ │ └── test_employee.py
│ └── test_tc.py
└── todo.txt
From my package directory, I want to be able to find both tests/test_tc.py and tests/db/test_employee.py. I'd prefer not to have to install a third-party library (nose or etc) or have to manually build a TestSuite to run this in.
Surely there's a way to tell unittest discover not to stop looking once it's found a test? python -m unittest discover -s tests will find tests/test_tc.py and python -m unittest discover -s tests/db will find tests/db/test_employee.py. Isn't there a way to find both?
In doing a bit of digging, it seems that as long as deeper modules remain importable, they'll be discovered via python -m unittest discover. The solution, then, was simply to add a __init__.py file to each directory to make them packages.
.
├── LICENSE
├── models
│ └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│ ├── db
│ │ ├── __init__.py # NEW
│ │ └── test_employee.py
│ ├── __init__.py # NEW
│ └── test_tc.py
└── todo.txt
So long as each directory has an __init__.py, python -m unittest discover can import the relevant test_* module.
If you're okay with adding a __init__.py file inside tests, you can put a load_tests function there that will handle discovery for you.
If a test package name (directory with __init__.py) matches the
pattern then the package will be checked for a 'load_tests' function. If
this exists then it will be called with loader, tests, pattern.
If load_tests exists then discovery does not recurse into the package,
load_tests is responsible for loading all tests in the package.
I'm far from confident that this is the best way, but one way to write that function would be:
import os
import pkgutil
import inspect
import unittest
# Add *all* subdirectories to this module's path
__path__ = [x[0] for x in os.walk(os.path.dirname(__file__))]
def load_tests(loader, suite, pattern):
for imp, modname, _ in pkgutil.walk_packages(__path__):
mod = imp.find_module(modname).load_module(modname)
for memname, memobj in inspect.getmembers(mod):
if inspect.isclass(memobj):
if issubclass(memobj, unittest.TestCase):
print("Found TestCase: {}".format(memobj))
for test in loader.loadTestsFromTestCase(memobj):
print(" Found Test: {}".format(test))
suite.addTest(test)
print("=" * 70)
return suite
Pretty ugly, I agree.
First you add all subdirectories to the test packages's path (Docs).
Then, you use pkgutil to walk the path, looking for packages or modules.
When it finds one, it then checks the module members to see whether they're classes, and if they're classes, whether they're subclasses of unittest.TestCase. If they are, the tests inside the classes are loaded into the test suite.
So now, from inside your project root, you can type
python -m unittest discover -p tests
Using the -p pattern switch. If all goes well, you'll see what I saw, which is something like:
Found TestCase: <class 'test_tc.TestCase'>
Found Test: testBar (test_tc.TestCase)
Found Test: testFoo (test_tc.TestCase)
Found TestCase: <class 'test_employee.TestCase'>
Found Test: testBar (test_employee.TestCase)
Found Test: testFoo (test_employee.TestCase)
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK
Which is what was expected, each of my two example files contained two tests, testFoo and testBar each.
Edit: After some more digging, it looks like you could specify this function as:
def load_tests(loader, suite, pattern):
for imp, modname, _ in pkgutil.walk_packages(__path__):
mod = imp.find_module(modname).load_module(modname)
for test in loader.loadTestsFromModule(mod):
print("Found Tests: {}".format(test._tests))
suite.addTests(test)
This uses the loader.loadTestsFromModule() method instead of the loader.loadTestsFromTestCase() method I used above. It still modifies the tests package path and walks it looking for modules, which I think is the key here.
The output looks a bit different now, since we're adding a found testsuite at a time to our main testsuite suite:
python -m unittest discover -p tests
Found Tests: [<test_tc.TestCase testMethod=testBar>, <test_tc.TestCase testMethod=testFoo>]
Found Tests: [<test_employee.TestCase testMethod=testBar>, <test_employee.TestCase testMethod=testFoo>]
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s
OK
But we still get the 4 tests we expected, in both classes, in both subdirectories.
The point using init.py, is that one may encounters side effects, like file not being the script file path. Using FOR DOS command can help (not found of DOS commands, but sometimes it helps
setlocal
set CWD=%CD%
FOR /R %%T in (*_test.py) do (
CD %%~pT
python %%T
)
CD %CWD%
endlocal
/R allows for walkthrough the hierarchy from current folder.
(expr) allows for selecting test files (I use _test.py)
%%~pT is $(dirname $T) in shell.
I saved and restore my original directory, as the .bat leaves me where it ends
setlocal ... endlocal to not pollute my environment with CWD.
Related
I have the following folder structure in my project:
my-project
src/
__init__.py
script.py
test/
__init__.py
test_script.py
Ideally I want to have a separate folder where all the unit tests go. My test_script.py looks something like this:
from src.script import my_object
class TestClass(unittest.TestCase):
def test_script_object(self):
# unit test here
pass
if __name__ == 'main':
unittest.main()
When I try to run the script (using python test_script.py) I get the following error:
Traceback (most recent call last):
File "test_script.py", line 4, in <module>
from src.script import my_object
ModuleNotFoundError: No module named 'src'
I was following the instructions from this other thread, and I even tried appending src to the sys path (which forces me to change how I do imports in the rest of the project). When I'm not trying to append to the sys path, both of my __init__.py files are empty.
I am using python 3.8.
Does anyone have any suggestions? I'm new to unit testing in python, so maybe there is a better structure or other conventions I'm not aware of. Thanks in advance for your help!
Generally, any instructions that have you modifying sys.path in order to run your tests are sending you in the wrong direction. Your testing tool should be able to discover both your tests and your application code without requiring that sort of hackery.
I generally use pytest for running my tests. I would structure your example like this:
my-project/
├── src
│ ├── conftest.py
│ └── myproject
│ ├── __init__.py
│ └── script.py
└── tests
├── __init__.py
└── test_script.py
Assuming that src/myproject/script.py looks like this:
def double(x: int):
return x*2
And tests/test_script.py look like this:
import myproject.script
def test_double():
res = myproject.script.double(2)
assert res == 4
I can run tests from the my-project directory by simply running pytest:
$ pytest
========================================== test session starts ==========================================
platform linux -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/lars/tmp/python/my-project
collected 1 item
tests/test_script.py . [100%]
=========================================== 1 passed in 0.00s ===========================================
In this setup, the file src/conftest.py is what allows pytest to automatically discover the location of your application code. Alternatively, you could instead specify that in your pyproject.toml file like this:
[tool.pytest.ini_options]
pythonpath = [
"src"
]
pytest also works if you write unittest-style tests as you show in your question; the above behavior would be identical if tests/test_script.py looked like:
import unittest
import myproject.script
class TestScript(unittest.TestCase):
def test_double(self):
res = myproject.script.double(2)
self.assertEqual(res, 4)
if __name__ == '__main__':
unittest.main()
(But I prefer pytest's simpler syntax.)
Possibly useful reading:
Good integration practices
If you really want to use unittest you can use the same directory layout, but you will need to update sys.path for it to run correctly. The easiest way to do that is:
PYTHONPATH=$PWD/src python -m unittest
This will automatically discover all your tests and run them. You can run a single test like this:
PYTHONPATH=$PWD/src python -m tests.test_script
You can avoid needing to modify sys.path if you use a simpler directory layout:
my-project/
├── myproject
│ ├── __init__.py
│ └── script.py
└── tests
├── __init__.py
└── test_script.py
Both pytest and python -m unittest, when run from the my-project directory, will correctly discover and run your tests without requiring any path modifications.
I'm creating a pytest unit test for a function in my software.
Before even starting to test, pyunit seems to be unable to import my "cache_offline" decorator which is indirectly imported when I import the function I'm testing in my test.
I'm using Anaconda embedding Python 3.7 and pytest 5.2.2
I tried to comment out the code where the decorator is applied to my functions, when I do so the pytest error disappear and the tests execute properly.
My test is in ./tests/scripts/test_scripts_helper.py and I run pytest at the project root .
Pytest finds properly my test (see the error message), so this is not the problem at hand here.
My test imports and wants to test the function read_tiff_tag from a package vorace.scripts_helper, which imports a function safe_mkdir from package vorace.core.misc, which imports the package vorace.core.vorace, in which 3 functions are decorated with the decorator cache_offline from package vorace.core.misc
I tried both running the tests using either py.test or python -m pytest at the root of my project.
My project have the following structure (simplified).
The code root is ./vorace
The tests root is ./tests
.
├── conftest.py
├── tests
│ ├── __init__.py
│ ├── scripts
│ │ ├── __init__.py
│ │ └── test_scripts_helper.py
│ └── tests_data
│ └── test_ROI.tif
└── vorace
├── __init__.py
├── core
│ ├── __init__.py
│ ├── misc.py
│ └── vorace.py
└── scripts
├── __init__.py
├── batch_analyzis.py
└── scripts_helper.py
I tried :
with and without empty __init__.py in each subfolder of the tests folder. -> no change
with and without an empty conftest.py at the root of the project. -> no change
executing a test which doesn't need any import in my test_scripts_helper.py file (with my test causing the problem being commented out) -> the test executes properly
I suspect kind of a circular import problem but I've always been told that it can't happen in python. Maybe the decorators are an exception to this rule ?
My vorace.core.misc code, with the decorator
from vorace.core import vorace
[...]
def cache_offline(cache_path=os.getcwd()):
[...]
def decorator(func):
[...]
def wrapper(*args, **kwargs):
[...]
return result
return wrapper
return decorator
def safe_mkdir(path):
[...]
One of the decorated functions in vorace.core.vorace
from vorace.core.misc import *
[...]
#cache_offline(cache_path=".cache")
def classify_clusters_by_connectivity(xyz_data):
[...]
[...]
The output from executing py.test in the project root
==================== test session starts ====================
platform linux -- Python 3.7.3, pytest-5.2.2, py-1.8.0, pluggy-0.12.0
rootdir: /home/flo/PycharmProjects/VorAce
plugins: arraydiff-0.3, openfiles-0.3.2, doctestplus-0.3.0, remotedata-0.3.1
collected 0 items / 1 errors
==================== ERRORS ====================
_________ ERROR collecting tests/scripts/test_scripts_helper.py _________
tests/scripts/test_scripts_helper.py:1: in <module>
import vorace.scripts.scripts_helper as sh
vorace/scripts/scripts_helper.py:6: in <module>
from vorace.core.misc import safe_mkdir
vorace/core/misc.py:8: in <module>
from vorace.core import vorace
vorace/core/vorace.py:91: in <module>
#cache_offline(cache_path=".cache")
E NameError: name 'cache_offline' is not defined
If I execute a simple 0 == 0 test in my tests/scripts/test_scripts_helper.py file without importing from my project, the test runs with success.
When you run ./tests/scripts/test_scripts_helper.py python automatically sets ./tests/scripts/ into PYTHONPATH but not sets any other directories, so all your imports from other custom files should fail.
Set all your working directories into PYTHONPATH environment variable.
Something like that on Linux shell.
PYTHONPATH="${PYTHONPATH}:$dir
EDIT: I finally got the confirmation of a circular import.
At the opposite of what I thought, importing particular names from a module
like in from x import y can be sensitive to circular imports, where import x can't.
To solve it, I just imported the module and use the syntax using the module prefixing the function call.
More information here : https://www.reddit.com/r/Python/comments/51hdup/from_import_vs_import_on_circular_import/
The problem what finally not related specifically to the decorator or pytest.
I worked around this issue by putting my cache_offline decorator in a separated package vorace.core.caching.py. Now I import this package only from vorace.core.vorace where the function needing to be decorated resides.
This way my decorator is artificially excluded from the code imported by my unit test, but still available to the rest of my code for a normal application execution.
However pytest still have an issue here, it should not fail to import. I'm still taking any answer that can explain why pytest fails to import my decorator, and I keep the resolve for an answer of this kind.
We using py.test. We try to put different conftest.py files in different folders to split our fixtures:
tests/api/
├── conftest.py
├── folder1
│ └── conftest.py
├── folder2
│ └── conftest.py
But when run the tests this error occurs:
____ ERROR collecting api/folder1/conftest.py ____
import file mismatch:
imported module 'conftest' has this __file__ attribute:
/tests/api/folder2/conftest.py
which is not the same as the test file we want to collect:
/tests/api/folder1/conftest.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
Why is that? How fix it?
PS. Removing __pycache__.pyc did not help.
PPS. __init__.py files already exist in each folder.
I had the same issue. To solve this you need to create python packages instead of directories. Then pytest will look at the conftest.py in your package instead of root directory. Hope, this will help you.
tests/api/
├── conftest.py
├── package1 # not folder
│ └── conftest.py
├── package2 # not folder
│ └── conftest.py
Your use case sounds like this example in the pytest documentation. Because of that I think it's possible to use conftest.pys at different levels to override fixtures.
The errors you're seeing may be related to incorrect imports. Is your test code importing from conftest files directly? Are your conftest files importing from your tests? Are any of your imports relative instead of absolute? If any of these are true, that may be your issue. I recommend only using absolute imports, and avoid imports between conftest.pys and test files.
Rename one (or both) of the test files Pytest is complaining about. Pytest is telling you in the error message to do this (i.e. change the basename, meaning don't name all your test files conftest.py). For example, you can fix it by doing:
tests/api/
├── conftest.py
├── folder1
│ └── test_conf1.py
├── folder2
│ └── test_conf2.py
In your case, the module names conflict (you have three conftest.pys). This is a quirk of Pytest AFAIK. Pytest could get around this by managing full package/module paths: but it doesn't do this (probably for good reason, but I do not maintain/contribute to pytest so I can't shed light on the issue). Pytest is a fantastic framework (it's even telling you exactly why it can't run your tests): I'm sure they have a good reason for not supporting this behavior.
You claim that you want to:
separate tests and fixtures by different functionalities.
So do that. Separating the test fixtures/functionalities has nothing to do with what you name the files.
I commonly run into this error when splitting up unit/integration/acceptance tests. I split them up so I can run my (fast) unit tests without having to run my (potentially slow) integration/acceptance tests. I might have some module, call it Abc. And I have something like:
tests/
├── unit
│ └── test_abc.py
├── integration
│ └── test_abc.py
But then pytest barfs with the identical error you've shown, and so I just rename integration/test_abc.py to integration/test_abc_integration.py and move on with my day. Like this:
tests/
├── unit
│ └── test_abc.py
├── integration
│ └── test_abc_integration.py
Is it annoying? A little. How long does the fix take? 5 whole seconds.
P.S. You might have to remove __pycache__ directories or you .pyc files for the first run after you get the error you've posted about (if you don't you'll just get the same error again even if you rename).
P.S.S. You can stop the Cpython interpreter (and most others) from writing out __pycache__ and .pyc files by calling python -B -m pytest .... The -B option makes the interpreter not save the bytecode to your filesystem. This results in some performance penalty whenever you run your test suite, but the penalty is usually very small (milage may vary). I typically use this option because I don't like the clutter in my repositories and the performance loss is typically negligible.
TL;DR
Here's an example repository that is set up as described in the first diagram (below): https://github.com/Poddster/package_problems
If you could please make it look like the second diagram in terms of project organisation and can still run the following commands, then you've answered the question:
$ git clone https://github.com/Poddster/package_problems.git
$ cd package_problems
<do your magic here>
$ nosetests
$ ./my_tool/my_tool.py
$ ./my_tool/t.py
$ ./my_tool/d.py
(or for the above commands, $ cd ./my_tool/ && ./my_tool.py is also acceptable)
Alternatively: Give me a different project structure that allows me to group together related files ('package'), run all of the files individually, import the files into other files in the same package, and import the packages/files into other package's files.
Current situation
I have a bunch of python files. Most of them are useful when callable from the command line i.e. they all use argparse and if __name__ == "__main__" to do useful things.
Currently I have this directory structure, and everything is working fine:
.
├── config.txt
├── docs/
│ ├── ...
├── my_tool.py
├── a.py
├── b.py
├── c.py
├── d.py
├── e.py
├── README.md
├── tests
│ ├── __init__.py
│ ├── a.py
│ ├── b.py
│ ├── c.py
│ ├── d.py
│ └── e.py
└── resources
├── ...
Some of the scripts import things from other scripts to do their work. But no script is merely a library, they are all invokable. e.g. I could invoke ./my_tool.py, ./a.by, ./b.py, ./c.py etc and they would do useful things for the user.
"my_tool.py" is the main script that leverages all of the other scripts.
What I want to happen
However I want to change the way the project is organised. The project itself represents an entire program useable by the user, and will be distributed as such, but I know that parts of it will be useful in different projects later so I want to try and encapsulate the current files into a package. In the immediate future I will also add other packages to this same project.
To facilitate this I've decided to re-organise the project to something like the following:
.
├── config.txt
├── docs/
│ ├── ...
├── my_tool
│ ├── __init__.py
│ ├── my_tool.py
│ ├── a.py
│ ├── b.py
│ ├── c.py
│ ├── d.py
│ ├── e.py
│ └── tests
│ ├── __init__.py
│ ├── a.py
│ ├── b.py
│ ├── c.py
│ ├── d.py
│ └── e.py
├── package2
│ ├── __init__.py
│ ├── my_second_package.py
| ├── ...
├── README.md
└── resources
├── ...
However, I can't figure out an project organisation that satisfies the following criteria:
All of the scripts are invokable on the command line (either as my_tool\a.py or cd my_tool && a.py)
The tests actually run :)
Files in package2 can do import my_tool
The main problem is with the import statements used by the packages and the tests.
Currently, all of the packages, including the tests, simply do import <module> and it's resolved correctly. But when jiggering things around it doesn't work.
Note that supporting py2.7 is a requirement so all of the files have from __future__ import absolute_import, ... at the top.
What I've tried, and the disastrous results
1
If I move the files around as shown above, but leave all of the import statements as they currently are:
$ ./my_tool/*.py works and they all run properly
$ nosetests run from the top directory doesn't work. The tests fail to import the packages scripts.
pycharm highlights import statements in red when editing those files :(
2
If I then change the test scripts to do:
from my_tool import x
$ ./my_tool/*.py still works and they all run properly
$ nosetests run from the top directory doesn't work. Then tests can import the correct scripts, but the imports in the scripts themselves fail when the test scripts import them.
pycharm highlights import statements in red in the main scripts still :(
3
If I keep the same structure and change everything to be from my_tool import then:
$ ./my_tool/*.py results in ImportErrors
$ nosetests runs everything ok.
pycharm doesn't complain about anything
e.g. of 1.:
Traceback (most recent call last):
File "./my_tool/a.py", line 34, in <module>
from my_tool import b
ImportError: cannot import name b
4
I also tried from . import x but that just ends up with ValueError: Attempted relative import in non-package for the direct running of scripts.
Looking at some other SO answers:
I can't just use python -m pkg.tests.core_test as
a) I don't have main.py. I guess I could have one?
b) I want to be able to run all of the scripts, not just main?
I've tried:
if __name__ == '__main__' and __package__ is None:
from os import sys, path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
but it didn't help.
I also tried:
__package__ = "my_tool"
from . import b
But received:
SystemError: Parent module 'loading_tool' not loaded, cannot perform relative import
adding import my_tool before from . import b just ends up back with ImportError: cannot import name b
Fix?
What's the correct set of magical incantations and directory layout to make all of this work?
Once you move to your desired configuration, the absolute imports you are using to load the modules that are specific to my_tool no longer work.
You need three modifications after you create the my_tool subdirectory and move the files into it:
Create my_tool/__init__.py. (You seem to already do this but I wanted to mention it for completeness.)
In the files directly under in my_tool: change the import statements to load the modules from the current package. So in my_tool.py change:
import c
import d
import k
import s
to:
from . import c
from . import d
from . import k
from . import s
You need to make a similar change to all your other files. (You mention having tried setting __package__ and then doing a relative import but setting __package__ is not needed.)
In the files located in my_tool/tests: change the import statements that import the code you want to test to relative imports that load from one package up in the hierarchy. So in test_my_tool.py change:
import my_tool
to:
from .. import my_tool
Similarly for all the other test files.
With the modifications above, I can run modules directly:
$ python -m my_tool.my_tool
C!
D!
F!
V!
K!
T!
S!
my_tool!
my_tool main!
|main tool!||detected||tar edit!||installed||keys||LOL||ssl connect||parse ASN.1||config|
$ python -m my_tool.k
F!
V!
K!
K main!
|keys||LOL||ssl connect||parse ASN.1|
and I can run tests:
$ nosetests
........
----------------------------------------------------------------------
Ran 8 tests in 0.006s
OK
Note that I can run the above both with Python 2.7 and Python 3.
Rather than make the various modules under my_tool be directly executable, I suggest using a proper setup.py file to declare entry points and let setup.py create these entry points when the package is installed. Since you intend to distribute this code, you should use a setup.py to formally package it anyway.
Modify the modules that can be invoked from the command line so that, taking my_tool/my_tool.py as example, instead of this:
if __name__ == "__main__":
print("my_tool main!")
print(do_something())
You have:
def main():
print("my_tool main!")
print(do_something())
if __name__ == "__main__":
main()
Create a setup.py file that contains the proper entry_points. For instance:
from setuptools import setup, find_packages
setup(
name="my_tool",
version="0.1.0",
packages=find_packages(),
entry_points={
'console_scripts': [
'my_tool = my_tool.my_tool:main'
],
},
author="",
author_email="",
description="Does stuff.",
license="MIT",
keywords=[],
url="",
classifiers=[
],
)
The file above instructs setup.py to create a script named my_tool that will invoke the main method in the module my_tool.my_tool. On my system, once the package is installed, there is a script located at /usr/local/bin/my_tool that invokes the main method in my_tool.my_tool. It produces the same output as running python -m my_tool.my_tool, which I've shown above.
Point 1
I believe it's working, so I don't comment on it.
Point 2
I always used tests at the same level as my_tool, not below it, but they should work if you do this at the top of each tests files (before importing my_tool or any other py file in the same directory)
import os
import sys
sys.path.insert(0, os.path.abspath(__file__).rsplit(os.sep, 2)[0])
Point 3
In my_second_package.py do this at the top (before importing my_tool)
import os
import sys
sys.path.insert(0,
os.path.abspath(__file__).rsplit(os.sep, 2)[0] + os.sep
+ 'my_tool')
Best regards,
JM
To run it from both command line and act like library while allowing nosetest to operate in a standard manner, I believe you will have to do a double up approach on Imports.
For example, the Python files will require:
try:
import f
except ImportError:
import tools.f as f
I went through and made a PR off the github you linked with all test cases working.
https://github.com/Poddster/package_problems/pull/1
Edit: Forgot the imports in __init__.py to be properly usable in other packages, added. Now should be able to do:
import tools
tools.c.do_something()
When playing with nose and trying to combine it with cython I can't quite get it all to work the way I'd like. The code is organised like this:
.
├── setup.py
└── src
├── calc
│ ├── factorial.py
│ ├── __init__.py
│ └── tests.py
└── cycalc
├── tests.py
└── triangle.pyx
Each of the tests.py contains 2 tests, one succeeds, one fails. The result of running setup.py nosetests is that only calc/tests.py are run. If I after this run nosetests3 src/cycalc the two tests in cycalc/tests.py are run. However, if I clean up all build files it fails because cycalc/triangle.pyx hasn't been built into a shared lib.
Then I tried adding the file src/cycalc/__init__.py, now setup.py nosetests picks up cycalc/tests.py but it fails to find the required module, it was placed in src.
How do I arrange my cython source and tests to make setup.py nosetests find everything it needs?
For nose to run your tests automatically you should add them into a folder called tests containing all your tests. Like this:
.
|-setup.py
|-src
|---calc
|------factorial.py
|------__init__.py
|---cycalc
|------triangle.pyx
|------__init__.py
|-tests
|---__init__.py
|---test_calc.py
|---test_cycalc.py
This way both tests will be run automatically with everything in the same path. If you remove the built files you need to run python setup.py build before the tests will work again.