The problem:
pytest (decided by policy) takes 1536 seconds to run the same test suite (585 tests) as nosetest, which runs in 71 seconds.
The pytest.ini files is:
[pytest]
python_files = tests_*.py *_tests.py
norecursedirs = .idea (pycharm env).
testpaths = tests
And the file is placed at the root of the project:
root
|-+ mod1
| |-- core.py
| |-- utils.py
|-+ mod2
| |-- core.py
| |-- utils2.py
|-+ tests
| |-- test_mod1
| |-- test_mod2
|-+ utils (don't test).
| |-- u1.py
| |-- u2.py
|- pytest.ini
|- readme.md
Things I've checked (following advice from the 14 other SO posts):
The number of Pass/Fails is the same.
When running the tests individually with pytests they take ~ 20ms.
When running the folder with pytests 10-20 tests take 14-15 seconds.
The test suite has one environment, there's no env or os magic. Just lots of technical logic.
Each test_xyz.py file has it's own isolated def setup and def teardown that creates/drop an sqlite database. The tests interact with the database, by adding new transactions and checking the additions. Example:
global db
def setup():
db = get_new_db()
def teardown():
pass
def test_01():
w = Widget(db) # create widget instance.
w.add_friend('a#b.com')
assert 'a#b.com' in w.friends()
Questions:
Do I really have to plaster #pytest.fixtures(scope='module') on the setup and teardown of every 585 tests? I hope not.
What can I do to get the runtime of pytest to be similar to nosetests?
I'm not sure why pytest chose to invoke the module setup function in a pytest_runtest_setup hook that runs once per each test instead of a module-scoped autouse fixture, but here it is:
#hookimpl(trylast=True)
def pytest_runtest_setup(item):
if is_potential_nosetest(item):
if not call_optional(item.obj, "setup"):
# call module level setup if there is no object level one
call_optional(item.parent.obj, "setup")
# XXX this implies we only call teardown when setup worked
item.session._setupstate.addfinalizer((lambda: teardown_nose(item)), item)
This means you'll need to rename the setup/teardown functions to setup_module()/teardown_module(). If you're on Linux/MacOS, you can use sed combined with grep for batch renaming:
$ grep -lr "\(def setup():\|def teardown():\)" | \
xargs sed -i 's/def setup():/def setup_module():/g;s/def teardown():/def teardown_module():/g'
Related
I have a problem in providing the correct path to mocker patcher. I have looked at couple stack threads but to no avail.
Project structure:
my_app
| src
| a
| mod.py
| tests
| test_mod.py
| b
| mod.py
| tests
| test_mod.py
Let's say that file my_app/src/a/mod.py contains:
from abc import x
def foo():
result = x.y()
return result
In test file I import mod.py via relative path. I also want to patch abc.x.y() function (imported in mod.py).
import pytest
from .. import mod
from ..mod import foo
def test_foo(mocker):
mocker.patch("mod.x.y", return_value=None)
assert foo() is None
The problem is that pytest does not find the "mod" module.
I have also tried using:
mocker.patch.object(mod, "x.y", return_value=None)
and while "mod" is found, "x.y" is not part of "mod".
How can I patch that function while staying contained inside "a" package (I do not want to provide a path through the whole project)?
I need to delete some folders created for testing.
In the main test folder, I created the a file test_teardown.py with the following content.
import shutil
import pytest
#pytest.fixture(scope="session")
def teardown():
yield
shutil.rmtree('tmp')
Yet, the tmp folder is not deleted after the testing session is completed. Am I using the fixture incorrectly?
File structure
+-- Project folder
| +-- tests
| | +-- __init__.py
| | +-- test_teardown.py
| | +-- Unit
| | | +-- __init__.py
| | | +-- test_moretests.py
Thanks to #MrBean Bremen
Adding autouse=True to the wrapper ensures that it's automatically involved.
Do make sure that your test_teardown.py is included in the used test folder.
Wrapper:
#pytest.fixture(scope='session', autouse=True)
def teardown():
yield
shutil.rmtree('tmp')
I have this little project where I use pytest and pytest-dependency with tox to develop integration tests on some code. Until now I used one base class (BTestClass) with some common tests in the root directory and the specific tests for each code component in a test_Component.py file next to it implementing a TestC class that inherits from BTestClass.
Everything worked fine until then. Now I want to add a BTestClass2 for another set of components. So I added another layer of inheritance, but now it doesn't work, pytest validates the common A tests but then skips the tests that depend on it. I have no idea why.
Here's the filesystem layout:
λ tree /F
Folder PATH listing
Volume serial number is F029-7357
C:.
│ B.py
│ requirements-tox.txt
│ tox.ini
│
├───app_C
│ └───tests
│ test_C.py
│
└───common
A.py
common\A.py
import pytest
class ATestClass():
#pytest.mark.dependency(name='test_a')
def test_a(self):
assert True
B.py
import pytest
from common.A import ATestClass
class BTestClass(ATestClass):
#pytest.mark.dependency(name='test_b', depends=['test_a'])
def test_b(self):
assert True
test_C.py
import pytest
import sys
sys.path.append('.')
from B import *
class TestC(BTestClass):
#pytest.mark.dependency(name='test_c', depends=['test_b'])
def test_c(self):
assert True
pytest output:
λ tox -- -rs
py38 installed: ...
py38 run-test-pre: PYTHONHASHSEED='367'
py38 run-test: commands[0] | pytest -x -v -rs
=============================================== test session starts ===============================================
platform win32 -- Python 3.8.1, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- ...\poc\.tox\py38\scripts\python.exe
cachedir: .tox\py38\.pytest_cache
rootdir: ...\poc
plugins: dependency-0.5.1
collected 3 items
app_C/tests/test_C.py::TestC::test_b SKIPPED [ 33%]
app_C/tests/test_C.py::TestC::test_c SKIPPED [ 66%]
app_C/tests/test_C.py::TestC::test_a PASSED [100%]
============================================= short test summary info =============================================
SKIPPED [1] .tox\py38\lib\site-packages\pytest_dependency.py:103: test_b depends on test_a
SKIPPED [1] .tox\py38\lib\site-packages\pytest_dependency.py:103: test_c depends on test_b
===================================== 1 passed, 2 skipped, 1 warning in 0.14s =====================================
_____________________________________________________ summary _____________________________________________________
py38: commands succeeded
congratulations :)
Any idea why test_b is skipped and not executed?
Edit: If I make BTestClass standalone, removing A / ATestClass from the picture, it works fine.
collected 2 items
app_C/tests/test_C.py::TestC::test_b PASSED [ 50%]
app_C/tests/test_C.py::TestC::test_c PASSED [100%]
In pytest-dependency, a dependency on another test implies that that test runs before the dependent test. If that is not the case (in your example test_b is run before test_a, because test_a is located in a subdirectory), the test is just skipped. pytest-dependency doesn't do any reordering of tests (unfortunately).
If you cannot easily establish the order in which tests are run via naming, you may use the pytest-ordering plugin to bring the tests into the needed order. In your case you could do:
class ATestClass:
#pytest.mark.dependency(name='test_a')
#pytest.mark.run(order=0)
def test_a(self):
assert True
...
class BTestClass(ATestClass):
#pytest.mark.dependency(name='test_b', depends=['test_a'])
#pytest.mark.run(order=1)
def test_b(self):
assert True
In this case, the tests are run in the order test_a - test_b - test_c, and all tests will run.
UPDATE:
You can also use pytest-order, which is a fork of pytest-ordering. If you use the pytest option --order-dependencies it will try to re-order the tests with dependencies created by pytest-dependencies, without the need to add extra marks.
Disclaimer: I'm the author of that fork.
****solved: added __init__.py to Test/ and renamed testcode.py to test_code.py. To run tests cd -> Zee and type pytest****
Structure:
|--Zee/
| |--Test/
| | |--__init__.py
| | |--test_code.py
| |--Codetotest/
| | |--code.py
in code.py
class Foo():
some code...
in testcode.py
from Codetotest.code import Foo
def test_foo():
assert ...
When I move to the Zee directory in my command line and run pytest Test/testcode.py I get ModuleNotFoundError: No module named Zee. How can I fix this?
I tried making Test a module by adding Test/__init__.py as suggested here. Ran from multiple directories, no dice.
Pytest version 5.3.4, imported from python 3.6
What I don't understand is, when I add __init__.py to Zee/, it gives me the same error
You need a __init__.py in the module directory.
Here's a typical project structure:
|--zee-project-directory/
| |--tests/
| | |--test_zee.py
| |--zee/
| | |--__init__.py
| | |--code.py
code.py
class Foo():
some code...
test_zee.py
from zee.code import Foo
def test_foo():
assert ...
I am at my wits end with trying to get all my unittest to run in Python. I have searched about 30 different posts and the unit test documentation but still cannot figure it out.
First I have two test classes that I can run each individually and all the tests pass:
File: unittest.subfolder1.TestObject1.py
class TestObject1(unittest.TestCase):
def test_case1(self):
...some code...
...some assertions...
if __name__ == '__main__':
unittest.main()
File: unittest.subfolder2.TestObject2.py
class TestObject2(unittest.TestCase):
def test_case1(self):
...some code...
...some assertions...
if __name__ == '__main__':
unittest.main()
Starting in the top level directory above 'unittest' I am trying to us unittest.discover to find and run all my tests:
import unittest
loader = unittest.TestLoader()
suite = loader.discover('unittest')
unittest.TextTestRunner().run(suite)
When I do this I get the error `ModuleNotFoundError: No module named 'subfolder1.TestObject1'
What am I doing wrong?
A good approach is to run all the tests in a subdirectory from the command line. In order to find the following files "TestObject1.py, TestObject2.py, ..." in subdirectories, you can run the following command in the command line:
python -m unittest discover -p 'Test*.py'
Additionally, the __init__.py is required within the import and module directories: Python unittest discovery with subfolders
The import unittest is required in the files unittest.subfolder1.TestObject1.py and unittest.subfolder2.TestObject2.py
It is also possible to define explicitly the directory where the discovery starts with the -s parameter:
python -m unittest discover [options]
-s directory Directory to start discovery ('.' default)
-p pattern Pattern to match test files ('test*.py' default)
In case you are using unittest2, it comes with a script unit2. The command line usage is:
unit2 discover unit2 -v test_module
Do not name your directory unittest, it may conflict with the standard library.
You also need to create a file named __init__.py in all of your directories (subfolder1, etc.), so they become packages and their content can be imported.
So I had to do my own workaround but at least I can get them all to run with the above file structure. It requires that I reinstantiate the TestLoader and the TestSuite each time I give it a new file path, so first I need to collect all relevant file paths in the unittest directory.
import os
import unittest
import traceback
class UnitTestLauncher(object):
def runTests(self):
#logging.INFO("Running unit tests...")
lsPaths = []
#Find all relevant subdirectories that contain unit tests
#Exclude 'unittest' directory, but include subdirectories, with code `path != 'unittest'`
for path,subdirs,files in os.walk('unittest'):
if "pycache" not in path and path != 'unittest':
lsPaths.append(path)
#loop through subdirectories and run individually
for path in lsPaths:
loader = unittest.TestLoader()
suite = unittest.TestSuite()
suite = loader.discover(path)
unittest.TextTestRunner().run(suite)
This solution is not perfect and each different directory comes out as a line of output so you have to look through each line manually for failed tests.
Old question but, oh, so current. I am new to Python, coming from strong typed languages and while the language itself is ok(ish), the conventions, tools and workarounds to make everything work in the ecosystem can drive you nuts. I struggled myself with running unit tests from separate subdirectories and this is the way I solved it.
First, the code you test, package it into a package. Organize your directories like this:
Work
|
+---PkToTest
| |
| +--- __init__.py
| +--- a.py
| +--- <other modules>.py
|
+---Tests (for PKToTest)
|
+--- test_a.py
PkToTest becomes a package due to the init.py file. In test_a.py make sure your sys.path will contain the path to PkToTest (absolute path not relative). I did that by:
import sys
sys.path.insert(0, "<absolute path to parent of PkTotest directory>")
import unittest
from PkToTest import a
class aTestSuite(unittest.TestCase):
def test1(self):
self.assertEqual(a.fnToTest(), ...)
Testing All Subdirectories
Given a structure of:
my_package
|
|
controller
|-- validator.py
|
validator
|-- controller.py
|
test
|-- controller
|
|-- __init__.py (MAKE SURE THIS EXISTS OR unittest MODULE WOULD NOT KNOW)
|-- test_controller.py
|
|-- validator
|
|-- __init__.py (MAKE SURE THIS EXISTS OR unittest MODULE WOULD NOT KNOW)
|-- test_validator.py
|
then just run
python -m unittest discover -s my_package/test
What this does is to test and -s means to start with the my_package/test as the starting directory
In my project all folders are folders (not modules) and they have the structure:
Folder > Subfolder > Subfolder > Tests > test_xxxx.py
Folder > Subfolder > Subfolder > xxxx.py
So i modified the answer from here, and also took a part from How do I run all Python unit tests in a directory? and came up with this:
import os, unittest
testFolderPaths = []
for path, subdirs, files in os.walk(os.getcwd()):
for file in files:
if file.startswith("test_") and file.endswith(".py"):
testFolderPaths.append(path)
for path in testFolderPaths:
print(f"Running tests from {path}...")
loader = unittest.TestLoader()
suite = loader.discover(path)
runner = unittest.TextTestRunner()
result = runner.run(suite)
print(f"RUN {result.testsRun} Tests. PASSED? {result.wasSuccessful()}")
If any tests fail it will throw and error showing which one exactly failed.