I would like to perform some sanity check when I run tests using pytest. Typically, I want to check that some executables are accessible to the tests, and that the options provided by the user on the command-line are valid.
The closest thing I found was to use a fixture such as:
#pytest.fixture(scope="session", autouse=True)
def sanity_check(request):
if not good:
sys.exit(0)
But this still runs all the tests. I'd like for the script to fail before attempting to run the tests.
You shouldn't need to validate the command line options explicitly; this will be done by the arg parser which will abort the execution early if necessary. As for conditions checking, you are not far from the solution. Use
pytest.exit to to an immediate abort
pytest.skip to skip all tests
pytest.xfail to fail all tests (this is an expected failure though, so it won't mark the whole execution as failed)
Example fixture:
#pytest.fixture(scope='session', autouse=True)
def precondition():
if not shutil.which('spam'):
# immediate shutdown
pytest.exit('Install spam before running this test suite.')
# or skip each test
# pytest.skip('Install spam before running this test suite.')
# or make it an expected failure
# pytest.xfail('Install spam before running this test suite.')
xdist compatibility
Invoking pytest.exit() in the test run with xdist will only crash the current worker and will not abort the main process. You have to move the check to a hook that is invoked before the runtestloop starts (so anything before the pytest_runtestloop hook), for example:
# conftest.py
def pytest_sessionstart(session):
if not shutil.which('spam'):
# immediate shutdown
pytest.exit('Install spam before running this test suite.')
If you want to run sanity check before whole test scenario then you can use conftest.py file - https://docs.pytest.org/en/2.7.3/plugins.html?highlight=re
Just add your function with the same scope and autouse option to conftest.py:
#pytest.fixture(scope="session", autouse=True)
def sanity_check(request):
if not good:
pytest.exit("Error message here")
Related
I would like to run a function at the end of all the tests.
A kind of global teardown function.
I found an example here and some clues here but it doesn't match my need. It runs the function at the beginning of the tests. I also saw the function pytest_runtest_teardown(), but it is called after each test.
Plus: if the function could be called only if all the tests passed, it would be great.
I found:
def pytest_sessionfinish(session, exitstatus):
""" whole test run finishes. """
exitstatus can be used to define which action to run. pytest docs about this
To run a function at the end of all the tests, use a pytest fixture with a "session" scope. Here is an example:
#pytest.fixture(scope="session", autouse=True)
def cleanup(request):
"""Cleanup a testing directory once we are finished."""
def remove_test_dir():
shutil.rmtree(TESTING_DIR)
request.addfinalizer(remove_test_dir)
The #pytest.fixture(scope="session", autouse=True) bit adds a pytest fixture which will run once every test session (which gets run every time you use pytest). The autouse=True tells pytest to run this fixture automatically (without being called anywhere else).
Within the cleanup function, we define the remove_test_dir and use the request.addfinalizer(remove_test_dir) line to tell pytest to run the remove_test_dir function once it is done (because we set the scope to "session", this will run once the entire testing session is done).
You can use the atexit module. For instance, if you want to report something at the end of all the test you need to add a report funtion as follows:
def report(report_dict=report_dict):
print("THIS IS AFTER TEST...")
for k, v in report_dict.items():
print(f"item for report: {k, v}")
then at the end of the module, you call atexit as follows:
atexit.register(report)
hoop this helps!
I have unit tests that runs with py.test for python 2.7 and py.test 3.0. My test directory is like this:
tests
---dir1
test1.py
-------sub-dir1-1
test-1-1.py
-------sub-dir1-2
test-1-2.py
---dir2
test2.py
-------sub-dir2-1
test-2-1.py
-------sub-dir2-2
test-2-2.py
I want all my tests to run a common setup and tear down function before and after test. I would like to do it with the least modification of all the test code.
Thanks
If I understand correctly, you can write a fixture that is session scoped (scope param) and make it used automatically (autouse param). I used a "yield fixture" as an example. Please note, that pytest.yield_fixture is deprecated since pytest 3.0 and pytest.fixture allows use of yield.
import pytest
#pytest.fixture(scope="session", autouse=True)
def callattr_ahead_of_alltests(request):
print 'run_pre_start'
yield
print 'run_after_finish'
It will run before first test (printing "run_pre_start") and part after yield will run after all the tests (printing "run_after_finish").
I've installed new pytest plugin (pytest-catchlog==1.2.2) and as much as I like it, it breaks my unit tests for logging module (e.g ValueError: I/O operation on closed file).
I would like to disable that plugin for test_logging.py file (or even a class or method), but I can't find any information on it.
The only option I found so far is to execute pytest twice: first time for test_logging.py onlywith catchlog disabled (py.test -p no:catchlog test_logging.py), and second time for all other test files.
Please let me know if I missed a pytest decorator, or any other way of disabling plugins in runtime.
You cannot selectively disable arbitrary plugins for selected tests. The plugins are loaded at a much earlier stage — when the pytest starts. And the plugins actually define what pytest does and how (i.e., command line options, test collection, filtering, etc).
In other words, it is too late to redefine the pytest's internal structure when it gets to the test execution.
Your best case is, indeed, to mark your tests with #pytest.mark.nocatchlog, and execute them separately:
pytest -m 'nocatchlog' -p no:catchlog # problematic tests with no plugin
pytest -m 'not nocatchlog` # all other tests
If those tests not under your control, i.e. if you cannot add marks, then you can only filter by expressions like -k test_logging or -k 'not test_logging' (i.e. by part of their node id).
Specifically for this pytest-catchlog plugin, you can make the same hooks as it does, and remove its log handler from the root logger (assuming that no other loggers were used explicitly):
conftest.py:
import pytest
def _disable_catchlog(item):
logger = logging.getLogger()
if item.catch_log_handler in logger.handlers:
logger.handlers.remove(item.catch_log_handler)
#pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_setup(item):
_disable_catchlog(item)
yield
#pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_call(item):
_disable_catchlog(item)
yield
#pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_teardown(item):
_disable_catchlog(item)
yield
So I have a directory with sub-directory of Acceptance Tests.
Most of my tests have no dependancies on each other, expect for one suite.
Is there a way I can tell nose when it reaches this class to execute the tests sequentially. Then once it hits the next class to enable multi-processing again?
This is nothing to do with fixtures in this test suite they simply can't run concurrently. They are executing APIs which affect other tests running at the same time.
Thanks in advance.
I would use nose attribute plugin to decorate tests that require disabling multiprocessing explicitly and run two nose commands: one with multiprocessing enabled, excluding sensitive tests, and one with multiprocessing disabled, including only sensitive tests. You would have to rely on CI framework should combine test results. Something like:
from unittest import TestCase
from nose.plugins.attrib import attr
#attr('sequential')
class MySequentialTestCase(TestCase):
def test_in_seq_1(self):
pass
def test_in_seq_2(self):
pass
class MyMultiprocessingTestCase(TestCase):
def test_in_parallel_1(self):
pass
def test_in_parallel_2(self):
pass
And run it like:
> nosetests -a '!sequential' --processes=10
test_in_parallel_1 (ms_test.MyMultiprocessingTestCase) ... ok
test_in_parallel_2 (ms_test.MyMultiprocessingTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.071s
OK
> nosetests -a sequential
test_in_seq_1 (ms_test.MySequentialTestCase) ... ok
test_in_seq_2 (ms_test.MySequentialTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
I am using unittest to test my Flask application, and nose to actually run the tests.
My first set of tests is to ensure the testing environment is clean and prevent running the tests on the Flask app's configured database. I'm confident that I've set up the test environment cleanly, but I'd like some assurance of that without running all the tests.
import unittest
class MyTestCase(unittest.TestCase):
def setUp(self):
# set some stuff up
pass
def tearDown(self):
# do the teardown
pass
class TestEnvironmentTest(MyTestCase):
def test_environment_is_clean(self):
# A failing test
assert 0 == 1
class SomeOtherTest(MyTestCase):
def test_foo(self):
# A passing test
assert 1 == 1
I'd like the TestEnvironmentTest to cause unittest or noseto bail if it fails, and prevent SomeOtherTest and any further tests from running. Is there some built-in method of doing so in either unittest (preferred) or nose that allows for that?
In order to get one test to execute first and only halt execution of the other tests in case of an error with that test, you'll need to put a call to the test in setUp() (because python does not guarantee test order) and then fail or skip the rest on failure.
I like skipTest() because it actually doesn't run the other tests whereas raising an exception seems to still attempt to run the tests.
def setUp(self):
# set some stuff up
self.environment_is_clean()
def environment_is_clean(self):
try:
# A failing test
assert 0 == 1
except AssertionError:
self.skipTest("Test environment is not clean!")
For your use case there's setUpModule() function:
If an exception is raised in a setUpModule then none of the tests in
the module will be run and the tearDownModule will not be run. If
the exception is a SkipTest exception then the module will be
reported as having been skipped instead of as an error.
Test your environment inside this function.
You can skip entire test cases by calling skipTest() in setUp(). This is a new feature in Python 2.7. Instead of failing the tests, it will simply skip them all.
I'm not quite sure whether it fits your needs, but you can make the execution of a second suite of unittests conditional on the result of a first suite of unittests:
envsuite = unittest.TestSuite()
moretests = unittest.TestSuite()
# fill suites with test cases ...
envresult = unittest.TextTestRunner().run(envsuite)
if envresult.wasSuccessful():
unittest.TextTestRunner().run(moretests)