I have following code in my tests module
def teardown_module():
clean_database()
def test1(): pass
def test2(): assert 0
and I want teardown_module() (some cleanup code) to be called only if some test failed. Otherwise (if all passed) this code shouldn't have to be called.
Can I do such a trick with PyTest?
You can. But it is a little bit of a hack.
As written here: http://pytest.org/latest/example/simple.html#making-test-result-information-available-in-fixtures
you do the following, to set up an attribute for saving the status of each phase of the testcall:
# content of conftest.py
import pytest
#pytest.mark.tryfirst
def pytest_runtest_makereport(item, call, __multicall__):
rep = __multicall__.execute()
setattr(item, "rep_" + rep.when, rep)
return rep
and in the fixture you just examine the condition on those attributes like this:
import pytest
#pytest.yield_fixture(scope="module", autouse=True)
def myfixture(request):
print "SETUP"
yield
# probably should not use "_collected" to iterate over test functions
if any(call.rep_call.outcome != "passed" for call in request.node._collected):
print "TEARDOWN"
This way if any of the tests associated with that module fixture is not "passed" (so "failed" or "skipped") then the condition holds.
The answer posted here and link to documentation was helpful but not sufficient for my needs. I needed a module teardown function to execute for each module independently if any test in that module (.py) file failed.
A complete sample project is available on GitHub
To start with, we need a hook to attach the test function result to
the test node. This is taken directly from the pytest docs:
# in conftest.py
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
# execute all other hooks to obtain the report object
outcome = yield
rep = outcome.get_result()
# set a report attribute for each phase of a call, which can
# be "setup", "call", "teardown"
var_name = "rep_" + rep.when
setattr(item, var_name, rep)
After that, we need another hook for the test case to find the module and
store itself there, so the module can easily find its test cases. Perhaps
there's a better way, but I was unable to find one.
# also in conftest.py
#pytest.fixture(scope="function", autouse=True)
def _testcase_exit(request):
yield
parent = request.node.parent
while not isinstance(parent, pytest.Module):
parent = parent.parent
try:
parent.test_nodes.append(request.node)
except AttributeError:
parent.test_nodes = [request.node]
Once we do that, it's nice to have a decorator function to have the module on
completion look through its test nodes, find if there are any failures, and
then if there were call the function associated with the decorator:
# also also in conftest.py
def module_error_teardown(f):
#wraps(f)
#pytest.fixture(scope="module", autouse=True)
def wrapped(request, *args, **kwargs):
yield
try:
test_nodes = request.node.test_nodes
except AttributeError:
test_nodes = []
something_failed = False
for x in test_nodes:
try:
something_failed |= x.rep_setup.failed
something_failed |= x.rep_call.failed
something_failed |= x.rep_teardown.failed
except AttributeError:
pass
if something_failed:
f(*args, **kwargs)
return wrapped
Now we have all the necessary framework to work with. Now, a test file with a failing test case is easy to write:
from conftest import module_error_teardown
def test_something_that_fails():
assert False, "Yes, it failed."
def test_something_else_that_fails():
assert False, "It failed again."
#module_error_teardown
def _this_gets_called_at_the_end_if_any_test_in_this_file_fails():
print('')
print("Here's where we would do module-level cleanup!")
Related
Assume we have:
#pytest.fixture()
def setup():
print('All set up!')
return True
def foo(setup):
print('I am using a fixture to set things up')
setup_done=setup
I'm looking for a way to get to know caller function name (in this case: foo) from within setup fixture.
So far I have tried:
import inspect
#pytest.fixture()
def setup():
daddy_function_name = inspect.stack()[1][3]
print(daddy_function_name)
print('All set up!')
return True
But what gets printed is: call_fixture_func
How do I get foo from printing daddy_function_name?
You can use the built-in request fixture in your own fixture:
The request fixture is a special fixture providing information of the requesting test function.
Its node attribute is the
Underlying collection node (depends on current request scope).
import pytest
#pytest.fixture()
def setup(request):
return request.node.name
def test_foo(setup):
assert setup == "test_foo"
I have a BaseTest class which has tear_down and I want to have inside tear_down a variable representing wether or not the test has failed.
I tried look at A LOT of older posts but I coulden't implement them as they were hooks or mixture of hook and fixture and something did not work on my end.
What is the best practice for doing that?
Last thing I've tried was -
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item):
outcome = yield
rep = outcome.get_result()
# set a report attribute for each phase of a call, which can
# be "setup", "call", "teardown"
setattr(item, "rep_" + rep.when, rep)
Then pass request fixture to teardown and inside use
has_failed = request.node.rep_call.failed
But request had no attributes at all, it was a method.
Also tried -
#pytest.fixture
def has_failed(request):
yield
return True if request.node.rep_call.failed else False
and pass it like that.
def teardown_method(self, has_failed):
And again, no attributes.
Isn't there a simple fixture to just do like request.test_status or something like that?
It's important that the teardown will have that bool parameter wether or not it failed and not do stuff outside the teardown.
Thanks!
There doesn't appear to be any super simple fixture offering the test report as a fixture. And I see what you mean: most examples of recording the test report are geared toward non-unittest use cases (including the official docs). However, we can adjust these examples to work with unittest TestCases.
There appears to be a private _testcase attribute on the item arg passed to pytest_runtest_makereport, which contains the instance of the TestCase. We can set an attribute on it, which can then be accessed within teardown_method.
# conftest.py
import pytest
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
report = outcome.get_result()
if report.when == 'call' and hasattr(item, '_testcase'):
item._testcase.did_pass = report.passed
And here's a dinky little example TestCase
import unittest
class DescribeIt(unittest.TestCase):
def setup_method(self, method):
self.did_pass = None
def teardown_method(self, method):
print('\nself.did_pass =', self.did_pass)
def test_it_works(self):
assert True
def test_it_doesnt_work(self):
assert False
When we run it, we find it prints the proper test failure/success bool
$ py.test --no-header --no-summary -qs
============================= test session starts =============================
collected 2 items
tests/tests.py::DescribeIt::test_it_doesnt_work FAILED
self.did_pass = False
tests/tests.py::DescribeIt::test_it_works PASSED
self.did_pass = True
========================= 1 failed, 1 passed in 0.02s =========================
I would like to avoid using the "test" prefix in classes and functions names and implement my own schema of the test parametrization.
I did the next code
test.py
import pytest
# class for inheritance to avoid "Test" prefix
class AtsClass:
__ATS_TEST_CLASS__ = True
# decorator to mark functions as tests (to avoid "Test" prefix)
def ats_test(f):
setattr(f, "__ATS_TEST_CLASS__", True)
return f
def test_1():
pass
#ats_test
def some_global_test():
pass
class MyClass(AtsClass):
def test_4(self):
pass
#ats_test
def some_func(self):
pass
conftest.py
import pytest
import inspect
# #pytest.hookimpl(hookwrapper=True)
def pytest_pycollect_makeitem(collector, name, obj):
# outcome = yield
# res = outcome.get_result()
if inspect.isclass(obj) and obj.__name__ != "AtsClass" and hasattr(obj, "__ATS_TEST_CLASS__") and obj.__ATS_TEST_CLASS__ == 1:
print("WE HAVE FOUND OUR CLASS")
return pytest.Class(name, parent=collector)
# outcome.force_result(pytest.Class(name, parent=collector))
if inspect.isfunction(obj) and hasattr(obj, "__ATS_TEST_CLASS__") and obj.__ATS_TEST_CLASS__ == 1:
print("WE HAVE FOUND OUR FUNCTION")
return pytest.Function(name, parent=collector)
# outcome.force_result([pytest.Function(name, parent=collector)])
def pytest_generate_tests(metafunc):
print("-->Generate: {}".format(metafunc.function.__name__))
In this case hook "pytest_pycollect_makeitem" creates test for function "some_global_test", but hook "pytest_generate_tests" is not executed for function "some_global_test".
I have found a solution, call collector._genfunctions(name, obj) from my hook. But I think it is not the right decision, cause _genfunctions is a private method and not declared.
Is there another way to solve my task?
So, nobody knows the answer and I decided to offer my solution (it can be useful for others):
class TestBaseClass:
__test__ = True
def mark_test(f):
setattr(f, "__test__", True)
return f
# using base class and decorator
class MyTestClass(TestBaseClass):
#mark_test
def some_func(self):
pass
Pytest uses attribute __test__ to detect nose-tests, so you can use nose-library or just use such base class and decorator.
If you want only to change prefix of tests you can set custom python_functionsand python_classes options at pytest.ini.
For more information follow a link.
So what I would like to achieve is mocking functions in various modules automatically with pytest. So I defined this in my conftest.py:
import sys
import __builtin__
from itertools import chain
# Fixture factory magic START
NORMAL_MOCKS = [
"logger", "error", "logging", "base_error", "partial"]
BUILTIN_MOCKS = ["exit"]
def _mock_factory(name, builtin):
def _mock(monkeypatch, request):
module = __builtin__ if builtin else request.node.module.MODULE
ret = Mock()
monkeypatch.setattr(module, name, ret)
return ret
return _mock
iterable = chain(
((el, False) for el in NORMAL_MOCKS),
((el, True) for el in BUILTIN_MOCKS))
for name, builtin in iterable:
fname = "mock_{name}".format(name=name)
_tmp_fn = pytest.fixture(name=fname)(_mock_factory(name, builtin))
_tmp_fn.__name__ = fname
setattr(
sys.modules[__name__],
"mock_{name}".format(name=name), _tmp_fn)
# Fixture normal factory magic END
This works and all, but I would like to omit the usage of the NORMAL_MOCKS and BUILTIN_MOCKS lists. So basically in a pytest hook I should be able to see that say there is a mock_foo fixture, but it's not registered yet, so I create a mock for it with the factory and register it. I just couldn't figure out how to do this. Basically I was looking into the pytest_runtest_setup function, but could not figure out how to do the actual fixture registration. So basically I would like to know with which hook/call can I register new fixture functions programatically from this hook.
One of the ways is to parameterize the tests at the collection/generation stage, i.e. before the test execution begins: https://docs.pytest.org/en/latest/example/parametrize.html
# conftest.py
import pytest
def mock_factory(name):
return name
def pytest_generate_tests(metafunc):
for name in metafunc.fixturenames:
if name.startswith('mock_'):
metafunc.parametrize(name, [mock_factory(name[5:])])
# test_me.py
def test_me(request, mock_it):
print(mock_it)
A very simple solution. But the downside is that the test is reported as parametrized when it actually is not:
$ pytest -s -v -ra
====== test session starts ======
test_me.py::test_me[it] PASSED
====== 1 passed in 0.01 seconds ======
To fully simulate the function args without the parametrization, you can make a less obvious trick:
# conftest.py
import pytest
def mock_factory(name):
return name
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item, nextitem):
for name in item.fixturenames:
if name.startswith('mock_') and name not in item.funcargs:
item.funcargs[name] = mock_factory(name[5:])
yield
The pytest_runtest_setup hook is also a good place for this, as long as I've just tried.
Note that you do not register the fixture in that case. It is too late for the fixture registration, as all the fixtures are gathered and prepared much earlier at the collection/parametrization stages. In this stage, you can only execute the tests and provide the values. It is your responsibility to calculate the fixture values and to destroy them afterward.
The snippet below is a pragmatic solution to "how to dynamically add fixtures".
Disclaimer: I don't have expertise on pytest. I'm not saying this is what pytest was designed for, I just looked at the source code and came up with this and it seems to work. The fact that I use "private" attributes means it might not work with all versions (currently I'm on pytest 7.1.3)
from _pytest.fixtures import FixtureDef
from _pytest.fixtures import SubRequest
import pytest
#pytest.fixture(autouse=True) # autouse is relevant, as then the fixture registration happens in-time. It's too late if requiring the fixture without autouse e.g. like `#pytest.mark.usefixtures("add_fixture_dynamically")`
def add_fixture_dynamically(request: SubRequest):
"""
Conditionally and dynamically adds another fixture. It's conditional on the presence of:
#pytest.mark.my_mark()
"""
marker = request.node.get_closest_marker("my_mark")
# don't register fixture if marker is not present:
if marker is None:
return
def your_fixture(): # the name of the fixture must match the parameter name, like other fixtures
return "hello"
# register the fixture just-in-time
request._fixturemanager._arg2fixturedefs[your_fixture.__name__] = [
FixtureDef(
argname=your_fixture.__name__,
func=your_fixture,
scope="function",
fixturemanager=request._fixturemanager,
baseid=None,
params=None,
),
]
yield # runs the test. Could be wrapped in try/except/finally
# suppress warning (works if this and `add_fixture_dynamically` are in `conftest.py`)
def pytest_configure(config):
"""Prevents printing of the warning 'PytestUnknownMarkWarning: Unknown pytest.mark.<fixture_name>'"""
config.addinivalue_line("markers", "my_mark")
#pytest.mark.my_mark()
def test_adding_fixture_dynamically(your_fixture):
assert your_fixture == "hello"
Suppose I have the following TestCase
class TestSomething(unittest.TestCase):
def test_a(self):
# Do some testing
def test_b(self):
# Do some other testing
Is it possible for me to run TestSomething.test_a if I have a reference to that test? What I am looking to do is something like:
def run_test(test):
# Somehow runs the test
# HERE IS THE PART I AM REALLY STUCK ON
run_test(TestSomething.test_a)
I know that it's an awkward thing to do for normal unit testing. What I am trying to do is provide a test to be run as an argument to a function decorator. Essentially:
#corresponding_test(TestSomething.test_a)
def my_function_a():
# Something here
And then in the decorator basically check if the test for that function passes before running the function.
OP clearly stated that the real world use case is more involved, but this still needs saying:
Disclaimer: This is not a good, standard way to run unit tests. If you use this code to run unit tests, you're [probably] doing it wrong.
That said, your question intrigued me, so I went ahead and wrote a working demo for you:
"""
The `only_if_test_passes` decorator can be used to run a function if and
only if the argument test (unbound `TestCase` method) passes.
"""
import inspect
from unittest import TestCase, TestResult
class TestError(Exception):
pass
class MyTests(TestCase):
def test_pass(self):
# This passes because nothing went wrong
pass
def test_fail(self):
self.fail('This test will always fail')
def only_if_test_passes(test_method):
# Comments are computed values when passed MyTests.test_pass
test_case_class = inspect._findclass(test_method) # MyTests
test_case_name = test_case_class.__name__ # 'MyTests'
test_name = test_method.__name__ # 'test_pass'
# Introspection for Python 2:
# test_case_class = test_method.im_class
# test_case_name = test_case_class.__name__ # Same as for Python 3
# test_name = test_method.if_func.func_name
def decorator(fn):
def decorated(*args, **kwargs):
test_result = TestResult()
case = test_case_class(test_name) # MyTests('test_pass')
case(test_result)
if test_result.wasSuccessful():
return fn(*args, **kwargs)
else:
raise TestError('Unit test failed: {}.{}'.format(
test_case_name, test_name))
return decorated
return decorator
#only_if_test_passes(MyTests.test_pass)
def this_will_run():
print('This should output')
#only_if_test_passes(MyTests.test_fail)
def this_wont_ever_run():
print("Don't bother; you'll never see this.")
if __name__ == "__main__":
this_will_run()
this_wont_ever_run()
gist
The introspection will be a little different in Python 2.
See also: unittest.TestCase docs