I am using py.test to write some tests and in my tests I utilize funcargs. These funcargs have their own setups and teardowns defined in the conftest.py like this:
conftest.py:
def pytest_funcarg__resource_name(request):
def setup():
# do setup
def teardown():
# do teardown
My problem is when someone uses CTRL+C to stop the test executions it leaves everything un-teardowned.
I know there is a hook pytest_keyboard_interrupt but I dont know what to do from there.
Sorry for the noobish question.
You don't provide a full example so maybe i am missing something. But here is an example of how it can work, using the request.cached_setup() helper:
def pytest_funcarg__res(request):
def setup():
print "res-setup"
def teardown(val):
print "res-teardown"
return request.cached_setup(setup, teardown)
def test_hello(res):
raise KeyboardInterrupt()
If you run this with "py.test" you get:
============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.2.5.dev4
plugins: xdist, bugzilla, pep8, cache
collected 1 items
tmp/test_keyboardinterrupt.py res-setup
res-teardown
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/home/hpk/p/pytest/tmp/test_keyboardinterrupt.py:10: KeyboardInterrupt
which shows that setup and teardown are called if a KeyboardInterrupt occurs during test execution.
Related
Our pytest environment has a lot of fixtures (mostly scope='function' and scope='module') that are doing something of the form:
#pytest.yield_fixture(scope='function')
def some_fixture():
... some object initialization ...
yield some_object
... teardown ...
We use the teardown phase of the fixture (after the yield) to delete some resources created specifically for the test.
However, if a test is failing, I don't want the teardown to execute so we will have the resources still exist for further debugging.
For example, here is a common scenario that repeats in all of our testing framework:
#pytest.yield_fixture(scope='function')
def obj_fixture():
obj = SomeObj.create()
yield obj
obj.delete()
def test_obj_some_field(obj_fixture):
assert obj_fixture.some_field is True
In this case, if the condition in the assert is True I want the obj.delete() to execute.
However, if the test is failing, I want pytest to skip the obj.delete() and anything else after the yield.
Thank you.
EDIT
I want the process to be done without altering the fixture and the tests code, I prefer an automatic process instead of doing this refactor in our whole testing codebase.
There's an example in the pytest docs about how to do this. The basic idea is that you need to capture this information in a hook function and add it to the test item, which is available on the test request, which is available to fixtures/tests via the request fixture.
For you, it would look something like this:
# conftest.py
import pytest
#pytest.hookimpl(tryfirst = True, hookwrapper = True)
def pytest_runtest_makereport(item, call):
# execute all other hooks to obtain the report object
outcome = yield
rep = outcome.get_result()
# set a report attribute for each phase of a call, which can
# be "setup", "call", "teardown"
setattr(item, "rep_" + rep.when, rep)
# test_obj.py
import pytest
#pytest.fixture()
def obj(request):
obj = 'obj'
yield obj
# setup succeeded, but the test itself ("call") failed
if request.node.rep_setup.passed and request.node.rep_call.failed:
print(' dont kill obj here')
else:
print(' kill obj here')
def test_obj(obj):
assert obj == 'obj'
assert False # force the test to fail
If you run this with pytest -s (to not let pytest capture output from fixtures), you'll see output like
foobar.py::test_obj FAILED dont kill obj here
which indicates that we're hitting the right branch of the conditional.
Teardown is intended to be executed independently of whether a test passed or failed.
So I suggest to either write your teardown code such, that it is robust enough to be executed whether the test passed or failed or to add the cleanup to the end of your test, so that it will only be called if no preceding assert failed and if no exception occurred before
Set a class-level flag to indicate pass/fail and check that in your teardown. This is not tested, but should give you the idea::
#pytest.yield_fixture(scope='function')
def obj_fixture():
obj = SomeObj.create()
yield obj
if this.passed:
obj.delete()
def test_obj_some_field(obj_fixture):
assert obj_fixture.some_field is True
this.passed = True
I use a Makefile to execute pytest so had an additional tool at my disposal. I too needed the cleanup of fixtures to happen only on success. I added the cleanup as a second command to my test method in my Makefile.
clean:
find . | grep -E "(__pycache__|\.pyc|\.pyo)" | xargs rm -rf
-rm database.db # the minus here allows this to fail quietly
database:
python -m create_database
lint:
black .
flake8 .
test: clean lint database
pytest -x -p no:warnings
rm -rf tests/mock/fixture_dir
I am trying to check the exit code I have for a script I'm writing in python3 on a Mac (10.14.4). When I run the test it doesn't fail which I think is wrong. But I can't see what it is that I've got wrong.
The test file looks like this:
import pytest
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import my_script
class TestMyScript():
def test_exit(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
assert pytest_wrapped_e.type == SystemExit
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
self.assertEqual(pytest_wrapped_e.exception.code, 42)
My script looks like:
#!/usr/bin/env python3
import sys
def main():
print('Hello World!')
sys.exit(0)
if __name__ == '__main__':
main()
The output I get is:
$ py.test -v
============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-3.10.1, py-1.8.0, pluggy-0.9.0 -- /usr/local/opt/python/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/robertpostill/software/gateway, inifile:
plugins: shutil-1.6.0
collected 2 items
test/test_git_refresh.py::TestGitRefresh::test_exit PASSED [ 50%]
test/test_git_refresh.py::TestGitRefresh::test_exit_code PASSED [100%]
=========================== 2 passed in 0.02 seconds ===========================
$
I would expect the second test(test_exit_code) to fail as the exit call is getting a code of 0, not 42. But for some reason, the assert is happy whatever value I put in the sys.exit call.
Good question, that's because your Asserts are never called (either of them). When exit() is called the program is done (at least within the with clause), it turns off the lights, packs up its bags, and goes home. No further functions will be called. To see this add an assert before and after you call main:
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
self.assertEqual(0, 1) # This will make it fail
my_script.main()
self.assertEqual(0, 1) # This will never be called because main `exits`
self.assertEqual(pytest_wrapped_e.exception.code, 42)
A test passes if no asserts fail and nothing breaks, so in your case both tests passed because the assert was never hit.
To fix this pull your asserts out of the with statement:
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
self.assertEqual(pytest_wrapped_e.exception.code, 42)
Though now you will need to fix the pytest syntax because you are missing some other stuff.
See: Testing sys.exit() with pytest
I am trying to write a py.test test with selenium to test a complex website. Because the setup is complex, the test can fail even before the actual test. But in such a case I want to be able to call a 'debug' function (maybe its a teardown function which I can use to debug things.
Example:
The test uses a fixture which returns a selenium webdriver
def test1(driver):
driver.get("my page")
...
other tests, login, etc etc
But now the call driver.get fails because of any reason. But in such a case I want to be able to investigate things like
def debug(driver):
driver.screenshot(...)
print(driver.page_source)
...
driver.quit()
(including the shutdown of the driver, as the browser would stay open) with the same driver instance as has been used in the test method.
Is there a way to do that?
The trickiest part is to pass the test result into the fixture, the rest is pretty much trivial. Following the pytests example Making test result information available in fixtures, add a custom hook in your conftest.py:
import pytest
#pytest.hookimpl(hookwrapper=True, tryfirst=True)
def pytest_runtest_makereport(item, call):
outcome = yield
rep = outcome.get_result()
setattr(item, "rep_" + rep.when, rep)
return rep
Now you can enhance your driver fixture with custom teardown logic in case the test fails:
# test_selenium.py
import pytest
from selenium import webdriver
def debug(driver):
print('debug: ', driver.title)
#pytest.fixture
def driver(request):
driver = webdriver.Firefox()
yield driver
if request.node.rep_setup.passed and request.node.rep_call.failed:
# this is the teardown code executed on test failure only
debug(driver)
# this is the teardown code that is always executed
driver.quit()
def test_fail(driver):
driver.get('http://wtf')
def test_ok(driver):
driver.get('https://www.google.de')
Running the tests yields:
$ pytest -sv
=============================== test session starts ===============================
platform darwin -- Python 3.6.3, ...
cachedir: .cache
rootdir: /Users/hoefling/projects/private/stackoverflow/so-48521762, inifile:
plugins: ...
collecting ... collected 2 items
test_spam.py::test_fail FAILED [ 50%]
debug Server Not Found
test_spam.py::test_ok PASSED [100%]
==================================== FAILURES =====================================
____________________________________ test_fail ____________________________________
driver = <selenium.webdriver.firefox.webdriver.WebDriver( ...
...
------------------------------- Captured log setup --------------------------------
remote_connection.py 474 DEBUG POST http://127.0.0.1:50319/session { ...
...
remote_connection.py 561 DEBUG Finished Request
======================== 1 failed, 1 passed in 7.23 seconds =======================
I'm trying to do setup and teardown modules using pytest-bdd.
I know with behave you can do a environment.py file with before_all and after_all modules. How do I do this in pytest-bdd
I have looked into "classic xunit-style setup" plugin and it didn't work when I tried it. (I know thats more related to py-test and not py-test bdd).
You could just declare a pytest.fixture with autouse=true and whatever scope you want. You can then use the request fixture to specify the teardown. E.g.:
#pytest.fixture(autouse=True, scope='module')
def setup(request):
# Setup code
def fin():
# Teardown code
request.addfinalizer(fin)
A simple-ish approach for me is to use a trivial fixture.
# This declaration can go in the project's confest.py:
#pytest.fixture
def context():
class Context(object):
pass
return Context()
#given('some given step')
def some_when_step(context):
context.uut = ...
#when('some when step')
def some_when_step(context):
context.result = context.uut...
Note: confest.py allows you to share fixtures between codes, and putting everything in one file gives me a static analysis warning anyway.
"pytest supports execution of fixture specific finalization code when the fixture goes out of scope. By using a yield statement instead of return, all the code after the yield statement serves as the teardown code:"
See: https://docs.pytest.org/en/latest/fixture.html
eg.
#pytest.fixture(scope="module")
def smtp_connection():
smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
yield smtp_connection # provide the fixture value
print("teardown smtp")
smtp_connection.close()
I try to collect my tests with py.test but it doesn't do so.
Do I have to provide additional options at the command line?
Py.test was executed in the directory of my .py-file. Are there any other requirements?
Are my tests named correctly? In my code I used 'Test-' for classes and 'test_' for methods.
The results from the terminal:
> py.test
=============================== in 0.00 seconds ============================
user#host:~/workspace/my_project/src/html_extractor$ py.test
============================= test session starts ===========================
platform linux2 -- Python 2.7.3 -- pytest-2.3.4
plugins: xdist
collected 0 items
My code under test:
class Factories(object):
TEST_LINKS = ['http://www.burgtheater.at',
'http://www.musikverein.at/']
def links(self):
for link in self.TEST_LINKS:
yield link
class TestHTMLExtractor(unittest.TestCase):
def setUp(self):
self.factory = Factories()
self.extractor = HTMLExtractor()
def test_extract_site_return(self):
for link in self.factory.links():
raw_html_page = self.extractor.run(link)
def test_whatever():
pass
if __name__ == '__main__':
unittest.main()
In the default configuration, the test file should be named test_<something>.py. See Changing standard (Python) test discovery.