I have a test framework that is performing multiple asserts and catching them (inherited someone else's code). The proprietary results report is correct, but as you may have guessed, pytest will mark these as passed:
test_something.py::TestSomething::test_that_should_fail PASSED
I've added an autouse fixture as such:
#pytest.fixture(autouse=True)
def run_after_tests(self):
yield
if self.actually_failed():
pytest.fail("Yay, failing when failures occur is cool!")
This solution works okay, except that it seems like the clean up happens after the test has already been marked as PASSED and a duplicate test is shown with an error.
Now pytest results look like this:
test_something.py::TestSomething::test_that_should_fail PASSED
test_something.py::TestSomething::test_that_should_fail ERROR
Is there a way to delay the evaluation of the test so it doesn't say it has passed?
I know this is a really stupid way of doing things and performing test evaluation at cleanup is not recommended, but there are too many tests that have been written this way and spending weeks to refactor the test is not feasible.
An alternative solution I've thought of is to write a decorator and then sed all the test cases and add it to the functions; but this is going to be my plan B if fixtures can't solve this.
Thanks!
As you've discovered, a fixture is a separate object from the test itself.
You'll need to modify the appropriate pytest hook. I haven't personally tested, but I believe placing the following code into your projects conftest.py will give you your desired result.
def check_for_failure(output) -> bool:
# define me
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
output = yield
if check_for_failure(output):
pytest.fail()
Related
I am going through pytest fixtures, and the following looks pretty similar, latest works pretty similar.
Yes, the readability is better in yield_fixure, however could someone let me know what exactly is the difference.
which should I use, in cases like mentioned below?
#pytest.fixture()
def open_browser(request):
print("Browser opened")
def close_browser():
print("browser closed")
request.addfinalizer(close_browser)
return "browser object"
#pytest.yield_fixture()
def open_browser():
print("Browser opened")
yield "browser object"
print("browser closed")
def test_google_search(open_browser):
print(open_browser)
print("test_google_search")
The only difference is in readability. I think (though I'm not 100% sure) the underlying behavior is identical (i.e. the cleanup after the yield statement is run as a finalizer). I always prefer using yield fixtures for cleanup, since it's more readable.
If you're using pytest <3.0, you'll still need to use pytest.yield_fixture to get that behavior. But if you're able to use pytest 3.0+, pytest.yield_fixture is deprecated and you can use pytest.fixture to get the same yield_fixture behavior.
Here are the explanatory docs:
Since pytest-3.0, fixtures using the normal fixture decorator can use
a yield statement to provide fixture values and execute teardown code,
exactly like yield_fixture in previous versions.
Marking functions as yield_fixture is still supported, but deprecated
and should not be used in new code.
addfinalizer has two key differences over yield:
It is possible to register multiple finalizer functions.
Finalizers will always be called regardless if the fixture setup
code raises an exception. This is handy to properly close all
resources created by a fixture even if one of them fails to be
created/acquired
From the pytest docs
I am running tests in two modes: with bare pytest and with pytest-xdist.
I have a heavy fixture that was defined with module scope. Inside this fixture, I have some optimization for the case when I am running tests with xdist:
#pytest.fixture(scope="module")
def myfixture(request):
if running_with_pytest:
pass
else:
pass
It works fine, but I also want to change fixture's scope to session(if I run tests with xdist).
How can I do this?
I gave a really good try to command line options, but it seems that fixture param scope only accepts string value, can't take callable function:
https://github.com/pytest-dev/pytest/issues/1682
I guess just need to wait until the issue is resolved.
But if you figure this one out please share your solution.
How can I find out from the current test if its the last to be run? (Python unittest / nosetests)
I have some specific fixture teardown to be done at the very end of the test run and it would be a lot easier if on a test by test basis I could just determine:
if last_test:
hard_fixture_teardown()
else:
soft_fixture_teardown()
I have a package teardown which would work perfectly but it seems very messy passing the fixture information back to the __init__.teardown_package().
You can use a combination of TestCase.tearDown() and TestCase.tearDownClass() to achieve this. tearDown() is called for each test method while tearDownClass() is called after all tests in the class have run.
As stated here unit test are not meant to have an order, unit tests depending on the order are either not well conceived or just have to be merged in a monolithic one. (merging separate functions in a single test is the accepted answer )
[edit after comment]
If the order is not important you can do this (quite messy, imo we are still forcing the boundaries of how unit tests should be used)
in every test package you put:
def tearDownModule():
xxx = int(os.getenv('XXX', '0')) + 1
if xxx == NUMBER_OF_TEST_PACKAGES:
print "hard tear down"
else:
print "not yet"
os.environ['XXX'] = str(xxx)
with NUMBER_OF_TEST_PACKAGES imported from somewhere global.
also if the order is not important I suppose that when used the fixture is only readed and not modified, if so you can setup that as a class method
#classmethod
def setUpClass(cls):
print "I'll take a lot of time here"
I'm new to testing and I would like to
1) test the login
2) create a folder
3) add content (a page) into the folder
I have each of the tests written and they work but obviously I would like to build ontop of each other, eg, in order to do 3 I need to do 1 then 2. In order to do 2 I need to do 1. This is my basic test structure:
class TestSelenium(unittest.TestCase):
def setUp(self):
# Create a new instance of the Firefox driver
self.driver = webdriver.Firefox()
def testLogin(self):
print '1'
...
def testFolderCreation(self):
print '2'
...
def testContentCreation(self):
print '3'
...
def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
unittest.main()
At first, I thought the tests would run in order and the 2nd function would continue off where the first one left off, but I've found this is not the case, it seems to be starting over with each test. I've also realized that they execute in reverse order. I get an output of 3,2,1 in the terminal. How should I achieve what I want? If I call the previous functions before I run the one I want, I feel like it's repetitively testing the same thing over and over since each one is a test (eg, in testContentCreation, I would call 'testLogin' then call testFolderCreation and inside testFolderCreation call testLogin. If I were to do more, the testLogin would've been called a number of times!). Should I instead turn the previous steps into regular non-test functions and in the final last one (the test function) call the previous ones in order? If I do it that way then I guess if any of the steps fail, the last one fails, there would be one big test function.
Any suggestions on how you should write this type of test?
Also, why are the tests running in reverse order?
Thanks!
You are seeing what you are seeing, I think, because you are making some incorrect assumptions about the assumptions unittest makes. Each test case is assumed to be a self-contained entity, so there is no run order imposed. In addition, SetUp() and TearDown() operate before and after each individual case. If you want global setup/teardown, you need to make classmethods named SetUpClass() and TearDownClass(). You may also want to look in to the TestSuite class. More here: http://docs.python.org/library/unittest.html
Keep in mind that when the unittest library does test discovery (reflects your testcase class to find the test cases to run), it is essentially limited to looking at the .__dict__ and dir() values for the object, which are inherently unordered.
Suppose I have a unittest like this
class TestABC(unittest.TestCase):
def setUp(self):
....
def test001_abc(self):
....
def test002_abc(self):
....
def test003_bac(self):
....
self.test001_abc()
But the output shows Run 3 tests in 10.962s. I believe that the last self.test001_bac() had been ran, but can we indicate this in the test count?
Thanks.
The last "test" was run as part of test_003_bac, so unittest does not see it as a separate test. I'm not sure what you're trying to achieve with this, but running one test inside another is not a good idea. nose supports the concept of Test Generators, which might do what you want.
Your question doesn't make sense. There are just three tests in your code; the fact that the last test calls another test doesn't magically "split" the test case into two, independent tests - test003_bac is still just one, single test.