I have the testcases structured in the following way.
app
test_login.py
class1
test_11
test_12
test_function1.py
class2
test_21
test_22
When I run "pytest.exe app", pytest is able to identify all the testcases, but it executes in random order. For example, test11, test22, test12 and so on
Is there any way I can change this and execute all testcases in a file::class first and then move on to another file::class?
it executes in random order
By default, tests are sorted by modules; inside the modules, tests are executed in the order they are specified. So you should get the rough order like this:
$ pytest --collect-only -q
test_function1.py::class2::test_21
test_function1.py::class2::test_22
test_login.py::class1::test_11
test_login.py::class1::test_12
Is there any way I can change this and execute all testcases in a file::class first and then move on to another file::class?
If you want to change the default execution order, you can do it in the pytest_collection_modifyitems hook. Example that reorders the collected tests by class name, then by test name:
# conftest.py
import operator
def pytest_collection_modifyitems(items):
items.sort(key=operator.attrgetter('cls.__name__', 'name'))
Now tests in test_login will be executed before those in test_function1 because the module names are not counted in the ordering anymore:
$ pytest --collect-only -q
test_login.py::class1::test_11
test_login.py::class1::test_12
test_function1.py::class2::test_21
test_function1.py::class2::test_22
Following code block solved my issue. Thanks #hoefling for your time and help.
# conftest.py
#pytest.hookimpl(hookwrapper=True)
def pytest_collection_modifyitems(items):
yield
items.sort(key=operator.attrgetter('cls.__name__'))
Related
I have a bunch of tests which I decided to put within a class, sample code is below:
class IntegrationTests:
#pytest.mark.integrationtest
#pytest.mark.asyncio
async def test_job(self):
assert await do_stuff()
However, when I try to run the tests:
pipenv run pytest -v -m integrationtest, they are not detected at all, where I got the following before moving them to a class:
5 passed, 4 deselected in 0.78 seconds
I now get this:
2 passed, 4 deselected in 0.51 seconds
Why does pytest not detect these tests? Are test classes not supported?
The name of the class needs to start with Test for the pytest discovery to find it.
class TestIntegration:
#pytest.mark.integrationtest
#pytest.mark.asyncio
async def test_job(self):
assert await do_stuff()
See Conventions for Python test discovery
Create a pytest.ini
From the docs:
In case you need to change the naming convention for test files, classes and tests, you can create a file pytest.ini, and set the options python_files, python_classes, and python_functions:
Example:
# content of pytest.ini
# Example 1: have pytest look for "check" instead of "test"
# can also be defined in tox.ini or setup.cfg file, although the section
# name in setup.cfg files should be "tool:pytest"
[pytest]
python_files = check_*.py
python_classes = *Tests
python_functions = *_check
In your case, if you don't want to change the name of the class IntegrationTests, set python_classes to *Tests.
Running tests inside a class
pytest /path/to/test_file_name.py::ClassName
Running a test inside a class
pytest /path/to/test_file_name.py::ClassName::test_name
To run all the tests under the class, "TestIntegration" you can use:
pytest -k TestIntegration
put the decorator above the class, the class with tests inside is like a group already.
#pytest.mark.smoke1
class TestClass():
I'm considering converting some unittest.TestCase tests into Pytest ones to take advantage of Pytest's fixtures. One feature of unittest that I wasn't able to easily find the equivalent of in Pytest, however, is the ability to create testing suites and run them. I currently often do something like this:
import unittest
class TestSomething(unittest.TestCase):
def test_1(self):
self.assertEqual("hello".upper(), "HELLO")
def test_2(self):
self.assertEqual(1+1, 2)
if __name__ == "__main__":
suite = unittest.TestSuite()
# suite.addTest(TestSomething('test_1'))
suite.addTest(TestSomething('test_2'))
runner = unittest.TextTestRunner()
runner.run(suite)
By commenting in and out the lines with addTest, I can easily select which tests to run. How would I do something similar with Pytest?
You can use the -k argument to run specific tests. For example
# put this in test.py
import unittest
class TestSomething(unittest.TestCase):
def test_1(self):
self.assertEqual("hello".upper(), "HELLO")
def test_2(self):
self.assertEqual(1+1, 2)
Running all tests in the class TestSomething can be done like this:
py.test test.py -k TestSomething
Running only test_2:
py.test test.py -k "TestSomething and test_2"
More examples in the documentation
Another way to go is to use special test names. These can be configures in the pytest.ini file.
# content of pytest.ini
# can also be defined in tox.ini or setup.cfg file, although the section
# name in setup.cfg files should be "tool:pytest"
[pytest]
python_files=check_*.py
python_classes=Check
python_functions=*_check
Another way is to take action in conftest.py. In this example the collect_ignore config variable is used. It is a list of test paths that are to be ignored. In this example test_somthing.py is always ignored for collection. test_other_module_py2.py is ignored if we are testing with a python 3.
# content of conftest.py
import sys
collect_ignore = ["test_something/test_something.py"]
if sys.version_info[0] > 2:
collect_ignore.append("test_other/test_other_module_py2.py")
Since pytest 2.6 it is also possible to omit classes from test registration like this:
# Will not be discovered as a test
class TestClass:
__test__ = False
These examples were loosely taken from the documentation of pytest chapter Changing standard (Python) test discovery
In addition to using -k filters, you can name specific test classes or cases you want to run,
py.test test.py::TestSomething::test_2
Would run just test_2
Think the best way to do this is to use custom pytest markers.
You should mark specific tests (which you want to run) with
#pytest.mark.mymarkername
And run only tests with the custom marker using command:
py.test -v -m mymarkername
Here you can find more info regarding markers:
http://doc.pytest.org/en/latest/example/markers.html
Building on mbatchkarov's answer, since the names of my tests can get quite lengthy, I would like to still be able to select tests by commenting in and out lines and hitting "Cntrl+B" in Sublime (or "Cntrl+R" using the Atom Runner). One way to do this is as follows:
import unittest
import pytest
class TestSomething(unittest.TestCase):
def test_1(self):
self.assertEqual("hello".upper(), "HELLO")
def test_2(self):
self.assertEqual(1+1, 2)
if __name__ == "__main__":
tests_to_run = []
# tests_to_run.append('TestSomething and test_1')
tests_to_run.append('TestSomething and test_2')
tests_to_run = " or ".join(tests_to_run)
args = [__file__, '-k', tests_to_run]
pytest.main(args)
The idea behind this is that because Pytest accepts a string expression to match tests (rather than just a list of tests), one must generate a list of expressions matching one test only, and concatenate them using or.
I have unit tests that runs with py.test for python 2.7 and py.test 3.0. My test directory is like this:
tests
---dir1
test1.py
-------sub-dir1-1
test-1-1.py
-------sub-dir1-2
test-1-2.py
---dir2
test2.py
-------sub-dir2-1
test-2-1.py
-------sub-dir2-2
test-2-2.py
I want all my tests to run a common setup and tear down function before and after test. I would like to do it with the least modification of all the test code.
Thanks
If I understand correctly, you can write a fixture that is session scoped (scope param) and make it used automatically (autouse param). I used a "yield fixture" as an example. Please note, that pytest.yield_fixture is deprecated since pytest 3.0 and pytest.fixture allows use of yield.
import pytest
#pytest.fixture(scope="session", autouse=True)
def callattr_ahead_of_alltests(request):
print 'run_pre_start'
yield
print 'run_after_finish'
It will run before first test (printing "run_pre_start") and part after yield will run after all the tests (printing "run_after_finish").
Is there any way I can execute the tests in the order in which they were written?
What happens in PyUnit is whenever I run tests it run in alphabetical order. This means even if I have written TestA after TestB, TestA will run before TestA. Which is creating problems for me.
import unittest
class SimpleTestCase(unittest.TestCase):
def testB(self):
print "Test B"
def testA(self):
print "Test A"
I want testB to execute before testA.
I found a solution for it using PyTest ordering plugin provided here.
Try py.test YourModuleName.py -vv in CLI and the test will run in the order they have appeared in your module (first testB and then testA)
I did the same thing and works fine for me.
Note: You need to install PyTest package and import it.
If these are unit tests then they should be completely isolated, so you should check for design flaws in those tests.
If you really need for some reason, to use a specific order, than you have three ways of achieving this in python:
Using unittest - change sorting method, described here.
Using Proboscis - use the following decorator #test(depends_on=[list of dependecies]), found here.
Using nose - nose executes its unit tests in the order in which they appear in the module file. More info here.
PyUnit uses TestLoaded which collects all testcases in a suite and runs them in alphabetical order
For example Test A runs before TEST B
In case you want to run TEST B first we have to create function and add Test B and then Test A
def suite():
suite = unittest.TestSuite()
suite.addTest(SimpleTestCase('test_B'))
suite.addTest(SimpleTestCase('test_A'))
return suite
Lets say I have the following testcases in different files
TestOne.py {tags: One, Two}
TestTwo.py {tags: Two}
TestThree.py {tags: Three}
Each of which inherits from unittest.TestCase. Is there any ability in python to embed metadata information within these files, so that I can have a main.py script to search for those tags and execute only those testcases?
For Eg: If I want to execute testcases with {tags: Two} then only testcases TestOne.py and TestTwo.py should be executed.
The py.test testing framework has support for meta data, via what they call markers.
For py.test test cases are functions that have names starting with "test", and which are in modules with names starting with "test". The tests themselves are simple assert statements. py.test can also run tests for the unittest library, and IIRC Nose tests.
The meta data consists of dynamically generated decorators for the test functions. The decorators have the form: #pytest.mark.my_meta_name. You can choose anything for my_meta_name. There are a few predefined markers that you can see with py.test --markers.
Here is an adapted snippet from their documentation:
# content of test_server.py
import pytest
#pytest.mark.webtest
def test_send_http():
pass # perform some webtest test for your app
def test_always_succeeds():
assert 2 == 3 - 1
def test_will_always_fail():
assert 4 == 5
You select marked tests with the -m command line option of the test runner. To selectively run test_send_http() you enter this into a shell:
py.test -v -m webtest
Of course it's more easy to define tags in the main module, but if it's important for you to save them with test files, it could be a good solution to define it in test files like this:
In TestOne.py:
test_tags = ['One', 'Two']
...
Then you can read all tags in the initialize function of your main module in this way:
test_modules = ['TestOne', 'TestTwo', 'TestThree']
test_tags_dict = {}
def initialize():
for module_name in test_modules:
module = import_string(module)
if hasattr(module, 'test_tags'):
for tag in module.test_tags:
if tag not in test_tags_dict:
test_tags_dict[tag] = []
test_tags_dict[tag].append(module)
So you can implement a run_test_with_tag function to run all tests for an specific tag:
def run_test_with_tag(tag):
for module in test_tags_dict.get(tag, []):
# Run module tests here ...