I have a number of test files, such as
test_func1.py
test_func2.py
test_func3.py
I know in advance that test_func3.py won't pass if I run Pytest in parallel, e.g. pytest -n8. The reason is that test_func3.py contains a number of parametrized tests that handle file i/o processes. Parallel writing to the same file leads to failures. In serial testing mode, all tests in this module passes.
I'm wondering how I can skip the whole module in case Pytest will be started with the option -n? My thinking is to apply the skipif marker. I need to check in my code whether the -n argument has been passed to pytest.
...>pytest # run all modules
...>pytest -n8 # skip module test_func3.py automatically
The pytest-xdist package supports four scheduling algorithms:
each
load
loadscope
loadfile
Calling pytest -n is a shortcut for load scheduling, i.e. the scheduler will load balance the tests across all workers.
Using loadfile scheduling, all test cases in a test file will be executed sequentially by the same worker.
pytest -n8 --dist=loadfile will do the trick. The drawback may be that the whole test suite execution may be slower than using load. The advantage is that all tests will be performed and no test will be skipped.
There may be a case of test that affects some service settings.
This test can not be run in parallel with any other test.
There is a way to skip individual test like this:
#pytest.mark.unparalleled
def test_to_skip(a_fixture):
if worker_id == "master":
pytest.skip('Can't run in parallel with anything')
The first drawback here is that it will be skipped, so you will need to run those test separately. For that matter you can put them in separate folder, or mark with some tag.
The second drawback is that any fixtures used in such will be initialized.
Old question, but xdist has a newer feature that may address the OP's answer.
Per the docs:
--dist loadgroup: Tests are grouped by the xdist_group mark. Groups are distributed to available workers as whole units. This guarantees that all tests with same xdist_group name run in the same worker.
#pytest.mark.xdist_group(name="group1")
def test1():
pass
class TestA:
#pytest.mark.xdist_group("group1")
def test2():
pass
This will make sure test1 and TestA::test2 will run in the same worker. Tests without the xdist_group mark are distributed normally as in the --dist=load mode.
Specifically, you could put the test functions in test_func3.py into the same xdist_group.
Related
I'm developing a test suite using pytest for a project of mine. Because of the nature of the project, I need to create a Pytest plugin that controls how the tests are being run; they are not run locally, but sent to a different process to run. (I know about xdist but I think it doesn't solve my problem.)
I've been writing my own Pytest plugin by overriding the various pytest_runtest_* methods. So far it's been progressing well. Here is where I've hit a wall: I want my implementations of pytest_runtest_setup, pytest_runtest_call and pytest_runtest_teardown to actually be responsible for doing the setup, call and teardown. They're going to do it in a different process. My problem is: After Pytest calls my pytest_runtest_setup, it also calls all the other pytest_runtest_setup down the line of plugins. This is because the hook specification for pytest_runtest_setup has firstresult=False.
I don't want this, because I don't want pytest_runtest_setup to actually run on the current process. I want to be responsible for running it on my own. I want to override how it's being run, not add to it. I want the other implementations of pytest_runtest_setup below my own to not be run.
How can I do this?
Generic “runtest” hooks
All runtest related hooks receive a pytest.Item object.
pytest_runtest_protocol(item, nextitem)[source]
implements the runtest_setup/call/teardown protocol for the given test item, including capturing exceptions and calling reporting hooks.
Parameters:
item – test item for which the runtest protocol is performed.
nextitem – the scheduled-to-be-next test item (or None if this is the end my friend). This argument is passed on to pytest_runtest_teardown().
Return boolean:
True if no further hook implementations should be invoked.
pytest_runtest_setup(item)[source]
called before pytest_runtest_call(item).
pytest_runtest_call(item)[source]
called to execute the test item.
pytest_runtest_teardown(item, nextitem)[source]
called after pytest_runtest_call.
Parameters: nextitem – the scheduled-to-be-next test item (None if no further test item is scheduled). This argument can be used to perform exact teardowns, i.e. calling just enough finalizers so that nextitem only needs to call setup-functions.
pytest_runtest_makereport(item, call)[source]
return a _pytest.runner.TestReport object for the given pytest.Item and _pytest.runner.CallInfo.
For deeper understanding you may look at the default implementation of these hooks in _pytest.runner and maybe also in _pytest.pdb which interacts with _pytest.capture and its input/output capturing in order to immediately drop into interactive debugging when a test failure occurs.
The _pytest.terminal reported specifically uses the reporting hook to print information about a test run.
I'm using pytest to run my tests, and testing my web application. My test file looks like
def test_logins():
# do stuff
def test_signups():
# do stuff
def testing_posting():
# do stuff
There are about 20 of them, and many of them have elements that run in constant time or rely on external HTTP requests, so it seems like it would lead to a large increase in testing speed if I could get pytest to start up 20 different mutliprocessing processes (one for each test) to run each testing function. Is this possible / reasonable / recommended?
I looked into xdist but splitting the tests so that they ran based on the amount of cores on my computer isn't what I want.
Also in case it's relevant, the bulk of the tests are done using python's requests library (although they will be moved to selenium eventually)
I would still recommend using pytest-xdist. And, as you mentioned already because your tests mostly do network IO, it's ok to start pytest with (much) more parallel processes than you have cores (like 20), it will be still beneficial, as GIL will not be preventing the speedup from the parallelization.
So you run it like:
py.test tests -n<number>
The additional benefit of xdist is that you can easily scale your test run to multiple machines with no effort.
For easier scaling among multiple machines, pytest-cloud can help a lot.
I've got a large number of tests written using Python unittest (Python 2.7.8) as a large TestSuite. Many of these tests invoke other programs. Sometimes these other programs dump core. When they do, I want to discover that and ensure the test fails. If some number of cores are dumped, I want to abort the entire test environment and exit rather than continuing: my total test suite has >6000 tests and if everything is dumping core it's useless (and dangerous: disk space etc.) to continue.
In order to ensure we look for coredumps after every test (so I have the best possible idea of what program/invocation dumped core) I decided to look for cores in tearDown(), which I am doing successfully. If I find a core, I can run an assert variant in tearDown() to specify that the test failed.
But I can't figure out how to just give up on all my testing completely from within tearDown() if I find too many cores. I even tried to run sys.exit("too many cores"), but unittest case.py catches every exception thrown by tearDown() except KeyboardInterrupt (if I try to raise that by hand my script hangs until I do a real ^C).
I thought about trying to call stop(), but this is a method on the result and I can't find any way to get access to the result object from within tearDown() (!).
So far my only option seems to be to invoke os._exit() which is really annoying because it keeps any results from being reported at all!
Is there really no facility in Python unittest.TestCase to tell the test environment to just stop right now, generate what results you have but don't run anything else?
Can you check how many cores have been dumped in setUp()? If so, you could just call self.skipTest('Too many cores dumped.') when things get bad.
If you don't want to look for cores in setUp(), you could probably use a class variable to hold the core dump count and check that instead.
While the unittesting philosophy is that tests can be ran in any order and it should pass, what if you're implementing an API where there is no other means of communicating with a server... and you need to test a certain very basic feature (such as delete) before you can do more complicated tasks? Is ordering the tests then reasonable?
If so, how can I do it with python's unittest module?
You already seem to realise that your unit tests should be independent. The only other reason I can see that you want to run the tests in some fixed order is that you want to stop running the suite if an early test fails. To do that, you can use the command-line option
-f, --failfast
Stop the test run on the first error or failure.
By the way, the tests are run in alphabetical order:
the order in which the various test cases will be run is determined by sorting the test function names with respect to the built-in ordering for strings.
(docs)
This question already has answers here:
Can Python's unittest test in parallel, like nose can?
(7 answers)
Closed 6 years ago.
I'm using python unittest in order to test some other external application but it takes too much time to run the test one by one.
I would like to know how can I speedup this process by using the power of multi-cores.
Can I tweak unittest to execute tests in parallel? How?
This question is not able python GIL limitation because in fact not the python code takes time but the external application that I execute, currently via os.system().
If your tests are not too involved, you may be able to run them using py.test which has support for distributed testing. If you are not running on Windows, then nose might also work for you.
The testtools package is an extension of unittest which supports running tests concurrently. It can be used with your old test classes that inherit unittest.TestCase.
For example:
import unittest
import testtools
class MyTester(unittest.TestCase):
# Tests...
suite = unittest.TestLoader().loadTestsFromTestCase(MyTester)
concurrent_suite = testtools.ConcurrentStreamTestSuite(lambda: ((case, None) for case in suite))
concurrent_suite.run(testtools.StreamResult())
Maybe you can run each test on a different process using the multiprocessing library. This implies that each unit test (or group of unit tests) should be independent and doesn't need to share the state.
It will open other processes, and will make use of other cores.
Check specifically the 'Using a pool of workers' on this page ( http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers)
EDIT: This module is included since version 2.6
As the #vinay-sajip suggested, a few non-core python packages like py.test and nose provided parallel execution of unit tests via multiprocessing lib right out of the box.
However, one thing to consider is that if you are testing a web app with database backend and majority of your test cases are relying on connecting to the same test database, then your unit test execution speed is bottlenecked on the DB not I/O per se. And using multiprocess won't speed it up.
Given that each unit test case requires an independent setup of the database schema + data, you cannot scale out the execution speed only on CPU but restricted with a single test database connection to a single test database server (otherwise the state of the data may interfere with other other while parallel executing each test case so on and so forth).