This question already has answers here:
Can Python's unittest test in parallel, like nose can?
(7 answers)
Closed 6 years ago.
I'm using python unittest in order to test some other external application but it takes too much time to run the test one by one.
I would like to know how can I speedup this process by using the power of multi-cores.
Can I tweak unittest to execute tests in parallel? How?
This question is not able python GIL limitation because in fact not the python code takes time but the external application that I execute, currently via os.system().
If your tests are not too involved, you may be able to run them using py.test which has support for distributed testing. If you are not running on Windows, then nose might also work for you.
The testtools package is an extension of unittest which supports running tests concurrently. It can be used with your old test classes that inherit unittest.TestCase.
For example:
import unittest
import testtools
class MyTester(unittest.TestCase):
# Tests...
suite = unittest.TestLoader().loadTestsFromTestCase(MyTester)
concurrent_suite = testtools.ConcurrentStreamTestSuite(lambda: ((case, None) for case in suite))
concurrent_suite.run(testtools.StreamResult())
Maybe you can run each test on a different process using the multiprocessing library. This implies that each unit test (or group of unit tests) should be independent and doesn't need to share the state.
It will open other processes, and will make use of other cores.
Check specifically the 'Using a pool of workers' on this page ( http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers)
EDIT: This module is included since version 2.6
As the #vinay-sajip suggested, a few non-core python packages like py.test and nose provided parallel execution of unit tests via multiprocessing lib right out of the box.
However, one thing to consider is that if you are testing a web app with database backend and majority of your test cases are relying on connecting to the same test database, then your unit test execution speed is bottlenecked on the DB not I/O per se. And using multiprocess won't speed it up.
Given that each unit test case requires an independent setup of the database schema + data, you cannot scale out the execution speed only on CPU but restricted with a single test database connection to a single test database server (otherwise the state of the data may interfere with other other while parallel executing each test case so on and so forth).
Related
I have a number of test files, such as
test_func1.py
test_func2.py
test_func3.py
I know in advance that test_func3.py won't pass if I run Pytest in parallel, e.g. pytest -n8. The reason is that test_func3.py contains a number of parametrized tests that handle file i/o processes. Parallel writing to the same file leads to failures. In serial testing mode, all tests in this module passes.
I'm wondering how I can skip the whole module in case Pytest will be started with the option -n? My thinking is to apply the skipif marker. I need to check in my code whether the -n argument has been passed to pytest.
...>pytest # run all modules
...>pytest -n8 # skip module test_func3.py automatically
The pytest-xdist package supports four scheduling algorithms:
each
load
loadscope
loadfile
Calling pytest -n is a shortcut for load scheduling, i.e. the scheduler will load balance the tests across all workers.
Using loadfile scheduling, all test cases in a test file will be executed sequentially by the same worker.
pytest -n8 --dist=loadfile will do the trick. The drawback may be that the whole test suite execution may be slower than using load. The advantage is that all tests will be performed and no test will be skipped.
There may be a case of test that affects some service settings.
This test can not be run in parallel with any other test.
There is a way to skip individual test like this:
#pytest.mark.unparalleled
def test_to_skip(a_fixture):
if worker_id == "master":
pytest.skip('Can't run in parallel with anything')
The first drawback here is that it will be skipped, so you will need to run those test separately. For that matter you can put them in separate folder, or mark with some tag.
The second drawback is that any fixtures used in such will be initialized.
Old question, but xdist has a newer feature that may address the OP's answer.
Per the docs:
--dist loadgroup: Tests are grouped by the xdist_group mark. Groups are distributed to available workers as whole units. This guarantees that all tests with same xdist_group name run in the same worker.
#pytest.mark.xdist_group(name="group1")
def test1():
pass
class TestA:
#pytest.mark.xdist_group("group1")
def test2():
pass
This will make sure test1 and TestA::test2 will run in the same worker. Tests without the xdist_group mark are distributed normally as in the --dist=load mode.
Specifically, you could put the test functions in test_func3.py into the same xdist_group.
I'm using pytest to run my tests, and testing my web application. My test file looks like
def test_logins():
# do stuff
def test_signups():
# do stuff
def testing_posting():
# do stuff
There are about 20 of them, and many of them have elements that run in constant time or rely on external HTTP requests, so it seems like it would lead to a large increase in testing speed if I could get pytest to start up 20 different mutliprocessing processes (one for each test) to run each testing function. Is this possible / reasonable / recommended?
I looked into xdist but splitting the tests so that they ran based on the amount of cores on my computer isn't what I want.
Also in case it's relevant, the bulk of the tests are done using python's requests library (although they will be moved to selenium eventually)
I would still recommend using pytest-xdist. And, as you mentioned already because your tests mostly do network IO, it's ok to start pytest with (much) more parallel processes than you have cores (like 20), it will be still beneficial, as GIL will not be preventing the speedup from the parallelization.
So you run it like:
py.test tests -n<number>
The additional benefit of xdist is that you can easily scale your test run to multiple machines with no effort.
For easier scaling among multiple machines, pytest-cloud can help a lot.
I have a few functions for performing actions in python, and I am debating whether or not it makes sense to write unit tests for these.
There is one function that resolves a host using the socket module. I don't think it makes sense to write a test for the socket module itself, but seems like it may make sense to write a functional test to see if DNS is working.
I have another function that opens a file and puts the contents into a list.
Does anyone have any guidance or links on performing unit tests/functional tests for items like this? Most of what I have read pertains to application level testing only.
Thank you in advance.
First of all, if you don't have tests at all, better start with high-level end-to-end functional tests and end with unit tests gathering coverage statistics on every new test you write.
While writing a test to a function which uses system or network libraries, you usually want to isolate your test, to make it independent and straight as much as possible by mocking out system and network calls (see Mock library).
By using mock library you can and should test how does your application/function respond to situations where there is a socket or system error.
Also see:
python unittest howto
How to Mock an HTTP request in a unit testing scenario in Python
An Introduction to Mocking in Python
Hope that helps.
I am using py.test (version 2.4, on Windows 7) with xdist to run a number of numerical regression and interface tests for a C++ library that provides a Python interface through a C module.
The number of tests has grown to ~2,000 over time, but we are running into some memory issues now. Whether using xdist or not, the memory usage of the python process running the tests seems to be ever increasing.
In single-process mode we have even seen a few issues of bad allocation errors, whereas with xdist total memory usage may bring down the OS (8 processes, each using >1GB towards the end).
Is this expected behaviour? Or did somebody else experience the same issue when using py.test for a large number of tests? Is there something I can do in tearDown(Class) to reduce the memory usage over time?
At the moment I cannot exclude the possibility of the problem lying somewhere inside the C/C++ code, but when running some long-running program using that code through the Python interface outside of py.test, I do see relatively constant memory usage over time. I also do not see any excessive memory usage when using nose instead of py.test (we are using py.test as we need junit-xml reporting to work with multiple processes)
py.test's memory usage will grow with the number of tests. Each test is collected before they are executed and for each test run a test report is stored in memory, which will be much larger for failures, so that all the information can be reported at the end. So to some extend this is expected and normal.
However I have no hard numbers and have never closely investigated this. We did run out of memory on some CI hosts ourselves before but just gave them more memory to solve it instead of investigating. Currently our CI hosts have 2G of mem and run about 3500 tests in one test run, it would probably work on half of that but might involve more swapping. Pypy is also a project that manages to run a huge test suite with py.test so this should certainly be possible.
If you suspect the C code to leak memory I recommend building a (small) test script which just tests the extension module API (with or without py.test) and invoke that in an infinite loop while gathering memory stats after every loop. After a few loops the memory should never increase anymore.
Try using --tb=no which should prevent pytest from accumulating stacks on every failure.
I have found that it's better to have your test runner run smaller instances of pytest in multiple processes, rather than one big pytest run, because of it's accumulation in memory of every error.
pytest should probably accumulate test results on-disk, rather than in ram.
We also experience similar problems. In our case we run about ~4600 test cases.
We use extensively pytest fixtures and we managed to save the few MB by scoping the fixtures slightly differently (scoping several from "session" to "class" of "function"). However we dropped in test performances.
I wrote an application server (using python & twisted) and I want to start writing some tests. But I do not want to use Twisted's Trial due to time constraints and not having time to play with it now. So here is what I have in mind: write a small test client that connects to the app server and makes the necessary requests (the communication protocol is some in-house XML), store in a static way the received XML and then write some tests on those static data using unitest.
My question is: Is this a correct approach and if yes, what kind of tests are covered with this approach?
Also, using this method has several disadvantages, like: not being able to access the database layer in order to build/rebuild the schema, when will the test client going to connect to the server: per each unit test or before running the test suite?
You should use Trial. It really isn't very hard. Trial's documentation could stand to be improved, but if you know how to use the standard library unit test, the only difference is that instead of writing
import unittest
you should write
from twisted.trial import unittest
... and then you can return Deferreds from your test_ methods. Pretty much everything else is the same.
The one other difference is that instead of building a giant test object at the bottom of your module and then running
python your/test_module.py
you can simply define your test cases and then run
trial your.test_module
If you don't care about reactor integration at all, in fact, you can just run trial on a set of existing Python unit tests. Trial supports the standard library 'unittest' module.
"My question is: Is this a correct approach?"
It's what you chose. You made a lot of excuses, so I'm assuming that your pretty well fixed on this course. It's not the best, but you've already listed all your reasons for doing it (and then asked follow-up questions on this specific course of action). "correct" doesn't enter into it anymore, so there's no answer to this question.
"what kind of tests are covered with this approach?"
They call it "black-box" testing. The application server is a black box that has a few inputs and outputs, and you can't test any of it's internals. It's considered one acceptable form of testing because it tests the bottom-line external interfaces for acceptable behavior.
If you have problems, it turns out to be useless for doing diagnostic work. You'll find that you need to also to white-box testing on the internal structures.
"not being able to access the database layer in order to build/rebuild the schema,"
Why not? This is Python. Write a separate tool that imports that layer and does database builds.
"when will the test client going to connect to the server: per each unit test or before running the test suite?"
Depends on the intent of the test. Depends on your use cases. What happens in the "real world" with your actual intended clients?
You'll want to test client-like behavior, making connections the way clients make connections.
Also, you'll want to test abnormal behavior, like clients dropping connections or doing things out of order, or unconnected.
I think you chose the wrong direction. It's true that the Trial docs is very light. But Trial is base on unittest and only add some stuff to deal with the reactor loop and the asynchronous calls (it's not easy to write tests that deal with deffers). All your tests that are not including deffer/asynchronous call will be exactly like normal unittest.
The Trial command is a test runner (a bit like nose), so you don't have to write test suites for your tests. You will save time with it. On top of that, the Trial command can output profiling and coverage information. Just do Trial -h for more info.
But in any way the first thing you should ask yourself is which kind of tests do you need the most, unit tests, integration tests or system tests (black-box). It's possible to do all with Trial but it's not necessary allways the best fit.
haven't used twisted before, and the twisted/trial documentation isn't stellar from what I just saw, but it'll likely take you 2-3 days to implement correctly the test system you describe above. Now, like I said I have no idea about Trial, but I GUESS you could probably get it working in 1-2 days, since you already have a Twisted application. Now if Trial gives you more coverage in less time, I'd go with Trial.
But remember this is just an answer from a very cursory look at the docs