I want to do tests with randomized parameters of a class with a very slow init method. The tests themself are very quick, but require a time consuming initialization step.
Of course. I do something like this:
#pytest.mark.parametrize("params", LIST_OF_RANDOMIZED_PARAMS)
def test_one(params):
state = very_slow_initialization(params)
assert state.fast_test()
#pytest.mark.parametrize("params", LIST_OF_RANDOMIZED_PARAMS)
def test_two(params):
state = very_slow_initialization(params)
assert state.another_fast_test()
From my unsuccessful tries so far I've learnt:
initializing a Testclass with a parametrized set_class(params) method is not supported
Using a fixture that initialized the class still calls the slow initialization every time
I could create a list with all initialized states in advance, however they demand a lot of memory. Furthermore sometimes I like to rune a lot of randomized tests overnight and just stop them the next morning. This this I would need to know precisely how many tests I should to so that all initializations are finished before that.
If possible I would prefer a solution that runs both tests for the first parameter, then runs both with the second parameter and so on.
There is probably a really simple solution for this.
pytest fixtures is a solution for you. Lifetime of fixture might be a single test, class, module or whole test session.
fixture management scales from simple unit to complex functional testing, allowing to parametrize fixtures and tests according to configuration and component options, or to re-use fixtures across function, class, module or whole test session scopes.
Per Fixture availability paragraph, you need to define feature in class, or on module level.
Consider using module-scoped ones (pay attention, that initialization launched only once):
import pytest
#pytest.fixture(scope="module")
def heavy_context():
# Use your LIST_OF_RANDOMIZED_PARAMS randomized parameters here
# to initialize whatever you want.
print("Slow fixture initialized")
return ["I'm heavy"]
def test_1(heavy_context):
print(f"\nUse of heavy context: {heavy_context[0]}")
def test_2(heavy_context):
print(f"\nUse of heavy context: {heavy_context[0]}")
Tests output:
...
collecting ... collected 2 items
test_basic.py::test_1 Slow fixture initialized
PASSED [ 50%]
Use of heavy context: I'm heavy
test_basic.py::test_2 PASSED [100%]
Use of heavy context: I'm heavy
Now, if you need it to be assertion safe (release resources even when test fails), consider creating heavy_context in a context-manager manner (much more details here: Fixture, Running multiple assert statements safely):
import pytest
#pytest.fixture(scope="module")
def heavy_context():
print("Slow context initialized")
obj = ["I'm heavy"]
# It is mandatory to put deinitialiation into "finally" scope
# otherwise in case of exception it won't be executed
try:
yield obj[0]
finally:
print("Slow context released")
def test_1(heavy_context):
# Pay attention, that in fact heavy_context now
# is what we initialized as 'obj' in heavy_context
# function.
print(f"\nUse of heavy context: {heavy_context}")
def test_2(heavy_context):
print(f"\nUse of heavy context: {heavy_context}")
Output:
collecting ... collected 2 items
test_basic.py::test_1 Slow context initialized
PASSED [ 50%]
Use of heavy context: I'm heavy
test_basic.py::test_2 PASSED [100%]
Use of heavy context: I'm heavy
Slow context released
============================== 2 passed in 0.01s ===============================
Process finished with exit code 0
Could you perhaps run the tests one after another without initializing the object again, e.g.:
#pytest.mark.parametrize("params", LIST_OF_RANDOMIZED_PARAMS)
def test_one(params):
state = very_slow_initialization(params)
assert state.fast_test()
assert state.another_fast_test()
or using separate functions for organization:
#pytest.mark.parametrize("params", LIST_OF_RANDOMIZED_PARAMS)
def test_main(params):
state = very_slow_initialization(params)
step_one(state)
step_two(state)
def step_one(state):
assert state.fast_test()
def step_two(state):
assert state.another_fast_test()
Although it's a test script, you can still use functions to organize your code. In the version with separate functions you may even declare a fixture, in case the state may be needed in other tests, too:
#pytest.fixture(scope="module", params=LIST_OF_RANDOMIZED_PARAMS)
def state(request):
return very_slow_initialization(request.param)
def test_main(state):
step_one(state)
step_two(state)
def step_one(state):
assert state.fast_test()
def step_two(state):
assert state.another_fast_test()
I hope I didn't do a mistake here, but it should work like this.
Related
I have a test framework that is performing multiple asserts and catching them (inherited someone else's code). The proprietary results report is correct, but as you may have guessed, pytest will mark these as passed:
test_something.py::TestSomething::test_that_should_fail PASSED
I've added an autouse fixture as such:
#pytest.fixture(autouse=True)
def run_after_tests(self):
yield
if self.actually_failed():
pytest.fail("Yay, failing when failures occur is cool!")
This solution works okay, except that it seems like the clean up happens after the test has already been marked as PASSED and a duplicate test is shown with an error.
Now pytest results look like this:
test_something.py::TestSomething::test_that_should_fail PASSED
test_something.py::TestSomething::test_that_should_fail ERROR
Is there a way to delay the evaluation of the test so it doesn't say it has passed?
I know this is a really stupid way of doing things and performing test evaluation at cleanup is not recommended, but there are too many tests that have been written this way and spending weeks to refactor the test is not feasible.
An alternative solution I've thought of is to write a decorator and then sed all the test cases and add it to the functions; but this is going to be my plan B if fixtures can't solve this.
Thanks!
As you've discovered, a fixture is a separate object from the test itself.
You'll need to modify the appropriate pytest hook. I haven't personally tested, but I believe placing the following code into your projects conftest.py will give you your desired result.
def check_for_failure(output) -> bool:
# define me
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
output = yield
if check_for_failure(output):
pytest.fail()
I have a couple of fixtures that do some initialization that is rather expensive. Some of those fixtures can take parameters, altering their behaviour slightly.
Because these are so expensive, I wanted to do initialisation of them once per test class. However, it does not destroy and reinit the fixtures on the next permutation of parameters.
See this example: https://gist.github.com/vhdirk/3d7bd632c8433eaaa481555a149168c2
I would expect that StuffStub would be a different instance when DBStub is recreated for parameters 'foo' and 'bar'.
Did I misunderstand something? Is this a bug?
I've recently encountered the same problem and wanted to share another solution. In my case the graph of fixtures that required regenerating for each parameter set was very deep and it's not so easy to control. An alternative is to bypass the pytest parametrization system and programmatically generate the test classes like so:
import pytest
import random
def make_test_class(name):
class TestFoo:
#pytest.fixture(scope="class")
def random_int(self):
return random.randint(1, 100)
def test_someting(self, random_int):
assert random_int and name == "foo"
return TestFoo
TestFooGood = make_test_class("foo")
TestFooBad = make_test_class("bar")
TestFooBad2 = make_test_class("wibble")
You can see from this that three tests are run, one passes (where "foo" == "foo") the other two fail, but you can see that the class scope fixtures have been recreated.
This is not a bug. There is no relation between the fixtures so one of them is not going to get called again just because the other one was due to having multiple params.
In your case db is called twice because db_factory that it uses has 2 params. The stuff fixture on the other hand is called only once because stuff_factory has only one item in params.
You should get what you expect if stuff would include db_factory as well without actually using its output (db_factory would not be called more than twice):
#pytest.fixture(scope="class")
def stuff(stuff_factory, db_factory):
return stuff_factory()
When parametrizing tests and fixtures in pytest, pytest seem to eagerly evaluate all parameters and to construct some test list datastructure before starting to execute the tests.
This is a problem in 2 situations:
when you have many parameter values (e.g. from a generator) - the generator and test itself may run fast but all those parameter values eat up all the memory
when parametrizing a fixture with different kind of expensive resources, where you only can afford to run one resource at the same time (e.g. because they listen on the same port or something like that)
Thus my question: Is it possibly to tell pytest to evaluate the parameters on the fly (i.e. lazily)?
EDIT: my first reaction would be "that is exactly what parametrized fixtures are for": a function-scoped fixture is a lazy value being called just before the test node is executed, and by parametrizing the fixture you can predefine as many variants (for example from a database key listing) as you like.
from pytest_cases import fixture_plus
#fixture_plus
def db():
return <todo>
#fixture_plus
#pytest.mark.parametrize("key", [<list_of keys>])
def sample(db, key):
return db.get(key)
def test_foo(sample):
return sample
That being said, in some (rare) situations you still need lazy values in a parametrize function, and you do not wish these to be the variants of a parametrized fixture. For those situations, there is now a solution also in pytest-cases, with lazy_value. With it, you can use functions in the parameter values, and these functions get called only when the test at hand is executed.
Here is an example showing two coding styles (switch the use_partial boolean arg to True to enable the other alternative)
from functools import partial
from random import random
import pytest
from pytest_cases import lazy_value
database = [random() for i in range(10)]
def get_param(i):
return database[i]
def make_param_getter(i, use_partial=False):
if use_partial:
return partial(get_param, i)
else:
def _get_param():
return database[i]
return _get_param
many_lazy_parameters = (make_param_getter(i) for i in range(10))
#pytest.mark.parametrize('a', [lazy_value(f) for f in many_lazy_parameters])
def test_foo(a):
print(a)
Note that lazy_value also has an id argument if you wish to customize the test ids. The default is to use the function __name__, and a support for partial functions is on the way.
You can parametrize fixtures the same way, but remember that you have to use #fixture_plus instead of #pytest.fixture. See pytest-cases documentation for details.
I'm the author of pytest-cases by the way ;)
As for your 2 question - proposed in comment link to manual seems like exactly what one should do. It allows "to setup expensive resources like DB connections or subprocess only when the actual test is run".
But as for 1 question it seems like such feature not implemented. You may directly pass generator to parametrize like so:
#pytest.mark.parametrize('data', data_gen)
def test_gen(data):
...
But pytest will list() of your generator -> RAM problems persists here as well.
I've also found some github issues than shed more light about why pytest not handle generator lazily. And it seems like a design problem. So "its not possible to correctly manage parametrization having a generator as value" because of
"pytest would have to collect all those tests with all the metadata...
collection happens always before test running".
There are also some refers to hypothesis or nose's yield-base tests in such cases. But if you still want to stick to pytest there are some workarounds:
If you somehow knew the number of generated params you may do the following:
import pytest
def get_data(N):
for i in range(N):
yield list(range(N))
N = 3000
data_gen = get_data(N)
#pytest.mark.parametrize('ind', range(N))
def test_yield(ind):
data = next(data_gen)
assert data
So here you parametrize over index (which is not so useful - just indicating pytest number of executions it must made) and generate data inside next run.
You may also wrap it to memory_profiler:
Results (46.53s):
3000 passed
Filename: run_test.py
Line # Mem usage Increment Line Contents
================================================
5 40.6 MiB 40.6 MiB #profile
6 def to_profile():
7 76.6 MiB 36.1 MiB pytest.main(['test.py'])
And compare with straightforward:
#pytest.mark.parametrize('data', data_gen)
def test_yield(data):
assert data
Which 'eats' much more memory:
Results (48.11s):
3000 passed
Filename: run_test.py
Line # Mem usage Increment Line Contents
================================================
5 40.7 MiB 40.7 MiB #profile
6 def to_profile():
7 409.3 MiB 368.6 MiB pytest.main(['test.py'])
If you want to parametrize your test over another params at the same time you may do a bit generalization of previous clause like so:
data_gen = get_data(N)
#pytest.fixture(scope='module', params=len_of_gen_if_known)
def fix():
huge_data_chunk = next(data_gen)
return huge_data_chunk
#pytest.mark.parametrize('other_param', ['aaa', 'bbb'])
def test_one(fix, other_param):
data = fix
...
So we use fixture here at module scope level in order to "preset" our data for parametrized test. Note that right here you may add another test and it will receive generated data as well. Simply add it after test_two:
#pytest.mark.parametrize('param2', [15, 'asdb', 1j])
def test_two(fix, param2):
data = fix
...
NOTE: if you do not know the number of generated data you may use this trick: set some approximate value (better if it be a bit higher than generated tests count) and 'mark' tests passed if it stops with StopIteration which will happen when all data generated already.
Another possibility is to use Factories as fixtures. Here you embed your generator into fixture and try yield in your test till it not ends. But here is another disadvantage - pytest will treat it as single test (with possibly bunch of checks inside) and will fail if one of generated data fails. Other words if compare to parametrize approach not all pytest statistic/features may be accessed.
And yet one another is to use pytest.main() in the loop something like so:
# data_generate
# set_up test
pytest.main(['test'])
Is not concerning iterators itself rather the way to save more Time/RAM if one has parametrized test:
Simply move some parametrization inside tests. Example:
#pytest.mark.parametrize("one", list_1)
#pytest.mark.parametrize("two", list_2)
def test_maybe_convert_objects(self, one, two):
...
Change to:
#pytest.mark.parametrize("one", list_1)
def test_maybe_convert_objects(self, one):
for two in list_2:
...
It's similar to factories but even more easy to implement. Also it not only reduce RAM multiple times but time for collecting metainfo as well. Drawbacks here - for pytest it would be one test for all two values. And it works smoothly with "simple" tests - if one have some special xmarks inside or something there might be problems.
I've also opened corresponding issue there might appear some additional info/tweaks about this problem.
You may find this workaround useful:
from datetime import datetime, timedelta
from time import sleep
import pytest
#pytest.mark.parametrize(
'lazy_params',
[
lambda: (datetime.now() - timedelta(days=1), datetime.now()),
lambda: (datetime.now(), datetime.now() + timedelta(days=1)),
],
)
def test_it(lazy_params):
yesterday, today = lazy_params()
print(f'\n{yesterday}\n{today}')
sleep(1)
assert yesterday < today
Sample output:
========================================================================= test session starts ==========================================================================
platform darwin -- Python 3.7.7, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/local/opt/python/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/apizarro/tmp
collected 2 items
test_that.py::test_it[<lambda>0]
2020-04-14 18:34:08.700531
2020-04-15 18:34:08.700550
PASSED
test_that.py::test_it[<lambda>1]
2020-04-15 18:34:09.702914
2020-04-16 18:34:09.702919
PASSED
========================================================================== 2 passed in 2.02s ===========================================================================
I need a tearDownClass(cls) method for the instance.
I mean one where I can refer to self (the instance), not to cls (the class).
A sort of tearDownTestCase(self).
My intention is to cleanup the database after ALL the test cases have been run.
tearDown(self) is executed at the end of every test and I don't want use it.
tearDownClass(cls) is executed once when ALL the tests finished but it does not contain a reference to self, and I need to access to a property of self (more precisely to a function).
There is a way to achieve this?
Python 3.6
Example of real scenario:
import unittest
'''
The real records saved in the database came from an external source (an API) so the ID is preassigned.
For the test I use everywhere a predefined fixed id, so the code result more clean.
'''
record_id = "TEST"
class RepositoryTest(unittest.TestCase):
def setUp(self):
# real initialization, reading connection string from config, database name, collection...
self.repository = None
# self._cleanup_record() # maybe here is executed too many unnecessary times
def tearDown(self):
# here is executed unnecessarily, because (where needed) the cleanup is eventually executed BEFORE the test (or in its beginning)
self._cleanup_record()
### pseudo (desired) method ###
def tearDownTestCase(self):
self._cleanup_record()
def tearDownClass(cls):
# self._cleanup_record() # self is not available
# rewrite the same code of initialization and _cleanup_record()
# I want to a void (or simplify this)
pass
# this is 1 of N tests
def test_save_record(self):
# cleanup (because I don't know in which state the database is)
self._cleanup_record() # almost every test require this, so it can be done in setUp()
# arrange
record = self._create_record()
# act
self.repository.save_record(record)
# assert
saved_record = self._get_record()
self.assertEquals(saved_record["my field"], record["my field"])
# utility methods
def _get_record(self):
# use self.repository and return the record with id = record_id
pass # return the record
def _create_record(self):
# use self.repository and create (and save) a record with id = record_id
return None # return the saved record
def _cleanup_record(self):
# use self.repository and delete the record with id = record_id (if exists)
pass
Doing the cleanup in the tearDown() method it results in:
setUp
.test 1
cleanup
test
cleanup (= redundant)
. . .
.test N
cleanup
test
cleanup
Instead I want this:
(and it is possible if a tearDownX() method is executed after ALL the tests are finished)
setUp
(test 1)
cleaup
test
. . .
(test N)
cleaup
test
tearDownX (self)
cleanup (final)
This is more or less how I ended in the design of the tests in the last years.
It try to be bullet proof against interrupted debug sessions (no cleanup) and dirty initial database state.
As a temporary solution I have replicated the cleanup method in the tearDownClass(cls) method, but I'm not happy. Ideally I could simply call the self._cleanup_record but this is not possible because tearDownClass is a class method.
I hope all that makes sense.
Thank you,
Alessandro
Yes, there is a pair of instance methods setUp and tearDown under the unittest.TestCase, executed before and after each test respectively.
From the docs:
setUp()
Method called to prepare the test fixture. This is called
immediately before calling the test method; other than AssertionError
or SkipTest, any exception raised by this method will be considered an
error rather than a test failure. The default implementation does
nothing.
tearDown()
Method called immediately after the test method has been
called and the result recorded. This is called even if the test method
raised an exception, so the implementation in subclasses may need to
be particularly careful about checking internal state. Any exception,
other than AssertionError or SkipTest, raised by this method will be
considered an additional error rather than a test failure (thus
increasing the total number of reported errors). This method will only
be called if the setUp() succeeds, regardless of the outcome of the
test method. The default implementation does nothing.
UPDATE (after comments)
Well, you probably have no choice but to re-design your code. You can make the db cleanup method a class method instead of instance method.
Anyways, since you shouldl't count on the test execution order, nor have your tests dependent on each other, it would be still smart to create the db fixture for each test with the setUp method and clean it with tearDown method after each test.
Another option would be using a mock for the db in your tests, so you won't need to worry about cleaning it up.
How can I find out from the current test if its the last to be run? (Python unittest / nosetests)
I have some specific fixture teardown to be done at the very end of the test run and it would be a lot easier if on a test by test basis I could just determine:
if last_test:
hard_fixture_teardown()
else:
soft_fixture_teardown()
I have a package teardown which would work perfectly but it seems very messy passing the fixture information back to the __init__.teardown_package().
You can use a combination of TestCase.tearDown() and TestCase.tearDownClass() to achieve this. tearDown() is called for each test method while tearDownClass() is called after all tests in the class have run.
As stated here unit test are not meant to have an order, unit tests depending on the order are either not well conceived or just have to be merged in a monolithic one. (merging separate functions in a single test is the accepted answer )
[edit after comment]
If the order is not important you can do this (quite messy, imo we are still forcing the boundaries of how unit tests should be used)
in every test package you put:
def tearDownModule():
xxx = int(os.getenv('XXX', '0')) + 1
if xxx == NUMBER_OF_TEST_PACKAGES:
print "hard tear down"
else:
print "not yet"
os.environ['XXX'] = str(xxx)
with NUMBER_OF_TEST_PACKAGES imported from somewhere global.
also if the order is not important I suppose that when used the fixture is only readed and not modified, if so you can setup that as a class method
#classmethod
def setUpClass(cls):
print "I'll take a lot of time here"