I'm looking to write unit tests for some code that uses an object's __subclasses__() method. Ultimately, I was trying to use __subclassess__() to keep track of classes dynamically imported into my application through a namespace package.
In some cases, my code base has test classes created within a pytest file. My problem is that when I run pytest in bulk on my source directory I find failures or errors in my code due to import polutions from these one-off test classes. This is because the pytest run maintains all of the imports as it runs through the source tree. On their own the tests pass fine, but in a sequence they fail, sometimes depending on the order in which they ran.
In my current code branch these __subclasses__() invocations are in the application code, but I have moved them out to tests here to demonstrate with a MVE:
In my/example.py
class MyClass(object):
def __init__(self):
pass
class MySubClass(MyClass):
def __init__(self):
super().__init__()
In my/test_subclass.py
from my.example import MyClass
class TestSubClass(MyClass):
def __init__(self):
super().__init__()
def test_TestSubClass():
assert issubclass(TestSubClass, MyClass)
In my/test_subclasses.py
from my.example import MySubClass, MyClass
def test_find_subclasses():
assert all([cls == MySubClass for cls in MyClass.__subclasses__()])
The result, when run on my machine, is that the test_find_subclasses() test fails due to the discovery of the TestSubClass when running after test_subclass.py:
def test_find_subclasses():
> assert all([cls == MySubClass for cls in MyClass.__subclasses__()])
E assert False
E + where False = all([True, False])
What is the best way to maintain a "clean" state during sequenced pytest runs so that I can avoid mangling imports?
As discussed in the comments, you probably don't want to hard-code the types that may extend MyClass, since you really can't predict what your application will need in the future. If you want to check subclassing, just check that it works at all:
def test_find_subclasses():
assert MySubClass in MyClass.__subclasses__()
Even more concisely, you could simply do
def test_find_subclasses():
assert issubclass(MySubClass, MyClass)
That being said, you can technically filter the classes you are looking through. In your particular case, you have a distinctive naming convention, so you can do something like
def only_production_classes(iterable):
return [cls for cls in iterable if not cls.__name__.lower().startswith('test')]
def test_find_subclasses():
assert all([cls == MySubClass for cls in only_production_classes(MyClass.__subclasses__())])
You can define only_production_classes using other criteria, like the module that the class appears in:
def only_production_classes(iterable):
return [cls for cls in iterable if not cls.__module__.lower().startswith('test_')]
You can't easily unlink class objects that have been loaded, so your idea of a "clean" test environment is not quite feasible. But you do have the option of filtering the data that you work with based on where it was imported from.
Related
My automation framework uses pytest setup/teardown type of testing instead of fixtures. I also have several levels of classes:
BaseClass - highest, all tests inhriet from it
FeatureClass - medium, all tests related to the programs feature inherit from it
TestClass - hold the actual tests
edit, for examples sake, i change the DB calls to a simple print
I want to add DB report in all setup/teardowns. i.e. i want that the general BaseClass setup_method will create a DB entry for the test and teardown_method will alter the entry with the results. i have tried but i can't seem to get out of method the values of currently running test during run time. is it possible even? and if not, how could i do it otherwise?
samples:
(in base.py)
class Base(object):
test_number = 0
def setup_method(self, method):
Base.test_number += 1
self.logger.info(color.Blue("STARTING TEST"))
self.logger.info(color.Blue("Current Test: {}".format(method.__name__)))
self.logger.info(color.Blue("Test Number: {}".format(self.test_number)))
# --->here i'd like to do something with the actual test parameters<---
self.logger.info("print parameters here")
def teardown_method(self, method):
self.logger.info(color.Blue("Current Test: {}".format(method.__name__)))
self.logger.info(color.Blue("Test Number: {}".format(self.test_number)))
self.logger.info(color.Blue("END OF TEST"))
(in my_feature.py)
class MyFeature(base.Base):
def setup_method(self, method):
# enable this feature in program
return True
(in test_my_feature.py)
class TestClass(my_feature.MyFeature):
#pytest.mark.parametrize("fragment_length", [1,5,10])
def test_my_first_test(self):
# do stuff that is changed based on fragment_length
assert verify_stuff(fragment_length)
so how can i get the parameters in setup_method, of the basic parent class of the testing framework?
The brief answer: NO, you cannot do this. And YES, you can work around it.
A bit longer: these unittest-style setups & teardowns are done only for compatibility with the unittest-style tests. They do not support the pytest's fixture, which make pytest nice.
Due to this, neither pytest nor pytest's unittest plugin provide the context for these setup/teardown methods. If you would have a request, function or some other contextual objects, you could get the fixture's values dynamically via request.getfuncargvalue('my_fixture_name').
However, all you have is self/cls, and method as the test method object itself (i.e. not the pytest's node).
If you look inside of the _pytest/unittest.py plugin, you will find this code:
class TestCaseFunction(Function):
_excinfo = None
def setup(self):
self._testcase = self.parent.obj(self.name)
self._fix_unittest_skip_decorator()
self._obj = getattr(self._testcase, self.name)
if hasattr(self._testcase, 'setup_method'):
self._testcase.setup_method(self._obj)
if hasattr(self, "_request"):
self._request._fillfixtures()
First, note that the setup_method() is called fully isolated from the pytest's object (e.g. self as the test node).
Second, note that the fixtures are prepared after the setup_method() is called. So even if you could access them, they will not be ready.
So, generally, you cannot do this without some trickery.
For the trickery, you have to define a pytest hook/hookwrapper once, and remember the pytest node being executed:
conftest.py or any other plugin:
import pytest
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item, nextitem):
item.cls._item = item
yield
test_me.py:
import pytest
class Base(object):
def setup_method(self, method):
length = self._item.callspec.getparam('fragment_length')
print(length)
class MyFeature(Base):
def setup_method(self, method):
super().setup_method(method)
class TestClass(MyFeature):
#pytest.mark.parametrize("fragment_length", [1,5,10])
def test_my_first_test(self, fragment_length):
# do stuff that is changed based on fragment_length
assert True # verify_stuff(fragment_length)
Also note that MyFeature.setup_method() must call the parent's super(...).setup_method() for obvious reasons.
The cls._item will be set on each callspec (i.e. each function call with each parameter). You can also put the item or the specific parameters into some other global state, if you wish.
Also be carefull not to save the field in the item.instance. The instance of the class will be created later, and you have to use setup_instance/teardown_instance method for that. Otherwise, the saved instance's field is not preserved and is not available as self._item in setup_method().
Here is the execution:
============ test session starts ============
......
collected 3 items
test_me.py::TestClass::test_my_first_test[1] 1
PASSED
test_me.py::TestClass::test_my_first_test[5] 5
PASSED
test_me.py::TestClass::test_my_first_test[10] 10
PASSED
============ 3 passed in 0.04 seconds ============
Problem
Yes I know you shouldn't send arguments to a unittest but in this case I have to so that it can access a piece of hardware within the current framework. For reasons for which the detail is irrelevant, I can't use a simple board = myBoard.getBoard() method within the tests, otherwise I wouldn't be here asking you all.
Horrible Attempt 1
class Test(object):
#staticmethod
def __call__(board):
_board = board
global _board
logging.info('Entering a Unit Test')
suite = unittest.TestLoader()
suite = suite.loadTestsFromTestCase(TestCase)
unittest.TextTestRunner().run(suite)
This method will be called upon to perform a unittest at somepoint. This is the way things are set out in the current framework, I'm just following the rules.
You can see here I assign a global variable (horrible). This can then be accessed by the test methods easily:
class TestCase(unittest.TestCase):
def runTest(self):
logging.critical('running')
def setUp(self):
logging.critical('setup')
def test_Me(self):
logging.critical('me')
The result is it won't enter runTest but it will setUp before each test_ method. Pretty much what I'd like, I can deal with it not entering runTest
Horrible Attempt 2
I'd like to do the above but without global variables so I tried this method:
class Test(object):
#staticmethod
def __call__(board):
logging.info('Entering a basic Unit Test')
suite = unittest.TestSuite() #creates a suite for testing
suite.addTest(BasicCase(board))
unittest.TextTestRunner().run(suite)
class BasicCase(unittest.TestCase):
def __init__(self, board):
self.board = board
logging.critical('init')
super(BasicCase, self).__init__()
def runTest(self):
logging.critical('runtest')
def setUp(self):
logging.critical('reset')
infra.Reset.__call__(self.board) #allows me to pass the board in a much nicer way
def test_Connect(self):
logging.critical('connection')
Problem with the above is that it will obviously only do init, setUp and runTest but it'll miss any test_ methods.
If I add:
suite.addTest(BasicCase(board).test_Connect)
Then it unnecessarily calls init twice and I have to add an extra member variable to test_Connect to accept the test object. I can deal with this too as I can return a list of the test_ methods but it just seems like a pain to do. Also I don't think it runs setUp every time it enters a new test_ method. Not ideal.
So ...
So is there some other method I can use that allows me to quickly run everything like a unittest but allows me to pass the board object?
If I should be using a different python module, then please say but I'm using unittest due to the lovely outputs and assert methods I get.
I've been fighting with nose and fixtures today and can't help but feel I'm doing something wrong.
I had all my tests written as functions and everything was working OK, but I want to move them to test classes as I think it's tidier and the tests can then be inherited from.
I create a connection to the testdb, the transaction to use, and a number of fixtures at a package level for all tests. This connection is used by repository classes in the application to load test data from the DB.
I tried to create my test classes with the class they are to test as an attribute. This was set in the __init__ function of the test class.
The problem I had is the class I was testing needs to be instantiated with data from my test DB, but Nose creates an instance of the test class before the package setup function is run. This mean that there are no fixtures when the test classes are created and the tests fail.
The only way I can get it to work is by adding a module setup function that defines the class I want to test as a global, then use that global in the test methods, but that seems quite hacky. Can anyone suggest a better way?
That isn't very clear, so here is an example of my test module...
from nose.tools import *
from myproject import repositories
from myproject import MyClass
def setup():
global class_to_test
db_data_repo = repositories.DBDataRepository()
db_data = db_data_repo.find(1)
class_to_test = MyClass(db_data)
class TestMyClass(object):
def test_result_should_be_an_float(self):
result = class_to_test.do_something()
assert isinstance(result, float)
If you know a better way to achieve this then please share :) thanks!
If you are moving toward unittest.TestCase class based testing, you might as well use setUp()/tearDown() method that will run before each test method, and setUpClass() to setup up class specific things, and nose will handle it just as unittest will, something like this:
class TestMyClass(unittest.TestCase):
#classmethod
def setUpClass(cls):
db_data_repo = repositories.DBDataRepository()
db_data = db_data_repo.find(1)
cls.class_to_test = MyClass(db_data)
def test_result_should_be_an_float(self):
result = self.class_to_test.do_something()
assert isinstance(result, float)
Not sure if this is a dupe or not. Here it goes.
I need to write some Python code that looks like:
class TestClass:
def test_case(self):
def get_categories(self):
return [“abc”,”bcd”]
# do the test here
and then have a test engine class that scans all these test classes, loads all the test_case functions and for each invokes get_categories to find out if the test belongs t the group of interest for the specific run.
The problem is that get_categories is not seen as an attribute of test_case, and even if I manually assign it
class TestClass:
def test_case(self):
def get_categories(self):
return [“abc”,”bcd”]
# do the test here
test_case.get_categories = get_categories
this is only going to happen when test_case first runs, too late for me.
The reason why this function can’t go on the class (or at least why I want it to be also available at the per-function level) is that a TestClass can have multiple test cases.
Since this is an already existing testing infrastructure, and the categories mechanism works (other than the categories-on-function scenario, which is of lesser importance), a rewrite is not in the plans.
Language tricks dearly appreciated.
Nested functions don't become attributes any more than any other assignment.
I suspect your test infrastructure is doing some severely weird things if this isn't supported (and uses old-style classes!), but you could just do this:
class TestClass:
def test_case(self):
# ...
def _get_categories(self):
return [...]
test_case.get_categories = _get_categories
del _get_categories
Class bodies are executable code like any other block.
What you need is nested classes. Functions aren't made to do what you are trying to do, so you have to move up a notch. Function attributes are mainly used as markup, whereas classes can have anything you want.
class TestClass(object):
class TestCase(object):
#classmethod
def get_categories(cls):
return ['abc', 'efg']
Note that I used #classmethod so that you could use it without instantiating TestCase(); modify if you want to do test_case = TestCase().
My test framework is currently based on a test-runner utility which itself is derived from the Eclipse pydev python test-runner. I'm switching to use Nose, which has many of the features of my custom test-runner but seems to be better quality code.
My test suite includes a number of abstract test-classes which previously never ran. The standard python testrunner (and my custom one) only ran instances of unittest.TestCase and unittest.TestSuite.
I've noticed that since I switched to Nose it's running just about anything which starts withthe name "test" which is annoying... because the naming convention we used for the test-mixins also looks like a test class to Nose. Previously these never ran as tests because they were not instances of TestCase or TestSuite.
Obviously I could re-name the methods to exclude the word "test" from their names... that would take a while because the test framework is very big and has a lot of inheritance. On the other hand it would be neat if there was a way to make Nose only see TestCases and TestSuites as being runnable... and nothing else.
Can this be done?
You could try to play with -m option for nosetests. From documentation:
A test class is a class defined in a
test module that matches testMatch or
is a subclass of unittest.TestCase
-m sets that testMatch, this way you can disable testing anything starting with test.
Another thing is that you can add __test__ = False to your test case class declaration, to mark it “not a test”.
If you want a truly abstract test class, you could just inherit the abstract class from object, then inherit in the testcases later.
For example:
class AbstractTestcases(object):
def test_my_function_does_something(self):
self.assertEquals("bar", self.func())
And then use it with:
class TestMyFooFunc(AbstractTestcases, unittest.TestCase):
def setUp(self):
self.func = lambda: "foo"
Then nosetests will pick up only the testcases in TestMyFooFunc and not those in AbstractTestCases.
You could use nose's --attr argument to specify an attribute posessed by the unittest.TestCase. For instance, I use:
nosetests --attr="assertAlmostEqual"
You could get even more careful by using and and or matching:
nosetests -A "assertAlmostEqual or addTest"
See unittest's documentation for a full list of methods/attributes, and Nose's description of the capabilities of the --attr plugin.
One addendum to #nailxx 's answer:
You could set __test__ = False in the parent class and then use a metaclass (see This question with some brilliant explanations) to set it back to True when subclassing.
(Finally, I found an excuse to use a metaclass!)
Although __test__ is a double underscore attribute, we have to explicitly set it to True, since not setting it would cause python just to lookup the attribute further up the MRO and evaluate it to False.
Thus, we need to check at class instantiation whether one of the parent classes has __test__ = False. If this is the case and the current class definition has not set __test__ itself, we shall add '__test__': True to the attributes dict.
The resulting code looks like this:
class TestWhenSubclassedMeta(type):
"""Metaclass that sets `__test__` back to `True` when subclassed.
Usage:
>>> class GenericTestCase(TestCase, metaclass=TestWhenSubclassed):
... __test__ = False
...
... def test_something(self):
... self.fail("This test is executed in a subclass, only.")
...
...
>>> class SpecificTestCase(GenericTestCase):
... pass
"""
def __new__(mcs, name, bases, attrs):
ATTR_NAME = '__test__'
VALUE_TO_RESET = False
RESET_VALUE = True
values = [getattr(base, ATTR_NAME) for base in bases
if hasattr(base, ATTR_NAME)]
# only reset if the first attribute is `VALUE_TO_RESET`
try:
first_value = values[0]
except IndexError:
pass
else:
if first_value == VALUE_TO_RESET and ATTR_NAME not in attrs:
attrs[ATTR_NAME] = RESET_VALUE
return super().__new__(mcs, name, bases, attrs)
One could extend this to some more implicit behaviour like “if the name starts with Abstract, set __test__ = False automatically”, but I for myself would keep the explicit assignment for clarity.
Let me paste simple unittests to demonstrate the behavior – and as a reminder that everybody should take the two minutes to test their code after introducing a feature.
from unittest import TestCase
from .base import TestWhenSubclassedMeta
class SubclassesTestCase(TestCase):
def test_subclass_resetted(self):
class Base(metaclass=TestWhenSubclassedMeta):
__test__ = False
class C(Base):
pass
self.assertTrue(C.__test__)
self.assertIn('__test__', C.__dict__)
def test_subclass_not_resetted(self):
class Base(metaclass=TestWhenSubclassedMeta):
__test__ = True
class C(Base):
pass
self.assertTrue(C.__test__)
self.assertNotIn('__test__', C.__dict__)
def test_subclass_attr_not_set(self):
class Base(metaclass=TestWhenSubclassedMeta):
pass
class C(Base):
pass
with self.assertRaises(AttributeError):
getattr(C, '__test__')
You can also use multiple inheritance on the test case level and let the base class inherit from object only. See this thread:
class MyBase(object):
def finishsetup(self):
self.z=self.x+self.y
def test1(self):
self.assertEqual(self.z, self.x+self.y)
def test2(self):
self.assert_(self.x > self.y)
class RealCase1(MyBase, unittest.TestCase):
def setUp(self):
self.x=10
self.y=5
MyBase.finishsetup(self)
class RealCase2(MyBase, unittest.TestCase):
def setUp(self):
self.x=42
self.y=13
MyBase.finishsetup(self)