I'm converting the test suite of a Python project from unittest to nose. The project's existing framework (based on unittest) is rather clunky, containing lots of heavily customised code for test discovery and running, so I'm trying to migrate to nose to make everything more streamlined.
I'm facing problems with the code that's generating test suites, however.
The project's framework has two ways of running tests. One is
class TestSomething(unittest.TestCase):
def setUp(self):
...
def test_x(self):
...
def test_y(self):
...
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(TestSomething))
which is the "straightforward" way, it's what all Nose examples and tutorials show, and it works. However, the second way is by defining a test class that contains all the test logic, then creating test cases in various subclasses that contain different setup configurations and inherit the tests from the superclass:
class TestSomething(unittest.TestCase):
def test_x(self):
...
def test_y(self):
...
class TestCase1(TestSomething):
def setUp(self):
...
class TestCase2(TestSomething):
def setUp(self):
...
suite = unittest.TestSuite()
cases = [TestCase1,TestCase2]
suite.addTests([unittest.makeSuite(case) for case in cases])
This is what Nose fails with. It tries to run the test methods first, which obviously doesn't work because there is no setUp() in the superclass and many of the variables used in test_x() and test_y() have not yet been defined.
I haven't found any examples of this done anywhere, and Nose's (rather sparse and difficult to navigate) documentation doesn't seem to mention it either. How can this be made to work with nose? Any help will be greatly appreciated.
First of all, as unutbu noted, you shouldn't give to TestSomething a name that starts with Test, because nose automatically treats such classes as test-cases. Also, nose runs all TestCase subclasses he finds, thus doing:
class Something(unittest.TestCase):
...
gives exactly the same results you are having. I think you should not inherit from TestCase and use that class as a mix-in:
class Something(object):
def test_x(self):
# here the various assertEqual etc. do not resolve, but you can use them
# as if they were present, since in real test-cases they will be inherited
# from unittest.TestCase.
...
...
class TestCase1(unittest.TestCase, Something):
def setUp(self):
...
An other way to do this is to set the class's __test__ attribute to False:
class TestSomething(unittest.TestCase):
__test__ = False
def test_x(self):
...
class TestCase1(TestSomething):
__test__ = True #must put this
def setUp(self):
...
Alternatively you can use nose.istest and nose.nottest to mark which class is a testcase and which one isn't:
#tools.nottest
class TestSomething(unittest.TestCase):
def test_x(self):
...
#tools.istest
class TestCase1(TestSomething):
sef setUp(self):
...
Related
house.py:
class House:
def is_habitable(self):
return True
def is_on_the_ground(self):
return True
conftest.py:
import pytest
from house import House
#pytest.fixture(scope='class')
def house():
return House()
test_house.py:
class TestHouse:
def test_habitability(self, house):
assert house.is_habitable()
def test_groundedness(self, house):
assert house.is_on_the_ground()
Up to that point, everything is being tested.
Now I add a subclass and override a method in house.py:
class House:
def is_habitable(self):
return True
def is_on_the_ground(self):
return True
class TreeHouse(House):
def is_on_the_ground(self):
return False
I also add a new fixture for that class in conftest.py:
import pytest
from house import House
from house import TreeHouse
#pytest.fixture(scope='class')
def house():
return House()
#pytest.fixture(scope='class')
def tree_house():
return TreeHouse()
I add a new test class for tree house in test_house.py:
class TestHouse:
def test_habitability(self, house):
assert house.is_habitable()
def test_groundedness(self, house):
assert house.is_on_the_ground()
class TestTreeHouse:
def test_groundedness(self, tree_house):
assert not tree_house.is_on_the_ground()
At that point, the code works, but there are cases that are not tested. For example, to be complete, I would need to test again the methods inherited from House in TreeHouse.
Rewriting the same tests from the TestHouse would not be DRY.
How do I test the inherited method of TreeHouse (in this case is_habitable) without duplicating code?
I would like something like re-testing the TreeHouse with the same tests that its super class runs through, but not for the methods / properties that are new or overridden.
After some research I came across contradictory sources. And after digging in the pytest documentation I fail to understand what applies to this scenario.
I am interested in the pytest way to do this. Please reference the docs and explain how that applies here.
One way to do this would be to use the fixture name house for all test methods (even if it's testing a TreeHouse), and override its value in each test context:
class TestTreeHouse(TestHouse):
#pytest.fixture
def house(self, tree_house):
return tree_house
def test_groundedness(self, house):
assert not house.is_on_the_ground()
Also note TestTreeHouse inherits from TestHouse. Since pytest merely enumerates methods of classes (i.e. there is no "registration" done with, say, a #pytest.test() decorator), all tests defined in TestHouse will be discovered in subclasses of it, without any further intervention.
You can use pytest parameterization to pass multiple arguments to the same test, in this case, the argument would most likely be the class being tested.
Problem
Yes I know you shouldn't send arguments to a unittest but in this case I have to so that it can access a piece of hardware within the current framework. For reasons for which the detail is irrelevant, I can't use a simple board = myBoard.getBoard() method within the tests, otherwise I wouldn't be here asking you all.
Horrible Attempt 1
class Test(object):
#staticmethod
def __call__(board):
_board = board
global _board
logging.info('Entering a Unit Test')
suite = unittest.TestLoader()
suite = suite.loadTestsFromTestCase(TestCase)
unittest.TextTestRunner().run(suite)
This method will be called upon to perform a unittest at somepoint. This is the way things are set out in the current framework, I'm just following the rules.
You can see here I assign a global variable (horrible). This can then be accessed by the test methods easily:
class TestCase(unittest.TestCase):
def runTest(self):
logging.critical('running')
def setUp(self):
logging.critical('setup')
def test_Me(self):
logging.critical('me')
The result is it won't enter runTest but it will setUp before each test_ method. Pretty much what I'd like, I can deal with it not entering runTest
Horrible Attempt 2
I'd like to do the above but without global variables so I tried this method:
class Test(object):
#staticmethod
def __call__(board):
logging.info('Entering a basic Unit Test')
suite = unittest.TestSuite() #creates a suite for testing
suite.addTest(BasicCase(board))
unittest.TextTestRunner().run(suite)
class BasicCase(unittest.TestCase):
def __init__(self, board):
self.board = board
logging.critical('init')
super(BasicCase, self).__init__()
def runTest(self):
logging.critical('runtest')
def setUp(self):
logging.critical('reset')
infra.Reset.__call__(self.board) #allows me to pass the board in a much nicer way
def test_Connect(self):
logging.critical('connection')
Problem with the above is that it will obviously only do init, setUp and runTest but it'll miss any test_ methods.
If I add:
suite.addTest(BasicCase(board).test_Connect)
Then it unnecessarily calls init twice and I have to add an extra member variable to test_Connect to accept the test object. I can deal with this too as I can return a list of the test_ methods but it just seems like a pain to do. Also I don't think it runs setUp every time it enters a new test_ method. Not ideal.
So ...
So is there some other method I can use that allows me to quickly run everything like a unittest but allows me to pass the board object?
If I should be using a different python module, then please say but I'm using unittest due to the lovely outputs and assert methods I get.
I am mocking out a database in some tests that I am doing. How would I create a setup method for the entire class, such that it runs each time an individual test within the class runs?
Example of what I am attempting to do.
from mocks import MockDB
class DBTests(unittest.TestCase):
def setup(self):
self.mock_db = MockDB()
def test_one(self):
#deal with self.mock_db
def test_two(self):
#deal with self.mock_db, as if nothing from test_one has happened
I'm assuming a teardown method would also be possible, but I can't find documentation that will do something like this.
If you are using Python unit test framework something like this is what you want:
class Test(unittest.TestCase):
def setUp(self):
self.mock_db = MockDB()
def tearDown(self):
pass # clean up
def test_1(self):
pass # test stuff
Documentation
With Nose, subclassing of TestCase works the same way as standard unittest -- setUp/tearDown are the same. From the nose docs
Test classes
A test class is a class defined in a test module that matches
testMatch or is a subclass of unittest.TestCase. All test classes are
run the same way: Methods in the class that match testMatch are
discovered, and a test case is constructed to run each method with a
fresh instance of the test class. Like unittest.TestCase subclasses,
other test classes can define setUp and tearDown methods that will be
run before and after each test method. Test classes that do not
descend from unittest.TestCase may also include generator methods and
class-level fixtures. Class-level setup fixtures may be named
setup_class, setupClass, setUpClass, setupAll or setUpAll; teardown
fixtures may be named teardown_class, teardownClass, tearDownClass,
teardownAll or tearDownAll. Class-level setup and teardown fixtures
must be class methods.
I'm using unittest to test my terminal interactive utility. I have 2 test cases with very similar contexts: one tests for correct output and other for correct handling of user commands in interactive mode. Although, both cases mock sys.stdout to suppress actual output (output is performed in process of interactive work as well).
Consider the following:
class StdoutOutputTestCase(unittest.TestCase):
"""Tests whether the stuff is printed correctly."""
def setUp(self):
self.patcher_stdout = mock.patch('sys.stdout', StringIO())
self.patcher_stdout.start()
# Do testing
def tearDown(self):
self.patcher_stdout.stop()
class UserInteractionTestCase(unittest.TestCase):
"""Tests whether user input is handled correctly."""
def setUp(self):
self.patcher_stdout = mock.patch('sys.stdout', StringIO())
self.patcher_stdout.start()
# Do testing
def tearDown(self):
self.patcher_stdout.stop()
What I don't like is that context setup is repeated here twice (for now; may be even more with time).
Is there a good way to set up common context for both cases? Can unittest.TestSuite help me? If yes, how? I couldn't find any example of common context setup.
I've also thought about defining a function setup_common_context, which would have been called from setUp of both cases, but it's still repetition.
I've solved this problem in my projects just by putting common setup code in a base class, and then putting test cases in derived classes. My derived class setUp and tearDown methods just call the super class implementations, and only do (de)initialization specific to those test cases. Also, keep in mind that you can put multiple tests in each test case, which might make sense if all the setup is the same.
class MyBaseTestCase(unittest.TestCase):
def setUp(self):
self.patcher_stdout = mock.patch('sys.stdout', StringIO())
self.patcher_stdout.start()
# Do **nothing**
def tearDown(self):
self.patcher_stdout.stop()
class StdoutOutputTestCase(MyBaseTestCase):
"""Tests whether the stuff is printed correctly."""
def setUp(self):
super(StdoutOutputTestCase, self).setUp()
# StdoutOutputTestCase specific set up code
# Do testing
def tearDown(self):
super(StdoutOutputTestCase, self).tearDown()
# StdoutOutputTestCase specific tear down code
class UserInteractionTestCase(MyBaseTestCase):
# Same pattern as StdoutOutputTestCase
My test framework is currently based on a test-runner utility which itself is derived from the Eclipse pydev python test-runner. I'm switching to use Nose, which has many of the features of my custom test-runner but seems to be better quality code.
My test suite includes a number of abstract test-classes which previously never ran. The standard python testrunner (and my custom one) only ran instances of unittest.TestCase and unittest.TestSuite.
I've noticed that since I switched to Nose it's running just about anything which starts withthe name "test" which is annoying... because the naming convention we used for the test-mixins also looks like a test class to Nose. Previously these never ran as tests because they were not instances of TestCase or TestSuite.
Obviously I could re-name the methods to exclude the word "test" from their names... that would take a while because the test framework is very big and has a lot of inheritance. On the other hand it would be neat if there was a way to make Nose only see TestCases and TestSuites as being runnable... and nothing else.
Can this be done?
You could try to play with -m option for nosetests. From documentation:
A test class is a class defined in a
test module that matches testMatch or
is a subclass of unittest.TestCase
-m sets that testMatch, this way you can disable testing anything starting with test.
Another thing is that you can add __test__ = False to your test case class declaration, to mark it “not a test”.
If you want a truly abstract test class, you could just inherit the abstract class from object, then inherit in the testcases later.
For example:
class AbstractTestcases(object):
def test_my_function_does_something(self):
self.assertEquals("bar", self.func())
And then use it with:
class TestMyFooFunc(AbstractTestcases, unittest.TestCase):
def setUp(self):
self.func = lambda: "foo"
Then nosetests will pick up only the testcases in TestMyFooFunc and not those in AbstractTestCases.
You could use nose's --attr argument to specify an attribute posessed by the unittest.TestCase. For instance, I use:
nosetests --attr="assertAlmostEqual"
You could get even more careful by using and and or matching:
nosetests -A "assertAlmostEqual or addTest"
See unittest's documentation for a full list of methods/attributes, and Nose's description of the capabilities of the --attr plugin.
One addendum to #nailxx 's answer:
You could set __test__ = False in the parent class and then use a metaclass (see This question with some brilliant explanations) to set it back to True when subclassing.
(Finally, I found an excuse to use a metaclass!)
Although __test__ is a double underscore attribute, we have to explicitly set it to True, since not setting it would cause python just to lookup the attribute further up the MRO and evaluate it to False.
Thus, we need to check at class instantiation whether one of the parent classes has __test__ = False. If this is the case and the current class definition has not set __test__ itself, we shall add '__test__': True to the attributes dict.
The resulting code looks like this:
class TestWhenSubclassedMeta(type):
"""Metaclass that sets `__test__` back to `True` when subclassed.
Usage:
>>> class GenericTestCase(TestCase, metaclass=TestWhenSubclassed):
... __test__ = False
...
... def test_something(self):
... self.fail("This test is executed in a subclass, only.")
...
...
>>> class SpecificTestCase(GenericTestCase):
... pass
"""
def __new__(mcs, name, bases, attrs):
ATTR_NAME = '__test__'
VALUE_TO_RESET = False
RESET_VALUE = True
values = [getattr(base, ATTR_NAME) for base in bases
if hasattr(base, ATTR_NAME)]
# only reset if the first attribute is `VALUE_TO_RESET`
try:
first_value = values[0]
except IndexError:
pass
else:
if first_value == VALUE_TO_RESET and ATTR_NAME not in attrs:
attrs[ATTR_NAME] = RESET_VALUE
return super().__new__(mcs, name, bases, attrs)
One could extend this to some more implicit behaviour like “if the name starts with Abstract, set __test__ = False automatically”, but I for myself would keep the explicit assignment for clarity.
Let me paste simple unittests to demonstrate the behavior – and as a reminder that everybody should take the two minutes to test their code after introducing a feature.
from unittest import TestCase
from .base import TestWhenSubclassedMeta
class SubclassesTestCase(TestCase):
def test_subclass_resetted(self):
class Base(metaclass=TestWhenSubclassedMeta):
__test__ = False
class C(Base):
pass
self.assertTrue(C.__test__)
self.assertIn('__test__', C.__dict__)
def test_subclass_not_resetted(self):
class Base(metaclass=TestWhenSubclassedMeta):
__test__ = True
class C(Base):
pass
self.assertTrue(C.__test__)
self.assertNotIn('__test__', C.__dict__)
def test_subclass_attr_not_set(self):
class Base(metaclass=TestWhenSubclassedMeta):
pass
class C(Base):
pass
with self.assertRaises(AttributeError):
getattr(C, '__test__')
You can also use multiple inheritance on the test case level and let the base class inherit from object only. See this thread:
class MyBase(object):
def finishsetup(self):
self.z=self.x+self.y
def test1(self):
self.assertEqual(self.z, self.x+self.y)
def test2(self):
self.assert_(self.x > self.y)
class RealCase1(MyBase, unittest.TestCase):
def setUp(self):
self.x=10
self.y=5
MyBase.finishsetup(self)
class RealCase2(MyBase, unittest.TestCase):
def setUp(self):
self.x=42
self.y=13
MyBase.finishsetup(self)