I have a suite of tests that I have loaded using TestLoader's (from the unittest module) loadTestsFromModule() method, i.e.,
suite = loader.loadTestsFromModule(module)
This gives me a perfectly ample list of tests that works fine. My problem is that the test harness I'm working with sometimes needs to skip certain tests based on various criteria. What I want to do is something like this:
for test in suite:
mark the test as 'to-skip' if it meets certain criteria
Note that I can't just remove the test from the list of tests because I want the unittest test runner to actually skip the tests, add them to the skipped count, and all of that jazz.
The unittest documentation suggests using decorators around the test methods or classes. Since I'm loading these tests from a module and determining to skip based on criteria not contained within the tests themselves, I can't really use decorators. Is there a way I can iterate over each individual test and some how mark it as a "to-skip" test without having to directly access the test class or methods within the class?
Using unittest.TestCase.skipTest:
import unittest
class TestFoo(unittest.TestCase):
def setUp(self): print('setup')
def tearDown(self): print('teardown')
def test_spam(self): pass
def test_egg(self): pass
def test_ham(self): pass
if __name__ == '__main__':
import sys
loader = unittest.loader.defaultTestLoader
runner = unittest.TextTestRunner(verbosity=2)
suite = loader.loadTestsFromModule(sys.modules['__main__'])
for ts in suite:
for t in ts:
if t.id().endswith('am'): # To skip `test_spam` and `test_ham`
setattr(t, 'setUp', lambda: t.skipTest('criteria'))
runner.run(suite)
prints
test_egg (__main__.TestFoo) ... setup
teardown
ok
test_ham (__main__.TestFoo) ... skipped 'criteria'
test_spam (__main__.TestFoo) ... skipped 'criteria'
----------------------------------------------------------------------
Ran 3 tests in 0.001s
OK (skipped=2)
----------------------------------------------------------------------
Ran 3 tests in 0.002s
OK (skipped=2)
UPDATE
Updated the code to patch setUp instead of test method. Otherwise, setUp/tearDown methods will be executed for test to be skipped.
NOTE
unittest.TestCase.skipTest (Test skipping) was introduced in Python 2.7, 3.1. So this method only work in Python 2.7+, 3.1+.
This is a bit of a hack, but because you only need to raise unittest.SkipTest you can walk through your suite and modify each test to raise it for you instead of running the actual test code:
import unittest
from unittest import SkipTest
class MyTestCase(unittest.TestCase):
def test_this_should_skip(self):
pass
def test_this_should_get_skipped_too(self):
pass
def _skip_test(reason):
raise SkipTest(reason)
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(MyTestCase)
for test in suite:
skipped_test_method = lambda: _skip_test("reason")
setattr(test, test._testMethodName, skipped_test_method)
unittest.TextTestRunner(verbosity=2).run(suite)
When I run this, this is the output I get:
test_this_should_get_skipped_too (__main__.MyTestCase) ... skipped 'reason'
test_this_should_skip (__main__.MyTestCase) ... skipped 'reason'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=2)
Google brought me here.
I found the easiest way to do this is by raising a SkipTest exception when your skip criteria is met.
from unittest.case import SkipTest
def test_this_foo(self):
if <skip conditsion>:
raise SkipTest
And that test will be marked as skipped.
Some observations:
A test is a callable object with a __call__(result) method
TestCase provides a higher-level interface, allowing test methods to throw a SkipTest exception to skip themselves
The skip decorators do exactly this
Skipped tests are recorded calling the TestResult.addSkip(test, reason) method.
So you just need to replace the to-be-skipped tests with a custom test that calls addSkip:
class Skipper(object):
def __init__(self, test, reason):
self.test = test
self.reason = reason
def __call__(self, result):
result.addSkip(self.test, self.reason)
Related
In nose, the teardown runs regardless if setup has completed successfully or the status of the test run.
I want to perform a task in teardown that is only executed if the test that just ran failed. Is there an easy way to retrieve the result of each individual test case and pass it to the teardown method to be interpreted?
class TestMyProgram:
def setup(self):
# setup code here
def teardown(self):
# teardown code here
# run this code if test failed
if test_result == 'FAIL':
# do something
def test_one(self):
# example test placeholder
pass
def test_two(self):
# example test placeholder
pass
You have to capture the state of the test, and pass it on to your teardown method. The state of the test is within nose code: you cannot access without writing a nose plugin. But even with plugin, you would have to write a custom rig to pass on the state to the teardown method. But if you are willing to break the structure of the code a little bit to accommodate your request, you might be able to do something like this:
def special_trardown(self, state):
# only state specific logic goes here
print state
def test_one_with_passing_state(self):
try:
test_one(self)
except AssertionError as err:
test_result = "FAIL"
self.special_teardown(test_result)
raise
Its not perfect, but it makes the flow of events obvious to other people looking at your tests. You can also wrap it up as decorator / context manager for more syntactic sugar.
I wrote a nosetest class to test a particular method - test_method()
WHen I run this module I noticed nosetests ran the other methods as we well - create_test_private_method.
I thought nosetests will test only methods that starts with test_.
import unittest
class test(unittest.TestCase):
def create_test_private_method(self):
self.assertEqual(1,1)
def test_method(self):
self.assertEqual(2,2)
Output:
create_test_private_method (nosetest.test) ... ok
test_method (nosetest.test) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.009s
OK
From nosetests docs:
Any python source file, directory or package that matches the testMatch regular expression (by default: (?:^|[b_.-])[Tt]est) will be collected as a test (or source for collection of tests).
To avoid such a behavior you can
rename your methods
decorate your methods with the nose.tools.nottest decorator (as Oleksiy pointed out)
define a custom tests selector.
I have a python unitest script with an double inheritance of the TestCase as follows:
import unittest
class Upper(unittest.TestCase):
def test_dummy(self):
pass
class Lower(Upper):
pass
if __name__ == '__main__':
unittest.main()
The idea is to define an upper class with a test method implementation, and derive from this class (in different subdirectories) which contain some additional setup functionality. In the end, there is one upper.py from which many different test_lower.py are derived. The test methods are ONLY implemented in upper.py.
Given the example above now I do a python test_example.py only to see that python is trying to run 2 tests! The script contains exactly one test, so why is unittest executing two tests?
I assume that unittest finds one test within Lower and one test in Upper somehow, but I only want to execute the test which is found in Lower (because of additional and required setup functionalities). How can I achieve this?
Context In the real case the two classes are defined in two different files, residing in two directories. Maybe this helps.
Unittest library iterates over a TestCase subclass attributes, finding methods which start with test_. In case of Lower test case it inherits a method from its parent, so it is executed twice - once inside Upper test case and second time inside Lower class.
If both of the test cases are meant to be run, the only solution i see is to make a take out test_dummy test to an other subclass of Upper
If you want a test to be run in a parent class, but skipped in some of its subclasses, try this:
import unittest
class Upper(unittest.TestCase):
def test_dummy(self):
pass
class Lower(Upper):
def test_dummy(self):
return # skip test
#unittest.skip # or this - but ugly
def test_dummy(self):
return # skip test
if __name__ == '__main__':
unittest.main()
UPDATE:
I think now i understand what you want: you want the test method to be run only in subclasses. Then i suggest you to inherit Upper from object, and the subclasses - both from Upper and TestCase:
import unittest
class Upper(object):
def test_dummy(self):
self.assertTrue(True)
class Lower(unittest.TestCase, Upper):
pass
if __name__ == '__main__':
unittest.main()
Running:
python test2.py -v
test_dummy (__main__.Lower) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Context: python 2.6.5 environment
I am using unittest.defaultTestLoader.loadTestsFromModule(module) to load tests.
However, when the following is loaded, the setUpClass method is not executed.
class MyTest(unittest.TestCase):
foo = None
def test_choice(self):
self.logger.info(' .. %s' % str(Full.foo))
self.assertTrue(1 == 1)
#classmethod
def setUpClass(cls):
logging.warn('setUpClass')
cls.foo = settings.INITIAL
The returned test suites shows that it returned:
<unittest.TestSuite
tests=[<unittest.TestSuite
tests=[<internal.tests.master.MyTest testMethod=test_choice>]>,
Basically under the 'test' package, there will be many tests modules. And I want the setUpClass & tearDownClass to work for each test suites. loadTestsFromModule does not satisfy my requirement. Are there other ways to achieve this ?
It works now after I upgraded to use unittest2.
The TestLoader.loadTestsFromModule method is just for loading the test cases from your module into a test suite, you should run your test suite if you want setUpClass & tearDownClass to be executed.
When using nosetests for Python it is possible to disable a unit test by setting the test function's __test__ attribute to false. I have implemented this using the following decorator:
def unit_test_disabled():
def wrapper(func):
func.__test__ = False
return func
return wrapper
#unit_test_disabled
def test_my_sample_test()
#code here ...
However, this has the side effect of calling wrapper as the unit test. Wrapper will always pass but it is included in nosetests output. Is there another way of structuring the decorator so that the test will not run and does not appear in nosetests output.
Nose already has a builtin decorator for this:
from nose.tools import nottest
#nottest
def test_my_sample_test()
#code here ...
Also check out the other goodies that nose provides: https://nose.readthedocs.org/en/latest/testing_tools.html
You can also use unittest.skip decorator:
import unittest
#unittest.skip("temporarily disabled")
class MyTestCase(unittest.TestCase):
...
There also is a skiptest plugin for nosetest, which will cause the test show in test output as skipped. Here is a decorator for that:
def skipped(func):
from nose.plugins.skip import SkipTest
def _():
raise SkipTest("Test %s is skipped" % func.__name__)
_.__name__ = func.__name__
return _
Example output:
$ nosetests tests
..........................................................................
..................................S.............
----------------------------------------------------------------------
Ran 122 tests in 2.160s
OK (SKIP=1)
You can just start the class, method or function name with an underscore and nose will ignore it.
#nottest has its uses but I find that it does not work well when classes derive from one another and some base classes must be ignored by nose. This happens often when I have a series of similar Django views to test. They often share characteristics that need testing. For instance, they are accessible only to users with certain permissions. Rather than write the same permission check for all of them, I put such shared test in an initial class from which the other classes derive. The problem though is that the base class is there only to be derived by the later classes and is not meant to be run on its own. Here's an example of the problem:
from unittest import TestCase
class Base(TestCase):
def test_something(self):
print "Testing something in " + self.__class__.__name__
class Derived(Base):
def test_something_else(self):
print "Testing something else in " + self.__class__.__name__
And the output from running nose on it:
$ nosetests test.py -s
Testing something in Base
.Testing something in Derived
.Testing something else in Derived
.
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
The Base class is included in the tests.
I cannot just slap #nottest on Base because it will mark the entire hierarchy. Indeed if you just add #nottest to the code above in front of class Base, then nose won't run any tests.
What I do is add an underscore in front of the base class:
from unittest import TestCase
class _Base(TestCase):
def test_something(self):
print "Testing something in " + self.__class__.__name__
class Derived(_Base):
def test_something_else(self):
print "Testing something else in " + self.__class__.__name__
And when running it _Base is ignored:
$ nosetests test3.py -s
Testing something in Derived
.Testing something else in Derived
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
This behavior is not well documented but the code that selects tests explicitly checks for an underscore at the start of class names.
A similar test is performed by nose on function and method names so it is possible to exclude them by adding an underscore at the start of the name.
I think you will also need to rename your decorator to something that has not got test in. The below only fails on the second test for me and the first does not show up in the test suite.
def unit_disabled(func):
def wrapper(func):
func.__test__ = False
return func
return wrapper
#unit_disabled
def test_my_sample_test():
assert 1 <> 1
def test2_my_sample_test():
assert 1 <> 1