When using nosetests for Python it is possible to disable a unit test by setting the test function's __test__ attribute to false. I have implemented this using the following decorator:
def unit_test_disabled():
def wrapper(func):
func.__test__ = False
return func
return wrapper
#unit_test_disabled
def test_my_sample_test()
#code here ...
However, this has the side effect of calling wrapper as the unit test. Wrapper will always pass but it is included in nosetests output. Is there another way of structuring the decorator so that the test will not run and does not appear in nosetests output.
Nose already has a builtin decorator for this:
from nose.tools import nottest
#nottest
def test_my_sample_test()
#code here ...
Also check out the other goodies that nose provides: https://nose.readthedocs.org/en/latest/testing_tools.html
You can also use unittest.skip decorator:
import unittest
#unittest.skip("temporarily disabled")
class MyTestCase(unittest.TestCase):
...
There also is a skiptest plugin for nosetest, which will cause the test show in test output as skipped. Here is a decorator for that:
def skipped(func):
from nose.plugins.skip import SkipTest
def _():
raise SkipTest("Test %s is skipped" % func.__name__)
_.__name__ = func.__name__
return _
Example output:
$ nosetests tests
..........................................................................
..................................S.............
----------------------------------------------------------------------
Ran 122 tests in 2.160s
OK (SKIP=1)
You can just start the class, method or function name with an underscore and nose will ignore it.
#nottest has its uses but I find that it does not work well when classes derive from one another and some base classes must be ignored by nose. This happens often when I have a series of similar Django views to test. They often share characteristics that need testing. For instance, they are accessible only to users with certain permissions. Rather than write the same permission check for all of them, I put such shared test in an initial class from which the other classes derive. The problem though is that the base class is there only to be derived by the later classes and is not meant to be run on its own. Here's an example of the problem:
from unittest import TestCase
class Base(TestCase):
def test_something(self):
print "Testing something in " + self.__class__.__name__
class Derived(Base):
def test_something_else(self):
print "Testing something else in " + self.__class__.__name__
And the output from running nose on it:
$ nosetests test.py -s
Testing something in Base
.Testing something in Derived
.Testing something else in Derived
.
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
The Base class is included in the tests.
I cannot just slap #nottest on Base because it will mark the entire hierarchy. Indeed if you just add #nottest to the code above in front of class Base, then nose won't run any tests.
What I do is add an underscore in front of the base class:
from unittest import TestCase
class _Base(TestCase):
def test_something(self):
print "Testing something in " + self.__class__.__name__
class Derived(_Base):
def test_something_else(self):
print "Testing something else in " + self.__class__.__name__
And when running it _Base is ignored:
$ nosetests test3.py -s
Testing something in Derived
.Testing something else in Derived
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
This behavior is not well documented but the code that selects tests explicitly checks for an underscore at the start of class names.
A similar test is performed by nose on function and method names so it is possible to exclude them by adding an underscore at the start of the name.
I think you will also need to rename your decorator to something that has not got test in. The below only fails on the second test for me and the first does not show up in the test suite.
def unit_disabled(func):
def wrapper(func):
func.__test__ = False
return func
return wrapper
#unit_disabled
def test_my_sample_test():
assert 1 <> 1
def test2_my_sample_test():
assert 1 <> 1
Related
Let's say that, for whatever reason, I want to write a set of test cases that start out from the premise of failing code.
Maybe because setting up a failure is really complicated, while the demonstrating a fixed state is simple. In my case, it's not even that, I wanted to start with a failed test and then show how to fix it, for documentation purposes.
I can decorate #unittest.expectedFailure on the base class.
But the the fixed subclasses blow up with unexpected success because the decoration is inherited.
Can I remove the expectedFailure somehow?
In the code itself, not in a command line argument?
While I use and appreciate pytest this is a question for regular unittest.
unittest.skipXXX is not what I want, I do want to run Test_ThisShouldFail's test.
import sys
import unittest
#unittest.expectedFailure
class Test_ThisShouldFail(unittest.TestCase):
""" for argument's sake, let's say
the configuration and testing is very complicated
and I want it fail by default.
Subclasses fix the issue but re-use the test code
"""
data = dict(a=1, b=2)
def test_evens(self):
for key, v in self.data.items():
self.assertFalse(v % 2, f"{key}:Odd on {v}")
## 👉???unittest.expectedSuccess????
class Test_ThisShouldWork(Test_ThisShouldFail):
""" how do I turn off the expected failure? """
def setUp(self):
self.data.update(a=10)
if __name__ == "__main__":
sys.exit(unittest.main())
output:
FAILED (expected failures=1, unexpected successes=1)
(venv) #explore$ py test_expectedfail.py -v
test_evens (__main__.Test_ThisShouldFail) ... expected failure
test_evens (__main__.Test_ThisShouldWork) ... unexpected success
----------------------------------------------------------------------
Ran 2 tests in 0.000s
FAILED (expected failures=1, unexpected successes=1)
this didn't work:
I was hoping the MRO would look at TurnItOff's blank unittest settings and use them. No such luck.
class TurnItOff(unittest.TestCase):
pass
class Test_ThisShouldWork(TurnItOff, Test_ThisShouldFail):
....
This relies on the internal implementation of unittest.expectedFailure, but works for a start:
def expectedSuccess(test_item):
test_item.__unittest_expecting_failure__ = False
return test_item
#unittest.expectedFailure
class TestFailure(unittest.TestCase):
def test_something(self):
self.assertTrue(False)
#expectedSuccess
class TestSuccess(TestFailure):
def test_something(self):
self.assertTrue(True)
Note that test_item can be both a class or a function, depending on where you put the decorator.
Tested with Python 2.7.12 on Linux.
I am trying to write a simple function that decorates a test_* method in unittest.TestCase. I know that methods that do not begin with test_ are not considered actual tests and are not invoked directly when the test is run. I then wondered what you happen if you took a test_ method and applied a naive decorator to it that made no attempt to preserve the name. I was expecting my test to be ignored and to have to modify my decorator to make sure that the function has a name beginning with test_. I was very, very surprised when the test ran anyway.
Here's the directory layout of the example
.
|-- add.py
|-- print_args.py
|-- test_add.py
`-- test_add_decorated.py
0 directories, 4 files
add.py is the library we are testing. It adds two things.
def add(x, y):
return x + y
print_args.py is a library containing a decorator that prints the args and kwargs of a function as a side effect. It is written as naively as possible and makes no attempt to preserve the name of the function.
def print_args(wrapped):
def wrapper(*args, **kwargs):
print [args, kwargs]
return apply(wrapped, args, kwargs)
return wrapper
Here is test_add.py which imports add.py and tests that 4 + 5 = 9. The __repr__ method of TestAdd is not directly relevant to this example, but will be in the next one.
import unittest
import add
class TestAdd(unittest.TestCase):
def __repr__(self):
return "I am a unittest.TestCase! Fear me!"
def test_add(self):
self.assertEqual(add.add(4, 5), 9)
if __name__ == "__main__":
unittest.main()
When we run test_add.py, we can see that one test ran and it passed.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Now we apply the decorator to the test_add method in test_add_decorated.py.
import unittest
import print_args
import add
class TestAdd(unittest.TestCase):
def __repr__(self):
return "I am a unittest.TestCase! Fear me!"
#print_args.print_args
def test_add(self):
self.assertEqual(add.add(4, 5), 9)
if __name__ == "__main__":
unittest.main()
Before I ran this, I was expecting to see no errors, but an indication that zero tests had run, since the test_add method's name should now be wrapper.
That is not what happened. The print_args decorator worked fine, we can see the args and kwargs in an array and an indication that one test ran successfully.
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
[(I am a unittest.TestCase! Fear me!,), {}]
So, my question is ... how did the unittest library figure out that it was supposed to run my decorated method?
The wrapper function's __name__ might be wrapper, but it's still TestAdd.test_add. As in, if you look at the TestAdd class's __dict__, you'll find the wrapper function bound to key 'test_add', and if you wanted to call the method, you would use the name test_add, not wrapper.
When unittest uses method names to determine whether a method is a test, it doesn't look at the __name__. It looks at what attribute name the method is associated with, and the decorator doesn't affect that.
I have a nose testcase already written and want to inherit the same object for another test case as I'm going to change just one parameter.
So, just to understand how it works I tried to simulate the following using 2 classes NoseTesting and NoseTestingInherit.
When I run this:
I don't see the print statement being printed.
When I run the code I get the following result:
----------------------------------------------------------------------
Ran 0 tests in 0.000s
I'm not sure which test method ran - Is it test_this_method_dup or test_this_method?
class NoseTesting():
def test_this_method_dup():
print "Test this method"
class NoseTestingInherit(NoseTesting):
def test_this_method():
print "Test this method"
New Code:-
import unittest
class NoseTesting(unittest.TestCase):
def test_this_method_dup(self,):
print "Test this method"
class NoseTestingInherit(NoseTesting):
def test_this_method(self,):
print "Test this method"
Output
:
test_this_method_dup (nosetesting.NoseTesting) ... ok
test_this_method (nosetesting.NoseTestingInherit) ... ok
test_this_method_dup (nosetesting.NoseTestingInherit) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.003s
OK
From the nosetest docs:
A test class is a class defined in a test module that matches testMatch or is a subclass of unittest.TestCase.
Your classes do not match that pattern.
As #IanAuld said, nosetest did not discover your tests. According to your example, you need not explicitly rewrite the method if you're not overriding the method since it's implicitly inherited.
class NoseTesting(unittest.TestCase):
def test_this_method_dup(self):
print "Test this method"
class NoseTestingInherit(NoseTesting):
# implicitly inherit test_this_method_dup()
# self.test_this_method_dup()
I have a suite of tests that I have loaded using TestLoader's (from the unittest module) loadTestsFromModule() method, i.e.,
suite = loader.loadTestsFromModule(module)
This gives me a perfectly ample list of tests that works fine. My problem is that the test harness I'm working with sometimes needs to skip certain tests based on various criteria. What I want to do is something like this:
for test in suite:
mark the test as 'to-skip' if it meets certain criteria
Note that I can't just remove the test from the list of tests because I want the unittest test runner to actually skip the tests, add them to the skipped count, and all of that jazz.
The unittest documentation suggests using decorators around the test methods or classes. Since I'm loading these tests from a module and determining to skip based on criteria not contained within the tests themselves, I can't really use decorators. Is there a way I can iterate over each individual test and some how mark it as a "to-skip" test without having to directly access the test class or methods within the class?
Using unittest.TestCase.skipTest:
import unittest
class TestFoo(unittest.TestCase):
def setUp(self): print('setup')
def tearDown(self): print('teardown')
def test_spam(self): pass
def test_egg(self): pass
def test_ham(self): pass
if __name__ == '__main__':
import sys
loader = unittest.loader.defaultTestLoader
runner = unittest.TextTestRunner(verbosity=2)
suite = loader.loadTestsFromModule(sys.modules['__main__'])
for ts in suite:
for t in ts:
if t.id().endswith('am'): # To skip `test_spam` and `test_ham`
setattr(t, 'setUp', lambda: t.skipTest('criteria'))
runner.run(suite)
prints
test_egg (__main__.TestFoo) ... setup
teardown
ok
test_ham (__main__.TestFoo) ... skipped 'criteria'
test_spam (__main__.TestFoo) ... skipped 'criteria'
----------------------------------------------------------------------
Ran 3 tests in 0.001s
OK (skipped=2)
----------------------------------------------------------------------
Ran 3 tests in 0.002s
OK (skipped=2)
UPDATE
Updated the code to patch setUp instead of test method. Otherwise, setUp/tearDown methods will be executed for test to be skipped.
NOTE
unittest.TestCase.skipTest (Test skipping) was introduced in Python 2.7, 3.1. So this method only work in Python 2.7+, 3.1+.
This is a bit of a hack, but because you only need to raise unittest.SkipTest you can walk through your suite and modify each test to raise it for you instead of running the actual test code:
import unittest
from unittest import SkipTest
class MyTestCase(unittest.TestCase):
def test_this_should_skip(self):
pass
def test_this_should_get_skipped_too(self):
pass
def _skip_test(reason):
raise SkipTest(reason)
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(MyTestCase)
for test in suite:
skipped_test_method = lambda: _skip_test("reason")
setattr(test, test._testMethodName, skipped_test_method)
unittest.TextTestRunner(verbosity=2).run(suite)
When I run this, this is the output I get:
test_this_should_get_skipped_too (__main__.MyTestCase) ... skipped 'reason'
test_this_should_skip (__main__.MyTestCase) ... skipped 'reason'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=2)
Google brought me here.
I found the easiest way to do this is by raising a SkipTest exception when your skip criteria is met.
from unittest.case import SkipTest
def test_this_foo(self):
if <skip conditsion>:
raise SkipTest
And that test will be marked as skipped.
Some observations:
A test is a callable object with a __call__(result) method
TestCase provides a higher-level interface, allowing test methods to throw a SkipTest exception to skip themselves
The skip decorators do exactly this
Skipped tests are recorded calling the TestResult.addSkip(test, reason) method.
So you just need to replace the to-be-skipped tests with a custom test that calls addSkip:
class Skipper(object):
def __init__(self, test, reason):
self.test = test
self.reason = reason
def __call__(self, result):
result.addSkip(self.test, self.reason)
I have a python unitest script with an double inheritance of the TestCase as follows:
import unittest
class Upper(unittest.TestCase):
def test_dummy(self):
pass
class Lower(Upper):
pass
if __name__ == '__main__':
unittest.main()
The idea is to define an upper class with a test method implementation, and derive from this class (in different subdirectories) which contain some additional setup functionality. In the end, there is one upper.py from which many different test_lower.py are derived. The test methods are ONLY implemented in upper.py.
Given the example above now I do a python test_example.py only to see that python is trying to run 2 tests! The script contains exactly one test, so why is unittest executing two tests?
I assume that unittest finds one test within Lower and one test in Upper somehow, but I only want to execute the test which is found in Lower (because of additional and required setup functionalities). How can I achieve this?
Context In the real case the two classes are defined in two different files, residing in two directories. Maybe this helps.
Unittest library iterates over a TestCase subclass attributes, finding methods which start with test_. In case of Lower test case it inherits a method from its parent, so it is executed twice - once inside Upper test case and second time inside Lower class.
If both of the test cases are meant to be run, the only solution i see is to make a take out test_dummy test to an other subclass of Upper
If you want a test to be run in a parent class, but skipped in some of its subclasses, try this:
import unittest
class Upper(unittest.TestCase):
def test_dummy(self):
pass
class Lower(Upper):
def test_dummy(self):
return # skip test
#unittest.skip # or this - but ugly
def test_dummy(self):
return # skip test
if __name__ == '__main__':
unittest.main()
UPDATE:
I think now i understand what you want: you want the test method to be run only in subclasses. Then i suggest you to inherit Upper from object, and the subclasses - both from Upper and TestCase:
import unittest
class Upper(object):
def test_dummy(self):
self.assertTrue(True)
class Lower(unittest.TestCase, Upper):
pass
if __name__ == '__main__':
unittest.main()
Running:
python test2.py -v
test_dummy (__main__.Lower) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK