We Keep Coding

How does the unittest library determine whether to run a decorated method?

Tested with Python 2.7.12 on Linux.
I am trying to write a simple function that decorates a test_* method in unittest.TestCase. I know that methods that do not begin with test_ are not considered actual tests and are not invoked directly when the test is run. I then wondered what you happen if you took a test_ method and applied a naive decorator to it that made no attempt to preserve the name. I was expecting my test to be ignored and to have to modify my decorator to make sure that the function has a name beginning with test_. I was very, very surprised when the test ran anyway.
Here's the directory layout of the example
.
|-- add.py
|-- print_args.py
|-- test_add.py
`-- test_add_decorated.py
0 directories, 4 files
add.py is the library we are testing. It adds two things.
def add(x, y):
return x + y
print_args.py is a library containing a decorator that prints the args and kwargs of a function as a side effect. It is written as naively as possible and makes no attempt to preserve the name of the function.
def print_args(wrapped):
def wrapper(*args, **kwargs):
print [args, kwargs]
return apply(wrapped, args, kwargs)
return wrapper
Here is test_add.py which imports add.py and tests that 4 + 5 = 9. The __repr__ method of TestAdd is not directly relevant to this example, but will be in the next one.
import unittest
import add
class TestAdd(unittest.TestCase):
def __repr__(self):
return "I am a unittest.TestCase! Fear me!"
def test_add(self):
self.assertEqual(add.add(4, 5), 9)
if __name__ == "__main__":
unittest.main()
When we run test_add.py, we can see that one test ran and it passed.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Now we apply the decorator to the test_add method in test_add_decorated.py.
import unittest
import print_args
import add
class TestAdd(unittest.TestCase):
def __repr__(self):
return "I am a unittest.TestCase! Fear me!"
#print_args.print_args
def test_add(self):
self.assertEqual(add.add(4, 5), 9)
if __name__ == "__main__":
unittest.main()
Before I ran this, I was expecting to see no errors, but an indication that zero tests had run, since the test_add method's name should now be wrapper.
That is not what happened. The print_args decorator worked fine, we can see the args and kwargs in an array and an indication that one test ran successfully.
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
[(I am a unittest.TestCase! Fear me!,), {}]
So, my question is ... how did the unittest library figure out that it was supposed to run my decorated method?

The wrapper function's __name__ might be wrapper, but it's still TestAdd.test_add. As in, if you look at the TestAdd class's __dict__, you'll find the wrapper function bound to key 'test_add', and if you wanted to call the method, you would use the name test_add, not wrapper.
When unittest uses method names to determine whether a method is a test, it doesn't look at the __name__. It looks at what attribute name the method is associated with, and the decorator doesn't affect that.

nosetests executing methods that not start with test

I wrote a nosetest class to test a particular method - test_method()
WHen I run this module I noticed nosetests ran the other methods as we well - create_test_private_method.
I thought nosetests will test only methods that starts with test_.
import unittest
class test(unittest.TestCase):
def create_test_private_method(self):
self.assertEqual(1,1)
def test_method(self):
self.assertEqual(2,2)
Output:
create_test_private_method (nosetest.test) ... ok
test_method (nosetest.test) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.009s
OK

From nosetests docs:
Any python source file, directory or package that matches the testMatch regular expression (by default: (?:^|[b_.-])[Tt]est) will be collected as a test (or source for collection of tests).
To avoid such a behavior you can
rename your methods
decorate your methods with the nose.tools.nottest decorator (as Oleksiy pointed out)
define a custom tests selector.

py.test skips test class if constructor is defined

I have following unittest code running via py.test.
Mere presence of the constructor make the entire class skip when running
py.test -v -s
collected 0 items / 1 skipped
Can anyone please explain to me this behaviour of py.test?
I am interested in understanding py.test behaviour, I know the constructor is not needed.
Thanks,
Zdenek
class TestClassName(object):
def __init__(self):
pass
def setup_method(self, method):
print "setup_method called"
def teardown_method(self, method):
print "teardown_method called"
def test_a(self):
print "test_a called"
assert 1 == 1
def test_b(self):
print "test_b called"
assert 1 == 1

The documentation for py.test says that py.test implements the following standard test discovery:
collection starts from the initial command line arguments which may be directories, filenames or test ids.
recurse into directories, unless they match norecursedirs
test_*.py or *_test.py files, imported by their package name.
Test prefixed test classes (without an __init__ method) [<-- notice this one here]
test_ prefixed test functions or methods are test items
So it's not that the constructor isn't needed, py.test just ignores classes that have a constructor. There is also a guide for changing the standard test discovery.

As already mentioned in the answer by Matti Lyra py.test purposely skips classes which have a constructor. The reason for this is that classes are only used for structural reasons in py.test and do not have any inherent behaviour, while when actually writing code it is the opposite and much rarer to not have an .__init__() method for a class. So in practice skipping a class with a constructor will likely be what was desired, usually it is just a class which happens to have a conflicting name.
Lastly py.test needs to instantiate the class in order to execute the tests. If the constructor takes any arguments it can't instantiate it, so again skipping is the right thing to do.

All the above answers clearly explain the underlying cause, I just thought to share my experience and workaround the warnings.
I got my test to work without the warnings by aliasing the imported Class
from app.core.utils import model_from_meta
from app.core.models import Panel, TestType as _TestType
from app.core.serializers import PanelSerializer, TestType as _TestTypeSerializer
def test_model_from_meta():
assert (Panel is model_from_meta(PanelSerializer))
assert (_TestType is model_from_meta(_TestTypeSerializer))
After importing the class using aliases the warnings no longer get printed
I hope this helps someone.

In my case, I just so happened to have a parameter's class names TestParams, which conflicts with pytest looking for classes beginning with the name test....
Solution: rename your own class
Source

Skip unittest test without decorator syntax

I have a suite of tests that I have loaded using TestLoader's (from the unittest module) loadTestsFromModule() method, i.e.,
suite = loader.loadTestsFromModule(module)
This gives me a perfectly ample list of tests that works fine. My problem is that the test harness I'm working with sometimes needs to skip certain tests based on various criteria. What I want to do is something like this:
for test in suite:
mark the test as 'to-skip' if it meets certain criteria
Note that I can't just remove the test from the list of tests because I want the unittest test runner to actually skip the tests, add them to the skipped count, and all of that jazz.
The unittest documentation suggests using decorators around the test methods or classes. Since I'm loading these tests from a module and determining to skip based on criteria not contained within the tests themselves, I can't really use decorators. Is there a way I can iterate over each individual test and some how mark it as a "to-skip" test without having to directly access the test class or methods within the class?

Using unittest.TestCase.skipTest:
import unittest
class TestFoo(unittest.TestCase):
def setUp(self): print('setup')
def tearDown(self): print('teardown')
def test_spam(self): pass
def test_egg(self): pass
def test_ham(self): pass
if __name__ == '__main__':
import sys
loader = unittest.loader.defaultTestLoader
runner = unittest.TextTestRunner(verbosity=2)
suite = loader.loadTestsFromModule(sys.modules['__main__'])
for ts in suite:
for t in ts:
if t.id().endswith('am'): # To skip `test_spam` and `test_ham`
setattr(t, 'setUp', lambda: t.skipTest('criteria'))
runner.run(suite)
prints
test_egg (__main__.TestFoo) ... setup
teardown
ok
test_ham (__main__.TestFoo) ... skipped 'criteria'
test_spam (__main__.TestFoo) ... skipped 'criteria'
----------------------------------------------------------------------
Ran 3 tests in 0.001s
OK (skipped=2)
----------------------------------------------------------------------
Ran 3 tests in 0.002s
OK (skipped=2)
UPDATE
Updated the code to patch setUp instead of test method. Otherwise, setUp/tearDown methods will be executed for test to be skipped.
NOTE
unittest.TestCase.skipTest (Test skipping) was introduced in Python 2.7, 3.1. So this method only work in Python 2.7+, 3.1+.

This is a bit of a hack, but because you only need to raise unittest.SkipTest you can walk through your suite and modify each test to raise it for you instead of running the actual test code:
import unittest
from unittest import SkipTest
class MyTestCase(unittest.TestCase):
def test_this_should_skip(self):
pass
def test_this_should_get_skipped_too(self):
pass
def _skip_test(reason):
raise SkipTest(reason)
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(MyTestCase)
for test in suite:
skipped_test_method = lambda: _skip_test("reason")
setattr(test, test._testMethodName, skipped_test_method)
unittest.TextTestRunner(verbosity=2).run(suite)
When I run this, this is the output I get:
test_this_should_get_skipped_too (__main__.MyTestCase) ... skipped 'reason'
test_this_should_skip (__main__.MyTestCase) ... skipped 'reason'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=2)

Google brought me here.
I found the easiest way to do this is by raising a SkipTest exception when your skip criteria is met.
from unittest.case import SkipTest
def test_this_foo(self):
if <skip conditsion>:
raise SkipTest
And that test will be marked as skipped.

Some observations:
A test is a callable object with a __call__(result) method
TestCase provides a higher-level interface, allowing test methods to throw a SkipTest exception to skip themselves
The skip decorators do exactly this
Skipped tests are recorded calling the TestResult.addSkip(test, reason) method.
So you just need to replace the to-be-skipped tests with a custom test that calls addSkip:
class Skipper(object):
def __init__(self, test, reason):
self.test = test
self.reason = reason
def __call__(self, result):
result.addSkip(self.test, self.reason)

Disabling Python nosetests

When using nosetests for Python it is possible to disable a unit test by setting the test function's __test__ attribute to false. I have implemented this using the following decorator:
def unit_test_disabled():
def wrapper(func):
func.__test__ = False
return func
return wrapper
#unit_test_disabled
def test_my_sample_test()
#code here ...
However, this has the side effect of calling wrapper as the unit test. Wrapper will always pass but it is included in nosetests output. Is there another way of structuring the decorator so that the test will not run and does not appear in nosetests output.

Nose already has a builtin decorator for this:
from nose.tools import nottest
#nottest
def test_my_sample_test()
#code here ...
Also check out the other goodies that nose provides: https://nose.readthedocs.org/en/latest/testing_tools.html

You can also use unittest.skip decorator:
import unittest
#unittest.skip("temporarily disabled")
class MyTestCase(unittest.TestCase):
...

There also is a skiptest plugin for nosetest, which will cause the test show in test output as skipped. Here is a decorator for that:
def skipped(func):
from nose.plugins.skip import SkipTest
def _():
raise SkipTest("Test %s is skipped" % func.__name__)
_.__name__ = func.__name__
return _
Example output:
$ nosetests tests
..........................................................................
..................................S.............
----------------------------------------------------------------------
Ran 122 tests in 2.160s
OK (SKIP=1)

You can just start the class, method or function name with an underscore and nose will ignore it.
#nottest has its uses but I find that it does not work well when classes derive from one another and some base classes must be ignored by nose. This happens often when I have a series of similar Django views to test. They often share characteristics that need testing. For instance, they are accessible only to users with certain permissions. Rather than write the same permission check for all of them, I put such shared test in an initial class from which the other classes derive. The problem though is that the base class is there only to be derived by the later classes and is not meant to be run on its own. Here's an example of the problem:
from unittest import TestCase
class Base(TestCase):
def test_something(self):
print "Testing something in " + self.__class__.__name__
class Derived(Base):
def test_something_else(self):
print "Testing something else in " + self.__class__.__name__
And the output from running nose on it:
$ nosetests test.py -s
Testing something in Base
.Testing something in Derived
.Testing something else in Derived
.
----------------------------------------------------------------------
Ran 3 tests in 0.000s
OK
The Base class is included in the tests.
I cannot just slap #nottest on Base because it will mark the entire hierarchy. Indeed if you just add #nottest to the code above in front of class Base, then nose won't run any tests.
What I do is add an underscore in front of the base class:
from unittest import TestCase
class _Base(TestCase):
def test_something(self):
print "Testing something in " + self.__class__.__name__
class Derived(_Base):
def test_something_else(self):
print "Testing something else in " + self.__class__.__name__
And when running it _Base is ignored:
$ nosetests test3.py -s
Testing something in Derived
.Testing something else in Derived
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
This behavior is not well documented but the code that selects tests explicitly checks for an underscore at the start of class names.
A similar test is performed by nose on function and method names so it is possible to exclude them by adding an underscore at the start of the name.

I think you will also need to rename your decorator to something that has not got test in. The below only fails on the second test for me and the first does not show up in the test suite.
def unit_disabled(func):
def wrapper(func):
func.__test__ = False
return func
return wrapper
#unit_disabled
def test_my_sample_test():
assert 1 <> 1
def test2_my_sample_test():
assert 1 <> 1