We Keep Coding

How to skip some test cases in test discovery?

In python 2.7, I use unittest module and write tests, while some of them are skipped with #unittest.skip.
My codes looks like:
import unittest
class MyTest(unittest.TestCase):
def test_1(self):
...
#unittest.skip
def test_2(self):
...
I have lots of such test files in a folder, and I use test discovery to run all these test files:
/%python_path/python -m unittest discover -s /%my_ut_folder% -p "*_unit_test.py"
This way, all *_unit_test.py files in the folder will be ran. In above codes, both test_1 and test_2 will be ran. What I want is, all test cases with #unittest.skip, e.g. test_2 in my above codes, should be skipped. How do I achieve this?
Any help or suggestion will be greatly appreciated!

Try adding a string argument to the #unittest.skip decorator, such as in the following:
import unittest
class TestThings(unittest.TestCase):
def test_1(self):
self.assertEqual(1,1)
#unittest.skip('skipping...')
def test_2(self):
self.assertEqual(2,4)
Running without the string argument in python 2.7 gives me the following:
.E
======================================================================
ERROR: test_2 (test_test.TestThings)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python2.7/functools.py", line 33, in update_wrapper
setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'TestThings' object has no attribute '__name__'
----------------------------------------------------------------------
Ran 2 tests in 0.001s
whereas running with text in python 2.7 gives me:
.s
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=1)
See https://docs.python.org/3/library/unittest.html or https://www.tutorialspoint.com/unittest_framework/unittest_framework_skip_test.htm for more details

Unexpected behavior from unittest.mock.patch

What is wrong with my code below?
I'm expecting assert call_func_once_with("b") to throw an error as call_func was passed 'a'. I confimed that the function was indeed called once and with argument 'a'.
from unittest.mock import Mock, patch
def call_func(x):
pass
#patch("__main__.call_func")
def test_call_func(call_func):
call_func("a")
assert call_func.called_once_with("b")
assert call_func.called == 1
print(call_func.call_args)
test_call_func()
Output:
call('a')

You're not the first person to notice strange things with these types of assertions (see Magic mock assert_called_once vs assert_called_once_with weird behaviour)
For what it's worth, I can only advise that you try to create a test class which inherits from unittest.TestCase and then use the assertEqual method to get more consistent test behaviour:
import unittest
from unittest.mock import patch, call
def call_func(x):
pass
class MyTests(unittest.TestCase):
#patch("__main__.call_func")
def test_call_func(self, call_func_mock):
call_func_mock("a")
# assert call_func_mock.called == 1
# assert call_func_mock.called_once_with("b")
self.assertEqual(call_func_mock.call_count, 1)
self.assertEqual(call_func_mock.call_args_list[0], call("b"))
print(call_func_mock.call_args)
unittest.main()
This gives the following (expected) results:
F
======================================================================
FAIL: test_call_func (__main__.MyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python36\lib\unittest\mock.py", line 1179, in patched
return func(*args, **keywargs)
File "C:/scratch.py", line 16, in test_call_func
self.assertEquals(call_func_mock.call_args_list[0], call("b"))
AssertionError: call('a') != call('b')
----------------------------------------------------------------------
Ran 1 test in 0.003s
FAILED (failures=1)
Process finished with exit code 1

Filter tests after discover

I'm currently running my tests like this:
tests = unittest.TestLoader().discover('tests')
unittest.TextTestRunner().run(tests)
Now I want to run a specific test knowing his name (like test_valid_user) but not knowing his class. If there is more than one test with such name than I would like to run all such tests. Is there any way to filter tests after discover?
Or maybe there are other solutions to this problem (please note that it shouldn't be done from command line)?

You can use the unittest.loader.TestLoader.testMethodPrefix instance variable to change the test methods filter according to a different prefix than "test".
Say you have a tests directory with this king of unit tests:
import unittest
class MyTest(unittest.TestCase):
def test_suite_1(self):
self.assertFalse("test_suite_1")
def test_suite_2(self):
self.assertFalse("test_suite_2")
def test_other(self):
self.assertFalse("test_other")
You can write your own discover function to discover only test functions starting with "test_suite_", for instance:
import unittest
def run_suite():
loader = unittest.TestLoader()
loader.testMethodPrefix = "test_suite_"
suite = loader.discover("tests")
result = unittest.TestResult()
suite.run(result)
for test, info in result.failures:
print(info)
if __name__ == '__main__':
run_suite()
remark: the argument "tests" in the discover method is a directory path, so you may need to write a fullpath.
As a result, you'll get:
Traceback (most recent call last):
File "/path/to/tests/test_my_module.py", line 8, in test_suite_1
self.assertFalse("test_suite_1")
AssertionError: 'test_suite_1' is not false
Traceback (most recent call last):
File "/path/to/tests/test_my_module.py", line 11, in test_suite_2
self.assertFalse("test_suite_2")
AssertionError: 'test_suite_2' is not false

Another simpler way, would be to use py.test with the -k option which does a test name keyword scan. It will run any tests whose name matches the keyword expression.
Although that is using the command-line which you didn't want, please not that you can call the command-line from your code using subprocess.call to pass any arguments you want dynamically.
E.g.: Assuming you have the following tests:
def test_user_gets_saved(self): pass
def test_user_gets_deleted(self): pass
def test_user_can_cancel(self): pass
You can call py.test from cli:
$ py.test -k "test_user"
Or from code:
return_code = subprocess.call('py.test -k "test_user"', shell=True)

There are two ways to run a single test method:
Command line:
$ python -m unittest test_module.TestClass.test_method
Using Python script:
import unittest
class TestMyCode(unittest.TestCase):
def setUp(self):
pass
def test_1(self):
self.assertTrue(True)
def test_2(self):
self.assertTrue(True)
if __name__ == '__main__':
testSuite = unittest.TestSuite()
testSuite.addTest(TestMyCode('test_1'))
runner=unittest.TextTestRunner()
runner.run(testSuite)
Output:
------------------------------------------------------------
Ran 1 test in 0.000s
OK

Python Unittest and object initialization

My Python version is 3.5.1
I have a simple code (tests.py):
import unittest
class SimpleObject(object):
array = []
class SimpleTestCase(unittest.TestCase):
def test_first(self):
simple_object = SimpleObject()
simple_object.array.append(1)
self.assertEqual(len(simple_object.array), 1)
def test_second(self):
simple_object = SimpleObject()
simple_object.array.append(1)
self.assertEqual(len(simple_object.array), 1)
if __name__ == '__main__':
unittest.main()
If I run it with command 'python tests.py' I will get the results:
.F
======================================================================
FAIL: test_second (__main__.SimpleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests.py", line 105, in test_second
self.assertEqual(len(simple_object.array), 1)
AssertionError: 2 != 1
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (failures=1)
Why it is happening? And how to fix it. I expect that each tests run will be independent (each test should pass), but it is not as we can see.

The array is shared by all instances of the class. If you want the array to be unique to an instance you need to put it in the class initializer:
class SimpleObject(object):
def __init__(self):
self.array = []
For more information take a look at this question: class variables is shared across all instances in python?

This can also be accomplished directly in unittest if you prefer to use only 1 class. Implement the setUp class. setUp runs before any of the tests are run as the class is instantiated. It is similar to init but conforms to the unittest library. Note the opposite is tearDown which is executed at the end of the test class if you need to build and deprecate objects. Example:
class SimpleObject(unitest.TestCase):
def setUp(self):
# I run first
self.array = []
def some_test(self):
self.assertTrue(self.array == [])
<some test code here>
def tearDown(self):
# I run last
self.array = []. # or whatever teardown code you need
enjoy!

Skipping unittest with decorator in python

I'm writing some unittest and found a rather curious behavior that nearly burned me.
The following test:
import unittest
class Test(unittest.TestCase):
#unittest.skip('Not ready yet')
def test_A(self):
self.assertTrue(False)
#unittest.skip
def test_B(self):
self.assertTrue(False)
def test_C(self):
self.assertTrue(False)
if __name__ == '__main__':
unittest.main()
results in:
test_A (__main__.Test) ... skipped 'Not ready yet'
test_B (__main__.Test) ... ok
test_C (__main__.Test) ... FAIL
======================================================================
FAIL: test_C (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./test.py", line 13, in test_C
self.assertTrue(False)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 3 tests in 0.000s
Using the decorator unittest.skip empty does skip the test but then reports it as passed. Therefore this skipped test could be easily forgotten the next day and stay in the skip state forever. What is the reason behind this skip, but report pass behavior?
In case it matters:
Python: 3.4.3 | Anaconda 2.3.0 (64-bit)
OS: RHEL 6.7

#decorator
def f(): ...
is equivalent to
def f(): ...
f = decorator(f)
and
#decorator(...)
def f(): ...
is equivalent to
def f(): ...
f = decorator(...)(f)
That means when you forget the skip reason, you get the effects of
def test_B(self): ...
test_B = unittest.skip(test_B)
The test method is passed as the skip reason, and the returned test decorator is assigned to test_B. When unittest tries to run test_B, the test decorator reports no assertion failures, so unittest thinks it's a passing test.
The inequivalence of #decorator and #decorator() is one of Python's design warts, but there isn't much we can do about it.