I'm currently running my tests like this:
tests = unittest.TestLoader().discover('tests')
unittest.TextTestRunner().run(tests)
Now I want to run a specific test knowing his name (like test_valid_user) but not knowing his class. If there is more than one test with such name than I would like to run all such tests. Is there any way to filter tests after discover?
Or maybe there are other solutions to this problem (please note that it shouldn't be done from command line)?
You can use the unittest.loader.TestLoader.testMethodPrefix instance variable to change the test methods filter according to a different prefix than "test".
Say you have a tests directory with this king of unit tests:
import unittest
class MyTest(unittest.TestCase):
def test_suite_1(self):
self.assertFalse("test_suite_1")
def test_suite_2(self):
self.assertFalse("test_suite_2")
def test_other(self):
self.assertFalse("test_other")
You can write your own discover function to discover only test functions starting with "test_suite_", for instance:
import unittest
def run_suite():
loader = unittest.TestLoader()
loader.testMethodPrefix = "test_suite_"
suite = loader.discover("tests")
result = unittest.TestResult()
suite.run(result)
for test, info in result.failures:
print(info)
if __name__ == '__main__':
run_suite()
remark: the argument "tests" in the discover method is a directory path, so you may need to write a fullpath.
As a result, you'll get:
Traceback (most recent call last):
File "/path/to/tests/test_my_module.py", line 8, in test_suite_1
self.assertFalse("test_suite_1")
AssertionError: 'test_suite_1' is not false
Traceback (most recent call last):
File "/path/to/tests/test_my_module.py", line 11, in test_suite_2
self.assertFalse("test_suite_2")
AssertionError: 'test_suite_2' is not false
Another simpler way, would be to use py.test with the -k option which does a test name keyword scan. It will run any tests whose name matches the keyword expression.
Although that is using the command-line which you didn't want, please not that you can call the command-line from your code using subprocess.call to pass any arguments you want dynamically.
E.g.: Assuming you have the following tests:
def test_user_gets_saved(self): pass
def test_user_gets_deleted(self): pass
def test_user_can_cancel(self): pass
You can call py.test from cli:
$ py.test -k "test_user"
Or from code:
return_code = subprocess.call('py.test -k "test_user"', shell=True)
There are two ways to run a single test method:
Command line:
$ python -m unittest test_module.TestClass.test_method
Using Python script:
import unittest
class TestMyCode(unittest.TestCase):
def setUp(self):
pass
def test_1(self):
self.assertTrue(True)
def test_2(self):
self.assertTrue(True)
if __name__ == '__main__':
testSuite = unittest.TestSuite()
testSuite.addTest(TestMyCode('test_1'))
runner=unittest.TextTestRunner()
runner.run(testSuite)
Output:
------------------------------------------------------------
Ran 1 test in 0.000s
OK
Related
I would like to upgrade my python test harness - which is based on Python's unittest module - from Python2 to Python3. However, the unittest.expectedFailure decorator doesn't seem to have the same effect anymore. In particular, the following code has different behavior depending on the Python version even though the specifications are virtually identical:
#!/usr/bin/env python2
#!/usr/bin/env python3
# Switch between the two lines above to get the different outcome
import unittest
class ComparisonTests(unittest.TestCase):
def runTest(self):
""" This method is needed even if empty """
def add_test(self, the_suite):
def testMain():
self.testFunc()
testMain = unittest.expectedFailure(testMain)
the_case = unittest.FunctionTestCase(testMain)
the_suite.addTest(the_case)
def testFunc(self):
self.assertTrue(False)
if __name__ == '__main__':
SUITE = unittest.TestSuite()
ComparisonTests().add_test(SUITE)
the_runner = unittest.TextTestRunner(verbosity=2)
the_runner.run(SUITE)
If I keep the first line (#!/usr/bin/env python2) and run on MacOS 10.14.1 and Python 2.7.15 then the output is the following:
unittest.case.FunctionTestCase (testMain) ... expected failure
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK (expected failures=1)
This is the behavior I expect. However, if I switch to the second line (#!/usr/bin/env python3) which will use Python 3.7.3 I get the following:
unittest.case.FunctionTestCase (testMain) ... FAIL
======================================================================
FAIL: unittest.case.FunctionTestCase (testMain)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./unittest_test_2.py", line 12, in testMain
self.testFunc()
File "./unittest_test_2.py", line 18, in testFunc
self.assertTrue(False)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
It looks like the unittest.expectedFailure decorator was ignored. Looking at the source code I can see a clear difference:
# Python 3.7 source:
def expectedFailure(test_item):
test_item.__unittest_expecting_failure__ = True
return test_item
# Python 2.7 source:
def expectedFailure(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
try:
func(*args, **kwargs)
except Exception:
raise _ExpectedFailure(sys.exc_info())
raise _UnexpectedSuccess
return wrapper
How can I define expected failures in the Python3 version of unittest ?
The Python 3 version of the unittest.expectedFailure decorator is expected to operate on a unittest test case and not on a method as it did in Python 2. So in order for the above test harness to work with Python 3 one needs to use the expectedFalure decorator on the_case as follows:
the_case = unittest.FunctionTestCase(testMain)
the_case = unittest.expectedFailure(the_case)
Is it possible to run tests from code using pytest? I did find pytest.main, but it's just a command line interface available from code. I would like to pass a test class / function from the code.
In unittest it's possible this way:
from unittest import TestLoader, TestCase, TestResult
class TestMy(TestCase):
def test_silly(self):
assert False
runner = TestLoader()
test_suite = runner.loadTestsFromTestCase(TestMy)
test_result = TestResult()
test_suite.run(test_result)
print(test_result)
Yes it's possible, that way for instance:
from pytest import main
class TestMy:
def test_silly(self):
assert False
main(['{}::{}'.format(__file__, TestMy.__name__)])
You can pass any argument to main as if called from command line.
I am able to pass command line arguments when running
python <filename>.py arg1
But when am trying to pass the command line arguments for running pytest it fails and gives error as below. Can you please advise.
pytest <filename>.py arg1
ERROR: file not found: arg1
EDIT:
For example am thinking of using it this way assuming I have passed an argument and am reading it via sys.argv:
import sys
arg = sys.argv[3]
def f():
return 3
def test_function():
assert f() == arg
Your pytest <filename>.py arg1 command is trying to call pytest on two modules <filename>.py and arg1 , But there is no module arg1.
If you want to pass some argument before running pytest then run the pytest from a python script after extracting your variable.
As others suggested though you would probably want to parameterize your tests in some other way, Try:Parameterized pytest.
# run.py
import pytest
import sys
def main():
# extract your arg here
print('Extracted arg is ==> %s' % sys.argv[2])
pytest.main([sys.argv[1]])
if __name__ == '__main__':
main()
call this using python run.py filename.py arg1
Here's the method I just cooked up from reading the parameterized pytest docs and hacking for a while... I don't know how stable or good it is going to be overall since I just got it working.
I did however check that HTML coverage generation works with this method.
add a file to your test directory for configuring the command-line args you want to pass:
tests\conftest.py
# this is just so we can pass --server and --port from the pytest command-line
def pytest_addoption(parser):
''' attaches optional cmd-line args to the pytest machinery '''
parser.addoption("--server", action="append", default=[], help="real server hostname/IP")
parser.addoption("--port", action="append", default=[], help="real server port number")
and then add a test file, with this special pytest_generate_tests function which is called when collecting a test function
tests\test_junk.py
def pytest_generate_tests(metafunc):
''' just to attach the cmd-line args to a test-class that needs them '''
server_from_cmd_line = metafunc.config.getoption("server")
port_from_cmd_line = metafunc.config.getoption("port")
print('command line passed for --server ({})'.format(server_from_cmd_line))
print('command line passed for --port ({})'.format(port_from_cmd_line))
# check if this function is in a test-class that needs the cmd-line args
if server_from_cmd_line and port_from_cmd_line and hasattr(metafunc.cls, 'real_server'):
# now set the cmd-line args to the test class
metafunc.cls.real_server = server_from_cmd_line[0]
metafunc.cls.real_port = int(port_from_cmd_line[0])
class TestServerCode(object):
''' test-class that might benefit from optional cmd-line args '''
real_server=None
real_port = None
def test_valid_string(self):
assert self.real_server!=None
assert self.real_port!=None
def test_other(self):
from mypackage import my_server_code
if self.real_server != None:
assert "couldn\'t find host" not in my_server_code.version(self.real_server, self.real_port)
then run (with HTML coverage, for example) with:
pytest tests\test_junk.py --server="abc" --port=123 --cov-report html --cov=mypackage
It seems monkeypatch also works.
Example:
import sys
def test_example(monkeypatch):
monkeypatch.setattr(sys, 'argv', ['/path/to/binary', 'opt1', '...'])
assert f() == '...'
def test_another():
# sys.argv is not modified here
assert f() != '...'
My Python version is 3.5.1
I have a simple code (tests.py):
import unittest
class SimpleObject(object):
array = []
class SimpleTestCase(unittest.TestCase):
def test_first(self):
simple_object = SimpleObject()
simple_object.array.append(1)
self.assertEqual(len(simple_object.array), 1)
def test_second(self):
simple_object = SimpleObject()
simple_object.array.append(1)
self.assertEqual(len(simple_object.array), 1)
if __name__ == '__main__':
unittest.main()
If I run it with command 'python tests.py' I will get the results:
.F
======================================================================
FAIL: test_second (__main__.SimpleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests.py", line 105, in test_second
self.assertEqual(len(simple_object.array), 1)
AssertionError: 2 != 1
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (failures=1)
Why it is happening? And how to fix it. I expect that each tests run will be independent (each test should pass), but it is not as we can see.
The array is shared by all instances of the class. If you want the array to be unique to an instance you need to put it in the class initializer:
class SimpleObject(object):
def __init__(self):
self.array = []
For more information take a look at this question: class variables is shared across all instances in python?
This can also be accomplished directly in unittest if you prefer to use only 1 class. Implement the setUp class. setUp runs before any of the tests are run as the class is instantiated. It is similar to init but conforms to the unittest library. Note the opposite is tearDown which is executed at the end of the test class if you need to build and deprecate objects. Example:
class SimpleObject(unitest.TestCase):
def setUp(self):
# I run first
self.array = []
def some_test(self):
self.assertTrue(self.array == [])
<some test code here>
def tearDown(self):
# I run last
self.array = []. # or whatever teardown code you need
enjoy!
I am trying to use unittest to automate a test case. However, when the test passes or fails, it would write the result to the console. Is there any way I can have unittest return some test result status code? This is because I would like to add another function in my test script to record the test result into our database. What is the best way to assess the test passed or failed programmatically?
This depends on how much information you need about the test results. If you just want to know if the test passed/failed using unittest.main :
"By default main calls sys.exit() with an exit code indicating success
or failure of the tests run"
So checking the return value (0=passed, non-0=failed) of your test script is enough to get a passed/failed answer.
If you need more details about the tests, you can skip the unittest.main() call and call the TestRunner.run method directly which returns a TestResult object describing the results. An example:
import unittest
from unittest import TextTestRunner
class TestExample(unittest.TestCase):
def test_pass(self):
self.assertEqual(1, 1, 'Expected 1 to equal 1')
def test_fail(self):
self.assertEqual(1, 2, 'uh-oh')
if __name__ == '__main__':
test_suite = unittest.TestLoader().loadTestsFromTestCase(TestExample)
test_result = TestRunner().run(test_suite)
... and you can now inspect the test_result variable to get more details about the test run:
>>> test_result.testsRun
2
>>> test_result.failures
[(<test_example.TestExample testMethod=test_fail>, 'Traceback (most recent call last):\n File "test_example.py", line 9, in test_fail\n self.assertEqual(1, 2, \'uh-oh\')\nAssertionError: uh-oh\n')]
>>> len(test_result.failures)
1
The properties of TestResult are documented here, and examples & options for running the test runner is explained here.