I am working on a python based testing system, that iterates through a set of python tests and run them one by one (there are unittests and pytests).
Is there a way that my testing system to understand the result of every individual tests and saves it to a dictionary with key [test_name] and value [test_status] for example. I imagine if the result of the test to be assigned to a variable for example:
test_status = "passed"
PS: all of the tests have a main(), that looks like that
# for unittests
def main():
unittest.main()
# for pytests
def main():
os.system("py.test -v {}".format(os.path.abspath(__file__)))
If I understood correctly, you want to actually run pytest or unittests as a command line tool and retrieve the results.
The straightforward way to do this would be to use JUnit xml output and parse it. For example in pytest:
pytest --junitxml=path
Maybe you want to consider using an automation server like Jenkins, that will run unittests and pytest tests separately and then collect the results.
I have found a solution for unittest framework:
The idea is to change the test ouput data to be not on the terminal console, but to a file. One way to do it is to add the following code to the tests:
if __name__ == '__main__':
# terminal command to run a specific test and save its output to a log file
# python [this_module_name] [log_file.log] [*tests]
parser = argparse.ArgumentParser()
parser.add_argument('test_log_file')
parser.add_argument('unittest_args', nargs='*')
args = parser.parse_args()
log_file = sys.argv[1]
# Now set the sys.argv to the unittest_args (leaving sys.argv[0] alone)
sys.argv[1:] = args.unittest_args
with open(log_file, "w") as f:
runner = unittest.TextTestRunner(f)
unittest.main(defaultTest=sys.argv[2:], exit=False, testRunner=runner)
and run it with command like this one:
python my_tests.py log_file.log class_name.test_1 class_name.test_2 ... test_n
Other way is with direct command that looks like this one:
python -m unittest [test_module_name].[test_class_name].[test_name] 2> [log_file_name]
# real command example:
python -m unittest my_tests.class_name.test_1 2> my_test_log_file.log
# real command example with multiple tests:
python -m unittest my_tests.class_name.test_1 my_tests.class_name.test_2 my_tests.class_name.test_3 2> my_test_log_file.log
The final part is to write a method, that reads this log file and get the result of the test. Those log files look something like that:
.FEs
======================================================================
ERROR: test_simulative_error (__main__.SimulativeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "unittest_tests.py", line 84, in test_simulative_error
raise ValueError
ValueError
======================================================================
FAIL: test_simulative_fail (__main__.SimulativeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "unittest_tests.py", line 81, in test_simulative_fail
assert False
AssertionError
----------------------------------------------------------------------
Ran 4 tests in 0.001s
FAILED (failures=1, errors=1, skipped=1)
So the final step is to open that log_file and read the first line that gives information how the test/tests have finished and save this info as you want.
Related
I have a file TestProtocol.py that has unittests. I can run that script and get test results for my 30 tests as expected. Now I want to run those tests from another file tester.py that is located in the same directory. Inside tester.py I tried import TestProtocol, but it runs 0 tests.
Then I found the documentation which says I should do something like this:
suite = unittest.TestLoader().discover(".", pattern = "*")
unittest.run(suite)
This should go through all files in the current directory . that match the pattern *, so all tests in all files. Unfortunately it again runs 0 tests.
There is a related QA that suggests to do
import TestProtocol
suite = unittest.findTestCases(TestProtocol)
unittest.run(suite)
but that also does not find any tests.
How do I import and run my tests?
You can try with following
# preferred module name would be test_protol as CamelCase convention are used for class name
import TestProtocol
# try to load all testcases from given module, hope your testcases are extending from unittest.TestCase
suite = unittest.TestLoader().loadTestsFromModule(TestProtocol)
# run all tests with verbosity
unittest.TextTestRunner(verbosity=2).run(suite)
Here is a full example
file 1: test_me.py
# file 1: test_me.py
import unittest
class TestMe(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
if __name__ == '__main__':
unittest.main()
file 2: test_other.py, put this under same directory
# file 2: test_other.py, put this under same directory
import unittest
import test_me
suite = unittest.TestLoader().loadTestsFromModule(test_me)
unittest.TextTestRunner(verbosity=2).run(suite)
run each file, it will show the same result
# python test_me.py - Ran 1 test in 0.000s
# python test_other.py - Ran 1 test in 0.000s
How can I make that my __main__ file prints are outputted, when I run tests? I mean prints from that file, not unittests files prints.
I have this sample structure (all files are in the same directory):
main.py:
import argparse
print('print me?') # no output
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('name')
args = parser.parse_args()
print(args.name) # no output
other.py:
def print_me():
print('ran print_me')
test.py:
import unittest
import sh
import other
class TestMain(unittest.TestCase):
def test_main(self):
print('test_main') # prints it.
sh.python3('main.py', 'test123')
def test_other(self):
print('test_other') # prints it.
other.print_me()
And I run it with python3 -m nose -s or python3 -m unittest, but it makes no difference, prints are not outputted from main.py, only the ones that are defined directly on test file. Here is what I do get:
user#user:~/python-programs/test_main$ python3 -m nose -s
test_main
.test_other
ran print_me
.
----------------------------------------------------------------------
Ran 2 tests in 0.040s
OK
P.S. Of course if I run main.py without using tests, then it prints normally (for example using python shell/interpreter and calling main.py with sh, just like in unittests)
sh.python3 starts new process and its output is not captured by nose. You can redirect the output printing the result from it:
print(sh.python3('main.py', 'test123'))
This might be really stupid, but I can't get it to work...
I'm want to use the such DLS in nose2 with python 2.7 in Linux.
I'm trying out the beginning of the example from the documentation http://nose2.readthedocs.org/en/latest/such_dsl.html (see code below) but it doesn't run the tests, no matter how I launch it from the command line.
My file is called test_something.py, it's the only file in the directory.
I've tried running from the command line with >> nose2 and >> nose2 --plugin nose2.plugins.layers, but I always get Ran 0 tests in 0.000s. With >> nose2 --plugin layers I get ImportError: No module named layers.
How am I supposed to run this test from the command line??
Thanks!
Code below:
import unittest
from nose2.tools import such
with such.A("system with complex setup") as it:
#it.has_setup
def setup():
print "Setup"
it.things = [1]
#it.has_teardown
def teardown():
print "Teardown"
it.things = []
#it.should("do something")
def test():
print "Test"
assert it.things
it.assertEqual(len(it.things), 1)
DOH!
I forgot to add it.createTests(globals()) at the end of the file!
I want to split my Python 3.4 unit tests in separate modules and still be able to control which tests to run or skip from the command line, as if all tests were located in the same file. I'm having trouble doing so.
According to the docs, command line arguments can be used to select which tests to run. For example:
TestSeqFunc.py:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import random
import unittest
class TestSequenceFunctions(unittest.TestCase):
def setUp(self):
self.seq = list(range(10))
def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, list(range(10)))
# should raise an exception for an immutable sequence
self.assertRaises(TypeError, random.shuffle, (1,2,3))
def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)
def test_sample(self):
with self.assertRaises(ValueError):
random.sample(self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)
if __name__ == '__main__':
unittest.main()
can be controlled with either:
./TestSeqFunc.py
to run all tests in the file,
./TestSeqFunc.py TestSequenceFunctions
to run all tests defined in the TestSequenceFunctions class, and finally:
./TestSeqFunc.py TestSequenceFunctions.test_sample
to run the specific test_sample() method.
The problem I have is that I cannot find an organization of files that will allow me to:
Have multiple modules containing multiple classes and methods in separate files
Use a kind of wrapper script that will give the same kind of control over which tests (module/file, class, method) to run.
The problem I have is I cannot find a way to emulate the python3 -m unittest behaviour using a run_tests.py script. For example, I want to be able to do:
Run all the tests in the current directory
So ./run_tests.py -v should do do the same as python3 -m unittest -v
Run one module (file):
./run_tests.py -v TestSeqFunc being equivalent to python3 -m unittest -v TestSeqFunc
Run one class:
./run_tests.py -v TestSeqFunc.TestSequenceFunctions being equivalent to python3 -m unittest -v TestSeqFunc.TestSequenceFunctions
Run specific methods from a class:
./run_tests.py -v TestSeqFunc.TestSequenceFunctions.test_sample being equivalent to python3 -m unittest -v TestSeqFunc.TestSequenceFunctions.test_sample
Note that I want to:
be able to pass arguments to unittests, for example the verbose flag used previously;
allow running specific modules, classes and even methods.
As of now, I use a suite() function in my run_all.py script which loads manually the modules and add their class to a suite using addTest(unittest.makeSuite(obj)). Then, my main() is simple:
if __name__ == '__main__':
unittest.main(defaultTest='suite')
But using this I cannot run specific tests. In the end, I might just execute python3 -m unittest <sys.argv> from inside the run_all.py script, but that would be inelegant...
Any suggestions?!
Thanks!
Here's my final run_all.py:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import unittest
import glob
test_pattern = 'validate_*.py'
if __name__ == '__main__':
# Find all files matching pattern
module_files = sorted(glob.glob(test_pattern))
module_names = [os.path.splitext(os.path.basename(module_file))[0] for module_file in module_files]
# Iterate over the found files
print('Importing:')
for module in module_names:
print(' ', module)
exec('import %s' % module)
print('Done!')
print()
unittest.main(defaultTest=module_names)
Notes:
I use exec() to simulate 'import modulename'. The issue is that using importlib (explained here for example) will import the module but will not create a namespace for the module content. When I type import os, an "os" namespace is created and I can then access os.path. By using importlib, I couldn't figure out a way to do create that namespace. Having such a namespace is required for unittest; you get these kind of errors:
Traceback (most recent call last):
File "./run_all.py", line 89, in <module>
unittest.main(argv=sys.argv)
File "~/usr/lib/python3.4/unittest/main.py", line 92, in __init__
self.parseArgs(argv)
File "~/usr/lib/python3.4/unittest/main.py", line 139, in parseArgs
self.createTests()
File "~/usr/lib/python3.4/unittest/main.py", line 146, in createTests
self.module)
File "~/usr/lib/python3.4/unittest/loader.py", line 146, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "~/usr/lib/python3.4/unittest/loader.py", line 146, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "~/usr/lib/python3.4/unittest/loader.py", line 114, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'validate_module1'
Hence the use of exec().
I have to add defaultTest=module_names or else main() defaults to all test classes inside the current file. Since there is no test class in run_all.py, nothing gets executed. So defaultTest must point to a list of all the modules name.
You can pass command-line arguments to unittest.main using the argv parameter:
The argv argument can be a list of options passed to the program, with
the first element being the program name. If not specified or None,
the values of sys.argv are used. (my emphasis)
So you should be able to use
if __name__ == '__main__':
unittest.main(defaultTest='suite')
without any change and be able to call your script with command-line arguments as desired.
I'm using contracts for Python to specify preconditons/postconditions/invariants. I'm also using doctests for doing unit testing.
I'd like to have all of my doctest unit tests run with contracts enabled, and I'd like to run my tests using nose. Unfortunately, if I run the tests with nose, it does not execute the pre/post/invariant assertions. I put a setup function in each .py file to make sure that contract.checkmod gets called
def setup():
import contract
contract.checkmod(__name__)
I can confirm that this function is being executed by nose before it runs the tests, but the contracts still don't get executed.
On the other hand, if I run the doctest by calling doctest.testmod, the pre/post/inv do get called:
def _test():
import contract
contract.checkmod(__name__)
import doctest
doctest.testmod()
if __name__=='__main__':
_test()
Here's an example of a Python script whose test will succeed if called directly, but failed if called with nose:
import os
def setup():
import contract
contract.checkmod(__name__)
def delete_file(path):
"""Delete a file. File must be present.
>>> import minimock
>>> minimock.mock('os.remove')
>>> minimock.mock('os.path.exists', returns=True)
>>> delete_file('/tmp/myfile.txt')
Called os.path.exists('/tmp/myfile.txt')
Called os.remove('/tmp/myfile.txt')
>>> minimock.restore()
pre: os.path.exists(path)
"""
os.remove(path)
if __name__ == '__main__':
setup()
import doctest
doctest.testmod()
When I run the above file standalone, the tests pass:
$ python contracttest.py -v
Trying:
import minimock
Expecting nothing
ok
Trying:
minimock.mock('os.remove')
Expecting nothing
ok
Trying:
minimock.mock('os.path.exists', returns=True)
Expecting nothing
ok
Trying:
delete_file('/tmp/myfile.txt')
Expecting:
Called os.path.exists('/tmp/myfile.txt')
Called os.remove('/tmp/myfile.txt')
ok
Trying:
minimock.restore()
Expecting nothing
ok
2 items had no tests:
__main__
__main__.setup
1 items passed all tests:
5 tests in __main__.delete_file
5 tests in 3 items.
5 passed and 0 failed.
Test passed.
Here it is with nose:
$ nosetests --with-doctest contracttest.py
F
======================================================================
FAIL: Doctest: contracttest.delete_file
----------------------------------------------------------------------
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 2131, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for contracttest.delete_file
File "/Users/lorin/Desktop/contracttest.py", line 10, in delete_file
----------------------------------------------------------------------
File "/Users/lorin/Desktop/contracttest.py", line 17, in contracttest.delete_file
Failed example:
delete_file('/tmp/myfile.txt')
Expected:
Called os.path.exists('/tmp/myfile.txt')
Called os.remove('/tmp/myfile.txt')
Got:
Called os.remove('/tmp/myfile.txt')
----------------------------------------------------------------------
Ran 1 test in 0.055s