We Keep Coding

Execute python script, get its print and logging output and assert unittest

Edit: Thanks #eemz for the idea to redesign the structure and use from unittest.mock import patch, but the problem persists.
So I just recently stumbled into unittest and I have a program which I normally start like this python run.py -config /path/to/config.file -y. I wanted to write a simple test in a separate test.py file: Execute the script, pass the mentioned arguments and get all of its output. I pass a prepared config file which is missing certain things, so the run.py will break and exactly log this error using logging.error: "xyz was missing in Config file!" (see example below). I'll get a few words from print() and then the logging instance kicks in and handles from there on. How do I get its output so I can check it? Feel free to rewrite this, as I'm still learning, please bear with me.
Simplified example:
run.py
import logging
def run(args):
< args.config = /path/to/config.file >
cnfg = Config(args.config)
cnfg.logger.info("Let's start with the rest of the code!") # This is NOT in 'output' of the unittest
< code >
if __name__ == "__main__":
print("Welcome! Starting execution.") # This is in 'output' of the unittest
< code to parse arguments 'args' >
run(args)
Config.py
import logging
class Config:
def __init__(self):
print("Creating logging instance, hold on ...") # This is in 'output' of the unittest
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler()
logger.addHandler(console_handler)
logger.info("Logging activated, let's go!") # This is NOT in 'output' of the unittest
self.logger = logger
if xyz not in config:
self.logger.error("xyz was missing in Config file!") # This is NOT in 'output' of the unittest
exit(1)
test.py
import unittest
from unittest.mock import patch
class TestConfigs(unittest.TestCase):
def test_xyz(self):
with patch('sys.stdout', new=StringIO()) as capture:
with self.assertRaises(SystemExit) as cm:
run("/p/to/f/missing/xyz/f", "", False, True)
output = capture.getvalue().strip()
self.assertEqual(cm.exception.code, 1)
# Following is working, because the print messages are in output
self.assertTrue("Welcome! Starting execution." in output)
# Following is NOT working, because the logging messages are not in output
self.assertTrue("xyz was missing in Config file!" in output)
if __name__ == "__main__":
unittest.main()

I would restructure run.py like this:
import logging
def main():
print("Welcome! Starting execution.")
Etc etc
if __name__ == "__main__":
main()
Then you can call the function run.main() in your unit test rather than forking a subprocess.
from io import StringIO
from unittest.mock import patch
import sys
import run
class etc etc
def test_run etc etc:
with patch('sys.stdout', new=StringIO()) as capture:
sys.argv = [‘run.py’, ‘-flag’, ‘-flag’, ‘-flag’]
run.main()
output = capture.getvalue().strip()
assert output == <whatever you expect it to be>
If you’re new to unit testing then you might not have seen mocks before. Effectively I am replacing stdout with a fake one to capture everything that gets sent there, so that I can pull it out later into the variable output.
In fact a second patch around sys.argv would be even better because what I’m doing here, an assignment to the real argv, will actually change it which will affect subsequent tests in the same file.

I ended up instantiating the logger of the main program with a specific name, so I could get the logger in the test.py again and assert that the logger was called with a specific text. I didn't know that I could just get the logger by using logging.getLogger("name") with the same name. Simplified example:
test.py
import unittest
from run import run
from unittest.mock import patch
main_logger = logging.getLogger("main_tool")
class TestConfigs(unittest.TestCase):
def test_xyz(self):
with patch('sys.stdout', new=StringIO()) as capture, \
self.assertRaises(SystemExit) as cm, \
patch.object(main_logger , "info") as mock_log1, \
patch.object(main_logger , "error") as mock_log2:
run("/path/to/file/missing/xyz.file")
output = capture.getvalue().strip()
self.assertTrue("Creating logging instance, hold on ..." in output)
mock_log1.assert_called_once_with("Logging activated, let's go!")
mock_log2.assert_called_once_with("xyz was missing in Config file!")
self.assertEqual(cm.exception.code, 1)
if __name__ == "__main__":
unittest.main()
run.py
def run(path: str):
cnfg = Config(path)
< code >
if __name__ == "__main__":
< code to parse arguments 'args' >
path = args.file_path
run(path)
Config.py
import logging
class Config:
def __init__(self, path: str):
print("Creating logging instance, hold on ...")
logger = logging.getLogger("main_tool")
console_handler = logging.StreamHandler()
logger.addHandler(console_handler)
logger.info("Logging activated, let's go!")
self.logger = logger
# Load file, simplified
config = load(path)
if xyz not in config:
self.logger.error("xyz was missing in Config file!")
exit(1)
This method seems to be very complicated and I got to this point by reading through a lot of other posts and the docs. Maybe some one knows a better way to achieve this.

Recognizing module level tests

How can I get test_greet to run in the below; note: test_one(when uncommented) is seen and run by the test runner; to be specific, I want the line unittest.main() to correctly pick up the module level test (test_greet).
import unittest
#class MyTests(unittest.TestCase):
# def test_one(self):
# assert 1==2
def test_greet():
assert 1==3
if __name__=="__main__":
unittest.main()

Let's say i have a file called MyTests.py as below:
import unittest
class MyTests(unittest.TestCase):
def test_greet(self):
self.assertEqual(1,3)
Then:
Open a CMD in the folder that MyTests.py exists
Run python -m unittest MyTests
Please note that, all your tests must have the test_ otherwise, it will not be run.

Python 3 unittest: How to extract results of tests?

I am using Python's (3.4.1) unittest module for my unit tests.
I load all my testing module files using imports and then run unittest.main():
import unittest
import testing_module1
import testing_module2
# [...]
if __name__ == '__main__':
unittest.main()
This works perfectly for me as it is simple and respect the command line arguments I use to control verbosity or which test(s) to run.
I want to continue to output the same information, but I would like to generate an XML file from the results. I tried xmlrunner (https://github.com/xmlrunner/unittest-xml-reporting/) but:
it does not output as much info to stdout as the standard runner;
it uses a specific format of the XML that doesn't suites me.
I would like to generate the XML (I don't mind doing it manually) with the format I need but with minimal change to how the tests are run.
What are my options?
I could write my own TestRunner but I don't want to re-write everything, I just want to add extra output to the actual runner with minimal code change.
I could inherit unittest.TextTestRunner but I fear that adding XML output to it would require re-writing every methods, loosing the advantage of inheritance in the first place.
I could try to extract the test results after the call to unittest.main() and parse it. The problem here is that unittest.main() seems to exit when it's done so any code after it is not executed.
Any suggestion?
Thanks!

I ended up writing two new classes, inheriting from unittest.TextTestResult and unittest.TextTestRunner. That way, I could run main like that:
unittest.main(testRunner=xmlrunner.XMLTestRunner(...))
I overloaded unittest.TextTestRunner's __init__ and those from unittest.TextTestResult:
addSuccess()
addError()
addFailure()
addSubTest()
For example:
def addSuccess(self, test):
super().addSuccess(test)
[... store the test into list, dictionary, whatever... ]
Since these add*() functions are called with the actual test, I can store them in a global list and parse them at the end of my XMLTestRunner.run():
def run(self, test):
result = super().run(test)
self.save_xml_report(result)
return result
Note that these functions are normally defined in /usr/lib/python3.4/unittest/runner.py.
Warning note: By using an actual object passed unittest.main()'s testRunner argument as shown, the command line arguments given when launching python are ignored. For example, increasing verbose level with -v argument is ignored. This is because the TestProgram class, defined in /usr/lib/python3.4/unittest/main.py, detects if unittest.main() was run with testRunner being a class or an object (see runTests() near the end of the file). If you give just a class like that:
unittest.main(testRunner=xmlrunner.XMLTestRunner)
then command line arguments are parsed. But you pass an instantiated object (like I need), runTests() will just use it as is. I thus had to parse arguments myself in my XMLTestRunner.__init__():
# Similar to what /usr/lib/python3.4/unittest/main.py's TestProgram._getParentArgParser() does.
import argparse
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument('-v', '--verbose', dest='verbosity',
action='store_const', const=2, default=1, # Add default=1, not present in _getParentArgParser()
help='Verbose output')
parser.add_argument('-q', '--quiet', dest='verbosity',
action='store_const', const=0,
help='Quiet output')
parser.add_argument('-f', '--failfast', dest='failfast',
action='store_true',
help='Stop on first fail or error')
parser.add_argument('-c', '--catch', dest='catchbreak',
action='store_true',
help='Catch ctrl-C and display results so far')
parser.add_argument('-b', '--buffer', dest='buffer',
action='store_true',
help='Buffer stdout and stderr during tests')

How does this work for you. Capture the output of unittest, which goes to sys.stderr, in a StringIO. Continue after unittest.main by adding `exit=False'. Read the captured output and process as you want. Proof of concept:
import contextlib
import io
import sys
import unittest
class Mytest(unittest.TestCase):
def test_true(self):
self.assertTrue(True)
#contextlib.contextmanager
def err_to(file):
old_err = sys.stderr
sys.stderr = file
yield
sys.stderr = old_err
if __name__ == '__main__':
result = io.StringIO()
with err_to(result):
unittest.main(exit=False)
result.seek(0)
print(result.read())
This prints (to sys.stdout)
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Note: contextlib has redirect_stdout, but not redirect_stderr. The above is simpler that the contextlib code. The above assumes that there are no exceptions not caught by unittest. See the contextlib.contextmanager doc for adding try: except: finally. I leave that to you.

I have faced the same issue with catching FAIL events from unittest lib. Following big_gie's answer, this code appeared:
File testFileName_1.py
import unittest
class TestClassToTestSth(unittest.TestCase):
def test_One(self):
self.AssertEqual(True, False, 'Hello world')
import unittest
from io import StringIO
import testFileName_1
def suites():
return [
# your testCase classes, for example
testFileName_1.TestClassToTestSth,
testFileName_445.TestClassToTestSomethingElse,
]
class TextTestResult(unittest.TextTestResult):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.slack = Slack('data-engineering-tests')
def addFailure(self, test, err):
super().addFailure(test, err)
# Whatever you want here
print(err, test)
print(self.failures)
class TextTestRunner(unittest.TextTestRunner):
resultclass = TextTestResult
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
loader = unittest.TestLoader()
suite = unittest.TestSuite()
stream = StringIO()
for test_case in suites():
suite.addTests(loader.loadTestsFromTestCase(test_case))
runner = TextTestRunner(stream=stream)
result = runner.run(suite)
stream.seek(0)
print(stream.read())

Run Python unittest so that nothing is printed if successful, only AssertionError() if fails

I have a test module in the standard unittest format
class my_test(unittest.TestCase):
def test_1(self):
[tests]
def test_2(self):
[tests]
etc....
My company has a proprietary test harness that will execute my module as a command line script, and which will catch any errors raised by my module, but requires that my module be mute if successful.
So, I am trying to find a way to run my test module naked, so that if all my tests pass then nothing is printed to the screen, and if a test fails with an AssertionError, that error gets piped through the standard Python error stack (just like any other error would in a normal Python script.)
The docs advocate using the unittest.main() function to run all the tests in a given module like
if __name__ == "__main__":
unittest.main()
The problem is that this wraps the test results in unittest's harness, so that even if all tests are successful, it still prints some fluff to the screen, and if there is an error, it's not simply dumped as a usual python error, but also dressed in the harness.
I've tried redirecting the output to an alternate stream using
with open('.LOG','a') as logf:
suite = unittest.TestLoader().loadTestsFromTestCase(my_test)
unittest.TextTestRunner(stream = logf).run(suite)
The problem here is that EVERYTHING gets piped to the log file (including all notice of errors). So when my companies harness runs the module, it complete's successfully because, as far as it can tell, no errors were raised (because they were all piped to the log file).
Any suggestions on how I can construct a test runner that suppresses all the fluff, and pipes errors through the normal Python error stack? As always, if you think there is a better way to approach this problem, please let me know.
EDIT:
Here is what I ended up using to resolve this. First, I added a "get_test_names()" method to my test class:
class my_test(unittest.TestCase):
etc....
#staticmethod
def get_test_names():
"""Return the names of all the test methods for this class."""
test_names = [ member[0] for memeber in inspect.getmembers(my_test)
if 'test_' in member[0] ]
Then I replaced my call to unittest.main() with the following:
# Unittest catches all errors raised by the test cases, and returns them as
# formatted strings inside a TestResult object. In order for the test
# harness to catch these errors they need to be re-raised, and so I am defining
# this CompareError class to do that.
# For each code error, a CompareError will be raised, with the original error
# stack as the argument. For test failures (i.e. assertion errors) an
# AssertionError is raised.
class CompareError(Exception):
def __init__(self,err):
self.err = err
def __str__(self):
return repr(self.err)
# Collect all tests into a TestSuite()
all_tests = ut.TestSuite()
for test in my_test.get_test_names():
all_tests.addTest(my_test(test))
# Define a TestResult object and run tests
results = ut.TestResult()
all_tests.run(results)
# Re-raise any script errors
for error in results.errors:
raise CompareError(error[1])
# Re-raise any test failures
for failure in results.failures:
raise AssertionError(failure[1])

I came up with this. If you are able to change the command line you might remove the internal io redirection.
import sys, inspect, traceback
# redirect stdout,
# can be replaced by testharness.py > /dev/null at console
class devnull():
def write(self, data):
pass
f = devnull()
orig_stdout = sys.stdout
sys.stdout = f
class TestCase():
def test_1(self):
print 'test_1'
def test_2(self):
raise AssertionError, 'test_2'
def test_3(self):
print 'test_3'
if __name__ == "__main__":
testcase = TestCase()
testnames = [ t[0] for t in inspect.getmembers(TestCase)
if t[0].startswith('test_') ]
for testname in testnames:
try:
getattr(testcase, testname)()
except AssertionError, e:
print >> sys.stderr, traceback.format_exc()
# restore
sys.stdout = orig_stdout

Log Unittest output to a text file

I am trying to log the output of tests to a text file. I am using the unittest module and want to log results into a text file instead of the screen. I have some script here to explain what has been tryied so far. This is the test script.
import unittest, sys
class TestOne(unittest.TestCase):
def setUp(self):
self.var = 'Tuesday'
def tearDown(self):
self.var = None
class BasicTestOne(TestOne):
def runTest(self):
TestOne.setUp(self)
self.assertEqual(self.var, 'Tuesday')
class AbsoluteMoveTestSuite(unittest.TestSuite):
# Tests to be tested by test suite
def makeAbsoluteMoveTestSuite():
suite = unittest.TestSuite()
suite.addTest(TestOne("BasicTestOne"))
return suite
def suite():
return unittest.makeSuite(TestOne)
if __name__ == '__main__':
unittest.main()
I have added this to the file but it doesn't seem to work.
log_file = 'log_file.txt'
sys.stout = sys.sterr = open(log_file, 'w')
return suite >> sys.stout
and also:
log_file = 'log_file.txt'
return suite >> open(log_file, 'w')
I have tried several different versions of this command.
if __name__ == '__main__':
unittest.main() >> open(log_file, 'w')
I have tried this. I want the log file to be stored and created inside the python script. I do not want to have to call python tests.py >> log_file.txt.
Thanks for any help

You can pass the text runner into the main method. The text runner must be set up to write to a file rather than the std.err as it wraps the stream in a decorator. The following worked for me in python 2.6
if __name__ == '__main__':
log_file = 'log_file.txt'
with open(log_file, "w") as f:
runner = unittest.TextTestRunner(f)
unittest.main(testRunner=runner)