We Keep Coding

how to get the traceback of failed test (unittest) in the tearDown() function

I want to save all the traceback of all the exceptions from every failed test I run to an external file.
Instead of using try and except in every test i would like to use the tearDown of unittest so it will be more generic.
something like:
import traceback
import unittest
class SomeTestCase(unittest.TestCase):
def setUp(self):
pass
def test_some_test(self):
self.assertTrue(False)
def tearDown(self):
with open(logger.txt, 'a') as doc:
doc.write(traceback.format_exc())
the problem is that the exception that you get in the test_some_test cant be called with traceback in the teardown (the tracebacke return None)
any suggestion?

So after I went over a big piece of the unittest code I found a solution!
The TestCase class has an _outcome attribute.
The _outcome has a list type attribute called errors.
errors[0][1] is a tuple that is exactly like the sys.exc_info() output that contains the traceback.
So now that I have this tuple that I can use in traceback.format_exception() just like traceback.format_exc() uses it and problem solved.
def tearDown(self):
try:
etype, value, tb = self._outcome.errors[0][1]
trace = ''.join(traceback.format_exception(etype=etype, value=value, tb=tb, limit=None))
date = '{date}\n'.format(date=str(datetime.datetime.now()))
name = '\n' + self._testMethodName + '-\n'
with open(self.logger, 'a') as doc:
doc.write(name + date + trace)
except:
pass

Can I work around UnitTest's SystemExit handler?

I'm rewriting a afl-fuzz (a C application) to Python. Since I don't have enough understanding of its inner workings, I would like to replicate its functionality as close as possible.
I'm trying to run a functional test of a routine that forks Python interpreter, runs execve and if it fails, reports failure to its caller by returning 42. The test runs nice outside of unittest, but fails when put into it:
#!/usr/bin/env python
import os
import sys
import unittest
def run_test():
x = os.fork()
if not x:
sys.exit(42)
waitpid_result, status = os.waitpid(x, os.WUNTRACED)
print(os.WEXITSTATUS(status))
class ForkFunctionalTest(unittest.TestCase):
def test_exercise_fork(self):
run_test()
if __name__ == '__main__':
print('Expecting "42" as output:')
run_test()
print('\nAnd here goes unexpected SystemExit error:')
unittest.main()
Here's how it fails:
Expecting "42" as output:
42
And here goes unexpected SystemExit error:
E
======================================================================
ERROR: test_exercise_fork (__main__.ForkFunctionalTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "afl-fuzz2.py", line 23, in test_exercise_fork
run_test()
File "afl-fuzz2.py", line 15, in run_test
sys.exit(42)
SystemExit: 42
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
1
.
----------------------------------------------------------------------
Ran 1 test in 0.014s
OK
Is there a way to make unittest work with this function without changing run_test? I tried os._exit instead of sys.exit(), but it made the program die in both processes.

It turns out that os._exit actually worked, but in my unit tests I needed to mock it out given that I mocked out os.fork. Silly mistake.

sys.exit() raises a SystemExit class exception, which, if not caught, quits the program. You can try to catch the exception:
def text_exercise_fork(self):
try:
run_test()
except SystemExit as e:
print(e.args[0])

Unit testing tornado applications: How to improve the display of error messages

I am using unittest to test a tornado app having several handlers, one of which raises an exception. If I run the following test code with python test.py:
# test.py
import unittest
import tornado.web
import tornado.testing
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write('Hello World') # handler works correctly
class HandlerWithError(tornado.web.RequestHandler):
def get(self):
raise Exception('Boom') # handler raises an exception
self.write('Hello World')
def make_app():
return tornado.web.Application([
(r'/main/', MainHandler),
(r'/error/', HandlerWithError),
])
class TornadoTestCase(tornado.testing.AsyncHTTPTestCase):
def get_app(self):
return make_app()
def test_main_handler(self):
response = self.fetch('/main/')
self.assertEqual(response.code, 200) # test should pass
def test_handler_with_error(self):
response = self.fetch('/error/')
self.assertEqual(response.code, 200) # test should fail with error
if __name__ == '__main__':
unittest.main()
the test output looks like:
ERROR:tornado.application:Uncaught exception GET /error/ (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:36590', method='GET', uri='/error/', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Connection': 'close', 'Host': 'localhost:3
6590', 'Accept-Encoding': 'gzip'})
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1332, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "test.py", line 13, in get
raise Exception('Boom') # handler raises an exception
Exception: Boom
ERROR:tornado.access:500 GET /error/ (127.0.0.1) 19.16ms
F.
======================================================================
FAIL: test_handler_with_error (__main__.TornadoTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/testing.py", line 118, in __call__
result = self.orig_method(*args, **kwargs)
File "test.py", line 33, in test_handler_with_error
self.assertEqual(response.code, 200) # test should fail with error
AssertionError: 500 != 200
----------------------------------------------------------------------
Ran 2 tests in 0.034s
FAILED (failures=1)
However, I would expect unittest to report an Error for the second test, instead of a failing assertion. Moreover, the fact that the traceback for the 'Boom' exception appears before the unittest test report and does not include a reference to the failing test function makes it difficult to find the source of the exception.
Any suggestions how to handle this situation?
Thanks in advance!
EDIT
What I find unexpected is the fact that test_handler_with_error actually arrives at making the assertEqual assertion, instead of throwing the error. For example, the following code does not execute the self.assertEqualstatement, and consequently reports an ERROR instead of a FAIL in the test output:
# simple_test.py
import unittest
def foo():
raise Exception('Boom')
return 'bar'
class SimpleTestCase(unittest.TestCase):
def test_failing_function(self):
result = foo()
self.assertEqual(result, 'bar')
if __name__ == '__main__':
unittest.main()

You can disable logging and only the test reports will appear:
logging.disable(logging.CRITICAL)
You can put that for example in
created TestCase subclass
test runner
More info How can I disable logging while running unit tests in Python Django?
Keep in mind that CI/CD systems actually use normalized report e.g. junit and then present it in more readable/elegant way - more info:
Python script to generate JUnit report from another testing result
How to output coverage XML with nosetests?

This is expected behavior. Your test itself asserts that the return code is HTTP 200, and since this is a formal assert that is false, the outcome is a "failure" instead of an "error". You can suppress logs as mentioned in kwaranuk's answer, but then you lose the information about what actually caused the HTTP 500 error.
Why does your code reach the assert, instead of throwing? It's because your test code does not call HandlerWithError.get. Your test code begins an asynchronous HTTP GET operation with an HTTP client provided by the AsyncHTTPTestCase class. (Check the source code of that class for details.) The event loop runs until HandlerWithError.get receives the request over a localhost socket, and responds on that socket with an HTTP 500. When HandlerWithError.get fails, it doesn't raise an exception into your test function, any more than a failure at Google.com would raise an exception: it merely results in an HTTP 500.
Welcome to the world of async! There's no easy way to neatly associate the assertion error and the traceback from HandlerWithError.get().

Unable to pass assertRaises test in Python

So, I have the most trivial in the world example. This is my class to be tested:
# My_Class.py
class My_Class(object):
#staticmethod
def doit(name, params):
try:
raise Exception("This is my error message")
except Exception:
print("Exception: I raised Exception")
And this is the tester itself:
# test.py
import unittest
from My_Class import My_Class
class Test_MyClass(unittest.TestCase):
def setUp(self):
self.my_class = My_Class()
def test_my_class(self):
name = "Abrakadabra"
params = {}
self.assertRaises(Exception, self.my_class.doit, name, params)
And this is what I see in the console, when I'm running my test.py:
$ nosetests test.py
F
======================================================================
FAIL: test_my_class (test.Test_MyClass)
----------------------------------------------------------------------
Traceback (most recent call last):
File ....
nose.proxy.AssertionError: Exception not raised by doit
-------------------- >> begin captured stdout << ---------------------
Exception: I raised Exception
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)
It is reaaly iteresting, because it is controversial. On the one hand the test says that "Exception not raised by doit", but one line below it clearly prints out a message from the Exception block. So, what I'm doing wrong here??? Thanks!

To directly answer your question, the reason why you are getting that message is because with this assertion:
self.assertRaises(Exception, self.my_class.doit, name, params)
You are testing to make sure an exception was raised. But your try/except suppresses this. If you actually remove your try/except your test will in fact pass, because now your method will raise.
Since you do not want to do this, what you should be doing instead is testing the behaviour of your method when an exception is raised. Ultimately, you want to make sure that your print method is called in your except. I have put together an example below to help understand this.
Keeping in mind what #user2357112 mentioned, which is very important to keep in mind when unittesting, here is an example to help expand on that to provide a practical use for what you are trying to do:
Let us just put together some method:
def some_method():
pass
We will now put this in to your staticmethod you defined as such:
# My_Class.py
class My_Class(object):
#staticmethod
def doit(name, params):
try:
some_method()
except Exception:
print("Exception: I raised Exception")
So now, when it comes to your unittesting, you want to test the behaviour of your method doit. With that in mind, what you will do in this case, is test that some_method will raise an exception and you will validate how your doit method behaves to that exception being raised.
At this point, I suggest taking a look at the documentation behind unittest and mock to get more familiar with what you can do with your testing, but here is an example using mock patching to test the behaviour of your code if an exception is being raised:
#patch('builtins.print')
#patch('__main__.some_method')
def test_my_class(self, m_some_method, m_print):
name = "Abrakadabra"
params = {}
# have the side_effect raise the exception when some_method is called in doit
m_some_method.side_effect = Exception()
self.my_class.doit(name, params)
# check to make sure you caught the exception by checking print was called
self.assertEqual(m_print.call_count, 1)
When you put it all together, the following is functional code that I ran on my end that you can play around with to understand what is happening:
def some_method():
pass
# My_Class.py
class My_Class(object):
#staticmethod
def doit(name, params):
try:
some_method()
except Exception:
print("Exception: I raised Exception")
# test.py
import unittest
from mock import patch
class Test_MyClass(unittest.TestCase):
def setUp(self):
self.my_class = My_Class()
#patch('builtins.print')
#patch('__main__.some_method')
def test_my_class(self, m_some_method, m_print):
name = "Abrakadabra"
params = {}
m_some_method.side_effect = Exception()
self.my_class.doit(name, params)
self.assertEqual(m_print.call_count, 1)
if __name__ == '__main__':
unittest.main()

assertRaises is an assertion about the function's visible behavior, not its internals. It asserts that the stated exception passes out of the function. Any exceptions that are handled inside the function are not assertRaises's concern.

assertRaises failed since there was actually no exception raised. Well, it was raised but handled with except inside the doit() method. The problem is here:
try:
raise Exception("This is my error message")
except Exception:
print("Exception: I raised Exception")
You are raising an exception and then catching it without re-raising. From a caller (assertRaises is the caller in your case) perspective, no errors were thrown during the function call. Re-raising an exception allows a caller to handle an exception as well. Put a raise after the print:
try:
raise Exception("This is my error message")
except Exception:
print("Exception: I raised Exception")
raise # re-raising
Also see Handling Exceptions.

How to fail a python unittest if the setUpClass throws exception

I am having little trouble using the python setUpClass.
For example consider the following case
class MyTest(unittest.case.TestCase):
#classmethod
def setUpClass(cls):
print "Test setup"
try:
1/0
except:
raise
#classmethod
def tearDownClass(cls):
print "Test teardown"
A couple of questions
Is the above code the right way to handle test setUpClass exceptions (by raising it so that the python unittest can take care of it), there are fail(), skip() methods, but those can only be used by test instances and not the test classes.
When there is a setUpClass exception, how can we ensure that tearDownClass runs (unittest doesn't run it, should we manualy call it).

You can call tearDownClass on an exception as Jeff points it out, but you may also implements the __del__(cls) method :
import unittest
class MyTest(unittest.case.TestCase):
#classmethod
def setUpClass(cls):
print "Test setup"
try:
1/0
except:
raise
#classmethod
def __del__(cls):
print "Test teardown"
def test_hello(cls):
print "Hello"
if __name__ == '__main__':
unittest.main()
Will have the following output :
Test setup
E
======================================================================
ERROR: setUpClass (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "my_test.py", line 8, in setUpClass
1/0
ZeroDivisionError: integer division or modulo by zero
----------------------------------------------------------------------
Ran 0 tests in 0.000s
FAILED (errors=1)
Test teardown
Note : you should be aware that the __del__ method will be called at the end of the program execution, which is maybe not what you want if you have a more than one test class.
Hope it helps

The best option would be is to add handler for the except which calls tearDownClass and re-raise exception.
#classmethod
def setUpClass(cls):
try:
super(MyTest, cls).setUpClass()
# setup routine...
except Exception: # pylint: disable = W0703
super(MyTest, cls).tearDownClass()
raise

import contextlib
class MyTestCase(unitest.TestCase):
#classmethod
def setUpClass(cls):
with contextlib.ExitStack() as stack:
# ensure teardown is called if error occurs
stack.callback(cls.tearDownClass)
# Do the things here!
# remove callback at the end if no error found
stack.pop_all()

use tearDownModule. It should be called after setUpClass runs.