We Keep Coding

how to get the traceback of failed test (unittest) in the tearDown() function

I want to save all the traceback of all the exceptions from every failed test I run to an external file.
Instead of using try and except in every test i would like to use the tearDown of unittest so it will be more generic.
something like:
import traceback
import unittest
class SomeTestCase(unittest.TestCase):
def setUp(self):
pass
def test_some_test(self):
self.assertTrue(False)
def tearDown(self):
with open(logger.txt, 'a') as doc:
doc.write(traceback.format_exc())
the problem is that the exception that you get in the test_some_test cant be called with traceback in the teardown (the tracebacke return None)
any suggestion?

So after I went over a big piece of the unittest code I found a solution!
The TestCase class has an _outcome attribute.
The _outcome has a list type attribute called errors.
errors[0][1] is a tuple that is exactly like the sys.exc_info() output that contains the traceback.
So now that I have this tuple that I can use in traceback.format_exception() just like traceback.format_exc() uses it and problem solved.
def tearDown(self):
try:
etype, value, tb = self._outcome.errors[0][1]
trace = ''.join(traceback.format_exception(etype=etype, value=value, tb=tb, limit=None))
date = '{date}\n'.format(date=str(datetime.datetime.now()))
name = '\n' + self._testMethodName + '-\n'
with open(self.logger, 'a') as doc:
doc.write(name + date + trace)
except:
pass

How to test if the right exception is raised and caught using unit testing?

I would like to test if a proper exception is raised and caught using unittest.
Below is my sample code:
def foo():
try:
raise Exception(ValueError)
except ValueError:
print "raised"
Here is what I would like to do:
self.assertRaises(ValueError, foo)
Since I caught the exception I'm not able to assertRaise the exception.
Am I missing something or is there any other way?
My full code:
#!/usr/bin/python
import unittest
def foo():
try:
raise ValueError
except ValueError, e:
print "caught"
class tester(unittest.TestCase):
def test1(self):
with self.assertRaises(ValueError):
foo()
def test2(self):
self.assertRaises(ValueError, foo)
if __name__ == '__main__':
unittest.main()

No, you cannot test things that only have effects inside the function from outside of it.
What you can do is split the code that raises the exception into its own function, and then test that to check that the exception is raised (with assertRaises). The original function can call that one, and if your normal tests on that work then apparently it catches the exception.
However, whether that is a good idea depends on what your function does. Why does it matter that the function raises an exception at some point and then catches it? Maybe one day you'll switch to some other way to do the same thing, with the same results. Shouldn't your tests keep working then?
If the exception thing is just an implementation detail, you probably shouldn't test it at all.

You can capture stdout and test for a string match. Pytest offer a fixture for that.
#!/usr/bin/python
import unittest
def foo():
try:
raise ValueError
except ValueError, e:
print "caught"
def test_foo(capfd):
foo()
out, err = capfd.readouterr()
assert out == "caught\n"
py.test test.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.9 -- py-1.4.28 -- pytest-2.7.1
rootdir: /tmp, inifile:
collected 1 items
test.py .
=========================== 1 passed in 0.01 seconds ===========================

Try return True or False from your function.
def foo():
try:
#work flow
return True
except ValueError:
#error handle
return False
self.assertFalse(foo())

Expecting Python assertRaises() to fail, but test always passes

I am attempting to get into the practice of TDD so I have been trying to start with Python's unittest module. I have written a test which I fully expect to fail, but it always passes! Below is the test code:
def test_file_not_found_exception(self):
invalid_config = "Testing/invalidExperiment.xml"
experiment_with_invalid_config = MyClass(invalid_config)
self.assertRaises(OSError, experiment_with_invalid_config.run_experiment())
and my class definition is as follows:
class MyClass:
def __init__(self, experiments, output_directory = ".")
self._experiments = experiments
self._output_directory = output_directory
def run_experiment(self):
try:
x = 2 # dummy statement
except OSError:
print "Experiment file not found"
except:
print "Unexpected Error"
I figure that the try block should always execute correctly so I am at a loss as to why my unittest keep passing. I am very new to OOP in Python so I may be making a very obvious mistake... Also, if I catch the exception in the try-except block of the run_experiment() method, should the assertRaises() call ever work?

You are calling the function MyClass.run_experiment and passing it's output to self.assertRaises, for which reason it does not properly register an error when it fails to raise.
Incorrect Example:
import unittest
class MyClass(object):
def __init__(self, config):
pass
def run_experiment(self): pass
class MainTest(unittest.TestCase):
def test_file_not_found_exception(self):
invalid_config = "Testing/invalidExperiment.xml"
experiment_with_invalid_config = MyClass(invalid_config)
self.assertRaises(OSError, experiment_with_invalid_config.run_experiment())
if __name__ == "__main__":
unittest.main()
Correct:
with self.assertRaises(OSError):
experiment_with_invalid_config.run_experiment()
Or
self.assertRaises(OSError, experiment_with_invalid_config.run_experiment)

Run Python unittest so that nothing is printed if successful, only AssertionError() if fails

I have a test module in the standard unittest format
class my_test(unittest.TestCase):
def test_1(self):
[tests]
def test_2(self):
[tests]
etc....
My company has a proprietary test harness that will execute my module as a command line script, and which will catch any errors raised by my module, but requires that my module be mute if successful.
So, I am trying to find a way to run my test module naked, so that if all my tests pass then nothing is printed to the screen, and if a test fails with an AssertionError, that error gets piped through the standard Python error stack (just like any other error would in a normal Python script.)
The docs advocate using the unittest.main() function to run all the tests in a given module like
if __name__ == "__main__":
unittest.main()
The problem is that this wraps the test results in unittest's harness, so that even if all tests are successful, it still prints some fluff to the screen, and if there is an error, it's not simply dumped as a usual python error, but also dressed in the harness.
I've tried redirecting the output to an alternate stream using
with open('.LOG','a') as logf:
suite = unittest.TestLoader().loadTestsFromTestCase(my_test)
unittest.TextTestRunner(stream = logf).run(suite)
The problem here is that EVERYTHING gets piped to the log file (including all notice of errors). So when my companies harness runs the module, it complete's successfully because, as far as it can tell, no errors were raised (because they were all piped to the log file).
Any suggestions on how I can construct a test runner that suppresses all the fluff, and pipes errors through the normal Python error stack? As always, if you think there is a better way to approach this problem, please let me know.
EDIT:
Here is what I ended up using to resolve this. First, I added a "get_test_names()" method to my test class:
class my_test(unittest.TestCase):
etc....
#staticmethod
def get_test_names():
"""Return the names of all the test methods for this class."""
test_names = [ member[0] for memeber in inspect.getmembers(my_test)
if 'test_' in member[0] ]
Then I replaced my call to unittest.main() with the following:
# Unittest catches all errors raised by the test cases, and returns them as
# formatted strings inside a TestResult object. In order for the test
# harness to catch these errors they need to be re-raised, and so I am defining
# this CompareError class to do that.
# For each code error, a CompareError will be raised, with the original error
# stack as the argument. For test failures (i.e. assertion errors) an
# AssertionError is raised.
class CompareError(Exception):
def __init__(self,err):
self.err = err
def __str__(self):
return repr(self.err)
# Collect all tests into a TestSuite()
all_tests = ut.TestSuite()
for test in my_test.get_test_names():
all_tests.addTest(my_test(test))
# Define a TestResult object and run tests
results = ut.TestResult()
all_tests.run(results)
# Re-raise any script errors
for error in results.errors:
raise CompareError(error[1])
# Re-raise any test failures
for failure in results.failures:
raise AssertionError(failure[1])

I came up with this. If you are able to change the command line you might remove the internal io redirection.
import sys, inspect, traceback
# redirect stdout,
# can be replaced by testharness.py > /dev/null at console
class devnull():
def write(self, data):
pass
f = devnull()
orig_stdout = sys.stdout
sys.stdout = f
class TestCase():
def test_1(self):
print 'test_1'
def test_2(self):
raise AssertionError, 'test_2'
def test_3(self):
print 'test_3'
if __name__ == "__main__":
testcase = TestCase()
testnames = [ t[0] for t in inspect.getmembers(TestCase)
if t[0].startswith('test_') ]
for testname in testnames:
try:
getattr(testcase, testname)()
except AssertionError, e:
print >> sys.stderr, traceback.format_exc()
# restore
sys.stdout = orig_stdout

Python Unit Testing: Automatically Running the Debugger when a test fails

Is there a way to automatically start the debugger at the point at which a unittest fails?
Right now I am just using pdb.set_trace() manually, but this is very tedious as I need to add it each time and take it out at the end.
For Example:
import unittest
class tests(unittest.TestCase):
def setUp(self):
pass
def test_trigger_pdb(self):
#this is the way I do it now
try:
assert 1==0
except AssertionError:
import pdb
pdb.set_trace()
def test_no_trigger(self):
#this is the way I would like to do it:
a=1
b=2
assert a==b
#magically, pdb would start here
#so that I could inspect the values of a and b
if __name__=='__main__':
#In the documentation the unittest.TestCase has a debug() method
#but I don't understand how to use it
#A=tests()
#A.debug(A)
unittest.main()

I think what you are looking for is nose. It works like a test runner for unittest.
You can drop into the debugger on errors, with the following command:
nosetests --pdb

import unittest
import sys
import pdb
import functools
import traceback
def debug_on(*exceptions):
if not exceptions:
exceptions = (AssertionError, )
def decorator(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except exceptions:
info = sys.exc_info()
traceback.print_exception(*info)
pdb.post_mortem(info[2])
return wrapper
return decorator
class tests(unittest.TestCase):
#debug_on()
def test_trigger_pdb(self):
assert 1 == 0
I corrected the code to call post_mortem on the exception instead of set_trace.

Third party test framework enhancements generally seem to include the feature (nose and nose2 were already mentioned in other answers). Some more:
pytest supports it.
pytest --pdb
Or if you use absl-py's absltest instead of unittest module:
name_of_test.py --pdb_post_mortem

A simple option is to just run the tests without result collection and letting the first exception crash down the stack (for arbitrary post mortem handling) by e.g.
try: unittest.findTestCases(__main__).debug()
except:
pdb.post_mortem(sys.exc_info()[2])
Another option: Override unittest.TextTestResult's addError and addFailure in a debug test runner for immediate post_mortem debugging (before tearDown()) - or for collecting and handling errors & tracebacks in an advanced way.
(Doesn't require extra frameworks or an extra decorator for test methods)
Basic example:
import unittest, pdb
class TC(unittest.TestCase):
def testZeroDiv(self):
1 / 0
def debugTestRunner(post_mortem=None):
"""unittest runner doing post mortem debugging on failing tests"""
if post_mortem is None:
post_mortem = pdb.post_mortem
class DebugTestResult(unittest.TextTestResult):
def addError(self, test, err):
# called before tearDown()
traceback.print_exception(*err)
post_mortem(err[2])
super(DebugTestResult, self).addError(test, err)
def addFailure(self, test, err):
traceback.print_exception(*err)
post_mortem(err[2])
super(DebugTestResult, self).addFailure(test, err)
return unittest.TextTestRunner(resultclass=DebugTestResult)
if __name__ == '__main__':
##unittest.main()
unittest.main(testRunner=debugTestRunner())
##unittest.main(testRunner=debugTestRunner(pywin.debugger.post_mortem))
##unittest.findTestCases(__main__).debug()

To apply #cmcginty's answer to the successor nose 2 (recommended by nose available on Debian-based systems via apt-get install nose2), you can drop into the debugger on failures and errors by calling
nose2
in your test directory.
For this, you need to have a suitable .unittest.cfg in your home directory or unittest.cfg in the project directory; it needs to contain the lines
[debugger]
always-on = True
errors-only = False

To address the comment in your code "In the documentation the unittest.TestCase has a debug() method but I don't understand how to use it", you can do something like this:
suite = unittest.defaultTestLoader.loadTestsFromModule(sys.modules[__name__])
suite.debug()
Individual test cases are created like:
testCase = tests('test_trigger_pdb') (where tests is a sub-class of TestCase as per your example). And then you can do testCase.debug() to debug one case.

Here's a built-in, no extra modules, solution:
import unittest
import sys
import pdb
####################################
def ppdb(e=None):
"""conditional debugging
use with: `if ppdb(): pdb.set_trace()`
"""
return ppdb.enabled
ppdb.enabled = False
###################################
class SomeTest(unittest.TestCase):
def test_success(self):
try:
pass
except Exception, e:
if ppdb(): pdb.set_trace()
raise
def test_fail(self):
try:
res = 1/0
#note: a `nosetests --pdb` run will stop after any exception
#even one without try/except and ppdb() does not not modify that.
except Exception, e:
if ppdb(): pdb.set_trace()
raise
if __name__ == '__main__':
#conditional debugging, but not in nosetests
if "--pdb" in sys.argv:
print "pdb requested"
ppdb.enabled = not sys.argv[0].endswith("nosetests")
sys.argv.remove("--pdb")
unittest.main()
call it with python myunittest.py --pdb and it will halt. Otherwise it won't.

Some solution above modifies business logic:
try: # <-- new code
original_code() # <-- changed (indented)
except Exception as e: # <-- new code
pdb.post_mortem(...) # <-- new code
To minimize changes to the original code, we can define a function decorator, and simply decorate the function that's throwing:
def pm(func):
import functools, pdb
#functools.wraps(func)
def func2(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
pdb.post_mortem(e.__traceback__)
raise
return func2
Use:
#pm
def test_xxx(...):
...

Buildt a module with a decorator which post mortems into every type of error except AssertionError. The decorator can be triggered by the logging root level
#!/usr/bin/env python3
'''
Decorator for getting post mortem on errors of a unittest TestCase
'''
import sys
import pdb
import functools
import traceback
import logging
import unittest
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.DEBUG)
def debug_on(log_level):
'''
Function decorator for post mortem debugging unittest functions.
Args:
log_level (int): logging levels coesponding to logging stl module
Usecase:
class tests(unittest.TestCase):
#debug_on(logging.root.level)
def test_trigger_pdb(self):
assert 1 == 0
'''
def decorator(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except BaseException as err:
info = sys.exc_info()
traceback.print_exception(*info)
if log_level < logging.INFO and type(err) != AssertionError:
pdb.post_mortem(info[2])
return wrapper
return decorator
class Debug_onTester(unittest.TestCase):
#debug_on(logging.root.level)
def test_trigger_pdb(self):
assert 1 == 0
if __name__ == '__main__':
unittest.main()