I'm rewriting a afl-fuzz (a C application) to Python. Since I don't have enough understanding of its inner workings, I would like to replicate its functionality as close as possible.
I'm trying to run a functional test of a routine that forks Python interpreter, runs execve and if it fails, reports failure to its caller by returning 42. The test runs nice outside of unittest, but fails when put into it:
#!/usr/bin/env python
import os
import sys
import unittest
def run_test():
x = os.fork()
if not x:
sys.exit(42)
waitpid_result, status = os.waitpid(x, os.WUNTRACED)
print(os.WEXITSTATUS(status))
class ForkFunctionalTest(unittest.TestCase):
def test_exercise_fork(self):
run_test()
if __name__ == '__main__':
print('Expecting "42" as output:')
run_test()
print('\nAnd here goes unexpected SystemExit error:')
unittest.main()
Here's how it fails:
Expecting "42" as output:
42
And here goes unexpected SystemExit error:
E
======================================================================
ERROR: test_exercise_fork (__main__.ForkFunctionalTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "afl-fuzz2.py", line 23, in test_exercise_fork
run_test()
File "afl-fuzz2.py", line 15, in run_test
sys.exit(42)
SystemExit: 42
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
1
.
----------------------------------------------------------------------
Ran 1 test in 0.014s
OK
Is there a way to make unittest work with this function without changing run_test? I tried os._exit instead of sys.exit(), but it made the program die in both processes.
It turns out that os._exit actually worked, but in my unit tests I needed to mock it out given that I mocked out os.fork. Silly mistake.
sys.exit() raises a SystemExit class exception, which, if not caught, quits the program. You can try to catch the exception:
def text_exercise_fork(self):
try:
run_test()
except SystemExit as e:
print(e.args[0])
Related
I have a dummy funtion to try exceptions:
def fun(n):
try:
if n <1:
raise ValueError
return 1
except:
pass
In my unit test I use:
import unittest
class TestFibonnacci(unittest.TestCase):
def test_values(self):
self.assertEqual(fun(1),1)
self.assertRaises(ValueError, fun(-1))
However I'm unable to get an answer I actually get:
An exception occurred
E....
ERROR: test_values (main.TestFibonnacci)
Traceback (most recent call last):
The traceback here
TypeError: 'NoneType' object is not callable
Ran 1 tests in 0.001s
What I'm doing wrong?
You are calling fun(-1) immediately, rather than letting self.assertRaises call it, so that it can catch the exception.
You have to pass the function and its arguments separately.
self.assertRaises(ValueError, fun, -1)
Alternatively, you can use assertRaises as a context manager.
with self.assertRaises(ValueError):
fun(-1)
The with statement captures the exception raised by fun and provides it to the __exit__ method of the value returned by assertRaises.
I am learning how to unit test with unittest in Python.
What I am trying to do is to raise a ValueError in the case a negative turn number, and display a message, such as 'turn cannot be negative'.
the code I have written so far is the following:
import unittest
from canvas import Game
class TestPlayer1(unittest.TestCase):
def setUp(self):
# Game objects
self.game_turn_0 = Game()
self.game_turn_5 = Game()
self.game_turn_negative = Game()
# values
self.game_turn_0.turn = 0
self.game_turn_5.turn = 5
self.game_turn_negative = -2
def test_get_turn(self):
self.assertEqual(self.game_turn_0.get_turn(), 0)
self.assertEqual(self.game_turn_5.get_turn(), 5)
with self.assertRaises(ValueError):
print('value error!')
if __name__ == '__main__':
unittest.main()
but the result is not as expected, as you can see:
Testing started at 09:55 ...
C:\Users\oricc\PycharmProjects\practisingDrowing\venv\Scripts\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2.3\helpers\pycharm\_jb_unittest_runner.py" --target test_canvas.TestPlayer1.test_get_turn
Launching unittests with arguments python -m unittest test_canvas.TestPlayer1.test_get_turn in C:\Users\oricc\PycharmProjects\practisingDrowing
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
value error!
Failure
Traceback (most recent call last):
File "C:\Users\oricc\AppData\Local\Programs\Python\Python37-32\lib\unittest\case.py", line 59, in testPartExecutor
yield
File "C:\Users\oricc\AppData\Local\Programs\Python\Python37-32\lib\unittest\case.py", line 628, in run
testMethod()
File "C:\Users\oricc\PycharmProjects\practisingDrowing\test_canvas.py", line 23, in test_get_turn
print('value error!')
File "C:\Users\oricc\AppData\Local\Programs\Python\Python37-32\lib\unittest\case.py", line 203, in __exit__
self._raiseFailure("{} not raised".format(exc_name))
File "C:\Users\oricc\AppData\Local\Programs\Python\Python37-32\lib\unittest\case.py", line 135, in _raiseFailure
raise self.test_case.failureException(msg)
AssertionError: ValueError not raised
Assertion failed
Assertion failed
Ran 1 test in 0.034s
FAILED (failures=1)
Process finished with exit code 1
Assertion failed
Assertion failed
Assertion failed
Assertion failed
I have watched a few videos by now, read the documentation on the python website, and read a few posts. I just can get the grasp.
I mean, I can't get how to use it in my case.
Could anyone explain to me how does this work?
Thanks
You use the assertRaises context manager around an operation that you expect to raise an error. If you look at your test code, can you see a line that should raise an error? I can't.
You are close - you have the general structure. But what you need is
def test_get_turn(self):
self.assertEqual(self.game_turn_0.get_turn(), 0)
self.assertEqual(self.game_turn_5.get_turn(), 5)
with self.assertRaises(ValueError):
self.game_turn_negative.get_turn()
I have only changed the last line. See how that is the operation that you are expecting to raise an exception?
Also, you have a bug in the setUp - you need to set self.game_turn_negative.turn = -2, not self.game_turn_negative = -2. Be sure to fix that too.
Now to check that the test is working, try running the test and see it pass, then change your code so that a negative turn value does not raise an exception, and run the test again. You should see that the test suite fails, with a complaint that a ValueError wasn't raise when it was expected.
with assertRaises is designed around the expectation that the exception will be raised within the with block. In this case the only code running within the with block is print('value error!') - which will never raise a ValueError. This then causes the assertion to fail as a ValueError was not raised.
To solve your problem you'll need to adjust your application so that when an invalid condition is detected, a ValueError is raised. You will then be able to catch the ValueError inside the assertRaises block. It's worth noting that there appears to be a problem with your assignment to self.game_turn_negative. For the game_turn_0 and game_turn_5 values you're assigning an integer value to the .turn attribute, rather than the top level variable. For game_turn_negative you're setting it to Game() and then later on setting it to -2 (rather than setting self.game_turn_negative.turn).
To fix your code, this may work:
import unittest
from canvas import Game
class TestPlayer1(unittest.TestCase):
def setUp(self):
# Game objects
self.game_turn_0 = Game()
self.game_turn_5 = Game()
self.game_turn_negative = Game()
# values
self.game_turn_0.turn = 0
self.game_turn_5.turn = 5
self.game_turn_negative.turn = -2
def test_get_turn(self):
self.assertEqual(self.game_turn_0.get_turn(), 0)
self.assertEqual(self.game_turn_5.get_turn(), 5)
with self.assertRaises(ValueError):
self.game_turn_negative.get_turn()
if __name__ == '__main__':
unittest.main()
class TestPlayer1(unittest.TestCase):
def setUp(self):
# Game objects
self.game_turn_0 = Game()
self.game_turn_5 = Game()
self.game_turn_negative = Game()
# values
self.game_turn_0.turn = 0
self.game_turn_5.turn = 5
self.game_turn_negative = -2
def test_get_turn(self):
self.assertEqual(self.game_turn_0.get_turn(), 0)
self.assertEqual(self.game_turn_5.get_turn(), 5)
with self.assertRaises(ValueError):
print('value error!')
if __name__ == '__main__':
unittest.main()
I would like to test if a proper exception is raised and caught using unittest.
Below is my sample code:
def foo():
try:
raise Exception(ValueError)
except ValueError:
print "raised"
Here is what I would like to do:
self.assertRaises(ValueError, foo)
Since I caught the exception I'm not able to assertRaise the exception.
Am I missing something or is there any other way?
My full code:
#!/usr/bin/python
import unittest
def foo():
try:
raise ValueError
except ValueError, e:
print "caught"
class tester(unittest.TestCase):
def test1(self):
with self.assertRaises(ValueError):
foo()
def test2(self):
self.assertRaises(ValueError, foo)
if __name__ == '__main__':
unittest.main()
No, you cannot test things that only have effects inside the function from outside of it.
What you can do is split the code that raises the exception into its own function, and then test that to check that the exception is raised (with assertRaises). The original function can call that one, and if your normal tests on that work then apparently it catches the exception.
However, whether that is a good idea depends on what your function does. Why does it matter that the function raises an exception at some point and then catches it? Maybe one day you'll switch to some other way to do the same thing, with the same results. Shouldn't your tests keep working then?
If the exception thing is just an implementation detail, you probably shouldn't test it at all.
You can capture stdout and test for a string match. Pytest offer a fixture for that.
#!/usr/bin/python
import unittest
def foo():
try:
raise ValueError
except ValueError, e:
print "caught"
def test_foo(capfd):
foo()
out, err = capfd.readouterr()
assert out == "caught\n"
py.test test.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.9 -- py-1.4.28 -- pytest-2.7.1
rootdir: /tmp, inifile:
collected 1 items
test.py .
=========================== 1 passed in 0.01 seconds ===========================
Try return True or False from your function.
def foo():
try:
#work flow
return True
except ValueError:
#error handle
return False
self.assertFalse(foo())
I have my own exceptions and i want to test farther fields in the ex other then the message.
Reading this thread i tried the idea of using a context. I wrote this generic function
def test_runtime_error(test, exception_type, message, display_parameter, path, callable_obj, *args):
pdb.set_trace()
with test.assertRaises(exception_type) as cx:
callable_obj(*args)
ex = cx.exception
test.assertEqual(ex.message,message)
test.assertEqual(ex.display_parameter,display_parameter)
test.assertEqual(ex.path,path)
The path and display_parameter are my own specific fields. I'm not inventing the wheel here, i took most of it from the source.
I'm using it like that
class ExceptionsTest(unittest.TestCase):
def test_something(self):
data = {"name" : "A"}
obj = MyModel.objects.get(pk=1)
test_runtime_error(self,CustomException, 'message', 'A', [], obj.create, data)
The arguments are passed correctly into the callable_obj. the function raises the expected exception. but right after the execution of callable_obj the function breaks and the exception is not fetched. BTW, when i ran the same code in the test it self it worked fine.
Whats wrong here ?
The issue here appears to be this line:
pdb.set_trace()
If you leave it in, but don't have import pdb, the code below will fail with:
E
======================================================================
ERROR: testRaises (__main__.ExceptionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./except.py", line 22, in testRaises
self.verifyComplexException(MyException, 'asdf', RaiseException, 'asdf')
File "./except.py", line 14, in verifyComplexException
pdb.set_trace()
NameError: global name 'pdb' is not defined
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
which matches your description. If you do add the import pdb line, it will drop into the debugger, which is a completely different behavior that cannot be confused for the exit with E or exit with F status, so it can't be that.
Here's a complete example based on this idea which works as intended (licensed under Apache 2.0; see my repo):
import unittest
class MyException(Exception):
def __init__(self, message):
self.message = message
def RaiseException(message):
raise MyException(message)
class ExceptionTest(unittest.TestCase):
def verifyComplexException(self, exception_class, message, callable, *args):
with self.assertRaises(exception_class) as cm:
callable(*args)
exception = cm.exception
self.assertEqual(exception.message, message)
def testRaises(self):
self.verifyComplexException(MyException, 'asdf', RaiseException, 'asdf')
if __name__ == '__main__':
unittest.main()
I am having little trouble using the python setUpClass.
For example consider the following case
class MyTest(unittest.case.TestCase):
#classmethod
def setUpClass(cls):
print "Test setup"
try:
1/0
except:
raise
#classmethod
def tearDownClass(cls):
print "Test teardown"
A couple of questions
Is the above code the right way to handle test setUpClass exceptions (by raising it so that the python unittest can take care of it), there are fail(), skip() methods, but those can only be used by test instances and not the test classes.
When there is a setUpClass exception, how can we ensure that tearDownClass runs (unittest doesn't run it, should we manualy call it).
You can call tearDownClass on an exception as Jeff points it out, but you may also implements the __del__(cls) method :
import unittest
class MyTest(unittest.case.TestCase):
#classmethod
def setUpClass(cls):
print "Test setup"
try:
1/0
except:
raise
#classmethod
def __del__(cls):
print "Test teardown"
def test_hello(cls):
print "Hello"
if __name__ == '__main__':
unittest.main()
Will have the following output :
Test setup
E
======================================================================
ERROR: setUpClass (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "my_test.py", line 8, in setUpClass
1/0
ZeroDivisionError: integer division or modulo by zero
----------------------------------------------------------------------
Ran 0 tests in 0.000s
FAILED (errors=1)
Test teardown
Note : you should be aware that the __del__ method will be called at the end of the program execution, which is maybe not what you want if you have a more than one test class.
Hope it helps
The best option would be is to add handler for the except which calls tearDownClass and re-raise exception.
#classmethod
def setUpClass(cls):
try:
super(MyTest, cls).setUpClass()
# setup routine...
except Exception: # pylint: disable = W0703
super(MyTest, cls).tearDownClass()
raise
import contextlib
class MyTestCase(unitest.TestCase):
#classmethod
def setUpClass(cls):
with contextlib.ExitStack() as stack:
# ensure teardown is called if error occurs
stack.callback(cls.tearDownClass)
# Do the things here!
# remove callback at the end if no error found
stack.pop_all()
use tearDownModule. It should be called after setUpClass runs.